Repository navigation
test(tmachine): add a K3s HA conformance lane with PostgreSQL and mTLS - #4081
Merged
Merged
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Drop the port-forward restart removed by the ClusterIP change, resolve the Service DNS name covered by the server certificate to the ClusterIP, bound each authenticated API wait attempt, and collect diagnostics from every gateway Pod. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
matthewgrossman
force-pushed
the
3529-tmachine-k3s-postgres-mtls/mg
branch
from
October 9, 2026 20:34
b6bd150 to
e772f89
Compare
Add the k3s-ha-tls installer to every core conformance matrix so Branch E2E, Release Dev, Release Tag, and manual Integration Tests cover three gateway replicas sharing PostgreSQL behind mTLS. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
🌿 Preview your docs: https://nvidia-preview-pr-4081.docs.buildwithfern.com/openshell |
TLS is part of the supported HA configuration once gateway replicas share PostgreSQL, so the installer name only needs to distinguish HA from the single-replica k3s lane. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…tgres-mtls/mg Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Add an optional Helm values overlay to the K3s installer so the k3s-ha installer stages PostgreSQL first and installs the three-replica TLS gateway directly, instead of installing a plaintext gateway and upgrading. Drop the redundant unauthenticated-users override and the install-time API wait, which the conformance playbook already performs. Share K3s installer inputs in the tmachine configuration. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
matthewgrossman
marked this pull request as ready for review
October 12, 2026 18:41
matthewgrossman
requested review from
a team,
derekwaynecarr,
mrunalp and
sjenning
as code owners
October 12, 2026 18:41
5 tasks
…tgres-mtls/mg Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
TaylorMutch
previously approved these changes
Oct 12, 2026
matthewgrossman
changed the title
Apply the OpenShell namespace instead of creating it so the staging play tolerates an existing namespace, and wait for the PostgreSQL rollout so gateway Pods do not start crash-looping against a missing database. Reflow and pluralize the K3s lane notes in CI.md. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
TaylorMutch
approved these changes
Oct 12, 2026
3 tasks
7 tasks
7 of 9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
OpenShell's newer testing path is tmachine. Each lane pairs a VM environment with an installer, and the shared conformance suite runs against it on every Branch E2E, Release Dev, and Release Tag run. Until now, tmachine couldn't set up an HA Kubernetes environment: the
k3sinstaller deploys a single plaintext gateway. HA was covered only by the older script-based kind harness (e2e:kubernetes:ha-rebalancing), which runs in plaintext, only when a PR has thetest:e2e-kuberneteslabel, and not in releases.This PR adds that environment for #3529, under the testing-strategy tracker #3954. The new
k3s-hainstaller runs three gateway replicas on shared PostgreSQL, with chart-generated TLS and a CLI that connects over mTLS. The conformance suite now runs against a real multi-replica gateway in every tmachine run.It also gives #3825 somewhere to put its HA tests. #3825 doesn't depend on this PR, but the lane already matches #3825's requirement that PostgreSQL HA run with TLS, so its
kubernetes_ha_operationsscenarios can move here later. The lane has already caught a real HA bug: concurrentsandbox stopconflicts between replicas, fixed by #4395.Related Issue
Closes #4443, a sub-issue of #3529 (HA and failover coverage), which #3954 (testing strategy) tracks.
Depended on #4395 (merged) for a green
sandbox_lifecyclescenario in thek3s-halane; this branch includes it.Changes
openshell-k3s.yamltakes an optional Helm values overlay (openshell_k3s_helm_values_overlay, empty by default). The plaink3slane installs exactly as before.k3s-hainstaller: before the base playbook runs, it applies the pinned PostgreSQL fixture. It then imports the base playbook with the HA values: 3 replicas, a Deployment, the external database, and TLS. The chart is installed once, with no plaintext install followed by an upgrade.allowUnauthenticatedUsersfor its development user.openshell.openshell.svc.cluster.localbut not the ClusterIP, so the installer maps that name to the ClusterIP in the guest's/etc/hosts.openshell-0, so failures under a Deployment are covered.k3s-hais added to the core conformance matrix in Branch E2E, Release Dev, Release Tag, and the Integration Tests defaults.CI.mddescribes both K3s lanes.nix run .#tmachine -- test ubuntu-k3s k3s-ha conformanceTesting
mise run pre-commitat8a8d6b142.fe6c9545e:k3s-ha, including baselinek3swith the new overlay hook.k3s-hainstall succeeds with a singlehelm install, and the mTLS gateway connects.k3s-haconformance scenarios pass.sandbox_lifecyclefails atsandbox stopwithconcurrent modification, the replica race that fix(server): retry stop lifecycle writes after concurrent updates #4395 fixes. The previous revisionba53cb5dafailed the same way.k3s-ha-tls): Branch E2E run 37995038846 at387026536. The stop race is intermittent; it hit the two most recent runs. Local cold-boot conformance passed 3/3 on arm64 at an earlier revision.k3s-hagreen after fix(server): retry stop lifecycle writes after concurrent updates #4395 merged: Branch E2E run 38295274656 at31a92f16f(merge ofmainwith fix(server): retry stop lifecycle writes after concurrent updates #4395) passes every lane, includingk3s-haandCore E2E result.8a8d6b142(PostgreSQL wait, rerunnable namespace step) passes every lane, includingk3s-haandCore E2E result. The first attempt failed one baselinek3sscenario on an unrelated Kubernetes APISendRequesterror duringsandbox stop; the rerun of that job passed.nix run .#tmachine -- test ubuntu-k3s k3s-ha conformancepassed on arm64 at8a8d6b142, with images frommainatf6a39833c(includes fix(server): retry stop lifecycle writes after concurrent updates #4395).Checklist