Repository navigation
Merge main into demo (after #52, #88, #89 and #85) - #96
Merged
sauagarwa merged 70 commits intoOct 12, 2026
Merged
Conversation
Consolidate Make targets and safe lifecycle scripts. Add deployment checks and update guides and tests.
Merge upstream/main after PR validatedpatterns-sandbox#79. Keep the clean Make workflow and remove obsolete prepare Job code.
…, execute H9 revocation drill Three gaps from PR validatedpatterns-sandbox#53 review (round 7) left open on main after merge: - verify_harness_image was only called from image_tree, which only runs when the harness volume gets (re)filled. A sandbox whose volume already holds the image never had the signature rechecked, so revoking the cosign identity had no effect on it. Moved the check into prepare_harness so it runs on every apply, regardless of whether the volume is current. - verify_harness only confirmed the OpenClaw bundle loaded (plugins list --json, bundleCapabilities flag), never that the individual MCP servers mcp.json declares are actually live. describe_harness_tree now returns mcpServers (declared names); verify_harness cross-checks them against `openclaw mcp status`, WARN not fail when the agent has no mcp status subcommand. - scripts/e2e-harness.sh's H9 revocation step only printed manual instructions and skipped. It now executes the drill: helm upgrade --set harnessEnabled=false, wait for the sandbox to come back without /sandbox/harness and a clean plugins.load.paths, then flip back to true and wait for the mount to return, restoring harnessEnabled=true on exit (including ^C). H3 gained the same live MCP server-name check. Rollback (HarnessVolume.fill backing up and restoring the previous tree on a failed import) was already correct on main; untouched here. Tests: tests/installer -k harness, 114 passed, 1 pre-existing unrelated failure (test_a_running_gateway_is_kept_when_nothing_changed, fails identically on clean main).
A harness volume reaches a sandbox through caller driver config, and allow_driver_config is a fleet-wide gateway switch: once it is on, any workspace user may submit a driver_config envelope, and OpenShell's resource admission never inspects who sent it or what mount mode it asks for (crates/openshell-core/src/resource_admission.rs knows nothing about the caller). That is the create-time window the harness review left open: the installer can detect a writable remount and recreate the sandbox, but nothing stopped the mount being created writable in the first place. The interceptor does see both, since the gateway hands it the whole CreateSandbox request and driver_config carries no secret field option, so it is not stripped from the interceptor view. Two rules go in through the same patch-the-upstream-source mechanism the profile-annotation workaround already uses: - only a platform admin identity may submit driver config at all; - every mount it carries must set read_only: true. CreateSandboxTemplate is bound as well, because workload_template is resolved after the interceptor seam, so a template would otherwise smuggle a writable mount past the CreateSandbox check. tests/charts/test_governance_interceptor_image.py keeps the Dockerfile and the BuildConfig's inlined copy in step, since the two must patch upstream identically.
…2e dedup Follow-up to 439f826: - verify_harness's MCP server check used plain substring containment (`s not in listed.out`), so a declared server named `search` would be wrongly treated as present if the output merely contained an unrelated server named `search-internal`. Switched to a word-boundary regex match. scripts/e2e-harness.sh already used `grep -qw` for the same check, so no change needed there. - Added test_a_revoked_signature_is_caught_on_a_later_apply: fills the volume once, marks the image unsigned, then re-applies with a fresh installer instance against the same (already current) ledger-backed volume. Confirms the signature check in 439f826 actually fires when no pull/export happens, not just on first fill. - Removed a redundant BOM_RELEASE default assignment inside the H9 drill block (already set at the top of the script). Tests: tests/installer -k harness, 115 passed, 1 pre-existing unrelated failure (test_a_running_gateway_is_kept_when_nothing_changed, fails identically on clean main).
…or GC prepare_harness added an image to self.harness_images before calling verify_harness_image on it. If the signature check then raised (revoked identity, no longer signed), the image was already in the keep-set for this apply, so cleanup_harness_images() never removed it -- a revoked image stayed on disk indefinitely instead of being garbage-collected. Swapped the order: verify first, then add to the keep-set. Tests: test_an_image_whose_signature_is_revoked_is_garbage_collected.
…t mcp status verify_harness's live MCP-server check called `openclaw mcp status`, which only lists servers added via `mcp add`/`mcp set`; it never reflects ones a plugin bundle declares in its own mcp.json. Caught live on a real cluster: the shipped ds-default bundle failed verify on every single apply, crash- looping saw-apply.service forever, even though the bundle loaded correctly. `openclaw plugins inspect <bundle> --runtime` is OpenClaw's actual documented way to check a bundle's live MCP servers (confirmed against docs.openclaw.ai); --runtime imports the module so it reflects real load state, not just declared config. No test exercised this code path before (the fake openshell had no handler for the command at all), which is why the bug shipped with "115 passed". Added a plugins inspect handler to the fake and a test proving verify still fails when a declared server is missing from the live listing.
…sion verify_harness's live MCP-server match used \bserver\b, but `-` is a non-word character in regex, so \bsearch\b still matches inside "search-internal" -- exactly the false positive 87d1e0d's word-boundary fix claimed to close. Server names here are kebab-case (saw-mcp-echo), so a hyphen must count as part of the token, not a boundary. Switched to a negative lookaround that treats [\w-] as the token's own characters. No test existed for the word-boundary behavior itself (87d1e0d only added a signature-revocation test); added one with a declared server name that is a prefix of a different name in the live listing.
Require the CLI API version pinned by the gateway BOM. Show Pattern VMs in saw-list and recover absent local gateway entries. Pass the sandbox password to the OpenClaw TUI. Document browser pairing for the older CUDA image.
Keep Pattern checks focused on tools and cluster capacity. Require CNV and RHBK only for the manual quickstart path. Remove an orphaned CNV CRD after owned teardown when no operator or HyperConverged resource remains. Document test user password access.
Correct Keycloak help, clarify SAW listing, and move internal checks after public commands. Keep designer approval pending.
Build UI route values from the selected SAW-BOM profiles and set the cluster domain for the VM proxy. Use uv for local Keycloak user tasks. Document the routes and test the profile selection.
Use the default Helm release list for SAW and Keycloak cleanup. Helm v4 includes all release states and rejects the --all flag.
…pr53-nvidia-gaps fix(harness): close signature/MCP/revocation and driver-config admission gaps
Mark only the CSV installed by the pattern-owned subscription. After uninstall, remove it only when no Pipelines subscription remains. This lets OLM resolve the operator on reinstall.
…ssuer The gateway fetches the OIDC issuer (Keycloak's route) at startup with the system trust store and exits when it cannot verify it. On a cluster whose apps certificate comes from its own ingress CA (found on an arm64 LaunchPad cluster), the gateway never started: "OIDC discovery request failed", and the install timed out. oidc.caBundle (PEM) goes into config.json; the installer writes it to /etc/pki/ca-trust/source/anchors on every run (cloud-init would run once per VM), runs update-ca-trust, and restarts the gateway when the trust store changed. Emptying the setting removes it. Co-Authored-By: Claude
… trust store docs/deployment-guide.md, "The issuer's certificate": when it is needed (self-signed *.apps or a private-CA external issuer), how to check from a VM, and how to set oidc.caBundle for the pattern and the quickstart. The dashboard and sandbox-UI oauth2-proxies get the VM's trust store, so setting dashboard.insecureSkipIssuerTlsVerify: false verifies the issuer with the configured CA. The default is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e fix make check-oidc-ca (also in check-prereqs) tells, before ./pattern.sh make install, whether workspace VMs will trust the issuer: OpenShift's self-signed *.apps default, or a custom certificate that does not verify against public CAs, means oidc.caBundle is needed. It saves the cluster's ingress CA and prints the overrides/saw-users.yaml snippet. ISSUER=<url> checks an external issuer. If it was missed, install now checks the issuer with the VM's trust store before starting the gateway and fails with the same instructions, instead of timing out on the gateway's port. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…luster Keycloak The saw-ingress-ca imperative job copies openshift-config-managed/ default-ingress-cert (public data) to Vault at secret/data/hub/cluster-ingress-ca when it changes. pattern-secrets syncs it into every user namespace as the Secret saw-ingress-ca (saw-users defaults.clusterCaSecret), and the VM mounts it like a provider Secret; the installer trusts its ca-bundle.crt. Only for the in-cluster Keycloak (oidc.issuerUrl empty); an explicit oidc.caBundle wins. With the issuer now trusted, saw-users no longer turns off the oauth2- proxies' issuer verification (insecureSkipIssuerTlsVerify). check-oidc-ca exits 3 for this case (handled by the pattern, the quickstart needs oidc.caBundle); the installer's error and the docs describe both paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pattern uninstall removes the managed gateway image stream. Restore the tag before the live test runner starts reinstall.
Found on the GB200: Argo CD's client-side apply copies each object into its kubectl.kubernetes.io/last-applied-configuration annotation, and annotations may hold at most 262144 bytes. The installer ConfigMap (apply_bom.py and friends) is about 261 KB rendered, so the sync failed with "metadata.annotations: Too long" and the VM kept its old installer. Main is already about 255 KB, so any growth hits it. Server-side apply keeps no such annotation; a ConfigMap itself may hold 1 MiB. The portal's per-user apps render saw-users too, so they get it as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mark the HyperConverged CRD while its pattern-owned namespace exists. Cleanup then checks the marker even if framework uninstall has already removed the namespace. Cover both deletion orders and unowned cases.
Use the CI Python environment for Keycloak user script tests. Keep uv for local runs. Set the harness test namespace so Make exports do not alter its fixture.
With ServerSideApply there is no last-applied annotation to diff against, so the fields KubeVirt defaults on the VM (firmware serial and uuid, machine type) kept saw-<user> OutOfSync. Found on the GB200; argocd.argoproj.io/compare-options: ServerSideDiff=true on the Application made it Synced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review: in the system trust store the issuer CA (the cluster's ingress CA, no name constraints) was a trust anchor for every TLS client on the VM, podman pulls included. Now: - The installer writes the CA to the runtime user's ~/.config/openshell/issuer-ca.pem and, with the public CAs, to issuer-trust.pem, and gives the gateway SSL_CERT_FILE=issuer-trust.pem in a drop-in (openshell-gateway.service.d/issuer-ca.conf). The system store is untouched; the anchor older builds wrote there is removed. - The oauth2-proxies mount issuer-ca.pem and use it only for the issuer (OAUTH2_PROXY_PROVIDER_CA_FILES) instead of getting the system bundle over /etc/ssl/certs; the dashboard BFF gets nothing. - A bundle with a private key is refused by the chart and the installer, one without a CA certificate by the installer; leaf certificates are dropped. The issuer pre-check verifies with issuer-trust.pem, so a CA that does not sign the issuer fails install. - saw-ingress-ca verifies Vault's certificate (service CA) before sending the root token, and skips a ConfigMap with a key in it. - Docs: the trust scope, and that a user's values are admin input. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review: the pre-check verified with the public CAs plus the extra CA, so on a cluster whose *.apps certificate is public the automatic ingress CA was still added to the gateway's trust although it signed nothing. select_issuer_ca now connects to the issuer and verifies with the extra CA alone: installed when it signs the issuer, skipped when the issuer is publicly trusted anyway, and install fails with the fix when neither verifies it. An unreachable issuer keeps the configured CA (the gateway reports it). Tests use a real HTTPS issuer on 127.0.0.1. saw-ingress-ca: stop with a clear message, before reading the root token, if the service CA file is missing. OpenShift puts service-ca.crt in every pod's default service-account volume, and the GB200 run reached Vault with verification, so this only guards against a different platform. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI (Python 3.14) failed the real-TLS issuer tests with "CA cert does not include key usage extension": since 3.13 ssl.create_default_context() sets VERIFY_X509_STRICT. The test CA now has keyUsage and a subject key identifier and the issuer certificate an authority key identifier, keyUsage and serverAuth, like OpenShift's ingress CA and certificate (which is why the same code passed on the GB200's Fedora 44 guest). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…idc-ca-trust Trust a private-CA OIDC issuer: oidc.caBundle, with manual steps
…ch-ci-images-to-ghcr ci: share image build recipes and switch project images to GHCR
…ster-policy feat(security): enforce intra-cluster NetworkPolicy and profile allowlist parity (Phase 2 Isolation)
global.governance.engine (values-global.yaml, or the Pattern's extraParameters) picks the service behind governance-interceptor.openshell-agents.svc:18081: the OpenShell governance interceptor (default, unchanged) or NVIDIA Agent Policy Fabric, which serves the same policy as a signed bundle, stamps provenance on every sandbox, refuses runtime policy changes and writes an audit trail. - charts/governance-interceptor: with apf, the signed bundle and trust root ConfigMaps (files/apf, generated per deployment), ExternalSecrets from Vault for the GHCR pull secret (toJson, trimmed), the signing seed and the Argo CD OCI repository, and an Argo CD Application for the private APF chart, released as governance-interceptor so gateways keep their endpoint. The NetworkPolicy selects the APF pod. The app prunes and the APF Application carries the resources finalizer, so switching engines either way removes the other engine's objects. - charts/openshell-saw: per-engine binding policy. APF declares one binding per phase, which allowlist rejects, so APF uses dynamic (its own 14 bindings, fail-open post-commit). Interceptor rendering is byte-identical, so existing VMs do not restart. saw-users passes global.governance to each user. - scripts/apf-bundle.py (make apf-keys, apf-bundle, apf-bundle-check): translates charts/governance-policy into APF SandboxPolicy/Provider documents using only the fields APF accepts, leaves out profiles APF cannot express (query auth: gemini), and signs exactly as apf-compile genfixture. The repository ships no key and no bundle. - Quickstart: make governance-apf and GOVERNANCE_ENGINE=apf. - values-secret.yaml.template: ghcr and apf-signing entries. - docs/apf.md, README setup steps, deployment guide; chart and script tests. Validated live on OpenShift: signature verified, policy injected with apf.nvidia.com provenance, runtime policy mutation refused, audit events recorded. Co-Authored-By: Claude
- Golden byte comparison of the interceptor-engine gateway.toml (default and with OIDC), rendered from main before the switch, so the "existing VMs don't restart" guarantee holds byte for byte, not only after parsing (review: mkhaas). - apf-bundle.py: note that apf-dev is only the chart's mount label; the key behind it is the operator's own (review: mkhaas). - Rebased on main with OpenShell 0.1.2 (validatedpatterns-sandbox#56): the engine switch sits in the schema-v2 gateway.toml, and the interceptor image stays the ghcr v0.1.2 one. The openai profile now names api.openai.com, which the APF bundle test expects. - docs/apf.md: what to check with OpenShell 0.1.x (extension protocol negotiation, NVIDIA/OpenShell#3929 annotations, no inference routing). Co-Authored-By: Claude
validatedpatterns-sandbox#78 lets a sandbox mount its harness volume through caller driver config, and relies on the patched governance interceptor to admit only the installer's read-only harness mount from an mTLS admin. APF has no such guard, so with engine apf any caller could attach the volume, writable. The chart now stops with a message when allowDriverConfig is set on the apf engine, instead of rendering a gateway without the guard. Harness bundles stay off by default, so default workspaces are unaffected. Co-Authored-By: Claude
The APF chart (0.2.0) pins images built against OpenShell 0.0.x. A 0.1.2 SAW gateway refuses them at startup: "gateway-interceptor extension 'governance' did not provide protocol metadata" (seen live), and with fail_closed the gateway stays down. APF 0.1.0 is the first build for OpenShell 0.1.x: its Describe returns the extension protocol metadata (PeerMetadata, protocol 1.0), and it no longer misreads the bundle's default SandboxPolicy as a provider profile. The chart stays at 0.2.0; only the gateway, policy-service and compile images move to 0.1.0, through apf.values (so the Argo CD Application and `make governance-apf` both get them). The images are multi-arch. Co-Authored-By: Claude
… 0.1.0 From the live test on OpenShift with OpenShell 0.1.2-rhaiv.0: APF puts the policy on a sandbox at creation and refuses changes afterwards, so a re-signed bundle reaches sandboxes created after it, not the existing ones. Adds the network_policies example that was verified (git clone from github.com refused, then allowed for a new sandbox after re-signing), how to find the real binaries to list, and what the provenance annotations record. Co-Authored-By: Claude
The portal rebuilt `global` for its saw-users values without `governance`, so portal-created workspaces rendered interceptor `allowlist` bindings even with the pattern on APF. Forward it like sshPublicKey; saw-users already passes it on to openshell-saw (machineGlobals). Co-Authored-By: Claude
- build: an unchanged policy is "up to date" only when the bundle still verifies against apf.pub. After `make apf-keys --force` it re-signs with a new revision instead of leaving a bundle APF rejects. - APF's anti-rollback counter is an emptyDir in the APF chart, and every bundle change rolls the pod, so a new pod accepts any signed revision. verify (make apf-bundle-check, and the committed-bundle test) now fails when policy_revision is not above every earlier commit of the bundle, so a git revert of the bundle is caught before Argo CD deploys it. build picks a revision above all of git history. - keygen --force tightens the seed to 0600 when the file already existed. Co-Authored-By: Claude
KubeVirt does not restart a running VM when its template changes; it marks it RestartRequired and the gateway keeps its old bindings. The docs said VMs restart on their own. `make saw-restart-pending` lists gateway VMs with RestartRequired and restarts them with CONFIRM=yes. docs/apf.md says to run it in both directions, how quickstart SAWs switch, how to re-sign after a new key, and where rollback is enforced. Co-Authored-By: Claude
validatedpatterns-sandbox#88 made networkPolicy.clusterNetworkCIDRs required (make saw-create and the pattern discover them), so the APF gateway tests' direct renders of openshell-saw need the same offline fixture main's chart tests use. Co-Authored-By: Claude
…pf-governance feat(governance): switch between the governance interceptor and APF
…tterns-sandbox#88, validatedpatterns-sandbox#89 and validatedpatterns-sandbox#85) Conflicts, resolved as in the feat/demo rebase: - values-secret.yaml.template: main's APF entries, then the demo's Slack and Gmail entries under its "Personal assistant" heading. - tests/charts/test_openshell_saw_chart.py: main (validatedpatterns-sandbox#88) moved the profile copy check to test_egress_profile_parity.py; it now lists the demo's gmail and slack copies there. - Makefile-quickstart: main's layout (validatedpatterns-sandbox#85); keycloak-add-user added to it. Main's Keycloak script no longer prints passwords, so its help says the password is kept in a Secret. Co-Authored-By: Claude
sauagarwa
force-pushed
the
sync/main-into-demo
branch
from
October 12, 2026 16:37
d0d3c09 to
88846fe
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge main into demo
demowas based on main at #79. This merge brings in current main (4fc1785), including #84, #85, #88, #89 and #52. It adds no new demo content: the next PR adds the LaunchPad doc and doc fixes.Please merge with "Create a merge commit", not squash or rebase. That way demo keeps main's commits, and later main → demo (or demo → main) merges stay conflict-free.
Conflicts
values-secret.yaml.template: both sides added entries after Brave. Kept main's APF entries, then the demo's Slack and Gmail entries, under the demo's "Personal assistant" heading.tests/charts/test_openshell_saw_chart.py: feat(security): enforce intra-cluster NetworkPolicy and profile allowlist parity (Phase 2 Isolation) #88 moved the provider-profile copy check totests/charts/test_egress_profile_parity.py. Removed it here, and added the demo'sgmailandslackcopies to the moved check.Makefile-quickstart: took main's layout (feat: clean Validated Pattern workflows (APPENG-5960) #85) and added the demo'skeycloak-add-usertarget to it. Main's Keycloak script no longer prints passwords (they stay in a Secret), so the target's help says so.Checks
feat/democontent that has been tested.ssh-keygen, and one NetworkPolicy render. CI covers those.egress.allowalready lists their hosts.0.1.2-rhaiv.0, engineinterceptor). A portal workspace installs and applies cleanly, with NemoClaw images from GHCR.