feat(providers): report bounded token-grant failures

[shiju-nv]

Summary

A refused dynamic token grant now gives authorized callers a bounded cause and recovery action through sandbox reads, while the workload receives a generic HTTP 502 and an opaque request ID. Reporting covers inspected HTTP protocols and selector-based forwarding, including failures before credential selection.

Related Issue

Closes #3319

Changes

• Classify each refused grant into one closed set of codes, each with a fixed recovery action from the existing provider recovery vocabulary. No cause retains remote descriptions, tokens or error chains.

  Code	Recovery	gRPC status
  provider_authorization_required	reauthorize	FAILED_PRECONDITION
  workload_identity_rejected	fix configuration	PERMISSION_DENIED
  configuration_error	fix configuration	FAILED_PRECONDITION
  timeout	retry	DEADLINE_EXCEEDED
  unavailable	retry	UNAVAILABLE
  unclassified	investigate	UNKNOWN

• Recognize a token service's provider_authorization_required only with the exact openshell.token-grant.v1 contract over verified HTTPS; a bare 401 stays unclassified. The gateway can also establish that cause from missing or expired stored subject credentials. HTTP 429 and 5xx without a recognized extension code are unavailable, as are connect failures, including TLS certificate or hostname verification failures. A failure after the connection is established is unclassified, and any timeout, including a slow response body, is timeout.

• The gateway returns a refused grant to the supervisor as a gRPC status with a single ErrorInfo detail (reason TOKEN_GRANT_FAILURE). The supervisor decodes it strictly. Any other status, such as details over 1024 bytes, more than one detail, or unknown metadata keys, codes or stages, becomes unclassified at the gateway_rpc stage.

• Carry one supervisor-minted request ID through both token-service hops, the gateway, refusal logs, status and workload response (x-openshell-request-id).

• Retain one SandboxStatus.last_token_grant_failure with a bounded destination and failure cause. Reuse authenticated status reports and sandbox reads, with no new public RPC or request history. Matching grant success and configuration/session resets clear it; unrelated success retains it. MCP endpoint status also retains its configured-endpoint cause.

• Proto additions, all on existing messages or new messages they reference:

  • EndpointFailure and EndpointFailureDomain (domain TOKEN_GRANT), carrying code, recovery action, provider, credential key, stage, optional upstream HTTP status and request ID.
  • TokenGrantStatus and TokenGrantFailureObservation (destination host and port plus an EndpointFailure).
  • ReportEndpointStatusRequest.token_grant_status = 8. Omitting it keeps the previous behavior; a present message with no last_failure clears the stored cause.
  • EndpointObservation.failure = 3, EndpointStatus.last_failure = 7 and SandboxStatus.last_token_grant_failure = 17.

• In Rust, the supervisor builds the proto EndpointFailure from a typed TokenGrantFailure through openshell_core::endpoint_failure::EndpointFailure. The gateway validates reported causes against the same closed sets and names the rejected field and reason (EndpointFailureValidationError) without echoing rejected values.

• Select the live credential snapshot on fallback forwarding so an idle socket cannot report an old grant under a new provider revision, and drop the per-connection credential copy that no request reads anymore. Expose the result through CLI JSON/YAML/text, the Go SDK and documentation.

• Add a real REST workload test with native CA verification at both grant hops, authenticated persisted reads, exact correlation, redaction, denied anonymous and supervisor-credential readers, and recovery.

Testing

Focused checks pass on the final head: core 586, server 747 (provider, policy, sandbox, session and schema modules), network 1120 (token grant, L7 and proxy modules) and CLI rendering tests; Go converter tests; strict Clippy on all targets of the five affected production packages; E2E target compilation; formatting and Markdown lint. The repository pre-commit hook (workspace formatting and Clippy on all targets) passed.

Final-head Branch Checks passed all 24 jobs on the first attempt. E2E passed with 59 successful jobs and six optional skipped jobs. The verified-HTTPS REST case passed in bundled Podman, and its target completed all three tests. Optional Windows, GPU and Helm contexts are not runtime proof.

The feature-enabled E2E target retains 13 preexisting Clippy diagnostics. Its library and a diagnostic run allowing only those known categories pass; no lint configuration was changed. The hosted default-feature strict Clippy gate passed.

• Checks appropriate to the affected code and behavior pass locally
• Unit tests added/updated
• E2E tests added/updated

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated

[github-actions]

Label test:e2e applied for e7d1a41. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.