Skip to content

bug(supervisor): literal and foreign-provider placeholders in conversation history block inference #2904

Description

@zredlined

User Story

As an agent running inside an OpenShell sandbox with provider-backed inference, I want to read documentation, source code, and other tool output containing credential-placeholder examples without terminating my inference session, so that I can safely complete long-running tasks.

Problem Statement

When an agent reads content using a tool, the tool's output is automatically added to the agent's conversation history. On the next model turn, the agent client serializes that history—including the tool output—into the outbound inference request body.

OpenShell intercepts and inspects that request before forwarding it to the configured model provider.

During long-horizon testing, an agent read OpenShell's policy-schema documentation. The tool output included this literal documentation example:

openshell:resolve:env:KEY

The following sequence occurred:

Agent reads documentation using a tool
  → documentation is returned as tool output
  → tool output is stored in the conversation history
  → the next model request includes that history
  → OpenShell intercepts and scans the request body
  → the literal example is classified as credential-bearing traffic
  → OpenShell blocks the request
  → the agent reports a stream disconnection

The agent did not request, print, or transmit an actual credential. It did not intentionally put credential material into a policy proposal. The reserved string entered the inference request only because ordinary documentation returned by a tool became part of the agent's conversation context.

After the first denial, the client retried using the same conversation. Because the conversation still contained the literal example, every retry was denied. Normal inference could not resume unless the affected context was discarded.

Although this reproduction uses OpenShell's documentation, the behavior is not documentation-specific. Any agent-visible content containing reserved placeholder syntax—including source code, logs, command output, repository files, or fetched pages—can be incorporated into conversation history and trigger the same false-positive denial.

The underlying issue is that syntactic marker detection does not distinguish an unbound literal string, an attached credential bound to another endpoint, and a credential usable at the request destination. This issue consolidates #3237; both reports exercise the same REST request-body guard but require distinct regression tests.

Suggested remediation

Before classifying a request body as credential-bearing, parse the placeholder and determine whether it references an active credential:

  • With request-body credential rewriting disabled, preserve fail-closed behavior for credentials bound to the destination host, port, and path.
  • Forward placeholders unchanged when trusted credential metadata establishes that the referenced key is unknown or the credential is outside the destination scope. Do not infer harmlessness from a failed value lookup.
  • If the marker is an unbound literal example, treat it as ordinary text.
  • If credential resolution is unavailable or fails unexpectedly, remain fail closed.
  • For a destination-bound credential reference in a model-bound request with body rewriting disabled, block the reference. Never substitute its secret into conversation content. Automatic redaction and enabling body rewriting are outside this fix.
  • Return a specific local diagnostic, such as credential_placeholder_in_request_body, instead of presenting only a generic stream disconnection.

Changing documentation or instructing agents not to print placeholder syntax may reduce the frequency, but it would not address the underlying issue because the same syntax can appear in arbitrary agent-visible content.

Impact / Why This Matters

This was observed in three of four long-horizon experiments:

  • After 7m53s in a 3-hour run.
  • After 1h40m46s in a 6-hour run.
  • After 4h14m56s in another 6-hour run.

In each case, hard inference failures began approximately 3–4 seconds after the placeholder example appeared in tool output.

Consequences included:

  • One 3-hour run accumulated 58 retries and 4,998 seconds of backoff, making the result operationally invalid.
  • One 6-hour run experienced 76 consecutive failed turns and did not recover before being interrupted.
  • Another 6-hour run terminated at approximately 4h15m.
  • OpenShell recorded 385 request-body credential denials in the final 30-minute log buffer of the 3-hour run.

The issue did not appear in shorter validation runs where the agent never encountered the reserved syntax. This makes the failure particularly disruptive to persistent agents: hours of successful work can be lost after an otherwise harmless documentation or source-code read.

The current workaround is to prevent agents from reading or printing any content containing reserved syntax, or to discard the complete conversation after the failure. That workaround is fragile because the syntax can appear in OpenShell's own public documentation and source code.

Acceptance Criteria

  • An unbound literal example such as openshell:resolve:env:KEY does not block a provider-credentialed model request.
  • With request-body credential rewriting disabled, a placeholder bound to the destination endpoint remains fail closed.
  • With request-body credential rewriting disabled, a placeholder for a credential bound to a different endpoint is forwarded unchanged.
  • A known but unusable credential is not treated as an unknown literal merely because resolution fails.
  • A real credential value is never forwarded to the model provider through the request body.
  • If the credential resolver is unavailable or returns an error, handling remains fail closed.
  • Canonical and provider-shaped placeholder formats receive equivalent bound-versus-literal handling.
  • Behavior is correct for fixed-length and chunked request bodies, including markers split across buffer boundaries.
  • A regression test covers tool output containing a literal placeholder followed by a model request carrying that output in its conversation history.
  • A blocked real credential reference produces an actionable diagnostic rather than only a generic stream-disconnection error.
  • OCSF denials identify destination-bound references versus unavailable classification metadata, without logging credential names, placeholder strings, secrets, or body contents.
  • Built-in codex + github providers permit POST /v1/responses after gh auth status output enters conversation history; equivalent coverage exercises claude-code and copilot conversation endpoints.
  • Documentation explains unbound literals and foreign-provider placeholders in multi-provider sandbox request bodies.

Reproduction Steps

  1. Create an OpenShell sandbox with a provider-backed, credentialed model endpoint.

  2. Run an agent inside the sandbox using that endpoint.

  3. Have the agent use a tool to read text containing this literal example:

    openshell:resolve:env:KEY
    

    KEY should not correspond to an attached credential.

  4. Continue the conversation so the tool result is included in the next model request.

  5. Observe that OpenShell permits the destination and HTTP method but denies the request body as credential-bearing traffic.

  6. Observe the client report a stream disconnection.

  7. Observe retries continue to fail because they resend the same conversation history.

Environment

  • OpenShell gateway: build from commit d9d38a561944dd041342c004dbfe6ff5b3c8d8de
  • Deployment: Docker on Linux ARM64
  • Agent client: Codex CLI 0.147.0
  • Inference integration: provider-backed Responses-compatible endpoint
  • Testing dates: 2026-08-21 through 2026-08-22

Logs

HTTP:POST [INFO] ALLOWED POST https://<model-provider-endpoint>/v1/responses
  [policy:<provider-policy> engine:l7]

NET:TRAFFIC [HIGH] DENIED <model-provider-endpoint>:443
  [reason:POST request body credential traffic denied]

FINDING:CREATE [HIGH] "Credential-bearing traffic cannot be inspected"
  [type:openshell.credentials.traffic_uninspectable]

The client then reports:

Reconnecting... 1/5
(stream disconnected before completion:
 error sending request to the configured Responses endpoint)

After retry 5/5:

stream disconnected before completion:
error sending request to the configured Responses endpoint

Consolidated reproduction from #3237

The original report above covers an unknown literal. #3237 covers an attached credential whose binding excludes the model endpoint:

  1. On OpenShell 0.0.116 with Providers v2 enabled and rootless Podman on Fedora 44, create providers:
    openshell provider create --name codex --type codex
    openshell provider create --name github --type github
    openshell sandbox create --provider codex --provider github -- codex
  2. Inside the agent session, run gh auth status. Its output includes a GitHub credential placeholder, for example Token: openshell:resolve:env:v<N>_GITHUB_***** (redacted display from the report).
  3. Send a follow-up prompt so this output enters the next POST /v1/responses body. The request is denied and retries repeat the denial.
  4. The GitHub credential is bound to github.com / api.github.com, not api.openai.com. With body rewriting disabled, the expected result is to forward the placeholder unchanged, without substituting a secret.

The same profile defaults affect claude-code and copilot conversation endpoints. The reported workaround uses a custom profile with allow_uninspected_credentials: true; built-in profiles are immutable, and this exception disables more protection than the false positive requires.

Both reproductions must remain covered independently: unknown-key classification and endpoint-mismatch classification are different paths. The original #3237 retains its full diagnostics and environment details for reference.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

state:acceptedA maintainer decided OpenShell should pursue this issuestate:agent-readyApproved for agent implementation

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions