Skip to content

bug(podman): sandbox remains Ready after relay or service forward becomes unavailable #3314

Open
Open
bug(podman): sandbox remains Ready after relay or service forward becomes unavailable#3314
Labels
state:needs-infoAssessment needs specific evidence or reproduction details

Description

@itdove
itdove
opened on Sep 14, 2026

Summary

This issue reports a possible OpenShell relay and forwarding lifecycle problem observed with the Podman compute driver. It is not specific to AI Guardian. AI Guardian exposed the problem because it uses openshell forward service to reach a daemon running inside the sandbox.

The failure is currently time-dependent and has not been reduced to a deterministic trigger.

Environment

  • OpenShell gateway: 0.0.116
  • Embedded sandbox supervisor in both affected containers: openshell-sandbox 0.0.116
  • Compute driver: rootless Podman
  • Host: Fedora Linux with SELinux enforcing, x86_64
  • Two long-lived sandboxes

No usernames, hostnames, home directories, repository paths, container IDs, sandbox UUIDs, credentials, or private URLs are included here.

Observed behavior

After several hours or overnight:

  • openshell sandbox list continued to report both sandboxes as Ready.
  • podman ps showed both corresponding containers as Up (healthy).
  • openshell sandbox exec and interactive connection attempts failed.
  • Host-side openshell forward service processes were no longer running.
  • openshell forward list showed stale or dead forward records.
  • No local listening socket remained for the recorded forward ports.
  • The gateway remained healthy, but the sandbox control and service-forward paths were unusable.

No OOM kill or unhealthy container state was observed.

Minimal OpenShell-only observation harness

The following example does not use AI Guardian. It creates a long-lived sandbox containing a simple loopback HTTP service, then forwards that service through OpenShell. Use a disposable sandbox and replace the local port if 43152 is already in use.

Terminal A:

openshell sandbox create \
  --name openshell-forward-repro \
  --from base \
  --no-auto-providers \
  -- \
  sh -lc 'exec python3 -m http.server 63152 --bind 127.0.0.1'

Terminal B, after the sandbox is ready:

openshell sandbox exec --name openshell-forward-repro --no-tty -- \
  sh -lc 'curl --fail --silent --show-error http://127.0.0.1:63152/'

Terminal C, start the host-side service forward and leave it running:

openshell forward service openshell-forward-repro \
  --target-host 127.0.0.1 \
  --target-port 63152 \
  --local 127.0.0.1:43152

In another shell, periodically test the forward and the sandbox control path:

curl --fail --silent --show-error http://127.0.0.1:43152/
openshell sandbox exec --name openshell-forward-repro --no-tty -- true
openshell sandbox list
openshell forward list

For a longer observation, leave the sandbox and forward running for several hours or overnight, then repeat the checks. The failure is considered reproduced if the container remains healthy and the sandbox remains Ready, but the forward process exits or the local port stops listening and subsequent exec or connection requests fail.

Sanitized state evidence

Gateway: openshell
Status: healthy
Version: 0.0.116
Compute drivers: podman (0.0.116)

NAME       PHASE
sandbox-a  Ready
sandbox-b  Ready

Forward records for the affected sandboxes: dead
Podman containers for the affected sandboxes: Up (healthy)

Relevant sanitized logs

The affected run included messages similar to:

[gateway] supervisor session: relay opened successfully
[gateway] relay stream: claimed pending relay, bridging
[sandbox] supervisor session: relay bridge failed
[sandbox] write to target failed: Broken pipe (os error 32)
[gateway] ForwardTcp: inbound stream ended
[sandbox] supervisor session: stream closed by supervisor
[sandbox] supervisor session: ended

During one manual stop/start transition, the sandbox log also contained:

WARN openshell_supervisor_process::netns: Failed to delete network namespace

This line may be secondary to the restart and is not asserted as the root cause.

Relation to existing reports

  • #2372 describes the SSH accept loop exiting after transient errors while the sandbox remains Ready.
  • #2337 describes a similar proxy accept-loop failure under file-descriptor exhaustion.
  • #2587 describes broken-pipe relay failures, although that report is VM-specific.

The 0.0.116 supervisor source contains retry/backoff handling for transient SSH accept errors and monitoring for an unexpected SSH accept-loop exit:

Therefore, this may be a different failure in the host-side openshell forward service or in the relay lifecycle. The captured logs do not contain EMFILE, so file-descriptor exhaustion is only a hypothesis and is not confirmed.

Expected behavior

  • A sandbox reported as Ready should have working exec, interactive connection, and required service-forwarding paths.
  • If a required relay or forward dies, OpenShell should either recover it or mark the sandbox degraded/error instead of leaving it Ready.
  • A detached or long-lived forward should have an actionable failure status and a supported recovery or reconciliation mechanism.
  • Restarting a sandbox should restore the complete control and forwarding path, or identify exactly which component could not be restored.

Questions

  1. Can the gateway or Podman supervisor detect and recover a lost relay or service-forward path?
  2. Is there a health signal that distinguishes a healthy Podman container from a sandbox whose SSH or forwarding relay is unusable?
  3. Should openshell forward service be supervised or automatically reconciled when the underlying relay closes?
  4. Are there known OpenShell 0.0.116 issues involving long-lived Podman sandboxes, repeated ForwardTcp traffic, or forward processes exiting without changing the sandbox phase?

Suggested diagnostics

If this is reproduced, please collect only sanitized output from:

openshell version
openshell gateway info
openshell sandbox list
openshell forward list
openshell logs openshell-forward-repro --since 30m
podman ps -a --filter label=openshell.managed=true

Please redact local usernames, hostnames, filesystem paths, container IDs, sandbox UUIDs, credentials, and private endpoints before attaching logs.

Activity

  1. github-actions
    added
    state:triage-neededOpened without agent diagnostics and needs triage
    on Sep 14, 2026

  2. politerealism commented on Sep 14, 2026

    @politerealism
    politerealism
    on Sep 14, 2026
    Contributor

    Hi Dominique — thanks for the thorough report, this is a great repro.

    On your service expose vs forward service question: I checked the CLI and server code, and there's a real architectural difference worth knowing. forward service is a Start/Stop/List model — it's a host-side CLI process that has to stay running to keep the tunnel alive, which lines up with what you observed ("Host-side openshell forward service processes were no longer running"). service expose, by contrast, is backed by a durable, gateway-persisted ServiceEndpoint resource with full CRUD (Expose/List/Get/Delete), managed entirely by the gateway rather than a process on your machine.

    So switching should plausibly be more stable for your case — but two things worth checking before you do:

    1. service expose is documented as exposing an HTTP service specifically. If the daemon AI Guardian talks to inside the sandbox isn't speaking HTTP, this may not be a usable substitute at all.
    2. It likely still depends on the same underlying supervisor-to-gateway relay session that appears implicated in this bug (the relay bridge failed/broken-pipe messages in your logs). Switching probably eliminates the specific "host process silently dies" failure mode you're seeing, but may not fully resolve the underlying relay-death class this issue and bug: SSH server accept loop exits permanently on transient errors (same class as #2337) #2372/bug(proxy): EMFILE stops the sandbox proxy while phase remains Ready #2337/VM sandbox SSH session disconnects with broken pipe #2587 are all pointing at — so I'd treat it as a plausible mitigation for your immediate case, not a confirmed fix for the root cause.

  3. politerealism commented on Sep 16, 2026

    @politerealism
    politerealism
    on Sep 16, 2026
    Contributor

    @sjenning can you take a look and triage this one? I can take on the work if needed.

  4. elezar commented on Oct 13, 2026

    @elezar
    elezar
    on Oct 13, 2026
    Member

    📋 triage-agent

    Assessment: credible first-hand report, but the overnight failure is not yet independently reproduced or isolated. The OpenShell-only harness and environment are useful. I inspected v0.0.116 and current source; this host cannot reproduce the Fedora/rootless-Podman setup because Podman is unavailable.

    Two paths need distinguishing:

    • In v0.0.116, service_forward_tcp polls sandbox readiness every two seconds and exits on any polling RPC error, a non-Ready phase, a fatal forwarding error, or a local accept error. A transient gateway RPC failure can therefore terminate the local forward. Ordinary per-connection relay failures are logged without necessarily stopping the listener.
    • That version already reconnects the supervisor session on unexpected EOF/error. Gateway session cleanup calls supervisor_session_disconnected, which should demote Ready to Provisioning. A dead host-side forward alone does not imply the sandbox is unhealthy; persistent Ready together with failed fresh exec needs additional evidence about the control session/SSH listener.

    Broken pipe alone does not establish resource exhaustion or a duplicate of #2372/#2337/#2587. A later confirmed EOF-propagation bug, #3724, was fixed by #3772; it caused hanging forwarded connections and is relevant to retesting, but does not establish the cause of the exited processes or stale readiness here. service expose also uses sandbox relays, so it is not a confirmed workaround for failed control access.

    Please capture, with secrets/private identifiers redacted:

    1. The host CLI version, Podman version, and exact forward launch command, including foreground/detached mode and any shell/service supervision.
    2. Timestamped forward stderr and exit status, exact errors from a fresh exec and connect attempt, and aligned gateway/sandbox logs spanning the first failure through subsequent reconnect attempts. Include session acceptance/closure, reconnect failures, and any failure to mark the sandbox disconnected.
    3. Whether restarting only the forward restores access; whether fresh exec remains broken; and whether sandbox stop/start restores both. Please retest on a current supported release, accounting for fix(relay): close outbound relay stream when target closes first #3772, or explain the upgrade constraint.

    Moving to state:needs-info. Once this separates local-forward termination from sandbox control-session failure, a maintainer can decide the scope of work. If it persists without a determinable trigger, recommend a focused spike into health-poll retry behavior, session/readiness reconciliation, and relay/connection-slot cleanup.

  5. elezar
    added
    state:needs-infoAssessment needs specific evidence or reproduction details
    and removed
    state:triage-neededOpened without agent diagnostics and needs triage
    on Oct 13, 2026

  6. politerealism commented on Oct 13, 2026

    @politerealism
    politerealism
    on Oct 13, 2026
    Contributor

    @itdove thanks for the detailed report and the harness. I tried it on Fedora with rootless Podman. I used the v0.1.3 RPMs because they include #3772, so this isn't a 0.0.116 repro. I couldn't leave it running overnight, so the overnight failure is not reproduced.

    What I saw

    • With forward service running, I stopped the gateway for ~20s. The forward exited immediately with sandbox '…' is no longer ready (phase: Provisioning); stopping service forward (status 1). The sandbox returned to Ready and fresh exec worked, but nothing restarted the forward. So a single trip out of Ready kills the forward permanently, which matches the first path @elezar described.
    • This isn't the overnight trigger. On Podman a gateway restart also stops and restarts the sandbox, so the phase change is expected there. I didn't see exec fail while the sandbox stayed Ready, which is the more worrying part of your report.
    • openshell forward list shows "No active forwards" while a forward service is running, so I don't think forward service is tracked there. That might explain the stale or dead records you saw.

    Retesting on v0.1.3

    --from base no longer reaches Ready unless you also pass --policy. The published community base:latest image (built 2026-05-29) ships /etc/openshell/policy.yaml with tls: terminate. Since #3414 that value is a hard error, so the supervisor reports the image policy as invalid and the sandbox stays in ConfigurationInvalid. The error shown by sandbox get doesn't say which field failed. I filed that separately as #4467. With a minimal policy file (filesystem and Landlock only) the harness works as written. Sandbox names are also limited to 19 characters, so openshell-forward-repro is rejected.

    Could you retry on a current release?

    If you can, run your harness on v0.1.3 or newer and, if the failure comes back, capture what @elezar listed: the exact host CLI and Podman versions and forward command, timestamped forward stderr and exit status, the exact exec error, and aligned gateway and sandbox logs from the first failure. It would also help to know whether restarting only the forward restores access, and whether exec stays broken after that.

    If it no longer happens on a current release, that's useful to know too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:needs-infoAssessment needs specific evidence or reproduction details

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions