Skip to content

A policy DNS address stops taking new connections after the trusted resolver's TTL, one second when that TTL is 0 #4674

Open

Description

@Kevinbeltran123

User Story

In a personal project, I use OpenShell to let an agent build projects with cargo and Gradle. Sometimes a build fails on a host my policy allows and passes when I run it again; the supervisor's log shows transparent TCP mapping is expired. It looks like a network fault rather than a policy decision, and widening the policy does not help.

Problem Statement

Policy DNS answers a sandbox's lookup with a synthetic address and publishes a mapping from it to the resolved destination. The answer's TTL and the mapping's lifetime are both the trusted resolver's TTL, clamped to between 1 and 30 seconds (crates/openshell-supervisor-network/src/policy_dns/mod.rs: MIN_MAPPING_TTL, MAX_MAPPING_TTL, clamp_mapping_ttl). After that, a new TCP connection to the address is refused (transparent TCP mapping is expired) until something looks the name up again, which publishes the same address anew; a connection already open is not affected.

Clients keep a resolved address longer than that: libcurl's DNS cache defaults to 60 s (cargo and git use libcurl) and the JDK's address cache to 30 s. A new connection they open to the kept address is refused, on a host the policy allows. With a resolver that answers TTL 0 (the VM's resolver under Colima, for pypi.org and github.com, every time), every mapping lives one second.

Neither the address's lifetime nor this denial is in the docs: the denial reasons in docs/observability/logging.mdx do not list transparent_tcp_mapping_denied.

Impact / Why This Matters

Builds fail now and then on hosts the policy allows, with what looks like a network fault. In two cargo builds on 0.1.4, cargo's two new connections to index.crates.io, 5.4 and 5.6 s after its lookup, were refused (it retried and passed); on 0.1.3, four cargo builds in a row met the same denial and two of them failed, and a Gradle build failed resolving its test classpath when the JVM connected 14.5 s after the lookup.

The client gets no response and the denial names no policy, so an operator cannot tell it from a missing rule without the supervisor's log, and widening the policy does not help. The design (#2712) fails closed on stale synthetic addresses, which is right; a one-second floor makes ordinary client caching stale.

Acceptance Criteria

  • A client that opens a new connection to an address the sandbox's resolver handed out, within a common client cache lifetime (60 s, for instance) and while the policy still allows the name, is not refused, whatever TTL the trusted resolver gives.
  • The docs say how long a synthetic address takes new connections, and list transparent_tcp_mapping_denied among the denial reasons.

Reproduction Steps

Uses only an OpenShell sandbox and the curl in its image; no third-party installs. Any image with curl and bash works. pypi.org's own TTL is 86400 s.

  1. Save policy.yaml:

    version: 1
    filesystem_policy:
      include_workdir: true
      read_only: [/usr, /lib, /etc]
      read_write: [/tmp, /dev/null]
    landlock:
      compatibility: hard_requirement
    process:
      run_as_user: "1000"
      run_as_group: "1000"
    network_policies:
      probe:
        endpoints:
          - host: pypi.org
            port: 443
            protocol: rest
            enforcement: enforce
            rules:
              - allow:
                  method: GET
                  path: /simple/pip/
        binaries:
          - path: /usr/bin/curl
  2. openshell sandbox create --name a5 --from <image-with-curl> --policy policy.yaml

  3. In the sandbox (--resolve keeps the address, as a client's DNS cache does). The 31 s wait is past the 30 s ceiling, so this fails whatever TTL the trusted resolver gives:

    ip=$(curl -sS -o /dev/null -w '%{remote_ip}' https://pypi.org/simple/pip/)
    echo "address: $ip"
    sleep 31
    curl -sS -o /dev/null -w 'after 31 s: %{http_code}\n' --resolve "pypi.org:443:$ip" https://pypi.org/simple/pip/
    echo "curl exit: $?"
    curl -sS -o /dev/null -w 'new lookup: %{http_code} via %{remote_ip}\n' https://pypi.org/simple/pip/

    → address: 198.18.0.2, after 31 s: 000, curl exit: 7, new lookup: 200 via 198.18.0.2 (the same address, published again).

  4. With a trusted resolver that answers TTL 0 (Colima's VM resolver here), the same happens after sleep 2, and even for a second request made right after the first one, once the first transfer took over a second.

  5. Control: two transfers in one curl invocation reuse their connection (num_connects 1, then 0) and both get 200; only new connections are refused.

Suggested UX (if applicable)

Not applicable to configuration or CLI. Two possible directions, for maintainers to judge: a floor on the mapping's lifetime separate from the answer's TTL, or an expired mapping refreshed on connect while the policy still allows the name (a synthetic address is not reassigned within a sandbox's life, #2712).

Environment

  • OpenShell: v0.1.4 (gateway, supervisor and sandbox images 0.1.4; also seen on v0.1.3 and v0.1.2). main has the same bounds; the only change to policy_dns/ since the tag is a type refactor (feat(sandbox): publish and activate sandbox backend protocol #3898).
  • OS: macOS 26.6.2 (Darwin 25.6.0)
  • Deployment: local gateway, Docker driver; Docker 29.5.2 via Colima 0.10.3 (the supervisor's /etc/resolv.conf points at the VM's resolver, which answers TTL 0)

Logs

CONFIG:PUBLISHED [INFO] Policy DNS mapped pypi.org resolved=151.101.0.223,151.101.64.223,151.101.128.223,151.101.192.223 synthetic=198.18.0.2 ports=443
WARN openshell_supervisor_network::proxy: Denied staged transparent connection destination=198.18.0.2:443 error=transparent destination mapping is unavailable: transparent TCP mapping is expired
NET:OPEN [MED] DENIED /usr/bin/curl(0) -> 198.18.0.2:443 [reason:transparent_tcp_mapping_denied]
CONFIG:PUBLISHED [INFO] Policy DNS mapped pypi.org resolved=151.101.0.223,151.101.64.223,151.101.128.223,151.101.192.223 synthetic=198.18.0.2 ports=443

curl: (7) Failed to connect to pypi.org port 443 after 0 ms: Could not connect to server
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions