bug: Kubernetes driver requests fail with SendRequest when reusing a nearly idle-timed-out connection #4451

Possibly seen elsewhere. #3758 (OpenShift single-node, OpenShell 0.1.1) reported healthy sandboxes being deleted about 5 minutes into startup after reconcile failed with ServiceError: client error (SendRequest). Its debug trace shows a similar sequence: hyper::Error(Io, Kind(TimedOut)), then reuse idle connection, then SendRequest about 10ms later. The reporter attributed it to API server stalls on "a suspiciously steady ~30s grid" and closed it as premature, so this is a possible match rather than a confirmed one.

We haven't found an upstream report of this exact failure. It might be specific to our configuration: the driver sets read_timeout to 30s, and this race seems to need a read timeout shorter than hyper-util's 90s pool idle timeout, because otherwise idle connections are closed before their read timer runs out. kube's default is 295s, and kube 4.0 removed the default entirely after concluding that read_timeout behaves as a connection idle timeout (kube#1798, kube#1945).

User Story

I run OpenShell's Kubernetes conformance lanes (tmachine K3s) for gateway changes such as #4081. I directly hit intermittent sandbox stop and sandbox delete failures with ServiceError: client error (SendRequest), and I need Kubernetes API requests from the gateway to succeed when the API server is healthy.

Problem Statement

The Kubernetes compute driver builds its request client with a 30s read_timeout (crates/openshell-driver-kubernetes/src/driver.rs). kube-client applies that timeout through hyper-timeout, whose read timer counts connection idle time, not request time:

• An idle pooled HTTP/1 connection keeps a read pending, so hyper-timeout starts the 30s read timer as soon as the connection goes idle.
• The timer resets only when a read completes. Writing a new request does not reset it, because kube-client never enables TimeoutConnector::set_reset_reader_on_write.
• A request sent on a connection that has been idle for T seconds therefore gets only 30 - T seconds to receive its response. When that remainder is shorter than the API server's latency, hyper fails the request with hyper::Error(Io, Kind(TimedOut)) and closes the connection. The driver reports that as ServiceError: client error (SendRequest).
• The API server then sees the client disconnect mid-request and logs context canceled, http: Handler timeout, and Post-timeout activity. Those lines are a consequence, not the cause.

Two things make this more likely than random chance:

• The driver's background reconcile loops run every 30s (SANDBOX_RUNTIME_RECONCILE_INTERVAL), the same as the read timeout. Connections opened for one tick's concurrent requests reach the end of their 30s budget right at the next tick, when concurrent traffic peaks again. A stop or delete running at that moment can be handed one of those connections.
• KubernetesDriverError::from_kube keeps only the top-level message, so the underlying TimedOut is never logged. That makes the failure look like a network or API server problem.

Impact / Why This Matters

• sandbox stop, sandbox delete, and sandbox create can fail with Internal on a healthy cluster. The stop path rolls back and the user must retry.
• Background paths are likely affected too. CI gateway logs show the error in reconcile paths, and gateway deletes healthy sandboxes at 5m bootstrap grace when apiserver transiently stalls (read timeout -> SendRequest -> fence reap) #3758 may be the same failure in a deployment: healthy sandboxes deleted at the 5m bootstrap grace after a reconcile SendRequest.
• It produces flaky Kubernetes CI lanes that need reruns.

Acceptance Criteria

• A Kubernetes API request reusing a pooled connection gets the full read timeout, regardless of how long the connection was idle.
• The read timeout still bounds a genuinely slow API response.
• A regression test reproduces the idle-connection failure without the fix.

Reproduction Steps

Deterministic reproduction with the same kube-client 0.99, hyper 1.9, and hyper-util 0.1.20 the driver uses: a kube client with read_timeout = 2s talking to a local HTTP server that answers the probe request in 300ms.

In a deployment, the failure appears intermittently on any Kubernetes gateway when a request lands in the last few milliseconds of a pooled connection's 30s window.

Environment

• OpenShell: main (CI at 8a8d6b142; possibly also 0.1.1 per gateway deletes healthy sandboxes at 5m bootstrap grace when apiserver transiently stalls (read timeout -> SendRequest -> fence reap) #3758)
• Runtime: Kubernetes compute driver; K3s v1.36 (tmachine CI), possibly OpenShift SNO (gateway deletes healthy sandboxes at 5m bootstrap grace when apiserver transiently stalls (read timeout -> SendRequest -> fence reap) #3758)
• kube 0.99.0, hyper 1.9.0, hyper-util 0.1.20, hyper-timeout 0.5.2

Logs

CI occurrences where this error failed the test (scan of failed Kubernetes jobs from 2026-07-01 to 2026-10-12, all attempts):

The same error also appears in the gateway logs of about 100 other failed Kubernetes jobs since 2026-09-05, mostly in background reconcile paths (Sandbox list failed, could not verify ... generation, Store reconciliation sweep failed). Those tests failed for other reasons, so they show the error is common in the background but don't prove it caused those failures. The scan found no occurrences in July or August logs. That may be related to the 30s reconcile loops added with the RFC 0012 sandbox and supervisor work in September, but we haven't confirmed it. For example, run 33981556004 (2026-09-05) logs failed with error client error (SendRequest) in the gateway.

Run 38303654729, attempt 1 (gateway and K3s share one VM clock):

20:46:14.199963Z INFO  reconcile: Listing sandboxes from Kubernetes
20:46:14.218518Z INFO  Listing sandboxes from Kubernetes
20:46:44.109439Z INFO  Prepared Kubernetes sandbox stop
20:46:44.136356Z INFO  supervisor session: stream closed by supervisor
20:46:44.219771Z ERROR kubernetes.stop_sandbox: kube_client::client::builder: failed with error client error (SendRequest)
k3s: 20:46:44.220163 etcd-client retrying of unary invoker failed ... KV/Range ... context canceled
k3s: 20:46:44.221799 apiserver was unable to write a JSON response: http: Handler timeout
k3s: 20:46:44.224783 "Post-timeout activity" method="GET" path="/api/v1/namespaces/openshell/pods/os-supervisor-<id>"

The failing request died 30.0s after the concurrent Listing sandboxes requests last read from the pool, and the client gave up before the API server finished its storage read.

CLI output:

Error:   × code: 'Internal error', message: "stop sandbox failed: ServiceError:
  │ client error (SendRequest)"

Suggested Fix

Set read_timeout = None on the driver's request client and add a per-request tower::timeout::TimeoutLayer through kube's ClientBuilder::with_layer. The timeout then starts when each request is sent, and every call path is covered without per-call changes. kube (through 4.2.0) does not expose hyper-timeout's set_reset_reader_on_write, so there is no plain config switch. Logging the full error chain in from_kube and retrying transport errors in stop and delete are possible follow-ups.