fix: send SSE comment keepalives so slow streams are not idle-killed - #188
Merged
Merged
Conversation
On slow hardware a streaming response can carry no bytes for minutes:
through a long prefill (the progress heartbeat is opt-in) and while a tool
call is buffered, since no delta is sent until it parses. Node/Bun clients
abort a body idle for ~300 s ("terminated"), and the agent's retry re-pays
the whole prefill. Observed with little-coder/pi on a 16 GB M2: a 4k-token
write tool call went silent after 16 tokens and died at +300 s, then every
retry timed out.
Emit ": keepalive" every 5 s on chat and text completion streams until the
stream terminates. SSE parsers ignore comment lines, and every yield is a
complete event, so a comment never lands inside another event.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
300 six-digit numbers measure 2,126 tokens with the gemma-4-e2b tokenizer and chat wrapper; 270 measure 1,916, keeping CI below the 2k-token budget. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Streaming responses now send an SSE comment (
: keepalive) every 5 s, so clients don't kill slow streams as idle.On slow hardware (a 35B-A3B MoE with SSD expert streaming on a 16 GB Mac, ~4–5 tok/s) a streaming response can carry no bytes for minutes:
X-SwiftLM-Prefill-Progress.ToolCallProcessorholds tool-call text until the call parses, so a 4k-tokenwritecall streams nothing for 15+ minutes.Node/Bun
fetchaborts a response body that has been idle for about 300 s (TypeError: terminated). The agent then retries and pays the whole prefill again. Observed with pi/little-coder: a run died on its first large file write, and every retry then timed out behind the abandoned generation. #170's early: connectedpreamble fixes the silence before headers are sent; this PR covers the silence after that.Change
startSSEKeepalive(_:interval:)yields": keepalive\r\n\r\n"every 5 s until the stream terminates or the task is cancelled.defercleanup.yieldis a complete event, so a comment can never land inside another event.Tests
ServerSSETests: comments are emitted while the stream is silent, and the keepalive task stops by itself once the stream finishes.Also in this PR
test: keep Test 38 prompt under 2k tokens for gemma-4-e2bshrinks the Test 38 streaming-TTFB prompt from 300 to 270 numbers. Measured with the gemma-4-e2b tokenizer plus chat wrapper, that's 1,916 tokens instead of 2,126, which keeps the gemma-e2b CI job below a 2k-token prompt budget. The change is identical in all three open PRs from this branch set, so it merges cleanly whichever lands first.🤖 Generated with Claude Code