Skip to content

Drain the accepted streams after a graceful GOAWAY - #31

Merged
samuel-williams-shopify merged 6 commits into
socketry:mainfrom
senid231:graceful-goaway-drain
Aug 31, 2026
Merged

Drain the accepted streams after a graceful GOAWAY#31
samuel-williams-shopify merged 6 commits into
socketry:mainfrom
senid231:graceful-goaway-drain

Conversation

@senid231

@senid231 senid231 commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

What this fixes

Connection#receive_goaway closes the connection immediately, whatever the error code. For a
graceful GOAWAY (error code 0) that is wrong: the peer is telling us it will not accept new
streams, but it is still processing the streams at or below last_stream_id and will send
their responses (RFC 9113 §6.8).

Because close! runs straight away, closed? is true, the caller stops reading, and
Connection#close then raises EOFError: Connection closed with N active stream(s)! into
exactly those streams — requests the peer has already processed, whose side effects have
already happened, and which cannot be retried if they are not idempotent.

nginx does this on the keepalive_requests-th request of every HTTP/2 connection (default
1000), on keepalive_time (default 1h), and on every reload, so a client with requests in
flight loses a burst of them each time.

The change

On GOAWAY with error code 0:

  • record that a GOAWAY was received instead of closing the connection;
  • close the streams above last_stream_id with Protocol::HTTP::RefusedError (unchanged) and
    remove them from the connection, so what remains is exactly the set of streams we are waiting
    for;
  • close the connection when the last of those streams completes, or immediately if there are
    none.

A non-zero error code still closes the connection immediately and raises GoawayError, as
before.

The state of the connection is decided before any stream's closed hook runs, so that a hook
which raises cannot leave the connection in an undecided state, and one which creates a stream
cannot defeat the "nothing left to drain" check.

Connection#create_stream now refuses to open a locally-initiated stream once a GOAWAY has
been received, since RFC 9113 §6.8 says receivers of a GOAWAY must not open additional
streams. Streams the peer initiates — a server pushing on a stream it already accepted — are
not covered by last_stream_id and remain legal, so they are still accepted.

Adds Connection#goaway_received?, which a client uses to stop offering the connection to new
requests once a GOAWAY has been received. The remaining lifecycle is expressed by the
concrete closed? state and the registered streams; no derived draining predicate is exposed.

Behaviour change

closed? is no longer true immediately after receiving a graceful GOAWAY — it becomes true
when the last accepted stream completes. Anything that gated new requests on !closed? should
gate on !goaway_received? instead; create_stream now enforces this for locally-initiated
streams.

The existing "client can handle graceful shutdown" test is updated to assert the new lifecycle,
and there are new tests for draining several accepted streams, for the case where there is
nothing to drain, for the create_stream restriction, for push promises arriving during a
drain, and for a stream callback which raises.

Details

The full write-up, a reproduction against real nginx, and the client-side half of the fix (pool
reuse and connection close) are in socketry/async-http#245. This change alone is not enough:
without it the connection is closed under the in-flight streams, and without the async-http
side the pool keeps handing the draining connection to new requests.

Types of Changes

  • Bug fix.

Contribution

Connection#receive_goaway closed the connection immediately, whatever the error
code. For a graceful GOAWAY (error code 0) that is wrong: the peer will not
accept new streams, but it is still processing the streams at or below
last_stream_id and will send their responses (RFC 9113 6.8). Closing at once
makes those requests fail with "Connection closed with N active stream(s)!"
even though the peer processed them, which for a non-idempotent request cannot
be retried.

On a graceful GOAWAY the connection now records that it is going away, refuses
and removes the streams above last_stream_id as before, and closes when the last
accepted stream completes. A non-zero error code is unchanged.

The state of the connection is decided before any stream's closed hook runs, so
a hook which raises cannot leave it undecided and one which creates a stream
cannot defeat the "nothing left to drain" check. create_stream refuses to open a
locally-initiated stream once a GOAWAY has been received; streams the peer
initiates are not covered by last_stream_id and stay legal. Connection#close
detaches the active streams before closing them, so a re-entrant close cannot
report a fabricated EOFError in place of the real error.

Adds Connection#goaway_received? and Connection#draining?, which a client needs
in order to stop offering the connection to new requests while it drains.
Assisted-By: devx/618580b0-d55f-4c2e-95b6-87e648f60543
Assisted-By: devx/618580b0-d55f-4c2e-95b6-87e648f60543
@samuel-williams-shopify

Copy link
Copy Markdown
Contributor

Removed the generic Connection#close stream-detachment change from this PR. It was intended to make close callbacks re-entrant-safe (and would make draining? false before callbacks), but it is not required for graceful GOAWAY handling and changes shutdown behavior outside this feature. It also had no focused regression coverage and could leave later detached streams unclosed if one callback raised. Verified with the original close behavior: protocol-http2 239/239 tests (1,194 assertions), async-http HTTP/2 17/17 tests (49 assertions). Any re-entrant close work should be handled separately with dedicated tests.

Assisted-By: devx/618580b0-d55f-4c2e-95b6-87e648f60543
Assisted-By: devx/618580b0-d55f-4c2e-95b6-87e648f60543
Assisted-By: devx/618580b0-d55f-4c2e-95b6-87e648f60543
@samuel-williams-shopify
samuel-williams-shopify merged commit c9521a2 into socketry:main Aug 31, 2026
20 checks passed
@senid231
senid231 deleted the graceful-goaway-drain branch August 31, 2026 07:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants