Skip to content

raise the datastream WebSocket message size limit (SCH-7098) - #110

Merged
bpapillon merged 1 commit into
mainfrom
sch-7098-datastream-ws-max-message-size
Sep 8, 2026
Merged

raise the datastream WebSocket message size limit (SCH-7098)#110
bpapillon merged 1 commit into
mainfrom
sch-7098-datastream-ws-max-message-size

Conversation

@schematic-orbit

Copy link
Copy Markdown
Contributor

websockets.connect() never passed max_size, so the datastream client kept the library's 1 MiB default. Syrto's rulesengine.Flags frame is 1,125,653 bytes, so the library closes the connection with a 1009 and the client reconnects straight into the same frame — ~1,000 reconnects an hour since they crossed 1 MiB on 2026-09-04. Flag checks stay correct because check_flag_with_entitlement falls back to REST, but the datastream never delivers anything. Python is the only SDK affected: schematic-node inherits ws's 100 MiB maxPayload and the Go replicator never calls SetReadLimit.

Pass an explicit 100 MiB max_size to match Node, exposed as max_message_size on ClientOptions, DataStreamClientOptions and DataStreamConfig.

Second change in the same file: _reconnect_attempts was cleared on every successful handshake, so a failure that always lands after connecting never escalated past the ~1s backoff floor. It now clears once the connection delivers a message or stays up 30 seconds. Consequence worth a look in review — a permanently poisoned connection now exhausts max_reconnect_attempts and fires on_error instead of churning forever.

Not addressed here: 740 rules already yields a 1.1 MB frame and Airia prod is ~10x that, so a client cap is not a durable answer. EnableCompression on the datastream upgrader in schematic-api would cut this payload ~10x — worth its own ticket.

websockets.connect() was left on the library's 1 MiB max_size default, so
a Flags payload from a large environment closes the connection with a
1009 and the client reconnects into the same frame forever. Pass an
explicit 100 MiB limit, matching the Node SDK's `ws` default, and make it
overridable.

Also stop clearing the reconnect backoff on the handshake alone: a
failure that always lands after connecting kept restarting at the ~1s
floor instead of escalating to the 30s cap. The backoff now clears once
the connection delivers a message or stays up 30 seconds.
@ryanechternacht ryanechternacht self-assigned this Sep 8, 2026
@ryanechternacht
ryanechternacht marked this pull request as ready for review September 8, 2026 13:22
@ryanechternacht
ryanechternacht requested a review from a team as a code owner September 8, 2026 13:22
@bpapillon
bpapillon merged commit 20d3a00 into main Sep 8, 2026
6 checks passed
@bpapillon
bpapillon deleted the sch-7098-datastream-ws-max-message-size branch September 8, 2026 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants