Skip to content

fix(llm): stream Bedrock ConverseStream events as they arrive - #167

Merged
wolfy-j merged 2 commits into
wippyai:masterfrom
VladKhomich:fix/bedrock-converse-stream
Oct 9, 2026
Merged

wolfy-j merged 2 commits into
wippyai:masterfrom
VladKhomich:fix/bedrock-converse-stream

Conversation

@VladKhomich

@VladKhomich VladKhomich commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Bedrock ConverseStream responses were buffered until the model finished, so callers received the entire answer in one burst. Parse and dispatch each event as bytes arrive. On wippy/llm 0.5.5 with Claude Haiku 4.5 in us-east-1 (three runs each), first-chunk latency changed from 9.7–12.7 seconds to about 1 second; chunks now arrive throughout the 10–12-second generation.

The reader also reports AWS exception messages through the existing error callback, skips every supported eventstream header value type without losing later headers, treats an empty read as pending data, and rejects a stream ending mid-frame. Impossible frame lengths fail immediately: a frame must include the 12-byte prelude and 4-byte CRC, and its headers must fit within that declared length. This prevents malformed zero-length frames from making the parser loop forever, including after a valid streamed event.

The InvokeModel path and public (content, err, result) return contract are unchanged. CRC values remain unverified, as before.

Validation: nine new streaming/parser tests cover per-event dispatch, split frames, empty reads, AWS exceptions, non-string headers, truncation, zero/undersized lengths, oversized headers, and malformed frames following valid content. The focused Bedrock client suite passes 13/13 tests; Bedrock namespace lint passes for 16 entries. Astra reviewed the frame-length follow-up and found no blockers. Full CI is rerunning on the updated branch.

@VladKhomich
VladKhomich force-pushed the fix/bedrock-converse-stream branch from 4232a73 to f7622ad Compare October 8, 2026 12:10
@wolfy-j
wolfy-j merged commit d623e6e into wippyai:master Oct 9, 2026
24 checks passed
@wippy-releases wippy-releases Bot mentioned this pull request Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants