egress: control and name the source IP the WAF sees (socks5h, egress-tagged parse failures); fail over before first block; Bun body fix - #176
Open
maxff77 wants to merge 9 commits into
Conversation
Qwen's WAF challenges POST /api/v2/files/parse per source IP, not per account. The 529s on qwen-next on 2026-09-16 were one WARP egress (104.28.222.16) challenged with all 236 accounts behind it; the logs said WAF_CAPTCHA and nothing about which proxy, so the branch was suspected before the IP was. describeEgress(account) returns the proxy an account leaves through (credentials masked) or 'direct'. Parse failures carry it in the message (`via socks5://host:port`) and as error.egress; the rate-limiter and breaker warnings name it too. Groundwork for keying both by egress once more than one egress exists (.scratch/egress-waf). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… on breaker-open too Review findings on 3ec9b9b: - describeEgress now returns protocol//host:port via new URL(); userinfo is dropped, not masked. The old regex (/\/\/[^@]*@/) cut at the FIRST "@", so "http://user:p@ss@proxy:3128" leaked "ss@proxy", and "http://proxy?token=a@b" lost its host. The regex survives only as the fallback when URL() throws, now anchored to the authority (/\/\/[^/?#]*@/). - assertParseBreakerClosed(account) names the egress like every other parse failure; both callers already had the account in hand. Without it, a burnt proxy was indistinguishable from a Qwen outage on the one log line that matters while the breaker is open. - Tests: "@" in password/query/path, unparseable URL still masked, breaker-open error carries egress and no credentials. expected-counts 1151 -> 1154. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Under Bun, Readable.toWeb/fromWeb drops undici's body error on a mid-body close: the SSE consumer hangs and the rejection escapes as an unhandled rejection, which the runtime turns into a process exit. qwen-next died 6x in 2.5 h this way (2026-09-16, UND_ERR_SOCKET "other side closed" at 5.7 KB into the stream, always via the socks5 hop). The proxy transport now talks to undici directly: stream responses get undici's own Node Readable, everything else is buffered in the transport, and the fetch wrapper returns a buffered Response. Transport failures are mapped onto the codes request.js already retries (UND_ERR_SOCKET -> ECONNRESET, connect/headers/body timeouts -> ETIMEDOUT/ECONNABORTED) and axios' timeout becomes headersTimeout/bodyTimeout, so the HTTP-proxy path no longer cuts slow first bytes at the 10 s pool default. Repro: tools/dev-probes/repro-transport.js under bun -> 3/3 caught, 0 escapes (before: 2 escaped rejections + hang per cut). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
socks5:// makes socks-proxy-agent resolve the target hostname locally and hand the proxy a bare IP, so the DNS query leaves through the host resolver while the TCP goes through the proxy — two egresses for one request. On qwen-next 76/80 upstream connections reached sing-box as bare IPs. socks5h:// (curl semantics) delegates resolution to the proxy: the agent reads the scheme itself and sets shouldLookup=false. Accept it in the backend validator, the dashboard regex it mirrors, the accounts API error text and the three locales. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…he first block Live on qwen-next (2026-09-16, 8 h): 3 of 70 /v1/messages streams ended in undici `UND_ERR_SOCKET: other side closed` with 63–90 KiB already read, and Claude Code showed "Connection lost mid-response". Until now that surfaced as a 500 with nothing the client could do. - upstream-error.js#isTransportInterruption: socket closes and timeouts as Node, undici and streams emit them. Excludes client cancels (AbortError / ERR_CANCELED) and anything carrying an HTTP response. - sse.js#consumeSSEStream counts bytes, and stamps upstreamBytesRead / upstreamEventCount on whatever error escapes — the controller uses it to decide whether a replay is safe, the [EGRESS] log to tell early from late cuts. - anthropic.js#handleAnthropicStream: one failover per request, only while no content block has been emitted (replaying after visible text would duplicate it), for transport cuts and for a RateLimited first frame. The burned account is paused (recordFailedAccount) before the pool is asked again with excludeEmails, so the retry never lands on it. - request.js#sendChatRequest forwards excludeEmails to the rotator. - The outer 429 handler no longer marks an account the failover already marked (error.accountFailureRecorded) — same guard chat.js already had. Without it, a single-account deployment logged the quota exhaustion twice. Tests: anthropic-midstream-failover (9), sse-interrupt-metrics, and the upstream-quota-429 harness now honours excludeEmails like the real rotator. Gate blessed at 1175 tests / 136 suites (was 1159/134). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…iled - request.js#isRetryableNetworkError delegates to upstream-error.js#isTransportInterruption and only adds the two connect-phase codes (ECONNREFUSED, EAI_AGAIN). Same guard against HTTP responses, same message sniff, one list to maintain. proxy-helper's UNDICI_ERROR_CODES stays: it is a mapping onto Node codes, not a classifier. - anthropic.js: when the stream dies and no failover is left, the error carries failedAccountEmail (email only) so the handler's catch marks the account that actually served -- same idiom as chat.js. Drops the `let ctx = null` hoist. - anthropic.js: attemptsMade counts only turns the model answered; failover iterations skip the increment instead of decrementing after the fact. Tests: +1 (isRetryableNetworkError); two failover assertions now check failedAccountEmail. Gate 1175/136 -> 1176/137. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…y stream without a yield-less generator CI on main (928f816, PR Rfym21#175) is red on four no-useless-assignment errors: `let promptTokens = 0` / `let completionTokens = 0` at the top of both Anthropic handlers are overwritten unconditionally after the upstream loop, so the zero is never read. Declare them `const` at the assignment instead. tests/sse-interrupt-metrics: the "failure before any byte" case built its stream from an async generator whose only statement is `throw`, which trips require-yield. A Readable that destroys itself on first read models the same thing (error before any data) without the generator. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…status polls A socket hang-up or ECONNRESET on the parse POST (or on /files/parse/status) used to surface as a bare axios error: no hint of which proxy the request left through, so a broken WARP sidecar and a broken direct route looked the same in the logs. Wrap both calls with the same helper the OSS/STS path already uses: the error keeps its code, gains `egress` (protocol//host:port, credentials dropped) and the message ends with `(file_id, via <egress>)`. Docs: the proxy sections listed HTTP/HTTPS/SOCKS5 only; `socks5h://` has been accepted for a while and is what qwen-next actually runs, so list it and say what the `h` buys (DNS resolved at the proxy). Test: transport failure on the parse POST through `socks5h://user:s3cr3t@lohari-warp-qwen:9091` -> code ECONNRESET, egress `socks5h://lohari-warp-qwen:9091`, secret absent from the message, zero status polls. Suite blessed 1176 -> 1177. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
maxff77
changed the title
Author
|
Heads-up: the CI run for this head is waiting on maintainer approval (fork PR → |
`docker-compose.next.yml` sets `API_KEY=${QWEN2API_ADMIN_KEY_V2},${QWEN2API_API_KEY}`.
Deployed without the admin variable in scope it expands to `,sk-client…`; the parser
filtered the empty slot away, so `apiKeys` held one key and `adminKey` silently became
the CLIENT key. The dashboard then answered "invalid API key" to the real admin key —
an auth symptom for what was a deploy bug. qwen-next lost its dashboard this way on
2026-09-16 (recreating with `doppler run --` restored both slots).
Nobody writes an empty slot on purpose, so parseApiKeys now throws instead, naming how
many slots are empty and which positions. An unset API_KEY is untouched: that is the
separate already-handled "no keys configured" case. parseApiKeys takes the raw value as
an argument (defaulting to process.env) and is exported so the refusal is testable.
Gate 1177 -> 1181.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Nine commits on top of
main(928f816), all around outbound egress and what the logs say when it breaks.socks5h://accepted (afd1a79). DNS resolves at the proxy instead of on the host.resolveProxyUrlused to reject it as invalid and fall back to DIRECT, which on a shared host means a different source IP than intended — and Qwen's WAF is per source IP.6807ac1,cb28e12,81b5526)./files/parse,/files/parse/status, the STS/OSS upload and the breaker-open path all attacherror.egress=protocol//host:port(credentials stripped) and end the message with(file_id, via <egress>). Before this, a dead SOCKS sidecar and a dead direct route produced the same bare axios error.b2ebd25). Under Bun the previous bridge closed the socket mid-response on large SSE bodies; the transport now hands undici's Node stream straight through./v1/messages: fail over before the first block (64f8593,7ae0af0). If the upstream stream dies before any content block was emitted, retry on another account instead of surfacingConnection lost mid-response; once a block has been sent the failure is delivered as before. One transport classifier (TRANSPORT_INTERRUPTION_CODES/_MESSAGE_RE) shared with the OpenAI path; the stream tags the account that failed.4b39af1): the fourno-useless-assignmenterrors that turnedmainred after fix(usage): report Qwen's real usage instead of the local estimate #175 (verify / regression, run 35110797317). Usage tokens declared where assigned; empty stream fails without a yield-less generator.API_KEYwith empty slots (dc54f99). A compose file that buildsAPI_KEY=${ADMIN},${CLIENT}and is deployed withoutADMINin scope expands to,sk-client…;parseApiKeysfiltered the empty slot away and silently promoted the CLIENT key to admin, so the dashboard answered "invalid API key" to the real admin key. An empty slot is never intentional, so it now throws at startup naming how many slots are empty and where. An unsetAPI_KEYis untouched (the separate "no keys configured" path)..env.exampleand the three READMEs list SOCKS5H and say what thehbuys.Tests
bun run lint: 0 errors.bun testgate: 1181 tests / 137 suites, 0 fail.tests/expected-counts.jsonblessed 1146/134 → 1181/137 (+35 tests, +3 suites). New suites:anthropic-midstream-failover,proxy-egress-describe,proxy-socks5h,proxy-transport-no-bridge,sse-interrupt-metrics,config-api-key-slots. Extended:upload-parse-status(transport failure on the parse POST names the egress, secret absent),upstream-quota-429.dc54f99): https://github.com/maxff77/qwen2api/actions/runs/35129316759 (green — lint + the full gate).Live
Running on a staging gateway behind a
socks5h://WARP sidecar since 2026-09-16 (imagehead-dc54f99):/v1/models200,/v1/messagesnon-stream 200 with real upstream usage, admin and client keys resolving to their own roles (403 for the client key on admin routes), no[ERROR]lines. The boot guard was exercised against the built image:API_KEY=",sk-client"exits 1 with the diagnostic instead of starting.🤖 Generated with Claude Code