feat(read): spread hot-record reads across the close group, and back off briefly from nodes that decline

[mickvandijke]

Three client changes so that many browsers reading the same chunks don't pile onto one node, so a full node isn't mistaken for a dead one, and so a full holder isn't reported as a missing record. See ADR-0023.

1. Spread first reads across the close group (client_engine/read.rs, native and browser)

• Before: both read paths ordered candidates by XOR distance, so every client sent its first GET for a record to the same nearest holder. The other six holders only got a request after the 1 s hedge.
• Now: both paths choose uniformly among the close group as currently known (the nearest CLOSE_GROUP_SIZE candidates), then fall back to nearest-first. Early reads pick among the unattempted members, and the ordinary path shuffles the close group at the front of its list.
• Unchanged: the hedge, the early-read allowance, the at-most-two-GETs rule and the absence rule. Known-peer fallbacks keep nearest-first order. Pointer reads already ask the whole close group, so they are unaffected.
• retrieve_progressive takes the choice as a parameter. Production passes uniform_choice, and tests pass nearest to keep their pinned order.

2. Back off briefly from a node that declined (browser/wasm_transport.rs, client_engine.rs)

• Before: a dial that a full node declined cost the 10 s channel-open timeout plus the 30-minute dial-failure cooldown.
• Now: a failed fresh dial is DialOutcome::Refused when the node answered it:
  • the browser declared the association failed within 3 s of ICE checks starting, before it ever connected and with ICE not connected. That is a transport refusal, sent by feat(webrtc): refuse full listeners with a STUN error instead of silence saorsa-transport#183. The test is "ICE not connected" rather than "ICE reads failed", because Chromium declares the association failed while its legacy iceConnectionState still reads "checking".
  • or ICE and DTLS completed, and the node then closed the channel, or the association died, before the PQ session was established.
• Refused endpoints are suppressed for 15 s. Every other failure keeps 30 minutes. That includes DTLS failing after ICE connected (a different node answered at the endpoint), an authentication failure, and a lane the client closed itself because of malformed data.
• A full dial-failure cache now evicts whichever record expires first, so a 15 s refusal never pushes out a 30-minute failure.
• The channel-open wait now ends as soon as the browser reports the association dead.

3. Wait out a refusal before reporting a record absent (client_engine/read.rs, data/client/chunk.rs, data/network.rs)

• Before: a refused holder is left out of discovery and reads for 15 s. A read that started while every holder it could find was being skipped retried once after 1 s, then returned Ok(None), and the browser said "record … not found". On the testnet every failed download ended this way.
• Now: after a round that finds nothing, the engine asks the adapter which peers it is skipping for a refusal, and how long each has left.
  • A skipped peer counts while it may hold the record: it did not answer NotFound itself, and fewer than a close group of nearer peers did.
  • While one counts, the round proves nothing. The read waits until the first refusal ends (at least the usual 1 s) and runs another round, up to 3 times.
  • If holders are still refusing after that, the read fails with network error: the nodes that may hold chunk … are busy; try again shortly instead of "not found".
  • A peer that was skipped when the round started but whose refusal ended during it counts as ready now. Otherwise a holder that discovery skipped could be mistaken for an absent record.
• Cost: at most 3 waits (45 s with the 15 s cooldown) and 3 extra lookups, and only for a read that would otherwise have failed.
• Unchanged: native transports report no refusals, so native reads behave exactly as before, including the single 1 s retry. Reads that fail for other reasons (timeouts, unreachable endpoints) still end in Ok(None).

Compatibility. Against nodes without saorsa-transport#183, a full node stays silent, so the client times out and applies the 30-minute cooldown exactly as before. Read spreading works with any node.

Heads-up: the ADR-0020 stack (#221) also changes client_engine/read.rs. Whichever lands second should keep the spreading rule for its holder choice.

Linear issue

Closes V2-1508

Risk tier

• T0 — docs / tooling / CI / pure UX-output. Repo CI only.
• T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
• T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
• T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Compatibility

• Wire: none. The same requests go to the same close group, in a different order.
• Storage: none
• API: the public BrowserNetwork trait gains refused_peers(), with a default that reports none, so existing adapters compile unchanged. No JS binding changes. A browser read whose holders keep refusing now rejects with a "busy" error rather than "not found".

Semver impact

• breaking
• feature
• fix

Test evidence

• cargo fmt --all -- --check and cargo clippy --all-targets --all-features -- -D warnings: clean.

• cargo clippy -p ant-core --target wasm32-unknown-unknown --no-default-features --features browser-wasm (and +test-utils) -- -D warnings: clean.

• cargo test -p ant-core --lib: 819 passed. New tests:

  • Over 7,000 trials, each close-group holder serves about 1/7 of first reads (±300), and peers beyond the close group keep nearest-first order.
  • Early reads choose within the known close group, then go nearest-first.
  • An out-of-range choice stays inside the close group.
  • An ineligible hint is never chosen over an eligible one.
  • A refusal expires after its shorter cooldown while a failure does not.
  • Eviction drops a refusal before a longer failure.
  • The cache lists live refusals with their time left, and drops them once they end.
  • With paused time: a holder that discovery skips for a 15 s refusal is read after exactly 15 s, although the seven peers discovery found instead all answered NotFound. Before this change, those answers proved the record absent.
  • A refusal that ends during a 2 s round is retried after the usual 1 s.
  • A holder refused on every attempt gives busy() after 3 waits (45 s), never Ok(None).
  • A refused peer beyond the close group does not delay an authoritative absence.

• WASM suite (node --import ./wasm-tests/setup-wasm.mjs --test ./wasm-tests/*.test.mjs): 211 passed, including the new capacity-refusal.test.mjs and refused-holder.test.mjs.

  • It covers a mock node that refuses at its transport and one that closes during the handshake.
  • Both fail the dial in under 2 s, are skipped inside the cooldown, and connect after it.
  • A third mock node that fails DTLS after ICE connected is not dialled again.
  • The test fails with refusal detection disabled, and also with an earlier "dead and fast" check that ignored ICE state.
  • refused-holder.test.mjs: a seed points at the only holder, which declines its first dial. The download completes after 15.1 s. Without section 3 it fails with record … not found, the testnet error.
  • A holder that declines every dial makes the download reject after 45.1 s with the "busy" error, which does not mention "not found".

• Real browsers, using a local 7-node devnet built against saorsa-transport#183 with limits of 2 sessions and 1 per IP:

  Browser	Turned away after (old transport → ci(release): ship version-matched release notes and rotate the changelog backlog #183)	16 s later, slot freed
  Chromium	10,011 ms → 67 ms	still blocked → connects in 132 ms
  WebKit	10,005 ms → 54 ms	still blocked → connects in 115 ms

  Playwright Firefox does not establish WebRTC Direct sessions with local nodes at all (independent of this change). Its failed dials took 5.0 s, outside the 3 s window, and correctly kept the 30-minute cooldown.

• Dev testnet A/B, 2026-10-13: 60 nodes and 64 browsers, each downloading the same 50 MiB file within 10 s. Full results are in the ADR's Validation section.

  • Spreading (section 1) cut the most popular holder's share of a chunk's first reads from 48% to 18–21%.
  • At the old node limits, this client and today's completed about the same share of downloads (61% and 66%). Every failed download ended in a false "record not found", which section 3 now addresses. With ant-node#250's limits, all 384 downloads completed.
  • Section 3 was added after the run and is tested only against the mocks above, not on the testnet.

New dependency

none. rand was already a dependency of ant-core.

ADR

ant-node ADR-0023, in WithAutonomi/ant-node#250.

Mitigation / rollback

Revert this PR's commits. That restores nearest-first reads, the single 30-minute dial-failure cooldown, and the single 1 s retry before a read reports not found. No stored state or wire format is involved.

🤖 Generated with Claude Code

[dirvine]

Hermes cross-repository review — provisional COMMENT

Reviewed commit: 4717922be49cfb373cb9c6b50f9f5788c7b62c03. The local checkout matches it and is clean. This review covers ant-client #225 alongside saorsa-transport #183 and ant-node #250.

Verdict: I have not confirmed a new blocking code defect in this diff. This is not an approval: the required review panel is incomplete, and Security Audit is still failing.

Code and contract checks

• read.rs shuffles the close group using the supplied uniform choice. It keeps peers outside that group in distance order. The progressive read still bounds concurrent GETs and preserves the final candidate-set absence rules.
• Refusing possible holders get a bounded retry path: at most three refusal waits, then a busy error instead of an absence result. The 45 seconds described in the code is the refusal-wait allowance, not a total read deadline including discovery and RPCs.
• The browser separates 15-second refusal suppression from the long failure cooldown. JavaScript does not receive the STUN 486 code directly: the code uses association/ICE state and a three-second timing window. Treat browser-version changes as a regression risk, not proof that every quick failure is a capacity refusal.

Evidence and remaining gates

• Independently executed at this head: cargo test --locked -p ant-core --lib: 819 passed, 0 failed.
• Built the browser WASM with browser-wasm,test-utils, then ran the generated WASM under Node with the repository's RTC test setup: 211 passed, 0 failed. This is not an independently run Chromium/WebKit or network test.
• ADR-0023 explicitly says the network comparison predates section 5, the refusal-wait policy. Rerun the old-limits arm with this final policy before treating the network results as validation of that fix. The reported Chromium/WebKit source-quota results are author evidence; Firefox refusal handling remains unverified in the ADR.
• Security Audit reports five quinn-proto advisories at 0.11.15. The logged solutions require at least 0.11.18 to cover all five. Cargo.lock is unchanged against the reviewed base: these affected dependency versions were not introduced by this diff. The audit remains a separate unresolved CI gate.

Panel participation and dissent

Qwen 3.8 completed correctness and adversarial roles. DS4 completed client correctness only; its adversarial call timed out. GLM 5.3 did not return a complete verdict in either role. Thus this is 2/3 models with substantive client input, but an incomplete role panel, not consensus.

Qwen's correctness role requested changes. Two allegations do not match the pinned code: DialFailures uses EndpointFailureCache<String> (wasm_transport.rs:191), so hex parsing is not a PeerId/type mismatch; earliest-expiry cache eviction is deliberate and has an explicit passing regression test (client_engine.rs:261-330), not an inverted implementation of oldest-entry eviction. The browser-state concern remains a runtime uncertainty, not a reproduced defect. DS4 returned UNCERTAIN. Neither reviewer's speculation is being presented as a confirmed finding.

Next checks: close or explicitly disposition the audit through the normal process; complete the missing named-model roles; run section 5 on a network before ADR acceptance. No merge, deployment or ADR acceptance was performed.