feat(read): read chunks through a get-or-closer lookup (ADR-0020)

[mickvandijke]

Native chunk reads now run their own iterative lookup, and ask each peer they query for the chunk as well as for closer peers. Browsers keep today's read. This implements ant-node ADR-0020.

How it works.

• The lookup runs on saorsa-core's public lookup engine, with the same alpha, grace period and deadline as saorsa-core's own lookup.
• Peers announcing get-or-closer/1 get one get-or-closer request. saorsa-core checks the peers in a Closer answer as a FIND_NODE answer. saorsa-core rejects a payload over the 64 KiB FIND_NODE limit before parsing it. A payload that fails any check fails only that peer's step.
• Every other peer gets FIND_NODE and a GET at once, in the same round trip. Every node in service today takes this route.
• The read stops at the first verified chunk.
• A peer whose content fails verification only drops out of the lookup. That covers a wrong address or a hash mismatch. Every queried peer is asked for the chunk, so ending the read on one bad answer would let any dishonest node on a lookup path fail downloads. The client logs a warning naming the peer. If no other peer sends the chunk, the ordinary read path runs, and its rule that bad content ends the read is unchanged.
• Likely holders are asked one at a time. Once the lookup knows at least 14 peers, a peer with fewer than a close group of known peers nearer the chunk is a likely holder.
  • Likely holders are asked closest first, one at a time.
  • One that answers with closer peers hands over at once; one still sending after 500 ms is hedged.
  • Far peers, and every peer of a cold lookup, are still asked in parallel.
• Requests belong to the read, not to the lookup round. A round's grace period therefore never cancels a chunk that is still arriving.
• If the lookup finds no holder, the existing retrieve_progressive path runs unchanged, with its absence rule and retry round.
• Browsers keep today's read. A browser variant was built and A/B'd on the dev testnet, and was not faster there (see Test evidence), so it is not part of this PR. Browser read tuning is in a separate PR, perf(browser): use the whole read budget and hedge after a DataChannel transfer #222.

Code layout.

• client_engine::kad_read: the policy, transport-neutral and tested on a scripted network.
• data::client::kad_get: the adapter, over saorsa-core and the chunk protocol.
• Test harness: in-process nodes now announce the token, so the e2e suite reads through get-or-closer.

Measurements. See the investigation: sequential reads on production, native.

	p50	p90	Whole 612 MB file
Before	5.95 s	13.0 s	199–205 s
After	~1 s	~2 s	49–55 s

Without the likely-holder gate, this read fetched 2.5–3.3 copies of a chunk on production, a trade-off accepted in the ADR. The gate trades a little latency for fewer copies (see Test evidence).

Stack, merge last:

1. feat(dht): expose lookup steps for caller-driven lookups saorsa-core#173
2. feat(chunk): add a get-or-closer request ant-protocol#44
3. feat(storage): answer get-or-closer requests (ADR-0020) ant-node#249
4. this PR

This PR pins all three, plus saorsa-transport, by git rev.

Linear issue

Closes V2-1358

Risk tier

• T0 — docs / tooling / CI / pure UX-output. Repo CI only.
• T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
• T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
• T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Compatibility

• Wire: sends the new GetOrCloserRequest only to peers whose user agent carries get-or-closer/1. Every other peer gets FIND_NODE and GET, both existing messages.
• Storage: none.
• API: additive. #[doc(hidden)] Client::chunk_get_kad_only, behind test-utils.

Semver impact

• breaking
• feature
• fix

Test evidence

• cargo test -p ant-core --lib: 822 passed. That includes 12 new kad_read tests on a scripted network and 1 in kad_get (25 consecutive runs clean):

  • a holder found by get-or-closer ends the read
  • older peers get FIND_NODE and a GET in the same step
  • a mixed network
  • a transfer outlives its round's grace period
  • an unreachable peer only leaves the lookup
  • corrupt content from one peer leaves the read to the others
  • corrupt content from the only holder is not-found, so the ordinary path decides
  • an error the adapter rejects still ends the read (the engine's fatal path)
  • the native policy treats bad content as that peer's failure
  • no holder reports not-found
  • likely holders are asked one at a time
  • a slow likely holder is hedged
  • a likely holder without the chunk hands over at once

• CI's e2e list (--features test-utils --test e2e_chunk … --test e2e_adr0004 -- --test-threads=1): 59 passed.

• test_chunk_get_or_closer_reads_from_upgraded_nodes checks that a network of announcing nodes is read with get-or-closer requests only, and no plain GET.

• cargo clippy -p ant-core --all-targets -- -D warnings, and with --features test-utils: clean.

• Live, against today's production nodes, which do not announce the token. ant file download of a 612 MB file took 56 s, against 242 s for the current client. The output was byte-identical, and all 150 chunks came through the new lookup's FIND_NODE-plus-GET route.

• Dev testnet comparison against main, matched (2026-10-09). Two 60-node fleets on the same layout:

  • 30% symmetric NAT across 9 regions.
  • 4 uploaders (50 MiB) and 2 verified downloaders per fleet, over a 60-minute window.
  • Both fleets on the rc-2026.10.1 release baseline. The clients differ only by the ADR-0020 commits.
  • Downloads paced to one every 40 s per downloader, so both fleets serve the same reads.

  	main	this stack	delta
  download p50 / p95 / max	20.5 / 28.6 / 35.5 s	9.7 / 15.5 / 20.4 s	−53% / −46% / −43%
  downloads ok / failed	176 / 0	176 / 0	same load
  read bytes nodes sent per byte downloaded	1.35	1.73	duplicate copies
  no-merkle uploads ok, p50 / p95	211, 32.9 / 45.6 s	211, 33.2 / 45.1 s	identical
  merkle uploads ok, p50 / p95	130, 54.1 / 66.2 s	112, 62.9 / 76.1 s	−14%, +16% / +15%
  node CPU mean	27.2%	26.3%
  node restarts	0	0

  • The routing works on a real network: since the redeploy, this stack's nodes served only get-or-closer requests (127 before T0), and main's only plain GETs.
  • The merkle upload gap is not attributed to this change:
    • It does not touch the merkle payment, quote or PUT path.
    • Non-merkle uploads, which run the same lookups and PUTs, are identical.
    • The kad fleet's chain (Anvil) droplet showed 2.8× the CPU steal of main's, and merkle uploads lean on it hardest.
    • A run with the code swapped between the two fleets' droplets would settle it.
  • An earlier, unmatched run is superseded. Its clients linked different saorsa-transport versions, and its downloaders ran closed-loop, so the faster fleet carried 1.85× the reads.
  • Full report: ant-testnet/state/comparisons/v2-1358-kad-vs-main-matched-60m-1009/README.md.
  • That run predates the likely-holder gate.

• Likely-holder gate, A/B on the same kad fleet. The client binary was swapped on two downloader droplets (syd1, fra1), and arms alternated over 50 MiB files not read before, with 5–6 downloads per arm in each cell. Each cell gives the change against asking every likely holder at once, as download time p50 and then bytes received:

  hedge	syd1, fetch cap 4	fra1, cap 4	syd1, cap 16	fra1, cap 16
  1 s	+13%, −36%	−2%, −33%	+15%, −28%	+18%, −16%
  500 ms (this PR)	+9%, −14%	−2%, −22%	+3%, −17%	+8%, −24%

  On these datacenter links bandwidth is not the bottleneck, so the gate shows its latency cost and not its benefit. Client CPU per download fell 10–17%.

• Browser variant, tried and dropped (2026-10-12). It ran this read driver over WebRTC Direct, with nodes answering the request inside chunk_protocol (a missing chunk got the node's browser find_node reply). The kad fleet's nodes were swapped in place to that build, keeping their data. Headless Chromium on two downloaders, 50 MiB private files, a fresh client per download. That WASM build was interleaved with this PR's, which reads as today, over 8 runs. Pairs are by round; the intervals resample whole runs:

  region	pairs	variant / today, mean	delta	faster in
  fra1	84	16.6 / 14.1 s	+17% (−1..+43%)	30 of 84
  syd1	85	36.3 / 38.2 s	−5% (−14..+6%)	45 of 85

  • Why, from traced reads. On 60 nodes, a warm browser already holds sessions with the close group, so today's read sends its first GET straight to a holder. The variant cannot shorten that. Its first round asks the 2–4 recent routes, which often do not hold the chunk. The holder is then asked a round later, after a dial and a wait for one of the browser's 4 read slots: about 0.5 s later per chunk on fra1. Its delivering request is faster, which offsets that only where transfers are long (syd1).
  • Seven variants, none committed, changed what the first round asks, the hedge, and which steps hold read slots. None beat the plain variant on fra1.
  • Full report: ant-testnet/state/comparisons/v2-1358-kad-vs-main-matched-60m-1009/README.md, sections "Browser get-or-closer reads" and "Why the browser read is not faster here".

• Browser (WASM) downloads are unchanged by this PR. Headless Chromium on the testnet's downloader droplets, 50 MiB private files, a fresh client per download, with main's and this PR's WASM builds interleaved:

  • Interleaved, 8 downloads each: fra1 17.4 s against 17.1 s p50, syd1 31.4 s against 30.8 s.
  • Pooled over every run, with 32–38 downloads per build per region, the means were within 2%.
  • Both builds took the same time against main's nodes and against this stack's nodes.
  • For WASM, those builds differed only in ant-protocol's two added enum variants.

New dependency

none

ADR

ADR-0020: Kademlia-style chunk GET: return the chunk or closer peers (ant-node). Measurements are in the investigation.

Mitigation / rollback

Client-only with no stored-format change: revert this PR. When the new lookup finds nothing, the existing read path still runs.

🤖 Generated with Claude Code

[dirvine]

Changes requested — consume a size-bounded closer-peer decoder

ant-core/src/data/client/kad_get.rs:106–115 passes untrusted peers directly to the new core decoder. The exact core revision pinned at root Cargo.toml:11 lacks the 65,536-byte pre-deserialisation limit; the outer chunk message permits up to 5 MiB. A local exact-head probe accepted 1,050,035 bytes containing 10,000 nodes. No crash or memory exhaustion was demonstrated.

This is the downstream exposure of the same P2 finding in saorsa-core#173, not a separate read-engine bug. Fix the public decoder in core, update this PR's core pin and lockfile (and node pin as the stack advances), then rerun focused and real-transport E2E tests. A retryable client-local size guard can mitigate this caller, but the public decoder still needs its own invariant.

No other introduced blocker was established in the read engine. Legacy fallback, content verification and native/browser separation were reviewed.

Reviewed commit: 7bc58ccb70081b8c5c911cc87a821108d8aadcf4.

Review method: six substantive seats across the linked set, including a completed z-ai/glm-5.2 API review recovered from a worker that timed out after saving its response; coordinator source checks and local execution. Initial functional/security/test seats recommended approval. The coordinator probe, independent adjudicator, API seat and replacement concurrency/release seats corroborate the missing decoder bound. This is not unanimous approval. The API seat recommended client approval with concerns; the coordinator withholds it because the exact client pin still exposes the defect.

Local verification (macOS, rustc 1.96.0): core closer-peers 3 passed and DHT-manager regression 143 passed; protocol all-features 108 passed; node get-or-closer unit tests 3 passed and real-transport E2E 1 passed; client kad_read unit tests 10 passed and real-transport E2E 1 passed. Formatting passed in all four repositories. No production/testnet deployment or merge performed. Draft git pins were verified with cargo metadata --locked; this is not published dependency closure. ADR-0020 remains Proposed.

[mickvandijke]

Thanks. This is addressed in two commits.

25d7c5a: bounded decoder.

• Pins saorsa-core fd37775 (the bounded decoder) and ant-node ad88647.
• A payload the decoder rejects surfaces as Error::Protocol. The kad read treats that as retryable, so only that peer's lookup step fails.
• I didn't add a separate client-side size check, since every closer-peers payload goes through the core decoder.

7d9a248: bad content drops only that peer. Tracing the rest of the answer path turned up a larger exposure in the same request.

• A Found that failed verification (wrong address or hash mismatch) raised InvalidData.
• That ended the read, and the ? in chunk_get_from_closest_peers skipped the fallback. The file download doesn't retry InvalidData either.
• Since get-or-closer goes to every queried peer, any dishonest node on a lookup path could fail downloads.
• Now such a peer only drops out of the lookup, and the client logs a warning naming it.
• If no other peer sends the chunk, the ordinary read runs, with its existing rule that bad content ends the read.
• New tests cover a corrupt peer alongside an honest holder, and a corrupt only-holder. Both fail under the old policy.
• ADR-0020 is updated to match (ant-node#249, 900ff66).

Checks:

• cargo test -p ant-core --lib: 822 passed, with the kad tests run 25 times clean.
• The real-transport e2e test_chunk_get_or_closer_reads_from_upgraded_nodes passed.
• Native, test-utils and WASM clippy are clean.
• CI on 25d7c5a was green (18/18). CI on 7d9a248 is running.

[dirvine]

Re-review — blockers resolved

Fixes verified

1. Core pin: saorsa-core now fd377754 (the bounded closer-peer decoder) in Cargo.toml and Cargo.lock. The first-round blocker (client passing untrusted peers to an unbounded decoder) is resolved — the decoder now rejects payloads over 64 KiB before postcard parsing.
2. ant-node pin: now ad886472 (ADR-0020 branch, carries the fixed core pin — see the coordination note on ant-node#249 about mid-branch vs head).
3. New: bad-content peers are dropped, read continues. is_retryable (kad_get.rs:155–160) now includes Error::InvalidData; note_bad_content logs and passes it through. In kad_read, a retryable failure drops the peer (Ok(None)); non-retryable local errors (e.g. Config) still abort. Tests: corrupt holder + honest slower holder → chunk found; corrupt-only holder → Ok(None) → fallback to ordinary read; adapter-rejected error ends the read.

Local verification at head 7d9a248b

• client-engine 12/12, adapter 1/1, e2e 1/1 ok (macOS). CI: E2E ubuntu/macos, Merkle E2E ubuntu/macos, unit tests, builds all green.
• CI red check: Security Audit fails — cargo audit reports 19 vulnerable dependencies, all inherited from the pinned saorsa-core git rev (e.g. tokio-rustls 0.24.1 RUSTSEC-2024-0336/0337, serde 1.0.228 RUSTSEC-2025-0049). Not introduced by this PR's code; resolves when the core rev is updated. Flagging for the core maintainer.

Verdict

APPROVE at head 7d9a248b, subject to: (a) the ant-node pin reconciliation (see #249), (b) the core cargo audit failures being tracked in the core rev. Panel: GLM 5.3 seat billing-failed (402) — 2/3 reviewed, provisional.