diff --git a/docs/adr/ADR-0024-fetch-requesters-stay-within-the-responder-share.md b/docs/adr/ADR-0024-fetch-requesters-stay-within-the-responder-share.md new file mode 100644 index 00000000..909a67f8 --- /dev/null +++ b/docs/adr/ADR-0024-fetch-requesters-stay-within-the-responder-share.md @@ -0,0 +1,282 @@ +# ADR-0024: Fetch requesters stay within the responder's per-peer share + +- **Status:** Proposed +- **Date:** 2026-10-13 +- **Decision owners:** +- **Reviewers:** +- **Supersedes:** none +- **Superseded by:** none +- **Related:** [ADR-0005](ADR-0005-replication-repair-hardening.md) (the repair fetch pipeline), + [ADR-0017](ADR-0017-bounded-fresh-offers-and-copy-free-sends.md) (send-path bounds) + +## Context + +A replication fetch is a request/response exchange between two nodes. The responder bounds what +one requester can occupy: + +- `FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER = 2` admitted requests per source, checked before the + 12-slot admission pool, in front of a 3-worker serving pool; +- a request past that share is not answered. The responder logs + `Fetch response dropped at admission: reason=per_peer_cap_full` and returns nothing. + +The requesting side never matched that share: + +- the fetch worker ran one fetch per hardware thread; +- it sent each one to the first verified holder of its key, in source-preference order (the + closest holder that answered Present); +- the fetch queue is nearest-first, so the keys at its head usually share the same few holders. + +When more than two fetch slots were available and several queued keys shared a holder, the worker +could send more than that holder's two-request share. A request arriving while two from the same +requester were still admitted was dropped. Each such drop cost the requester: + +- the full fetch timeout (30 s) before moving on; +- a `ConnectionTimeout` trust event, recorded by the transport against the holder that dropped it, + although that holder had done nothing wrong; +- a retry that went to the next holder without regard to how busy that holder was. + +On the production fleet this was common, according to the per-service cumulative counters: + +| period | inbound fetch requests dropped at admission | +|---|---| +| on 0.21.0 | 36.6% | +| on 0.22.0 | 44.3% | +| 25-hour window containing a wave of 177 freshly installed nodes filling their ranges | 65.8% overall by cumulative counters; 3.22M WARN-line drops came from the new nodes, about 2.8 per chunk attributed to them | + +In that window, 68.5% of drops happened with only 2 of the responder's 12 admission slots in use, +both held by the refused peer. + +## Decision Drivers + +- No wire, storage or production public API change, so the fix ships as a behavioural change on + the release train. +- Do not weaken the responder's protection against one peer occupying the whole serving pool. +- A key must never be lost, tracked twice, or released from bootstrap accounting early because of + scheduling. +- Keep the queue's nearest-first priority and each key's source preference whenever they do not + conflict with the share. +- No scenario may drain more slowly than before the change by more than one fetch timeout, in + particular the keys of a holder that has stopped answering. + +## Considered Options + +1. **Raise the responder's per-peer share.** Rejected. It hides the symptom, lets one requester take + all three serving workers, which is what the share exists to prevent, and still drops requesters + with more threads than the new share. +2. **Answer an over-share request with an explicit "busy" reply.** Deferred. It needs a new + response variant, which peers that cannot decode it treat as malformed and charge an application + failure, so it is a wire change with its own rollout. It removes the cost of a drop, not the + cause. +3. **Limit fetch concurrency to 2.** Rejected. It serialises a node's whole repair and bootstrap fill + through one share even when other holders are idle. +4. **Ship the scheduler switched off, behind a per-node switch, for a later canary.** Rejected. It + merges code that no node runs, so its risks are deferred rather than measured. The risks were + simulated instead (below), and the one regression found was fixed before merge. +5. **Keep at most the share in flight per answering holder on the requesting side, and give each + free slot to the nearest queued key that can start now.** Chosen. + +## Decision + +The fetch worker keeps at most `FETCH_MAX_IN_FLIGHT_PER_SOURCE` fetches in flight to any one source +peer that is answering. The constant is defined equal to `FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER`, +the share every deployed responder grants. It is the worker's only scheduling path. + +**Dispatch.** Each free fetch slot goes to the nearest queued key that can start now. In the key's +source-preference order, over its untried sources, it starts on: + +1. the first source that is answering and below the limit; +2. otherwise the first silent source (below), past the limit. + +**Parking.** + +- A key whose untried sources are all answering and at the limit is parked: it leaves the fetch + heap and is indexed under each of those sources. +- It becomes a candidate again as soon as any of them has a free share or falls silent, and it + competes with the heap on distance. +- Each key leaves the heap once, so a startable key further down is reached in the same pass, and + no source ordering can hide a startable key. + +**Retries.** + +- A failed fetch moves to its next untried source by the same rule. If every untried source is + answering and at the limit, it is parked rather than sent to a full holder. +- It keeps the sources it already tried, so none is asked twice in one verification cycle, and it + keeps its verification retry reservation. +- When no untried source is left, or the fetch queue is full, it returns to verification as before, + so the fetch queue's hard bound holds. + +**Silent sources.** The limit protects a holder's admission share, and only a holder that answers +has one. Without an exception, the keys held only by a holder that stopped answering would be tried +two per fetch timeout instead of one per free slot: four times more slowly than before on a node +with eight slots. + +- Each attempt records whether its source answered. Requests and answers take places in one + process-wide sequence, as a request is sent and as an answer returns, so they can be ordered + exactly, whatever order their outcomes are applied in and whatever the clock reads. +- An answer reaches the scheduler as soon as it arrives, before its chunk is checked and stored, + and again with the attempt's outcome, to no further effect. A timeout that an answer already + outdates therefore never marks the source while that answer's chunk is still being written. +- A source is **silent** while a request sent to it after its latest answer has gone without any + response (a timeout or connection failure). Any answer received after that request was sent ends + it. +- A silent source is fetched from past the limit, as before the change, until it answers again, + or until this node has learned of no newer unanswered request to it for two fetch timeouts (at + least one second). A holder that stays dead while this node keeps fetching from it therefore + stays silent, and one this node stopped fetching from is probed under the limit when it is next + used, marked again only if a request sent then also goes unanswered. +- The table of observations is bounded at 8,192 sources. It drops only entries that hold no mark and + whose source has no fetch in flight. If that frees nothing, a source not in the table counts as + silent for one mark lifetime, which is again the behaviour before the change, never a slower one. + +**Source limit.** + +- The sources a queued or in-flight key is associated with, tried ones included, are bounded by + `close_group_size + paid_list_close_group_size + MAX_HINT_SOURCES_PER_KEY` from the running + configuration. +- That never truncates a verification round's own sources, and it bounds the parked index. +- A re-advertisement of a queued or in-flight key merges into the verification entry kept for its + retry under `MAX_HINT_SOURCES_PER_KEY`, the bound a pending entry has, in the same order: replica + claimants first, displacing paid-only advertisers other than the key's capacity owner, then + paid-only advertisers while there is room. Before the change that merge had no bound. + +**Dispatch summary.** The fetch worker logs `replication fetch dispatch summary` every 5 minutes and +at shutdown, with: + +- the parked count, the index size and the oldest parked age; +- queued and in-flight fetches, and sources at the limit; +- parks and retry parks; +- attempts by outcome, bytes stored, and attempt latency; +- the longest fill-pass lock wait and hold, and the longest parked-index maintenance after a + departure; +- silent sources now and at peak, marks, clears, expiries, ignored stale timeouts, starts on silent + sources, unreachable attempts, tracked sources and table overflows. + +Responder admission, the fetch timeout, fetch concurrency and the wire protocol are unchanged. + +## Risk simulation + +A local 12-node network, eight fetch slots per node and a 5 s fetch timeout, timed each scenario on +this change and on a build with the limit removed and no silent-source handling, which schedules +exactly as before the change: each key to its first untried source. + +| scenario | this change | before the change | +|---|---|---| +| fill: 48 keys held by 3 answering holders | 0.36 s | 10.08 s | +| a non-answering holder first among 3 sources, 2 answering: 24 keys | 5.1 s | 20.1 s | +| a non-answering sole holder: 16 keys | 15.0 s | 10.15 s | + +The third row is the accepted penalty: the first two requests to a holder that just went silent take +one fetch timeout to show it, and the rest then go out past the limit. It is paid once each time +this node starts fetching from a silent holder again after two fetch timeouts without trying it. +For `N` keys held only by that holder, `C ≥ 2` fetch slots and fetch timeout `T`, draining takes +`T × (1 + ceil((N − 2) / C))` against `T × ceil(N / C)` before the change. With one fetch slot the +first wave is one request, and the bound is the same with 1 in place of 2. Without the +silent-source exception the same scenario took 40.1 s. + +The scale benchmark at the fetch queue's hard bound (`scale_benchmark_at_the_fetch_queue_bound`, +release build, 131,072 parked keys) stayed within the thresholds declared before it ran, apart from +dropping the queues (below): + +| sources per key | one park pass, over repeated runs | peak process RSS | +|---|---|---| +| 1 | 89 to 99 ms | 197 MiB | +| 8 | 207 to 216 ms | 342 MiB | +| 35 (the limit) | 754 to 960 ms | 1.13 GiB | + +- Waking and the stats read took under 1 ms; a departure took at most 107 ms. +- Dropping the queues at the limit took 274 to 315 ms over repeated runs. The scheduler before the + silent-source handling took 226 to 295 ms, and the same time as this code in alternating runs on + one machine. It was first held to 250 ms, which a loaded machine did not meet. It happens once, + when the engine shuts down, outside any lock, so it is now held to 1 s. +- The silent-source check makes the park pass about 5% slower than before. + +## Consequences + +### Positive + +- A requester on this release keeps its own in-flight fetches to one answering holder within that + holder's share. That removes the ordinary over-share bursts and their timeout and trust cost; it + does not cover stale work the responder still holds, described below. +- A fill or repair can finish sooner, because fetch slots no longer spend 30 s on over-share + requests that will not be answered. +- Responders should log fewer `per_peer_cap_full` drops as requesters upgrade. +- The dispatch summary shows the scheduler's queue state, lock times and silent sources in + production. + +### Negative / Trade-offs + +- It is not a guarantee of zero refusals. A request this node has already timed out on can still + hold its share at the responder until a worker sheds it, and a new request sent in that window + is refused. Closing that window needs option 2. During the 25-hour wave, responders shed 31,172 + admitted requests after waiting 30 s for a worker; the production data do not show how often + that window caused a later refusal. +- The requester counts a fetch as in flight for its whole attempt, local checks and the local write + included, which is longer than the responder holds its share. A holder can therefore sit below + its share while this node waits, and a fetch whose local work stalls holds its holder's share for + as long as it holds its fetch slot. +- A key whose answering holders are all busy waits in the queue, or goes to a lower-ranked holder + with room, instead of being sent to its first holder. +- Keys held only by a holder that just stopped answering take one fetch timeout longer to drain + than before the change, as in the simulation. A holder that is alive but so overloaded that it + times out is fetched from as before the change while its mark lasts. +- Recovering the slots dropped requests used to waste raises a filling node's instantaneous + bandwidth, disk and CPU use. Each responder still serves at most its share per requester with + three workers; the data a joining node needs to move is unchanged. +- At the source limit, the parked index can reach 4.6 million entries and about 1.1 GiB of process + memory; at 8 sources per key, about 340 MiB. +- Requesters on older releases still overshoot until they upgrade, so the network-wide drop rate + falls only as the release spreads, and the traffic pattern changes during that time. +- There is no runtime switch. Undoing the change means a release without it. + +### Neutral / Operational + +- The effect shows in the responders' cumulative counters (`replication responder admission + summary`): `fetch_admission_dropped_per_peer_share` relative to `fetch_request_rx_count` should + fall as the fleet and the community upgrade. +- Before release, a dev testnet run with a join wave of fresh nodes should compare drain time, + stored bytes, admission drops, trust events and host load against the previous release. + +## Validation + +**Unit tests on the queue:** + +- no answering source exceeds the share; +- preference order is kept; +- a startable key behind hundreds of blocked keys starts in the same pass; +- a freed share starts the nearest parked key, whatever its source; +- parked and retried keys keep their payload, tried sources and retry reservation; +- departures and merges keep the parked index exact; +- the source limit holds across park, retry and merge; +- a silent source is fetched from past the share, an answering source with room is preferred, and + parked keys start when a full source falls silent; +- whether a source is silent does not depend on the order its outcomes are applied in; marks are + set, cleared and expired as described, and the table bound keeps every guard and mark; +- an answer reported when it arrives keeps a source within its share when an earlier request to it + times out while the answered fetch is still storing its chunk; known only from the outcome, the + same answer comes too late and the source is sent every waiting key; +- re-advertisements keep a queued or in-flight key's retry entry within the hint bound, with its + capacity owner; +- sixteen keys held only by a silent holder, with eight slots, start in exactly three waves of 2, 8 + and 6; +- a randomised run of queue operations, re-advertisements and source contacts keeps every + invariant, the retry entries' bound included. + +**End-to-end tests:** + +- with eight fetch slots and twelve keys directed to one holder that holds what it admits, the + requester's own pipeline records at most two in-flight fetches to that holder; with the limit + removed it records eight and the test fails; +- with a 5 s fetch timeout and sixteen keys directed to a holder that answers nothing, the requester + sends past the share once the first requests time out and the keys leave its pipeline within four + timeouts (15.1 s measured); with the silent-source exception removed the test fails. + +**Revisit if:** + +- the responder's per-peer share changes; +- a busy reply is introduced; +- the fleet counters keep showing a high dropped share once most requesters run this release; +- the dispatch summary shows sustained table overflows or a large count of silent sources. + +## Notes for AI-assisted work + +AI tools may help draft this ADR, but **must not mark it Accepted without human review**. Accepted ADRs are immutable: create a new superseding ADR rather than editing an Accepted ADR. diff --git a/docs/adr/ADR-0025-fetch-busy-reply.md b/docs/adr/ADR-0025-fetch-busy-reply.md new file mode 100644 index 00000000..ff78d275 --- /dev/null +++ b/docs/adr/ADR-0025-fetch-busy-reply.md @@ -0,0 +1,276 @@ +# ADR-0025: Tell a fetch requester "busy" instead of refusing silently + +- **Status:** Proposed +- **Date:** 2026-10-15 +- **Decision owners:** +- **Reviewers:** +- **Supersedes:** none +- **Superseded by:** none +- **Related:** [ADR-0005](ADR-0005-replication-repair-hardening.md) (the repair fetch pipeline), + [ADR-0017](ADR-0017-bounded-fresh-offers-and-copy-free-sends.md) (responder admission bounds), + ant-node #252 / ADR-0024 (requesters stay within the responder's per-peer share) + +## Context + +A replication fetch is a request/response exchange. The responder admits at most two +outstanding fetches per requesting peer (`FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER`) and twelve +overall (`FETCH_RESPONDER_MAX_OUTSTANDING`). A request past either limit is refused at admission: +the responder logs `Fetch response dropped at admission` and sends nothing back. + +Silence is expensive for both sides: + +- the requester waits its whole fetch timeout (30 s) before moving on to the next holder; +- saorsa-core records a `ConnectionTimeout` trust event against the responder, which did nothing + wrong; +- the requester learns nothing about why, so it keeps choosing that holder for new keys. + +On the production fleet this was the common case, not an edge. As measured for #252 from the +responders' own cumulative admission counters, 36.6% of inbound fetch requests on 0.21.0 and 44.3% +on 0.22.0 were refused at admission. In the 25 hours around a wave of 177 newly installed nodes +filling their ranges (2026-10-11 20:00 to 2026-10-12 21:00 UTC), the WARN line counted 3.38M +refusals, 95.2% of them to the new nodes. + +ant-node #252 removes most of the per-peer over-share by keeping a requester's own fetches within +each answering holder's share. It cannot remove every refusal: a request the requester has +already timed out on still holds its share at the responder until the responder sheds it, the +twelve-slot pool can be full of other peers' requests, a holder #252 currently counts as silent +may be sent requests past its share until an answer clears the mark or the mark expires and the +holder is probed under the share again, and older requesters keep over-sharing until they +upgrade. #252's description leaves the first case to a busy reply, and its ADR defers that reply +as a wire change with its own rollout. + +## Decision Drivers + +- Most peers run older versions for weeks after any release. An old node must decode the same + messages it always did and never be sent a `Busy` it cannot decode. +- A delivered `Busy` within its sender's neutral allowance must not cost it trust; a refusal that + cannot be answered stays the old silence. +- A holder must not be able to keep the data, pass its storage audits and answer `Busy` to every + fetch for good at no cost: neutrality has to be bounded per holder, across keys. +- The added reply traffic must be bounded: no request storm from requesters retrying on every busy + answer, and no echo amplification at responders. +- One requester scheduler: #252's per-source scheduler reads which holders recently answered busy, + rather than a second scheduler choosing sources. + +## Considered Options + +1. **Keep silence; rely on #252 alone.** Leaves every remaining refusal at 30 s and a false + trust charge. +2. **Reply with the existing `FetchResponse::Error`.** Decodable by old requesters, but they + charge the responder an `ApplicationFailure` for it and fail over at once, so it trades a + timeout for an immediate hop storm. +3. **Add a field to `FetchRequest`.** Old responders would ignore a trailing field, but a current + responder would have to read old requests without it, which postcard reports as an error, not a + missing field. A hand-written tolerant deserializer can paper over that, but it cannot tell a + missing flag from a malformed one and only stays correct while `FetchRequest` is the last value + in every message that carries it. +4. **A capability trailer after the whole message, and an appended `FetchResponse::Busy` + variant (chosen).** +5. **Busy with requester-side retry rotation and budgets.** Reviewed and rejected: hopping on every + busy answer turns each refusal into an immediate new request, so a requester whose holders all + refuse asks once per round trip instead of once per fetch timeout; it competes with `tried` + state; and charging at budget expiry punishes honest overloaded holders once per key per cycle. +6. **Busy trust-neutral without a limit.** The first version of this change. Rejected in review: a + holder could answer `Busy` forever, putting off every fetch at no trust cost while it kept + answering storage audits. A per-key retry limit or a longer cooldown does not fix that, because + the same holder can be asked about any number of keys. + +## Decision + +We will let a requester say it understands a busy reply, and have a responder that refuses such a +request at admission say `Busy` at once, within bounds, instead of staying silent. + +**Wire.** `FetchRequest` keeps its serialized layout (`key`). A requester that understands busy +replies appends the 5-byte fetch capability trailer, `antf` and a flags byte with bit 0 set, +immediately after the postcard message. `ReplicationMessage::decode` uses `take_from_bytes` and +sets `FetchRequest::busy_reply` (not serialized) only when that trailer follows a fetch request; +bytes after the flags byte are reserved and ignored. Older nodes decode with `from_bytes`, which +ignores everything after the message, so they read the key and refuse silently as before. +`FetchResponse::Busy { key }` is appended after `Success`, `NotFound` and `Error`, whose +discriminants are unchanged, and is only ever sent in answer to a request carrying the trailer. + +**Responder.** Every saorsa-core message is signed with ML-DSA-65, so a busy reply costs a +signature and about 5 KiB on the wire, not a few bytes. On an admission refusal of a request that +asked, the responder takes a busy-reply send permit (48 at once), then a grant from +`FetchBusyReplyLimiter`: per peer, two back to back and then one a second, with at most two in +flight; overall, 32 back to back and then 16 a second (at most 48 replies, about 250 KiB, in any +one second; about 85 KiB/s sustained). Both rate allowances are tested and spent together under +one lock, a denial spends neither, and the per-peer table is capped at 4096 entries, forgetting +only peers whose allowance has refilled and that have nothing in flight. Within all bounds it +sends `Busy` on the detached task tracker, abandons the send after 2 s and races shutdown; +otherwise it refuses silently, exactly as before. The `Fetch response dropped at admission` +WARN line and its counters are unchanged, with `capability=asked|not_asked` and +`reply=scheduled|none` fields added. + +**Requester.** Every fetch request this node's fetch worker sends asks for busy replies. A `Busy` +for the requested key carries no trust event while its holder is within its neutral allowance +(`FETCH_BUSY_NEUTRAL_ALLOWANCE`): eight busy answers in a row to this requester, counted across all +keys, since the holder last served it. The attempt then waits 5-10 s (uniform jitter), never past +its own fetch timeout, still holding its fetch slot, and ends as `SourceFailed`, so the queue moves +on to the next untried holder exactly as after a timeout. Past the allowance, each further busy +answer is handled as the silent refusal it replaces: the attempt holds its slot until its fetch +timeout, counted from when it began, runs out, never longer however late the answer came, and +then the requester records one `ApplicationFailure` of weight 1.0, which saorsa-core scores +exactly as the `ConnectionTimeout` a silent refusal earned (one failed response of unit weight). +An attempt cancelled by shutdown before then is not charged. Silence's timeout starts once any +dial is done, so after a cold dial the charge can come up to that dial's time sooner than +silence's would have. Only serving this requester once, with a chunk that passes its integrity +checks, restores the allowance; time does not, so a holder asked only rarely still uses it up. +The allowance is per requester: each node judges only its own fetches. + +`SourceBusyTracker` remembers the busy holder, and #252's per-source scheduler reads it: a new key +starts, and a failed key retries, on the first free holder in preference order (below this +node's per-source share and not silent) that is not cooling down. A cooling holder comes after +every other free holder, and still before a silent one, which costs a whole fetch timeout. It +reorders and never withholds, so it lowers how often a busy holder is asked but is not a hard +per-holder bound; the hard bound is the busy wait per fetch slot. The cooldown is 1, 2, 4, then 8 times the full jittered wait drawn for that busy answer, +the multiplier rising with each further busy answer before the holder next serves this +requester, capped at 60 s, and a later busy answer never shortens a running cooldown; once it +runs out the holder is no longer put last. Serving once clears the holder's record, so its next +busy answer starts the multiplier again; after 10 minutes without a busy answer, the next one +starts the multiplier again too, but leaves what the holder owes against its allowance as it was. +Records go only when their holder serves: not for age, and not to make room. When the +4096-entry table is full, a holder it does not already track is charged for each busy answer, +as for silence, and is not put last. A `Busy` for any other key is treated as malformed, like +every other key mismatch. + +**Order.** This change is built on #252 and merges after it; until #252 merges, its branch +carries #252's commit. #252's per-source share is what removes most per-peer over-share and +bounds how many fetches one requester has open at one answering holder; without it, a requester +with many fetch slots still over-shares, and most of those refusals exceed the per-peer +busy-reply allowance and stay silent. In the combined build: +- #252's source choice puts a cooling holder after every other free holder; +- a busy answer counts as an answer for #252's silent-source tracking, as every answer does; +- a busy wait keeps its attempt in flight, counted against the holder's share. + +## Consequences + +### Positive + +- When a refused requester receives `Busy` within the responder's neutral allowance, it moves on + after 5-10 s, or sooner if its own fetch timeout runs out first, instead of 30 s, without + charging the responder. +- A holder that just refused is not chosen for a key while another free holder for that key is + not cooling. +- Old nodes decode exactly the messages they always did: a request carries 5 bytes after the + message that they never read, and they are never sent `Busy`. Rollback is a release without the + change. +- With #252, each busy wait keeps one attempt counted against that holder's two-request share for + its duration, reducing new requests and stopping them when both share slots are occupied; the + busy answer also counts as an answer for #252's silent-source logic. + +### Negative / Trade-offs + +- New outbound traffic at responders: at most 16 signed busy replies a second sustained, about + 85 KiB/s, and 48 (about 250 KiB) in any one second, plus that many signatures. Today a refusal + sends nothing. Each refused request already cost its sender the same signed message. +- A requester asks a refusing holder up to once per busy wait per fetch slot instead of once per + fetch timeout `T`: at most `C / min(5 s, T)` small requests with `C` hardware-thread fetch + slots, against `C / T` today, in the case where every holder refuses everything; with the + default 30 s timeout, `C / 5 s` against `C / 30 s`. Capping `C` is a separate decision. +- Refusals beyond the per-peer busy-reply allowance stay silent and are still charged by saorsa-core. + Without #252, a requester that over-shares by more than two requests at once still meets some + silence. +- Busy is trust-neutral only within the allowance. A holder that never serves gets eight free busy + answers in a row per requester, then each further refusal costs it what silence cost, so past + the allowance it does no better than before this change. An honest holder that stays overloaded + past the allowance is likewise charged per refusal, as it was before this change. A holder that + serves one fetch in nine is never charged: bounding selective service more tightly would need a + service audit with reserved capacity, which is not part of this decision. +- The count does not decay with time. An honest holder that, over any length of time, answers one + requester busy nine times without serving it once in between is charged for the ninth, as + silence would have charged every one. +- The count lives in memory, so a requester restart starts every holder afresh: at most eight free + busy answers per holder per run of the requester. Keeping it across restarts would need a new + on-disk format. A holder does not control when a requester restarts. +- A full table keeps every count, so it charges holders it does not track yet from their first + busy answer, as silence did. That takes 4096 holders each answering this requester busy + without serving it since; a node fetches from its neighbourhood, far fewer peers, and the table + empties on restart. As with the global busy-reply allowance, what identities filling it on + purpose win is the old silence for everyone else, not a new failure. +- Sixteen identities each spending their per-peer second can use up the global allowance and + deny busy replies to everyone else, and stalled peers can hold the 48 send permits for up to + 2 s each. What they win is the old silence for those peers, not a new failure. The global + allowance is kept small on purpose, because it is what bounds the signed bytes a full node + spends on refusals. + +### Neutral / Operational + +- New fields on the `replication responder admission summary (cumulative)` line: the cumulative + counters + `fetch_busy_reply_{not_asked,send_pool_full,rate_limited,peer_in_flight,sent,send_failed,timed_out,cancelled}`, + `fetch_busy_received`, `fetch_busy_charged` (busy answers charged past the neutral allowance) and + `fetch_busy_cooling_skips`, and the gauge `fetch_busy_sources_tracked`, + the current size of the requester's busy-holder table. + `not_asked` against the rest is the share of refused requests that did not ask for a busy reply: + a proxy, and an upper bound, for how much refused traffic still comes from older versions (a + request from a current node can also not ask, through the API or a malformed trailer). + +## Validation + +- Unit tests: + - mirrors of the previous types decode current requests, and the unchanged answers 0-2; + - a current node reads an older request as not asking; + - trailer parsing: exact, wrong magic, truncated, unknown bits, reserved bytes, padding; + - an older requester cannot decode `Busy`; + - limiter: burst, refill, global cap, no spend on a denial, in-flight limit, bounded table; + - the busy wait's range and its cap at the fetch timeout, and that a busy answer, however late + and whether charged or not, never holds the slot past the attempt's fetch timeout; + - tracker: cooldown, escalation, cap, not shortened by a later busy answer, reset, expiry, + bounded table; + - neutral allowance: a sole holder that answers busy every 5 s for over an hour without serving + is charged for every answer after the eighth; two answers at once + spend two; a holder that serves once between spells of eight is never charged; serving + restores the whole allowance and another holder serving does not; answers an hour apart + still use it up; a full table forgets no count while one more holder than it holds takes + turns with the rest, charges that newcomer instead, and makes room once a holder serves; + - #252's dispatch and retry pass over a cooling holder for another free one, still take it + before a silent one, and never withhold it; + - a busy reply is not counted as served bytes. +- End to end on a local network, made deterministic with a test-only fetch serve gate (the same + one #252 adds): + - a refused request that asks is told `Busy` at once; + - one that does not ask is refused silently; + - the held requests are served once the gate opens; + - the requester's own fetch worker, refused because its two-request share at the holder is + full, records the holder as cooling, keeps the attempt for at least the minimum busy wait, + then gives the key up, and the holder's trust at the requester never drops; + - a holder that keeps refusing the requester's fetch worker, one new key after another, costs + nothing at the requester for eight busy answers, and is charged for the ninth only as that + attempt's fetch timeout runs out; once it serves the worker a chunk, what it owes is + cleared; + - #252's two per-source share tests pass on the combined build. Its silent-holder test turns + off that holder's busy replies with a test-only switch, so it still models a holder that + answers nothing: one that answers busy is answering, and stays under #252's cap. +- Measured on real nodes with the test-only gate, in `tests/e2e/fetch_busy_ab.rs` (opt-in): 20 + requesters each within their two-request share, and the first of five holders with its 12-slot + pool full for 2 s, the refusal case left after #252, with 28 refusals in every run below: + - the combined #252 + #253 build, on main b0e29c32: all 40 chunks in 10.2 s, 10.0 s, 10.0 s + and 10.0 s, no holder charged; + - #252 alone (447c000d, with the file's two test-only counters added): 60.4 s, 60.5 s and + 60.8 s, with 14 requesters charging the first holder and 8 the second each time, and 0 to 7 + the others. + + One more run of #252 alone stopped before measuring, at the test's check that every key was + newly queued (38 of 40). +- Earlier, without #252, on main a6e3388e: the same pool case took 90.3 s in one run on that + main (2026-10-16), with 14, 9, 9, 4 and 3 requesters charging the five holders, and 9.4 s to + 10.5 s in four runs, no holder charged, with this change alone (2026-10-16 and 2026-10-20). A + single requester over-asking three holders, the case #252 removes, took 90.2 s, 90.2 s and + 90.3 s on that main, and 35.4 s to 39.2 s in six runs with this change alone; that measurement + left the file once #252 made the case impossible. +- A local, unpublished discrete-event model of the 177-node join wave and 24 more scenarios + informed the design, including the 5-10 s wait. Its source and raw output are not part of this + change, so its results are not offered as validation. +- In production: + - `fetch_busy_reply_sent` against `fetch_admission_dropped_*`; + - the fall in `connection_timeout` trust events between fleet nodes; + - `fetch_busy_reply_rate_limited` and `fetch_busy_reply_send_pool_full` staying small; + - `fetch_busy_charged` against `fetch_busy_received`: how often holders stay busy past the + neutral allowance. + + Revisit the 5-10 s wait and the allowances against measured admission hold times. + +## Notes for AI-assisted work + +AI tools may help draft this ADR, but **must not mark it Accepted without human review**. Accepted ADRs are immutable: create a new superseding ADR rather than editing an Accepted ADR. diff --git a/docs/adr/README.md b/docs/adr/README.md index b58e9a32..90646772 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -40,3 +40,5 @@ See [`TOOLING.md`](./TOOLING.md) for `adrs`, `adr-kit`, and AI harness setup. - [ADR-0018: Pointer ownership transfer by final redirection](./ADR-0018-pointer-transfer-by-final-redirection.md) - [ADR-0019: Pointer audits serve the record round 1 bound](./ADR-0019-pointer-audits-serve-the-record-round-one-bound.md) - [ADR-0022: Remove the LMDB Chunk Store](./ADR-0022-remove-the-lmdb-chunk-store.md) +- [ADR-0024: Fetch requesters stay within the responder's per-peer share](./ADR-0024-fetch-requesters-stay-within-the-responder-share.md) +- [ADR-0025: Tell a fetch requester "busy" instead of refusing silently](./ADR-0025-fetch-busy-reply.md) diff --git a/src/replication/audit_metrics.rs b/src/replication/audit_metrics.rs index de8f77a1..281476b0 100644 --- a/src/replication/audit_metrics.rs +++ b/src/replication/audit_metrics.rs @@ -400,6 +400,25 @@ static RESPONDER_ADMISSION_DROPS: [AtomicU64; N_RESPONDER_CLASSES * N_ADMISSION_ static RESPONDER_STALENESS_SHEDS: [AtomicU64; N_RESPONDER_CLASSES] = [const { AtomicU64::new(0) }; N_RESPONDER_CLASSES]; +/// Fetch admission refusals by what became of the busy reply, indexed by +/// [`FetchBusyReplyOutcome::index`]. +static FETCH_BUSY_REPLIES: [AtomicU64; FetchBusyReplyOutcome::COUNT] = + [const { AtomicU64::new(0) }; FetchBusyReplyOutcome::COUNT]; + +/// Busy answers this node received to its own fetches. +static FETCH_BUSY_RECEIVED: AtomicU64 = AtomicU64::new(0); + +/// Busy answers this node charged, because their source had used up its +/// neutral allowance without serving it. +static FETCH_BUSY_CHARGED: AtomicU64 = AtomicU64::new(0); + +/// Times this node, choosing where to start or retry a fetch, passed over a +/// holder that had recently answered it busy. +static FETCH_BUSY_COOLING_SKIPS: AtomicU64 = AtomicU64::new(0); + +/// Holders this node currently remembers as having answered it busy. +static FETCH_BUSY_SOURCES_TRACKED: AtomicU64 = AtomicU64::new(0); + static SERIAL_QUEUE_OVERFLOW_DROPS: [AtomicU64; protocol::N_REPLICATION_VARIANTS] = [const { AtomicU64::new(0) }; protocol::N_REPLICATION_VARIANTS]; @@ -532,6 +551,111 @@ pub fn record_responder_staleness_shed(class: ReplicationResponderClass) { } } +/// What became of a fetch refused at admission, as far as a busy reply goes. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum FetchBusyReplyOutcome { + /// The requester did not ask for busy replies (an older node): silence, + /// as before. + NotAsked, + /// Every busy-reply send permit was in use: silence. + SendPoolFull, + /// Over the per-peer or global busy-reply rate: silence. + RateLimited, + /// A busy reply to the same peer was still being sent: silence. + PeerInFlight, + /// The busy reply was handed to the transport. + Sent, + /// The transport refused the busy reply. + SendFailed, + /// The transport did not take the busy reply in time; abandoned. + TimedOut, + /// Shutdown began before the busy reply was sent. + Cancelled, +} + +impl FetchBusyReplyOutcome { + const COUNT: usize = 8; + + const fn index(self) -> usize { + match self { + Self::NotAsked => 0, + Self::SendPoolFull => 1, + Self::RateLimited => 2, + Self::PeerInFlight => 3, + Self::Sent => 4, + Self::SendFailed => 5, + Self::TimedOut => 6, + Self::Cancelled => 7, + } + } +} + +/// Count what became of one fetch refusal's busy reply. +pub fn record_fetch_busy_reply(outcome: FetchBusyReplyOutcome) { + if let Some(counter) = FETCH_BUSY_REPLIES.get(outcome.index()) { + counter.fetch_add(1, Ordering::Relaxed); + } +} + +/// Fetch refusals with this busy-reply outcome since process start. +#[must_use] +#[cfg_attr(not(feature = "logging"), allow(dead_code))] +pub fn fetch_busy_replies_total(outcome: FetchBusyReplyOutcome) -> u64 { + FETCH_BUSY_REPLIES + .get(outcome.index()) + .map_or(0, |counter| counter.load(Ordering::Relaxed)) +} + +/// Count a busy answer to one of this node's own fetches. +pub fn record_fetch_busy_received() { + FETCH_BUSY_RECEIVED.fetch_add(1, Ordering::Relaxed); +} + +/// Busy answers to this node's own fetches since process start. +#[must_use] +#[cfg_attr(not(feature = "logging"), allow(dead_code))] +pub fn fetch_busy_received_total() -> u64 { + FETCH_BUSY_RECEIVED.load(Ordering::Relaxed) +} + +/// Count a busy answer charged because its source was past its neutral +/// allowance. +pub fn record_fetch_busy_charged() { + FETCH_BUSY_CHARGED.fetch_add(1, Ordering::Relaxed); +} + +/// Busy answers this node charged since process start. +#[must_use] +#[cfg_attr(not(feature = "logging"), allow(dead_code))] +pub fn fetch_busy_charged_total() -> u64 { + FETCH_BUSY_CHARGED.load(Ordering::Relaxed) +} + +/// Count a fetch started or retried on a later holder because an earlier one +/// had recently answered busy. +pub fn record_fetch_busy_cooling_skip() { + FETCH_BUSY_COOLING_SKIPS.fetch_add(1, Ordering::Relaxed); +} + +/// Fetches started or retried past a cooling holder since process start. +#[must_use] +#[cfg_attr(not(feature = "logging"), allow(dead_code))] +pub fn fetch_busy_cooling_skips_total() -> u64 { + FETCH_BUSY_COOLING_SKIPS.load(Ordering::Relaxed) +} + +/// Set how many holders this node remembers as having answered it busy. +pub fn set_fetch_busy_sources_tracked(count: usize) { + FETCH_BUSY_SOURCES_TRACKED.store(u64::try_from(count).unwrap_or(u64::MAX), Ordering::Relaxed); +} + +/// Holders this node remembers as having answered it busy. +#[must_use] +#[cfg_attr(not(feature = "logging"), allow(dead_code))] +pub fn fetch_busy_sources_tracked() -> u64 { + FETCH_BUSY_SOURCES_TRACKED.load(Ordering::Relaxed) +} + /// One class's admission pressure, read together for a summary line. #[cfg_attr(not(feature = "logging"), allow(dead_code))] struct ResponderAdmissionSnapshot { @@ -640,6 +764,21 @@ pub fn log_responder_admission_summary() { fetch_admission_dropped_global_pool = fetch.global_pool, fetch_admission_dropped_per_peer_share = fetch.per_peer_share, fetch_staleness_shed = fetch.staleness_shed, + fetch_busy_reply_not_asked = fetch_busy_replies_total(FetchBusyReplyOutcome::NotAsked), + fetch_busy_reply_send_pool_full = + fetch_busy_replies_total(FetchBusyReplyOutcome::SendPoolFull), + fetch_busy_reply_rate_limited = + fetch_busy_replies_total(FetchBusyReplyOutcome::RateLimited), + fetch_busy_reply_peer_in_flight = + fetch_busy_replies_total(FetchBusyReplyOutcome::PeerInFlight), + fetch_busy_reply_sent = fetch_busy_replies_total(FetchBusyReplyOutcome::Sent), + fetch_busy_reply_send_failed = fetch_busy_replies_total(FetchBusyReplyOutcome::SendFailed), + fetch_busy_reply_timed_out = fetch_busy_replies_total(FetchBusyReplyOutcome::TimedOut), + fetch_busy_reply_cancelled = fetch_busy_replies_total(FetchBusyReplyOutcome::Cancelled), + fetch_busy_received = fetch_busy_received_total(), + fetch_busy_charged = fetch_busy_charged_total(), + fetch_busy_cooling_skips = fetch_busy_cooling_skips_total(), + fetch_busy_sources_tracked = fetch_busy_sources_tracked(), verification_admission_dropped_global_pool = verification.global_pool, verification_admission_dropped_per_peer_share = verification.per_peer_share, verification_staleness_shed = verification.staleness_shed, diff --git a/src/replication/fetch_busy.rs b/src/replication/fetch_busy.rs new file mode 100644 index 00000000..536bd891 --- /dev/null +++ b/src/replication/fetch_busy.rs @@ -0,0 +1,953 @@ +//! The fetch busy reply: what each side does when a responder cannot admit a +//! fetch. +//! +//! A responder that refuses a fetch at admission used to stay silent. The +//! requester then waited its whole fetch timeout, and saorsa-core charged the +//! responder a connection timeout it had not earned. A requester that says it +//! understands [`FetchResponse::Busy`] is now told at once instead. +//! +//! Two bounds keep the new reply from costing more than the silence it +//! replaces: +//! +//! - **Responder:** a busy reply is a send the node never made before, caused +//! by a request it is refusing, and like every message it is signed, so it +//! costs a signature and about 5 KiB on the wire. [`FetchBusyReplyLimiter`] +//! caps those sends per peer and overall, and how many to one peer may be in +//! flight at once. +//! Past any cap a refusal is silent again, which is exactly what every +//! refusal was before. +//! - **Requester:** a busy answer is not a reason to ask again at once. +//! [`busy_wait`] keeps the attempt, and so its fetch slot, for a few seconds +//! before the key moves on, so a requester asks at most once per wait per +//! slot instead of as fast as answers come back. The attempt then ends as a +//! source failure, exactly as the silent timeout did, minus the trust charge +//! while the source is within its neutral allowance (see below). +//! [`SourceBusyTracker`] also puts a busy source last when choosing where to +//! start or retry a key, for longer each time it stays busy, so a requester +//! does not keep picking a holder that just said it cannot take more. +//! +//! A busy answer costs no trust only while it can still be passing overload. +//! A source gets [`FETCH_BUSY_NEUTRAL_ALLOWANCE`] busy answers in a row, across +//! all keys, before it next serves this requester. Past that, until it serves +//! once, each busy answer is handled as the silent refusal it replaces: the +//! attempt holds its slot until its fetch timeout runs out, and the source is +//! then charged the same trust. +//! Without that bound a holder could keep the data, answer its storage audits +//! and say busy to every fetch for good, at no cost. +//! +//! [`FetchResponse::Busy`]: crate::replication::protocol::FetchResponse::Busy + +use std::collections::HashMap; +use std::sync::Arc; +use std::time::{Duration, Instant}; + +use parking_lot::Mutex; +use saorsa_core::identity::PeerId; + +use super::audit_metrics; + +/// Shortest time a requester waits after a busy answer before moving on. +/// +/// A responder refuses because its admitted fetches, each up to a 4 MiB chunk +/// sent over a home upload link, are still being served. A share frees in +/// seconds, not milliseconds, so asking again sooner would mostly be refused +/// again. +pub const FETCH_BUSY_WAIT_MIN: Duration = Duration::from_secs(5); + +/// Longest time a requester waits after a busy answer before moving on. +/// +/// Waits are spread uniformly between the two bounds, so requesters refused +/// together do not all come back together. +pub const FETCH_BUSY_WAIT_MAX: Duration = Duration::from_secs(10); + +/// The jitter a requester adds to [`FETCH_BUSY_WAIT_MIN`]: drawn uniformly from +/// zero to this, inclusive. +pub const FETCH_BUSY_WAIT_SPREAD: Duration = + FETCH_BUSY_WAIT_MAX.saturating_sub(FETCH_BUSY_WAIT_MIN); + +/// Busy answers in a row that a source may give this requester, before it +/// next serves it, at no trust cost. +/// +/// Honest overload passes: a share frees in seconds. A source that keeps +/// answering busy cannot be told apart from one that keeps the data and will +/// not serve it. So past this many, until the source serves this requester +/// once, each busy answer is handled as the silent refusal it replaces: the +/// attempt holds its slot until its fetch timeout, counted from when it +/// began, runs out, and the source is then charged the same trust. Silence's +/// timeout starts once any dial is done, so after a cold dial the charge can +/// come up to that dial's time sooner. Only serving restores the allowance; +/// time does not. +/// +/// The count lives in memory, so a restart of the requester starts every +/// source afresh: at most this many free busy answers per source per run. +/// +/// With two answers a round, one for each request in a peer's reply burst, +/// eight is four rounds of being put last: minutes, when the requester has +/// other holders to ask, and well under a minute when it has none. +pub const FETCH_BUSY_NEUTRAL_ALLOWANCE: u32 = 8; + +/// Longest a responder spends handing one busy reply to the transport. +/// +/// A reply that cannot be handed over in this long is abandoned: the requester +/// then gets the silence it would have had anyway. Without it, a peer that is +/// slow to accept data could hold a send permit indefinitely. +pub const FETCH_BUSY_REPLY_SEND_TIMEOUT: Duration = Duration::from_secs(2); + +/// Busy replies one peer may receive back to back, and the most to it that +/// may be in flight at once. +/// +/// The same as the per-peer fetch admission share, so a requester that sent +/// one share too many at once hears about each refusal. +const PEER_BURST: u32 = 2; + +/// Sustained rate of busy replies to one peer: one per this interval. +/// +/// A requester that waits [`FETCH_BUSY_WAIT_MIN`] after each busy answer never +/// comes near it. A peer that ignores the answers and keeps sending gets one +/// reply a second, and silence for the rest. +const PEER_INTERVAL: Duration = Duration::from_secs(1); + +/// Busy replies this node may send back to back across all peers. +/// +/// Every message is signed, which adds about 5 KiB to the wire and a +/// signature's worth of CPU, so the global allowance is what bounds the cost. +/// Burst and rate together allow at most 48 replies, about 250 KiB, in any one +/// second, and 16 a second, about 85 KiB/s, sustained. The burst lets a node +/// that becomes full answer a whole pool's worth of peers at once. +const GLOBAL_BURST: u32 = 32; + +/// Sustained rate of busy replies across all peers: one per this interval, +/// 16 a second. +const GLOBAL_INTERVAL: Duration = Duration::from_micros(62_500); + +/// Most peers whose busy-reply allowance is tracked at once. +/// +/// A peer whose allowance has fully refilled and has no reply in flight +/// carries no information and is dropped first. If every tracked peer is still +/// being limited, a new peer gets silence rather than an entry, so the table +/// cannot grow without bound. +const MAX_TRACKED_PEERS: usize = 4096; + +/// Longest a source that keeps answering busy is put last. +const MAX_SOURCE_COOLDOWN: Duration = Duration::from_mins(1); + +/// Busy answers after which a source's cooldown stops doubling: the wait, +/// then two, four and eight times it, capped at [`MAX_SOURCE_COOLDOWN`]. +const MAX_COOLDOWN_DOUBLINGS: u32 = 3; + +/// A source that has not answered busy for this long starts a new busy spell +/// with its next busy answer: its cooldown starts again from the wait, so one +/// busy spell is not held against its scheduling for good. What it owes +/// against its neutral allowance does not start again. +const BUSY_SPELL_GAP: Duration = Duration::from_mins(10); + +/// Most sources whose busy record is kept at once. +/// +/// Records go only when their source serves, so a full table forgets nothing +/// to make room. A source it does not already hold is then charged for every +/// busy answer, as for silence, and is not put last. A node fetches from its +/// neighbourhood, far fewer peers than this, and the table empties on restart. +const MAX_TRACKED_SOURCES: usize = 4096; + +/// How long to wait after a busy answer, given `remaining`, the time left +/// before the attempt's fetch timeout would have expired. +/// +/// The wait is [`FETCH_BUSY_WAIT_MIN`] plus `jitter`, which the caller draws +/// from `0..=FETCH_BUSY_WAIT_SPREAD` and which is clamped to that range. It +/// never outlasts `remaining`, so a busy answer never leaves a key waiting +/// longer than the silent refusal it replaces. (A fetch timeout shorter than +/// the minimum wait, which only tests configure, shortens the wait with it.) +#[must_use] +pub fn busy_wait(remaining: Duration, jitter: Duration) -> Duration { + FETCH_BUSY_WAIT_MIN + .saturating_add(jitter.min(FETCH_BUSY_WAIT_SPREAD)) + .min(remaining) +} + +/// When an attempt answered busy at `now` lets go of its fetch slot, given +/// `wait`, the busy wait drawn for it, and `deadline`, when the attempt's fetch +/// timeout runs out. +/// +/// Within the neutral allowance it is once the busy wait is over; past it, at +/// the deadline, as silence would have held it. Either way it is never after +/// `deadline`, so however late a busy answer comes, the slot is held no longer +/// than silence would have held it. A fixed time rather than a length, so +/// nothing between the answer and the sleep can stretch it. +#[must_use] +pub fn busy_wake(answer: BusyAnswer, wait: Duration, now: Instant, deadline: Instant) -> Instant { + if answer.charged { + deadline + } else { + now.checked_add(wait) + .map_or(deadline, |done| done.min(deadline)) + } +} + +/// A generic cell rate limiter: `burst` events back to back, then one per +/// `interval`. +/// +/// It keeps only the theoretical arrival time of the next event, so it needs no +/// floating point and no refill timer. +#[derive(Debug, Default, Clone, Copy)] +struct Gcra { + /// When the allowance is next fully refilled; `None` until first used. + tat: Option, +} + +impl Gcra { + /// The arrival time a further event would move this limiter to, if it is + /// allowed now. + fn admit(self, now: Instant, interval: Duration, burst: u32) -> Option { + let tat = self.tat.map_or(now, |tat| tat.max(now)); + let next = tat.checked_add(interval)?; + (next.saturating_duration_since(now) <= interval.saturating_mul(burst)).then_some(next) + } + + /// Whether the allowance has fully refilled. + fn is_idle(self, now: Instant) -> bool { + self.tat.is_none_or(|tat| tat <= now) + } +} + +/// One peer's busy-reply allowance. +#[derive(Debug, Default, Clone, Copy)] +struct PeerAllowance { + rate: Gcra, + /// Busy replies to this peer being sent now. + in_flight: u32, +} + +impl PeerAllowance { + /// Forgetting it loses nothing: its allowance is full and nothing is in + /// flight. + fn is_idle(self, now: Instant) -> bool { + self.in_flight == 0 && self.rate.is_idle(now) + } +} + +#[derive(Debug, Default)] +struct LimiterState { + global: Gcra, + peers: HashMap, +} + +/// Why a busy reply was not allowed. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum BusyReplyDenied { + /// Over the per-peer or global rate, or the table of peers is full. + RateLimited, + /// The most busy replies to this peer that may be in flight are still + /// being sent. + PeerInFlight, +} + +/// Bounds how many busy replies a responder sends: per peer and overall, by +/// rate, and how many to one peer may be in flight at once. +/// +/// Synchronous and short, so it can be asked on the serial message loop. +#[derive(Debug, Default)] +pub struct FetchBusyReplyLimiter { + state: Mutex, +} + +impl FetchBusyReplyLimiter { + /// A limiter with every allowance full. + #[must_use] + pub fn new() -> Self { + Self::default() + } + + /// Allow one busy reply to `peer` at `now`, if its allowance, the global + /// one and its in-flight limit all permit. Both rate allowances are spent + /// together on a grant and neither on a denial. The returned grant holds + /// one of the peer's in-flight places until it is dropped, which the caller + /// does when the send ends. + /// + /// # Errors + /// + /// Returns why the reply is not allowed. + pub fn try_acquire( + self: &Arc, + peer: &PeerId, + now: Instant, + ) -> Result { + let mut state = self.state.lock(); + if !state.peers.contains_key(peer) && state.peers.len() >= MAX_TRACKED_PEERS { + state.peers.retain(|_, allowance| !allowance.is_idle(now)); + if state.peers.len() >= MAX_TRACKED_PEERS { + return Err(BusyReplyDenied::RateLimited); + } + } + let allowance = state.peers.get(peer).copied().unwrap_or_default(); + if allowance.in_flight >= PEER_BURST { + return Err(BusyReplyDenied::PeerInFlight); + } + let peer_next = allowance + .rate + .admit(now, PEER_INTERVAL, PEER_BURST) + .ok_or(BusyReplyDenied::RateLimited)?; + let global_next = state + .global + .admit(now, GLOBAL_INTERVAL, GLOBAL_BURST) + .ok_or(BusyReplyDenied::RateLimited)?; + state.global.tat = Some(global_next); + state.peers.insert( + *peer, + PeerAllowance { + rate: Gcra { + tat: Some(peer_next), + }, + in_flight: allowance.in_flight.saturating_add(1), + }, + ); + Ok(BusyReplyGrant { + limiter: Arc::clone(self), + peer: *peer, + }) + } + + fn release(&self, peer: &PeerId) { + if let Some(allowance) = self.state.lock().peers.get_mut(peer) { + allowance.in_flight = allowance.in_flight.saturating_sub(1); + } + } + + /// How many peers are tracked now. + #[cfg(test)] + fn tracked_peers(&self) -> usize { + self.state.lock().peers.len() + } +} + +/// A granted busy reply. Holds one of the peer's in-flight places until +/// dropped. +#[derive(Debug)] +pub struct BusyReplyGrant { + limiter: Arc, + peer: PeerId, +} + +impl Drop for BusyReplyGrant { + fn drop(&mut self) { + self.limiter.release(&self.peer); + } +} + +/// What one busy answer means for its source at this requester. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct BusyAnswer { + /// How long the source is now put last. + pub cooling_for: Duration, + /// Whether the answer is past the source's neutral allowance, and so is + /// handled as the silent refusal it replaces. + pub charged: bool, +} + +/// One source's busy answers to this requester since it last served it. +#[derive(Debug, Clone, Copy)] +struct SourceBusy { + /// Busy answers in its current busy spell, which set its cooldown. + strikes: u32, + /// Busy answers since it last served this requester, however long ago: + /// what spends its neutral allowance. + unserved: u32, + /// Until when it is put last. + cooling_until: Instant, + /// When it last answered busy. + last_busy: Instant, +} + +impl SourceBusy { + fn is_cooling(self, now: Instant) -> bool { + now < self.cooling_until + } + + fn spell_over(self, now: Instant) -> bool { + !self.is_cooling(now) && now.saturating_duration_since(self.last_busy) >= BUSY_SPELL_GAP + } +} + +/// Which sources recently answered this requester busy, until when they +/// should be asked last, and which have used up their neutral allowance. +/// +/// Mostly local scheduling: a holder that just refused is the worst place to +/// send the next fetch, and one that keeps refusing is put last for longer +/// each time. The fetch scheduler reads it to start or retry a key on another +/// free holder first, and never to withhold one: a key whose every free holder +/// is cooling still starts on the first of them, and then waits out that +/// holder's busy answer like any other. So per requester it lowers how often +/// a busy holder is asked; the hard bound is the busy wait per fetch slot. +/// +/// The same record counts a source's busy answers in a row, across all keys, +/// against [`FETCH_BUSY_NEUTRAL_ALLOWANCE`]. It does not touch trust itself; +/// it tells the caller which busy answer is past the allowance. Only serving +/// clears the record, and so restores the allowance: neither time nor a full +/// table does. The table is not saved, so a restart clears it. +/// Synchronous and short, so it is safe to consult while holding the fetch +/// queues lock. +#[derive(Debug, Default)] +pub struct SourceBusyTracker { + sources: Mutex>, +} + +impl SourceBusyTracker { + /// A tracker that knows no busy source. + #[must_use] + pub fn new() -> Self { + Self::default() + } + + /// Record a busy answer from `source` at `now`. `wait` is the busy wait + /// before any cap from the attempt's own timeout. + /// + /// Returns how long the source is now put last: `wait` the first time, + /// doubling with each further busy answer in the same busy spell, up to a + /// minute. A cooldown already running is never shortened, so when it + /// outlasts the new one, what is left of it is returned. Also returns + /// whether this answer is past the source's + /// [`FETCH_BUSY_NEUTRAL_ALLOWANCE`]. When the table is full and holds no + /// record of `source`, the answer is charged and nothing is recorded. + pub fn record_busy(&self, source: &PeerId, now: Instant, wait: Duration) -> BusyAnswer { + let mut sources = self.sources.lock(); + if !sources.contains_key(source) && sources.len() >= MAX_TRACKED_SOURCES { + // Forgetting any record would restore its source's allowance, so + // with no room this answer gets none. + return BusyAnswer { + cooling_for: Duration::ZERO, + charged: true, + }; + } + let record = sources.entry(*source).or_insert(SourceBusy { + strikes: 0, + unserved: 0, + cooling_until: now, + last_busy: now, + }); + if record.spell_over(now) { + record.strikes = 0; + } + record.strikes = record.strikes.saturating_add(1); + record.unserved = record.unserved.saturating_add(1); + let doublings = record.strikes.saturating_sub(1).min(MAX_COOLDOWN_DOUBLINGS); + let cooldown = wait + .saturating_mul(1u32 << doublings) + .min(MAX_SOURCE_COOLDOWN); + let until = now.checked_add(cooldown).unwrap_or(now); + record.cooling_until = record.cooling_until.max(until); + record.last_busy = now; + let answer = BusyAnswer { + cooling_for: record.cooling_until.saturating_duration_since(now), + charged: record.unserved > FETCH_BUSY_NEUTRAL_ALLOWANCE, + }; + audit_metrics::set_fetch_busy_sources_tracked(sources.len()); + answer + } + + /// `source` served this requester: forget its busy record, which restores + /// its neutral allowance. + pub fn record_served(&self, source: &PeerId) { + let mut sources = self.sources.lock(); + if sources.remove(source).is_some() { + audit_metrics::set_fetch_busy_sources_tracked(sources.len()); + } + } + + /// Whether `source` should be asked last at `now`. + #[must_use] + pub fn is_cooling(&self, source: &PeerId, now: Instant) -> bool { + self.sources + .lock() + .get(source) + .is_some_and(|record| record.is_cooling(now)) + } + + /// Busy answers from `source` since it last served this requester, as far + /// as it is remembered. + #[must_use] + pub fn unserved_busy(&self, source: &PeerId) -> u32 { + self.sources + .lock() + .get(source) + .map_or(0, |record| record.unserved) + } + + /// When `source` last answered busy, if it is still remembered. + #[must_use] + pub fn last_busy(&self, source: &PeerId) -> Option { + self.sources + .lock() + .get(source) + .map(|record| record.last_busy) + } + + /// How many sources are tracked now. + #[must_use] + pub fn len(&self) -> usize { + self.sources.lock().len() + } + + /// Whether no source is tracked. + #[must_use] + pub fn is_empty(&self) -> bool { + self.sources.lock().is_empty() + } +} + +#[cfg(test)] +#[allow(clippy::unwrap_used, clippy::expect_used)] +mod tests { + use super::*; + + fn peer(i: u32) -> PeerId { + let mut bytes = [0u8; 32]; + for (byte, value) in bytes.iter_mut().zip(i.to_le_bytes()) { + *byte = value; + } + PeerId::from_bytes(bytes) + } + + fn limiter() -> Arc { + Arc::new(FetchBusyReplyLimiter::new()) + } + + /// Grant and immediately end the send, as a reply that went out at once. + fn grant(limiter: &Arc, p: &PeerId, now: Instant) -> bool { + limiter.try_acquire(p, now).is_ok() + } + + #[test] + fn busy_wait_spans_the_jitter_range() { + let remaining = Duration::from_secs(30); + assert_eq!(busy_wait(remaining, Duration::ZERO), FETCH_BUSY_WAIT_MIN); + assert_eq!( + busy_wait(remaining, FETCH_BUSY_WAIT_SPREAD), + FETCH_BUSY_WAIT_MAX + ); + assert_eq!( + busy_wait(remaining, Duration::from_secs(2)), + FETCH_BUSY_WAIT_MIN + Duration::from_secs(2) + ); + // Jitter past the spread is clamped, not added. + assert_eq!( + busy_wait(remaining, Duration::from_hours(1)), + FETCH_BUSY_WAIT_MAX + ); + } + + #[test] + fn busy_wait_never_outlasts_the_fetch_timeout() { + assert_eq!( + busy_wait(Duration::from_secs(3), FETCH_BUSY_WAIT_SPREAD), + Duration::from_secs(3) + ); + assert_eq!( + busy_wait(Duration::ZERO, FETCH_BUSY_WAIT_SPREAD), + Duration::ZERO + ); + } + + #[test] + fn a_busy_answer_never_holds_the_slot_past_the_fetch_timeout() { + let neutral = BusyAnswer { + cooling_for: FETCH_BUSY_WAIT_MIN, + charged: false, + }; + let charged = BusyAnswer { + cooling_for: FETCH_BUSY_WAIT_MIN, + charged: true, + }; + let sent = Instant::now(); + let deadline = sent + Duration::from_secs(30); + // A prompt answer: the busy wait within the allowance, the rest of the + // timeout past it. + assert_eq!( + busy_wake(neutral, FETCH_BUSY_WAIT_MAX, sent, deadline), + sent + FETCH_BUSY_WAIT_MAX + ); + assert_eq!( + busy_wake(charged, FETCH_BUSY_WAIT_MAX, sent, deadline), + deadline + ); + // A late answer, a second before the deadline, holds the slot for that + // second and no more; one after it, not at all. + for now in [ + sent + Duration::from_secs(29), + deadline + Duration::from_secs(5), + ] { + for answer in [neutral, charged] { + assert_eq!( + busy_wake(answer, FETCH_BUSY_WAIT_MAX, now, deadline), + deadline + ); + } + } + } + + #[test] + fn a_peer_gets_its_burst_then_one_reply_per_interval() { + let limiter = limiter(); + let p = peer(1); + let t0 = Instant::now(); + for _ in 0..PEER_BURST { + assert!(grant(&limiter, &p, t0)); + } + assert_eq!( + limiter.try_acquire(&p, t0).err(), + Some(BusyReplyDenied::RateLimited) + ); + assert!(!grant(&limiter, &p, t0 + PEER_INTERVAL / 2)); + assert!(grant(&limiter, &p, t0 + PEER_INTERVAL)); + assert!(!grant(&limiter, &p, t0 + PEER_INTERVAL)); + // A long quiet spell refills only up to the burst, never beyond it. + let later = t0 + PEER_INTERVAL * 100; + for _ in 0..PEER_BURST { + assert!(grant(&limiter, &p, later)); + } + assert!(!grant(&limiter, &p, later)); + } + + #[test] + fn at_most_a_burst_in_flight_per_peer() { + let limiter = limiter(); + let p = peer(1); + let t0 = Instant::now(); + let held: Vec = (0..PEER_BURST) + .map(|_| limiter.try_acquire(&p, t0).expect("within the burst")) + .collect(); + // Long enough later that the rate allows more, but the sends have not + // ended: the in-flight limit refuses. + let later = t0 + PEER_INTERVAL * 10; + assert_eq!( + limiter.try_acquire(&p, later).err(), + Some(BusyReplyDenied::PeerInFlight) + ); + // Another peer is not affected. + assert!(grant(&limiter, &peer(2), later)); + drop(held); + assert!(grant(&limiter, &p, later)); + } + + #[test] + fn one_peer_exhausting_its_allowance_leaves_others_theirs() { + let limiter = limiter(); + let t0 = Instant::now(); + for _ in 0..10 { + let _ = grant(&limiter, &peer(1), t0); + } + assert!(grant(&limiter, &peer(2), t0)); + } + + #[test] + fn the_global_allowance_caps_every_peer_together() { + let limiter = limiter(); + let t0 = Instant::now(); + let granted = (0..1_000u32) + .filter(|i| grant(&limiter, &peer(*i), t0)) + .count(); + assert_eq!(granted, usize::try_from(GLOBAL_BURST).unwrap()); + // Refills at the global rate: one second later, one second's worth. + let t1 = t0 + Duration::from_secs(1); + let granted = (1_000..2_000u32) + .filter(|i| grant(&limiter, &peer(*i), t1)) + .count(); + let per_second = Duration::from_secs(1).as_nanos() / GLOBAL_INTERVAL.as_nanos(); + assert_eq!(granted, usize::try_from(per_second).unwrap()); + } + + #[test] + fn a_global_refusal_spends_nothing_from_the_peer() { + let limiter = limiter(); + let t0 = Instant::now(); + for i in 0..GLOBAL_BURST { + assert!(grant(&limiter, &peer(i), t0)); + } + let p = peer(100_000); + assert!(!grant(&limiter, &p, t0)); + assert_eq!( + limiter.tracked_peers(), + usize::try_from(GLOBAL_BURST).unwrap() + ); + // Once the global allowance refills, the peer still has its whole burst. + let t1 = t0 + Duration::from_secs(1); + for _ in 0..PEER_BURST { + assert!(grant(&limiter, &p, t1)); + } + } + + #[test] + fn the_peer_table_stays_bounded_and_forgets_only_idle_peers() { + let limiter = limiter(); + let mut now = Instant::now(); + let mut next_peer = 0u32; + // Fill the table with replies still in flight, pacing to stay inside + // the global allowance. + let mut held = Vec::new(); + while held.len() < MAX_TRACKED_PEERS { + match limiter.try_acquire(&peer(next_peer), now) { + Ok(grant) => { + held.push(grant); + next_peer += 1; + } + Err(_) => now += GLOBAL_INTERVAL, + } + } + assert_eq!(limiter.tracked_peers(), MAX_TRACKED_PEERS); + // No entry can be forgotten, so a newcomer is refused rather than + // tracked, although the global allowance has long refilled. + let newcomer = peer(u32::MAX); + let later = now + Duration::from_secs(10); + assert_eq!( + limiter.try_acquire(&newcomer, later).err(), + Some(BusyReplyDenied::RateLimited) + ); + assert_eq!(limiter.tracked_peers(), MAX_TRACKED_PEERS); + // Once those replies end, idle entries make room. + drop(held); + assert!(grant(&limiter, &newcomer, later)); + assert!(limiter.tracked_peers() <= MAX_TRACKED_PEERS); + } + + #[test] + fn a_peer_with_a_reply_in_flight_is_never_forgotten() { + let limiter = limiter(); + let t0 = Instant::now(); + let held: Vec = (0..PEER_BURST) + .map(|_| limiter.try_acquire(&peer(0), t0).expect("grant")) + .collect(); + let mut now = t0; + for i in 1..u32::try_from(MAX_TRACKED_PEERS).unwrap() { + while !grant(&limiter, &peer(i), now) { + now += GLOBAL_INTERVAL; + } + } + // Long after, everyone else is idle and a full table sheds them, but + // never the peer whose reply is still in flight. + let later = now + Duration::from_hours(1); + assert!(grant(&limiter, &peer(u32::MAX), later)); + assert_eq!( + limiter.try_acquire(&peer(0), later).err(), + Some(BusyReplyDenied::PeerInFlight) + ); + drop(held); + assert!(grant(&limiter, &peer(0), later)); + } + + const WAIT: Duration = Duration::from_secs(5); + + #[test] + fn a_busy_source_cools_for_the_wait() { + let tracker = SourceBusyTracker::new(); + let (a, b) = (peer(1), peer(2)); + let t0 = Instant::now(); + assert!(!tracker.is_cooling(&a, t0)); + assert_eq!(tracker.record_busy(&a, t0, WAIT).cooling_for, WAIT); + assert!(tracker.is_cooling(&a, t0)); + assert!(!tracker.is_cooling(&b, t0)); + // Cooling ends after the wait. + assert!(!tracker.is_cooling(&a, t0 + WAIT)); + } + + #[test] + fn a_source_that_stays_busy_cools_longer_up_to_the_cap() { + let tracker = SourceBusyTracker::new(); + let a = peer(1); + let t0 = Instant::now(); + let cooldowns: Vec = (0..6) + .map(|_| tracker.record_busy(&a, t0, WAIT).cooling_for) + .collect(); + assert_eq!( + cooldowns, + vec![WAIT, WAIT * 2, WAIT * 4, WAIT * 8, WAIT * 8, WAIT * 8] + ); + // A longer wait reaches the cap. + let b = peer(2); + let long = Duration::from_secs(10); + for _ in 0..4 { + tracker.record_busy(&b, t0, long); + } + assert_eq!( + tracker.record_busy(&b, t0, long).cooling_for, + MAX_SOURCE_COOLDOWN + ); + } + + #[test] + fn a_running_cooldown_is_never_shortened() { + let tracker = SourceBusyTracker::new(); + let a = peer(1); + let t0 = Instant::now(); + for _ in 0..4 { + tracker.record_busy(&a, t0, Duration::from_secs(10)); + } + // Cooling until t0 + 60 s now. A later busy answer with a tiny wait + // would end far sooner; the running cooldown stands, and what is left + // of it is what the caller is told. + assert_eq!( + tracker + .record_busy(&a, t0 + Duration::from_secs(1), Duration::from_millis(1)) + .cooling_for, + Duration::from_secs(59) + ); + assert!(tracker.is_cooling(&a, t0 + Duration::from_secs(59))); + } + + #[test] + fn serving_once_clears_the_record() { + let tracker = SourceBusyTracker::new(); + let a = peer(1); + let t0 = Instant::now(); + for _ in 0..3 { + tracker.record_busy(&a, t0, WAIT); + } + tracker.record_served(&a); + assert!(!tracker.is_cooling(&a, t0)); + assert!(tracker.is_empty()); + assert_eq!(tracker.record_busy(&a, t0, WAIT).cooling_for, WAIT); + } + + #[test] + fn a_new_busy_spell_starts_the_cooldown_over() { + let tracker = SourceBusyTracker::new(); + let a = peer(1); + let t0 = Instant::now(); + for _ in 0..3 { + tracker.record_busy(&a, t0, WAIT); + } + // Long after its last busy answer, the next one starts a new spell: + // its cooldown is the first one's again. + let later = t0 + BUSY_SPELL_GAP + MAX_SOURCE_COOLDOWN; + assert_eq!(tracker.record_busy(&a, later, WAIT).cooling_for, WAIT); + assert_eq!(tracker.unserved_busy(&a), 4); + } + + /// A holder that keeps the data but answers busy to every fetch, one + /// every busy wait for over an hour. The record is per source, so these + /// answers count together whichever keys they answer. Only its first + /// `FETCH_BUSY_NEUTRAL_ALLOWANCE` answers are free. + #[test] + fn a_holder_that_never_serves_is_charged_past_its_allowance() { + let tracker = SourceBusyTracker::new(); + let holder = peer(1); + let t0 = Instant::now(); + let neutral = usize::try_from(FETCH_BUSY_NEUTRAL_ALLOWANCE).unwrap(); + let charged: Vec = (0..1_000u32) + .map(|i| tracker.record_busy(&holder, t0 + WAIT * i, WAIT).charged) + .collect(); + assert!(charged.iter().take(neutral).all(|c| !c)); + assert!(charged.iter().skip(neutral).all(|c| *c)); + } + + /// Two answers at once, one for each request in the reply burst, spend + /// two of the allowance: it counts answers, not rounds. + #[test] + fn answers_that_arrive_together_each_spend_the_allowance() { + let tracker = SourceBusyTracker::new(); + let holder = peer(1); + let t0 = Instant::now(); + for _ in 0..FETCH_BUSY_NEUTRAL_ALLOWANCE { + assert!(!tracker.record_busy(&holder, t0, WAIT).charged); + } + assert!(tracker.record_busy(&holder, t0, WAIT).charged); + } + + /// Honest overload comes and goes: a holder that serves this requester + /// once between busy spells, each as long as the allowance, is never + /// charged however many spells there are. + #[test] + fn a_holder_that_serves_between_busy_spells_is_never_charged() { + let tracker = SourceBusyTracker::new(); + let holder = peer(1); + let mut now = Instant::now(); + for _ in 0..100 { + for _ in 0..FETCH_BUSY_NEUTRAL_ALLOWANCE { + assert!(!tracker.record_busy(&holder, now, WAIT).charged); + now += WAIT; + } + tracker.record_served(&holder); + } + } + + /// Serving once restores the whole allowance, and another source serving + /// does not. + #[test] + fn serving_once_restores_the_allowance() { + let tracker = SourceBusyTracker::new(); + let (holder, other) = (peer(1), peer(2)); + let t0 = Instant::now(); + for _ in 0..FETCH_BUSY_NEUTRAL_ALLOWANCE { + tracker.record_busy(&holder, t0, WAIT); + } + // Another holder serving changes nothing for this one. + tracker.record_served(&other); + assert!(tracker.record_busy(&holder, t0, WAIT).charged); + tracker.record_served(&holder); + for _ in 0..FETCH_BUSY_NEUTRAL_ALLOWANCE { + assert!(!tracker.record_busy(&holder, t0, WAIT).charged); + } + assert!(tracker.record_busy(&holder, t0, WAIT).charged); + } + + /// Time does not restore the allowance: a holder asked only rarely, long + /// after each busy spell has ended, still uses it up across spells and is + /// charged until it serves. + #[test] + fn time_alone_never_restores_the_allowance() { + let tracker = SourceBusyTracker::new(); + let holder = peer(1); + let t0 = Instant::now(); + let apart = BUSY_SPELL_GAP * 6; + let charged: Vec = (0..=FETCH_BUSY_NEUTRAL_ALLOWANCE + 2) + .map(|i| { + let answer = tracker.record_busy(&holder, t0 + apart * i, WAIT); + // Every answer starts a new spell, so each cools only the wait. + assert_eq!(answer.cooling_for, WAIT); + answer.charged + }) + .collect(); + let neutral = usize::try_from(FETCH_BUSY_NEUTRAL_ALLOWANCE).unwrap(); + assert!(charged.iter().take(neutral).all(|c| !c)); + assert!(charged.iter().skip(neutral).all(|c| *c)); + } + + /// A full table forgets nothing, so many holders taking turns to answer + /// busy cannot push one another's records out to start again. A newcomer + /// is charged instead, and a tracked holder keeps counting. + #[test] + fn a_full_table_forgets_nothing_and_charges_newcomers() { + let tracker = SourceBusyTracker::new(); + let t0 = Instant::now(); + let max = u32::try_from(MAX_TRACKED_SOURCES).unwrap(); + for i in 0..max { + assert!(!tracker.record_busy(&peer(i), t0, WAIT).charged); + } + assert_eq!(tracker.len(), MAX_TRACKED_SOURCES); + // One more holder than the table holds, all taking turns, long apart: + // every later round is charged for the newcomer, and nobody's count + // starts again. + let newcomer = peer(max); + let mut now = t0; + for _ in 0..=FETCH_BUSY_NEUTRAL_ALLOWANCE { + now += BUSY_SPELL_GAP * 2; + assert_eq!( + tracker.record_busy(&newcomer, now, WAIT), + BusyAnswer { + cooling_for: Duration::ZERO, + charged: true, + } + ); + for i in 0..max { + tracker.record_busy(&peer(i), now, WAIT); + } + } + assert_eq!(tracker.len(), MAX_TRACKED_SOURCES); + assert!(!tracker.is_cooling(&newcomer, now)); + assert_eq!(tracker.unserved_busy(&newcomer), 0); + assert!( + (0..max).all(|i| tracker.unserved_busy(&peer(i)) == FETCH_BUSY_NEUTRAL_ALLOWANCE + 2) + ); + assert!(tracker.record_busy(&peer(0), now, WAIT).charged); + // Serving makes room again. + tracker.record_served(&peer(0)); + assert!(!tracker.record_busy(&newcomer, now, WAIT).charged); + } +} diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 62937413..a18084b8 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -22,6 +22,7 @@ pub mod bootstrap; pub mod commitment; pub mod commitment_state; pub mod config; +pub mod fetch_busy; pub mod fresh; pub mod neighbor_sync; pub mod paid_list; @@ -42,6 +43,8 @@ use std::fmt; use std::num::NonZeroUsize; use std::ops::ControlFlow; use std::path::{Path, PathBuf}; +#[cfg(any(test, feature = "test-utils"))] +use std::sync::atomic::AtomicBool; use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::Arc; use std::time::{Duration, Instant, SystemTime}; @@ -56,12 +59,13 @@ use parking_lot::Mutex; use rand::rngs::StdRng; use rand::{Rng, SeedableRng}; use tokio::sync::broadcast::error::RecvError; -use tokio::sync::{mpsc, Notify, RwLock, Semaphore}; +use tokio::sync::{mpsc, Notify, OwnedSemaphorePermit, RwLock, Semaphore}; use tokio::task::JoinHandle; use tokio_util::sync::CancellationToken; use tokio_util::task::TaskTracker; use crate::ant_protocol::XorName; +use crate::client::xor_distance; use crate::error::{Error, Result}; use crate::payment::{ PaymentVerifier, VerificationContext, MAX_PAYMENT_PROOF_SIZE_BYTES, @@ -72,7 +76,7 @@ use crate::pointer::{FinalStateWitness, PointerService}; use crate::replication::audit::AuditTickResult; use crate::replication::audit_coordinator::AuditChallengeCoordinator; use crate::replication::audit_metrics::{ - AuditResponderClass, AuditResponderDropReason, AuditResponderMetrics, + AuditResponderClass, AuditResponderDropReason, AuditResponderMetrics, FetchBusyReplyOutcome, ReplicationResponderClass, ResponderAdmissionCeiling, }; use crate::replication::commitment::{commitment_hash, StorageCommitment}; @@ -88,6 +92,10 @@ use crate::replication::config::{ SUBTREE_AUDIT_PROTOCOL_ID, SUBTREE_ROUND1_WORK_BURST_BYTES, SUBTREE_ROUND1_WORK_REFILL_BYTES_PER_SEC, SUBTREE_SESSION_TTL, }; +use crate::replication::fetch_busy::{ + busy_wait, busy_wake, BusyReplyDenied, BusyReplyGrant, FetchBusyReplyLimiter, + SourceBusyTracker, FETCH_BUSY_REPLY_SEND_TIMEOUT, FETCH_BUSY_WAIT_SPREAD, +}; use crate::replication::paid_list::PaidList; use crate::replication::protocol::{ FreshReplicationResponse, NeighborSyncResponse, ReplicationMessage, ReplicationMessageBody, @@ -95,7 +103,12 @@ use crate::replication::protocol::{ }; use crate::replication::quorum::KeyVerificationOutcome; use crate::replication::recent_provers::RecentProvers; -use crate::replication::scheduling::{CapacityDisplacement, DeferralOutcome, ReplicationQueues}; +#[cfg(any(test, feature = "test-utils"))] +pub use crate::replication::scheduling::FetchPipelineSnapshot; +use crate::replication::scheduling::{ + fetch_source_limit, CapacityDisplacement, Contact, DeferralOutcome, FetchDispatchStats, + FetchRetry, ReplicationQueues, +}; use crate::replication::storage_commitment_audit::PointerBindings; use crate::replication::types::{ AuditFailureReason, BootstrapClaimObservation, BootstrapState, FailureEvidence, @@ -116,6 +129,16 @@ use saorsa_pqc::api::sig::{MlDsaSecretKey, MlDsaVariant}; /// counted: it is not a saturation signal. static FIRST_AUDIT_INGRESS_DROPPED: AtomicU64 = AtomicU64::new(0); +/// One sequence over every fetch request this process sends and every answer +/// it receives, so the fetch scheduler can order a request against an answer +/// even when both read the same clock value. See [`Contact`]. +static FETCH_CONTACT_SEQUENCE: AtomicU64 = AtomicU64::new(0); + +/// The next place in [`FETCH_CONTACT_SEQUENCE`]. +fn next_fetch_contact() -> u64 { + FETCH_CONTACT_SEQUENCE.fetch_add(1, Ordering::Relaxed) +} + /// Record one ingress-full drop. Called by the payment verifier's `try_send` /// sites; read by the drainer's periodic summary. pub(crate) fn note_monetized_ingress_drop() { @@ -1129,6 +1152,31 @@ const FETCH_RESPONDER_MAX_OUTSTANDING: usize = FETCH_RESPONDER_WORKER_LIMIT * 4; /// of the three workers available to other peers under a single-source flood. const FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER: u32 = 2; +/// Maximum busy replies being sent at once. +/// +/// [`FetchBusyReplyLimiter`] bounds how often busy replies are sent; this +/// bounds how many sends are open at once, so peers that are slow to take a +/// reply cannot pile up send tasks. Past it, a refusal falls back to the +/// silence every refusal got before busy replies existed. +const FETCH_BUSY_REPLY_MAX_OUTSTANDING: usize = FETCH_RESPONDER_MAX_OUTSTANDING * 4; +/// Maximum fetches this node keeps in flight to any one source peer that is +/// answering. +/// +/// The share every responder grants one requester +/// ([`FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER`], unchanged since it was +/// introduced). A request past that share is dropped without an answer, so +/// sending it costs the full fetch timeout and a timeout trust event against a +/// holder that did nothing wrong. Staying within it removes those bursts; it +/// cannot rule out every refusal, because a request this node has already +/// timed out on may still be holding its share at the responder until a worker +/// sheds it. +/// +/// A source whose request got no answer, and which has not answered since that +/// request was sent, has no share to stay within. It is fetched from without +/// this cap, as before the cap existed, until it answers again or two fetch +/// timeouts pass, so a holder that went silent does not hold its keys back. +const FETCH_MAX_IN_FLIGHT_PER_SOURCE: usize = FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER as usize; + /// Maximum verification batches served concurrently. /// /// Point lookups are fast, but a batch can contain 8,192 of them. Two @@ -1222,7 +1270,9 @@ fn paid_notify_payment_context() -> VerificationContext { } /// Boxed future type for in-flight fetch tasks. -type FetchFuture = Pin)> + Send>>; +/// A fetch task's key, its outcome (`None` when the task panicked), and when +/// the task finished. +type FetchFuture = Pin, Instant)> + Send>>; /// Fresh-offer keys currently being handled, each with the proofs still to try. /// @@ -1667,6 +1717,8 @@ pub struct ReplicationEngine { payment_verifier: Arc, /// Replication pipeline queues. queues: Arc>, + /// Fetches the worker runs at once: one per hardware thread. + fetch_concurrency: usize, /// Neighbor sync cycle state. sync_state: Arc>, /// Per-peer sync history (for `RepairOpportunity`). @@ -1787,6 +1839,24 @@ pub struct ReplicationEngine { fetch_responder_admission_semaphore: Arc, /// Per-source fetch responder counts for flood-fair admission. fetch_responder_inflight: Arc>>, + /// Test-only hold on serving admitted fetch requests. + #[cfg(any(test, feature = "test-utils"))] + fetch_serve_gate: Arc, + /// Test-only count of this engine's fetch refusals because its admission + /// pool was full. + #[cfg(any(test, feature = "test-utils"))] + fetch_pool_full_refusals: Arc, + /// Test-only: refuse every fetch past admission silently, without a busy + /// reply, as a holder that has stopped answering would. + #[cfg(any(test, feature = "test-utils"))] + fetch_busy_replies_off: Arc, + /// Permits for busy replies being sent to fetch requests refused at + /// admission. + fetch_busy_reply_semaphore: Arc, + /// Per-peer and global rate bound on those busy replies. + fetch_busy_reply_limiter: Arc, + /// Sources that recently answered this node's own fetches busy. + fetch_busy_sources: Arc, /// Worker permits for lookup-heavy verification batches. verification_responder_worker_semaphore: Arc, /// Admission permits bounding verification workers and queued waiters. @@ -1872,7 +1942,7 @@ impl ReplicationEngine { /// /// Returns an error if the `PaidList` LMDB environment cannot be opened /// or if the configuration fails validation. - #[allow(clippy::too_many_arguments)] + #[allow(clippy::too_many_arguments, clippy::too_many_lines)] pub async fn new( config: ReplicationConfig, p2p_node: Arc, @@ -1902,13 +1972,25 @@ impl ReplicationEngine { let (monetized_pin_tx, monetized_pin_rx) = mpsc::channel(config::FIRST_AUDIT_INGRESS_CAPACITY); + let source_limit = + fetch_source_limit(config.close_group_size, config.paid_list_close_group_size); + // Fetches record busy answers here; the scheduler reads it to put a + // holder that just answered busy last. + let fetch_busy_sources = Arc::new(SourceBusyTracker::new()); + let engine = Self { config: Arc::clone(&config), p2p_node, storage, paid_list, payment_verifier, - queues: Arc::new(RwLock::new(ReplicationQueues::new())), + queues: Arc::new(RwLock::new( + ReplicationQueues::new() + .with_fetch_source_limit(source_limit) + .with_fetch_request_timeout(config.fetch_request_timeout) + .with_busy_sources(Arc::clone(&fetch_busy_sources)), + )), + fetch_concurrency: max_parallel_fetch(), sync_state: Arc::new(RwLock::new(initial_neighbors)), sync_history: Arc::new(RwLock::new(HashMap::new())), audit_on_gossip_cooldown: Arc::new(RwLock::new(HashMap::new())), @@ -1940,6 +2022,15 @@ impl ReplicationEngine { FETCH_RESPONDER_MAX_OUTSTANDING, )), fetch_responder_inflight: Arc::new(RwLock::new(HashMap::new())), + #[cfg(any(test, feature = "test-utils"))] + fetch_serve_gate: Arc::new(FetchServeGate::default()), + #[cfg(any(test, feature = "test-utils"))] + fetch_pool_full_refusals: Arc::new(AtomicU64::new(0)), + #[cfg(any(test, feature = "test-utils"))] + fetch_busy_replies_off: Arc::new(AtomicBool::new(false)), + fetch_busy_reply_semaphore: Arc::new(Semaphore::new(FETCH_BUSY_REPLY_MAX_OUTSTANDING)), + fetch_busy_reply_limiter: Arc::new(FetchBusyReplyLimiter::new()), + fetch_busy_sources, verification_responder_worker_semaphore: Arc::new(Semaphore::new( VERIFICATION_RESPONDER_WORKER_LIMIT, )), @@ -2172,6 +2263,54 @@ impl ReplicationEngine { queues.enqueue_fetch(key, distance, sources) } + /// Test-only: whether this engine puts `source` last when choosing where + /// to start or retry a fetch, because it recently answered busy. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn fetch_source_cooling_for_test(&self, source: &PeerId) -> bool { + self.fetch_busy_sources.is_cooling(source, Instant::now()) + } + + /// Test-only: when `source` last answered this engine's fetches busy, if + /// it is still remembered. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn fetch_source_last_busy_for_test(&self, source: &PeerId) -> Option { + self.fetch_busy_sources.last_busy(source) + } + + /// Test-only: busy answers `source` has given this engine's fetches since + /// it last served one, which spend its neutral allowance. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn fetch_source_unserved_busy_for_test(&self, source: &PeerId) -> u32 { + self.fetch_busy_sources.unserved_busy(source) + } + + /// Test-only: fetch requests this engine has admitted and not yet finished + /// serving. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn fetch_admitted_for_test(&self) -> usize { + FETCH_RESPONDER_MAX_OUTSTANDING + .saturating_sub(self.fetch_responder_admission_semaphore.available_permits()) + } + + /// Test-only: fetch requests this engine has refused because its + /// admission pool was full. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn fetch_pool_full_refusals_for_test(&self) -> u64 { + self.fetch_pool_full_refusals.load(Ordering::Relaxed) + } + + /// Test-only: from now on, refuse fetches past admission silently, with no + /// busy reply, as a holder that has stopped answering would. + #[cfg(any(test, feature = "test-utils"))] + pub fn turn_off_fetch_busy_replies_for_test(&self) { + self.fetch_busy_replies_off.store(true, Ordering::Relaxed); + } + /// Test-only: whether `key` is still tracked in any fetch-pipeline stage /// (pending verification, fetch queue, or in-flight fetch). #[cfg(any(test, feature = "test-utils"))] @@ -2194,6 +2333,35 @@ impl ReplicationEngine { self.queues.read().await.fetch_queued_or_in_flight(key) } + /// Test-only: run `concurrency` fetches at once instead of one per hardware + /// thread. Call before [`Self::start`], which reads it. Per engine, so it + /// affects no other node or test. + #[cfg(any(test, feature = "test-utils"))] + pub fn set_fetch_concurrency_for_test(&mut self, concurrency: usize) { + self.fetch_concurrency = concurrency; + } + + /// Test-only: hold this engine's admitted fetch requests before they are + /// served, until [`Self::open_fetch_serve_gate_for_test`]. + #[cfg(any(test, feature = "test-utils"))] + pub fn close_fetch_serve_gate_for_test(&self) { + self.fetch_serve_gate.close(); + } + + /// Test-only: release this engine's held fetch requests and stop holding + /// new ones. + #[cfg(any(test, feature = "test-utils"))] + pub fn open_fetch_serve_gate_for_test(&self) { + self.fetch_serve_gate.open(); + } + + /// Test-only: every key queued for fetch and every in-flight fetch with + /// the source it was sent to, read under one lock. + #[cfg(any(test, feature = "test-utils"))] + pub async fn fetch_pipeline_snapshot_for_test(&self) -> FetchPipelineSnapshot { + self.queues.read().await.fetch_pipeline_snapshot() + } + /// Test-only: place `key` into pending verification as though `hinter` had /// just advertised it with a replica hint. Returns whether it was admitted. /// @@ -2434,6 +2602,12 @@ impl ReplicationEngine { } } + // A test may shut down with the fetch serve gate still closed. The + // requests it holds wait on the gate before they reach a worker pool, + // so release them, or the drain below never finishes. + #[cfg(any(test, feature = "test-utils"))] + self.fetch_serve_gate.open(); + // Close the responder worker pools. Detached tasks still queued for a // worker take the `Err` arm of `acquire_owned()` and exit immediately // rather than waiting for a slot that will never free up — without @@ -3063,6 +3237,14 @@ impl ReplicationEngine { let fetch_responder_admission_semaphore = Arc::clone(&self.fetch_responder_admission_semaphore); let fetch_responder_inflight = Arc::clone(&self.fetch_responder_inflight); + #[cfg(any(test, feature = "test-utils"))] + let fetch_serve_gate = Arc::clone(&self.fetch_serve_gate); + #[cfg(any(test, feature = "test-utils"))] + let fetch_pool_full_refusals = Arc::clone(&self.fetch_pool_full_refusals); + #[cfg(any(test, feature = "test-utils"))] + let fetch_busy_replies_off = Arc::clone(&self.fetch_busy_replies_off); + let fetch_busy_reply_semaphore = Arc::clone(&self.fetch_busy_reply_semaphore); + let fetch_busy_reply_limiter = Arc::clone(&self.fetch_busy_reply_limiter); let verification_responder_worker_semaphore = Arc::clone(&self.verification_responder_worker_semaphore); let verification_responder_admission_semaphore = @@ -3119,6 +3301,14 @@ impl ReplicationEngine { fetch_responder_worker_semaphore, fetch_responder_admission_semaphore, fetch_responder_inflight, + #[cfg(any(test, feature = "test-utils"))] + fetch_serve_gate, + #[cfg(any(test, feature = "test-utils"))] + fetch_pool_full_refusals, + #[cfg(any(test, feature = "test-utils"))] + fetch_busy_replies_off, + fetch_busy_reply_semaphore, + fetch_busy_reply_limiter, verification_responder_worker_semaphore, verification_responder_admission_semaphore, verification_responder_inflight, @@ -3881,72 +4071,103 @@ impl ReplicationEngine { let is_bootstrapping = Arc::clone(&self.is_bootstrapping); let bootstrap_complete_notify = Arc::clone(&self.bootstrap_complete_notify); let detached_task_tracker = self.detached_task_tracker.clone(); - let concurrency = max_parallel_fetch(); + let busy_sources = Arc::clone(&self.fetch_busy_sources); + let concurrency = self.fetch_concurrency; info!("Fetch worker concurrency set to {concurrency} (hardware threads)"); let handle = tokio::spawn(async move { - // Each in-flight future yields (key, Option) so we - // always recover the key — even if the inner task panics. + // Each in-flight future yields (key, Option, when it + // finished), so we always recover the key, even if the inner task + // panics. let mut in_flight = FuturesUnordered::::new(); + let mut dispatch_window = FetchDispatchWindow::new(); + // Each answer as it arrives, ahead of its fetch's outcome. + let (answers, mut early_answers) = mpsc::unbounded_channel(); loop { - // Fill up to `concurrency` slots from the queue. - { - let mut q = queues.write().await; - while in_flight.len() < concurrency { - let Some(candidate) = q.dequeue_fetch() else { - break; - }; - let fetch_key = candidate.key; - let Some(&source) = candidate.sources.first() else { - warn!( - "Fetch candidate {} has no sources; requeueing for verification", - hex::encode(fetch_key) - ); - let _ = q.requeue_candidate_for_verification( - candidate, - config.verification_request_timeout, - ); - continue; - }; - q.start_dequeued_fetch(candidate, source); - - let p2p = Arc::clone(&p2p); - let storage = Arc::clone(&storage); - let config = Arc::clone(&config); - let token = shutdown.clone(); - let tracker = detached_task_tracker.clone(); - in_flight.push(Box::pin(async move { - // Tracked so shutdown() still awaits the task if - // this awaiter is dropped (e.g. the worker is - // aborted): it holds Arc and must - // not outlive the engine. - let handle = tracker.spawn(async move { - // Cancel-aware: abort when the engine shuts down. - tokio::select! { - () = token.cancelled() => FetchOutcome { - key: fetch_key, - result: FetchResult::SourceFailed, - }, - outcome = execute_single_fetch( - p2p, storage, config, fetch_key, source, - ) => outcome, - } - }); - match handle.await { - Ok(outcome) => (outcome.key, Some(outcome)), - Err(e) => { - error!( - "Fetch task for {} panicked: {e}", - hex::encode(fetch_key) - ); - (fetch_key, None) - } + // Fill free slots from the queue, never sending more than + // `FETCH_MAX_IN_FLIGHT_PER_SOURCE` of this node's fetches to any + // one answering source at once. + let lock_requested = Instant::now(); + let mut q = queues.write().await; + let lock_acquired = Instant::now(); + record_early_answers(&mut q, &mut early_answers); + let free_slots = concurrency.saturating_sub(in_flight.len()); + let dispatch = q.dispatch_fetches(free_slots, FETCH_MAX_IN_FLIGHT_PER_SOURCE); + let parked = dispatch.parked; + for candidate in dispatch.sourceless { + warn!( + "Fetch candidate {} has no untried sources; requeueing for verification", + hex::encode(candidate.key) + ); + let _ = q.requeue_candidate_for_verification( + candidate, + config.verification_request_timeout, + ); + } + for (fetch_key, source) in dispatch.started { + let p2p = Arc::clone(&p2p); + let storage = Arc::clone(&storage); + let config = Arc::clone(&config); + let busy_sources = Arc::clone(&busy_sources); + let answers = answers.clone(); + let token = shutdown.clone(); + let tracker = detached_task_tracker.clone(); + in_flight.push(Box::pin(async move { + // Tracked so shutdown() still awaits the task if + // this awaiter is dropped (e.g. the worker is + // aborted): it holds Arc and must + // not outlive the engine. + let handle = tracker.spawn(async move { + // Cancel-aware: abort when the engine shuts down. + // Shutdown is checked first, so a fetch that is + // ready to charge a busy source at the same + // moment is cancelled uncharged instead. + tokio::select! { + biased; + () = token.cancelled() => FetchOutcome { + key: fetch_key, + source, + result: FetchResult::SourceFailed, + contact: Contact::Unknown, + }, + outcome = execute_single_fetch( + p2p, + storage, + config, + busy_sources, + fetch_key, + source, + answers, + ) => outcome, } - })); - } - } // release queues write lock + }); + let joined = handle.await; + let completed_at = Instant::now(); + match joined { + Ok(outcome) => (outcome.key, Some(outcome), completed_at), + Err(e) => { + error!("Fetch task for {} panicked: {e}", hex::encode(fetch_key)); + (fetch_key, None, completed_at) + } + } + })); + } + drop(q); + dispatch_window.parks += parked; + dispatch_window.record_fill( + lock_acquired.duration_since(lock_requested), + lock_acquired.elapsed(), + ); + if dispatch_window.age() >= FETCH_DISPATCH_SUMMARY_INTERVAL { + let stats_requested = Instant::now(); + let stats = queues + .write() + .await + .take_fetch_dispatch_stats(FETCH_MAX_IN_FLIGHT_PER_SOURCE); + dispatch_window.log_and_reset(&stats, stats_requested.elapsed()); + } if in_flight.is_empty() { // No work — wait for new items or shutdown. @@ -3961,22 +4182,33 @@ impl ReplicationEngine { // Wait for the next fetch to complete and process the result. tokio::select! { () = shutdown.cancelled() => break, - Some((key, maybe_outcome)) = in_flight.next() => { + Some((key, maybe_outcome, completed_at)) = in_flight.next() => { let mut q = queues.write().await; + record_early_answers(&mut q, &mut early_answers); + let attempt = attempt_latency(q.in_flight_started_at(&key), completed_at); + dispatch_window.record_outcome(maybe_outcome.as_ref(), attempt); let terminal = if let Some(outcome) = maybe_outcome { + q.record_fetch_contact(outcome.source, outcome.contact); match apply_fetch_result( &mut q, &key, &outcome.result, config.verification_request_timeout, + xor_distance(&key, p2p.peer_id().as_bytes()), ) { FetchFollowUp::Terminal => true, FetchFollowUp::RequeuedForVerification => false, + FetchFollowUp::Parked => { + dispatch_window.retry_parks += 1; + false + } FetchFollowUp::RetryFrom(next_peer) => { // Spawn a new fetch task for the next source. let p2p = Arc::clone(&p2p); let storage = Arc::clone(&storage); let config = Arc::clone(&config); + let busy_sources = Arc::clone(&busy_sources); + let answers = answers.clone(); let token = shutdown.clone(); let tracker = detached_task_tracker.clone(); let fetch_key = key; @@ -3985,23 +4217,36 @@ impl ReplicationEngine { // initial fetch spawn above. let handle = tracker.spawn(async move { tokio::select! { + biased; () = token.cancelled() => FetchOutcome { key: fetch_key, + source: next_peer, result: FetchResult::SourceFailed, + contact: Contact::Unknown, }, outcome = execute_single_fetch( - p2p, storage, config, fetch_key, next_peer, + p2p, + storage, + config, + busy_sources, + fetch_key, + next_peer, + answers, ) => outcome, } }); - match handle.await { - Ok(outcome) => (outcome.key, Some(outcome)), + let joined = handle.await; + let completed_at = Instant::now(); + match joined { + Ok(outcome) => { + (outcome.key, Some(outcome), completed_at) + } Err(e) => { error!( "Fetch task for {} panicked: {e}", hex::encode(fetch_key) ); - (fetch_key, None) + (fetch_key, None, completed_at) } } })); @@ -4043,7 +4288,22 @@ impl ReplicationEngine { // Cancel and drain remaining in-flight fetches on shutdown. // The CancellationToken is already cancelled by this point, so // spawned tasks will see cancellation via their select! branches. - while in_flight.next().await.is_some() {} + // Their outcomes are counted but not applied, so the final + // summary's in-flight count still includes them. + let mut drained = Vec::new(); + while let Some(resolved) = in_flight.next().await { + drained.push(resolved); + } + let stats_requested = Instant::now(); + let mut q = queues.write().await; + for (key, maybe_outcome, completed_at) in &drained { + let attempt = attempt_latency(q.in_flight_started_at(key), *completed_at); + dispatch_window.record_outcome(maybe_outcome.as_ref(), attempt); + } + let stats = q.take_fetch_dispatch_stats(FETCH_MAX_IN_FLIGHT_PER_SOURCE); + drop(q); + dispatch_window.shutdown = true; + dispatch_window.log_and_reset(&stats, stats_requested.elapsed()); debug!("Fetch worker shut down"); }); self.task_handles.push(handle); @@ -4589,6 +4849,18 @@ struct ReplicationMessageHandlerContext { fetch_responder_worker_semaphore: Arc, fetch_responder_admission_semaphore: Arc, fetch_responder_inflight: Arc>>, + /// Test-only hold on serving admitted fetch requests. + #[cfg(any(test, feature = "test-utils"))] + fetch_serve_gate: Arc, + /// Test-only count of this engine's fetch refusals because its admission + /// pool was full. + #[cfg(any(test, feature = "test-utils"))] + fetch_pool_full_refusals: Arc, + /// Test-only: refuse silently, without busy replies. + #[cfg(any(test, feature = "test-utils"))] + fetch_busy_replies_off: Arc, + fetch_busy_reply_semaphore: Arc, + fetch_busy_reply_limiter: Arc, verification_responder_worker_semaphore: Arc, verification_responder_admission_semaphore: Arc, verification_responder_inflight: Arc>>, @@ -7112,6 +7384,45 @@ pub fn verification_requests_for_key_from_for_test(requester: &PeerId, key: &Xor }) } +/// Test-only: holds an engine's admitted fetch requests before they are served. +/// +/// A held request keeps its admission slot and its requester's per-peer share, +/// exactly as a request waiting for a worker does, so a test can read what a +/// requester has outstanding without racing the speed at which the host serves +/// a chunk. One per engine, so closing it affects no other node or test. +#[cfg(any(test, feature = "test-utils"))] +#[derive(Debug, Default)] +struct FetchServeGate { + closed: AtomicBool, + opened: Notify, +} + +#[cfg(any(test, feature = "test-utils"))] +impl FetchServeGate { + fn close(&self) { + self.closed.store(true, Ordering::Release); + } + + fn open(&self) { + self.closed.store(false, Ordering::Release); + self.opened.notify_waiters(); + } + + /// Wait while the gate is closed. The wakeup is registered before the flag + /// is read, so an open that lands between the two is not missed. + async fn wait_open(&self) { + loop { + let opened = self.opened.notified(); + tokio::pin!(opened); + opened.as_mut().enable(); + if !self.closed.load(Ordering::Acquire) { + return; + } + opened.await; + } + } +} + async fn handle_verification_request( source: &PeerId, request: &protocol::VerificationRequest, @@ -7351,11 +7662,42 @@ async fn dispatch_fetch_request( ReplicationResponderClass::Fetch, failure.reason.into(), ); + #[cfg(any(test, feature = "test-utils"))] + if failure.reason == ResponderRejectReason::GlobalPoolFull { + ctx.fetch_pool_full_refusals.fetch_add(1, Ordering::Relaxed); + } + let busy_reply = fetch_busy_reply_permit(&source, request.busy_reply, ctx); + let capability = if request.busy_reply { + "asked" + } else { + "not_asked" + }; + let reply = if busy_reply.is_some() { + "scheduled" + } else { + "none" + }; + // Same line and counter as before busy replies existed, so the + // existing refusal counts carry on. `capability` says whether the + // requester asked for a busy reply, and `reply` whether one was + // scheduled; whether it was delivered is in the busy-reply counters. warn!( responder_class = "fetch", source = %source, + capability, + reply, "Fetch response dropped at admission: {failure}" ); + if let Some((permit, grant)) = busy_reply { + send_fetch_busy_reply( + source, + request.key, + ctx, + request_id, + rr_message_id, + (permit, grant), + ); + } return Ok(()); } }; @@ -7365,8 +7707,12 @@ async fn dispatch_fetch_request( let p2p_node = Arc::clone(&ctx.p2p_node); let request_timeout = ctx.config.fetch_request_timeout; let rr_message_id = rr_message_id.map(ToOwned::to_owned); + #[cfg(any(test, feature = "test-utils"))] + let serve_gate = Arc::clone(&ctx.fetch_serve_gate); ctx.detached_task_tracker.spawn(async move { let _guard = guard; + #[cfg(any(test, feature = "test-utils"))] + serve_gate.wait_open().await; let Ok(_worker_permit) = worker_semaphore.acquire_owned().await else { debug!( responder_class = "fetch", @@ -7402,6 +7748,95 @@ async fn dispatch_fetch_request( Ok(()) } +/// Whether a fetch refused at admission may be answered `Busy`: the send +/// permit and the limiter's grant to do it with. +/// +/// Only a requester that asked for busy replies gets one, and only while a +/// send permit is free, within the limiter's per-peer and global allowances, +/// and while fewer than the limiter's in-flight limit of busy replies to that +/// peer (two) are still being sent. Every other +/// refusal stays silent, exactly as all of them were before busy replies +/// existed. +fn fetch_busy_reply_permit( + source: &PeerId, + busy_reply: bool, + ctx: &ReplicationMessageHandlerContext, +) -> Option<(OwnedSemaphorePermit, BusyReplyGrant)> { + if !busy_reply { + audit_metrics::record_fetch_busy_reply(FetchBusyReplyOutcome::NotAsked); + return None; + } + #[cfg(any(test, feature = "test-utils"))] + if ctx.fetch_busy_replies_off.load(Ordering::Relaxed) { + return None; + } + // The send permit first: if none is free, the reply would not be sent, so + // the rate allowance is left unspent. + let Ok(permit) = Arc::clone(&ctx.fetch_busy_reply_semaphore).try_acquire_owned() else { + audit_metrics::record_fetch_busy_reply(FetchBusyReplyOutcome::SendPoolFull); + return None; + }; + match ctx + .fetch_busy_reply_limiter + .try_acquire(source, Instant::now()) + { + Ok(grant) => Some((permit, grant)), + Err(denied) => { + audit_metrics::record_fetch_busy_reply(match denied { + BusyReplyDenied::RateLimited => FetchBusyReplyOutcome::RateLimited, + BusyReplyDenied::PeerInFlight => FetchBusyReplyOutcome::PeerInFlight, + }); + None + } + } +} + +/// Tell a requester its fetch was refused at admission, off the serial message +/// loop. +/// +/// The send runs on the detached task tracker, holding the send permit and +/// one of the peer's in-flight places until it ends, so a slow or stalled peer +/// can neither block other replication traffic nor hold more permits than its +/// in-flight limit (two). It +/// is abandoned after [`FETCH_BUSY_REPLY_SEND_TIMEOUT`], and it races the +/// engine's shutdown, so `shutdown()` never waits on a peer that is slow to +/// take the reply. +fn send_fetch_busy_reply( + source: PeerId, + key: XorName, + ctx: &ReplicationMessageHandlerContext, + request_id: u64, + rr_message_id: Option<&str>, + held: (OwnedSemaphorePermit, BusyReplyGrant), +) { + let p2p_node = Arc::clone(&ctx.p2p_node); + let shutdown = ctx.shutdown.clone(); + let rr_message_id = rr_message_id.map(ToOwned::to_owned); + ctx.detached_task_tracker.spawn(async move { + let _held = held; + let delivery = tokio::time::timeout( + FETCH_BUSY_REPLY_SEND_TIMEOUT, + send_replication_response_checked( + &source, + &p2p_node, + request_id, + ReplicationMessageBody::FetchResponse(protocol::FetchResponse::Busy { key }), + rr_message_id.as_deref(), + None, + ), + ); + let outcome = tokio::select! { + () = shutdown.cancelled() => FetchBusyReplyOutcome::Cancelled, + sent = delivery => match sent { + Ok(true) => FetchBusyReplyOutcome::Sent, + Ok(false) => FetchBusyReplyOutcome::SendFailed, + Err(_) => FetchBusyReplyOutcome::TimedOut, + }, + }; + audit_metrics::record_fetch_busy_reply(outcome); + }); +} + fn request_is_stale(received_at: Instant, timeout: Duration) -> bool { received_at.elapsed() >= timeout } @@ -7666,15 +8101,7 @@ async fn send_replication_response_checked( return false; } }; - // V2-684: per-peer served-bytes attribution for the heavy serve paths. - // `FetchResponse` carries ~99% of served bytes; `NeighborSyncResponse` is - // included for completeness. Other response variants (verification/audit/ - // commitment) are intentionally excluded — the round-2 audit reply is now a - // few-KB verified slice (V2-685), not a full-chunk transfer, so it is light. - if matches!( - msg.body, - ReplicationMessageBody::FetchResponse(_) | ReplicationMessageBody::NeighborSyncResponse(_) - ) { + if counts_as_served(&msg.body) { protocol::record_served(peer, encoded.len()); } let protocol = protocol.unwrap_or_else(|| response_protocol_for(&msg.body)); @@ -7692,6 +8119,26 @@ async fn send_replication_response_checked( true } +/// Whether a response's bytes count towards the peer it is sent to in the +/// per-peer served-bytes attribution (V2-684). +/// +/// That attribution covers the heavy serve paths. `FetchResponse` carries ~99% +/// of served bytes; `NeighborSyncResponse` is included for completeness. Other +/// response variants (verification/audit/commitment) are intentionally +/// excluded — the round-2 audit reply is now a few-KB verified slice (V2-685), +/// not a full-chunk transfer, so it is light. A busy reply serves nothing, so +/// it is left out too. +const fn counts_as_served(body: &ReplicationMessageBody) -> bool { + matches!( + body, + ReplicationMessageBody::FetchResponse( + protocol::FetchResponse::Success { .. } + | protocol::FetchResponse::NotFound { .. } + | protocol::FetchResponse::Error { .. } + ) | ReplicationMessageBody::NeighborSyncResponse(_) + ) +} + async fn record_sent_replica_hints( peer: &PeerId, hints: &[neighbor_sync::SentReplicaHint], @@ -8931,7 +9378,10 @@ async fn complete_bootstrap( /// Result classification for a single fetch attempt. enum FetchResult { /// Data fetched, integrity-checked, and stored successfully. - Stored, + Stored { + /// Size of the stored chunk, for the per-source scheduler's summary. + bytes: usize, + }, /// Content-address integrity check failed — do not retry. IntegrityFailed, /// Source failed (network error or non-success response) — retryable. @@ -8979,7 +9429,159 @@ enum FetchResult { /// worker loop to update queue state. struct FetchOutcome { key: XorName, + /// The source this attempt was sent to. + source: PeerId, result: FetchResult, + /// Whether `source` answered, for the per-source cap's exemption of + /// sources that stopped answering. + contact: Contact, +} + +/// How often the fetch worker reports its dispatching. +const FETCH_DISPATCH_SUMMARY_INTERVAL: Duration = Duration::from_mins(5); + +/// How long an attempt that started at `started` took, up to when its task +/// finished rather than when the worker got to its outcome. +fn attempt_latency(started: Option, completed_at: Instant) -> Option { + started.map(|started| completed_at.saturating_duration_since(started)) +} + +/// What the fetch worker observed about its own dispatching since the last +/// summary. +#[derive(Default)] +struct FetchDispatchWindow { + /// When this window began. + started: Option, + /// Longest wait for the queues write lock before a fill pass. + fill_lock_wait_max: Duration, + /// Longest hold of the queues write lock by a fill pass, spawning included. + fill_lock_hold_max: Duration, + /// Fill passes in this window. + fills: u64, + /// Queued keys parked by fill passes in this window. + parks: usize, + /// Failed fetches parked because every untried source was full. + retry_parks: u64, + /// Attempts by outcome. + stored: u64, + already_held: u64, + no_longer_responsible: u64, + source_failed: u64, + integrity_failed: u64, + local_write_failed: u64, + panicked: u64, + /// Attempts cancelled by shutdown before they resolved. + cancelled: u64, + /// Bytes of chunks stored by attempts in this window. + stored_bytes: u64, + /// Attempt latency: longest, and total over `timed_attempts`. + attempt_max: Duration, + attempt_total: Duration, + timed_attempts: u64, + /// Set for the summary logged at shutdown, whose in-flight count includes + /// attempts drained without being applied. + shutdown: bool, +} + +impl FetchDispatchWindow { + fn new() -> Self { + Self { + started: Some(Instant::now()), + ..Self::default() + } + } + + /// Time since this window began. + fn age(&self) -> Duration { + self.started + .map_or(Duration::ZERO, |started| started.elapsed()) + } + + fn record_fill(&mut self, wait: Duration, hold: Duration) { + self.fill_lock_wait_max = self.fill_lock_wait_max.max(wait); + self.fill_lock_hold_max = self.fill_lock_hold_max.max(hold); + self.fills += 1; + } + + /// Count one resolved attempt: `None` when its task panicked. A cancelled + /// attempt is counted as cancelled whatever result it carries, and its + /// cut-short latency is left out. + fn record_outcome(&mut self, outcome: Option<&FetchOutcome>, attempt: Option) { + if outcome.is_some_and(|outcome| outcome.contact == Contact::Unknown) { + self.cancelled += 1; + return; + } + match outcome.map(|outcome| &outcome.result) { + Some(FetchResult::Stored { bytes }) => { + self.stored += 1; + self.stored_bytes = self + .stored_bytes + .saturating_add(u64::try_from(*bytes).unwrap_or(u64::MAX)); + } + Some(FetchResult::AlreadyHeld) => self.already_held += 1, + Some(FetchResult::NoLongerResponsible) => self.no_longer_responsible += 1, + Some(FetchResult::SourceFailed) => self.source_failed += 1, + Some(FetchResult::IntegrityFailed) => self.integrity_failed += 1, + Some(FetchResult::LocalWriteFailed) => self.local_write_failed += 1, + None => self.panicked += 1, + } + if let Some(attempt) = attempt { + self.attempt_max = self.attempt_max.max(attempt); + self.attempt_total = self.attempt_total.saturating_add(attempt); + self.timed_attempts += 1; + } + } + + /// Log this window with the queues' state, then start a new window. + /// `stats_lock` is how long reading that state took, lock wait included. + fn log_and_reset(&mut self, stats: &FetchDispatchStats, stats_lock: Duration) { + let attempt_mean_ms = u32::try_from(self.timed_attempts) + .ok() + .filter(|count| *count > 0) + .map_or(0, |count| (self.attempt_total / count).as_millis()); + info!( + per_source_cap = FETCH_MAX_IN_FLIGHT_PER_SOURCE, + window_secs = self.age().as_secs(), + parked = stats.parked, + parked_associations = stats.parked_associations, + oldest_parked_secs = stats.oldest_parked.map_or(0, |age| age.as_secs()), + queued = stats.queued, + in_flight = stats.in_flight, + saturated_sources = stats.saturated_sources, + fills = self.fills, + parks = self.parks, + retry_parks = self.retry_parks, + stored = self.stored, + stored_bytes = self.stored_bytes, + already_held = self.already_held, + no_longer_responsible = self.no_longer_responsible, + source_failed = self.source_failed, + integrity_failed = self.integrity_failed, + local_write_failed = self.local_write_failed, + panicked = self.panicked, + cancelled = self.cancelled, + shutdown = self.shutdown, + attempt_max_ms = self.attempt_max.as_millis(), + attempt_mean_ms, + fill_lock_wait_max_ms = self.fill_lock_wait_max.as_millis(), + fill_lock_hold_max_ms = self.fill_lock_hold_max.as_millis(), + parked_maintenance_max_ms = stats.parked_maintenance_max.as_millis(), + tracked_sources = stats.tracked_sources, + silent_sources = stats.silent_sources, + health_overflow_active = stats.overflow_active, + silent_peak = stats.health.silent_peak, + silent_starts = stats.health.silent_starts, + unreachable = stats.health.unreachable, + silent_marks = stats.health.marks, + silent_clears = stats.health.clears, + silent_expired = stats.health.expired_marks, + silent_ignored_stale = stats.health.ignored_stale, + health_overflows = stats.health.overflows, + stats_lock_ms = stats_lock.as_millis(), + "replication fetch dispatch summary" + ); + *self = Self::new(); + } } /// What the fetch worker must do next for a key whose fetch attempt resolved. @@ -8993,6 +9595,11 @@ enum FetchFollowUp { RequeuedForVerification, /// Retry immediately from this next untried verified source. RetryFrom(PeerId), + /// Every untried source already has this node's per-source share in + /// flight, so the key went back to the fetch queue, parked, keeping its + /// tried sources and retry reservation. Not terminal: the key is still in + /// the pipeline, and the next fill pass starts it once a share frees. + Parked, } /// Apply a resolved [`FetchResult`] to the queues and report the follow-up. @@ -9013,19 +9620,26 @@ fn apply_fetch_result( key: &XorName, result: &FetchResult, verification_retry_after: Duration, + distance: XorName, ) -> FetchFollowUp { match result { - FetchResult::Stored | FetchResult::NoLongerResponsible | FetchResult::AlreadyHeld => { + FetchResult::Stored { .. } + | FetchResult::NoLongerResponsible + | FetchResult::AlreadyHeld => { q.complete_fetch(key); FetchFollowUp::Terminal } FetchResult::IntegrityFailed | FetchResult::SourceFailed => { - if let Some(next_peer) = q.retry_fetch(key) { - FetchFollowUp::RetryFrom(next_peer) - } else if q.requeue_fetch_for_verification(key, verification_retry_after) { - FetchFollowUp::RequeuedForVerification - } else { - FetchFollowUp::Terminal + match q.retry_or_park_fetch(key, distance, FETCH_MAX_IN_FLIGHT_PER_SOURCE) { + FetchRetry::Next(next_peer) => FetchFollowUp::RetryFrom(next_peer), + FetchRetry::Parked => FetchFollowUp::Parked, + FetchRetry::Exhausted => { + if q.requeue_fetch_for_verification(key, verification_retry_after) { + FetchFollowUp::RequeuedForVerification + } else { + FetchFollowUp::Terminal + } + } } } // No `retry_fetch`: the local write is what failed, so every other @@ -9072,7 +9686,6 @@ async fn is_storage_admitted( .await } -#[allow(clippy::too_many_lines)] /// Execute a single fetch request against `source` for `key`. /// /// Handles encoding, network I/O, integrity checking, storage, and trust @@ -9090,19 +9703,69 @@ async fn execute_single_fetch( p2p_node: Arc, storage: Arc, config: Arc, + busy_sources: Arc, key: XorName, source: PeerId, + answers: mpsc::UnboundedSender<(PeerId, Contact)>, ) -> FetchOutcome { + let mut contact = Contact::NotAttempted; + let result = fetch_attempt( + p2p_node, + storage, + config, + &busy_sources, + key, + source, + &mut contact, + &answers, + ) + .await; + FetchOutcome { + key, + source, + result, + contact, + } +} + +/// Record every answer a fetch has reported since the last call. +/// +/// A fetch reports its source's answer as soon as it arrives, before checking +/// and storing the chunk, so the per-source cap never treats an answering +/// source as silent while that work is still under way. The same answer comes +/// again with the fetch's outcome, which records it a second time to no +/// effect. Each is reported while its fetch is in flight, and the worker calls +/// this before applying any outcome, so it is recorded before its fetch ends. +fn record_early_answers( + queues: &mut ReplicationQueues, + early_answers: &mut mpsc::UnboundedReceiver<(PeerId, Contact)>, +) { + while let Ok((source, contact)) = early_answers.try_recv() { + queues.record_fetch_contact(source, contact); + } +} + +/// The body of [`execute_single_fetch`]: the attempt's result, with `contact` +/// set to whether the source answered, stamped when the request returned. An +/// answer is also sent on `answers` at once. +#[allow(clippy::too_many_lines, clippy::too_many_arguments)] +async fn fetch_attempt( + p2p_node: Arc, + storage: Arc, + config: Arc, + busy_sources: &SourceBusyTracker, + key: XorName, + source: PeerId, + contact: &mut Contact, + answers: &mpsc::UnboundedSender<(PeerId, Contact)>, +) -> FetchResult { let self_id = *p2p_node.peer_id(); if !is_storage_admitted(&self_id, &key, &p2p_node, &config).await { debug!( "Skipping fetch for {}: no longer in the storage-admission group", hex::encode(key) ); - return FetchOutcome { - key, - result: FetchResult::NoLongerResponsible, - }; + return FetchResult::NoLongerResponsible; } // Possession, then capacity — both before the dial, and in that order. @@ -9116,10 +9779,7 @@ async fn execute_single_fetch( "Skipping fetch for {}: already held locally", hex::encode(key) ); - return FetchOutcome { - key, - result: FetchResult::AlreadyHeld, - }; + return FetchResult::AlreadyHeld; } // Capacity is checked before the dial rather than after the bytes arrive. @@ -9141,13 +9801,13 @@ async fn execute_single_fetch( "Skipping fetch for {}: local storage cannot accept a write: {e}", hex::encode(key) ); - return FetchOutcome { - key, - result: FetchResult::LocalWriteFailed, - }; + return FetchResult::LocalWriteFailed; } - let request = protocol::FetchRequest { key }; + let request = protocol::FetchRequest { + key, + busy_reply: true, + }; let msg = ReplicationMessage { request_id: rand::thread_rng().gen::(), body: ReplicationMessageBody::FetchRequest(request), @@ -9157,13 +9817,12 @@ async fn execute_single_fetch( Ok(data) => data, Err(e) => { warn!("Failed to encode fetch request: {e}"); - return FetchOutcome { - key, - result: FetchResult::SourceFailed, - }; + return FetchResult::SourceFailed; } }; + let sent_at = Instant::now(); + let sent = next_fetch_contact(); let result = p2p_node .send_request( &source, @@ -9172,6 +9831,18 @@ async fn execute_single_fetch( config.fetch_request_timeout, ) .await; + *contact = if result.is_ok() { + Contact::Answered { + order: next_fetch_contact(), + } + } else { + Contact::Unreachable { sent } + }; + if matches!(contact, Contact::Answered { .. }) { + // Fails only once the fetch worker has stopped, when no scheduling + // decision is left to inform. + let _ = answers.send((source, *contact)); + } match result { Ok(response) => { @@ -9182,10 +9853,7 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - return FetchOutcome { - key, - result: FetchResult::SourceFailed, - }; + return FetchResult::SourceFailed; }; match resp_msg.body { @@ -9209,10 +9877,7 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - return FetchOutcome { - key, - result: FetchResult::IntegrityFailed, - }; + return FetchResult::IntegrityFailed; } // Enforce chunk size invariant on fetched data. @@ -9231,10 +9896,7 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - return FetchOutcome { - key, - result: FetchResult::IntegrityFailed, - }; + return FetchResult::IntegrityFailed; } // Content-address integrity check. @@ -9251,12 +9913,13 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - return FetchOutcome { - key, - result: FetchResult::IntegrityFailed, - }; + return FetchResult::IntegrityFailed; } + // It served a valid chunk, so any earlier busy answers no + // longer count against it when choosing where to start. + busy_sources.record_served(&source); + // Responsibility can lapse during the network round-trip. // The bandwidth is already spent; declining the write is // what still avoids the disk write and the later @@ -9268,10 +9931,7 @@ async fn execute_single_fetch( "Fetched {} but responsibility lapsed in transit; not storing", hex::encode(key) ); - return FetchOutcome { - key, - result: FetchResult::NoLongerResponsible, - }; + return FetchResult::NoLongerResponsible; } if let Err(e) = storage.put(&resp_key, &data).await { @@ -9285,16 +9945,10 @@ async fn execute_single_fetch( "Failed to store fetched record {}: {e}", hex::encode(resp_key) ); - return FetchOutcome { - key, - result: FetchResult::LocalWriteFailed, - }; + return FetchResult::LocalWriteFailed; } - FetchOutcome { - key, - result: FetchResult::Stored, - } + FetchResult::Stored { bytes: data.len() } } ReplicationMessageBody::FetchResponse( ref response @ (protocol::FetchResponse::NotFound { .. } @@ -9320,23 +9974,70 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - FetchOutcome { - key, - result: FetchResult::SourceFailed, + FetchResult::SourceFailed + } + ReplicationMessageBody::FetchResponse(protocol::FetchResponse::Busy { + key: resp_key, + }) if resp_key == key => { + // The source refused at admission and said so, where it + // used to stay silent. A prompt answer to passing overload + // carries no trust event. Waiting here, still holding this + // fetch slot, before the key moves on is what keeps a busy + // answer from becoming a fast retry loop: this slot asks at + // most once per wait, where silence made it once per fetch + // timeout. The attempt then ends exactly as that timeout + // did, so nothing downstream changes. The worker races this + // whole task against shutdown, so the wait does not delay it. + audit_metrics::record_fetch_busy_received(); + let jitter = + rand::thread_rng().gen_range(Duration::ZERO..=FETCH_BUSY_WAIT_SPREAD); + // The holder is put last for at least the full jittered + // wait, even when this attempt's own deadline cuts the + // sleep short. + let full_wait = busy_wait(Duration::MAX, jitter); + let answer = busy_sources.record_busy(&source, Instant::now(), full_wait); + // Past its neutral allowance, a source that keeps saying + // busy without serving is handled as the silence it + // replaces: the attempt holds its slot until its fetch + // timeout runs out, never longer, however late the answer + // came, and only then is the source charged the unit-weight + // failure saorsa-core records for a timeout. A fetch + // cancelled by shutdown before then is not charged. + let deadline = sent_at + .checked_add(config.fetch_request_timeout) + .unwrap_or(sent_at); + let now = Instant::now(); + let wake = busy_wake(answer, full_wait, now, deadline); + let charged = answer.charged; + debug!( + "Fetch: source {source} busy for {}; moving on in {} ms, \ + putting it last for {} ms, charged: {charged}", + hex::encode(key), + wake.saturating_duration_since(now).as_millis(), + answer.cooling_for.as_millis() + ); + tokio::time::sleep_until(wake.into()).await; + if charged { + audit_metrics::record_fetch_busy_charged(); + p2p_node + .report_trust_event( + &source, + TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), + ) + .await; } + FetchResult::SourceFailed } _ => { - // Unexpected message type — treat as malformed. + // Unexpected message type, or a busy answer for another + // key — treat as malformed. p2p_node .report_trust_event( &source, TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - FetchOutcome { - key, - result: FetchResult::SourceFailed, - } + FetchResult::SourceFailed } } } @@ -9344,10 +10045,7 @@ async fn execute_single_fetch( debug!("Fetch request to {source} failed: {e}"); // No ApplicationFailure here — P2PNode::send_request() already // reports ConnectionTimeout / ConnectionFailed to the TrustEngine. - FetchOutcome { - key, - result: FetchResult::SourceFailed, - } + FetchResult::SourceFailed } } } @@ -10537,6 +11235,7 @@ mod tests { use crate::replication::audit::AuditTickResult; use crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL; use crate::replication::recent_provers::RecentProvers; + use crate::replication::scheduling::MAX_FETCH_QUEUE; use crate::replication::types::{AuditFailureReason, FailureEvidence}; use lru::LruCache; use saorsa_core::identity::PeerId; @@ -13287,6 +13986,7 @@ mod tests { request_id: 1, body: ReplicationMessageBody::FetchRequest(protocol::FetchRequest { key: test_key(1), + busy_reply: true, }), }, rr_message_id: None, @@ -13779,8 +14479,13 @@ mod tests { drive_key_in_flight(&mut q, key, vec![test_peer(0x01), test_peer(0x02)]); assert_eq!(q.retry_reserved_slot_count(), 1); - let follow_up = - apply_fetch_result(&mut q, &key, &FetchResult::NoLongerResponsible, RETRY_AFTER); + let follow_up = apply_fetch_result( + &mut q, + &key, + &FetchResult::NoLongerResponsible, + RETRY_AFTER, + key, + ); assert_eq!( follow_up, @@ -13810,12 +14515,15 @@ mod tests { // Both variants must walk the identical terminal gate so the // battle-tested Stored accounting (retry-slot release + bootstrap // pending-set shrink in the caller) covers lapsed responsibility too. - for result in [FetchResult::Stored, FetchResult::NoLongerResponsible] { + for result in [ + FetchResult::Stored { bytes: 0 }, + FetchResult::NoLongerResponsible, + ] { let mut q = ReplicationQueues::new(); let key = test_key(0xCD); drive_key_in_flight(&mut q, key, vec![test_peer(0x03)]); - let follow_up = apply_fetch_result(&mut q, &key, &result, RETRY_AFTER); + let follow_up = apply_fetch_result(&mut q, &key, &result, RETRY_AFTER, key); assert_eq!(follow_up, FetchFollowUp::Terminal); assert!(!q.contains_key(&key)); @@ -13833,7 +14541,8 @@ mod tests { let second = test_peer(0x05); drive_key_in_flight(&mut q, key, vec![first, second]); - let follow_up = apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER); + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); assert_eq!( follow_up, FetchFollowUp::RetryFrom(second), @@ -13841,7 +14550,8 @@ mod tests { ); assert_eq!(q.in_flight_count(), 1, "retry keeps the key in flight"); - let follow_up = apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER); + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); assert_eq!( follow_up, FetchFollowUp::RequeuedForVerification, @@ -13855,6 +14565,269 @@ mod tests { ); } + /// A retry passes over an untried source that recently answered busy when + /// another untried source is free, and still takes it when it is the only + /// one left: busy holders go last, never out. + #[test] + fn source_failure_retries_on_a_holder_that_is_not_cooling_first() { + const RETRY_AFTER: Duration = Duration::from_mins(1); + + let mut q = ReplicationQueues::new(); + let key = test_key(0xE1); + let (first, cooling, free) = (test_peer(0x11), test_peer(0x12), test_peer(0x13)); + drive_key_in_flight(&mut q, key, vec![first, cooling, free]); + q.busy_sources() + .record_busy(&cooling, Instant::now(), Duration::from_mins(1)); + + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); + assert_eq!(follow_up, FetchFollowUp::RetryFrom(free)); + + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); + assert_eq!( + follow_up, + FetchFollowUp::RetryFrom(cooling), + "a cooling holder is still tried once it is the last untried one" + ); + + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); + assert_eq!(follow_up, FetchFollowUp::RequeuedForVerification); + } + + /// A busy reply serves nothing, so it never counts as served bytes; the + /// answers that did before still do. + #[test] + fn a_busy_reply_is_not_counted_as_served() { + let key = test_key(0x77); + assert!(!counts_as_served(&ReplicationMessageBody::FetchResponse( + protocol::FetchResponse::Busy { key } + ))); + for response in [ + protocol::FetchResponse::Success { + key, + data: vec![1, 2, 3], + }, + protocol::FetchResponse::NotFound { key }, + protocol::FetchResponse::Error { + key, + reason: "io".to_string(), + }, + ] { + assert!(counts_as_served(&ReplicationMessageBody::FetchResponse( + response + ))); + } + assert!(counts_as_served( + &ReplicationMessageBody::NeighborSyncResponse(protocol::NeighborSyncResponse { + replica_hints: vec![], + paid_hints: vec![], + bootstrapping: false, + rejected_keys: vec![], + commitment: None, + }) + )); + assert!(!counts_as_served( + &ReplicationMessageBody::VerificationResponse(protocol::VerificationResponse { + results: vec![], + }) + )); + } + + /// Put two unrelated fetches in flight to `source`, filling its share. + fn saturate_source(q: &mut ReplicationQueues, source: PeerId, first_key_byte: u8) { + for offset in 0..2u8 { + let key = test_key(first_key_byte + offset); + q.start_fetch(key, source, vec![source]); + } + } + + #[test] + fn source_failure_with_every_untried_source_full_parks_the_key() { + const RETRY_AFTER: Duration = Duration::from_mins(1); + + let mut q = ReplicationQueues::new(); + let key = test_key(0x2A); + let (first, second) = (test_peer(0x01), test_peer(0x02)); + assert_eq!(drive_key_in_flight(&mut q, key, vec![first, second]), first); + saturate_source(&mut q, second, 0xE0); + let reserved = q.retry_reserved_slot_count(); + + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); + + assert_eq!( + follow_up, + FetchFollowUp::Parked, + "the only untried source is at its share, so the key must wait rather \ + than be sent a request it will drop" + ); + assert!( + q.contains_key(&key), + "a parked key is still in the pipeline" + ); + assert!(q.fetch_queued_or_in_flight(&key)); + assert_eq!( + q.in_flight_count(), + 2, + "only the two saturating fetches remain" + ); + assert_eq!( + q.pending_count(), + 0, + "parking is not a verification requeue" + ); + assert_eq!(q.retry_reserved_slot_count(), reserved); + } + + #[test] + fn an_answer_reported_before_its_fetch_ends_keeps_a_timed_out_source_capped() { + const RETRY_AFTER: Duration = Duration::from_mins(1); + const SLOTS: usize = 8; + + let source = test_peer(0x0A); + let (dropped, answered) = (test_key(0x50), test_key(0x51)); + let waiting: Vec = (0x60..0x64).map(test_key).collect(); + // Both of `source`'s shares are in flight and four keys only it holds + // wait behind them. + let setup = || { + let mut q = ReplicationQueues::new(); + q.start_fetch(dropped, source, vec![source]); + q.start_fetch(answered, source, vec![source]); + for key in &waiting { + assert!(q.enqueue_fetch(*key, *key, vec![source])); + } + q + }; + let in_flight_to_source = |q: &ReplicationQueues| { + q.fetch_pipeline_snapshot() + .in_flight + .iter() + .filter(|(_, peer)| *peer == source) + .count() + }; + // The first request is dropped and times out. The second, sent after + // it, has been answered and is still checking and storing its chunk. + let dropped_sent = next_fetch_contact(); + let _answered_sent = next_fetch_contact(); + let answer = Contact::Answered { + order: next_fetch_contact(), + }; + let timeout = Contact::Unreachable { sent: dropped_sent }; + + // Known only from the second fetch's outcome, the answer comes too + // late: the timeout marks `source` silent and the next pass sends it + // every waiting key, past its share. + let mut late = setup(); + late.record_fetch_contact(source, timeout); + apply_fetch_result( + &mut late, + &dropped, + &FetchResult::SourceFailed, + RETRY_AFTER, + dropped, + ); + late.dispatch_fetches(SLOTS, FETCH_MAX_IN_FLIGHT_PER_SOURCE); + assert_eq!(in_flight_to_source(&late), 1 + waiting.len()); + + // Reported when it arrives, the answer is recorded before the timeout, + // which it outdates, so `source` stays within its share. + let (answers, mut early_answers) = mpsc::unbounded_channel(); + let mut q = setup(); + assert!(answers.send((source, answer)).is_ok()); + record_early_answers(&mut q, &mut early_answers); + q.record_fetch_contact(source, timeout); + apply_fetch_result( + &mut q, + &dropped, + &FetchResult::SourceFailed, + RETRY_AFTER, + dropped, + ); + record_early_answers(&mut q, &mut early_answers); + let dispatch = q.dispatch_fetches(SLOTS, FETCH_MAX_IN_FLIGHT_PER_SOURCE); + assert_eq!(dispatch.started.len(), 1, "one share is free"); + assert_eq!(in_flight_to_source(&q), FETCH_MAX_IN_FLIGHT_PER_SOURCE); + + // The same answer again, with the second fetch's outcome, changes + // nothing. + q.record_fetch_contact(source, answer); + assert_eq!( + q.dispatch_fetches(SLOTS, FETCH_MAX_IN_FLIGHT_PER_SOURCE) + .started + .len(), + 0 + ); + assert_eq!(in_flight_to_source(&q), FETCH_MAX_IN_FLIGHT_PER_SOURCE); + } + + #[test] + fn the_dispatch_window_times_resolved_attempts_and_counts_cancelled_ones() { + let (key, source) = (test_key(0x31), test_peer(0x01)); + let outcome = |result: FetchResult, contact: Contact| FetchOutcome { + key, + source, + result, + contact, + }; + let started = Instant::now(); + let at = |millis: u64| started + Duration::from_millis(millis); + let mut window = FetchDispatchWindow::new(); + + // Two attempts resolve at different times; the worker reaches both + // later, which must not count. + let stored = outcome( + FetchResult::Stored { bytes: 10 }, + Contact::Answered { order: 2 }, + ); + window.record_outcome(Some(&stored), attempt_latency(Some(started), at(100))); + let failed = outcome(FetchResult::SourceFailed, Contact::Unreachable { sent: 1 }); + window.record_outcome(Some(&failed), attempt_latency(Some(started), at(300))); + // A cancelled attempt is counted as cancelled, not by its placeholder + // result, and its cut-short time is left out. + let cancelled = outcome(FetchResult::SourceFailed, Contact::Unknown); + window.record_outcome(Some(&cancelled), attempt_latency(Some(started), at(5_000))); + + assert_eq!((window.stored, window.source_failed), (1, 1)); + assert_eq!((window.cancelled, window.timed_attempts), (1, 2)); + assert_eq!(window.attempt_max, Duration::from_millis(300)); + assert_eq!(window.attempt_total, Duration::from_millis(400)); + assert_eq!(window.stored_bytes, 10); + assert_eq!(attempt_latency(None, at(100)), None); + } + + #[test] + fn source_failure_with_a_full_fetch_queue_requeues_for_verification() { + const RETRY_AFTER: Duration = Duration::from_mins(1); + + let mut q = ReplicationQueues::new(); + let key = test_key(0x2B); + let (first, second) = (test_peer(0x01), test_peer(0x02)); + drive_key_in_flight(&mut q, key, vec![first, second]); + saturate_source(&mut q, second, 0xE0); + let filler_source = test_peer(0x09); + let mut filler = 0u32; + while q.fetch_queue_count() < MAX_FETCH_QUEUE { + let mut filler_key = [0xC0u8; 32]; + filler_key[..4].copy_from_slice(&filler.to_be_bytes()); + assert!(q.enqueue_fetch(filler_key, filler_key, vec![filler_source])); + filler += 1; + } + + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); + + assert_eq!( + follow_up, + FetchFollowUp::RequeuedForVerification, + "with no room to park, the key returns to verification on its own \ + reserved slot instead of overfilling the fetch queue" + ); + assert_eq!(q.fetch_queue_count(), MAX_FETCH_QUEUE); + assert_eq!(q.pending_count(), 1); + } + #[test] fn source_failure_without_retry_metadata_is_terminal() { const RETRY_AFTER: Duration = Duration::from_mins(1); @@ -13868,7 +14841,8 @@ mod tests { let candidate = q.dequeue_fetch().expect("enqueued key must dequeue"); q.start_dequeued_fetch(candidate, source); - let follow_up = apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER); + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); assert_eq!(follow_up, FetchFollowUp::Terminal); assert!(!q.contains_key(&key)); @@ -13883,7 +14857,8 @@ mod tests { let source = test_peer(0x0B); drive_key_in_flight(&mut q, key, vec![source, test_peer(0x0C)]); - let follow_up = apply_fetch_result(&mut q, &key, &FetchResult::AlreadyHeld, RETRY_AFTER); + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::AlreadyHeld, RETRY_AFTER, key); assert_eq!( follow_up, @@ -13912,8 +14887,13 @@ mod tests { let third = test_peer(0x09); drive_key_in_flight(&mut q, key, vec![first, second, third]); - let follow_up = - apply_fetch_result(&mut q, &key, &FetchResult::LocalWriteFailed, RETRY_AFTER); + let follow_up = apply_fetch_result( + &mut q, + &key, + &FetchResult::LocalWriteFailed, + RETRY_AFTER, + key, + ); assert_eq!( follow_up, @@ -13946,8 +14926,13 @@ mod tests { let candidate = q.dequeue_fetch().expect("enqueued key must dequeue"); q.start_dequeued_fetch(candidate, source); - let follow_up = - apply_fetch_result(&mut q, &key, &FetchResult::LocalWriteFailed, RETRY_AFTER); + let follow_up = apply_fetch_result( + &mut q, + &key, + &FetchResult::LocalWriteFailed, + RETRY_AFTER, + key, + ); assert_eq!(follow_up, FetchFollowUp::Terminal); assert!(!q.contains_key(&key)); diff --git a/src/replication/protocol.rs b/src/replication/protocol.rs index ebb4644a..d98eec80 100644 --- a/src/replication/protocol.rs +++ b/src/replication/protocol.rs @@ -22,6 +22,35 @@ use super::config::MAX_SUBTREE_AUDIT_MESSAGE_SIZE; /// Used in [`AuditResponse::Digests`] for keys the peer does not hold. pub const ABSENT_KEY_DIGEST: [u8; 32] = [0u8; 32]; +/// Opens the fetch capability trailer that may follow an encoded +/// [`FetchRequest`] message. +/// +/// A requester states what it understands here, after the end of the postcard +/// message, because every older node decodes with postcard's `from_bytes`, +/// which ignores whatever follows the message. A field inside the message +/// would instead shift or break their decoding. The magic keeps a stray byte +/// after a message from ever reading as a capability. +pub const FETCH_CAPABILITY_MAGIC: [u8; 4] = *b"antf"; + +/// Bit in the fetch capability flags byte: the requester understands +/// [`FetchResponse::Busy`]. +pub const FETCH_CAPABILITY_BUSY_REPLY: u8 = 0b0000_0001; + +/// Length of the fetch capability trailer: the magic and one flags byte. +const FETCH_CAPABILITY_TRAILER_LEN: usize = FETCH_CAPABILITY_MAGIC.len() + 1; + +/// The fetch capability flags in `trailer`, the bytes after a decoded message. +/// +/// Zero unless `trailer` opens with [`FETCH_CAPABILITY_MAGIC`] and has a flags +/// byte after it. Bytes after the flags byte are reserved and ignored, so a +/// later version can extend the trailer without this one rejecting it. +fn fetch_capability_flags(trailer: &[u8]) -> u8 { + match trailer.strip_prefix(FETCH_CAPABILITY_MAGIC.as_slice()) { + Some([flags, ..]) => *flags, + _ => 0, + } +} + // --------------------------------------------------------------------------- // Top-level envelope // --------------------------------------------------------------------------- @@ -68,8 +97,24 @@ impl ReplicationMessage { // The buffer is sized exactly, and an oversized body is refused // before anything is allocated. Chunk-carrying bodies run to several // MiB and are held while they are sent. - let max_size = ceiling_for(family_of_variant(self.body.variant_index())); - let bytes = encode_exact(self, max_size).map_err(|e| match e { + // A fetch request that asks for busy replies carries the capability + // trailer after the message, so its message must leave room for it + // under the same ceiling the decoder applies to the whole input. + let asks_busy_reply = matches!( + &self.body, + ReplicationMessageBody::FetchRequest(FetchRequest { + busy_reply: true, + .. + }) + ); + let trailer_len = if asks_busy_reply { + FETCH_CAPABILITY_TRAILER_LEN + } else { + 0 + }; + let max_size = + ceiling_for(family_of_variant(self.body.variant_index())).saturating_sub(trailer_len); + let mut bytes = encode_exact(self, max_size).map_err(|e| match e { ExactEncodeError::Serialize(e) => { ReplicationProtocolError::SerializationFailed(e.to_string()) } @@ -81,6 +126,14 @@ impl ReplicationMessage { } })?; + // The capability trailer goes immediately after the message, where + // older decoders never look. + if asks_busy_reply { + bytes.reserve_exact(trailer_len); + bytes.extend_from_slice(&FETCH_CAPABILITY_MAGIC); + bytes.push(FETCH_CAPABILITY_BUSY_REPLY); + } + // V2-623: cumulative per-variant tx accounting. Every replication send // funnels through here, so this is the single tx choke point. record_tx(&self.body, bytes.len()); @@ -120,8 +173,14 @@ impl ReplicationMessage { max_size, }); } - let message: Self = postcard::from_bytes(data) + // `take_from_bytes` rather than `from_bytes` only to see what follows + // the message. Both ignore it otherwise, so nothing that decoded before + // decodes differently now. + let (mut message, trailer): (Self, &[u8]) = postcard::take_from_bytes(data) .map_err(|e| ReplicationProtocolError::DeserializationFailed(e.to_string()))?; + if let ReplicationMessageBody::FetchRequest(request) = &mut message.body { + request.busy_reply = fetch_capability_flags(trailer) & FETCH_CAPABILITY_BUSY_REPLY != 0; + } // V2-623: cumulative per-variant rx accounting. Every replication // receive funnels through here, so this is the single rx choke point. @@ -977,6 +1036,18 @@ pub struct VerificationResponse { pub struct FetchRequest { /// The key of the record to fetch. pub key: XorName, + /// Whether the requester understands [`FetchResponse::Busy`]. + /// + /// Not part of the serialized struct, whose layout is unchanged since the + /// first release: it travels in the fetch capability trailer that + /// [`ReplicationMessage::encode`] appends after the whole message, and + /// [`ReplicationMessage::decode`] reads back. Older nodes decode with + /// postcard's `from_bytes`, which stops at the end of the message and never + /// sees the trailer, so they keep refusing silently. A responder answers + /// `Busy` only when this is set, so an older requester, which never sends + /// the trailer, is never sent a variant it cannot decode. + #[serde(skip)] + pub busy_reply: bool, } /// Response to a fetch request. @@ -1004,6 +1075,16 @@ pub enum FetchResponse { /// Human-readable error description. reason: String, }, + /// The responder could not admit the request right now and did not + /// serve it. Nothing is implied about whether it holds the record. + /// + /// APPENDED so the discriminants of the variants above are unchanged. Sent + /// only in answer to a [`FetchRequest`] with `busy_reply` set; an older + /// requester never sets it, so it is never sent a value it cannot decode. + Busy { + /// The requested key. + key: XorName, + }, } // --------------------------------------------------------------------------- @@ -1740,7 +1821,10 @@ mod tests { paid_list_check_indices: vec![], }), ReplicationMessageBody::VerificationResponse(VerificationResponse { results: vec![] }), - ReplicationMessageBody::FetchRequest(FetchRequest { key: z }), + ReplicationMessageBody::FetchRequest(FetchRequest { + key: z, + busy_reply: true, + }), ReplicationMessageBody::FetchResponse(FetchResponse::NotFound { key: z }), ReplicationMessageBody::AuditChallenge(AuditChallenge { challenge_id: 1, @@ -2392,7 +2476,10 @@ mod tests { fn fetch_request_roundtrip() { let msg = ReplicationMessage { request_id: 9, - body: ReplicationMessageBody::FetchRequest(FetchRequest { key: [0x40; 32] }), + body: ReplicationMessageBody::FetchRequest(FetchRequest { + key: [0x40; 32], + busy_reply: true, + }), }; let encoded = msg.encode().expect("encode should succeed"); let decoded = ReplicationMessage::decode(&encoded).expect("decode should succeed"); @@ -2400,6 +2487,7 @@ mod tests { assert_eq!(decoded.request_id, 9); if let ReplicationMessageBody::FetchRequest(req) = decoded.body { assert_eq!(req.key, [0x40; 32]); + assert!(req.busy_reply); } else { panic!("expected FetchRequest"); } @@ -2471,6 +2559,283 @@ mod tests { } } + #[test] + fn fetch_response_busy_roundtrip() { + let msg = ReplicationMessage { + request_id: 13, + body: ReplicationMessageBody::FetchResponse(FetchResponse::Busy { key: [0x71; 32] }), + }; + let encoded = msg.encode().expect("encode should succeed"); + let decoded = ReplicationMessage::decode(&encoded).expect("decode should succeed"); + + assert_eq!(decoded.request_id, 13); + if let ReplicationMessageBody::FetchResponse(FetchResponse::Busy { key }) = decoded.body { + assert_eq!(key, [0x71; 32]); + } else { + panic!("expected FetchResponse::Busy"); + } + } + + // -- the fetch busy reply across versions ------------------------------- + // + // `Legacy*` mirror the fetch types and the envelope as every release + // before the busy reply encoded them. The seven unit variants stand in for + // the body variants before `FetchRequest`; only their count matters, since + // postcard writes an enum as its variant index followed by the payload. + + #[derive(Serialize, Deserialize)] + struct LegacyFetchRequest { + key: XorName, + } + + #[derive(Serialize, Deserialize)] + enum LegacyFetchResponse { + Success { + key: XorName, + #[serde(with = "serde_bytes")] + data: Vec, + }, + NotFound { + key: XorName, + }, + Error { + key: XorName, + reason: String, + }, + } + + #[derive(Serialize, Deserialize)] + enum LegacyBody { + V0, + V1, + V2, + V3, + V4, + V5, + V6, + FetchRequest(LegacyFetchRequest), + FetchResponse(LegacyFetchResponse), + } + + #[derive(Serialize, Deserialize)] + struct LegacyMessage { + request_id: u64, + body: LegacyBody, + } + + fn legacy_fetch_request_bytes(key: XorName) -> Vec { + postcard::to_stdvec(&LegacyMessage { + request_id: 21, + body: LegacyBody::FetchRequest(LegacyFetchRequest { key }), + }) + .expect("encode") + } + + /// The mirror must match the real layout, or every test below proves + /// nothing. A current request is an older one's bytes exactly, plus the + /// capability trailer when it asks for a busy reply. + #[test] + fn legacy_mirror_matches_the_current_layout() { + let legacy = legacy_fetch_request_bytes([0x22; 32]); + let plain = ReplicationMessage { + request_id: 21, + body: ReplicationMessageBody::FetchRequest(FetchRequest { + key: [0x22; 32], + busy_reply: false, + }), + } + .encode() + .expect("encode"); + assert_eq!(plain, legacy); + + let asking = ReplicationMessage { + request_id: 21, + body: ReplicationMessageBody::FetchRequest(FetchRequest { + key: [0x22; 32], + busy_reply: true, + }), + } + .encode() + .expect("encode"); + let mut expected = legacy; + expected.extend_from_slice(&FETCH_CAPABILITY_MAGIC); + expected.push(FETCH_CAPABILITY_BUSY_REPLY); + assert_eq!(asking, expected); + } + + /// An older responder decodes a current request: the trailer comes after + /// the message, which `from_bytes` never reads, so it sees the key and + /// keeps refusing silently. + #[test] + fn older_responder_decodes_a_busy_aware_request() { + let encoded = ReplicationMessage { + request_id: 22, + body: ReplicationMessageBody::FetchRequest(FetchRequest { + key: [0x23; 32], + busy_reply: true, + }), + } + .encode() + .expect("encode"); + let decoded: LegacyMessage = postcard::from_bytes(&encoded).expect("older decoder accepts"); + assert_eq!(decoded.request_id, 22); + match decoded.body { + LegacyBody::FetchRequest(LegacyFetchRequest { key }) => assert_eq!(key, [0x23; 32]), + _ => panic!("expected the legacy FetchRequest variant"), + } + } + + /// A current responder decodes an older requester's request, which has no + /// trailer, as one that did not ask for a busy reply. + #[test] + fn older_request_decodes_without_busy_reply() { + let decoded = ReplicationMessage::decode(&legacy_fetch_request_bytes([0x24; 32])) + .expect("current decoder accepts an older request"); + match decoded.body { + ReplicationMessageBody::FetchRequest(FetchRequest { key, busy_reply }) => { + assert_eq!(key, [0x24; 32]); + assert!(!busy_reply); + } + _ => panic!("expected FetchRequest"), + } + } + + /// Only the magic followed by a flags byte with the busy bit set asks for a + /// busy reply. Unknown flag bits and bytes after the flags are reserved and + /// ignored; anything else after the message reads as no capability, as it + /// always has. + #[test] + fn only_the_trailer_with_the_busy_bit_asks_for_a_busy_reply() { + let magic = FETCH_CAPABILITY_MAGIC.to_vec(); + let with = |tail: &[u8]| { + let mut t = magic.clone(); + t.extend_from_slice(tail); + t + }; + let base = legacy_fetch_request_bytes([0x25; 32]); + for (trailer, expected) in [ + (vec![], false), + (with(&[FETCH_CAPABILITY_BUSY_REPLY]), true), + (with(&[0]), false), + (with(&[0b1111_1111]), true), + (with(&[0b1111_1110]), false), + (with(&[FETCH_CAPABILITY_BUSY_REPLY, 0xAA, 0xBB]), true), + (magic.clone(), false), + (magic.get(..3).expect("short magic").to_vec(), false), + ( + vec![b'a', b'n', b't', b'g', FETCH_CAPABILITY_BUSY_REPLY], + false, + ), + (vec![FETCH_CAPABILITY_BUSY_REPLY], false), + (vec![0u8; 64], false), + ] { + let mut bytes = base.clone(); + bytes.extend_from_slice(&trailer); + let decoded = ReplicationMessage::decode(&bytes).expect("decodes"); + match decoded.body { + ReplicationMessageBody::FetchRequest(req) => { + assert_eq!(req.key, [0x25; 32]); + assert_eq!(req.busy_reply, expected, "trailer {trailer:?}"); + } + _ => panic!("expected FetchRequest"), + } + } + } + + /// The trailer means something only after a fetch request. After any other + /// body it is ignored exactly like every other trailing byte. + #[test] + fn other_bodies_ignore_a_capability_trailer() { + let mut bytes = ReplicationMessage { + request_id: 25, + body: ReplicationMessageBody::FetchResponse(FetchResponse::NotFound { + key: [0x29; 32], + }), + } + .encode() + .expect("encode"); + bytes.extend_from_slice(&FETCH_CAPABILITY_MAGIC); + bytes.push(FETCH_CAPABILITY_BUSY_REPLY); + let decoded = ReplicationMessage::decode(&bytes).expect("decodes"); + assert_eq!(decoded.request_id, 25); + assert!(matches!( + decoded.body, + ReplicationMessageBody::FetchResponse(FetchResponse::NotFound { key }) if key == [0x29; 32] + )); + } + + /// A request cut short inside the key is still malformed. + #[test] + fn a_truncated_key_is_still_rejected() { + let bytes = legacy_fetch_request_bytes([0x26; 32]); + let truncated = bytes.get(..bytes.len() - 1).expect("non-empty"); + assert!(ReplicationMessage::decode(truncated).is_err()); + } + + /// The pre-existing answers keep their discriminants, so an older requester + /// still decodes every answer it is ever sent. + #[test] + fn older_requester_decodes_the_unchanged_answers() { + for (body, check) in [ + ( + FetchResponse::Success { + key: [0x27; 32], + data: vec![1, 2, 3], + }, + 0u8, + ), + (FetchResponse::NotFound { key: [0x27; 32] }, 1), + ( + FetchResponse::Error { + key: [0x27; 32], + reason: "io".to_string(), + }, + 2, + ), + ] { + let encoded = ReplicationMessage { + request_id: 23, + body: ReplicationMessageBody::FetchResponse(body), + } + .encode() + .expect("encode"); + let decoded: LegacyMessage = + postcard::from_bytes(&encoded).expect("older decoder accepts"); + let got = match decoded.body { + LegacyBody::FetchResponse(LegacyFetchResponse::Success { key, data }) => { + assert_eq!(key, [0x27; 32]); + assert_eq!(data, vec![1, 2, 3]); + 0 + } + LegacyBody::FetchResponse(LegacyFetchResponse::NotFound { key }) => { + assert_eq!(key, [0x27; 32]); + 1 + } + LegacyBody::FetchResponse(LegacyFetchResponse::Error { key, reason }) => { + assert_eq!(key, [0x27; 32]); + assert_eq!(reason, "io"); + 2 + } + _ => panic!("expected a legacy FetchResponse"), + }; + assert_eq!(got, check); + } + } + + /// Why the responder gates `Busy` on the request's flag: an older requester + /// cannot decode it, and an undecodable answer costs the responder a trust + /// event at that requester. + #[test] + fn older_requester_cannot_decode_busy() { + let encoded = ReplicationMessage { + request_id: 24, + body: ReplicationMessageBody::FetchResponse(FetchResponse::Busy { key: [0x28; 32] }), + } + .encode() + .expect("encode"); + assert!(postcard::from_bytes::(&encoded).is_err()); + } + // === Audit roundtrips === #[test] @@ -2620,7 +2985,10 @@ mod tests { // fetched chunk needs. let mut core_body = ReplicationMessage { request_id: 1, - body: ReplicationMessageBody::FetchRequest(FetchRequest { key: [0u8; 32] }), + body: ReplicationMessageBody::FetchRequest(FetchRequest { + key: [0u8; 32], + busy_reply: false, + }), } .encode() .expect("a small core body encodes"); diff --git a/src/replication/scheduling.rs b/src/replication/scheduling.rs index e296606e..4d30e184 100644 --- a/src/replication/scheduling.rs +++ b/src/replication/scheduling.rs @@ -5,13 +5,18 @@ //! time, with strict dedup across all three stages. use std::cmp::Ordering; -use std::collections::{BTreeMap, BinaryHeap, HashMap, HashSet, VecDeque}; +use std::collections::{BTreeMap, BTreeSet, BinaryHeap, HashMap, HashSet, VecDeque}; +use std::sync::Arc; use std::time::{Duration, Instant}; use crate::logging::debug; -use crate::ant_protocol::XorName; -use crate::replication::config::VERIFICATION_RETRY_BACKOFF_MAX; +use crate::ant_protocol::{XorName, CLOSE_GROUP_SIZE}; +use crate::replication::audit_metrics; +use crate::replication::config::{ + FETCH_REQUEST_TIMEOUT, PAID_LIST_CLOSE_GROUP_SIZE, VERIFICATION_RETRY_BACKOFF_MAX, +}; +use crate::replication::fetch_busy::SourceBusyTracker; use crate::replication::types::{ FetchCandidate, FetchOrder, FetchPayload, VerificationEntry, VerificationState, }; @@ -94,6 +99,32 @@ pub const MAX_FETCH_QUEUE: usize = 131_072; /// decides *which* sources survive the cap. pub const MAX_HINT_SOURCES_PER_KEY: usize = 8; +/// Maximum distinct source peers one key may be associated with while it is +/// queued or in flight for fetch, counting the sources it was already fetched +/// from this cycle, for the given close-group widths. +/// +/// A verification round names at most its quorum targets and its paid-list +/// targets as holders, and the replica-hint fallbacks added to them come from a +/// pending entry already bounded by [`MAX_HINT_SOURCES_PER_KEY`], so this never +/// truncates a source list a round produced. It bounds what later +/// advertisements can merge into a queued or in-flight key, and with it the +/// parked-fetch index, which holds one entry per untried source of each parked +/// key. +#[must_use] +pub(crate) const fn fetch_source_limit( + close_group_size: usize, + paid_list_close_group_size: usize, +) -> usize { + close_group_size + .saturating_add(paid_list_close_group_size) + .saturating_add(MAX_HINT_SOURCES_PER_KEY) +} + +/// [`fetch_source_limit`] at the production close-group widths, and the limit +/// [`ReplicationQueues::new`] starts with. +pub(crate) const MAX_FETCH_SOURCES_PER_KEY: usize = + fetch_source_limit(CLOSE_GROUP_SIZE, PAID_LIST_CLOSE_GROUP_SIZE); + /// Outcome of [`ReplicationQueues::add_pending_verify`]. /// /// Distinguishes "the key is already being handled" from "the key was @@ -174,9 +205,13 @@ pub struct InFlightEntry { pub key: XorName, /// The peer we are currently fetching from. pub source: PeerId, - /// When the fetch started. + /// When the current attempt started: the first, or under the per-source + /// scheduler the latest retry. pub started_at: Instant, - /// All verified sources for this key. + /// Verified sources for this key not yet tried before this episode's + /// current attempt. A key that was parked between attempts keeps the + /// sources it already tried only in `tried`, so everything the key is + /// associated with is the union of the two. pub all_sources: Vec, /// Sources already attempted (failed or in progress). pub tried: HashSet, @@ -184,6 +219,409 @@ pub struct InFlightEntry { pub retry_verification: Option, } +// --------------------------------------------------------------------------- +// Per-source fetch dispatch +// --------------------------------------------------------------------------- + +/// A queued key none of whose untried sources had a free share when it was +/// last considered for dispatch. +/// +/// Its payload stays in `fetch_payloads`; only its place in line moves here +/// from the fetch heap. +#[derive(Debug, Clone)] +struct ParkedFetch { + /// XOR distance from self to the key, its priority once startable again. + distance: XorName, + /// Untried sources whose next free share could start the key, in the + /// payload's source-preference order. Each one indexes the key in + /// `parked_by_source`. + wake_sources: Vec, + /// When the key was parked, for the dispatch summary's oldest-parked age. + parked_at: Instant, +} + +/// Fetches started, and candidates found with no untried source, by one +/// [`ReplicationQueues::dispatch_fetches`] pass. +#[derive(Debug, Default)] +pub(crate) struct FetchDispatch { + /// Each started key with the source it was sent to. Already in flight. + pub(crate) started: Vec<(XorName, PeerId)>, + /// Dequeued candidates with no untried source left. The caller decides + /// what to do with them, exactly as for a sourceless dequeue. + pub(crate) sourceless: Vec, + /// Heap keys this pass parked because every untried source was full. + pub(crate) parked: usize, +} + +/// The per-source scheduler's queue state, for its periodic summary. +/// +/// Read only by that summary's log line, which a build without logging +/// compiles out. +#[derive(Debug, Clone, Copy)] +#[cfg_attr(not(feature = "logging"), allow(dead_code))] +pub(crate) struct FetchDispatchStats { + /// Keys parked behind full sources. + pub(crate) parked: usize, + /// How long the longest-parked key has waited, if any is parked. + pub(crate) oldest_parked: Option, + /// Keys queued for fetch, parked ones included. + pub(crate) queued: usize, + /// Fetches in flight. + pub(crate) in_flight: usize, + /// Sources holding at least the per-source cap of this node's fetches. + pub(crate) saturated_sources: usize, + /// Entries in the parked index: one per untried source of each parked key. + pub(crate) parked_associations: usize, + /// Longest parked-index maintenance after a departed source, since the last + /// read. + pub(crate) parked_maintenance_max: Duration, + /// Sources in the health table before the summary's purge. + pub(crate) tracked_sources: usize, + /// Tracked sources silent now. + pub(crate) silent_sources: usize, + /// Whether the health table is overflowing, so every untracked source + /// also counts as silent. + pub(crate) overflow_active: bool, + /// Source-health events since the last read. + pub(crate) health: SourceHealthCounters, +} + +/// What [`ReplicationQueues::retry_or_park_fetch`] did with a failed fetch. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum FetchRetry { + /// Retry now from this untried source, which has a free share. + Next(PeerId), + /// Every untried source is at its share: the key went back to the fetch + /// queue, parked, keeping its tried sources and retry reservation. + Parked, + /// No untried source is left, or the fetch queue is full. The key stays + /// in flight, with its failed source now marked tried, for the caller's + /// verification requeue. + Exhausted, +} + +// --------------------------------------------------------------------------- +// Fetch source health +// --------------------------------------------------------------------------- + +/// How many fetch timeouts a source stays silent after this node last learned +/// of an unanswered request to it. +const SILENCE_LIFETIME_TIMEOUTS: u32 = 2; + +/// Shortest time a mark lasts, whatever the fetch timeout, so a zero or +/// tiny configured timeout cannot turn silent sources off. +const MIN_SILENCE_LIFETIME: Duration = Duration::from_secs(1); + +/// Most fetch sources whose answering this node tracks at once. +/// +/// Far above the sources one node fetches from within a silence lifetime. At +/// the bound, entries that neither hold a mark nor guard an in-flight fetch +/// are dropped. If that frees nothing, a source not in the table counts as +/// silent for one silence lifetime: fetched from as it was before the +/// per-source cap, never more slowly. +const MAX_TRACKED_FETCH_SOURCES: usize = 8_192; + +/// Whether a fetch attempt reached its source. +/// +/// Requests and answers carry their place in one process-wide sequence, taken +/// as the request is sent and as its answer returns, so the table can tell +/// which came first however the fetch worker orders their outcomes and +/// whatever the clock reads. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum Contact { + /// The attempt ended before a request was sent. + NotAttempted, + /// The source answered, whatever the answer then turned out to be. + Answered { + /// Where the answer's arrival falls in the sequence. + order: u64, + }, + /// The request got no answer: a timeout or a connection failure. + Unreachable { + /// Where the request's sending falls in the sequence. + sent: u64, + }, + /// The attempt was cancelled at shutdown, so nothing is known. + Unknown, +} + +/// What this node has observed of one source's answering. +#[derive(Debug, Clone, Copy, Default)] +struct SourceHealth { + /// Sequence place of the latest answer from this source. + last_answer: Option, + /// Sequence place of the latest-sent request to this source that got no + /// answer. + last_unanswered: Option, + /// When this node learned of that request's failure, while the request is + /// newer than the latest answer and the mark has not expired. + marked_since: Option, +} + +impl SourceHealth { + /// Whether a request sent after this source's latest answer went + /// unanswered. It depends only on the two latest events, so it is the same + /// whatever order the outcomes were applied in. + fn unanswered_since_answer(&self) -> bool { + self.last_unanswered + .is_some_and(|sent| self.last_answer.is_none_or(|answer| sent > answer)) + } +} + +/// Source-health events since the dispatch summary last read them. +/// +/// Read only by that summary's log line, which a build without logging +/// compiles out. +#[derive(Debug, Clone, Copy, Default)] +#[cfg_attr(not(feature = "logging"), allow(dead_code))] +pub(crate) struct SourceHealthCounters { + /// Attempts whose request got no answer. + pub(crate) unreachable: u64, + /// Sources that became silent. + pub(crate) marks: u64, + /// Marks cleared by an answer. + pub(crate) clears: u64, + /// Marks that ran out their lifetime. + pub(crate) expired_marks: u64, + /// Unanswered requests sent before their source's latest answer. + pub(crate) ignored_stale: u64, + /// Fetches started or retried on a silent source, outside the cap. + pub(crate) silent_starts: u64, + /// Times the table was full of marks and in-flight guards. + pub(crate) overflows: u64, + /// Most tracked sources silent at once. + pub(crate) silent_peak: usize, +} + +/// What this node has observed of each fetch source's answering. +/// +/// The per-source cap keeps this node within the admission share a responder +/// grants it, and only a responder that answers has a share to keep within. A +/// source is silent while the latest-sent request to it that went unanswered +/// was sent after its latest answer: it is fetched from without the cap, as +/// before the cap existed. Any answer received after that request was sent +/// ends it. So does learning of no newer unanswered request for one silence +/// lifetime, so a source this node has stopped trying is probed again under +/// the cap. +#[derive(Debug)] +struct SourceHealthTable { + entries: HashMap, + /// Every mark as `(marked_since, source)`, so marks expire in order. + marks_by_since: BTreeSet<(Instant, PeerId)>, + /// How long a mark lasts without newer evidence. + silence_lifetime: Duration, + /// When the table last overflowed. For one silence lifetime after, a + /// source not in the table counts as silent. + overflow_since: Option, + counters: SourceHealthCounters, +} + +impl SourceHealthTable { + fn new(fetch_request_timeout: Duration) -> Self { + Self { + entries: HashMap::new(), + marks_by_since: BTreeSet::new(), + silence_lifetime: fetch_request_timeout + .saturating_mul(SILENCE_LIFETIME_TIMEOUTS) + .max(MIN_SILENCE_LIFETIME), + overflow_since: None, + counters: SourceHealthCounters::default(), + } + } + + /// Whether something that began at `since` is still within one silence + /// lifetime at `now`. + fn within_lifetime(&self, since: Instant, now: Instant) -> bool { + now.saturating_duration_since(since) < self.silence_lifetime + } + + fn overflowing(&self, now: Instant) -> bool { + self.overflow_since + .is_some_and(|since| self.within_lifetime(since, now)) + } + + /// Whether `source` counts as silent at `now`. + fn is_silent(&self, source: &PeerId, now: Instant) -> bool { + self.entries.get(source).map_or_else( + || self.overflowing(now), + |health| { + health + .marked_since + .is_some_and(|since| self.within_lifetime(since, now)) + }, + ) + } + + /// Drop the marks that ran out their lifetime by `now`. + fn expire(&mut self, now: Instant) { + while let Some(&(since, source)) = self.marks_by_since.first() { + if self.within_lifetime(since, now) { + break; + } + self.marks_by_since.pop_first(); + if let Some(health) = self.entries.get_mut(&source) { + if health.marked_since == Some(since) { + health.marked_since = None; + } + } + self.counters.expired_marks += 1; + } + } + + /// Drop every entry that neither holds a mark nor guards an in-flight + /// fetch to its source. Every contact, an answer reported ahead of its + /// fetch's outcome included, is applied while its own fetch is still in + /// flight, so none can arrive for a dropped source's past requests: + /// anything later is newer than all of its history. Call after + /// [`Self::expire`]. + fn purge(&mut self, in_flight: &HashMap) { + let guarded: HashSet = in_flight.values().map(|entry| entry.source).collect(); + self.entries + .retain(|source, health| health.marked_since.is_some() || guarded.contains(source)); + } + + /// The entry for `source`, purging the table at the bound. `None` when the + /// table cannot take it, which starts an overflow. + fn entry( + &mut self, + source: PeerId, + now: Instant, + in_flight: &HashMap, + ) -> Option<&mut SourceHealth> { + if !self.entries.contains_key(&source) && self.entries.len() >= MAX_TRACKED_FETCH_SOURCES { + if self.overflowing(now) { + return None; + } + self.purge(in_flight); + if self.entries.len() >= MAX_TRACKED_FETCH_SOURCES { + self.overflow_since = Some(now); + self.counters.overflows += 1; + return None; + } + } + Some(self.entries.entry(source).or_default()) + } + + /// Apply, at `now`, one attempt's contact with `source`. + fn record( + &mut self, + source: PeerId, + contact: Contact, + now: Instant, + in_flight: &HashMap, + ) { + self.expire(now); + match contact { + Contact::NotAttempted | Contact::Unknown => {} + Contact::Answered { order } => { + let Some(health) = self.entry(source, now, in_flight) else { + return; + }; + health.last_answer = health.last_answer.max(Some(order)); + let cleared = if health.unanswered_since_answer() { + None + } else { + health.marked_since.take() + }; + if let Some(since) = cleared { + self.marks_by_since.remove(&(since, source)); + self.counters.clears += 1; + } + } + Contact::Unreachable { sent } => { + self.counters.unreachable += 1; + let Some(health) = self.entry(source, now, in_flight) else { + return; + }; + let stale = health.last_answer.is_some_and(|answer| sent < answer); + let newest = health.last_unanswered.is_none_or(|latest| sent > latest); + if newest { + health.last_unanswered = Some(sent); + } + // The newest unanswered request, if newer than every answer, + // marks the source from now: from when this node learned of it. + let (replaced, marked) = if newest && health.unanswered_since_answer() { + (health.marked_since.replace(now), true) + } else { + (None, false) + }; + if stale { + self.counters.ignored_stale += 1; + } + if marked { + if let Some(since) = replaced { + self.marks_by_since.remove(&(since, source)); + } else { + self.counters.marks += 1; + } + self.marks_by_since.insert((now, source)); + self.counters.silent_peak = + self.counters.silent_peak.max(self.marks_by_since.len()); + } + } + } + } + + /// The counters since the last call, and the table's size before the + /// purge that follows. + fn take_counters( + &mut self, + now: Instant, + in_flight: &HashMap, + ) -> (SourceHealthCounters, usize) { + self.expire(now); + let tracked = self.entries.len(); + self.purge(in_flight); + let counters = std::mem::take(&mut self.counters); + self.counters.silent_peak = self.marks_by_since.len(); + (counters, tracked) + } +} + +/// What lets a parked key start now: an untried source below the per-source +/// cap, or a silent one. +#[derive(Clone, Copy)] +struct Startable<'a> { + in_flight: &'a HashMap, + per_source_cap: usize, + now: Instant, +} + +/// The next queued key, taken off the heap or out of the parked set. +/// +/// Returned and matched at once, never stored, so the size gap between the +/// variants costs nothing; boxing would add an allocation per dispatched fetch. +#[allow(clippy::large_enum_variant)] +enum TakenFetch { + /// The key, rejoined with its payload. + Candidate(FetchCandidate), + /// An inconsistent entry was dropped instead; try again. + Skipped, + /// The heap is empty and no parked key was asked for. + Empty, +} + +/// How [`ReplicationQueues::dispatch_fetches`] may start one candidate. +enum SourceChoice { + /// This untried source has a free share. + Start(PeerId), + /// Untried sources exist, but every one is at its share. + AllBusy, + /// No untried source is left. + NoSource, +} + +/// Test-only: every key queued or parked for fetch, and every in-flight fetch +/// with its source, read under one lock. +#[cfg(any(test, feature = "test-utils"))] +#[derive(Debug, Clone, Default)] +pub struct FetchPipelineSnapshot { + /// Keys queued for fetch, on the heap or parked. + pub queued: Vec, + /// In-flight fetches with the source each one was sent to. + pub in_flight: Vec<(XorName, PeerId)>, +} + // --------------------------------------------------------------------------- // Central queue manager // --------------------------------------------------------------------------- @@ -205,10 +643,11 @@ pub struct ReplicationQueues { /// Capacity-bounded by [`MAX_PENDING_VERIFY`]. At capacity, max-min sender /// accounting reclaims borrowed entries before rejecting new work. pending_verify: HashMap, - /// Nearest-first priority order over the keys in `fetch_payloads`. + /// Nearest-first priority order over the queued keys that are not parked. /// /// Holds ordering data only: every entry has exactly one matching - /// `fetch_payloads` entry, and vice versa. Splitting the mutable half of a + /// `fetch_payloads` entry, and every `fetch_payloads` entry is represented + /// exactly once, here or in `fetch_parked`. Splitting the mutable half of a /// queued fetch out into `fetch_payloads` keeps this heap's ordering /// immutable for as long as a key is queued, so a source merge never has /// to rebuild it. @@ -221,8 +660,34 @@ pub struct ReplicationQueues { /// Capacity-bounded by [`MAX_FETCH_QUEUE`]: enqueues are dropped once /// full, preventing unbounded growth under a network hint flood. fetch_payloads: HashMap, + /// Queued keys taken off `fetch_queue` because every untried source was at + /// its per-source share. Every key in `fetch_payloads` is in exactly one of + /// `fetch_queue` and this map. + fetch_parked: HashMap, + /// Exact reverse index over `fetch_parked`: `(distance, key)` is in the set + /// for `source` if and only if `key` is parked with `source` among its + /// wake sources. Ordered so the nearest parked key per source is `first()`; + /// empty sets are removed. + parked_by_source: HashMap>, + /// Sources a queued key was already fetched from in the current + /// verification cycle. Present only for a key that went back to the queue + /// from flight, and carried into its next in-flight entry. + fetch_tried: HashMap>, /// Active downloads keyed by `XorName`. in_flight_fetch: HashMap, + /// Most distinct sources a queued or in-flight key may be associated with, + /// tried ones included. See [`fetch_source_limit`]. + max_fetch_sources: usize, + /// Longest parked-index maintenance after a departed source since the + /// dispatch summary last read it. + parked_maintenance_max: Duration, + /// Which fetch sources stopped answering, so the per-source cap can leave + /// them out. + source_health: SourceHealthTable, + /// Which fetch sources recently answered busy, so a key starts or retries + /// on another free source first. Shared with the fetches that record the + /// busy answers. + busy_sources: Arc, /// Reverse index for removing a departed peer from every pending hint /// without scanning the entire pending table. pending_keys_by_source: HashMap>, @@ -272,7 +737,14 @@ impl ReplicationQueues { pending_verify: HashMap::new(), fetch_queue: BinaryHeap::new(), fetch_payloads: HashMap::new(), + fetch_parked: HashMap::new(), + parked_by_source: HashMap::new(), + fetch_tried: HashMap::new(), in_flight_fetch: HashMap::new(), + max_fetch_sources: MAX_FETCH_SOURCES_PER_KEY, + parked_maintenance_max: Duration::ZERO, + source_health: SourceHealthTable::new(FETCH_REQUEST_TIMEOUT), + busy_sources: Arc::new(SourceBusyTracker::new()), pending_keys_by_source: HashMap::new(), capacity_owner_by_key: HashMap::new(), pending_keys_by_owner: HashMap::new(), @@ -286,6 +758,53 @@ impl ReplicationQueues { } } + /// Use `limit` as the most distinct sources a queued or in-flight key may + /// be associated with, for close-group widths other than the production + /// ones. See [`fetch_source_limit`]. + #[must_use] + pub(crate) fn with_fetch_source_limit(mut self, limit: usize) -> Self { + self.set_fetch_source_limit(limit); + self + } + + /// Change the source limit. Only for queues no key has entered yet, such + /// as an engine's before it starts. + pub(crate) fn set_fetch_source_limit(&mut self, limit: usize) { + self.max_fetch_sources = limit.max(1); + } + + /// Use `timeout`, the fetch request timeout, to time how long a source + /// that stopped answering stays silent. Only for queues no fetch has + /// resolved in yet. + #[must_use] + pub(crate) fn with_fetch_request_timeout(mut self, timeout: Duration) -> Self { + self.source_health = SourceHealthTable::new(timeout); + self + } + + /// Read which sources recently answered busy from `busy_sources`, the + /// tracker this node's fetches record those answers in. + #[must_use] + pub(crate) fn with_busy_sources(mut self, busy_sources: Arc) -> Self { + self.busy_sources = busy_sources; + self + } + + /// The tracker of sources that recently answered busy. + #[cfg(test)] + pub(crate) fn busy_sources(&self) -> &SourceBusyTracker { + &self.busy_sources + } + + /// Apply whether a fetch attempt's `source` answered. Called for every + /// resolved attempt, before the attempt's result moves its key on, and for + /// an answer as soon as it arrives, while its attempt is still in flight. + /// Applying the same answer twice changes nothing. + pub(crate) fn record_fetch_contact(&mut self, source: PeerId, contact: Contact) { + self.source_health + .record(source, contact, Instant::now(), &self.in_flight_fetch); + } + // ----------------------------------------------------------------------- // PendingVerify // ----------------------------------------------------------------------- @@ -302,6 +821,7 @@ impl ReplicationQueues { /// source-side retry) MUST treat this as still-outstanding work, not as /// "done". Without this distinction a bootstrap snapshot whose hints /// are capacity-rejected would silently mark itself drained. + #[allow(clippy::too_many_lines)] pub fn add_pending_verify( &mut self, key: XorName, @@ -373,32 +893,66 @@ impl ReplicationQueues { return AdmissionResult::AlreadyPresent; } // Merging a source touches only `FetchPayload`, which no ordering - // reads, so the fetch heap is left completely untouched here. + // reads, so the fetch heap is left completely untouched here. A source + // this key was already fetched from this cycle is not merged back: it + // would be asked twice. A parked key can start from a merged source as + // soon as that source has a free share, so it is indexed there too. if let Some(payload) = self.fetch_payloads.get_mut(&key) { + let tried = self.fetch_tried.get(&key); + let mut merged = Vec::new(); for source in &entry.replica_hint_sources { - if !payload.sources.contains(source) { + // Payload sources and tried sources are disjoint, so this is + // the size of everything the key has been associated with. + if payload.sources.len() + tried.map_or(0, HashSet::len) >= self.max_fetch_sources { + break; + } + if !payload.sources.contains(source) && !tried.is_some_and(|t| t.contains(source)) { payload.sources.push(*source); + merged.push(*source); + } + } + if let Some(parked) = self.fetch_parked.get_mut(&key) { + for source in merged { + parked.wake_sources.push(source); + self.parked_by_source + .entry(source) + .or_default() + .insert((parked.distance, key)); } } if let Some(retry) = &mut payload.retry_verification { - retry - .replica_hint_sources - .extend(entry.replica_hint_sources); - retry.hint_sources.extend(entry.hint_sources); + Self::merge_retry_hints( + retry, + &entry.replica_hint_sources, + &entry.hint_sources, + self.capacity_owner_by_key.get(&key).copied(), + ); } return AdmissionResult::AlreadyPresent; } if let Some(in_flight) = self.in_flight_fetch.get_mut(&key) { + let mut associated = in_flight.all_sources.len() + + in_flight + .tried + .iter() + .filter(|peer| !in_flight.all_sources.contains(peer)) + .count(); for source in &entry.replica_hint_sources { - if !in_flight.all_sources.contains(source) { + if associated >= self.max_fetch_sources { + break; + } + if !in_flight.all_sources.contains(source) && !in_flight.tried.contains(source) { in_flight.all_sources.push(*source); + associated += 1; } } if let Some(retry) = &mut in_flight.retry_verification { - retry - .replica_hint_sources - .extend(entry.replica_hint_sources); - retry.hint_sources.extend(entry.hint_sources); + Self::merge_retry_hints( + retry, + &entry.replica_hint_sources, + &entry.hint_sources, + self.capacity_owner_by_key.get(&key).copied(), + ); } return AdmissionResult::AlreadyPresent; } @@ -652,10 +1206,7 @@ impl ReplicationQueues { key: &XorName, protected_owner: Option, ) -> bool { - let victim = entry.hint_sources.iter().copied().find(|source| { - !entry.replica_hint_sources.contains(source) && protected_owner != Some(*source) - }); - let Some(victim) = victim else { + let Some(victim) = Self::paid_only_victim(entry, protected_owner) else { return false; }; entry.hint_sources.remove(&victim); @@ -668,6 +1219,53 @@ impl ReplicationQueues { true } + /// The paid-only advertiser [`Self::displace_paid_only_source`] would drop + /// from `entry`: never a replica claimant, never `protected_owner`. + fn paid_only_victim( + entry: &VerificationEntry, + protected_owner: Option, + ) -> Option { + entry.hint_sources.iter().copied().find(|source| { + !entry.replica_hint_sources.contains(source) && protected_owner != Some(*source) + }) + } + + /// Merge a re-advertisement into the verification entry a fetch-stage key + /// keeps for a later retry, under the bound and in the order + /// [`Self::add_pending_verify`] applies to a pending entry: replica + /// claimants first, displacing paid-only advertisers other than + /// `protected_owner` when the set is full, then paid-only advertisers + /// while it is below [`MAX_HINT_SOURCES_PER_KEY`]. A fetch-stage key is in + /// no pending index, so only the entry changes. + fn merge_retry_hints( + retry: &mut VerificationEntry, + replica_hint_sources: &HashSet, + hint_sources: &HashSet, + protected_owner: Option, + ) { + for source in replica_hint_sources { + if retry.replica_hint_sources.contains(source) { + continue; + } + if !retry.hint_sources.contains(source) + && retry.hint_sources.len() >= MAX_HINT_SOURCES_PER_KEY + { + let Some(victim) = Self::paid_only_victim(retry, protected_owner) else { + break; + }; + retry.hint_sources.remove(&victim); + } + retry.replica_hint_sources.insert(*source); + retry.hint_sources.insert(*source); + } + for source in hint_sources { + if retry.hint_sources.len() >= MAX_HINT_SOURCES_PER_KEY { + break; + } + retry.hint_sources.insert(*source); + } + } + /// Reserve a pending slot for `key`, charged to its capacity owner. /// /// Must be called while `capacity_owner_by_key` still holds `key`, which is @@ -951,8 +1549,9 @@ impl ReplicationQueues { .retain(|key, _| !orphaned_fetch_keys.contains(key)); self.fetch_queue .retain(|order| !orphaned_fetch_keys.contains(&order.key)); - orphaned.extend(orphaned_fetch_keys); } + self.drop_departed_source_from_parked(source, &orphaned_fetch_keys); + orphaned.extend(orphaned_fetch_keys); for entry in self.in_flight_fetch.values_mut() { entry.all_sources.retain(|peer| peer != source); @@ -1114,7 +1713,7 @@ impl ReplicationQueues { &mut self, key: XorName, distance: XorName, - sources: Vec, + mut sources: Vec, retry_verification: Option, ) -> bool { if self.pending_verify.contains_key(&key) @@ -1130,6 +1729,7 @@ impl ReplicationQueues { ); return false; } + sources.truncate(self.max_fetch_sources); self.fetch_payloads.insert( key, FetchPayload { @@ -1138,6 +1738,10 @@ impl ReplicationQueues { }, ); self.fetch_queue.push(FetchOrder { key, distance }); + // A fresh enqueue starts a new verification cycle's fetch, so nothing + // asked in an earlier cycle is excluded. Cleared only now, after the + // insert succeeded: a refused enqueue leaves an existing entry alone. + self.fetch_tried.remove(&key); true } @@ -1184,39 +1788,84 @@ impl ReplicationQueues { enqueued } - /// Dequeue the nearest fetch candidate. + /// Dequeue the nearest fetch candidate, whether it is on the fetch heap or + /// parked. /// /// Returns `None` when the queue is empty. Silently skips candidates /// that are somehow already in-flight. Concurrency is enforced by the /// fetch worker, not by this method. /// /// A returned candidate may carry a live verification retry-slot - /// reservation. Callers must consume it with - /// [`Self::start_dequeued_fetch`], [`Self::discard_fetch_candidate`], or + /// reservation, and the sources it was already fetched from this cycle. + /// Callers must consume it with [`Self::start_dequeued_fetch`], + /// [`Self::discard_fetch_candidate`], or /// [`Self::requeue_candidate_for_verification`] so that reservation is /// either transferred, released, or restored to `pending_verify`. pub fn dequeue_fetch(&mut self) -> Option { - while let Some(order) = self.fetch_queue.pop() { - let Some(payload) = self.fetch_payloads.remove(&order.key) else { - debug_assert!( - false, - "fetch order without payload for {}", - hex::encode(order.key) - ); - continue; - }; - let candidate = FetchCandidate { - key: order.key, - distance: order.distance, - sources: payload.sources, - retry_verification: payload.retry_verification, + loop { + let parked = self.nearest_parked(None); + let candidate = match self.take_queued(self.parked_if_nearer(parked)) { + TakenFetch::Candidate(candidate) => candidate, + TakenFetch::Skipped => continue, + TakenFetch::Empty => return None, }; if !self.in_flight_fetch.contains_key(&candidate.key) { return Some(candidate); } - self.release_retry_slot_for_candidate(&candidate); + self.drop_duplicate_of_in_flight(&candidate); } - None + } + + /// `parked`'s key when it is nearer than the heap head, or the heap is + /// empty; `None` to take the heap head. + fn parked_if_nearer(&self, parked: Option<(XorName, XorName)>) -> Option { + let heap_head = self + .fetch_queue + .peek() + .map(|order| (order.distance, order.key)); + match (heap_head, parked) { + (Some(head), Some(parked)) if parked < head => Some(parked.1), + (None, Some(parked)) => Some(parked.1), + _ => None, + } + } + + /// Remove a queued key from where it waits, the parked key `parked` or + /// otherwise the heap head, and rejoin it with its payload. + fn take_queued(&mut self, parked: Option) -> TakenFetch { + let (key, distance) = if let Some(key) = parked { + let Some(distance) = self.unpark(&key) else { + self.purge_parked_index_entry(&key); + return TakenFetch::Skipped; + }; + (key, distance) + } else { + let Some(order) = self.fetch_queue.pop() else { + return TakenFetch::Empty; + }; + (order.key, order.distance) + }; + let Some(payload) = self.fetch_payloads.remove(&key) else { + debug_assert!( + false, + "queued fetch without payload for {}", + hex::encode(key) + ); + return TakenFetch::Skipped; + }; + TakenFetch::Candidate(FetchCandidate { + key, + distance, + sources: payload.sources, + retry_verification: payload.retry_verification, + }) + } + + /// Discard a dequeued candidate whose key is somehow already in flight, + /// releasing what it carried. + fn drop_duplicate_of_in_flight(&mut self, candidate: &FetchCandidate) { + self.release_retry_slot_for_candidate(candidate); + self.fetch_tried.remove(&candidate.key); } /// Number of keys waiting in the fetch queue. @@ -1253,8 +1902,41 @@ impl ReplicationQueues { all_sources: Vec, retry_verification: Option, ) { - let mut tried = HashSet::new(); + self.start_fetch_after(key, source, all_sources, retry_verification, HashSet::new()); + } + + /// Start `key` from `source`, carrying the sources it was already fetched + /// from this cycle into the in-flight entry's tried set. + fn start_fetch_after( + &mut self, + key: XorName, + source: PeerId, + mut all_sources: Vec, + retry_verification: Option, + mut tried: HashSet, + ) { tried.insert(source); + // Keep the distinct union of tried and listed sources within the + // limit by dropping untried sources from the end of the list. Every + // dispatched start is already within it; this binds direct starts. + let union = all_sources.len() + + tried + .iter() + .filter(|peer| !all_sources.contains(peer)) + .count(); + let mut excess = union.saturating_sub(self.max_fetch_sources); + if excess > 0 { + let mut kept = Vec::with_capacity(all_sources.len()); + for peer in all_sources.into_iter().rev() { + if excess > 0 && !tried.contains(&peer) { + excess -= 1; + } else { + kept.push(peer); + } + } + kept.reverse(); + all_sources = kept; + } let replaced = self.in_flight_fetch.insert( key, InFlightEntry { @@ -1271,8 +1953,9 @@ impl ReplicationQueues { } } - /// Consume a dequeued fetch candidate and transfer its retry reservation - /// into the in-flight entry. + /// Consume a dequeued fetch candidate and transfer its retry reservation, + /// and the sources it was already fetched from this cycle, into the + /// in-flight entry. pub fn start_dequeued_fetch(&mut self, candidate: FetchCandidate, source: PeerId) { let FetchCandidate { key, @@ -1280,7 +1963,8 @@ impl ReplicationQueues { retry_verification, .. } = candidate; - self.start_fetch_with_retry(key, source, sources, retry_verification); + let tried = self.fetch_tried.remove(&key).unwrap_or_default(); + self.start_fetch_after(key, source, sources, retry_verification, tried); } /// Mark a fetch as completed (success or permanent failure). @@ -1300,6 +1984,7 @@ impl ReplicationQueues { retry_verification, .. } = candidate; + self.fetch_tried.remove(&key); if retry_verification.is_some() { self.release_retry_slot(&key); self.capacity_owner_by_key.remove(&key); @@ -1329,73 +2014,498 @@ impl ReplicationQueues { } } - /// Consume a dequeued candidate and restore its verification entry for a - /// later retry when retry metadata exists. - pub fn requeue_candidate_for_verification( + // ----------------------------------------------------------------------- + // Per-source dispatch + // ----------------------------------------------------------------------- + + /// Start up to `slots` queued fetches without putting more than + /// `per_source_cap` of this node's fetches in flight to any one source + /// that is answering. + /// + /// Each slot goes to the nearest queued key that can start now: one with + /// an untried source below the cap, or an untried silent source. It starts + /// on the first untried source below the cap that is not silent, in its + /// source-preference order, one that recently answered busy only when no + /// other is below the cap, or else on its first untried silent source, + /// past the cap. A heap key that cannot start is parked under each of its + /// untried sources, so it leaves the heap once and a startable key behind + /// it is reached in the same pass; a parked key becomes a candidate again + /// as soon as one of those sources has a free share or falls silent, and + /// competes on distance with the heap. + /// + /// With `per_source_cap = usize::MAX` nothing parks and every key starts on + /// its first untried source that is not silent, in plain nearest-first + /// order. + pub(crate) fn dispatch_fetches( &mut self, - candidate: FetchCandidate, - retry_after: Duration, - ) -> bool { + slots: usize, + per_source_cap: usize, + ) -> FetchDispatch { + let now = Instant::now(); + self.source_health.expire(now); + let mut dispatch = FetchDispatch::default(); + let mut in_flight = self.in_flight_counts_by_source(None); + let mut best_parked = self.nearest_parked(Some(Startable { + in_flight: &in_flight, + per_source_cap, + now, + })); + while dispatch.started.len() < slots { + let from_parked = self.parked_if_nearer(best_parked); + let candidate = match self.take_queued(from_parked) { + TakenFetch::Candidate(candidate) => Some(candidate), + TakenFetch::Skipped => None, + TakenFetch::Empty => break, + }; + let mut started = false; + if let Some(candidate) = candidate { + if self.in_flight_fetch.contains_key(&candidate.key) { + self.drop_duplicate_of_in_flight(&candidate); + } else { + let startable = Startable { + in_flight: &in_flight, + per_source_cap, + now, + }; + match self.first_startable_source(&candidate, startable) { + SourceChoice::Start(source) => { + if self.source_health.is_silent(&source, now) { + self.source_health.counters.silent_starts += 1; + } + *in_flight.entry(source).or_default() += 1; + dispatch.started.push((candidate.key, source)); + self.start_dequeued_fetch(candidate, source); + started = true; + } + SourceChoice::NoSource => dispatch.sourceless.push(candidate), + SourceChoice::AllBusy => { + if from_parked.is_none() { + dispatch.parked += 1; + } + self.park_candidate(candidate); + } + } + } + } + // A start changes the per-source counts, and a key taken from the + // parked set changes the set itself. A heap key that parked is + // indexed only under full sources that are not silent, so it + // cannot change the answer. + if started || from_parked.is_some() { + best_parked = self.nearest_parked(Some(Startable { + in_flight: &in_flight, + per_source_cap, + now, + })); + } + } + dispatch + } + + /// Put a dequeued candidate back in the queue, parked under its untried + /// sources. + fn park_candidate(&mut self, candidate: FetchCandidate) { let FetchCandidate { key, + distance, + sources, retry_verification, - .. } = candidate; - let Some(mut verification) = retry_verification else { - return false; - }; - verification.state = VerificationState::PendingVerify; - verification.verified_sources.clear(); - verification.tried_sources.clear(); - verification.next_verify_at = Instant::now() + retry_after; - - let owner = self - .capacity_owner_by_key - .get(&key) - .copied() - .or_else(|| self.least_loaded_source(&verification.hint_sources)); - let Some(owner) = owner else { - self.release_retry_slot(&key); - return false; - }; - // Release before re-inserting: the reservation is charged to the owner - // recorded now, which `insert_pending_owned_unchecked` may replace. - self.release_retry_slot(&key); - self.insert_pending_owned_unchecked(key, verification, owner, true); - true + self.fetch_payloads.insert( + key, + FetchPayload { + sources, + retry_verification, + }, + ); + self.park(key, distance); } - /// Complete an exhausted fetch and restore its verification entry for a - /// later retry when retry metadata exists. - pub fn requeue_fetch_for_verification(&mut self, key: &XorName, retry_after: Duration) -> bool { - let Some(mut entry) = self.in_flight_fetch.remove(key) else { - return false; - }; - let Some(mut verification) = entry.retry_verification.take() else { - self.capacity_owner_by_key.remove(key); - return false; + /// Mark the current fetch attempt as failed and move `key` on, without + /// sending to a source that is answering and already has `per_source_cap` + /// of this node's fetches in flight. + /// + /// The next untried source with a free share that is not silent is + /// returned to retry from now, as [`Self::retry_fetch`] would, one that + /// recently answered busy only when no other is free, or else the first + /// untried silent source, past the cap. If every untried source is + /// answering and at its share, the key goes back to the fetch queue + /// parked, with only its untried sources and with its tried set kept, so + /// no source is asked twice this cycle; its retry reservation travels with + /// it. If no untried source is left, or the fetch queue is at + /// [`MAX_FETCH_QUEUE`], the in-flight entry is left for the caller's + /// verification requeue, which keeps `MAX_FETCH_QUEUE` a hard bound. + pub(crate) fn retry_or_park_fetch( + &mut self, + key: &XorName, + distance: XorName, + per_source_cap: usize, + ) -> FetchRetry { + let now = Instant::now(); + self.source_health.expire(now); + let in_flight = self.in_flight_counts_by_source(Some(key)); + let queue_full = self.fetch_payloads.len() >= MAX_FETCH_QUEUE; + let Some(entry) = self.in_flight_fetch.get_mut(key) else { + return FetchRetry::Exhausted; }; - verification.state = VerificationState::PendingVerify; - verification.verified_sources.clear(); - verification.tried_sources.clear(); - verification.next_verify_at = Instant::now() + retry_after; - - let owner = self - .capacity_owner_by_key - .get(key) + entry.tried.insert(entry.source); + let untried: Vec = entry + .all_sources + .iter() + .filter(|peer| !entry.tried.contains(peer)) .copied() - .or_else(|| self.least_loaded_source(&verification.hint_sources)); - let Some(owner) = owner else { - self.release_retry_slot(key); - return false; + .collect(); + if untried.is_empty() { + return FetchRetry::Exhausted; + } + let health = &self.source_health; + let busy = &self.busy_sources; + let is_free = |peer: &PeerId| { + !health.is_silent(peer, now) + && in_flight.get(peer).copied().unwrap_or(0) < per_source_cap }; - // Release before re-inserting: the reservation is charged to the owner - // recorded now, which `insert_pending_owned_unchecked` may replace. - self.release_retry_slot(key); + // A free source that recently answered busy goes after every other + // free one, and still before a silent one. + let rested = untried + .iter() + .find(|peer| is_free(peer) && !busy.is_cooling(peer, now)) + .copied(); + if let Some(rested) = rested { + if untried + .iter() + .take_while(|peer| **peer != rested) + .any(&is_free) + { + audit_metrics::record_fetch_busy_cooling_skip(); + } + } + let free = rested.or_else(|| untried.iter().find(|peer| is_free(peer)).copied()); + let silent = || { + untried + .iter() + .find(|peer| health.is_silent(peer, now)) + .copied() + }; + let next = free.or_else(silent); + if let Some(next) = next { + if free.is_none() { + self.source_health.counters.silent_starts += 1; + } + entry.source = next; + entry.tried.insert(next); + entry.started_at = now; + return FetchRetry::Next(next); + } + if queue_full { + return FetchRetry::Exhausted; + } + let Some(entry) = self.in_flight_fetch.remove(key) else { + return FetchRetry::Exhausted; + }; + self.fetch_payloads.insert( + *key, + FetchPayload { + sources: untried, + retry_verification: entry.retry_verification, + }, + ); + self.fetch_tried.insert(*key, entry.tried); + self.park(*key, distance); + FetchRetry::Parked + } + + /// In-flight fetches per source, optionally leaving out `exclude`'s own + /// entry. + fn in_flight_counts_by_source(&self, exclude: Option<&XorName>) -> HashMap { + let mut counts = HashMap::new(); + for (key, entry) in &self.in_flight_fetch { + if exclude != Some(key) { + *counts.entry(entry.source).or_default() += 1; + } + } + counts + } + + /// The first untried source of `candidate`, in its source-preference + /// order, that is not silent and is below the cap, or else its first + /// untried silent source. A source below the cap that recently answered + /// busy is passed over while another one below the cap is not cooling, + /// and still comes before a silent one: it answers, if only to say busy. + fn first_startable_source( + &self, + candidate: &FetchCandidate, + startable: Startable<'_>, + ) -> SourceChoice { + let tried = self.fetch_tried.get(&candidate.key); + let mut any_untried = false; + let mut first_silent = None; + let mut first_cooling = None; + for source in &candidate.sources { + if tried.is_some_and(|t| t.contains(source)) { + continue; + } + any_untried = true; + if self.source_health.is_silent(source, startable.now) { + first_silent = first_silent.or(Some(*source)); + } else if startable.in_flight.get(source).copied().unwrap_or(0) + < startable.per_source_cap + { + if !self.busy_sources.is_cooling(source, startable.now) { + if first_cooling.is_some() { + audit_metrics::record_fetch_busy_cooling_skip(); + } + return SourceChoice::Start(*source); + } + first_cooling = first_cooling.or(Some(*source)); + } + } + match (first_cooling.or(first_silent), any_untried) { + (Some(source), _) => SourceChoice::Start(source), + (None, true) => SourceChoice::AllBusy, + (None, false) => SourceChoice::NoSource, + } + } + + /// Park a queued key whose payload is in `fetch_payloads`, indexed under + /// each of its untried sources. A key with no untried source goes back on + /// the heap instead, so it is never parked with nothing to wake it. + fn park(&mut self, key: XorName, distance: XorName) { + self.unpark(&key); + let tried = self.fetch_tried.get(&key); + let wake_sources: Vec = self + .fetch_payloads + .get(&key) + .map(|payload| { + payload + .sources + .iter() + .filter(|source| !tried.is_some_and(|t| t.contains(source))) + .copied() + .collect() + }) + .unwrap_or_default(); + if wake_sources.is_empty() { + self.fetch_queue.push(FetchOrder { key, distance }); + return; + } + for source in &wake_sources { + self.parked_by_source + .entry(*source) + .or_default() + .insert((distance, key)); + } + self.fetch_parked.insert( + key, + ParkedFetch { + distance, + wake_sources, + parked_at: Instant::now(), + }, + ); + } + + /// The per-source scheduler's state for its periodic summary, resetting + /// the maintenance maximum and source-health counters it reports and + /// purging the health table. Walks the parked set, the index's per-source + /// sets, the health table and the in-flight fetches once each. + pub(crate) fn take_fetch_dispatch_stats( + &mut self, + per_source_cap: usize, + ) -> FetchDispatchStats { + let now = Instant::now(); + let parked_maintenance_max = std::mem::take(&mut self.parked_maintenance_max); + let (health, tracked_sources) = + self.source_health.take_counters(now, &self.in_flight_fetch); + FetchDispatchStats { + tracked_sources, + silent_sources: self.source_health.marks_by_since.len(), + overflow_active: self.source_health.overflowing(now), + health, + parked_associations: self.parked_by_source.values().map(BTreeSet::len).sum(), + parked_maintenance_max, + parked: self.fetch_parked.len(), + oldest_parked: self + .fetch_parked + .values() + .map(|parked| now.saturating_duration_since(parked.parked_at)) + .max(), + queued: self.fetch_payloads.len(), + in_flight: self.in_flight_fetch.len(), + saturated_sources: self + .in_flight_counts_by_source(None) + .values() + .filter(|count| **count >= per_source_cap) + .count(), + } + } + + /// The parked-state part of [`Self::remove_hint_source`]: orphaned keys + /// leave it entirely, and a parked key can no longer start from the + /// departed source. One left with no source to wake it goes back on the + /// heap, where dispatch hands it out as sourceless, exactly as a queued key + /// that lost its last source. Timed for the dispatch summary. + fn drop_departed_source_from_parked( + &mut self, + source: &PeerId, + orphaned_fetch_keys: &HashSet, + ) { + if self.fetch_parked.is_empty() && self.fetch_tried.is_empty() { + return; + } + let started = Instant::now(); + for key in orphaned_fetch_keys { + self.unpark(key); + self.fetch_tried.remove(key); + } + if let Some(parked_here) = self.parked_by_source.remove(source) { + for (distance, key) in parked_here { + let Some(parked) = self.fetch_parked.get_mut(&key) else { + continue; + }; + parked.wake_sources.retain(|peer| peer != source); + if parked.wake_sources.is_empty() { + self.fetch_parked.remove(&key); + self.fetch_queue.push(FetchOrder { key, distance }); + } + } + } + self.parked_maintenance_max = self.parked_maintenance_max.max(started.elapsed()); + } + + /// Take `key` out of the parked state and the index, returning its + /// distance. The payload is left in `fetch_payloads`. + fn unpark(&mut self, key: &XorName) -> Option { + let parked = self.fetch_parked.remove(key)?; + for source in &parked.wake_sources { + if let Some(keys) = self.parked_by_source.get_mut(source) { + keys.remove(&(parked.distance, *key)); + if keys.is_empty() { + self.parked_by_source.remove(source); + } + } + } + Some(parked.distance) + } + + /// The nearest parked key, as `(distance, key)`. With `startable`, only + /// keys indexed under a source below the cap or silent count, so the + /// result is the nearest parked key that could start now. + fn nearest_parked(&self, startable: Option>) -> Option<(XorName, XorName)> { + self.parked_by_source + .iter() + .filter(|(source, _)| { + startable.is_none_or(|startable| { + startable.in_flight.get(source).copied().unwrap_or(0) < startable.per_source_cap + || self.source_health.is_silent(source, startable.now) + }) + }) + .filter_map(|(_, keys)| keys.first().copied()) + .min() + } + + /// Defensive: drop every index entry for `key` when the index names a key + /// that is not parked. Unreachable while the index stays exact; without it + /// such an entry would be offered forever. + fn purge_parked_index_entry(&mut self, key: &XorName) { + debug_assert!( + false, + "parked index names an unparked key {}", + hex::encode(key) + ); + self.parked_by_source.retain(|_, keys| { + keys.retain(|(_, parked)| parked != key); + !keys.is_empty() + }); + } + + /// Test-only: every queued or parked key, and every in-flight fetch with + /// its source, read together. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn fetch_pipeline_snapshot(&self) -> FetchPipelineSnapshot { + FetchPipelineSnapshot { + queued: self.fetch_payloads.keys().copied().collect(), + in_flight: self + .in_flight_fetch + .iter() + .map(|(key, entry)| (*key, entry.source)) + .collect(), + } + } + + /// Consume a dequeued candidate and restore its verification entry for a + /// later retry when retry metadata exists. + pub fn requeue_candidate_for_verification( + &mut self, + candidate: FetchCandidate, + retry_after: Duration, + ) -> bool { + let FetchCandidate { + key, + retry_verification, + .. + } = candidate; + self.fetch_tried.remove(&key); + let Some(mut verification) = retry_verification else { + return false; + }; + verification.state = VerificationState::PendingVerify; + verification.verified_sources.clear(); + verification.tried_sources.clear(); + verification.next_verify_at = Instant::now() + retry_after; + + let owner = self + .capacity_owner_by_key + .get(&key) + .copied() + .or_else(|| self.least_loaded_source(&verification.hint_sources)); + let Some(owner) = owner else { + self.release_retry_slot(&key); + return false; + }; + // Release before re-inserting: the reservation is charged to the owner + // recorded now, which `insert_pending_owned_unchecked` may replace. + self.release_retry_slot(&key); + self.insert_pending_owned_unchecked(key, verification, owner, true); + true + } + + /// Complete an exhausted fetch and restore its verification entry for a + /// later retry when retry metadata exists. + pub fn requeue_fetch_for_verification(&mut self, key: &XorName, retry_after: Duration) -> bool { + let Some(mut entry) = self.in_flight_fetch.remove(key) else { + return false; + }; + let Some(mut verification) = entry.retry_verification.take() else { + self.capacity_owner_by_key.remove(key); + return false; + }; + verification.state = VerificationState::PendingVerify; + verification.verified_sources.clear(); + verification.tried_sources.clear(); + verification.next_verify_at = Instant::now() + retry_after; + + let owner = self + .capacity_owner_by_key + .get(key) + .copied() + .or_else(|| self.least_loaded_source(&verification.hint_sources)); + let Some(owner) = owner else { + self.release_retry_slot(key); + return false; + }; + // Release before re-inserting: the reservation is charged to the owner + // recorded now, which `insert_pending_owned_unchecked` may replace. + self.release_retry_slot(key); self.insert_pending_owned_unchecked(*key, verification, owner, true); true } + /// When `key`'s current fetch attempt started, if it is in flight. + pub(crate) fn in_flight_started_at(&self, key: &XorName) -> Option { + self.in_flight_fetch.get(key).map(|entry| entry.started_at) + } + /// Number of in-flight fetches. #[must_use] pub fn in_flight_count(&self) -> usize { @@ -1516,6 +2626,8 @@ fn max_min_allocations( mod tests { use super::*; use crate::replication::config::CAPACITY_BLOCKED_RETRY; + use rand::rngs::StdRng; + use rand::{Rng, SeedableRng}; /// Build a `PeerId` from a single byte (zero-padded to 32 bytes). fn peer_id_from_byte(b: u8) -> PeerId { @@ -3026,4 +4138,1549 @@ mod tests { ); assert_eq!(queues.pending_count(), 1); } + + // -- per-source fetch dispatch --------------------------------------------- + + const CAP: usize = 2; + + fn peer(b: u8) -> PeerId { + peer_id_from_byte(b) + } + + /// A key whose byte order matches the numeric order of `distance`. + fn key_at(distance: u32) -> XorName { + let mut name = [0u8; 32]; + name[..4].copy_from_slice(&distance.to_be_bytes()); + name + } + + /// Queue `key` at `distance` with `sources`, keyed at its own distance so + /// tests can reason about nearest-first order directly. + fn queue(queues: &mut ReplicationQueues, distance: u32, sources: &[PeerId]) -> XorName { + let key = key_at(distance); + assert!(queues.enqueue_fetch(key, key, sources.to_vec())); + key + } + + /// Queue `key` through pending verification and promotion, so it carries a + /// verification retry reservation owned by `hinter`. + fn queue_promoted( + queues: &mut ReplicationQueues, + distance: u32, + hinter: u8, + sources: &[PeerId], + ) -> XorName { + let key = key_at(distance); + assert!(queues + .add_pending_verify(key, test_entry(hinter)) + .admitted()); + assert!(queues.promote_pending_to_fetch(key, key, sources.to_vec())); + key + } + + fn in_flight_to(queues: &ReplicationQueues, source: &PeerId) -> usize { + queues + .in_flight_fetch + .values() + .filter(|entry| entry.source == *source) + .count() + } + + fn heap_keys(queues: &ReplicationQueues) -> HashSet { + queues.fetch_queue.iter().map(|order| order.key).collect() + } + + /// I1 and I2: every queued payload is on the heap or parked, never both, + /// and the reverse index is exactly what `fetch_parked` implies. + fn assert_queue_invariants(queues: &ReplicationQueues) { + let heap = heap_keys(queues); + assert_eq!(heap.len(), queues.fetch_queue.len(), "duplicate heap entry"); + for key in queues.fetch_payloads.keys() { + let on_heap = heap.contains(key); + let parked = queues.fetch_parked.contains_key(key); + assert!( + on_heap != parked, + "queued key must be on exactly one of heap/parked" + ); + } + for key in heap.iter().chain(queues.fetch_parked.keys()) { + assert!( + queues.fetch_payloads.contains_key(key), + "order without payload" + ); + } + let mut expected: HashMap> = HashMap::new(); + for (key, parked) in &queues.fetch_parked { + assert!( + !parked.wake_sources.is_empty(), + "parked key with nothing to wake it" + ); + for source in &parked.wake_sources { + expected + .entry(*source) + .or_default() + .insert((parked.distance, *key)); + } + } + assert_eq!( + queues.parked_by_source, expected, + "parked index is not exact" + ); + for key in queues.fetch_tried.keys() { + assert!( + queues.fetch_payloads.contains_key(key), + "tried sources kept for a key that is not queued" + ); + } + } + + #[test] + fn dispatch_spreads_fetches_across_sources_within_the_cap() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let keys: Vec = (0..8).map(|d| queue(&mut queues, d, &[a, b, c])).collect(); + + let dispatch = queues.dispatch_fetches(8, CAP); + + assert_eq!(dispatch.started.len(), 6, "two per source, three sources"); + assert!(dispatch.sourceless.is_empty()); + for source in [a, b, c] { + assert_eq!(in_flight_to(&queues, &source), CAP); + } + // Preference order is kept: the nearest keys take the first source. + let expected: Vec<(XorName, PeerId)> = keys + .iter() + .take(6) + .zip([a, a, b, b, c, c]) + .map(|(key, source)| (*key, source)) + .collect(); + assert_eq!(dispatch.started, expected); + assert_eq!(queues.fetch_parked.len(), 2); + assert_queue_invariants(&queues); + } + + #[test] + fn uncapped_dispatch_keeps_the_first_source_order() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + for d in 0..5 { + queue(&mut queues, d, &[a, b]); + } + + let dispatch = queues.dispatch_fetches(8, usize::MAX); + + assert_eq!(dispatch.started.len(), 5); + assert!(dispatch.started.iter().all(|(_, source)| *source == a)); + assert!(queues.fetch_parked.is_empty()); + assert_queue_invariants(&queues); + } + + #[test] + fn a_startable_key_behind_blocked_keys_starts_in_the_same_pass() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + queue(&mut queues, 0, &[a]); + queue(&mut queues, 1, &[a]); + assert_eq!(queues.dispatch_fetches(8, CAP).started.len(), 2); + for d in 10..310 { + queue(&mut queues, d, &[a]); + } + let deeper = queue(&mut queues, 1_000, &[b]); + + let dispatch = queues.dispatch_fetches(8, CAP); + + assert_eq!(dispatch.started, vec![(deeper, b)]); + assert_eq!( + queues.fetch_parked.len(), + 300, + "every blocked key left the heap" + ); + assert!(queues.fetch_queue.is_empty()); + assert_eq!( + queues.fetch_queue_count(), + 300, + "parked keys are still queued" + ); + assert_queue_invariants(&queues); + } + + #[test] + fn a_freed_share_starts_the_nearest_parked_key_for_that_source() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + let first = queue(&mut queues, 0, &[a]); + queue(&mut queues, 1, &[a]); + for d in [40, 20, 30] { + queue(&mut queues, d, &[a]); + } + assert_eq!(queues.dispatch_fetches(8, CAP).started.len(), 2); + assert_eq!(queues.fetch_parked.len(), 3); + + queues.complete_fetch(&first); + let dispatch = queues.dispatch_fetches(8, CAP); + + assert_eq!(dispatch.started, vec![(key_at(20), a)]); + assert_eq!(in_flight_to(&queues, &a), CAP); + assert_queue_invariants(&queues); + } + + #[test] + fn the_nearest_startable_parked_key_wins_whatever_its_source() { + let mut queues = ReplicationQueues::new(); + let sources: Vec = (1..=5).map(peer).collect(); + // Saturate every source with two blockers, then park one key per source. + let mut blockers = Vec::new(); + for (index, source) in sources.iter().enumerate() { + let base = u32::try_from(index).unwrap() * 2; + blockers.push(queue(&mut queues, base, &[*source])); + blockers.push(queue(&mut queues, base + 1, &[*source])); + } + assert_eq!(queues.dispatch_fetches(10, CAP).started.len(), 10); + let parked_distances = [500, 100, 400, 200, 300]; + for (source, distance) in sources.iter().zip(parked_distances) { + queue(&mut queues, distance, &[*source]); + } + assert_eq!(queues.dispatch_fetches(5, CAP).started, Vec::new()); + assert_eq!(queues.fetch_parked.len(), 5); + + // Free one share on every source: the two slots go to the two nearest. + for pair in blockers.chunks(2) { + queues.complete_fetch(&pair[0]); + } + let dispatch = queues.dispatch_fetches(2, CAP); + + assert_eq!( + dispatch.started, + vec![(key_at(100), sources[1]), (key_at(200), sources[3])] + ); + assert_queue_invariants(&queues); + } + + #[test] + fn a_parked_key_starts_on_its_preferred_free_source() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + let blockers: Vec = (0..4) + .map(|d| queue(&mut queues, d, &[if d < 2 { a } else { b }])) + .collect(); + assert_eq!(queues.dispatch_fetches(4, CAP).started.len(), 4); + let key = queue(&mut queues, 50, &[a, b]); + assert_eq!(queues.dispatch_fetches(1, CAP).started, Vec::new()); + + // Free b first, then a: both have room when dispatch runs. + queues.complete_fetch(&blockers[2]); + queues.complete_fetch(&blockers[0]); + let dispatch = queues.dispatch_fetches(1, CAP); + + assert_eq!(dispatch.started, vec![(key, a)]); + assert_queue_invariants(&queues); + } + + #[test] + fn retry_moves_to_a_free_untried_source() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + queue(&mut queues, 10, &[b]); + queue(&mut queues, 11, &[b]); + let key = queue(&mut queues, 0, &[a, b, c]); + assert_eq!(queues.dispatch_fetches(3, CAP).started.len(), 3); + assert_eq!(queues.in_flight_fetch[&key].source, a); + + let retry = queues.retry_or_park_fetch(&key, key, CAP); + + assert_eq!(retry, FetchRetry::Next(c), "b is at its share"); + let entry = &queues.in_flight_fetch[&key]; + assert_eq!(entry.source, c); + assert_eq!(entry.tried, HashSet::from([a, c])); + assert_queue_invariants(&queues); + } + + #[test] + fn retry_with_every_untried_source_full_parks_and_keeps_its_history() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = queue_promoted(&mut queues, 0, 9, &[a, b, c]); + let blockers: Vec = (10..14) + .map(|d| queue(&mut queues, d, &[if d < 12 { b } else { c }])) + .collect(); + assert_eq!(queues.dispatch_fetches(5, CAP).started.len(), 5); + assert_eq!(queues.in_flight_fetch[&key].source, a); + let reserved = queues.retry_reserved_slot_count(); + let owner = queues.capacity_owner_by_key.get(&key).copied(); + let reserved_by_owner = queues.retry_reserved_by_owner.clone(); + assert_eq!(reserved_by_owner.get(&peer(9)), Some(&1)); + + let retry = queues.retry_or_park_fetch(&key, key, CAP); + + assert_eq!(retry, FetchRetry::Parked); + assert!(!queues.in_flight_fetch.contains_key(&key)); + assert_eq!(queues.fetch_payloads[&key].sources, vec![b, c]); + assert!(queues.fetch_payloads[&key].retry_verification.is_some()); + assert_eq!(queues.fetch_tried[&key], HashSet::from([a])); + assert_eq!(queues.fetch_parked[&key].wake_sources, vec![b, c]); + assert_eq!(queues.retry_reserved_slot_count(), reserved); + assert_eq!(queues.capacity_owner_by_key.get(&key).copied(), owner); + assert_eq!(queues.retry_reserved_by_owner, reserved_by_owner); + assert!(queues.contains_key(&key)); + assert_queue_invariants(&queues); + + // A share on b frees: the key restarts there, still remembering a. + queues.complete_fetch(&blockers[0]); + let dispatch = queues.dispatch_fetches(1, CAP); + assert_eq!(dispatch.started, vec![(key, b)]); + assert_eq!(queues.in_flight_fetch[&key].tried, HashSet::from([a, b])); + assert!(!queues.fetch_tried.contains_key(&key)); + assert_eq!(queues.retry_reserved_slot_count(), reserved); + assert_eq!(queues.capacity_owner_by_key.get(&key).copied(), owner); + assert_eq!(queues.retry_reserved_by_owner, reserved_by_owner); + assert_queue_invariants(&queues); + } + + #[test] + fn a_parked_key_wakes_through_any_of_its_sources_beyond_the_eighth() { + let mut queues = ReplicationQueues::new(); + let sources: Vec = (1..=9).map(peer).collect(); + let mut blockers = Vec::new(); + for (index, source) in sources.iter().enumerate() { + let base = u32::try_from(index).unwrap() * 2; + blockers.push(queue(&mut queues, base, &[*source])); + blockers.push(queue(&mut queues, base + 1, &[*source])); + } + assert_eq!(queues.dispatch_fetches(18, CAP).started.len(), 18); + let key = queue(&mut queues, 100, &sources); + assert_eq!(queues.dispatch_fetches(1, CAP).started, Vec::new()); + assert_eq!(queues.fetch_parked[&key].wake_sources, sources); + + // Only the ninth source frees a share. + queues.complete_fetch(&blockers[16]); + let dispatch = queues.dispatch_fetches(1, CAP); + + assert_eq!(dispatch.started, vec![(key, sources[8])]); + assert_queue_invariants(&queues); + } + + #[test] + fn retry_with_no_untried_source_is_exhausted_and_left_in_flight() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + let key = queue(&mut queues, 0, &[a]); + assert_eq!(queues.dispatch_fetches(1, CAP).started.len(), 1); + + assert_eq!( + queues.retry_or_park_fetch(&key, key, CAP), + FetchRetry::Exhausted + ); + assert!(queues.in_flight_fetch.contains_key(&key)); + assert_queue_invariants(&queues); + } + + #[test] + fn retry_with_a_full_fetch_queue_is_exhausted_rather_than_overfilling_it() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + let key = queue(&mut queues, 0, &[a, b]); + queue(&mut queues, 1, &[b]); + queue(&mut queues, 2, &[b]); + assert_eq!(queues.dispatch_fetches(3, CAP).started.len(), 3); + let mut filler = 1_000_u32; + while queues.fetch_queue_count() < MAX_FETCH_QUEUE { + queue(&mut queues, filler, &[peer(7)]); + filler += 1; + } + + assert_eq!( + queues.retry_or_park_fetch(&key, key, CAP), + FetchRetry::Exhausted + ); + assert!(queues.in_flight_fetch.contains_key(&key)); + assert_eq!(queues.fetch_queue_count(), MAX_FETCH_QUEUE); + } + + /// Park `key` (sources a, b, c) with a already tried and b, c full. + fn park_after_trying_a( + queues: &mut ReplicationQueues, + a: PeerId, + b: PeerId, + c: PeerId, + ) -> XorName { + let key = queue(queues, 0, &[a, b, c]); + for d in 10..14 { + queue(queues, d, &[if d < 12 { b } else { c }]); + } + assert_eq!(queues.dispatch_fetches(5, CAP).started.len(), 5); + assert_eq!( + queues.retry_or_park_fetch(&key, key, CAP), + FetchRetry::Parked + ); + key + } + + #[test] + fn a_tried_source_readvertised_for_a_parked_key_is_not_merged_back() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + + let result = queues.add_pending_verify(key, test_entry(1)); + + assert_eq!(result, AdmissionResult::AlreadyPresent); + assert_eq!(queues.fetch_payloads[&key].sources, vec![b, c]); + assert_eq!(queues.fetch_parked[&key].wake_sources, vec![b, c]); + assert_queue_invariants(&queues); + } + + #[test] + fn a_new_source_merged_into_a_parked_key_can_start_it() { + let mut queues = ReplicationQueues::new(); + let (a, b, c, d) = (peer(1), peer(2), peer(3), peer(4)); + let key = park_after_trying_a(&mut queues, a, b, c); + + assert_eq!( + queues.add_pending_verify(key, test_entry(4)), + AdmissionResult::AlreadyPresent + ); + assert_eq!(queues.fetch_parked[&key].wake_sources, vec![b, c, d]); + assert_queue_invariants(&queues); + + let dispatch = queues.dispatch_fetches(1, CAP); + assert_eq!(dispatch.started, vec![(key, d)]); + assert_eq!(queues.in_flight_fetch[&key].tried, HashSet::from([a, d])); + assert_queue_invariants(&queues); + } + + #[test] + fn an_in_flight_merge_skips_tried_sources_and_respects_the_limit() { + let mut queues = ReplicationQueues::new().with_fetch_source_limit(4); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + // Restart it on b, carrying a as tried: associated sources are a, b, c. + let blocker = queues + .in_flight_fetch + .iter() + .find(|(_, entry)| entry.source == b) + .map(|(key, _)| *key) + .unwrap(); + queues.complete_fetch(&blocker); + assert_eq!(queues.dispatch_fetches(1, CAP).started, vec![(key, b)]); + + // a was tried: not merged. 5 is new and fits (4 associated). 6 does not. + for hinter in [1, 5, 6] { + queues.add_pending_verify(key, test_entry(hinter)); + } + + assert_eq!( + queues.in_flight_fetch[&key].all_sources, + vec![b, c, peer(5)] + ); + assert_queue_invariants(&queues); + } + + /// A re-advertisement from `sender` that claims no replica. + fn paid_only_entry(sender: u8) -> VerificationEntry { + let mut entry = test_entry(sender); + entry.replica_hint_sources.clear(); + entry + } + + /// Every fetch-stage key's retry entry is within the hint bound, as a + /// pending entry is, and still holds the key's capacity owner. + fn assert_retry_entries_bounded(queues: &ReplicationQueues) { + let retries = queues + .fetch_payloads + .iter() + .filter_map(|(key, payload)| Some((key, payload.retry_verification.as_ref()?))) + .chain( + queues + .in_flight_fetch + .iter() + .filter_map(|(key, entry)| Some((key, entry.retry_verification.as_ref()?))), + ); + for (key, retry) in retries { + assert!( + retry.hint_sources.len() <= MAX_HINT_SOURCES_PER_KEY, + "{} retry entry over the hint bound", + hex::encode(key) + ); + assert!(retry.replica_hint_sources.is_subset(&retry.hint_sources)); + if let Some(owner) = queues.capacity_owner_by_key.get(key) { + assert!( + retry.hint_sources.contains(owner), + "{} retry entry lost its capacity owner", + hex::encode(key) + ); + } + } + } + + #[test] + fn readvertisements_keep_a_fetch_stage_retry_entry_within_the_hint_bound() { + let mut queues = ReplicationQueues::new(); + let (owner, holder) = (peer(1), peer(2)); + let key = key_at(0); + // The owner advertised the key without claiming a replica, so only + // the capacity accounting keeps it in the set. + assert!(queues + .add_pending_verify(key, paid_only_entry(1)) + .admitted()); + assert!(queues.promote_pending_to_fetch(key, key, vec![holder])); + + // Queued: paid-only advertisers fill the set to the bound, then replica + // claimants displace every one of them except the owner. + for sender in 20..40 { + queues.add_pending_verify(key, paid_only_entry(sender)); + } + for sender in 40..60 { + queues.add_pending_verify(key, test_entry(sender)); + } + assert_retry_entries_bounded(&queues); + let retry_sizes = queues + .fetch_payloads + .get(&key) + .and_then(|payload| payload.retry_verification.as_ref()) + .map(|retry| (retry.hint_sources.len(), retry.replica_hint_sources.len())); + assert_eq!( + retry_sizes, + Some((MAX_HINT_SOURCES_PER_KEY, MAX_HINT_SOURCES_PER_KEY - 1)), + "a full set keeps the owner and fills the rest with replica claimants" + ); + + // In flight: the same bound. + assert_eq!(queues.dispatch_fetches(1, CAP).started, vec![(key, holder)]); + for sender in 60..80 { + queues.add_pending_verify(key, test_entry(sender)); + } + assert_retry_entries_bounded(&queues); + + // Back in pending verification, the restored entry keeps the bound and + // the owner its reservation was charged to. + assert!(queues.requeue_fetch_for_verification(&key, Duration::ZERO)); + let pending = queues.get_pending(&key).map(|entry| { + ( + entry.hint_sources.len(), + entry.hint_sources.contains(&owner), + ) + }); + assert_eq!(pending, Some((MAX_HINT_SOURCES_PER_KEY, true))); + assert_eq!(queues.capacity_owner_by_key.get(&key), Some(&owner)); + } + + #[test] + fn source_lists_are_capped_at_the_limit_including_tried_sources() { + let mut queues = ReplicationQueues::new().with_fetch_source_limit(3); + let sources: Vec = (1..=6).map(peer).collect(); + let key = queue(&mut queues, 0, &sources); + assert_eq!(queues.fetch_payloads[&key].sources, sources[..3].to_vec()); + + // Park it with one tried source: payload b, c plus tried a is the limit. + let mut queues = ReplicationQueues::new().with_fetch_source_limit(3); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + queues.add_pending_verify(key, test_entry(4)); + assert_eq!(queues.fetch_payloads[&key].sources, vec![b, c]); + assert_queue_invariants(&queues); + + // A direct start with an oversized list is cut to the limit too. + let mut queues = ReplicationQueues::new().with_fetch_source_limit(2); + let direct = key_at(9); + queues.start_fetch(direct, sources[0], sources.clone()); + assert_eq!( + queues.in_flight_fetch[&direct].all_sources, + sources[..2].to_vec() + ); + + // A start from a source outside the list counts that source too. + let mut queues = ReplicationQueues::new().with_fetch_source_limit(3); + let outside = peer(9); + queues.start_fetch(direct, outside, sources[..4].to_vec()); + let entry = &queues.in_flight_fetch[&direct]; + assert_eq!(entry.all_sources, sources[..2].to_vec()); + let union: HashSet<&PeerId> = entry.all_sources.iter().chain(&entry.tried).collect(); + assert_eq!(union.len(), 3); + } + + #[test] + fn a_parked_key_that_loses_its_last_wake_source_returns_to_the_heap() { + let mut queues = ReplicationQueues::new(); + let (b, c) = (peer(2), peer(3)); + queue(&mut queues, 0, &[b]); + queue(&mut queues, 1, &[b]); + queue(&mut queues, 2, &[c]); + // Hinted by 9, so the retry metadata survives b leaving. + let key = queue_promoted(&mut queues, 10, 9, &[b]); + let started = queues.dispatch_fetches(4, CAP).started; + assert!(started.iter().all(|(started_key, _)| *started_key != key)); + assert!(queues.fetch_parked.contains_key(&key)); + + let orphaned = queues.remove_hint_source(&b); + + assert!(!orphaned.contains(&key)); + assert!(!queues.fetch_parked.contains_key(&key)); + assert!(heap_keys(&queues).contains(&key)); + assert_queue_invariants(&queues); + let dispatch = queues.dispatch_fetches(4, CAP); + assert!(dispatch + .sourceless + .iter() + .any(|candidate| candidate.key == key)); + } + + #[test] + fn an_orphaned_parked_key_leaves_every_structure() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + queues.remove_hint_source(&b); + assert!( + queues.fetch_parked.contains_key(&key), + "c can still wake it" + ); + + let orphaned = queues.remove_hint_source(&c); + + assert!(orphaned.contains(&key)); + assert!(!queues.fetch_payloads.contains_key(&key)); + assert!(!queues.fetch_parked.contains_key(&key)); + assert!(!queues.fetch_tried.contains_key(&key)); + assert!(!queues.contains_key(&key)); + assert_queue_invariants(&queues); + } + + #[test] + fn tried_sources_survive_dequeue_and_move_into_the_next_start() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + + let candidate = queues.dequeue_fetch().unwrap(); + assert_eq!( + candidate.key, key, + "the parked key is the nearest queued key" + ); + assert!(queues.fetch_tried.contains_key(&key)); + queues.start_dequeued_fetch(candidate, c); + + assert_eq!(queues.in_flight_fetch[&key].tried, HashSet::from([a, c])); + assert!(!queues.fetch_tried.contains_key(&key)); + assert_queue_invariants(&queues); + } + + #[test] + fn tried_sources_are_dropped_with_a_discarded_or_requeued_candidate() { + for requeue in [false, true] { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + let candidate = queues.dequeue_fetch().unwrap(); + assert_eq!(candidate.key, key); + if requeue { + queues.requeue_candidate_for_verification(candidate, Duration::from_secs(1)); + } else { + queues.discard_fetch_candidate(candidate); + } + assert!(!queues.fetch_tried.contains_key(&key)); + assert_queue_invariants(&queues); + } + } + + #[test] + fn tried_sources_are_kept_by_a_refused_enqueue_and_cleared_by_a_fresh_one() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + assert!(!queues.enqueue_fetch(key, key, vec![a]), "already queued"); + assert_eq!(queues.fetch_tried[&key], HashSet::from([a])); + + let stray = key_at(77); + queues.fetch_tried.insert(stray, HashSet::from([a])); + assert!(queues.enqueue_fetch(stray, stray, vec![a])); + assert!(!queues.fetch_tried.contains_key(&stray)); + } + + #[test] + fn a_duplicate_of_an_in_flight_key_is_dropped_with_its_tried_sources() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + let key = key_at(3); + queues.start_fetch(key, a, vec![a]); + // Force the impossible state the defensive skip exists for. + queues.fetch_payloads.insert( + key, + FetchPayload { + sources: vec![a], + retry_verification: None, + }, + ); + queues.fetch_queue.push(FetchOrder { key, distance: key }); + queues.fetch_tried.insert(key, HashSet::from([a])); + + assert!(queues.dequeue_fetch().is_none()); + assert!(!queues.fetch_tried.contains_key(&key)); + assert!(queues.in_flight_fetch.contains_key(&key)); + } + + #[test] + fn dequeue_returns_a_parked_key_nearer_than_the_heap_head() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + queue(&mut queues, 0, &[a]); + queue(&mut queues, 1, &[a]); + let parked = queue(&mut queues, 5, &[a]); + assert_eq!(queues.dispatch_fetches(3, CAP).started.len(), 2); + let farther = queue(&mut queues, 9, &[peer(2)]); + + assert_eq!(queues.dequeue_fetch().map(|c| c.key), Some(parked)); + assert_eq!(queues.dequeue_fetch().map(|c| c.key), Some(farther)); + } + + #[test] + fn a_parked_key_holds_bootstrap_drain_open() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + queue(&mut queues, 0, &[a]); + queue(&mut queues, 1, &[a]); + let parked = queue(&mut queues, 2, &[a]); + queues.dispatch_fetches(3, CAP); + assert!(queues.fetch_parked.contains_key(&parked)); + + assert!(!queues.is_bootstrap_work_empty(&HashSet::from([parked]))); + } + + // -- silent sources --------------------------------------------------------- + + /// A fetch request timeout for the table tests: a mark lasts twice this. + const TEST_TIMEOUT: Duration = Duration::from_secs(10); + + /// A peer distinct from every `peer(b)` and every other index. + fn distinct_peer(index: u32) -> PeerId { + let mut bytes = [0xA5_u8; 32]; + bytes[..4].copy_from_slice(&index.to_be_bytes()); + PeerId::from_bytes(bytes) + } + + /// `base` moved forward by `millis`. + fn at(base: Instant, millis: u64) -> Instant { + base + Duration::from_millis(millis) + } + + fn unreachable(sent: u64) -> Contact { + Contact::Unreachable { sent } + } + + fn answered(order: u64) -> Contact { + Contact::Answered { order } + } + + /// Mark `source` silent from now with a request at sequence place `sent`. + fn silence(queues: &mut ReplicationQueues, source: PeerId, sent: u64) { + queues.record_fetch_contact(source, unreachable(sent)); + } + + fn in_flight_entry(key: XorName, source: PeerId, started_at: Instant) -> InFlightEntry { + InFlightEntry { + key, + source, + started_at, + all_sources: vec![source], + tried: HashSet::from([source]), + retry_verification: None, + } + } + + /// The health table's mark index matches its entries exactly, and only a + /// source with an unanswered request since its latest answer is marked. + fn assert_health_invariants(table: &SourceHealthTable) { + assert!(table.entries.len() <= MAX_TRACKED_FETCH_SOURCES); + let marked: BTreeSet<(Instant, PeerId)> = table + .entries + .iter() + .filter_map(|(source, health)| health.marked_since.map(|since| (since, *source))) + .collect(); + assert_eq!(table.marks_by_since, marked, "mark index is not exact"); + for health in table.entries.values() { + assert!( + health.marked_since.is_none() || health.unanswered_since_answer(), + "a mark outlived the evidence for it" + ); + } + } + + #[test] + fn an_unanswered_request_marks_its_source_until_the_mark_expires() { + let none = HashMap::new(); + let a = peer(1); + let t0 = Instant::now(); + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + + table.record(a, unreachable(1), at(t0, 100), &none); + assert!(table.is_silent(&a, at(t0, 100))); + assert!(table.is_silent(&a, at(t0, 20_099))); + assert!( + !table.is_silent(&a, at(t0, 20_100)), + "a mark lasts two fetch timeouts from when the failure was learned" + ); + + table.expire(at(t0, 20_100)); + assert_eq!(table.counters.expired_marks, 1); + assert!(table.marks_by_since.is_empty()); + + // An older unanswered request learned after the expiry is not new + // evidence: the source stays under the cap for its probe. + table.record(a, unreachable(0), at(t0, 20_200), &none); + assert!(!table.is_silent(&a, at(t0, 20_200))); + + // A request sent later that also goes unanswered marks it again. + table.record(a, unreachable(2), at(t0, 30_200), &none); + assert!(table.is_silent(&a, at(t0, 30_200))); + assert_eq!(table.counters.marks, 2); + assert_eq!(table.counters.unreachable, 3); + assert_health_invariants(&table); + } + + #[test] + fn a_source_that_keeps_failing_stays_silent_past_one_lifetime() { + let none = HashMap::new(); + let a = peer(1); + let t0 = Instant::now(); + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + + for (sent, millis) in [(1, 0), (2, 15_000), (3, 30_000)] { + table.record(a, unreachable(sent), at(t0, millis), &none); + } + + assert!(table.is_silent(&a, at(t0, 49_999))); + assert!(!table.is_silent(&a, at(t0, 50_000))); + assert_eq!( + table.counters.marks, 1, + "one mark, refreshed by newer failures" + ); + assert_eq!(table.marks_by_since.len(), 1); + assert_health_invariants(&table); + } + + #[test] + fn a_failure_learned_late_still_marks_for_a_whole_lifetime() { + let none = HashMap::new(); + let a = peer(1); + let learned = at(Instant::now(), 600_000); + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + + table.record(a, unreachable(1), learned, &none); + + assert!(table.is_silent(&a, learned)); + assert!(table.is_silent(&a, at(learned, 19_999))); + assert!(!table.is_silent(&a, at(learned, 20_000))); + } + + /// Every order of applying `events` to a fresh table, all learned at + /// `now`, so only the sequence places can decide. + fn apply_in_every_order(events: &[Contact], now: Instant) -> Vec { + let none = HashMap::new(); + let a = peer(1); + let mut orders: Vec> = vec![Vec::new()]; + for _ in events { + orders = orders + .into_iter() + .flat_map(|prefix| { + (0..events.len()) + .filter(|index| !prefix.contains(index)) + .map(|index| { + let mut next = prefix.clone(); + next.push(index); + next + }) + .collect::>() + }) + .collect(); + } + orders + .iter() + .map(|order| { + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + for event in order.iter().filter_map(|index| events.get(*index)) { + table.record(a, *event, now, &none); + } + assert_health_invariants(&table); + table + }) + .collect() + } + + #[test] + fn whether_a_source_is_silent_does_not_depend_on_the_order_outcomes_arrive() { + let a = peer(1); + let now = Instant::now(); + let lifetime_ms = + u64::try_from((TEST_TIMEOUT * SILENCE_LIFETIME_TIMEOUTS).as_millis()).unwrap(); + + // Unanswered at 1, an answer at 2, unanswered again at 3: silent, as + // the request sent after the answer went unanswered, for one lifetime. + let silent = [unreachable(1), answered(2), unreachable(3)]; + let tables = apply_in_every_order(&silent, now); + assert_eq!(tables.len(), 6); + for table in &tables { + assert!(table.is_silent(&a, now)); + assert!(table.is_silent(&a, at(now, lifetime_ms - 1))); + assert!(!table.is_silent(&a, at(now, lifetime_ms))); + } + + // Unanswered at 1 and 2, then an answer at 3: answering. + let answering = [unreachable(1), unreachable(2), answered(3)]; + for table in &apply_in_every_order(&answering, now) { + assert!(!table.is_silent(&a, now)); + } + } + + #[test] + fn an_answer_older_than_the_unanswered_request_neither_clears_nor_blocks_its_mark() { + let none = HashMap::new(); + let a = peer(1); + let t0 = Instant::now(); + + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + table.record(a, unreachable(5), t0, &none); + table.record(a, answered(4), t0, &none); + assert!(table.is_silent(&a, t0)); + assert_eq!(table.counters.clears, 0); + + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + table.record(a, answered(4), t0, &none); + table.record(a, unreachable(5), t0, &none); + assert!(table.is_silent(&a, t0)); + assert_eq!(table.counters.ignored_stale, 0); + + // A request sent before the latest answer is stale and cannot mark. + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + table.record(a, answered(6), t0, &none); + table.record(a, unreachable(5), t0, &none); + assert!(!table.is_silent(&a, t0)); + assert_eq!((table.counters.marks, table.counters.ignored_stale), (0, 1)); + assert_health_invariants(&table); + } + + #[test] + fn an_expired_mark_is_kept_only_while_its_source_has_a_fetch_in_flight() { + let t0 = Instant::now(); + let (guarded, idle) = (peer(1), peer(2)); + let key = key_at(0); + let in_flight = HashMap::from([(key, in_flight_entry(key, guarded, t0))]); + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + table.record(guarded, unreachable(1), t0, &in_flight); + table.record(idle, unreachable(2), t0, &in_flight); + + let later = at(t0, 20_000); + let (counters, tracked) = table.take_counters(later, &in_flight); + + assert_eq!((counters.marks, counters.expired_marks, tracked), (2, 2, 2)); + assert!(table.entries.contains_key(&guarded)); + assert!(!table.entries.contains_key(&idle)); + // The guarded source keeps its history: its older request is no new + // evidence, its newer one marks it again. + table.record(guarded, unreachable(0), later, &in_flight); + assert!(!table.is_silent(&guarded, later)); + table.record(guarded, unreachable(3), later, &in_flight); + assert!(table.is_silent(&guarded, later)); + assert_health_invariants(&table); + } + + #[test] + fn attempts_that_sent_nothing_or_were_cancelled_change_nothing() { + let none = HashMap::new(); + let a = peer(1); + let t0 = Instant::now(); + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + + table.record(a, Contact::NotAttempted, t0, &none); + table.record(a, Contact::Unknown, t0, &none); + + assert!(table.entries.is_empty()); + assert!(!table.is_silent(&a, t0)); + assert_eq!(table.counters.unreachable, 0); + } + + #[test] + fn a_zero_fetch_timeout_still_leaves_silent_sources_a_lifetime() { + let none = HashMap::new(); + let a = peer(1); + let t0 = Instant::now(); + let mut table = SourceHealthTable::new(Duration::ZERO); + + table.record(a, unreachable(1), t0, &none); + + assert!(table.is_silent(&a, t0)); + assert!(table.is_silent(&a, at(t0, 999))); + assert!(!table.is_silent(&a, at(t0, 1_000))); + } + + /// A table at its bound: `guarded` answered and has a fetch in flight, + /// `idle` answered and has none, and every other entry is marked. + fn full_table( + t0: Instant, + marked: usize, + ) -> ( + SourceHealthTable, + HashMap, + PeerId, + PeerId, + ) { + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + let guarded = distinct_peer(0); + let idle = distinct_peer(1); + let key = key_at(0); + let in_flight = HashMap::from([(key, in_flight_entry(key, guarded, t0))]); + table.record(guarded, answered(1), t0, &in_flight); + if marked < MAX_TRACKED_FETCH_SOURCES - 1 { + table.record(idle, answered(1), t0, &in_flight); + } + for index in 0..marked { + let source = distinct_peer(u32::try_from(index).unwrap() + 2); + table.record(source, unreachable(1), t0, &in_flight); + } + assert_eq!(table.entries.len(), MAX_TRACKED_FETCH_SOURCES); + (table, in_flight, guarded, idle) + } + + #[test] + fn a_full_table_makes_room_by_dropping_answers_that_guard_nothing() { + let t0 = Instant::now(); + let (mut table, in_flight, guarded, idle) = full_table(t0, MAX_TRACKED_FETCH_SOURCES - 2); + let newcomer = distinct_peer(u32::MAX); + + table.record(newcomer, unreachable(1), t0, &in_flight); + + assert!(table.is_silent(&newcomer, t0)); + assert!( + table.entries.contains_key(&guarded), + "an in-flight guard stays" + ); + assert!(!table.entries.contains_key(&idle)); + assert_eq!(table.counters.overflows, 0); + assert_health_invariants(&table); + } + + #[test] + fn a_table_full_of_marks_and_guards_counts_untracked_sources_as_silent() { + let t0 = Instant::now(); + let (mut table, in_flight, guarded, _) = full_table(t0, MAX_TRACKED_FETCH_SOURCES - 1); + let newcomer = distinct_peer(u32::MAX); + let bystander = distinct_peer(u32::MAX - 1); + + table.record(newcomer, unreachable(1), t0, &in_flight); + + assert_eq!(table.counters.overflows, 1); + assert!(!table.entries.contains_key(&newcomer)); + assert!( + table.is_silent(&newcomer, t0), + "an untracked source that stopped answering is fetched from as before the cap" + ); + assert!(table.is_silent(&bystander, t0)); + assert!( + table.entries.contains_key(&guarded), + "an in-flight guard is never dropped" + ); + assert!( + !table.is_silent(&guarded, t0), + "a tracked source keeps its own state" + ); + let lifetime = TEST_TIMEOUT * SILENCE_LIFETIME_TIMEOUTS; + assert!( + !table.is_silent(&bystander, t0 + lifetime), + "the overflow lasts one silence lifetime" + ); + assert_health_invariants(&table); + } + + #[test] + fn a_silent_source_is_fetched_from_past_the_cap() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + for d in 0..6 { + queue(&mut queues, d, &[a]); + } + assert_eq!(queues.dispatch_fetches(8, CAP).started.len(), CAP); + assert_eq!(queues.fetch_parked.len(), 4); + + silence(&mut queues, a, 1); + let dispatch = queues.dispatch_fetches(8, CAP); + + assert_eq!( + dispatch.started.len(), + 4, + "every parked key starts on the silent source" + ); + assert_eq!(in_flight_to(&queues, &a), 6); + assert!(queues.fetch_parked.is_empty()); + assert_eq!(queues.source_health.counters.silent_starts, 4); + assert_queue_invariants(&queues); + } + + #[test] + fn a_sole_silent_holder_drains_one_timeout_after_it_is_noticed() { + // Sixteen keys held only by `a`, eight fetch slots, and every request + // to `a` unanswered: one wave of two at the share, then the rest past + // it, eight per wave, as many waves as before the cap plus one. + let mut queues = ReplicationQueues::new(); + let a = peer(1); + for d in 0..16 { + queue(&mut queues, d, &[a]); + } + let mut sequence = 0; + let mut waves = Vec::new(); + loop { + let started = queues.dispatch_fetches(8, CAP).started; + if started.is_empty() { + break; + } + waves.push(started.len()); + // Every request in the wave times out. + for (key, source) in started { + sequence += 1; + queues.record_fetch_contact(source, unreachable(sequence)); + assert_eq!( + queues.retry_or_park_fetch(&key, key, CAP), + FetchRetry::Exhausted + ); + queues.complete_fetch(&key); + } + assert_queue_invariants(&queues); + } + assert_eq!(waves, vec![2, 8, 6]); + assert_eq!(queues.fetch_queue_count(), 0); + } + + #[test] + fn an_answering_source_with_room_is_preferred_over_a_silent_one() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + silence(&mut queues, a, 1); + let keys: Vec = (0..3).map(|d| queue(&mut queues, d, &[a, b])).collect(); + + let dispatch = queues.dispatch_fetches(8, CAP); + + let expected: Vec<(XorName, PeerId)> = keys.iter().copied().zip([b, b, a]).collect(); + assert_eq!( + dispatch.started, expected, + "b until its share is full, then the silent a rather than parking" + ); + assert!(queues.fetch_parked.is_empty()); + assert_queue_invariants(&queues); + } + + #[test] + fn retry_prefers_an_answering_source_with_room_then_a_silent_one() { + let (a, b, c) = (peer(1), peer(2), peer(3)); + for c_full in [false, true] { + let mut queues = ReplicationQueues::new(); + let key = queue(&mut queues, 0, &[a, b, c]); + if c_full { + queue(&mut queues, 10, &[c]); + queue(&mut queues, 11, &[c]); + } + let started = if c_full { 3 } else { 1 }; + assert_eq!(queues.dispatch_fetches(3, CAP).started.len(), started); + assert_eq!( + queues.in_flight_fetch.get(&key).map(|entry| entry.source), + Some(a) + ); + silence(&mut queues, b, 1); + + let retry = queues.retry_or_park_fetch(&key, key, CAP); + + let expected = if c_full { b } else { c }; + assert_eq!(retry, FetchRetry::Next(expected)); + assert_eq!( + queues.source_health.counters.silent_starts, + u64::from(c_full) + ); + assert_queue_invariants(&queues); + } + } + + /// Record that `source` just answered this node busy. + fn cool(queues: &ReplicationQueues, source: PeerId) { + queues + .busy_sources + .record_busy(&source, Instant::now(), Duration::from_mins(1)); + } + + #[test] + fn dispatch_puts_a_source_that_answered_busy_after_the_other_free_ones() { + let mut queues = ReplicationQueues::new(); + let (a, b, s) = (peer(1), peer(2), peer(3)); + silence(&mut queues, s, 1); + cool(&queues, a); + let keys: Vec = (0..5).map(|d| queue(&mut queues, d, &[a, b, s])).collect(); + + let dispatch = queues.dispatch_fetches(8, CAP); + + let expected: Vec<(XorName, PeerId)> = keys.iter().copied().zip([b, b, a, a, s]).collect(); + assert_eq!( + dispatch.started, expected, + "b until its share is full, then the busy a, never withheld, before the silent s" + ); + assert!(queues.fetch_parked.is_empty()); + assert_queue_invariants(&queues); + } + + #[test] + fn retry_puts_a_source_that_answered_busy_after_the_other_free_ones() { + let mut queues = ReplicationQueues::new(); + let (a, b, c, s) = (peer(1), peer(2), peer(3), peer(4)); + let key = queue(&mut queues, 0, &[a, b, c, s]); + assert_eq!(queues.dispatch_fetches(1, CAP).started, vec![(key, a)]); + silence(&mut queues, s, 1); + cool(&queues, b); + + assert_eq!( + queues.retry_or_park_fetch(&key, key, CAP), + FetchRetry::Next(c), + "the free c before the busy b" + ); + assert_eq!( + queues.retry_or_park_fetch(&key, key, CAP), + FetchRetry::Next(b), + "the busy b, never withheld, before the silent s" + ); + assert_eq!( + queues.retry_or_park_fetch(&key, key, CAP), + FetchRetry::Next(s) + ); + assert_eq!( + queues.retry_or_park_fetch(&key, key, CAP), + FetchRetry::Exhausted + ); + assert_queue_invariants(&queues); + } + + #[test] + fn parked_keys_start_on_a_source_that_falls_silent_while_the_others_stay_full() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + for d in 0..4 { + queue(&mut queues, d, &[if d < 2 { a } else { b }]); + } + assert_eq!(queues.dispatch_fetches(4, CAP).started.len(), 4); + let parked: Vec = (10..20).map(|d| queue(&mut queues, d, &[a, b])).collect(); + assert_eq!(queues.dispatch_fetches(8, CAP).started, Vec::new()); + assert_eq!(queues.fetch_parked.len(), 10); + + silence(&mut queues, b, 1); + let dispatch = queues.dispatch_fetches(8, CAP); + + // a stays full and answering: every slot goes to b, and the pass ends. + let expected: Vec<(XorName, PeerId)> = parked.iter().take(8).map(|key| (*key, b)).collect(); + assert_eq!(dispatch.started, expected); + assert_eq!(queues.fetch_parked.len(), 2); + assert_queue_invariants(&queues); + } + + #[test] + fn an_answer_puts_a_silent_source_back_under_the_cap() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + for d in 0..6 { + queue(&mut queues, d, &[a]); + } + silence(&mut queues, a, 1); + assert_eq!(queues.dispatch_fetches(3, CAP).started.len(), 3); + + queues.record_fetch_contact(a, answered(2)); + let dispatch = queues.dispatch_fetches(8, CAP); + + assert_eq!( + dispatch.started, + Vec::new(), + "a answers again and already has three fetches in flight" + ); + assert_eq!(queues.fetch_parked.len(), 3); + assert_queue_invariants(&queues); + } + + #[test] + fn a_departure_keeps_what_was_observed_of_the_source() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + let key = queue(&mut queues, 0, &[a]); + assert_eq!(queues.dispatch_fetches(1, CAP).started, vec![(key, a)]); + silence(&mut queues, a, 1); + + queues.remove_hint_source(&a); + + assert!(queues.source_health.is_silent(&a, Instant::now())); + // The attempt still in flight answers after the departure: applied + // like any other outcome. + queues.record_fetch_contact(a, answered(2)); + assert!(!queues.source_health.is_silent(&a, Instant::now())); + // A late unanswered request sent before that answer cannot mark it. + queues.record_fetch_contact(a, unreachable(1)); + assert!(!queues.source_health.is_silent(&a, Instant::now())); + assert_health_invariants(&queues.source_health); + } + + #[test] + fn the_summary_drops_answers_that_guard_nothing_and_keeps_marks() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = queue(&mut queues, 0, &[a]); + assert_eq!(queues.dispatch_fetches(1, CAP).started, vec![(key, a)]); + queues.record_fetch_contact(a, answered(1)); + queues.record_fetch_contact(b, answered(2)); + silence(&mut queues, c, 3); + + let stats = queues.take_fetch_dispatch_stats(CAP); + + assert_eq!(stats.tracked_sources, 3); + assert_eq!(stats.silent_sources, 1); + assert!(!stats.overflow_active); + assert_eq!(stats.health.marks, 1); + assert_eq!(stats.health.silent_peak, 1); + let kept: HashSet = queues.source_health.entries.keys().copied().collect(); + assert_eq!( + kept, + HashSet::from([a, c]), + "a guards a fetch in flight and c is marked" + ); + assert_eq!(queues.source_health.counters.marks, 0, "counters restart"); + assert_eq!( + queues.source_health.counters.silent_peak, 1, + "the peak restarts at the sources silent now" + ); + assert_health_invariants(&queues.source_health); + } + + #[test] + #[allow(clippy::too_many_lines)] + fn queue_invariants_hold_under_random_operations() { + let mut rng = StdRng::seed_from_u64(0x1514); + let mut queues = ReplicationQueues::new().with_fetch_source_limit(5); + let sources: Vec = (1..=6).map(peer).collect(); + let mut sequence = 0_u64; + for step in 0..4_000_u32 { + match rng.gen_range(0..10) { + 0 | 1 => { + let count = rng.gen_range(1..=4); + let mut chosen: Vec = Vec::new(); + while chosen.len() < count { + let source = sources[rng.gen_range(0..sources.len())]; + if !chosen.contains(&source) { + chosen.push(source); + } + } + let key = key_at(rng.gen_range(0..400)); + if rng.gen_bool(0.25) { + // Through verification, so it carries a retry entry. + let hinter = rng.gen_range(1..=30); + if queues + .add_pending_verify(key, test_entry(hinter)) + .admitted() + { + queues.promote_pending_to_fetch(key, key, chosen); + } + } else { + queues.enqueue_fetch(key, key, chosen); + } + } + 2 => { + // As the fetch worker does, a candidate with no untried + // source goes back to verification. + let dispatch = queues.dispatch_fetches(rng.gen_range(0..6), CAP); + for candidate in dispatch.sourceless { + queues.requeue_candidate_for_verification(candidate, Duration::ZERO); + } + } + 3 => { + if let Some(key) = queues.in_flight_fetch.keys().min().copied() { + queues.complete_fetch(&key); + } + } + 4 => { + let mut in_flight: Vec = + queues.in_flight_fetch.keys().copied().collect(); + in_flight.sort_unstable(); + if !in_flight.is_empty() { + let key = in_flight[rng.gen_range(0..in_flight.len())]; + if queues.retry_or_park_fetch(&key, key, CAP) == FetchRetry::Exhausted { + queues.complete_fetch(&key); + } + } + } + 5 => { + let mut waiting: Vec = queues.fetch_payloads.keys().copied().collect(); + waiting.sort_unstable(); + if !waiting.is_empty() { + let key = waiting[rng.gen_range(0..waiting.len())]; + let hinter = rng.gen_range(1..=30); + let entry = if rng.gen_bool(0.5) { + test_entry(hinter) + } else { + paid_only_entry(hinter) + }; + queues.add_pending_verify(key, entry); + } + } + 6 => { + if rng.gen_range(0..10) == 0 { + queues.remove_hint_source(&sources[rng.gen_range(0..sources.len())]); + } + } + 7 => { + if let Some(candidate) = queues.dequeue_fetch() { + match candidate.sources.first().copied() { + Some(source) if rng.gen_bool(0.5) => { + queues.start_dequeued_fetch(candidate, source); + } + _ => queues.discard_fetch_candidate(candidate), + } + } + } + _ => { + // A source answers, or a request to it goes unanswered, + // applied a little out of sequence order. + if let Some(source) = sources.get(rng.gen_range(0..sources.len())).copied() { + sequence += 1; + let place = sequence.saturating_sub(rng.gen_range(0..4)); + let contact = if rng.gen_bool(0.5) { + Contact::Answered { order: place } + } else { + Contact::Unreachable { sent: place } + }; + queues.record_fetch_contact(source, contact); + } + } + } + assert_queue_invariants(&queues); + assert_health_invariants(&queues.source_health); + assert_retry_entries_bounded(&queues); + for (key, entry) in &queues.in_flight_fetch { + let associated: HashSet<&PeerId> = + entry.all_sources.iter().chain(entry.tried.iter()).collect(); + assert!( + associated.len() <= 5, + "step {step}: {} over the source limit", + hex::encode(key) + ); + } + for (key, payload) in &queues.fetch_payloads { + let tried = queues.fetch_tried.get(key).map_or(0, HashSet::len); + assert!( + payload.sources.len() + tried <= 5, + "step {step}: queued over the limit" + ); + } + } + } + + /// Scale benchmark for the per-source scheduler at the fetch queue's hard + /// bound. Runs only when `ANT_FETCH_SCALE_BENCH` names a sources-per-key + /// count (it allocates hundreds of MiB and takes seconds); otherwise it + /// returns at once. One count per process, so `/usr/bin/time -l` reports + /// that scenario's peak memory: + /// + /// `ANT_FETCH_SCALE_BENCH=8 cargo test --release --features test-utils --lib + /// scale_benchmark -- --nocapture` + /// + /// Thresholds, declared before the release runs: one pass parking every key + /// holds the queue for at most 1 s at up to 8 sources per key and at most + /// 5 s at the 35-source limit; waking, the stats read and a departure each + /// stay under 250 ms; the process's peak RSS is at most 1 GiB at 8 sources + /// per key and 3 GiB at 35. Dropping the queues was first held to 250 ms + /// too, but measured 274 to 315 ms at 35 sources per key, and 226 to 295 ms + /// with the scheduler before the silent-source handling. It happens once, + /// when the engine shuts down, outside any lock, so it is held to 1 s. + #[test] + fn scale_benchmark_at_the_fetch_queue_bound() { + let Some(sources_per_key) = std::env::var("ANT_FETCH_SCALE_BENCH") + .ok() + .and_then(|raw| raw.parse::().ok()) + else { + return; + }; + let park_limit = if sources_per_key <= 8 { + Duration::from_secs(1) + } else { + Duration::from_secs(5) + }; + { + let other_limit = Duration::from_millis(250); + let teardown_limit = Duration::from_secs(1); + let sources: Vec = (0..sources_per_key) + .map(|index| { + let mut bytes = [0u8; 32]; + bytes[..8].copy_from_slice(&(index as u64 + 1).to_be_bytes()); + PeerId::from_bytes(bytes) + }) + .collect(); + let mut queues = ReplicationQueues::new(); + // Saturate every source with this node's own fetches first. + let mut blockers = Vec::new(); + for (index, source) in sources.iter().enumerate() { + for offset in 0..CAP { + let key = key_at(u32::MAX - u32::try_from(index * CAP + offset).unwrap()); + queues.start_fetch(key, *source, vec![*source]); + blockers.push(key); + } + } + let mut distance = 0u32; + while queues.fetch_queue_count() < MAX_FETCH_QUEUE { + let key = key_at(distance); + assert!(queues.enqueue_fetch(key, key, sources.clone())); + distance += 1; + } + + let started = Instant::now(); + let dispatch = queues.dispatch_fetches(8, CAP); + let park_pass = started.elapsed(); + assert_eq!(dispatch.started, Vec::new()); + assert_eq!(queues.fetch_parked.len(), MAX_FETCH_QUEUE); + + let associations: usize = queues.parked_by_source.values().map(BTreeSet::len).sum(); + + let started = Instant::now(); + let stats = queues.take_fetch_dispatch_stats(CAP); + let stats_pass = started.elapsed(); + assert_eq!(stats.parked, MAX_FETCH_QUEUE); + + let started = Instant::now(); + queues.complete_fetch(&blockers[0]); + let wake = queues.dispatch_fetches(8, CAP); + let wake_pass = started.elapsed(); + assert_eq!(wake.started.len(), 1); + + let started = Instant::now(); + queues.remove_hint_source(&sources[0]); + let departure = started.elapsed(); + + let started = Instant::now(); + drop(queues); + let teardown = started.elapsed(); + + println!( + "FETCH-SCALE sources_per_key={sources_per_key} keys={MAX_FETCH_QUEUE} \ + associations={associations} park_pass_ms={} stats_ms={} wake_ms={} \ + departure_ms={} teardown_ms={}", + park_pass.as_millis(), + stats_pass.as_millis(), + wake_pass.as_millis(), + departure.as_millis(), + teardown.as_millis() + ); + assert!( + park_pass <= park_limit, + "park pass {park_pass:?} over {park_limit:?}" + ); + for (name, took) in [ + ("stats", stats_pass), + ("wake", wake_pass), + ("departure", departure), + ] { + assert!( + took <= other_limit, + "{name} took {took:?}, over {other_limit:?}" + ); + } + assert!( + teardown <= teardown_limit, + "teardown took {teardown:?}, over {teardown_limit:?}" + ); + } + } } diff --git a/tests/e2e/fetch_busy_ab.rs b/tests/e2e/fetch_busy_ab.rs new file mode 100644 index 00000000..36e537c2 --- /dev/null +++ b/tests/e2e/fetch_busy_ab.rs @@ -0,0 +1,206 @@ +//! Opt-in measurement, not a pass/fail gate: how long requesters take to +//! fetch a batch of chunks while the first of their holders keeps its whole +//! fetch admission pool full, and what that costs the holders in the +//! requesters' trust tables. +//! +//! Every requester stays within its two-request share at each holder, as +//! #252's per-source scheduler keeps it, so the refusals measured are the ones +//! left after #252: a holder whose twelve-slot pool is full of other peers' +//! requests. Besides the test-only fetch serve gate, which #252 has too, it +//! reads two test-only counters, `fetch_admitted_for_test` and +//! `fetch_pool_full_refusals_for_test`. With those two patched in, the same +//! file measures #252 without the busy reply and with it. +//! +//! Run with `ANT_FETCH_BUSY_AB=1 cargo test --test e2e --features test-utils +//! fetch_busy_ab -- --nocapture --test-threads=1`. Without the variable it +//! returns at once. + +#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] + +use super::testnet::TestNetworkConfig; +use super::TestHarness; +use ant_node::client::compute_address; +use ant_node::ReplicationConfig; +use serial_test::serial; +use std::time::{Duration, Instant}; + +const FILL_DEADLINE: Duration = Duration::from_mins(15); +const POLL: Duration = Duration::from_millis(100); + +/// Nodes in the pool measurement's network. +const POOL_NODES: usize = 25; +/// A close group this wide makes every node responsible for every key, so +/// each requester stores what it fetches. +const POOL_CLOSE_GROUP: usize = 24; +/// The holders, in the source order every chunk uses. +const POOL_HOLDERS: [usize; 5] = [3, 4, 5, 6, 7]; +/// Chunks per requester: its per-peer share, so it never over-asks a holder. +const POOL_KEYS_PER_REQUESTER: usize = 2; +const POOL_CHUNK_LEN: usize = 64 * 1024; +/// How long the first holder keeps its pool full, once it is full, before it +/// serves what it holds. +const POOL_GATE_HOLD: Duration = Duration::from_secs(2); +/// A holder's fetch admission pool: `FETCH_RESPONDER_MAX_OUTSTANDING`. +const POOL_SLOTS: usize = 12; +/// Bound on reaching the precondition before measuring. +const POOL_SETUP_DEADLINE: Duration = Duration::from_mins(1); + +#[tokio::test] +#[serial] +async fn fetch_busy_ab_pool() { + if std::env::var("ANT_FETCH_BUSY_AB").is_err() { + return; + } + let config = TestNetworkConfig { + node_count: POOL_NODES, + replication_config: Some(ReplicationConfig { + close_group_size: POOL_CLOSE_GROUP, + ..ReplicationConfig::default() + }), + ..TestNetworkConfig::default() + }; + let harness = TestHarness::setup_with_config(config).await.expect("setup"); + harness.warmup_dht().await.expect("warmup"); + + let holders: Vec<_> = POOL_HOLDERS + .iter() + .map(|&i| { + *harness + .test_node(i) + .and_then(|n| n.p2p_node.as_ref()) + .expect("holder p2p") + .peer_id() + }) + .collect(); + let requesters: Vec = (0..POOL_NODES) + .filter(|i| !POOL_HOLDERS.contains(i)) + .collect(); + + let mut keys_by_requester = Vec::with_capacity(requesters.len()); + for (r, _) in requesters.iter().enumerate() { + let mut keys = Vec::with_capacity(POOL_KEYS_PER_REQUESTER); + for k in 0..POOL_KEYS_PER_REQUESTER { + let content: Vec = (0..POOL_CHUNK_LEN) + .map(|i| u8::try_from((i * 13 + r * 101 + k * 7) % 251).unwrap()) + .collect(); + let key = compute_address(&content); + for &h in &POOL_HOLDERS { + harness + .test_node(h) + .and_then(|n| n.ant_protocol.as_ref()) + .expect("holder protocol") + .storage() + .put(&key, &content) + .await + .expect("put"); + } + keys.push(key); + } + keys_by_requester.push(keys); + } + + let first_holder = POOL_HOLDERS + .first() + .and_then(|&i| harness.test_node(i)) + .and_then(|n| n.replication_engine.as_ref()) + .expect("first holder engine"); + first_holder.close_fetch_serve_gate_for_test(); + + let trust_of = |i: usize| -> Vec { + let p2p = harness + .test_node(i) + .and_then(|n| n.p2p_node.as_ref()) + .expect("requester p2p"); + holders.iter().map(|h| p2p.peer_trust(h)).collect() + }; + let trust_before: Vec> = requesters.iter().map(|&i| trust_of(i)).collect(); + + let total = requesters.len() * POOL_KEYS_PER_REQUESTER; + let refusals_before = first_holder.fetch_pool_full_refusals_for_test(); + let started = Instant::now(); + let mut enqueued = 0usize; + for (&i, keys) in requesters.iter().zip(&keys_by_requester) { + let engine = harness + .test_node(i) + .and_then(|n| n.replication_engine.as_ref()) + .expect("requester engine"); + for key in keys { + if engine.enqueue_fetch_for_test(*key, holders.clone()).await { + enqueued += 1; + } + } + } + assert_eq!( + enqueued, total, + "every key must be queued for the measurement" + ); + + // The measurement is only about a full pool: wait until the first holder + // holds a whole pool's worth, keep it full, and require that it refused + // the rest before letting it serve. + let setup_deadline = Instant::now() + POOL_SETUP_DEADLINE; + while first_holder.fetch_admitted_for_test() < POOL_SLOTS { + assert!( + Instant::now() < setup_deadline, + "the first holder never filled its pool: {} admitted", + first_holder.fetch_admitted_for_test() + ); + tokio::time::sleep(POLL).await; + } + tokio::time::sleep(POOL_GATE_HOLD).await; + let refused_while_full = first_holder + .fetch_pool_full_refusals_for_test() + .saturating_sub(refusals_before); + assert!( + refused_while_full > 0, + "the first holder refused nothing for a full pool while it was full" + ); + first_holder.open_fetch_serve_gate_for_test(); + let mut stored = 0usize; + while started.elapsed() < FILL_DEADLINE { + stored = 0; + for (&i, keys) in requesters.iter().zip(&keys_by_requester) { + let storage = harness + .test_node(i) + .and_then(|n| n.ant_protocol.as_ref()) + .expect("requester protocol") + .storage(); + for key in keys { + if storage.exists(key).unwrap_or(false) { + stored += 1; + } + } + } + if stored == total { + break; + } + tokio::time::sleep(POLL).await; + } + let elapsed = started.elapsed(); + + // Trust lost per holder, summed over requesters. Each connection-timeout + // charge from a neutral 0.5 costs about 0.06. + let trust_after: Vec> = requesters.iter().map(|&i| trust_of(i)).collect(); + let drops = |h: usize| { + trust_before + .iter() + .zip(&trust_after) + .filter_map(move |(b, a)| Some(b.get(h)? - a.get(h)?)) + }; + let lost: Vec = (0..holders.len()) + .map(|h| drops(h).map(|d| d.max(0.0)).sum()) + .collect(); + let charged: Vec = (0..holders.len()) + .map(|h| drops(h).filter(|d| *d > 0.0).count()) + .collect(); + + println!( + "FETCH_BUSY_AB_POOL requesters={} keys={total} enqueued={enqueued} stored={stored} \ + refused_while_full={refused_while_full} elapsed_ms={} \ + requesters_charging_each_holder={charged:?} trust_lost_per_holder={lost:?}", + requesters.len(), + elapsed.as_millis(), + ); + + harness.teardown().await.expect("teardown"); +} diff --git a/tests/e2e/fetch_busy_reply.rs b/tests/e2e/fetch_busy_reply.rs new file mode 100644 index 00000000..b4bbcabc --- /dev/null +++ b/tests/e2e/fetch_busy_reply.rs @@ -0,0 +1,443 @@ +//! A fetch responder answers `Busy` to a refused request that asked for it, +//! stays silent to one that did not, and a requester takes a busy answer +//! without charging the holder, until the holder has answered busy past its +//! neutral allowance without serving. +//! +//! The unit tests pin the wire format, the limiter, the wait and the queue +//! transitions. What they cannot show is the whole path between two real +//! nodes: the capability trailer surviving the transport, admission refusing +//! past the per-peer share, the reply riding the request/response channel +//! back, and the requester's fetch worker acting on it. +//! +//! The holder's fetch serve gate makes every case deterministic. With the gate +//! closed, an admitted request is held before it is served, keeping its share, +//! so of three concurrent requests from one peer exactly two are admitted and +//! exactly one is refused, whatever order they arrive in. + +#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] + +use super::TestHarness; +use ant_node::client::compute_address; +use ant_node::replication::config::{FETCH_REQUEST_TIMEOUT, REPLICATION_PROTOCOL_ID}; +use ant_node::replication::fetch_busy::{ + FETCH_BUSY_NEUTRAL_ALLOWANCE, FETCH_BUSY_WAIT_MAX, FETCH_BUSY_WAIT_MIN, +}; +use ant_node::replication::protocol::{ + FetchRequest, FetchResponse, ReplicationMessage, ReplicationMessageBody, +}; +use futures::stream::{FuturesUnordered, StreamExt}; +use saorsa_core::identity::PeerId; +use saorsa_core::P2PNode; +use serial_test::serial; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::time::{Duration, Instant}; + +const HOLDER_INDEX: usize = 3; +const REQUESTER_INDEX: usize = 4; +/// Requests admitted per requesting peer: the responder's per-peer share. +const SHARE: usize = 2; +/// Long enough that a held request is never timed out before the test opens +/// the gate; a timeout would charge the holder and spoil the trust checks. +const HELD_TIMEOUT: Duration = Duration::from_mins(1); +/// How long a request that should be refused silently is given. +const SILENCE_TIMEOUT: Duration = Duration::from_secs(3); +/// Bound on how long anything that must happen promptly may take. +const PROMPTLY: Duration = Duration::from_secs(10); +const POLL: Duration = Duration::from_millis(50); +const CHUNK_LEN: usize = 64 * 1024; +/// Just over the holder's per-peer busy reply interval of one a second, so +/// each of the requester's refused fetches is answered busy, not silently. +const BUSY_REPLY_SPACING: Duration = Duration::from_millis(1_100); + +#[derive(Debug, Clone, PartialEq, Eq)] +enum Answer { + Success, + Busy, + Silent, + Other, +} + +async fn fetch( + sender: &P2PNode, + holder: &PeerId, + key: [u8; 32], + busy_reply: bool, + timeout: Duration, +) -> Answer { + let msg = ReplicationMessage { + request_id: rand_request_id(), + body: ReplicationMessageBody::FetchRequest(FetchRequest { key, busy_reply }), + }; + let Ok(response) = sender + .send_request( + holder, + REPLICATION_PROTOCOL_ID, + msg.encode().expect("encode"), + timeout, + ) + .await + else { + return Answer::Silent; + }; + match ReplicationMessage::decode(&response.data).map(|m| m.body) { + Ok(ReplicationMessageBody::FetchResponse(FetchResponse::Success { key: k, .. })) + if k == key => + { + Answer::Success + } + Ok(ReplicationMessageBody::FetchResponse(FetchResponse::Busy { key: k })) if k == key => { + Answer::Busy + } + _ => Answer::Other, + } +} + +fn rand_request_id() -> u64 { + static NEXT: AtomicU64 = AtomicU64::new(9_000); + NEXT.fetch_add(1, Ordering::Relaxed) +} + +struct Setup { + harness: TestHarness, + key: [u8; 32], + holder: PeerId, +} + +/// A network whose holder has its fetch serve gate closed, and the holder. +async fn gated_holder() -> (TestHarness, PeerId) { + let harness = TestHarness::setup_minimal().await.expect("setup"); + harness.warmup_dht().await.expect("warmup"); + let holder_node = harness.test_node(HOLDER_INDEX).expect("holder"); + let holder = *holder_node.p2p_node.as_ref().expect("holder p2p").peer_id(); + holder_node + .replication_engine + .as_ref() + .expect("holder engine") + .close_fetch_serve_gate_for_test(); + (harness, holder) +} + +/// Put a chunk made from `seed` straight into the holder's store. +async fn put_on_holder(harness: &TestHarness, seed: usize) -> [u8; 32] { + let content: Vec = (0..CHUNK_LEN) + .map(|i| u8::try_from((i + seed * 7) % 251).unwrap()) + .collect(); + let key = compute_address(&content); + harness + .test_node(HOLDER_INDEX) + .and_then(|n| n.ant_protocol.as_ref()) + .expect("holder protocol") + .storage() + .put(&key, &content) + .await + .expect("put"); + key +} + +async fn setup() -> Setup { + let (harness, holder) = gated_holder().await; + let key = put_on_holder(&harness, 0).await; + Setup { + harness, + key, + holder, + } +} + +fn open_gate(harness: &TestHarness) { + harness + .test_node(HOLDER_INDEX) + .and_then(|n| n.replication_engine.as_ref()) + .expect("holder engine") + .open_fetch_serve_gate_for_test(); +} + +/// Of `SHARE + 1` concurrent requests that ask for busy replies, the one the +/// holder refuses is answered `Busy` at once, the held ones are served once +/// the gate opens, and a request that does not ask is refused silently. (The +/// per-peer and global busy allowances are pinned exactly by the limiter's +/// unit tests; over a real network their timing is not controllable.) +#[tokio::test] +#[serial] +async fn a_refused_fetch_is_answered_busy_only_when_it_asks() { + let Setup { + harness, + key, + holder, + } = setup().await; + let requester = harness + .test_node(REQUESTER_INDEX) + .and_then(|n| n.p2p_node.as_ref()) + .expect("requester p2p"); + + let mut pending: FuturesUnordered<_> = (0..=SHARE) + .map(|_| fetch(requester, &holder, key, true, HELD_TIMEOUT)) + .collect(); + let started = Instant::now(); + let first = tokio::time::timeout(PROMPTLY, pending.next()) + .await + .expect("the refused request is answered promptly") + .expect("a request"); + assert_eq!(first, Answer::Busy, "the refused request is told busy"); + assert!(started.elapsed() < PROMPTLY); + + // The share is still held, so a request that does not ask for busy + // replies is refused, and silently, exactly as before busy replies. + assert_eq!( + fetch(requester, &holder, key, false, SILENCE_TIMEOUT).await, + Answer::Silent + ); + + open_gate(&harness); + let rest: Vec = tokio::time::timeout(PROMPTLY, pending.collect()) + .await + .expect("held requests are served once the gate opens"); + assert_eq!(rest, vec![Answer::Success; SHARE]); + + harness.teardown().await.expect("teardown"); +} + +/// The requester's own fetch worker, refused by a holder whose share is full, +/// gets a busy answer, puts that holder last for later fetches, waits before +/// giving the key up, and charges the holder nothing. Before busy replies the +/// same refusal was silent and cost the holder a connection-timeout trust +/// event at the requester. +#[tokio::test] +#[serial] +async fn the_requester_takes_a_busy_answer_without_charging_the_holder() { + let Setup { + harness, + key, + holder, + } = setup().await; + let requester_node = harness.test_node(REQUESTER_INDEX).expect("requester"); + let requester = requester_node.p2p_node.as_ref().expect("requester p2p"); + let engine = requester_node + .replication_engine + .as_ref() + .expect("requester engine"); + let trust_before = requester.peer_trust(&holder); + + // Fill this requester's share at the holder: two held, one told busy. + let mut held: FuturesUnordered<_> = (0..=SHARE) + .map(|_| fetch(requester, &holder, key, true, HELD_TIMEOUT)) + .collect(); + let first = tokio::time::timeout(PROMPTLY, held.next()) + .await + .expect("the refused request is answered promptly") + .expect("a request"); + assert_eq!(first, Answer::Busy); + assert!(!engine.fetch_source_cooling_for_test(&holder)); + + // Trust is sampled throughout, so a charge cannot hide behind a later + // unrelated success: any dip below the starting value fails. + let mut lowest_trust = trust_before; + let sample = |lowest: &mut f64| *lowest = lowest.min(requester.peer_trust(&holder)); + + // The fetch worker's request is refused too, and told busy. + assert!(engine.enqueue_fetch_for_test(key, vec![holder]).await); + let deadline = Instant::now() + PROMPTLY; + while !engine.fetch_source_cooling_for_test(&holder) { + assert!( + Instant::now() < deadline, + "the fetch worker never recorded the busy answer" + ); + sample(&mut lowest_trust); + tokio::time::sleep(POLL).await; + } + + // The attempt holds its slot through the busy wait, then gives the key up + // (its only source refused), exactly as a timeout would have, sooner. The + // wait is timed from when the busy answer was recorded, just before it + // starts, to when this fetch attempt is first seen gone, watched with the + // gate still closed so nothing else delays the observation. + let busy_at = engine + .fetch_source_last_busy_for_test(&holder) + .expect("the busy answer was recorded"); + let deadline = Instant::now() + FETCH_BUSY_WAIT_MAX + PROMPTLY; + while engine.fetch_queued_or_in_flight_for_test(&key).await { + assert!( + Instant::now() < deadline, + "the fetch attempt outlived its busy wait" + ); + sample(&mut lowest_trust); + tokio::time::sleep(POLL).await; + } + let attempt_lasted = busy_at.elapsed(); + assert!( + attempt_lasted >= FETCH_BUSY_WAIT_MIN, + "the key was given up {attempt_lasted:?} after the busy answer, before the busy wait" + ); + + open_gate(&harness); + let rest: Vec = tokio::time::timeout(PROMPTLY, held.collect()) + .await + .expect("held requests are served once the gate opens"); + assert_eq!(rest, vec![Answer::Success; SHARE]); + sample(&mut lowest_trust); + assert!( + lowest_trust >= trust_before, + "a busy answer must not cost the holder trust: {trust_before} fell to {lowest_trust}" + ); + + harness.teardown().await.expect("teardown"); +} + +/// A holder that keeps answering busy, across many keys and without serving, +/// costs it nothing at the requester's fetch worker for +/// `FETCH_BUSY_NEUTRAL_ALLOWANCE` answers in a row. The next is charged, as its +/// silence would have been, once that attempt's fetch timeout has run out. +/// Serving the requester once then clears what the holder owes. +/// +/// The holder stores nothing until the end. A chunk put straight into its +/// store has no payment behind it, so once replication starts, about a minute +/// in, the holder's replica hint for it is rejected and charged, and this test +/// runs longer than that. +#[tokio::test] +#[serial] +async fn a_holder_that_keeps_answering_busy_is_charged_past_its_allowance() { + let (harness, holder) = gated_holder().await; + let requester_node = harness.test_node(REQUESTER_INDEX).expect("requester"); + let requester = requester_node.p2p_node.as_ref().expect("requester p2p"); + let engine = requester_node + .replication_engine + .as_ref() + .expect("requester engine"); + let trust_before = requester.peer_trust(&holder); + + // Fill this requester's share at the holder, so the holder refuses every + // fetch the worker sends it. Each attempt below runs to its end before the + // next starts, so the held requests must outlast all of them: a timeout + // would charge the holder. They are never answered. The gate stays closed + // for longer than the holder's fetch timeout, so once it opens the holder + // sheds them as stale. + let held_key = compute_address(b"held"); + let held_timeout = (BUSY_REPLY_SPACING + FETCH_BUSY_WAIT_MAX + PROMPTLY) + * (FETCH_BUSY_NEUTRAL_ALLOWANCE + 1) + + FETCH_REQUEST_TIMEOUT; + let mut held: FuturesUnordered<_> = (0..=SHARE) + .map(|_| fetch(requester, &holder, held_key, true, held_timeout)) + .collect(); + let first = tokio::time::timeout(PROMPTLY, held.next()) + .await + .expect("the refused request is answered promptly") + .expect("a request"); + assert_eq!(first, Answer::Busy); + + // Busy answers from the worker, each for a new key, until the holder has + // used up its allowance: the allowance belongs to the holder, not to a key. + // Each attempt runs to its end before the next key is queued, so any + // charge would come from the attempt just made. + let mut next_key = 0u32; + let mut lowest_trust = trust_before; + loop { + let owed = engine.fetch_source_unserved_busy_for_test(&holder); + if owed >= FETCH_BUSY_NEUTRAL_ALLOWANCE { + break; + } + tokio::time::sleep(BUSY_REPLY_SPACING).await; + let other_key = compute_address(&next_key.to_le_bytes()); + next_key += 1; + assert!(engine.enqueue_fetch_for_test(other_key, vec![holder]).await); + let deadline = Instant::now() + FETCH_REQUEST_TIMEOUT + PROMPTLY; + while engine.fetch_queued_or_in_flight_for_test(&other_key).await { + lowest_trust = lowest_trust.min(requester.peer_trust(&holder)); + assert!( + Instant::now() < deadline, + "the attempt after {owed} busy answers never ended" + ); + tokio::time::sleep(POLL).await; + } + lowest_trust = lowest_trust.min(requester.peer_trust(&holder)); + assert!( + engine.fetch_source_unserved_busy_for_test(&holder) > owed, + "the attempt after {owed} busy answers was not answered busy" + ); + assert!( + lowest_trust >= trust_before, + "a busy answer within the allowance cost the holder trust: \ + {trust_before} fell to {lowest_trust}" + ); + } + assert_eq!( + engine.fetch_source_unserved_busy_for_test(&holder), + FETCH_BUSY_NEUTRAL_ALLOWANCE + ); + + // The next answer is past the allowance. Its attempt holds its slot until + // its fetch timeout runs out, costing nothing meanwhile, and the holder is + // charged only as it ends. + tokio::time::sleep(BUSY_REPLY_SPACING).await; + let last_key = compute_address(&next_key.to_le_bytes()); + assert!(engine.enqueue_fetch_for_test(last_key, vec![holder]).await); + let deadline = Instant::now() + PROMPTLY; + while engine.fetch_source_unserved_busy_for_test(&holder) <= FETCH_BUSY_NEUTRAL_ALLOWANCE { + assert!( + Instant::now() < deadline, + "the answer past the allowance never came" + ); + tokio::time::sleep(POLL).await; + } + let answered_at = Instant::now(); + // The answer was seen within `PROMPTLY` of the request, so the attempt + // holds its slot, uncharged, for at least the rest: still well past any + // busy wait. + let not_before = FETCH_REQUEST_TIMEOUT.saturating_sub(PROMPTLY); + let deadline = answered_at + FETCH_REQUEST_TIMEOUT + PROMPTLY; + while engine.fetch_queued_or_in_flight_for_test(&last_key).await { + if answered_at.elapsed() < not_before { + lowest_trust = lowest_trust.min(requester.peer_trust(&holder)); + } + assert!( + Instant::now() < deadline, + "the attempt past the allowance never ended" + ); + tokio::time::sleep(POLL).await; + } + let attempt_lasted = answered_at.elapsed(); + assert!( + attempt_lasted >= not_before, + "the attempt past the allowance ended {attempt_lasted:?} after its answer, \ + before its fetch timeout" + ); + assert!( + lowest_trust >= trust_before, + "the answer past the allowance was charged before its attempt ended: \ + {trust_before} fell to {lowest_trust}" + ); + let trust_after = requester.peer_trust(&holder); + assert!( + trust_after < trust_before, + "the answer past the allowance was not charged: trust stayed {trust_after}" + ); + drop(held); + + // Serving once clears what the holder owes. The chunk is new, so the + // requester cannot already hold it, and it is fetched only after the + // holder has shed the held requests and so has room for it. + let fresh_key = put_on_holder(&harness, 1).await; + open_gate(&harness); + let holder_engine = harness + .test_node(HOLDER_INDEX) + .and_then(|n| n.replication_engine.as_ref()) + .expect("holder engine"); + let deadline = Instant::now() + PROMPTLY; + while holder_engine.fetch_admitted_for_test() > 0 { + assert!( + Instant::now() < deadline, + "the holder never let go of the held requests" + ); + tokio::time::sleep(POLL).await; + } + assert!(engine.enqueue_fetch_for_test(fresh_key, vec![holder]).await); + let deadline = Instant::now() + PROMPTLY; + while engine.fetch_source_unserved_busy_for_test(&holder) > 0 { + assert!( + Instant::now() < deadline, + "serving the requester did not clear what the holder owes" + ); + tokio::time::sleep(POLL).await; + } + + harness.teardown().await.expect("teardown"); +} diff --git a/tests/e2e/fetch_per_source_share.rs b/tests/e2e/fetch_per_source_share.rs new file mode 100644 index 00000000..b4c32d2a --- /dev/null +++ b/tests/e2e/fetch_per_source_share.rs @@ -0,0 +1,332 @@ +//! A fetching node keeps no more of its fetches in flight to one answering +//! holder than that holder will admit from it. +//! +//! A fetch responder admits two outstanding requests per requesting peer and +//! refuses a third. Before busy replies it answered that third with nothing +//! at all, and it still does when it cannot send a busy reply, so a requester +//! that sends more than two at once to one holder can lose the rest to its own +//! fetch timeout. The fetch +//! worker used to do exactly that: it ran one fetch per slot and sent every one +//! to the first verified holder of its key, so a node whose queued keys shared +//! one holder had all of them outstanding there at once. +//! +//! The observation is made on the requester, which is what the scheduler +//! governs, and made deterministic by holding the holder's work in place: the +//! holder's test-only serve gate holds every fetch request it admits before +//! serving it, so whatever the requester sends stays outstanding for as long as +//! the test looks at it, however fast the host serves a chunk. Every node runs +//! eight fetch slots, more than one holder's share on any host. With the +//! per-source limit removed the requester holds all eight in flight to the one +//! holder and this test fails. Both hooks are per engine, so no other node or +//! test is affected. +//! +//! The limit is a share of a holder that answers. A holder whose requests go +//! unanswered has no share to keep within, and keeping its keys at two per +//! fetch timeout would drain them four times more slowly than before the limit. +//! The second test closes the holder's gate for good, with a short fetch +//! timeout, and checks that once the first requests time out the requester +//! sends past the share and the keys leave its fetch pipeline in a bounded time. +//! +//! Both tests tear the network down with the holder's gate still closed and +//! requests held at it, which the holder's engine shutdown must release. + +#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] + +use super::testnet::TestNetworkConfig; +use super::verify_storage_gate::find_key_for_target; +use super::TestHarness; +use ant_node::client::XorName; +use ant_node::replication::config::storage_admission_width; +use ant_node::{ReplicationConfig, ReplicationEngine}; +use saorsa_core::identity::PeerId; +use serial_test::serial; +use std::collections::HashSet; +use std::time::{Duration, Instant}; + +/// Above `storage_admission_width` (9), so the network has a real close group. +const NODE_COUNT: usize = 12; +/// Production close-group size. +const CLOSE_GROUP_SIZE: usize = 7; +/// Node whose fetch worker is observed. +const REQUESTER_INDEX: usize = 5; +/// The one node hosting every chunk. +const HOLDER_INDEX: usize = 6; +/// More keys than fetch slots, all held by the one holder. +const KEY_COUNT: usize = 12; +/// Fetch slots per node, above the per-source share on every host. +const FETCH_CONCURRENCY: usize = 8; +/// The responder's per-peer fetch share. +const PER_SOURCE_SHARE: usize = 2; +/// How long the requester's fetch pipeline must stay empty before the test +/// starts, so earlier traffic cannot occupy its slots. +const QUIET_FOR: Duration = Duration::from_secs(1); +/// Upper bound on waiting for that quiet period. +const QUIET_DEADLINE: Duration = Duration::from_secs(30); +/// How long the requester is observed with the gate closed. Far below the 30 s +/// fetch timeout, so no held request times out and frees its slot early. +const OBSERVATION_WINDOW: Duration = Duration::from_secs(3); +const OBSERVATION_POLL: Duration = Duration::from_millis(50); +/// The silent-holder test's fetch timeout, short so silence is seen in seconds. +const SILENT_FETCH_TIMEOUT: Duration = Duration::from_secs(5); +/// Keys held only by the silent holder: two fetch slots' worth. +const SILENT_KEY_COUNT: usize = 16; +/// The longest the silent holder's keys may take to leave the requester's +/// fetch pipeline: four fetch timeouts. This scheduler takes three, one more +/// than fetching as before the limit, to see the silence; keeping the holder +/// at its share would take eight (sixteen keys, two per timeout). The exact +/// three waves are checked without timing in the scheduler's unit tests. +const SILENT_DRAIN_LIMIT: Duration = Duration::from_secs(20); +/// The longest tearing the network down may take with requests still held at +/// the holder's closed serve gate. Without the release at engine shutdown it +/// waits for them for ever. +const TEARDOWN_LIMIT: Duration = Duration::from_secs(120); + +#[tokio::test] +#[serial] +async fn requester_keeps_at_most_its_share_in_flight_to_one_holder() { + let max_in_flight = max_in_flight_to_one_holder().await; + assert!( + max_in_flight <= PER_SOURCE_SHARE, + "the requester had {max_in_flight} fetches outstanding at one holder, which \ + admits {PER_SOURCE_SHARE} from it and answers the rest with nothing" + ); + assert_eq!( + max_in_flight, PER_SOURCE_SHARE, + "the requester never filled the holder's share, so the run says nothing \ + about the limit: the worker did not dispatch" + ); +} + +#[tokio::test] +#[serial] +async fn a_holder_that_stops_answering_is_fetched_from_past_its_share() { + let harness = start_network(SILENT_FETCH_TIMEOUT).await; + let chunks = chunks_for_requester(&harness, "silent-holder", SILENT_KEY_COUNT).await; + let keys: HashSet = chunks.iter().map(|(_, key)| *key).collect(); + + let requester = harness.test_node(REQUESTER_INDEX).expect("requester"); + let requester_engine = requester + .replication_engine + .as_ref() + .expect("requester engine"); + let holder = harness.test_node(HOLDER_INDEX).expect("holder"); + let holder_peer = *holder.p2p_node.as_ref().expect("holder p2p").peer_id(); + let holder_engine = holder.replication_engine.as_ref().expect("holder engine"); + + wait_until_fetch_pipeline_is_quiet(requester_engine).await; + + // The holder answers none of these fetches, ever: it holds the ones it + // admits, and refuses the rest without even a busy reply. + holder_engine.close_fetch_serve_gate_for_test(); + holder_engine.turn_off_fetch_busy_replies_for_test(); + let started = Instant::now(); + for key in &keys { + assert!( + requester_engine + .enqueue_fetch_for_test(*key, vec![holder_peer]) + .await, + "every key must be queued for the silent holder alone" + ); + } + + let mut max_in_flight_to_holder = 0; + let drained_after = loop { + let snapshot = requester_engine.fetch_pipeline_snapshot_for_test().await; + let in_flight_to_holder = snapshot + .in_flight + .iter() + .filter(|(key, source)| keys.contains(key) && *source == holder_peer) + .count(); + max_in_flight_to_holder = max_in_flight_to_holder.max(in_flight_to_holder); + let pending = snapshot + .queued + .iter() + .chain(snapshot.in_flight.iter().map(|(key, _)| key)) + .filter(|key| keys.contains(*key)) + .count(); + if pending == 0 { + break started.elapsed(); + } + assert!( + started.elapsed() < SILENT_DRAIN_LIMIT, + "{pending} of {SILENT_KEY_COUNT} keys held only by a silent holder were still \ + in the fetch pipeline after {SILENT_DRAIN_LIMIT:?}" + ); + tokio::time::sleep(OBSERVATION_POLL).await; + }; + println!( + "SILENT-HOLDER-RESULT drained_after={drained_after:?} \ + max_in_flight_to_holder={max_in_flight_to_holder} timeout={SILENT_FETCH_TIMEOUT:?}" + ); + + teardown_with_the_gate_closed(harness).await; + assert!( + drained_after < SILENT_DRAIN_LIMIT, + "the silent holder's keys took {drained_after:?} to leave the fetch pipeline" + ); + assert!( + max_in_flight_to_holder > PER_SOURCE_SHARE, + "the requester kept a holder that answers nothing at its share of \ + {PER_SOURCE_SHARE}, which drains that holder's keys at two per timeout" + ); +} + +/// Build the network with `fetch_request_timeout` and warm its DHT. +async fn start_network(fetch_request_timeout: Duration) -> TestHarness { + let config = TestNetworkConfig { + node_count: NODE_COUNT, + replication_config: Some(ReplicationConfig { + close_group_size: CLOSE_GROUP_SIZE, + fetch_request_timeout, + ..ReplicationConfig::default() + }), + fetch_concurrency: Some(FETCH_CONCURRENCY), + ..TestNetworkConfig::default() + }; + let harness = TestHarness::setup_with_config(config) + .await + .expect("setup per-source share network"); + harness.warmup_dht().await.expect("warmup"); + harness +} + +/// `count` distinct chunks the requester is responsible for storing. +async fn chunks_for_requester( + harness: &TestHarness, + label: &str, + count: usize, +) -> Vec<(Vec, XorName)> { + let width = storage_admission_width(CLOSE_GROUP_SIZE); + let mut chunks = Vec::with_capacity(count); + for index in 0..count { + let label = format!("{label}-{index}"); + chunks.push(find_key_for_target(harness, REQUESTER_INDEX, width, true, &label).await); + } + let keys: HashSet<&XorName> = chunks.iter().map(|(_, key)| key).collect(); + assert_eq!(keys.len(), count, "every key must be distinct"); + chunks +} + +/// Build the network, direct twelve keys at one holder while that holder holds +/// what it admits, and return the most fetches the requester had in flight to +/// it. +async fn max_in_flight_to_one_holder() -> usize { + let harness = start_network(ReplicationConfig::default().fetch_request_timeout).await; + let chunks = chunks_for_requester(&harness, "per-source-share", KEY_COUNT).await; + let keys: HashSet = chunks.iter().map(|(_, key)| *key).collect(); + + let requester = harness.test_node(REQUESTER_INDEX).expect("requester"); + let requester_engine = requester + .replication_engine + .as_ref() + .expect("requester engine"); + let holder = harness.test_node(HOLDER_INDEX).expect("holder"); + let holder_peer = *holder.p2p_node.as_ref().expect("holder p2p").peer_id(); + let holder_engine = holder.replication_engine.as_ref().expect("holder engine"); + let holder_storage = holder + .ant_protocol + .as_ref() + .expect("holder protocol") + .storage(); + + // Start from an idle requester, so its fetch slots are all free. + wait_until_fetch_pipeline_is_quiet(requester_engine).await; + + // Hold every fetch the holder admits, then make the chunks available. + holder_engine.close_fetch_serve_gate_for_test(); + for (content, key) in &chunks { + holder_storage.put(key, content).await.expect("host chunk"); + } + let before = requester_engine.fetch_pipeline_snapshot_for_test().await; + let unrelated: Vec = before + .queued + .iter() + .chain(before.in_flight.iter().map(|(key, _)| key)) + .filter(|key| !keys.contains(*key)) + .copied() + .collect(); + assert!( + unrelated.is_empty(), + "unrelated fetch work could occupy the requester's slots: {unrelated:?}" + ); + for key in &keys { + // A replica hint may already have queued the key; either way it is + // headed to the one holder. + let _ = requester_engine + .enqueue_fetch_for_test(*key, vec![holder_peer]) + .await; + } + + let max_in_flight_to_holder = + observe_max_in_flight_to(requester_engine, &keys, &holder_peer).await; + println!( + "PER-SOURCE-SHARE-RESULT max_in_flight_to_holder={max_in_flight_to_holder} \ + concurrency={FETCH_CONCURRENCY} keys={KEY_COUNT}" + ); + + teardown_with_the_gate_closed(harness).await; + max_in_flight_to_holder +} + +/// Tear the network down while the holder's serve gate is still closed. +async fn teardown_with_the_gate_closed(harness: TestHarness) { + let torn_down = tokio::time::timeout(TEARDOWN_LIMIT, harness.teardown()).await; + assert!( + matches!(torn_down, Ok(Ok(()))), + "teardown with requests held at a closed serve gate did not finish within \ + {TEARDOWN_LIMIT:?}: {torn_down:?}" + ); +} + +/// Wait until `engine` has nothing queued or in flight for fetch, continuously +/// for [`QUIET_FOR`]. +async fn wait_until_fetch_pipeline_is_quiet(engine: &ReplicationEngine) { + let deadline = tokio::time::Instant::now() + QUIET_DEADLINE; + let mut quiet_since = tokio::time::Instant::now(); + loop { + let snapshot = engine.fetch_pipeline_snapshot_for_test().await; + let now = tokio::time::Instant::now(); + if !snapshot.queued.is_empty() || !snapshot.in_flight.is_empty() { + quiet_since = now; + } else if now.duration_since(quiet_since) >= QUIET_FOR { + return; + } + assert!( + now < deadline, + "the requester's fetch pipeline never went quiet: {snapshot:?}" + ); + tokio::time::sleep(OBSERVATION_POLL).await; + } +} + +/// The most of `keys` that `engine` had in flight to `holder` at once over +/// [`OBSERVATION_WINDOW`], failing if any other fetch work appears. +async fn observe_max_in_flight_to( + engine: &ReplicationEngine, + keys: &HashSet, + holder: &PeerId, +) -> usize { + let deadline = tokio::time::Instant::now() + OBSERVATION_WINDOW; + let mut max_in_flight = 0; + while tokio::time::Instant::now() < deadline { + let snapshot = engine.fetch_pipeline_snapshot_for_test().await; + let unrelated: Vec<&XorName> = snapshot + .queued + .iter() + .chain(snapshot.in_flight.iter().map(|(key, _)| key)) + .filter(|key| !keys.contains(*key)) + .collect(); + assert!( + unrelated.is_empty(), + "unrelated fetch work appeared during the observation: {unrelated:?}" + ); + let to_holder = snapshot + .in_flight + .iter() + .filter(|(key, source)| keys.contains(key) && source == holder) + .count(); + max_in_flight = max_in_flight.max(to_holder); + tokio::time::sleep(OBSERVATION_POLL).await; + } + max_in_flight +} diff --git a/tests/e2e/mod.rs b/tests/e2e/mod.rs index ec2a74c7..5ecd86d5 100644 --- a/tests/e2e/mod.rs +++ b/tests/e2e/mod.rs @@ -69,12 +69,21 @@ mod security_attacks; #[cfg(test)] mod subtree_audit_testnet; +#[cfg(test)] +mod fetch_busy_ab; + +#[cfg(test)] +mod fetch_busy_reply; + #[cfg(test)] mod fetch_local_write_guard; #[cfg(test)] mod fetch_responsibility_recheck; +#[cfg(test)] +mod fetch_per_source_share; + #[cfg(test)] mod first_audit_ab; diff --git a/tests/e2e/replication.rs b/tests/e2e/replication.rs index 6ae3fb5b..06855160 100644 --- a/tests/e2e/replication.rs +++ b/tests/e2e/replication.rs @@ -1203,7 +1203,10 @@ async fn test_fetch_request_returns_record() { .expect("put"); // Send fetch request from B to A - let request = FetchRequest { key: address }; + let request = FetchRequest { + key: address, + busy_reply: false, + }; let msg = ReplicationMessage { request_id: 99, body: ReplicationMessageBody::FetchRequest(request), @@ -2030,7 +2033,10 @@ async fn test_fetch_not_found() { let peer_a = *p2p_a.peer_id(); let missing_key = [0xEE; 32]; - let request = FetchRequest { key: missing_key }; + let request = FetchRequest { + key: missing_key, + busy_reply: false, + }; let msg = ReplicationMessage { request_id: 77, body: ReplicationMessageBody::FetchRequest(request), @@ -2366,7 +2372,10 @@ async fn test_fetch_returns_error_for_corrupt_key() { let peer_a = *p2p_a.peer_id(); let fake_key = [0x00; 32]; - let request = FetchRequest { key: fake_key }; + let request = FetchRequest { + key: fake_key, + busy_reply: false, + }; let msg = ReplicationMessage { request_id: 4000, body: ReplicationMessageBody::FetchRequest(request), diff --git a/tests/e2e/testnet.rs b/tests/e2e/testnet.rs index ca33a901..051e42b4 100644 --- a/tests/e2e/testnet.rs +++ b/tests/e2e/testnet.rs @@ -230,6 +230,10 @@ pub struct TestNetworkConfig { /// use this to shorten timers — e.g. the ADR-0003 possession-check delay. pub replication_config: Option, + /// Fetches each node's worker runs at once. `None` uses one per hardware + /// thread. + pub fetch_concurrency: Option, + /// Optional per-node storage disk-reserve overrides. /// /// Tests can set a node's reserve above available disk space to make its @@ -283,6 +287,7 @@ impl Default for TestNetworkConfig { payment_enforcement: false, evm_network: None, replication_config: None, + fetch_concurrency: None, storage_disk_reserve_overrides: HashMap::new(), } } @@ -1388,6 +1393,9 @@ impl TestNetwork { .await { Ok(mut engine) => { + if let Some(concurrency) = self.config.fetch_concurrency { + engine.set_fetch_concurrency_for_test(concurrency); + } // Pointers replicate through the same engine (ADR-0016), // which the service also asks before a final state // (ADR-0018).