From b8e7dd41f029b1658cb2200268a6517601116f48 Mon Sep 17 00:00:00 2001 From: grumbach Date: Thu, 15 Oct 2026 18:08:23 +0900 Subject: [PATCH 01/10] feat(replication)!: answer a refused fetch with busy instead of silence A fetch responder admits two outstanding requests per requesting peer and twelve overall, and refuses the rest at admission without a reply. The requester then waits its whole 30 s fetch timeout, and saorsa-core charges the responder a connection-timeout trust event it did not earn. As measured for #252 from the responders' cumulative admission counters, this was 36.6% of inbound fetch requests on 0.21.0 and 44.3% on 0.22.0, and the WARN line counted 3.38M refusals in the 25 hours of the 2026-10-11 join wave. A requester now says it understands a busy reply, and a responder that refuses such a request answers `FetchResponse::Busy` at once, within bounds. Wire: `FetchRequest` keeps its serialized layout. A requester that understands busy replies appends a 5-byte capability trailer ("antf" and a flags byte) after the postcard message. Older nodes decode with `from_bytes`, which ignores it, so they refuse silently as before. A current node reads it with `take_from_bytes`. `FetchResponse::Busy` is appended after the existing variants, whose discriminants are unchanged, and is only sent in answer to a request with the trailer, so an older requester never receives it. Responder: a busy reply needs a free send permit (48), the per-peer allowance (two back to back, then one a second, at most two in flight) and the global one (32 back to back, then 16 a second; every message is signed, so about 5 KiB each). It is sent off the serial message loop, abandoned after 2 s and raced against shutdown. Anything outside those bounds is the old silence. The refusal WARN line keeps its text and counters and gains `capability` and `reply` fields; new cumulative counters report each busy-reply outcome. Requester: the fetch worker's requests ask for busy replies. A busy answer for the requested key carries no trust event. The attempt waits 5-10 s (jittered, never past its own fetch timeout) while holding its fetch slot, then ends as a source failure, so the queue moves on exactly as after a timeout. The busy holder is put last when choosing where to start or retry a key while its cooldown runs: the wait, doubling with each further busy answer up to 60 s. Serving once resets that doubling, and so do 10 minutes without a busy answer. Merge after #252: its per-source share removes most per-peer over-share, and the refusals that remain are what this answers. A test-only fetch serve gate, the same one #252 adds, makes the end-to-end tests deterministic. ADR-0025 records the decision. BREAKING CHANGE: `replication::protocol::FetchRequest` gains a public `busy_reply` field (not serialized) and `FetchResponse` gains a `Busy` variant, so code constructing the struct or matching the enum exhaustively must be updated. The wire format stays compatible with older nodes in both directions. --- docs/adr/ADR-0025-fetch-busy-reply.md | 209 ++++++++ docs/adr/README.md | 1 + src/replication/audit_metrics.rs | 121 +++++ src/replication/fetch_busy.rs | 745 ++++++++++++++++++++++++++ src/replication/mod.rs | 534 +++++++++++++++++- src/replication/protocol.rs | 380 ++++++++++++- src/replication/scheduling.rs | 21 +- tests/e2e/fetch_busy_ab.rs | 299 +++++++++++ tests/e2e/fetch_busy_reply.rs | 266 +++++++++ tests/e2e/mod.rs | 6 + tests/e2e/replication.rs | 15 +- 11 files changed, 2557 insertions(+), 40 deletions(-) create mode 100644 docs/adr/ADR-0025-fetch-busy-reply.md create mode 100644 src/replication/fetch_busy.rs create mode 100644 tests/e2e/fetch_busy_ab.rs create mode 100644 tests/e2e/fetch_busy_reply.rs diff --git a/docs/adr/ADR-0025-fetch-busy-reply.md b/docs/adr/ADR-0025-fetch-busy-reply.md new file mode 100644 index 00000000..091d97c8 --- /dev/null +++ b/docs/adr/ADR-0025-fetch-busy-reply.md @@ -0,0 +1,209 @@ +# ADR-0025: Tell a fetch requester "busy" instead of refusing silently + +- **Status:** Proposed +- **Date:** 2026-10-15 +- **Decision owners:** +- **Reviewers:** +- **Supersedes:** none +- **Superseded by:** none +- **Related:** [ADR-0005](ADR-0005-replication-repair-hardening.md) (the repair fetch pipeline), + [ADR-0017](ADR-0017-bounded-fresh-offers-and-copy-free-sends.md) (responder admission bounds), + ant-node #252 / ADR-0024 (requesters stay within the responder's per-peer share) + +## Context + +A replication fetch is a request/response exchange. The responder admits at most two +outstanding fetches per requesting peer (`FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER`) and twelve +overall (`FETCH_RESPONDER_MAX_OUTSTANDING`). A request past either limit is refused at admission: +the responder logs `Fetch response dropped at admission` and sends nothing back. + +Silence is expensive for both sides: + +- the requester waits its whole fetch timeout (30 s) before moving on to the next holder; +- saorsa-core records a `ConnectionTimeout` trust event against the responder, which did nothing + wrong; +- the requester learns nothing about why, so it keeps choosing that holder for new keys. + +On the production fleet this was the common case, not an edge. As measured for #252 from the +responders' own cumulative admission counters, 36.6% of inbound fetch requests on 0.21.0 and 44.3% +on 0.22.0 were refused at admission. In the 25 hours around a wave of 177 newly installed nodes +filling their ranges (2026-10-11 20:00 to 2026-10-12 21:00 UTC), the WARN line counted 3.38M +refusals, 95.2% of them to the new nodes. + +ant-node #252 removes most of the per-peer over-share by keeping a requester's own fetches within +each answering holder's share. It cannot remove every refusal: a request the requester has +already timed out on still holds its share at the responder until the responder sheds it, the +twelve-slot pool can be full of other peers' requests, and older requesters keep over-sharing +until they upgrade. Its ADR names a busy reply, which is a wire change, as the remaining piece. + +## Decision Drivers + +- Most peers run older versions for weeks after any release. An old node must decode the same + messages it always did and never be sent a `Busy` it cannot decode. +- A delivered `Busy` must not cost an honest responder trust; a refusal that cannot be answered + stays the old silence. +- The added reply traffic must be bounded: no request storm from requesters retrying on every busy + answer, and no echo amplification at responders. +- No new requester scheduling state that would collide with #252's per-source scheduler. + +## Considered Options + +1. **Keep silence; rely on #252 alone.** Leaves every remaining refusal at 30 s and a false + trust charge. +2. **Reply with the existing `FetchResponse::Error`.** Decodable by old requesters, but they + charge the responder an `ApplicationFailure` for it and fail over at once, so it trades a + timeout for an immediate hop storm. +3. **Add a field to `FetchRequest`.** Old responders would ignore a trailing field, but a current + responder would have to read old requests without it, which postcard reports as an error, not a + missing field. A hand-written tolerant deserializer can paper over that, but it cannot tell a + missing flag from a malformed one and only stays correct while `FetchRequest` is the last value + in every message that carries it. +4. **A capability trailer after the whole message, and an appended `FetchResponse::Busy` + variant (chosen).** +5. **Busy with requester-side retry rotation and budgets.** Reviewed and rejected: hopping on every + busy answer multiplies requests about tenfold in the worst case, competes with `tried` state, + and charging at budget expiry punishes honest overloaded holders once per key per cycle. + +## Decision + +We will let a requester say it understands a busy reply, and have a responder that refuses such a +request at admission say `Busy` at once, within bounds, instead of staying silent. + +**Wire.** `FetchRequest` keeps its serialized layout (`key`). A requester that understands busy +replies appends the 5-byte fetch capability trailer, `antf` and a flags byte with bit 0 set, +immediately after the postcard message. `ReplicationMessage::decode` uses `take_from_bytes` and +sets `FetchRequest::busy_reply` (not serialized) only when that trailer follows a fetch request; +bytes after the flags byte are reserved and ignored. Older nodes decode with `from_bytes`, which +ignores everything after the message, so they read the key and refuse silently as before. +`FetchResponse::Busy { key }` is appended after `Success`, `NotFound` and `Error`, whose +discriminants are unchanged, and is only ever sent in answer to a request carrying the trailer. + +**Responder.** Every saorsa-core message is signed with ML-DSA-65, so a busy reply costs a +signature and about 5 KiB on the wire, not a few bytes. On an admission refusal of a request that +asked, the responder takes a busy-reply send permit (48 at once), then a grant from +`FetchBusyReplyLimiter`: per peer, two back to back and then one a second, with at most two in +flight; overall, 32 back to back and then 16 a second (at most 48 replies, about 250 KiB, in any +one second; about 85 KiB/s sustained). Both rate allowances are tested and spent together under +one lock, a denial spends neither, and the per-peer table is capped at 4096 entries, forgetting +only peers whose allowance has refilled and that have nothing in flight. Within all bounds it +sends `Busy` on the detached task tracker, abandons the send after 2 s and races shutdown; +otherwise it refuses silently, exactly as before. The `Fetch response dropped at admission` +WARN line and its counters are unchanged, with `capability=asked|not_asked` and +`reply=scheduled|none` fields added. + +**Requester.** Every fetch request this node's fetch worker sends asks for busy replies. A `Busy` for the requested key carries +no trust event. The attempt then waits 5-10 s (uniform jitter), never past its own fetch timeout, +still holding its fetch slot, and ends as `SourceFailed`, so the queue moves on to the next untried +holder exactly as after a timeout. `SourceBusyTracker` remembers the busy holder: a new key starts, +and a failed key retries, on the first holder in preference order that is not cooling down, +falling back to the first when all are. It reorders and never withholds, so it lowers how often a +busy holder is asked but is not a hard per-holder bound; the hard bound is the busy wait per fetch +slot. The cooldown is the full jittered wait, doubling with each further busy answer before the +holder next serves this requester, up to 60 s, and is never shortened while it runs; once it runs +out the holder is no longer put last. Serving once clears the holder's record, so its next busy +answer starts the doubling again; a record with no busy answer for 10 minutes does the same. When +the 4096-entry table is full, a holder that is no longer cooling is forgotten first. A `Busy` for any other key is +treated as malformed, like every other key mismatch. + +**Order.** This is merged after #252. #252's per-source share is what removes most per-peer +over-share and bounds how many fetches one requester has open at one holder; without it, a +requester with many fetch slots still over-shares, and most of those refusals exceed the per-peer +busy allowance and stay silent. On rebase, #252's source choice also skips cooling holders. + +## Consequences + +### Positive + +- When a refused requester receives `Busy`, it moves on after 5-10 s instead of 30 s without + charging the responder. +- A holder that just refused is not handed this requester's next keys while another holder that is + not cooling exists. +- Old nodes decode exactly the messages they always did: a request carries 5 bytes after the + message that they never read, and they are never sent `Busy`. Rollback is a release without the + change. +- With #252, each busy wait keeps one attempt counted against that holder's two-request share for + its duration, reducing new requests and stopping them when both share slots are occupied; the + busy answer also counts as an answer for #252's silent-source logic. + +### Negative / Trade-offs + +- New outbound traffic at responders: at most 16 signed busy replies a second sustained, about + 85 KiB/s, and 48 (about 250 KiB) in any one second, plus that many signatures. Today a refusal + sends nothing. Each refused request already cost its sender the same signed message. +- A requester asks a refusing holder up to once per 5 s per fetch slot instead of once per 30 s: + at most `C / 5 s` small requests with `C` hardware-thread fetch slots, against `C / 30 s` today, + in the case where every holder refuses everything. Capping `C` is a separate decision. +- Refusals beyond the per-peer busy allowance stay silent and are still charged by saorsa-core. + Without #252, a requester that over-shares by more than two requests at once still meets some + silence; the simulation below gives the residual. +- Busy is trust-neutral. A holder that tries to answer busy to everything can still be charged when + a request did not ask, a reply was suppressed, or a scheduled send did not complete; otherwise it + is passed over for new keys for up to 60 s at a time and remains subject to storage audits. + Penalising "never serves" would need a service audit with reserved capacity, which is not part + of this decision. +- Sixteen identities each spending their per-peer second can use up the global allowance and + deny busy replies to everyone else, and stalled peers can hold the 48 send permits for up to + 2 s each. What they win is the old silence for those peers, not a new failure. The global + allowance is kept small on purpose, because it is what bounds the signed bytes a full node + spends on refusals. + +### Neutral / Operational + +- New counters on the `replication responder admission summary (cumulative)` line: + `fetch_busy_reply_{not_asked,send_pool_full,rate_limited,peer_in_flight,sent,send_failed,timed_out,cancelled}`, + `fetch_busy_received`, `fetch_busy_cooling_skips` and `fetch_busy_sources_tracked`. + `not_asked` against the rest is the share of refused requests that did not ask for a busy reply: + a proxy, and an upper bound, for how much refused traffic still comes from older versions (a + request from a current node can also not ask, through the API or a malformed trailer). + +## Validation + +- Unit tests: + - mirrors of the previous types decode current requests, and the unchanged answers 0-2; + - a current node reads an older request as not asking; + - trailer parsing: exact, wrong magic, truncated, unknown bits, reserved bytes, padding; + - an older requester cannot decode `Busy`; + - limiter: burst, refill, global cap, no spend on a denial, in-flight limit, bounded table; + - the busy wait's range and its cap at the fetch timeout; + - tracker: cooldown, escalation, cap, never shortened, reset, expiry, bounded table; + - a retry passes over a cooling holder but still takes it last; + - a busy reply is not counted as served bytes. +- End to end on a local network, made deterministic with a test-only fetch serve gate (the same + one #252 adds): + - a refused request that asks is told `Busy` at once; + - one that does not ask is refused silently; + - the held requests are served once the gate opens; + - the requester's own fetch worker, refused by a full holder, records the holder as cooling, + keeps the attempt for at least the minimum busy wait, then gives the key up, and the holder's + trust at the requester never drops. +- Measured on real nodes against upstream main with the same test-only gate, in + `tests/e2e/fetch_busy_ab.rs` (opt-in): + - 20 requesters each within their two-request share, the first of five holders with its + 12-slot pool full for 2 s (28 refusals in both builds), which is the refusal left after #252: + all 40 chunks in 10.5 s, no holder charged, against 60.4 s with 14 requesters charging the + first holder and 4 the second; + - one requester with 8 fetch slots over-asking three holders, which #252 removes: 39.7 s, 39.4 s + and 36.7 s with only the first holder charged (4 charges), against 60.2 s, 90.3 s and 90.2 s with + up to all three charged. +- A discrete-event simulation of the 177-node join wave and 24 more scenarios (dead holders, + holders that always answer busy, saturated pools, mixed versions, 1-64 fetch slots, + single-holder keys, limiter and wait sensitivity), comparing today, this change, #252-style + capping, and both. It reproduces the real-node run of this change and overstates the cost of + today's silence, so it gives directions, not numbers. In it no key is stranded in any scenario. + With nobody answering busy it behaves exactly as today. With #252-style capping, the busy reply + removes every false charge in the background-load, overload, always-busy and dense scenarios; + what remains elsewhere comes from requests a slow holder sheds unanswered after the fetch timeout + and from requesters that do not ask. Without capping it raises requests per chunk + by up to 12% in some scenarios and lowers it by up to 17% in others (with a 1-2 s wait instead of +5-10 s it rises 16%, which is why the wait is longer), and keys with a single holder go back to +verification about twice as often. +- In production: + - `fetch_busy_reply_sent` against `fetch_admission_dropped_*`; + - the fall in `connection_timeout` trust events between fleet nodes; + - `fetch_busy_reply_rate_limited` and `send_pool_full` staying small. + + Revisit the 5-10 s wait and the allowances against measured admission hold times. + +## Notes for AI-assisted work + +AI tools may help draft this ADR, but **must not mark it Accepted without human review**. Accepted ADRs are immutable: create a new superseding ADR rather than editing an Accepted ADR. diff --git a/docs/adr/README.md b/docs/adr/README.md index b58e9a32..092768d4 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -40,3 +40,4 @@ See [`TOOLING.md`](./TOOLING.md) for `adrs`, `adr-kit`, and AI harness setup. - [ADR-0018: Pointer ownership transfer by final redirection](./ADR-0018-pointer-transfer-by-final-redirection.md) - [ADR-0019: Pointer audits serve the record round 1 bound](./ADR-0019-pointer-audits-serve-the-record-round-one-bound.md) - [ADR-0022: Remove the LMDB Chunk Store](./ADR-0022-remove-the-lmdb-chunk-store.md) +- [ADR-0025: Tell a fetch requester "busy" instead of refusing silently](./ADR-0025-fetch-busy-reply.md) diff --git a/src/replication/audit_metrics.rs b/src/replication/audit_metrics.rs index de8f77a1..0989d5b2 100644 --- a/src/replication/audit_metrics.rs +++ b/src/replication/audit_metrics.rs @@ -400,6 +400,21 @@ static RESPONDER_ADMISSION_DROPS: [AtomicU64; N_RESPONDER_CLASSES * N_ADMISSION_ static RESPONDER_STALENESS_SHEDS: [AtomicU64; N_RESPONDER_CLASSES] = [const { AtomicU64::new(0) }; N_RESPONDER_CLASSES]; +/// Fetch admission refusals by what became of the busy reply, indexed by +/// [`FetchBusyReplyOutcome::index`]. +static FETCH_BUSY_REPLIES: [AtomicU64; FetchBusyReplyOutcome::COUNT] = + [const { AtomicU64::new(0) }; FetchBusyReplyOutcome::COUNT]; + +/// Busy answers this node received to its own fetches. +static FETCH_BUSY_RECEIVED: AtomicU64 = AtomicU64::new(0); + +/// Times this node, choosing where to start or retry a fetch, passed over a +/// holder that had recently answered it busy. +static FETCH_BUSY_COOLING_SKIPS: AtomicU64 = AtomicU64::new(0); + +/// Holders this node currently remembers as having answered it busy. +static FETCH_BUSY_SOURCES_TRACKED: AtomicU64 = AtomicU64::new(0); + static SERIAL_QUEUE_OVERFLOW_DROPS: [AtomicU64; protocol::N_REPLICATION_VARIANTS] = [const { AtomicU64::new(0) }; protocol::N_REPLICATION_VARIANTS]; @@ -532,6 +547,98 @@ pub fn record_responder_staleness_shed(class: ReplicationResponderClass) { } } +/// What became of a fetch refused at admission, as far as a busy reply goes. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum FetchBusyReplyOutcome { + /// The requester did not ask for busy replies (an older node): silence, + /// as before. + NotAsked, + /// Every busy-reply send permit was in use: silence. + SendPoolFull, + /// Over the per-peer or global busy-reply rate: silence. + RateLimited, + /// A busy reply to the same peer was still being sent: silence. + PeerInFlight, + /// The busy reply was handed to the transport. + Sent, + /// The transport refused the busy reply. + SendFailed, + /// The transport did not take the busy reply in time; abandoned. + TimedOut, + /// Shutdown began before the busy reply was sent. + Cancelled, +} + +impl FetchBusyReplyOutcome { + const COUNT: usize = 8; + + const fn index(self) -> usize { + match self { + Self::NotAsked => 0, + Self::SendPoolFull => 1, + Self::RateLimited => 2, + Self::PeerInFlight => 3, + Self::Sent => 4, + Self::SendFailed => 5, + Self::TimedOut => 6, + Self::Cancelled => 7, + } + } +} + +/// Count what became of one fetch refusal's busy reply. +pub fn record_fetch_busy_reply(outcome: FetchBusyReplyOutcome) { + if let Some(counter) = FETCH_BUSY_REPLIES.get(outcome.index()) { + counter.fetch_add(1, Ordering::Relaxed); + } +} + +/// Fetch refusals with this busy-reply outcome since process start. +#[must_use] +#[cfg_attr(not(feature = "logging"), allow(dead_code))] +pub fn fetch_busy_replies_total(outcome: FetchBusyReplyOutcome) -> u64 { + FETCH_BUSY_REPLIES + .get(outcome.index()) + .map_or(0, |counter| counter.load(Ordering::Relaxed)) +} + +/// Count a busy answer to one of this node's own fetches. +pub fn record_fetch_busy_received() { + FETCH_BUSY_RECEIVED.fetch_add(1, Ordering::Relaxed); +} + +/// Busy answers to this node's own fetches since process start. +#[must_use] +#[cfg_attr(not(feature = "logging"), allow(dead_code))] +pub fn fetch_busy_received_total() -> u64 { + FETCH_BUSY_RECEIVED.load(Ordering::Relaxed) +} + +/// Count a fetch started or retried on a later holder because an earlier one +/// had recently answered busy. +pub fn record_fetch_busy_cooling_skip() { + FETCH_BUSY_COOLING_SKIPS.fetch_add(1, Ordering::Relaxed); +} + +/// Fetches started or retried past a cooling holder since process start. +#[must_use] +#[cfg_attr(not(feature = "logging"), allow(dead_code))] +pub fn fetch_busy_cooling_skips_total() -> u64 { + FETCH_BUSY_COOLING_SKIPS.load(Ordering::Relaxed) +} + +/// Set how many holders this node remembers as having answered it busy. +pub fn set_fetch_busy_sources_tracked(count: usize) { + FETCH_BUSY_SOURCES_TRACKED.store(u64::try_from(count).unwrap_or(u64::MAX), Ordering::Relaxed); +} + +/// Holders this node remembers as having answered it busy. +#[must_use] +#[cfg_attr(not(feature = "logging"), allow(dead_code))] +pub fn fetch_busy_sources_tracked() -> u64 { + FETCH_BUSY_SOURCES_TRACKED.load(Ordering::Relaxed) +} + /// One class's admission pressure, read together for a summary line. #[cfg_attr(not(feature = "logging"), allow(dead_code))] struct ResponderAdmissionSnapshot { @@ -640,6 +747,20 @@ pub fn log_responder_admission_summary() { fetch_admission_dropped_global_pool = fetch.global_pool, fetch_admission_dropped_per_peer_share = fetch.per_peer_share, fetch_staleness_shed = fetch.staleness_shed, + fetch_busy_reply_not_asked = fetch_busy_replies_total(FetchBusyReplyOutcome::NotAsked), + fetch_busy_reply_send_pool_full = + fetch_busy_replies_total(FetchBusyReplyOutcome::SendPoolFull), + fetch_busy_reply_rate_limited = + fetch_busy_replies_total(FetchBusyReplyOutcome::RateLimited), + fetch_busy_reply_peer_in_flight = + fetch_busy_replies_total(FetchBusyReplyOutcome::PeerInFlight), + fetch_busy_reply_sent = fetch_busy_replies_total(FetchBusyReplyOutcome::Sent), + fetch_busy_reply_send_failed = fetch_busy_replies_total(FetchBusyReplyOutcome::SendFailed), + fetch_busy_reply_timed_out = fetch_busy_replies_total(FetchBusyReplyOutcome::TimedOut), + fetch_busy_reply_cancelled = fetch_busy_replies_total(FetchBusyReplyOutcome::Cancelled), + fetch_busy_received = fetch_busy_received_total(), + fetch_busy_cooling_skips = fetch_busy_cooling_skips_total(), + fetch_busy_sources_tracked = fetch_busy_sources_tracked(), verification_admission_dropped_global_pool = verification.global_pool, verification_admission_dropped_per_peer_share = verification.per_peer_share, verification_staleness_shed = verification.staleness_shed, diff --git a/src/replication/fetch_busy.rs b/src/replication/fetch_busy.rs new file mode 100644 index 00000000..e6be386f --- /dev/null +++ b/src/replication/fetch_busy.rs @@ -0,0 +1,745 @@ +//! The fetch busy reply: what each side does when a responder cannot admit a +//! fetch. +//! +//! A responder that refuses a fetch at admission used to stay silent. The +//! requester then waited its whole fetch timeout, and saorsa-core charged the +//! responder a connection timeout it had not earned. A requester that says it +//! understands [`FetchResponse::Busy`] is now told at once instead. +//! +//! Two bounds keep the new reply from costing more than the silence it +//! replaces: +//! +//! - **Responder:** a busy reply is a send the node never made before, caused +//! by a request it is refusing, and like every message it is signed, so it +//! costs a signature and about 5 KiB on the wire. [`FetchBusyReplyLimiter`] +//! caps those sends per peer and overall, and how many to one peer may be in +//! flight at once. +//! Past any cap a refusal is silent again, which is exactly what every +//! refusal was before. +//! - **Requester:** a busy answer is not a reason to ask again at once. +//! [`busy_wait`] keeps the attempt, and so its fetch slot, for a few seconds +//! before the key moves on, so a requester asks at most once per wait per +//! slot instead of as fast as answers come back. The attempt then ends as a +//! source failure, exactly as the silent timeout did, minus the trust charge. +//! [`SourceBusyTracker`] also puts a busy source last when choosing where to +//! start or retry a key, for longer each time it stays busy, so a requester +//! does not keep picking a holder that just said it cannot take more. +//! +//! [`FetchResponse::Busy`]: crate::replication::protocol::FetchResponse::Busy + +use std::collections::HashMap; +use std::sync::Arc; +use std::time::{Duration, Instant}; + +use parking_lot::Mutex; +use saorsa_core::identity::PeerId; + +use super::audit_metrics; + +/// Shortest time a requester waits after a busy answer before moving on. +/// +/// A responder refuses because its admitted fetches, each up to a 4 MiB chunk +/// sent over a home upload link, are still being served. A share frees in +/// seconds, not milliseconds, so asking again sooner would mostly be refused +/// again. +pub const FETCH_BUSY_WAIT_MIN: Duration = Duration::from_secs(5); + +/// Longest time a requester waits after a busy answer before moving on. +/// +/// Waits are spread uniformly between the two bounds, so requesters refused +/// together do not all come back together. +pub const FETCH_BUSY_WAIT_MAX: Duration = Duration::from_secs(10); + +/// The jitter a requester adds to [`FETCH_BUSY_WAIT_MIN`]: drawn uniformly from +/// zero to this, inclusive. +pub const FETCH_BUSY_WAIT_SPREAD: Duration = + FETCH_BUSY_WAIT_MAX.saturating_sub(FETCH_BUSY_WAIT_MIN); + +/// Longest a responder spends handing one busy reply to the transport. +/// +/// A reply that cannot be handed over in this long is abandoned: the requester +/// then gets the silence it would have had anyway. Without it, a peer that is +/// slow to accept data could hold a send permit indefinitely. +pub const FETCH_BUSY_REPLY_SEND_TIMEOUT: Duration = Duration::from_secs(2); + +/// Busy replies one peer may receive back to back, and the most to it that +/// may be in flight at once. +/// +/// The same as the per-peer fetch admission share, so a requester that sent +/// one share too many at once hears about each refusal. +const PEER_BURST: u32 = 2; + +/// Sustained rate of busy replies to one peer: one per this interval. +/// +/// A requester that waits [`FETCH_BUSY_WAIT_MIN`] after each busy answer never +/// comes near it. A peer that ignores the answers and keeps sending gets one +/// reply a second, and silence for the rest. +const PEER_INTERVAL: Duration = Duration::from_secs(1); + +/// Busy replies this node may send back to back across all peers. +/// +/// Every message is signed, which adds about 5 KiB to the wire and a +/// signature's worth of CPU, so the global allowance is what bounds the cost. +/// Burst and rate together allow at most 48 replies, about 250 KiB, in any one +/// second, and 16 a second, about 85 KiB/s, sustained. The burst lets a node +/// that becomes full answer a whole pool's worth of peers at once. +const GLOBAL_BURST: u32 = 32; + +/// Sustained rate of busy replies across all peers: one per this interval, +/// 16 a second. +const GLOBAL_INTERVAL: Duration = Duration::from_micros(62_500); + +/// Most peers whose busy-reply allowance is tracked at once. +/// +/// A peer whose allowance has fully refilled and has no reply in flight +/// carries no information and is dropped first. If every tracked peer is still +/// being limited, a new peer gets silence rather than an entry, so the table +/// cannot grow without bound. +const MAX_TRACKED_PEERS: usize = 4096; + +/// Longest a source that keeps answering busy is put last. +const MAX_SOURCE_COOLDOWN: Duration = Duration::from_secs(60); + +/// Busy answers after which a source's cooldown stops doubling: the wait, +/// then two, four and eight times it, capped at [`MAX_SOURCE_COOLDOWN`]. +const MAX_COOLDOWN_DOUBLINGS: u32 = 3; + +/// How long a source's busy record is kept after its last busy answer. Long +/// enough to remember a holder that stays busy across many cooldowns; short +/// enough that one busy spell is not held against it for good. +const SOURCE_RECORD_TTL: Duration = Duration::from_secs(600); + +/// Most sources whose busy record is kept at once. +const MAX_TRACKED_SOURCES: usize = 4096; + +/// How long to wait after a busy answer, given `remaining`, the time left +/// before the attempt's fetch timeout would have expired. +/// +/// The wait is [`FETCH_BUSY_WAIT_MIN`] plus `jitter`, which the caller draws +/// from `0..=FETCH_BUSY_WAIT_SPREAD` and which is clamped to that range. It +/// never outlasts `remaining`, so a busy answer never leaves a key waiting +/// longer than the silent refusal it replaces. (A fetch timeout shorter than +/// the minimum wait, which only tests configure, shortens the wait with it.) +#[must_use] +pub fn busy_wait(remaining: Duration, jitter: Duration) -> Duration { + FETCH_BUSY_WAIT_MIN + .saturating_add(jitter.min(FETCH_BUSY_WAIT_SPREAD)) + .min(remaining) +} + +/// A generic cell rate limiter: `burst` events back to back, then one per +/// `interval`. +/// +/// It keeps only the theoretical arrival time of the next event, so it needs no +/// floating point and no refill timer. +#[derive(Debug, Default, Clone, Copy)] +struct Gcra { + /// When the allowance is next fully refilled; `None` until first used. + tat: Option, +} + +impl Gcra { + /// The arrival time a further event would move this limiter to, if it is + /// allowed now. + fn admit(self, now: Instant, interval: Duration, burst: u32) -> Option { + let tat = self.tat.map_or(now, |tat| tat.max(now)); + let next = tat.checked_add(interval)?; + (next.saturating_duration_since(now) <= interval.saturating_mul(burst)).then_some(next) + } + + /// Whether the allowance has fully refilled. + fn is_idle(self, now: Instant) -> bool { + self.tat.is_none_or(|tat| tat <= now) + } +} + +/// One peer's busy-reply allowance. +#[derive(Debug, Default, Clone, Copy)] +struct PeerAllowance { + rate: Gcra, + /// Busy replies to this peer being sent now. + in_flight: u32, +} + +impl PeerAllowance { + /// Forgetting it loses nothing: its allowance is full and nothing is in + /// flight. + fn is_idle(self, now: Instant) -> bool { + self.in_flight == 0 && self.rate.is_idle(now) + } +} + +#[derive(Debug, Default)] +struct LimiterState { + global: Gcra, + peers: HashMap, +} + +/// Why a busy reply was not allowed. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum BusyReplyDenied { + /// Over the per-peer or global rate, or the table of peers is full. + RateLimited, + /// The most busy replies to this peer that may be in flight are still + /// being sent. + PeerInFlight, +} + +/// Bounds how many busy replies a responder sends: per peer and overall, by +/// rate, and how many to one peer may be in flight at once. +/// +/// Synchronous and short, so it can be asked on the serial message loop. +#[derive(Debug, Default)] +pub struct FetchBusyReplyLimiter { + state: Mutex, +} + +impl FetchBusyReplyLimiter { + /// A limiter with every allowance full. + #[must_use] + pub fn new() -> Self { + Self::default() + } + + /// Allow one busy reply to `peer` at `now`, if its allowance, the global + /// one and its in-flight limit all permit. Both rate allowances are spent + /// together on a grant and neither on a denial. The returned grant holds + /// one of the peer's in-flight places until it is dropped, which the caller + /// does when the send ends. + /// + /// # Errors + /// + /// Returns why the reply is not allowed. + pub fn try_acquire( + self: &Arc, + peer: &PeerId, + now: Instant, + ) -> Result { + let mut state = self.state.lock(); + if !state.peers.contains_key(peer) && state.peers.len() >= MAX_TRACKED_PEERS { + state.peers.retain(|_, allowance| !allowance.is_idle(now)); + if state.peers.len() >= MAX_TRACKED_PEERS { + return Err(BusyReplyDenied::RateLimited); + } + } + let allowance = state.peers.get(peer).copied().unwrap_or_default(); + if allowance.in_flight >= PEER_BURST { + return Err(BusyReplyDenied::PeerInFlight); + } + let peer_next = allowance + .rate + .admit(now, PEER_INTERVAL, PEER_BURST) + .ok_or(BusyReplyDenied::RateLimited)?; + let global_next = state + .global + .admit(now, GLOBAL_INTERVAL, GLOBAL_BURST) + .ok_or(BusyReplyDenied::RateLimited)?; + state.global.tat = Some(global_next); + state.peers.insert( + *peer, + PeerAllowance { + rate: Gcra { + tat: Some(peer_next), + }, + in_flight: allowance.in_flight.saturating_add(1), + }, + ); + Ok(BusyReplyGrant { + limiter: Arc::clone(self), + peer: *peer, + }) + } + + fn release(&self, peer: &PeerId) { + if let Some(allowance) = self.state.lock().peers.get_mut(peer) { + allowance.in_flight = allowance.in_flight.saturating_sub(1); + } + } + + /// How many peers are tracked now. + #[cfg(test)] + fn tracked_peers(&self) -> usize { + self.state.lock().peers.len() + } +} + +/// A granted busy reply. Holds one of the peer's in-flight places until +/// dropped. +#[derive(Debug)] +pub struct BusyReplyGrant { + limiter: Arc, + peer: PeerId, +} + +impl Drop for BusyReplyGrant { + fn drop(&mut self) { + self.limiter.release(&self.peer); + } +} + +/// One source's recent busy answers to this requester. +#[derive(Debug, Clone, Copy)] +struct SourceBusy { + /// Busy answers since it last served this requester. + strikes: u32, + /// Until when it is put last. + cooling_until: Instant, + /// When it last answered busy. + last_busy: Instant, +} + +impl SourceBusy { + fn is_cooling(self, now: Instant) -> bool { + now < self.cooling_until + } + + fn expired(self, now: Instant) -> bool { + !self.is_cooling(now) && now.saturating_duration_since(self.last_busy) >= SOURCE_RECORD_TTL + } +} + +/// Which sources recently answered this requester busy, and until when they +/// should be asked last. +/// +/// A busy answer is not a fault, so nothing here touches trust. It is local +/// scheduling: a holder that just refused is the worst place to send the next +/// fetch, and one that keeps refusing is put last for longer each time. +/// Serving once clears its record. It orders sources and never withholds one: +/// a key whose every holder is cooling still starts on the first of them, and +/// then waits out that holder's busy answer like any other. So per requester +/// it lowers how often a busy holder is asked; the hard bound is the busy wait +/// per fetch slot. Synchronous and short, so it is safe to consult while +/// holding the fetch queues lock. +#[derive(Debug, Default)] +pub struct SourceBusyTracker { + sources: Mutex>, +} + +impl SourceBusyTracker { + /// A tracker that knows no busy source. + #[must_use] + pub fn new() -> Self { + Self::default() + } + + /// Record a busy answer from `source` at `now`. `wait` is the busy wait + /// before any cap from the attempt's own timeout. Returns how long the + /// source is now put last: `wait` the first time, doubling with each further + /// busy answer before it serves again, up to a minute. A + /// cooldown already running is never shortened. + pub fn record_busy(&self, source: &PeerId, now: Instant, wait: Duration) -> Duration { + let mut sources = self.sources.lock(); + if sources + .get(source) + .is_some_and(|record| record.expired(now)) + { + sources.remove(source); + } + if !sources.contains_key(source) && sources.len() >= MAX_TRACKED_SOURCES { + sources.retain(|_, record| !record.expired(now)); + if sources.len() >= MAX_TRACKED_SOURCES { + // Prefer to forget a source that is not cooling. Forgetting + // one that is ends its cooldown early; the busy wait still + // bounds how often it is asked. + let victim = sources + .iter() + .min_by_key(|(_, record)| (record.is_cooling(now), record.cooling_until)) + .map(|(peer, _)| *peer); + if let Some(peer) = victim { + sources.remove(&peer); + } + } + } + let record = sources.entry(*source).or_insert(SourceBusy { + strikes: 0, + cooling_until: now, + last_busy: now, + }); + record.strikes = record.strikes.saturating_add(1); + let doublings = record.strikes.saturating_sub(1).min(MAX_COOLDOWN_DOUBLINGS); + let cooldown = wait + .saturating_mul(1u32 << doublings) + .min(MAX_SOURCE_COOLDOWN); + let until = now.checked_add(cooldown).unwrap_or(now); + record.cooling_until = record.cooling_until.max(until); + record.last_busy = now; + audit_metrics::set_fetch_busy_sources_tracked(sources.len()); + cooldown + } + + /// `source` served this requester: forget its busy record. + pub fn record_served(&self, source: &PeerId) { + let mut sources = self.sources.lock(); + if sources.remove(source).is_some() { + audit_metrics::set_fetch_busy_sources_tracked(sources.len()); + } + } + + /// Whether `source` should be asked last at `now`. + #[must_use] + pub fn is_cooling(&self, source: &PeerId, now: Instant) -> bool { + self.sources + .lock() + .get(source) + .is_some_and(|record| record.is_cooling(now)) + } + + /// Of `candidates`, in their order of preference, the first that is not + /// cooling at `now`, or the first of them when all are. Counts a skip when + /// that passes over the first candidate. + pub fn choose<'a, I>(&self, candidates: I, now: Instant) -> Option<&'a PeerId> + where + I: IntoIterator, + { + let tracked = self.sources.lock(); + let mut first = None; + for candidate in candidates { + let first_seen = *first.get_or_insert(candidate); + if tracked + .get(candidate) + .is_none_or(|record| !record.is_cooling(now)) + { + if !std::ptr::eq(first_seen, candidate) { + audit_metrics::record_fetch_busy_cooling_skip(); + } + return Some(candidate); + } + } + first + } + + /// When `source` last answered busy, if it is still remembered. + #[must_use] + pub fn last_busy(&self, source: &PeerId) -> Option { + self.sources + .lock() + .get(source) + .map(|record| record.last_busy) + } + + /// How many sources are tracked now. + #[must_use] + pub fn len(&self) -> usize { + self.sources.lock().len() + } + + /// Whether no source is tracked. + #[must_use] + pub fn is_empty(&self) -> bool { + self.sources.lock().is_empty() + } +} + +#[cfg(test)] +#[allow(clippy::unwrap_used, clippy::expect_used)] +mod tests { + use super::*; + + fn peer(i: u32) -> PeerId { + let mut bytes = [0u8; 32]; + for (byte, value) in bytes.iter_mut().zip(i.to_le_bytes()) { + *byte = value; + } + PeerId::from_bytes(bytes) + } + + fn limiter() -> Arc { + Arc::new(FetchBusyReplyLimiter::new()) + } + + /// Grant and immediately end the send, as a reply that went out at once. + fn grant(limiter: &Arc, p: &PeerId, now: Instant) -> bool { + limiter.try_acquire(p, now).is_ok() + } + + #[test] + fn busy_wait_spans_the_jitter_range() { + let remaining = Duration::from_secs(30); + assert_eq!(busy_wait(remaining, Duration::ZERO), FETCH_BUSY_WAIT_MIN); + assert_eq!( + busy_wait(remaining, FETCH_BUSY_WAIT_SPREAD), + FETCH_BUSY_WAIT_MAX + ); + assert_eq!( + busy_wait(remaining, Duration::from_secs(2)), + FETCH_BUSY_WAIT_MIN + Duration::from_secs(2) + ); + // Jitter past the spread is clamped, not added. + assert_eq!( + busy_wait(remaining, Duration::from_secs(3600)), + FETCH_BUSY_WAIT_MAX + ); + } + + #[test] + fn busy_wait_never_outlasts_the_fetch_timeout() { + assert_eq!( + busy_wait(Duration::from_secs(3), FETCH_BUSY_WAIT_SPREAD), + Duration::from_secs(3) + ); + assert_eq!( + busy_wait(Duration::ZERO, FETCH_BUSY_WAIT_SPREAD), + Duration::ZERO + ); + } + + #[test] + fn a_peer_gets_its_burst_then_one_reply_per_interval() { + let limiter = limiter(); + let p = peer(1); + let t0 = Instant::now(); + for _ in 0..PEER_BURST { + assert!(grant(&limiter, &p, t0)); + } + assert_eq!( + limiter.try_acquire(&p, t0).err(), + Some(BusyReplyDenied::RateLimited) + ); + assert!(!grant(&limiter, &p, t0 + PEER_INTERVAL / 2)); + assert!(grant(&limiter, &p, t0 + PEER_INTERVAL)); + assert!(!grant(&limiter, &p, t0 + PEER_INTERVAL)); + // A long quiet spell refills only up to the burst, never beyond it. + let later = t0 + PEER_INTERVAL * 100; + for _ in 0..PEER_BURST { + assert!(grant(&limiter, &p, later)); + } + assert!(!grant(&limiter, &p, later)); + } + + #[test] + fn at_most_a_burst_in_flight_per_peer() { + let limiter = limiter(); + let p = peer(1); + let t0 = Instant::now(); + let held: Vec = (0..PEER_BURST) + .map(|_| limiter.try_acquire(&p, t0).expect("within the burst")) + .collect(); + // Long enough later that the rate allows more, but the sends have not + // ended: the in-flight limit refuses. + let later = t0 + PEER_INTERVAL * 10; + assert_eq!( + limiter.try_acquire(&p, later).err(), + Some(BusyReplyDenied::PeerInFlight) + ); + // Another peer is not affected. + assert!(grant(&limiter, &peer(2), later)); + drop(held); + assert!(grant(&limiter, &p, later)); + } + + #[test] + fn one_peer_exhausting_its_allowance_leaves_others_theirs() { + let limiter = limiter(); + let t0 = Instant::now(); + for _ in 0..10 { + let _ = grant(&limiter, &peer(1), t0); + } + assert!(grant(&limiter, &peer(2), t0)); + } + + #[test] + fn the_global_allowance_caps_every_peer_together() { + let limiter = limiter(); + let t0 = Instant::now(); + let granted = (0..1_000u32) + .filter(|i| grant(&limiter, &peer(*i), t0)) + .count(); + assert_eq!(granted, usize::try_from(GLOBAL_BURST).unwrap()); + // Refills at the global rate: one second later, one second's worth. + let t1 = t0 + Duration::from_secs(1); + let granted = (1_000..2_000u32) + .filter(|i| grant(&limiter, &peer(*i), t1)) + .count(); + let per_second = Duration::from_secs(1).as_nanos() / GLOBAL_INTERVAL.as_nanos(); + assert_eq!(granted, usize::try_from(per_second).unwrap()); + } + + #[test] + fn a_global_refusal_spends_nothing_from_the_peer() { + let limiter = limiter(); + let t0 = Instant::now(); + for i in 0..GLOBAL_BURST { + assert!(grant(&limiter, &peer(i), t0)); + } + let p = peer(100_000); + assert!(!grant(&limiter, &p, t0)); + assert_eq!( + limiter.tracked_peers(), + usize::try_from(GLOBAL_BURST).unwrap() + ); + // Once the global allowance refills, the peer still has its whole burst. + let t1 = t0 + Duration::from_secs(1); + for _ in 0..PEER_BURST { + assert!(grant(&limiter, &p, t1)); + } + } + + #[test] + fn the_peer_table_stays_bounded_and_forgets_only_idle_peers() { + let limiter = limiter(); + let mut now = Instant::now(); + let mut next_peer = 0u32; + // Fill the table with replies still in flight, pacing to stay inside + // the global allowance. + let mut held = Vec::new(); + while held.len() < MAX_TRACKED_PEERS { + match limiter.try_acquire(&peer(next_peer), now) { + Ok(grant) => { + held.push(grant); + next_peer += 1; + } + Err(_) => now += GLOBAL_INTERVAL, + } + } + assert_eq!(limiter.tracked_peers(), MAX_TRACKED_PEERS); + // No entry can be forgotten, so a newcomer is refused rather than + // tracked, although the global allowance has long refilled. + let newcomer = peer(u32::MAX); + let later = now + Duration::from_secs(10); + assert_eq!( + limiter.try_acquire(&newcomer, later).err(), + Some(BusyReplyDenied::RateLimited) + ); + assert_eq!(limiter.tracked_peers(), MAX_TRACKED_PEERS); + // Once those replies end, idle entries make room. + drop(held); + assert!(grant(&limiter, &newcomer, later)); + assert!(limiter.tracked_peers() <= MAX_TRACKED_PEERS); + } + + #[test] + fn a_peer_with_a_reply_in_flight_is_never_forgotten() { + let limiter = limiter(); + let t0 = Instant::now(); + let held: Vec = (0..PEER_BURST) + .map(|_| limiter.try_acquire(&peer(0), t0).expect("grant")) + .collect(); + let mut now = t0; + for i in 1..u32::try_from(MAX_TRACKED_PEERS).unwrap() { + while !grant(&limiter, &peer(i), now) { + now += GLOBAL_INTERVAL; + } + } + // Long after, everyone else is idle and a full table sheds them, but + // never the peer whose reply is still in flight. + let later = now + Duration::from_secs(3600); + assert!(grant(&limiter, &peer(u32::MAX), later)); + assert_eq!( + limiter.try_acquire(&peer(0), later).err(), + Some(BusyReplyDenied::PeerInFlight) + ); + drop(held); + assert!(grant(&limiter, &peer(0), later)); + } + + const WAIT: Duration = Duration::from_secs(5); + + #[test] + fn a_busy_source_is_put_last_while_it_cools() { + let tracker = SourceBusyTracker::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let sources = [a, b, c]; + let t0 = Instant::now(); + assert_eq!(tracker.choose(&sources, t0), Some(&a)); + assert_eq!(tracker.record_busy(&a, t0, WAIT), WAIT); + assert!(tracker.is_cooling(&a, t0)); + assert_eq!(tracker.choose(&sources, t0), Some(&b)); + // Cooling ends after the wait. + assert_eq!(tracker.choose(&sources, t0 + WAIT), Some(&a)); + } + + #[test] + fn when_every_source_cools_the_first_is_still_chosen() { + let tracker = SourceBusyTracker::new(); + let (a, b) = (peer(1), peer(2)); + let t0 = Instant::now(); + tracker.record_busy(&a, t0, WAIT); + tracker.record_busy(&b, t0, WAIT); + assert_eq!(tracker.choose(&[a, b], t0), Some(&a)); + assert_eq!(tracker.choose(&[], t0), None); + } + + #[test] + fn a_source_that_stays_busy_cools_longer_up_to_the_cap() { + let tracker = SourceBusyTracker::new(); + let a = peer(1); + let t0 = Instant::now(); + let cooldowns: Vec = (0..6).map(|_| tracker.record_busy(&a, t0, WAIT)).collect(); + assert_eq!( + cooldowns, + vec![WAIT, WAIT * 2, WAIT * 4, WAIT * 8, WAIT * 8, WAIT * 8] + ); + // A longer wait reaches the cap. + let b = peer(2); + let long = Duration::from_secs(10); + for _ in 0..4 { + tracker.record_busy(&b, t0, long); + } + assert_eq!(tracker.record_busy(&b, t0, long), MAX_SOURCE_COOLDOWN); + } + + #[test] + fn a_running_cooldown_is_never_shortened() { + let tracker = SourceBusyTracker::new(); + let a = peer(1); + let t0 = Instant::now(); + for _ in 0..4 { + tracker.record_busy(&a, t0, Duration::from_secs(10)); + } + // Cooling until t0 + 60 s now. A later busy answer with a tiny wait + // would end far sooner; the running cooldown stands. + tracker.record_busy(&a, t0 + Duration::from_secs(1), Duration::from_millis(1)); + assert!(tracker.is_cooling(&a, t0 + Duration::from_secs(59))); + } + + #[test] + fn serving_once_clears_the_record() { + let tracker = SourceBusyTracker::new(); + let a = peer(1); + let t0 = Instant::now(); + for _ in 0..3 { + tracker.record_busy(&a, t0, WAIT); + } + tracker.record_served(&a); + assert!(!tracker.is_cooling(&a, t0)); + assert!(tracker.is_empty()); + assert_eq!(tracker.record_busy(&a, t0, WAIT), WAIT); + } + + #[test] + fn a_stale_record_starts_over() { + let tracker = SourceBusyTracker::new(); + let a = peer(1); + let t0 = Instant::now(); + for _ in 0..3 { + tracker.record_busy(&a, t0, WAIT); + } + // Long after its last busy answer, the next one counts as the first. + let later = t0 + SOURCE_RECORD_TTL + MAX_SOURCE_COOLDOWN; + assert_eq!(tracker.record_busy(&a, later, WAIT), WAIT); + } + + #[test] + fn the_source_table_stays_bounded_and_spares_cooling_sources() { + let tracker = SourceBusyTracker::new(); + let t0 = Instant::now(); + // One source whose cooldown has ended, the rest still cooling. + tracker.record_busy(&peer(0), t0, Duration::from_millis(1)); + let t1 = t0 + Duration::from_secs(1); + let max = u32::try_from(MAX_TRACKED_SOURCES).unwrap(); + for i in 1..max { + tracker.record_busy(&peer(i), t1, WAIT); + } + assert_eq!(tracker.len(), MAX_TRACKED_SOURCES); + let newcomer = peer(u32::MAX); + tracker.record_busy(&newcomer, t1, WAIT); + assert_eq!(tracker.len(), MAX_TRACKED_SOURCES); + assert!(tracker.is_cooling(&newcomer, t1)); + // The one that had stopped cooling made room; every cooling one stayed. + assert!((1..max).all(|i| tracker.is_cooling(&peer(i), t1))); + // Long after, expired records make room before anything is displaced. + let later = t1 + SOURCE_RECORD_TTL + MAX_SOURCE_COOLDOWN; + tracker.record_busy(&peer(u32::MAX - 1), later, WAIT); + assert_eq!(tracker.len(), 1); + } +} diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 62937413..7288160e 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -22,6 +22,7 @@ pub mod bootstrap; pub mod commitment; pub mod commitment_state; pub mod config; +pub mod fetch_busy; pub mod fresh; pub mod neighbor_sync; pub mod paid_list; @@ -42,6 +43,8 @@ use std::fmt; use std::num::NonZeroUsize; use std::ops::ControlFlow; use std::path::{Path, PathBuf}; +#[cfg(any(test, feature = "test-utils"))] +use std::sync::atomic::AtomicBool; use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::Arc; use std::time::{Duration, Instant, SystemTime}; @@ -56,7 +59,7 @@ use parking_lot::Mutex; use rand::rngs::StdRng; use rand::{Rng, SeedableRng}; use tokio::sync::broadcast::error::RecvError; -use tokio::sync::{mpsc, Notify, RwLock, Semaphore}; +use tokio::sync::{mpsc, Notify, OwnedSemaphorePermit, RwLock, Semaphore}; use tokio::task::JoinHandle; use tokio_util::sync::CancellationToken; use tokio_util::task::TaskTracker; @@ -72,7 +75,7 @@ use crate::pointer::{FinalStateWitness, PointerService}; use crate::replication::audit::AuditTickResult; use crate::replication::audit_coordinator::AuditChallengeCoordinator; use crate::replication::audit_metrics::{ - AuditResponderClass, AuditResponderDropReason, AuditResponderMetrics, + AuditResponderClass, AuditResponderDropReason, AuditResponderMetrics, FetchBusyReplyOutcome, ReplicationResponderClass, ResponderAdmissionCeiling, }; use crate::replication::commitment::{commitment_hash, StorageCommitment}; @@ -88,6 +91,10 @@ use crate::replication::config::{ SUBTREE_AUDIT_PROTOCOL_ID, SUBTREE_ROUND1_WORK_BURST_BYTES, SUBTREE_ROUND1_WORK_REFILL_BYTES_PER_SEC, SUBTREE_SESSION_TTL, }; +use crate::replication::fetch_busy::{ + busy_wait, BusyReplyDenied, BusyReplyGrant, FetchBusyReplyLimiter, SourceBusyTracker, + FETCH_BUSY_REPLY_SEND_TIMEOUT, FETCH_BUSY_WAIT_SPREAD, +}; use crate::replication::paid_list::PaidList; use crate::replication::protocol::{ FreshReplicationResponse, NeighborSyncResponse, ReplicationMessage, ReplicationMessageBody, @@ -1129,6 +1136,14 @@ const FETCH_RESPONDER_MAX_OUTSTANDING: usize = FETCH_RESPONDER_WORKER_LIMIT * 4; /// of the three workers available to other peers under a single-source flood. const FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER: u32 = 2; +/// Maximum busy replies being sent at once. +/// +/// [`FetchBusyReplyLimiter`] bounds how often busy replies are sent; this +/// bounds how many sends are open at once, so peers that are slow to take a +/// reply cannot pile up send tasks. Past it, a refusal falls back to the +/// silence every refusal got before busy replies existed. +const FETCH_BUSY_REPLY_MAX_OUTSTANDING: usize = FETCH_RESPONDER_MAX_OUTSTANDING * 4; + /// Maximum verification batches served concurrently. /// /// Point lookups are fast, but a batch can contain 8,192 of them. Two @@ -1787,6 +1802,20 @@ pub struct ReplicationEngine { fetch_responder_admission_semaphore: Arc, /// Per-source fetch responder counts for flood-fair admission. fetch_responder_inflight: Arc>>, + /// Test-only hold on serving admitted fetch requests. + #[cfg(any(test, feature = "test-utils"))] + fetch_serve_gate: Arc, + /// Test-only count of this engine's fetch refusals because its admission + /// pool was full. + #[cfg(any(test, feature = "test-utils"))] + fetch_pool_full_refusals: Arc, + /// Permits for busy replies being sent to fetch requests refused at + /// admission. + fetch_busy_reply_semaphore: Arc, + /// Per-peer and global rate bound on those busy replies. + fetch_busy_reply_limiter: Arc, + /// Sources that recently answered this node's own fetches busy. + fetch_busy_sources: Arc, /// Worker permits for lookup-heavy verification batches. verification_responder_worker_semaphore: Arc, /// Admission permits bounding verification workers and queued waiters. @@ -1940,6 +1969,13 @@ impl ReplicationEngine { FETCH_RESPONDER_MAX_OUTSTANDING, )), fetch_responder_inflight: Arc::new(RwLock::new(HashMap::new())), + #[cfg(any(test, feature = "test-utils"))] + fetch_serve_gate: Arc::new(FetchServeGate::default()), + #[cfg(any(test, feature = "test-utils"))] + fetch_pool_full_refusals: Arc::new(AtomicU64::new(0)), + fetch_busy_reply_semaphore: Arc::new(Semaphore::new(FETCH_BUSY_REPLY_MAX_OUTSTANDING)), + fetch_busy_reply_limiter: Arc::new(FetchBusyReplyLimiter::new()), + fetch_busy_sources: Arc::new(SourceBusyTracker::new()), verification_responder_worker_semaphore: Arc::new(Semaphore::new( VERIFICATION_RESPONDER_WORKER_LIMIT, )), @@ -2172,6 +2208,53 @@ impl ReplicationEngine { queues.enqueue_fetch(key, distance, sources) } + /// Test-only: hold this engine's admitted fetch requests before they are + /// served, until [`Self::open_fetch_serve_gate_for_test`]. + #[cfg(any(test, feature = "test-utils"))] + pub fn close_fetch_serve_gate_for_test(&self) { + self.fetch_serve_gate.close(); + } + + /// Test-only: release this engine's held fetch requests and stop holding + /// new ones. + #[cfg(any(test, feature = "test-utils"))] + pub fn open_fetch_serve_gate_for_test(&self) { + self.fetch_serve_gate.open(); + } + + /// Test-only: whether this engine puts `source` last when choosing where + /// to start or retry a fetch, because it recently answered busy. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn fetch_source_cooling_for_test(&self, source: &PeerId) -> bool { + self.fetch_busy_sources.is_cooling(source, Instant::now()) + } + + /// Test-only: when `source` last answered this engine's fetches busy, if + /// it is still remembered. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn fetch_source_last_busy_for_test(&self, source: &PeerId) -> Option { + self.fetch_busy_sources.last_busy(source) + } + + /// Test-only: fetch requests this engine has admitted and not yet finished + /// serving. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn fetch_admitted_for_test(&self) -> usize { + FETCH_RESPONDER_MAX_OUTSTANDING + .saturating_sub(self.fetch_responder_admission_semaphore.available_permits()) + } + + /// Test-only: fetch requests this engine has refused because its + /// admission pool was full. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn fetch_pool_full_refusals_for_test(&self) -> u64 { + self.fetch_pool_full_refusals.load(Ordering::Relaxed) + } + /// Test-only: whether `key` is still tracked in any fetch-pipeline stage /// (pending verification, fetch queue, or in-flight fetch). #[cfg(any(test, feature = "test-utils"))] @@ -2434,6 +2517,12 @@ impl ReplicationEngine { } } + // A test may shut down with the fetch serve gate still closed. The + // requests it holds wait on the gate before they reach a worker pool, + // so release them, or the drain below never finishes. + #[cfg(any(test, feature = "test-utils"))] + self.fetch_serve_gate.open(); + // Close the responder worker pools. Detached tasks still queued for a // worker take the `Err` arm of `acquire_owned()` and exit immediately // rather than waiting for a slot that will never free up — without @@ -3063,6 +3152,12 @@ impl ReplicationEngine { let fetch_responder_admission_semaphore = Arc::clone(&self.fetch_responder_admission_semaphore); let fetch_responder_inflight = Arc::clone(&self.fetch_responder_inflight); + #[cfg(any(test, feature = "test-utils"))] + let fetch_serve_gate = Arc::clone(&self.fetch_serve_gate); + #[cfg(any(test, feature = "test-utils"))] + let fetch_pool_full_refusals = Arc::clone(&self.fetch_pool_full_refusals); + let fetch_busy_reply_semaphore = Arc::clone(&self.fetch_busy_reply_semaphore); + let fetch_busy_reply_limiter = Arc::clone(&self.fetch_busy_reply_limiter); let verification_responder_worker_semaphore = Arc::clone(&self.verification_responder_worker_semaphore); let verification_responder_admission_semaphore = @@ -3119,6 +3214,12 @@ impl ReplicationEngine { fetch_responder_worker_semaphore, fetch_responder_admission_semaphore, fetch_responder_inflight, + #[cfg(any(test, feature = "test-utils"))] + fetch_serve_gate, + #[cfg(any(test, feature = "test-utils"))] + fetch_pool_full_refusals, + fetch_busy_reply_semaphore, + fetch_busy_reply_limiter, verification_responder_worker_semaphore, verification_responder_admission_semaphore, verification_responder_inflight, @@ -3881,6 +3982,7 @@ impl ReplicationEngine { let is_bootstrapping = Arc::clone(&self.is_bootstrapping); let bootstrap_complete_notify = Arc::clone(&self.bootstrap_complete_notify); let detached_task_tracker = self.detached_task_tracker.clone(); + let busy_sources = Arc::clone(&self.fetch_busy_sources); let concurrency = max_parallel_fetch(); info!("Fetch worker concurrency set to {concurrency} (hardware threads)"); @@ -3899,7 +4001,11 @@ impl ReplicationEngine { break; }; let fetch_key = candidate.key; - let Some(&source) = candidate.sources.first() else { + // A source that just answered this node busy is the + // worst place to start a new key; start on the first + // that is not cooling down, or the first if all are. + let Some(&source) = busy_sources.choose(&candidate.sources, Instant::now()) + else { warn!( "Fetch candidate {} has no sources; requeueing for verification", hex::encode(fetch_key) @@ -3915,6 +4021,7 @@ impl ReplicationEngine { let p2p = Arc::clone(&p2p); let storage = Arc::clone(&storage); let config = Arc::clone(&config); + let busy_sources = Arc::clone(&busy_sources); let token = shutdown.clone(); let tracker = detached_task_tracker.clone(); in_flight.push(Box::pin(async move { @@ -3930,7 +4037,7 @@ impl ReplicationEngine { result: FetchResult::SourceFailed, }, outcome = execute_single_fetch( - p2p, storage, config, fetch_key, source, + p2p, storage, config, busy_sources, fetch_key, source, ) => outcome, } }); @@ -3969,6 +4076,7 @@ impl ReplicationEngine { &key, &outcome.result, config.verification_request_timeout, + &busy_sources, ) { FetchFollowUp::Terminal => true, FetchFollowUp::RequeuedForVerification => false, @@ -3977,6 +4085,7 @@ impl ReplicationEngine { let p2p = Arc::clone(&p2p); let storage = Arc::clone(&storage); let config = Arc::clone(&config); + let busy_sources = Arc::clone(&busy_sources); let token = shutdown.clone(); let tracker = detached_task_tracker.clone(); let fetch_key = key; @@ -3990,7 +4099,12 @@ impl ReplicationEngine { result: FetchResult::SourceFailed, }, outcome = execute_single_fetch( - p2p, storage, config, fetch_key, next_peer, + p2p, + storage, + config, + busy_sources, + fetch_key, + next_peer, ) => outcome, } }); @@ -4589,6 +4703,15 @@ struct ReplicationMessageHandlerContext { fetch_responder_worker_semaphore: Arc, fetch_responder_admission_semaphore: Arc, fetch_responder_inflight: Arc>>, + /// Test-only hold on serving admitted fetch requests. + #[cfg(any(test, feature = "test-utils"))] + fetch_serve_gate: Arc, + /// Test-only count of this engine's fetch refusals because its admission + /// pool was full. + #[cfg(any(test, feature = "test-utils"))] + fetch_pool_full_refusals: Arc, + fetch_busy_reply_semaphore: Arc, + fetch_busy_reply_limiter: Arc, verification_responder_worker_semaphore: Arc, verification_responder_admission_semaphore: Arc, verification_responder_inflight: Arc>>, @@ -7112,6 +7235,45 @@ pub fn verification_requests_for_key_from_for_test(requester: &PeerId, key: &Xor }) } +/// Test-only: holds an engine's admitted fetch requests before they are served. +/// +/// A held request keeps its admission slot and its requester's per-peer share, +/// exactly as a request waiting for a worker does, so a test can read what a +/// requester has outstanding without racing the speed at which the host serves +/// a chunk. One per engine, so closing it affects no other node or test. +#[cfg(any(test, feature = "test-utils"))] +#[derive(Debug, Default)] +struct FetchServeGate { + closed: AtomicBool, + opened: Notify, +} + +#[cfg(any(test, feature = "test-utils"))] +impl FetchServeGate { + fn close(&self) { + self.closed.store(true, Ordering::Release); + } + + fn open(&self) { + self.closed.store(false, Ordering::Release); + self.opened.notify_waiters(); + } + + /// Wait while the gate is closed. The wakeup is registered before the flag + /// is read, so an open that lands between the two is not missed. + async fn wait_open(&self) { + loop { + let opened = self.opened.notified(); + tokio::pin!(opened); + opened.as_mut().enable(); + if !self.closed.load(Ordering::Acquire) { + return; + } + opened.await; + } + } +} + async fn handle_verification_request( source: &PeerId, request: &protocol::VerificationRequest, @@ -7351,11 +7513,42 @@ async fn dispatch_fetch_request( ReplicationResponderClass::Fetch, failure.reason.into(), ); + #[cfg(any(test, feature = "test-utils"))] + if failure.reason == ResponderRejectReason::GlobalPoolFull { + ctx.fetch_pool_full_refusals.fetch_add(1, Ordering::Relaxed); + } + let busy_reply = fetch_busy_reply_permit(&source, request.busy_reply, ctx); + let capability = if request.busy_reply { + "asked" + } else { + "not_asked" + }; + let reply = if busy_reply.is_some() { + "scheduled" + } else { + "none" + }; + // Same line and counter as before busy replies existed, so the + // existing refusal counts carry on. `capability` says whether the + // requester asked for a busy reply, and `reply` whether one was + // scheduled; whether it was delivered is in the busy-reply counters. warn!( responder_class = "fetch", source = %source, + capability, + reply, "Fetch response dropped at admission: {failure}" ); + if let Some((permit, grant)) = busy_reply { + send_fetch_busy_reply( + source, + request.key, + ctx, + request_id, + rr_message_id, + (permit, grant), + ); + } return Ok(()); } }; @@ -7365,8 +7558,12 @@ async fn dispatch_fetch_request( let p2p_node = Arc::clone(&ctx.p2p_node); let request_timeout = ctx.config.fetch_request_timeout; let rr_message_id = rr_message_id.map(ToOwned::to_owned); + #[cfg(any(test, feature = "test-utils"))] + let serve_gate = Arc::clone(&ctx.fetch_serve_gate); ctx.detached_task_tracker.spawn(async move { let _guard = guard; + #[cfg(any(test, feature = "test-utils"))] + serve_gate.wait_open().await; let Ok(_worker_permit) = worker_semaphore.acquire_owned().await else { debug!( responder_class = "fetch", @@ -7402,6 +7599,91 @@ async fn dispatch_fetch_request( Ok(()) } +/// Whether a fetch refused at admission may be answered `Busy`: the send +/// permit and the limiter's grant to do it with. +/// +/// Only a requester that asked for busy replies gets one, and only while a +/// send permit is free, within the limiter's per-peer and global allowances, +/// and while fewer than the limiter's in-flight limit of busy replies to that +/// peer (two) are still being sent. Every other +/// refusal stays silent, exactly as all of them were before busy replies +/// existed. +fn fetch_busy_reply_permit( + source: &PeerId, + busy_reply: bool, + ctx: &ReplicationMessageHandlerContext, +) -> Option<(OwnedSemaphorePermit, BusyReplyGrant)> { + if !busy_reply { + audit_metrics::record_fetch_busy_reply(FetchBusyReplyOutcome::NotAsked); + return None; + } + // The send permit first: if none is free, the reply would not be sent, so + // the rate allowance is left unspent. + let Ok(permit) = Arc::clone(&ctx.fetch_busy_reply_semaphore).try_acquire_owned() else { + audit_metrics::record_fetch_busy_reply(FetchBusyReplyOutcome::SendPoolFull); + return None; + }; + match ctx + .fetch_busy_reply_limiter + .try_acquire(source, Instant::now()) + { + Ok(grant) => Some((permit, grant)), + Err(denied) => { + audit_metrics::record_fetch_busy_reply(match denied { + BusyReplyDenied::RateLimited => FetchBusyReplyOutcome::RateLimited, + BusyReplyDenied::PeerInFlight => FetchBusyReplyOutcome::PeerInFlight, + }); + None + } + } +} + +/// Tell a requester its fetch was refused at admission, off the serial message +/// loop. +/// +/// The send runs on the detached task tracker, holding the send permit and +/// one of the peer's in-flight places until it ends, so a slow or stalled peer +/// can neither block other replication traffic nor hold more permits than its +/// in-flight limit (two). It +/// is abandoned after [`FETCH_BUSY_REPLY_SEND_TIMEOUT`], and it races the +/// engine's shutdown, so `shutdown()` never waits on a peer that is slow to +/// take the reply. +fn send_fetch_busy_reply( + source: PeerId, + key: XorName, + ctx: &ReplicationMessageHandlerContext, + request_id: u64, + rr_message_id: Option<&str>, + held: (OwnedSemaphorePermit, BusyReplyGrant), +) { + let p2p_node = Arc::clone(&ctx.p2p_node); + let shutdown = ctx.shutdown.clone(); + let rr_message_id = rr_message_id.map(ToOwned::to_owned); + ctx.detached_task_tracker.spawn(async move { + let _held = held; + let delivery = tokio::time::timeout( + FETCH_BUSY_REPLY_SEND_TIMEOUT, + send_replication_response_checked( + &source, + &p2p_node, + request_id, + ReplicationMessageBody::FetchResponse(protocol::FetchResponse::Busy { key }), + rr_message_id.as_deref(), + None, + ), + ); + let outcome = tokio::select! { + () = shutdown.cancelled() => FetchBusyReplyOutcome::Cancelled, + sent = delivery => match sent { + Ok(true) => FetchBusyReplyOutcome::Sent, + Ok(false) => FetchBusyReplyOutcome::SendFailed, + Err(_) => FetchBusyReplyOutcome::TimedOut, + }, + }; + audit_metrics::record_fetch_busy_reply(outcome); + }); +} + fn request_is_stale(received_at: Instant, timeout: Duration) -> bool { received_at.elapsed() >= timeout } @@ -7666,15 +7948,7 @@ async fn send_replication_response_checked( return false; } }; - // V2-684: per-peer served-bytes attribution for the heavy serve paths. - // `FetchResponse` carries ~99% of served bytes; `NeighborSyncResponse` is - // included for completeness. Other response variants (verification/audit/ - // commitment) are intentionally excluded — the round-2 audit reply is now a - // few-KB verified slice (V2-685), not a full-chunk transfer, so it is light. - if matches!( - msg.body, - ReplicationMessageBody::FetchResponse(_) | ReplicationMessageBody::NeighborSyncResponse(_) - ) { + if counts_as_served(&msg.body) { protocol::record_served(peer, encoded.len()); } let protocol = protocol.unwrap_or_else(|| response_protocol_for(&msg.body)); @@ -7692,6 +7966,26 @@ async fn send_replication_response_checked( true } +/// Whether a response's bytes count towards the peer it is sent to in the +/// per-peer served-bytes attribution (V2-684). +/// +/// That attribution covers the heavy serve paths. `FetchResponse` carries ~99% +/// of served bytes; `NeighborSyncResponse` is included for completeness. Other +/// response variants (verification/audit/commitment) are intentionally +/// excluded — the round-2 audit reply is now a few-KB verified slice (V2-685), +/// not a full-chunk transfer, so it is light. A busy reply serves nothing, so +/// it is left out too. +const fn counts_as_served(body: &ReplicationMessageBody) -> bool { + matches!( + body, + ReplicationMessageBody::FetchResponse( + protocol::FetchResponse::Success { .. } + | protocol::FetchResponse::NotFound { .. } + | protocol::FetchResponse::Error { .. } + ) | ReplicationMessageBody::NeighborSyncResponse(_) + ) +} + async fn record_sent_replica_hints( peer: &PeerId, hints: &[neighbor_sync::SentReplicaHint], @@ -9013,6 +9307,7 @@ fn apply_fetch_result( key: &XorName, result: &FetchResult, verification_retry_after: Duration, + busy_sources: &SourceBusyTracker, ) -> FetchFollowUp { match result { FetchResult::Stored | FetchResult::NoLongerResponsible | FetchResult::AlreadyHeld => { @@ -9020,7 +9315,12 @@ fn apply_fetch_result( FetchFollowUp::Terminal } FetchResult::IntegrityFailed | FetchResult::SourceFailed => { - if let Some(next_peer) = q.retry_fetch(key) { + // The next untried source in preference order, passing over any + // that recently answered this node busy unless all of them have. + let now = Instant::now(); + if let Some(next_peer) = + q.retry_fetch_with(key, |untried| busy_sources.choose(untried, now).copied()) + { FetchFollowUp::RetryFrom(next_peer) } else if q.requeue_fetch_for_verification(key, verification_retry_after) { FetchFollowUp::RequeuedForVerification @@ -9090,6 +9390,7 @@ async fn execute_single_fetch( p2p_node: Arc, storage: Arc, config: Arc, + busy_sources: Arc, key: XorName, source: PeerId, ) -> FetchOutcome { @@ -9147,7 +9448,10 @@ async fn execute_single_fetch( }; } - let request = protocol::FetchRequest { key }; + let request = protocol::FetchRequest { + key, + busy_reply: true, + }; let msg = ReplicationMessage { request_id: rand::thread_rng().gen::(), body: ReplicationMessageBody::FetchRequest(request), @@ -9164,6 +9468,7 @@ async fn execute_single_fetch( } }; + let sent_at = Instant::now(); let result = p2p_node .send_request( &source, @@ -9257,6 +9562,10 @@ async fn execute_single_fetch( }; } + // It served a valid chunk, so any earlier busy answers no + // longer count against it when choosing where to start. + busy_sources.record_served(&source); + // Responsibility can lapse during the network round-trip. // The bandwidth is already spent; declining the write is // what still avoids the disk write and the later @@ -9325,8 +9634,46 @@ async fn execute_single_fetch( result: FetchResult::SourceFailed, } } + ReplicationMessageBody::FetchResponse(protocol::FetchResponse::Busy { + key: resp_key, + }) if resp_key == key => { + // The source refused at admission and said so, where it + // used to stay silent. That is an honest, prompt answer, so + // it carries no trust event. Waiting here, still holding + // this fetch slot, before the key moves on is what keeps a + // busy answer from becoming a fast retry loop: this slot asks + // at most once per wait, where silence made it once per fetch + // timeout. The attempt then ends exactly as that timeout + // did, so nothing downstream changes. The worker races this + // whole task against shutdown, so the wait does not delay it. + audit_metrics::record_fetch_busy_received(); + let jitter = + rand::thread_rng().gen_range(Duration::ZERO..=FETCH_BUSY_WAIT_SPREAD); + // The holder is put last for at least the full jittered + // wait, even when this attempt's own deadline cuts the + // sleep short. + let full_wait = busy_wait(Duration::MAX, jitter); + let remaining = config + .fetch_request_timeout + .saturating_sub(sent_at.elapsed()); + let wait = full_wait.min(remaining); + let cooldown = busy_sources.record_busy(&source, Instant::now(), full_wait); + debug!( + "Fetch: source {source} busy for {}; moving on in {} ms, \ + passing it over for new keys for {} ms", + hex::encode(key), + wait.as_millis(), + cooldown.as_millis() + ); + tokio::time::sleep(wait).await; + FetchOutcome { + key, + result: FetchResult::SourceFailed, + } + } _ => { - // Unexpected message type — treat as malformed. + // Unexpected message type, or a busy answer for another + // key — treat as malformed. p2p_node .report_trust_event( &source, @@ -13287,6 +13634,7 @@ mod tests { request_id: 1, body: ReplicationMessageBody::FetchRequest(protocol::FetchRequest { key: test_key(1), + busy_reply: true, }), }, rr_message_id: None, @@ -13779,8 +14127,13 @@ mod tests { drive_key_in_flight(&mut q, key, vec![test_peer(0x01), test_peer(0x02)]); assert_eq!(q.retry_reserved_slot_count(), 1); - let follow_up = - apply_fetch_result(&mut q, &key, &FetchResult::NoLongerResponsible, RETRY_AFTER); + let follow_up = apply_fetch_result( + &mut q, + &key, + &FetchResult::NoLongerResponsible, + RETRY_AFTER, + &SourceBusyTracker::new(), + ); assert_eq!( follow_up, @@ -13815,7 +14168,13 @@ mod tests { let key = test_key(0xCD); drive_key_in_flight(&mut q, key, vec![test_peer(0x03)]); - let follow_up = apply_fetch_result(&mut q, &key, &result, RETRY_AFTER); + let follow_up = apply_fetch_result( + &mut q, + &key, + &result, + RETRY_AFTER, + &SourceBusyTracker::new(), + ); assert_eq!(follow_up, FetchFollowUp::Terminal); assert!(!q.contains_key(&key)); @@ -13833,7 +14192,13 @@ mod tests { let second = test_peer(0x05); drive_key_in_flight(&mut q, key, vec![first, second]); - let follow_up = apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER); + let follow_up = apply_fetch_result( + &mut q, + &key, + &FetchResult::SourceFailed, + RETRY_AFTER, + &SourceBusyTracker::new(), + ); assert_eq!( follow_up, FetchFollowUp::RetryFrom(second), @@ -13841,7 +14206,13 @@ mod tests { ); assert_eq!(q.in_flight_count(), 1, "retry keeps the key in flight"); - let follow_up = apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER); + let follow_up = apply_fetch_result( + &mut q, + &key, + &FetchResult::SourceFailed, + RETRY_AFTER, + &SourceBusyTracker::new(), + ); assert_eq!( follow_up, FetchFollowUp::RequeuedForVerification, @@ -13855,6 +14226,91 @@ mod tests { ); } + /// A retry passes over an untried source that recently answered busy when + /// another untried source is free, and still takes it when it is the only + /// one left: busy holders go last, never out. + #[test] + fn source_failure_retries_on_a_holder_that_is_not_cooling_first() { + const RETRY_AFTER: Duration = Duration::from_mins(1); + + let mut q = ReplicationQueues::new(); + let key = test_key(0xE1); + let (first, cooling, free) = (test_peer(0x11), test_peer(0x12), test_peer(0x13)); + drive_key_in_flight(&mut q, key, vec![first, cooling, free]); + let busy_sources = SourceBusyTracker::new(); + busy_sources.record_busy(&cooling, Instant::now(), Duration::from_mins(1)); + + let follow_up = apply_fetch_result( + &mut q, + &key, + &FetchResult::SourceFailed, + RETRY_AFTER, + &busy_sources, + ); + assert_eq!(follow_up, FetchFollowUp::RetryFrom(free)); + + let follow_up = apply_fetch_result( + &mut q, + &key, + &FetchResult::SourceFailed, + RETRY_AFTER, + &busy_sources, + ); + assert_eq!( + follow_up, + FetchFollowUp::RetryFrom(cooling), + "a cooling holder is still tried once it is the last untried one" + ); + + let follow_up = apply_fetch_result( + &mut q, + &key, + &FetchResult::SourceFailed, + RETRY_AFTER, + &busy_sources, + ); + assert_eq!(follow_up, FetchFollowUp::RequeuedForVerification); + } + + /// A busy reply serves nothing, so it never counts as served bytes; the + /// answers that did before still do. + #[test] + fn a_busy_reply_is_not_counted_as_served() { + let key = test_key(0x77); + assert!(!counts_as_served(&ReplicationMessageBody::FetchResponse( + protocol::FetchResponse::Busy { key } + ))); + for response in [ + protocol::FetchResponse::Success { + key, + data: vec![1, 2, 3], + }, + protocol::FetchResponse::NotFound { key }, + protocol::FetchResponse::Error { + key, + reason: "io".to_string(), + }, + ] { + assert!(counts_as_served(&ReplicationMessageBody::FetchResponse( + response + ))); + } + assert!(counts_as_served( + &ReplicationMessageBody::NeighborSyncResponse(protocol::NeighborSyncResponse { + replica_hints: vec![], + paid_hints: vec![], + bootstrapping: false, + rejected_keys: vec![], + commitment: None, + }) + )); + assert!(!counts_as_served( + &ReplicationMessageBody::VerificationResponse(protocol::VerificationResponse { + results: vec![], + }) + )); + } + #[test] fn source_failure_without_retry_metadata_is_terminal() { const RETRY_AFTER: Duration = Duration::from_mins(1); @@ -13868,7 +14324,13 @@ mod tests { let candidate = q.dequeue_fetch().expect("enqueued key must dequeue"); q.start_dequeued_fetch(candidate, source); - let follow_up = apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER); + let follow_up = apply_fetch_result( + &mut q, + &key, + &FetchResult::SourceFailed, + RETRY_AFTER, + &SourceBusyTracker::new(), + ); assert_eq!(follow_up, FetchFollowUp::Terminal); assert!(!q.contains_key(&key)); @@ -13883,7 +14345,13 @@ mod tests { let source = test_peer(0x0B); drive_key_in_flight(&mut q, key, vec![source, test_peer(0x0C)]); - let follow_up = apply_fetch_result(&mut q, &key, &FetchResult::AlreadyHeld, RETRY_AFTER); + let follow_up = apply_fetch_result( + &mut q, + &key, + &FetchResult::AlreadyHeld, + RETRY_AFTER, + &SourceBusyTracker::new(), + ); assert_eq!( follow_up, @@ -13912,8 +14380,13 @@ mod tests { let third = test_peer(0x09); drive_key_in_flight(&mut q, key, vec![first, second, third]); - let follow_up = - apply_fetch_result(&mut q, &key, &FetchResult::LocalWriteFailed, RETRY_AFTER); + let follow_up = apply_fetch_result( + &mut q, + &key, + &FetchResult::LocalWriteFailed, + RETRY_AFTER, + &SourceBusyTracker::new(), + ); assert_eq!( follow_up, @@ -13946,8 +14419,13 @@ mod tests { let candidate = q.dequeue_fetch().expect("enqueued key must dequeue"); q.start_dequeued_fetch(candidate, source); - let follow_up = - apply_fetch_result(&mut q, &key, &FetchResult::LocalWriteFailed, RETRY_AFTER); + let follow_up = apply_fetch_result( + &mut q, + &key, + &FetchResult::LocalWriteFailed, + RETRY_AFTER, + &SourceBusyTracker::new(), + ); assert_eq!(follow_up, FetchFollowUp::Terminal); assert!(!q.contains_key(&key)); diff --git a/src/replication/protocol.rs b/src/replication/protocol.rs index ebb4644a..d98eec80 100644 --- a/src/replication/protocol.rs +++ b/src/replication/protocol.rs @@ -22,6 +22,35 @@ use super::config::MAX_SUBTREE_AUDIT_MESSAGE_SIZE; /// Used in [`AuditResponse::Digests`] for keys the peer does not hold. pub const ABSENT_KEY_DIGEST: [u8; 32] = [0u8; 32]; +/// Opens the fetch capability trailer that may follow an encoded +/// [`FetchRequest`] message. +/// +/// A requester states what it understands here, after the end of the postcard +/// message, because every older node decodes with postcard's `from_bytes`, +/// which ignores whatever follows the message. A field inside the message +/// would instead shift or break their decoding. The magic keeps a stray byte +/// after a message from ever reading as a capability. +pub const FETCH_CAPABILITY_MAGIC: [u8; 4] = *b"antf"; + +/// Bit in the fetch capability flags byte: the requester understands +/// [`FetchResponse::Busy`]. +pub const FETCH_CAPABILITY_BUSY_REPLY: u8 = 0b0000_0001; + +/// Length of the fetch capability trailer: the magic and one flags byte. +const FETCH_CAPABILITY_TRAILER_LEN: usize = FETCH_CAPABILITY_MAGIC.len() + 1; + +/// The fetch capability flags in `trailer`, the bytes after a decoded message. +/// +/// Zero unless `trailer` opens with [`FETCH_CAPABILITY_MAGIC`] and has a flags +/// byte after it. Bytes after the flags byte are reserved and ignored, so a +/// later version can extend the trailer without this one rejecting it. +fn fetch_capability_flags(trailer: &[u8]) -> u8 { + match trailer.strip_prefix(FETCH_CAPABILITY_MAGIC.as_slice()) { + Some([flags, ..]) => *flags, + _ => 0, + } +} + // --------------------------------------------------------------------------- // Top-level envelope // --------------------------------------------------------------------------- @@ -68,8 +97,24 @@ impl ReplicationMessage { // The buffer is sized exactly, and an oversized body is refused // before anything is allocated. Chunk-carrying bodies run to several // MiB and are held while they are sent. - let max_size = ceiling_for(family_of_variant(self.body.variant_index())); - let bytes = encode_exact(self, max_size).map_err(|e| match e { + // A fetch request that asks for busy replies carries the capability + // trailer after the message, so its message must leave room for it + // under the same ceiling the decoder applies to the whole input. + let asks_busy_reply = matches!( + &self.body, + ReplicationMessageBody::FetchRequest(FetchRequest { + busy_reply: true, + .. + }) + ); + let trailer_len = if asks_busy_reply { + FETCH_CAPABILITY_TRAILER_LEN + } else { + 0 + }; + let max_size = + ceiling_for(family_of_variant(self.body.variant_index())).saturating_sub(trailer_len); + let mut bytes = encode_exact(self, max_size).map_err(|e| match e { ExactEncodeError::Serialize(e) => { ReplicationProtocolError::SerializationFailed(e.to_string()) } @@ -81,6 +126,14 @@ impl ReplicationMessage { } })?; + // The capability trailer goes immediately after the message, where + // older decoders never look. + if asks_busy_reply { + bytes.reserve_exact(trailer_len); + bytes.extend_from_slice(&FETCH_CAPABILITY_MAGIC); + bytes.push(FETCH_CAPABILITY_BUSY_REPLY); + } + // V2-623: cumulative per-variant tx accounting. Every replication send // funnels through here, so this is the single tx choke point. record_tx(&self.body, bytes.len()); @@ -120,8 +173,14 @@ impl ReplicationMessage { max_size, }); } - let message: Self = postcard::from_bytes(data) + // `take_from_bytes` rather than `from_bytes` only to see what follows + // the message. Both ignore it otherwise, so nothing that decoded before + // decodes differently now. + let (mut message, trailer): (Self, &[u8]) = postcard::take_from_bytes(data) .map_err(|e| ReplicationProtocolError::DeserializationFailed(e.to_string()))?; + if let ReplicationMessageBody::FetchRequest(request) = &mut message.body { + request.busy_reply = fetch_capability_flags(trailer) & FETCH_CAPABILITY_BUSY_REPLY != 0; + } // V2-623: cumulative per-variant rx accounting. Every replication // receive funnels through here, so this is the single rx choke point. @@ -977,6 +1036,18 @@ pub struct VerificationResponse { pub struct FetchRequest { /// The key of the record to fetch. pub key: XorName, + /// Whether the requester understands [`FetchResponse::Busy`]. + /// + /// Not part of the serialized struct, whose layout is unchanged since the + /// first release: it travels in the fetch capability trailer that + /// [`ReplicationMessage::encode`] appends after the whole message, and + /// [`ReplicationMessage::decode`] reads back. Older nodes decode with + /// postcard's `from_bytes`, which stops at the end of the message and never + /// sees the trailer, so they keep refusing silently. A responder answers + /// `Busy` only when this is set, so an older requester, which never sends + /// the trailer, is never sent a variant it cannot decode. + #[serde(skip)] + pub busy_reply: bool, } /// Response to a fetch request. @@ -1004,6 +1075,16 @@ pub enum FetchResponse { /// Human-readable error description. reason: String, }, + /// The responder could not admit the request right now and did not + /// serve it. Nothing is implied about whether it holds the record. + /// + /// APPENDED so the discriminants of the variants above are unchanged. Sent + /// only in answer to a [`FetchRequest`] with `busy_reply` set; an older + /// requester never sets it, so it is never sent a value it cannot decode. + Busy { + /// The requested key. + key: XorName, + }, } // --------------------------------------------------------------------------- @@ -1740,7 +1821,10 @@ mod tests { paid_list_check_indices: vec![], }), ReplicationMessageBody::VerificationResponse(VerificationResponse { results: vec![] }), - ReplicationMessageBody::FetchRequest(FetchRequest { key: z }), + ReplicationMessageBody::FetchRequest(FetchRequest { + key: z, + busy_reply: true, + }), ReplicationMessageBody::FetchResponse(FetchResponse::NotFound { key: z }), ReplicationMessageBody::AuditChallenge(AuditChallenge { challenge_id: 1, @@ -2392,7 +2476,10 @@ mod tests { fn fetch_request_roundtrip() { let msg = ReplicationMessage { request_id: 9, - body: ReplicationMessageBody::FetchRequest(FetchRequest { key: [0x40; 32] }), + body: ReplicationMessageBody::FetchRequest(FetchRequest { + key: [0x40; 32], + busy_reply: true, + }), }; let encoded = msg.encode().expect("encode should succeed"); let decoded = ReplicationMessage::decode(&encoded).expect("decode should succeed"); @@ -2400,6 +2487,7 @@ mod tests { assert_eq!(decoded.request_id, 9); if let ReplicationMessageBody::FetchRequest(req) = decoded.body { assert_eq!(req.key, [0x40; 32]); + assert!(req.busy_reply); } else { panic!("expected FetchRequest"); } @@ -2471,6 +2559,283 @@ mod tests { } } + #[test] + fn fetch_response_busy_roundtrip() { + let msg = ReplicationMessage { + request_id: 13, + body: ReplicationMessageBody::FetchResponse(FetchResponse::Busy { key: [0x71; 32] }), + }; + let encoded = msg.encode().expect("encode should succeed"); + let decoded = ReplicationMessage::decode(&encoded).expect("decode should succeed"); + + assert_eq!(decoded.request_id, 13); + if let ReplicationMessageBody::FetchResponse(FetchResponse::Busy { key }) = decoded.body { + assert_eq!(key, [0x71; 32]); + } else { + panic!("expected FetchResponse::Busy"); + } + } + + // -- the fetch busy reply across versions ------------------------------- + // + // `Legacy*` mirror the fetch types and the envelope as every release + // before the busy reply encoded them. The seven unit variants stand in for + // the body variants before `FetchRequest`; only their count matters, since + // postcard writes an enum as its variant index followed by the payload. + + #[derive(Serialize, Deserialize)] + struct LegacyFetchRequest { + key: XorName, + } + + #[derive(Serialize, Deserialize)] + enum LegacyFetchResponse { + Success { + key: XorName, + #[serde(with = "serde_bytes")] + data: Vec, + }, + NotFound { + key: XorName, + }, + Error { + key: XorName, + reason: String, + }, + } + + #[derive(Serialize, Deserialize)] + enum LegacyBody { + V0, + V1, + V2, + V3, + V4, + V5, + V6, + FetchRequest(LegacyFetchRequest), + FetchResponse(LegacyFetchResponse), + } + + #[derive(Serialize, Deserialize)] + struct LegacyMessage { + request_id: u64, + body: LegacyBody, + } + + fn legacy_fetch_request_bytes(key: XorName) -> Vec { + postcard::to_stdvec(&LegacyMessage { + request_id: 21, + body: LegacyBody::FetchRequest(LegacyFetchRequest { key }), + }) + .expect("encode") + } + + /// The mirror must match the real layout, or every test below proves + /// nothing. A current request is an older one's bytes exactly, plus the + /// capability trailer when it asks for a busy reply. + #[test] + fn legacy_mirror_matches_the_current_layout() { + let legacy = legacy_fetch_request_bytes([0x22; 32]); + let plain = ReplicationMessage { + request_id: 21, + body: ReplicationMessageBody::FetchRequest(FetchRequest { + key: [0x22; 32], + busy_reply: false, + }), + } + .encode() + .expect("encode"); + assert_eq!(plain, legacy); + + let asking = ReplicationMessage { + request_id: 21, + body: ReplicationMessageBody::FetchRequest(FetchRequest { + key: [0x22; 32], + busy_reply: true, + }), + } + .encode() + .expect("encode"); + let mut expected = legacy; + expected.extend_from_slice(&FETCH_CAPABILITY_MAGIC); + expected.push(FETCH_CAPABILITY_BUSY_REPLY); + assert_eq!(asking, expected); + } + + /// An older responder decodes a current request: the trailer comes after + /// the message, which `from_bytes` never reads, so it sees the key and + /// keeps refusing silently. + #[test] + fn older_responder_decodes_a_busy_aware_request() { + let encoded = ReplicationMessage { + request_id: 22, + body: ReplicationMessageBody::FetchRequest(FetchRequest { + key: [0x23; 32], + busy_reply: true, + }), + } + .encode() + .expect("encode"); + let decoded: LegacyMessage = postcard::from_bytes(&encoded).expect("older decoder accepts"); + assert_eq!(decoded.request_id, 22); + match decoded.body { + LegacyBody::FetchRequest(LegacyFetchRequest { key }) => assert_eq!(key, [0x23; 32]), + _ => panic!("expected the legacy FetchRequest variant"), + } + } + + /// A current responder decodes an older requester's request, which has no + /// trailer, as one that did not ask for a busy reply. + #[test] + fn older_request_decodes_without_busy_reply() { + let decoded = ReplicationMessage::decode(&legacy_fetch_request_bytes([0x24; 32])) + .expect("current decoder accepts an older request"); + match decoded.body { + ReplicationMessageBody::FetchRequest(FetchRequest { key, busy_reply }) => { + assert_eq!(key, [0x24; 32]); + assert!(!busy_reply); + } + _ => panic!("expected FetchRequest"), + } + } + + /// Only the magic followed by a flags byte with the busy bit set asks for a + /// busy reply. Unknown flag bits and bytes after the flags are reserved and + /// ignored; anything else after the message reads as no capability, as it + /// always has. + #[test] + fn only_the_trailer_with_the_busy_bit_asks_for_a_busy_reply() { + let magic = FETCH_CAPABILITY_MAGIC.to_vec(); + let with = |tail: &[u8]| { + let mut t = magic.clone(); + t.extend_from_slice(tail); + t + }; + let base = legacy_fetch_request_bytes([0x25; 32]); + for (trailer, expected) in [ + (vec![], false), + (with(&[FETCH_CAPABILITY_BUSY_REPLY]), true), + (with(&[0]), false), + (with(&[0b1111_1111]), true), + (with(&[0b1111_1110]), false), + (with(&[FETCH_CAPABILITY_BUSY_REPLY, 0xAA, 0xBB]), true), + (magic.clone(), false), + (magic.get(..3).expect("short magic").to_vec(), false), + ( + vec![b'a', b'n', b't', b'g', FETCH_CAPABILITY_BUSY_REPLY], + false, + ), + (vec![FETCH_CAPABILITY_BUSY_REPLY], false), + (vec![0u8; 64], false), + ] { + let mut bytes = base.clone(); + bytes.extend_from_slice(&trailer); + let decoded = ReplicationMessage::decode(&bytes).expect("decodes"); + match decoded.body { + ReplicationMessageBody::FetchRequest(req) => { + assert_eq!(req.key, [0x25; 32]); + assert_eq!(req.busy_reply, expected, "trailer {trailer:?}"); + } + _ => panic!("expected FetchRequest"), + } + } + } + + /// The trailer means something only after a fetch request. After any other + /// body it is ignored exactly like every other trailing byte. + #[test] + fn other_bodies_ignore_a_capability_trailer() { + let mut bytes = ReplicationMessage { + request_id: 25, + body: ReplicationMessageBody::FetchResponse(FetchResponse::NotFound { + key: [0x29; 32], + }), + } + .encode() + .expect("encode"); + bytes.extend_from_slice(&FETCH_CAPABILITY_MAGIC); + bytes.push(FETCH_CAPABILITY_BUSY_REPLY); + let decoded = ReplicationMessage::decode(&bytes).expect("decodes"); + assert_eq!(decoded.request_id, 25); + assert!(matches!( + decoded.body, + ReplicationMessageBody::FetchResponse(FetchResponse::NotFound { key }) if key == [0x29; 32] + )); + } + + /// A request cut short inside the key is still malformed. + #[test] + fn a_truncated_key_is_still_rejected() { + let bytes = legacy_fetch_request_bytes([0x26; 32]); + let truncated = bytes.get(..bytes.len() - 1).expect("non-empty"); + assert!(ReplicationMessage::decode(truncated).is_err()); + } + + /// The pre-existing answers keep their discriminants, so an older requester + /// still decodes every answer it is ever sent. + #[test] + fn older_requester_decodes_the_unchanged_answers() { + for (body, check) in [ + ( + FetchResponse::Success { + key: [0x27; 32], + data: vec![1, 2, 3], + }, + 0u8, + ), + (FetchResponse::NotFound { key: [0x27; 32] }, 1), + ( + FetchResponse::Error { + key: [0x27; 32], + reason: "io".to_string(), + }, + 2, + ), + ] { + let encoded = ReplicationMessage { + request_id: 23, + body: ReplicationMessageBody::FetchResponse(body), + } + .encode() + .expect("encode"); + let decoded: LegacyMessage = + postcard::from_bytes(&encoded).expect("older decoder accepts"); + let got = match decoded.body { + LegacyBody::FetchResponse(LegacyFetchResponse::Success { key, data }) => { + assert_eq!(key, [0x27; 32]); + assert_eq!(data, vec![1, 2, 3]); + 0 + } + LegacyBody::FetchResponse(LegacyFetchResponse::NotFound { key }) => { + assert_eq!(key, [0x27; 32]); + 1 + } + LegacyBody::FetchResponse(LegacyFetchResponse::Error { key, reason }) => { + assert_eq!(key, [0x27; 32]); + assert_eq!(reason, "io"); + 2 + } + _ => panic!("expected a legacy FetchResponse"), + }; + assert_eq!(got, check); + } + } + + /// Why the responder gates `Busy` on the request's flag: an older requester + /// cannot decode it, and an undecodable answer costs the responder a trust + /// event at that requester. + #[test] + fn older_requester_cannot_decode_busy() { + let encoded = ReplicationMessage { + request_id: 24, + body: ReplicationMessageBody::FetchResponse(FetchResponse::Busy { key: [0x28; 32] }), + } + .encode() + .expect("encode"); + assert!(postcard::from_bytes::(&encoded).is_err()); + } + // === Audit roundtrips === #[test] @@ -2620,7 +2985,10 @@ mod tests { // fetched chunk needs. let mut core_body = ReplicationMessage { request_id: 1, - body: ReplicationMessageBody::FetchRequest(FetchRequest { key: [0u8; 32] }), + body: ReplicationMessageBody::FetchRequest(FetchRequest { + key: [0u8; 32], + busy_reply: false, + }), } .encode() .expect("a small core body encodes"); diff --git a/src/replication/scheduling.rs b/src/replication/scheduling.rs index e296606e..5ce83aa3 100644 --- a/src/replication/scheduling.rs +++ b/src/replication/scheduling.rs @@ -1311,14 +1311,29 @@ impl ReplicationQueues { /// Returns the next source peer if one is available, or `None` if all /// sources have been exhausted. pub fn retry_fetch(&mut self, key: &XorName) -> Option { + self.retry_fetch_with(key, |untried| untried.first().copied()) + } + + /// Like [`Self::retry_fetch`], with `choose` picking the next source from + /// the untried ones, given in source-preference order. A pick that is not + /// among them is ignored and the first untried source is used instead, so + /// `choose` can reorder the untried sources but never revive a tried one. + pub fn retry_fetch_with(&mut self, key: &XorName, choose: F) -> Option + where + F: FnOnce(&[PeerId]) -> Option, + { let entry = self.in_flight_fetch.get_mut(key)?; entry.tried.insert(entry.source); - let next = entry + let untried: Vec = entry .all_sources .iter() - .find(|p| !entry.tried.contains(p)) - .copied(); + .filter(|p| !entry.tried.contains(p)) + .copied() + .collect(); + let next = choose(&untried) + .filter(|pick| untried.contains(pick)) + .or_else(|| untried.first().copied()); if let Some(next_peer) = next { entry.source = next_peer; diff --git a/tests/e2e/fetch_busy_ab.rs b/tests/e2e/fetch_busy_ab.rs new file mode 100644 index 00000000..739bbb10 --- /dev/null +++ b/tests/e2e/fetch_busy_ab.rs @@ -0,0 +1,299 @@ +//! Opt-in measurement, not a pass/fail gate: how long one node takes to fetch +//! a batch of chunks whose holders it over-asks, and what that costs the +//! holders in the requester's trust table. +//! +//! The requester fetches every chunk with the same three holders in the same +//! order, so its fetch worker, one slot per hardware thread, sends more than a +//! holder's two-request share to the first holder at once. That is the +//! over-share the busy reply answers. The file uses only test hooks that exist +//! on `main` too, so the same file measures a build without the busy reply and +//! a build with it. +//! +//! A second measurement, [`fetch_busy_ab_pool`], has every requester stay +//! within its share, as #252 makes them, and fills the first holder's +//! twelve-slot pool instead: the refusals that remain after #252. +//! +//! Run with `ANT_FETCH_BUSY_AB=1 cargo test --test e2e --features test-utils +//! fetch_busy_ab -- --nocapture --test-threads=1`. Without the variable both +//! return at once. The pool measurement uses the test-only fetch serve gate, +//! which a build without the busy reply needs patched in to compare. + +#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] + +use super::testnet::TestNetworkConfig; +use super::TestHarness; +use ant_node::client::compute_address; +use ant_node::ReplicationConfig; +use serial_test::serial; +use std::time::{Duration, Instant}; + +const REQUESTER_INDEX: usize = 4; +const HOLDER_INDICES: [usize; 3] = [1, 2, 3]; +const CHUNKS: usize = 48; +const CHUNK_LEN: usize = 256 * 1024; +const FILL_DEADLINE: Duration = Duration::from_mins(15); +const POLL: Duration = Duration::from_millis(100); + +#[tokio::test] +#[serial] +async fn fetch_busy_ab() { + if std::env::var("ANT_FETCH_BUSY_AB").is_err() { + return; + } + let harness = TestHarness::setup_minimal().await.expect("setup"); + harness.warmup_dht().await.expect("warmup"); + + let holders: Vec<_> = HOLDER_INDICES + .iter() + .map(|&i| { + *harness + .test_node(i) + .and_then(|n| n.p2p_node.as_ref()) + .expect("holder p2p") + .peer_id() + }) + .collect(); + + let mut keys = Vec::with_capacity(CHUNKS); + for c in 0..CHUNKS { + let content: Vec = (0..CHUNK_LEN) + .map(|i| u8::try_from((i * 31 + c * 7) % 251).unwrap()) + .collect(); + let key = compute_address(&content); + for &h in &HOLDER_INDICES { + harness + .test_node(h) + .and_then(|n| n.ant_protocol.as_ref()) + .expect("holder protocol") + .storage() + .put(&key, &content) + .await + .expect("put"); + } + keys.push(key); + } + + let requester = harness.test_node(REQUESTER_INDEX).expect("requester"); + let p2p = requester.p2p_node.as_ref().expect("requester p2p"); + let engine = requester + .replication_engine + .as_ref() + .expect("requester engine"); + let storage = requester + .ant_protocol + .as_ref() + .expect("requester protocol") + .storage(); + + let trust_before: Vec = holders.iter().map(|h| p2p.peer_trust(h)).collect(); + let started = Instant::now(); + let mut enqueued = 0usize; + for key in &keys { + if engine.enqueue_fetch_for_test(*key, holders.clone()).await { + enqueued += 1; + } + } + + let mut stored = 0usize; + while started.elapsed() < FILL_DEADLINE { + stored = 0; + for key in &keys { + if storage.exists(key).unwrap_or(false) { + stored += 1; + } + } + if stored == CHUNKS { + break; + } + tokio::time::sleep(POLL).await; + } + let elapsed = started.elapsed(); + let trust_after: Vec = holders.iter().map(|h| p2p.peer_trust(h)).collect(); + + println!( + "FETCH_BUSY_AB slots={} chunks={CHUNKS} enqueued={enqueued} stored={stored} \ + elapsed_ms={} trust_before={trust_before:?} trust_after={trust_after:?}", + std::thread::available_parallelism().map_or(0, std::num::NonZero::get), + elapsed.as_millis(), + ); + + harness.teardown().await.expect("teardown"); +} + +/// Nodes in the pool measurement's network. +const POOL_NODES: usize = 25; +/// A close group this wide makes every node responsible for every key, so +/// each requester stores what it fetches. +const POOL_CLOSE_GROUP: usize = 24; +/// The holders, in the source order every chunk uses. +const POOL_HOLDERS: [usize; 5] = [3, 4, 5, 6, 7]; +/// Chunks per requester: its per-peer share, so it never over-asks a holder. +const POOL_KEYS_PER_REQUESTER: usize = 2; +const POOL_CHUNK_LEN: usize = 64 * 1024; +/// How long the first holder keeps its pool full, once it is full, before it +/// serves what it holds. +const POOL_GATE_HOLD: Duration = Duration::from_secs(2); +/// A holder's fetch admission pool: `FETCH_RESPONDER_MAX_OUTSTANDING`. +const POOL_SLOTS: usize = 12; +/// Bound on reaching the precondition before measuring. +const POOL_SETUP_DEADLINE: Duration = Duration::from_secs(60); + +#[tokio::test] +#[serial] +async fn fetch_busy_ab_pool() { + if std::env::var("ANT_FETCH_BUSY_AB").is_err() { + return; + } + let config = TestNetworkConfig { + node_count: POOL_NODES, + replication_config: Some(ReplicationConfig { + close_group_size: POOL_CLOSE_GROUP, + ..ReplicationConfig::default() + }), + ..TestNetworkConfig::default() + }; + let harness = TestHarness::setup_with_config(config).await.expect("setup"); + harness.warmup_dht().await.expect("warmup"); + + let holders: Vec<_> = POOL_HOLDERS + .iter() + .map(|&i| { + *harness + .test_node(i) + .and_then(|n| n.p2p_node.as_ref()) + .expect("holder p2p") + .peer_id() + }) + .collect(); + let requesters: Vec = (0..POOL_NODES) + .filter(|i| !POOL_HOLDERS.contains(i)) + .collect(); + + let mut keys_by_requester = Vec::with_capacity(requesters.len()); + for (r, _) in requesters.iter().enumerate() { + let mut keys = Vec::with_capacity(POOL_KEYS_PER_REQUESTER); + for k in 0..POOL_KEYS_PER_REQUESTER { + let content: Vec = (0..POOL_CHUNK_LEN) + .map(|i| u8::try_from((i * 13 + r * 101 + k * 7) % 251).unwrap()) + .collect(); + let key = compute_address(&content); + for &h in &POOL_HOLDERS { + harness + .test_node(h) + .and_then(|n| n.ant_protocol.as_ref()) + .expect("holder protocol") + .storage() + .put(&key, &content) + .await + .expect("put"); + } + keys.push(key); + } + keys_by_requester.push(keys); + } + + let first_holder = POOL_HOLDERS + .first() + .and_then(|&i| harness.test_node(i)) + .and_then(|n| n.replication_engine.as_ref()) + .expect("first holder engine"); + first_holder.close_fetch_serve_gate_for_test(); + + let trust_of = |i: usize| -> Vec { + let p2p = harness + .test_node(i) + .and_then(|n| n.p2p_node.as_ref()) + .expect("requester p2p"); + holders.iter().map(|h| p2p.peer_trust(h)).collect() + }; + let trust_before: Vec> = requesters.iter().map(|&i| trust_of(i)).collect(); + + let total = requesters.len() * POOL_KEYS_PER_REQUESTER; + let refusals_before = first_holder.fetch_pool_full_refusals_for_test(); + let started = Instant::now(); + let mut enqueued = 0usize; + for (&i, keys) in requesters.iter().zip(&keys_by_requester) { + let engine = harness + .test_node(i) + .and_then(|n| n.replication_engine.as_ref()) + .expect("requester engine"); + for key in keys { + if engine.enqueue_fetch_for_test(*key, holders.clone()).await { + enqueued += 1; + } + } + } + assert_eq!( + enqueued, total, + "every key must be queued for the measurement" + ); + + // The measurement is only about a full pool: wait until the first holder + // holds a whole pool's worth, keep it full, and require that it refused + // the rest before letting it serve. + let setup_deadline = Instant::now() + POOL_SETUP_DEADLINE; + while first_holder.fetch_admitted_for_test() < POOL_SLOTS { + assert!( + Instant::now() < setup_deadline, + "the first holder never filled its pool: {} admitted", + first_holder.fetch_admitted_for_test() + ); + tokio::time::sleep(POLL).await; + } + tokio::time::sleep(POOL_GATE_HOLD).await; + let refused_while_full = first_holder + .fetch_pool_full_refusals_for_test() + .saturating_sub(refusals_before); + assert!( + refused_while_full > 0, + "the first holder refused nothing for a full pool while it was full" + ); + first_holder.open_fetch_serve_gate_for_test(); + let mut stored = 0usize; + while started.elapsed() < FILL_DEADLINE { + stored = 0; + for (&i, keys) in requesters.iter().zip(&keys_by_requester) { + let storage = harness + .test_node(i) + .and_then(|n| n.ant_protocol.as_ref()) + .expect("requester protocol") + .storage(); + for key in keys { + if storage.exists(key).unwrap_or(false) { + stored += 1; + } + } + } + if stored == total { + break; + } + tokio::time::sleep(POLL).await; + } + let elapsed = started.elapsed(); + + // Trust lost per holder, summed over requesters. Each connection-timeout + // charge from a neutral 0.5 costs about 0.06. + let trust_after: Vec> = requesters.iter().map(|&i| trust_of(i)).collect(); + let drops = |h: usize| { + trust_before + .iter() + .zip(&trust_after) + .filter_map(move |(b, a)| Some(b.get(h)? - a.get(h)?)) + }; + let lost: Vec = (0..holders.len()) + .map(|h| drops(h).map(|d| d.max(0.0)).sum()) + .collect(); + let charged: Vec = (0..holders.len()) + .map(|h| drops(h).filter(|d| *d > 0.0).count()) + .collect(); + + println!( + "FETCH_BUSY_AB_POOL requesters={} keys={total} enqueued={enqueued} stored={stored} \ + refused_while_full={refused_while_full} elapsed_ms={} \ + requesters_charging_each_holder={charged:?} trust_lost_per_holder={lost:?}", + requesters.len(), + elapsed.as_millis(), + ); + + harness.teardown().await.expect("teardown"); +} diff --git a/tests/e2e/fetch_busy_reply.rs b/tests/e2e/fetch_busy_reply.rs new file mode 100644 index 00000000..5419888f --- /dev/null +++ b/tests/e2e/fetch_busy_reply.rs @@ -0,0 +1,266 @@ +//! A fetch responder answers `Busy` to a refused request that asked for it, +//! stays silent to one that did not, and a requester takes a busy answer +//! without charging the holder. +//! +//! The unit tests pin the wire format, the limiter, the wait and the queue +//! transitions. What they cannot show is the whole path between two real +//! nodes: the capability trailer surviving the transport, admission refusing +//! past the per-peer share, the reply riding the request/response channel +//! back, and the requester's fetch worker acting on it. +//! +//! The holder's fetch serve gate makes every case deterministic. With the gate +//! closed, an admitted request is held before it is served, keeping its share, +//! so of three concurrent requests from one peer exactly two are admitted and +//! exactly one is refused, whatever order they arrive in. + +#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] + +use super::TestHarness; +use ant_node::client::compute_address; +use ant_node::replication::config::REPLICATION_PROTOCOL_ID; +use ant_node::replication::fetch_busy::{FETCH_BUSY_WAIT_MAX, FETCH_BUSY_WAIT_MIN}; +use ant_node::replication::protocol::{ + FetchRequest, FetchResponse, ReplicationMessage, ReplicationMessageBody, +}; +use futures::stream::{FuturesUnordered, StreamExt}; +use saorsa_core::identity::PeerId; +use saorsa_core::P2PNode; +use serial_test::serial; +use std::time::{Duration, Instant}; + +const HOLDER_INDEX: usize = 3; +const REQUESTER_INDEX: usize = 4; +/// Requests admitted per requesting peer: the responder's per-peer share. +const SHARE: usize = 2; +/// Long enough that a held request is never timed out before the test opens +/// the gate; a timeout would charge the holder and spoil the trust checks. +const HELD_TIMEOUT: Duration = Duration::from_secs(60); +/// How long a request that should be refused silently is given. +const SILENCE_TIMEOUT: Duration = Duration::from_secs(3); +/// Bound on how long anything that must happen promptly may take. +const PROMPTLY: Duration = Duration::from_secs(10); +const POLL: Duration = Duration::from_millis(50); +const CHUNK_LEN: usize = 64 * 1024; + +#[derive(Debug, Clone, PartialEq, Eq)] +enum Answer { + Success, + Busy, + Silent, + Other, +} + +async fn fetch( + sender: &P2PNode, + holder: &PeerId, + key: [u8; 32], + busy_reply: bool, + timeout: Duration, +) -> Answer { + let msg = ReplicationMessage { + request_id: rand_request_id(), + body: ReplicationMessageBody::FetchRequest(FetchRequest { key, busy_reply }), + }; + let Ok(response) = sender + .send_request( + holder, + REPLICATION_PROTOCOL_ID, + msg.encode().expect("encode"), + timeout, + ) + .await + else { + return Answer::Silent; + }; + match ReplicationMessage::decode(&response.data).map(|m| m.body) { + Ok(ReplicationMessageBody::FetchResponse(FetchResponse::Success { key: k, .. })) + if k == key => + { + Answer::Success + } + Ok(ReplicationMessageBody::FetchResponse(FetchResponse::Busy { key: k })) if k == key => { + Answer::Busy + } + _ => Answer::Other, + } +} + +fn rand_request_id() -> u64 { + use std::sync::atomic::{AtomicU64, Ordering}; + static NEXT: AtomicU64 = AtomicU64::new(9_000); + NEXT.fetch_add(1, Ordering::Relaxed) +} + +struct Setup { + harness: TestHarness, + key: [u8; 32], + holder: PeerId, +} + +async fn setup() -> Setup { + let harness = TestHarness::setup_minimal().await.expect("setup"); + harness.warmup_dht().await.expect("warmup"); + let holder_node = harness.test_node(HOLDER_INDEX).expect("holder"); + let content: Vec = (0..CHUNK_LEN) + .map(|i| u8::try_from(i % 251).unwrap()) + .collect(); + let key = compute_address(&content); + holder_node + .ant_protocol + .as_ref() + .expect("holder protocol") + .storage() + .put(&key, &content) + .await + .expect("put"); + let holder = *holder_node.p2p_node.as_ref().expect("holder p2p").peer_id(); + holder_node + .replication_engine + .as_ref() + .expect("holder engine") + .close_fetch_serve_gate_for_test(); + Setup { + harness, + key, + holder, + } +} + +fn open_gate(harness: &TestHarness) { + harness + .test_node(HOLDER_INDEX) + .and_then(|n| n.replication_engine.as_ref()) + .expect("holder engine") + .open_fetch_serve_gate_for_test(); +} + +/// Of `SHARE + 1` concurrent requests that ask for busy replies, the one the +/// holder refuses is answered `Busy` at once, the held ones are served once +/// the gate opens, and a request that does not ask is refused silently. (The +/// per-peer and global busy allowances are pinned exactly by the limiter's +/// unit tests; over a real network their timing is not controllable.) +#[tokio::test] +#[serial] +async fn a_refused_fetch_is_answered_busy_only_when_it_asks() { + let Setup { + harness, + key, + holder, + } = setup().await; + let requester = harness + .test_node(REQUESTER_INDEX) + .and_then(|n| n.p2p_node.as_ref()) + .expect("requester p2p"); + + let mut pending: FuturesUnordered<_> = (0..=SHARE) + .map(|_| fetch(requester, &holder, key, true, HELD_TIMEOUT)) + .collect(); + let started = Instant::now(); + let first = tokio::time::timeout(PROMPTLY, pending.next()) + .await + .expect("the refused request is answered promptly") + .expect("a request"); + assert_eq!(first, Answer::Busy, "the refused request is told busy"); + assert!(started.elapsed() < PROMPTLY); + + // The share is still held, so a request that does not ask for busy + // replies is refused, and silently, exactly as before busy replies. + assert_eq!( + fetch(requester, &holder, key, false, SILENCE_TIMEOUT).await, + Answer::Silent + ); + + open_gate(&harness); + let rest: Vec = tokio::time::timeout(PROMPTLY, pending.collect()) + .await + .expect("held requests are served once the gate opens"); + assert_eq!(rest, vec![Answer::Success; SHARE]); + + harness.teardown().await.expect("teardown"); +} + +/// The requester's own fetch worker, refused by a holder whose share is full, +/// gets a busy answer, puts that holder last for later fetches, waits before +/// giving the key up, and charges the holder nothing. Before busy replies the +/// same refusal was silent and cost the holder a connection-timeout trust +/// event at the requester. +#[tokio::test] +#[serial] +async fn the_requester_takes_a_busy_answer_without_charging_the_holder() { + let Setup { + harness, + key, + holder, + } = setup().await; + let requester_node = harness.test_node(REQUESTER_INDEX).expect("requester"); + let requester = requester_node.p2p_node.as_ref().expect("requester p2p"); + let engine = requester_node + .replication_engine + .as_ref() + .expect("requester engine"); + let trust_before = requester.peer_trust(&holder); + + // Fill this requester's share at the holder: two held, one told busy. + let mut held: FuturesUnordered<_> = (0..=SHARE) + .map(|_| fetch(requester, &holder, key, true, HELD_TIMEOUT)) + .collect(); + let first = tokio::time::timeout(PROMPTLY, held.next()) + .await + .expect("the refused request is answered promptly") + .expect("a request"); + assert_eq!(first, Answer::Busy); + assert!(!engine.fetch_source_cooling_for_test(&holder)); + + // Trust is sampled throughout, so a charge cannot hide behind a later + // unrelated success: any dip below the starting value fails. + let mut lowest_trust = trust_before; + let sample = |lowest: &mut f64| *lowest = lowest.min(requester.peer_trust(&holder)); + + // The fetch worker's request is refused too, and told busy. + assert!(engine.enqueue_fetch_for_test(key, vec![holder]).await); + let deadline = Instant::now() + PROMPTLY; + while !engine.fetch_source_cooling_for_test(&holder) { + assert!( + Instant::now() < deadline, + "the fetch worker never recorded the busy answer" + ); + sample(&mut lowest_trust); + tokio::time::sleep(POLL).await; + } + + // The attempt holds its slot through the busy wait, then gives the key up + // (its only source refused), exactly as a timeout would have, sooner. The + // wait is timed from when the busy answer was recorded, just before it + // starts, to when this fetch attempt is first seen gone, watched with the + // gate still closed so nothing else delays the observation. + let busy_at = engine + .fetch_source_last_busy_for_test(&holder) + .expect("the busy answer was recorded"); + let deadline = Instant::now() + FETCH_BUSY_WAIT_MAX + PROMPTLY; + while engine.fetch_queued_or_in_flight_for_test(&key).await { + assert!( + Instant::now() < deadline, + "the fetch attempt outlived its busy wait" + ); + sample(&mut lowest_trust); + tokio::time::sleep(POLL).await; + } + let attempt_lasted = busy_at.elapsed(); + assert!( + attempt_lasted >= FETCH_BUSY_WAIT_MIN, + "the key was given up {attempt_lasted:?} after the busy answer, before the busy wait" + ); + + open_gate(&harness); + let rest: Vec = tokio::time::timeout(PROMPTLY, held.collect()) + .await + .expect("held requests are served once the gate opens"); + assert_eq!(rest, vec![Answer::Success; SHARE]); + sample(&mut lowest_trust); + assert!( + lowest_trust >= trust_before, + "a busy answer must not cost the holder trust: {trust_before} fell to {lowest_trust}" + ); + + harness.teardown().await.expect("teardown"); +} diff --git a/tests/e2e/mod.rs b/tests/e2e/mod.rs index ec2a74c7..e375b108 100644 --- a/tests/e2e/mod.rs +++ b/tests/e2e/mod.rs @@ -69,6 +69,12 @@ mod security_attacks; #[cfg(test)] mod subtree_audit_testnet; +#[cfg(test)] +mod fetch_busy_ab; + +#[cfg(test)] +mod fetch_busy_reply; + #[cfg(test)] mod fetch_local_write_guard; diff --git a/tests/e2e/replication.rs b/tests/e2e/replication.rs index 6ae3fb5b..06855160 100644 --- a/tests/e2e/replication.rs +++ b/tests/e2e/replication.rs @@ -1203,7 +1203,10 @@ async fn test_fetch_request_returns_record() { .expect("put"); // Send fetch request from B to A - let request = FetchRequest { key: address }; + let request = FetchRequest { + key: address, + busy_reply: false, + }; let msg = ReplicationMessage { request_id: 99, body: ReplicationMessageBody::FetchRequest(request), @@ -2030,7 +2033,10 @@ async fn test_fetch_not_found() { let peer_a = *p2p_a.peer_id(); let missing_key = [0xEE; 32]; - let request = FetchRequest { key: missing_key }; + let request = FetchRequest { + key: missing_key, + busy_reply: false, + }; let msg = ReplicationMessage { request_id: 77, body: ReplicationMessageBody::FetchRequest(request), @@ -2366,7 +2372,10 @@ async fn test_fetch_returns_error_for_corrupt_key() { let peer_a = *p2p_a.peer_id(); let fake_key = [0x00; 32]; - let request = FetchRequest { key: fake_key }; + let request = FetchRequest { + key: fake_key, + busy_reply: false, + }; let msg = ReplicationMessage { request_id: 4000, body: ReplicationMessageBody::FetchRequest(request), From 6abba5a79a7f04be40e11aa12c4a638148989ace Mon Sep 17 00:00:00 2001 From: grumbach Date: Fri, 16 Oct 2026 15:57:36 +0900 Subject: [PATCH 02/10] fix(replication): log the cooldown a busy holder actually gets `SourceBusyTracker::record_busy` never shortens a cooldown that is already running, but it returned the newly computed cooldown, so when an earlier, longer one was still running the requester's debug line reported less than the holder was really put last for. It now returns what is left of the cooldown in force. ant-node only logs the return value, so scheduling and wire behaviour do not change; callers of the public method now get the remaining cooldown in force. The never-shortened test now checks the value too. The same debug line said the holder was passed over "for new keys", but the cooldown applies both when a key starts and when it moves on to another holder. It now says the holder is put last. ADR-0025: - the context adds the refusals #252 leaves at holders it currently counts as silent, and says #252's ADR defers a busy reply rather than naming it as the remaining piece; - the trade-off on holders that answer busy to everything says they are put behind every holder that is not cooling, on retries too, and names the `fetch_busy_reply_send_pool_full` counter in full; - the requester section states the cooldown as 1, 2, 4 and 8 times the jittered wait drawn for each busy answer, capped at 60 s, says that a later busy answer never shortens a running cooldown, and gives the table's eviction order when every holder in it is cooling; - the request-rate trade-off is given for any fetch timeout, not only the default 30 s; - the requester end-to-end test is described as filling that requester's own two-request share at the holder; - `fetch_busy_sources_tracked` is listed as a gauge, apart from the cumulative counters; - the positive consequence notes that the busy wait can be cut short by the attempt's own fetch timeout; - the rejected retry-rotation option states its cost as the mechanism (one request per round trip instead of one per fetch timeout) rather than an estimate; - the order section records that this change and #252 conflict and have not been built or measured together yet; - the real-node measurements are rerun on upstream main a6e3388e, and the local simulation, which is not published, is named only as design input rather than offered as validation. --- docs/adr/ADR-0025-fetch-busy-reply.md | 100 ++++++++++++++------------ src/replication/fetch_busy.rs | 16 +++-- src/replication/mod.rs | 2 +- 3 files changed, 65 insertions(+), 53 deletions(-) diff --git a/docs/adr/ADR-0025-fetch-busy-reply.md b/docs/adr/ADR-0025-fetch-busy-reply.md index 091d97c8..f6f5a8c3 100644 --- a/docs/adr/ADR-0025-fetch-busy-reply.md +++ b/docs/adr/ADR-0025-fetch-busy-reply.md @@ -33,8 +33,11 @@ refusals, 95.2% of them to the new nodes. ant-node #252 removes most of the per-peer over-share by keeping a requester's own fetches within each answering holder's share. It cannot remove every refusal: a request the requester has already timed out on still holds its share at the responder until the responder sheds it, the -twelve-slot pool can be full of other peers' requests, and older requesters keep over-sharing -until they upgrade. Its ADR names a busy reply, which is a wire change, as the remaining piece. +twelve-slot pool can be full of other peers' requests, a holder #252 currently counts as silent +may be sent requests past its share until an answer clears the mark or the mark expires and the +holder is probed under the share again, and older requesters keep over-sharing until they +upgrade. #252's description leaves the first case to a busy reply, and its ADR defers that reply +as a wire change with its own rollout. ## Decision Drivers @@ -61,8 +64,9 @@ until they upgrade. Its ADR names a busy reply, which is a wire change, as the r 4. **A capability trailer after the whole message, and an appended `FetchResponse::Busy` variant (chosen).** 5. **Busy with requester-side retry rotation and budgets.** Reviewed and rejected: hopping on every - busy answer multiplies requests about tenfold in the worst case, competes with `tried` state, - and charging at budget expiry punishes honest overloaded holders once per key per cycle. + busy answer turns each refusal into an immediate new request, so a requester whose holders all + refuse asks once per round trip instead of once per fetch timeout; it competes with `tried` + state; and charging at budget expiry punishes honest overloaded holders once per key per cycle. ## Decision @@ -98,24 +102,28 @@ holder exactly as after a timeout. `SourceBusyTracker` remembers the busy holder and a failed key retries, on the first holder in preference order that is not cooling down, falling back to the first when all are. It reorders and never withholds, so it lowers how often a busy holder is asked but is not a hard per-holder bound; the hard bound is the busy wait per fetch -slot. The cooldown is the full jittered wait, doubling with each further busy answer before the -holder next serves this requester, up to 60 s, and is never shortened while it runs; once it runs -out the holder is no longer put last. Serving once clears the holder's record, so its next busy -answer starts the doubling again; a record with no busy answer for 10 minutes does the same. When -the 4096-entry table is full, a holder that is no longer cooling is forgotten first. A `Busy` for any other key is -treated as malformed, like every other key mismatch. +slot. The cooldown is 1, 2, 4, then 8 times the full jittered wait drawn for that busy answer, +the multiplier rising with each further busy answer before the holder next serves this +requester, capped at 60 s, and a later busy answer never shortens a running cooldown; once it +runs out the holder is no longer put last. Serving once clears the holder's record, so its next +busy answer starts the multiplier again; a record with no busy answer for 10 minutes does the +same. When the 4096-entry table is full, a holder that is no longer cooling is forgotten first, +or, if every holder in it is cooling, the one whose cooldown ends soonest. A `Busy` for any +other key is treated as malformed, like every other key mismatch. **Order.** This is merged after #252. #252's per-source share is what removes most per-peer -over-share and bounds how many fetches one requester has open at one holder; without it, a -requester with many fetch slots still over-shares, and most of those refusals exceed the per-peer -busy allowance and stay silent. On rebase, #252's source choice also skips cooling holders. +over-share and bounds how many fetches one requester has open at one answering holder; without +it, a requester with many fetch slots still over-shares, and most of those refusals exceed the +per-peer busy allowance and stay silent. On rebase, #252's source choice also puts cooling holders +last. The two changes conflict in `src/replication/mod.rs` and have not yet been built or measured +together; the gates and the real-node measurements below will be rerun on the rebased change. ## Consequences ### Positive -- When a refused requester receives `Busy`, it moves on after 5-10 s instead of 30 s without - charging the responder. +- When a refused requester receives `Busy`, it moves on after 5-10 s, or sooner if its own fetch + timeout runs out first, instead of 30 s, without charging the responder. - A holder that just refused is not handed this requester's next keys while another holder that is not cooling exists. - Old nodes decode exactly the messages they always did: a request carries 5 bytes after the @@ -130,15 +138,18 @@ busy allowance and stay silent. On rebase, #252's source choice also skips cooli - New outbound traffic at responders: at most 16 signed busy replies a second sustained, about 85 KiB/s, and 48 (about 250 KiB) in any one second, plus that many signatures. Today a refusal sends nothing. Each refused request already cost its sender the same signed message. -- A requester asks a refusing holder up to once per 5 s per fetch slot instead of once per 30 s: - at most `C / 5 s` small requests with `C` hardware-thread fetch slots, against `C / 30 s` today, - in the case where every holder refuses everything. Capping `C` is a separate decision. +- A requester asks a refusing holder up to once per busy wait per fetch slot instead of once per + fetch timeout `T`: at most `C / min(5 s, T)` small requests with `C` hardware-thread fetch + slots, against `C / T` today, in the case where every holder refuses everything; with the + default 30 s timeout, `C / 5 s` against `C / 30 s`. Capping `C` is a separate decision. - Refusals beyond the per-peer busy allowance stay silent and are still charged by saorsa-core. Without #252, a requester that over-shares by more than two requests at once still meets some - silence; the simulation below gives the residual. + silence. - Busy is trust-neutral. A holder that tries to answer busy to everything can still be charged when - a request did not ask, a reply was suppressed, or a scheduled send did not complete; otherwise it - is passed over for new keys for up to 60 s at a time and remains subject to storage audits. + a request did not ask, a reply was suppressed, or a scheduled send did not complete; otherwise, + when this requester starts or retries a key, it is put behind every holder that is not cooling + (when all are cooling, the first is still tried), for up to 60 s at a time, and remains subject + to storage audits. Penalising "never serves" would need a service audit with reserved capacity, which is not part of this decision. - Sixteen identities each spending their per-peer second can use up the global allowance and @@ -149,9 +160,11 @@ busy allowance and stay silent. On rebase, #252's source choice also skips cooli ### Neutral / Operational -- New counters on the `replication responder admission summary (cumulative)` line: +- New fields on the `replication responder admission summary (cumulative)` line: the cumulative + counters `fetch_busy_reply_{not_asked,send_pool_full,rate_limited,peer_in_flight,sent,send_failed,timed_out,cancelled}`, - `fetch_busy_received`, `fetch_busy_cooling_skips` and `fetch_busy_sources_tracked`. + `fetch_busy_received` and `fetch_busy_cooling_skips`, and the gauge `fetch_busy_sources_tracked`, + the current size of the requester's busy-holder table. `not_asked` against the rest is the share of refused requests that did not ask for a busy reply: a proxy, and an upper bound, for how much refused traffic still comes from older versions (a request from a current node can also not ask, through the API or a malformed trailer). @@ -165,7 +178,8 @@ busy allowance and stay silent. On rebase, #252's source choice also skips cooli - an older requester cannot decode `Busy`; - limiter: burst, refill, global cap, no spend on a denial, in-flight limit, bounded table; - the busy wait's range and its cap at the fetch timeout; - - tracker: cooldown, escalation, cap, never shortened, reset, expiry, bounded table; + - tracker: cooldown, escalation, cap, not shortened by a later busy answer, reset, expiry, + bounded table; - a retry passes over a cooling holder but still takes it last; - a busy reply is not counted as served bytes. - End to end on a local network, made deterministic with a test-only fetch serve gate (the same @@ -173,34 +187,26 @@ busy allowance and stay silent. On rebase, #252's source choice also skips cooli - a refused request that asks is told `Busy` at once; - one that does not ask is refused silently; - the held requests are served once the gate opens; - - the requester's own fetch worker, refused by a full holder, records the holder as cooling, - keeps the attempt for at least the minimum busy wait, then gives the key up, and the holder's - trust at the requester never drops. -- Measured on real nodes against upstream main with the same test-only gate, in + - the requester's own fetch worker, refused because its two-request share at the holder is + full, records the holder as cooling, keeps the attempt for at least the minimum busy wait, + then gives the key up, and the holder's trust at the requester never drops. +- Measured on real nodes against upstream main a6e3388e with the same test-only gate, in `tests/e2e/fetch_busy_ab.rs` (opt-in): - 20 requesters each within their two-request share, the first of five holders with its - 12-slot pool full for 2 s (28 refusals in both builds), which is the refusal left after #252: - all 40 chunks in 10.5 s, no holder charged, against 60.4 s with 14 requesters charging the - first holder and 4 the second; - - one requester with 8 fetch slots over-asking three holders, which #252 removes: 39.7 s, 39.4 s - and 36.7 s with only the first holder charged (4 charges), against 60.2 s, 90.3 s and 90.2 s with - up to all three charged. -- A discrete-event simulation of the 177-node join wave and 24 more scenarios (dead holders, - holders that always answer busy, saturated pools, mixed versions, 1-64 fetch slots, - single-holder keys, limiter and wait sensitivity), comparing today, this change, #252-style - capping, and both. It reproduces the real-node run of this change and overstates the cost of - today's silence, so it gives directions, not numbers. In it no key is stranded in any scenario. - With nobody answering busy it behaves exactly as today. With #252-style capping, the busy reply - removes every false charge in the background-load, overload, always-busy and dense scenarios; - what remains elsewhere comes from requests a slow holder sheds unanswered after the fetch timeout - and from requesters that do not ask. Without capping it raises requests per chunk - by up to 12% in some scenarios and lowers it by up to 17% in others (with a 1-2 s wait instead of -5-10 s it rises 16%, which is why the wait is longer), and keys with a single holder go back to -verification about twice as often. + 12-slot pool full for 2 s (26 refusals with this change, 28 on main), one of the refusal + cases left after #252: all 40 chunks in 10.5 s, no holder charged, against 90.3 s with 14, + 9, 9, 4 and 3 requesters charging the five holders (one run each; on the previous main + 8d25f0c3 it was 60.4 s with 14 and 4 requesters charging the first two); + - one requester with 8 fetch slots over-asking three holders, which #252 removes: 38.1 s, 39.2 s + and 37.2 s with only the first holder charged (4 charges), against 90.2 s, 90.2 s and 90.3 s + with all three charged (6 charges on the first, 1 to 4 on the others). +- A local, unpublished discrete-event model of the 177-node join wave and 24 more scenarios + informed the design, including the 5-10 s wait. Its source and raw output are not part of this + change, so its results are not offered as validation. - In production: - `fetch_busy_reply_sent` against `fetch_admission_dropped_*`; - the fall in `connection_timeout` trust events between fleet nodes; - - `fetch_busy_reply_rate_limited` and `send_pool_full` staying small. + - `fetch_busy_reply_rate_limited` and `fetch_busy_reply_send_pool_full` staying small. Revisit the 5-10 s wait and the allowances against measured admission hold times. diff --git a/src/replication/fetch_busy.rs b/src/replication/fetch_busy.rs index e6be386f..b651b304 100644 --- a/src/replication/fetch_busy.rs +++ b/src/replication/fetch_busy.rs @@ -325,8 +325,9 @@ impl SourceBusyTracker { /// Record a busy answer from `source` at `now`. `wait` is the busy wait /// before any cap from the attempt's own timeout. Returns how long the /// source is now put last: `wait` the first time, doubling with each further - /// busy answer before it serves again, up to a minute. A - /// cooldown already running is never shortened. + /// busy answer before it serves again, up to a minute. A cooldown already + /// running is never shortened, so when it outlasts the new one, what is + /// left of it is returned. pub fn record_busy(&self, source: &PeerId, now: Instant, wait: Duration) -> Duration { let mut sources = self.sources.lock(); if sources @@ -363,8 +364,9 @@ impl SourceBusyTracker { let until = now.checked_add(cooldown).unwrap_or(now); record.cooling_until = record.cooling_until.max(until); record.last_busy = now; + let cooling_for = record.cooling_until.saturating_duration_since(now); audit_metrics::set_fetch_busy_sources_tracked(sources.len()); - cooldown + cooling_for } /// `source` served this requester: forget its busy record. @@ -687,8 +689,12 @@ mod tests { tracker.record_busy(&a, t0, Duration::from_secs(10)); } // Cooling until t0 + 60 s now. A later busy answer with a tiny wait - // would end far sooner; the running cooldown stands. - tracker.record_busy(&a, t0 + Duration::from_secs(1), Duration::from_millis(1)); + // would end far sooner; the running cooldown stands, and what is left + // of it is what the caller is told. + assert_eq!( + tracker.record_busy(&a, t0 + Duration::from_secs(1), Duration::from_millis(1)), + Duration::from_secs(59) + ); assert!(tracker.is_cooling(&a, t0 + Duration::from_secs(59))); } diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 7288160e..3d694c2f 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -9660,7 +9660,7 @@ async fn execute_single_fetch( let cooldown = busy_sources.record_busy(&source, Instant::now(), full_wait); debug!( "Fetch: source {source} busy for {}; moving on in {} ms, \ - passing it over for new keys for {} ms", + putting it last for {} ms", hex::encode(key), wait.as_millis(), cooldown.as_millis() From a5490a9368693439e3743394bd001f6e628a747b Mon Sep 17 00:00:00 2001 From: grumbach Date: Tue, 20 Oct 2026 16:04:51 +0900 Subject: [PATCH 03/10] fix(replication): bound how long a holder may stay busy at no cost A busy answer was trust-neutral without limit. A holder that kept the data and answered its storage audits could refuse every fetch with Busy for good at no cost: the requester's SourceBusyTracker only put it last and never withheld it. Each holder now gets FETCH_BUSY_NEUTRAL_ALLOWANCE, eight busy answers in a row to a requester, counted across all keys, before it next serves that requester. Past the allowance, each further busy answer is handled as the silent refusal it replaces. The attempt holds its slot until its fetch timeout runs out, never longer however late the answer came, and the holder is then charged one unit-weight ApplicationFailure, which saorsa-core scores the same as the ConnectionTimeout that silence earned. The timeout counts from before any dial, while silence's counts from after it, so after a cold dial the charge can come up to the dial's time sooner. An attempt cancelled by shutdown first is not charged. Only a served chunk restores the allowance. Time does not, and the 4096-entry table never forgets a record to make room: when it is full, a holder it does not track yet is charged as for silence. Ten minutes without a busy answer still restart the cooldown multiplier, but not the count. The count lives in memory, so a requester restart starts every holder afresh. SourceBusyTracker::record_busy now returns BusyAnswer { cooling_for, charged }, and busy_wake says when the attempt then lets go of its slot. The admission summary line gains a fetch_busy_charged counter. New unit tests cover: - a late answer never holding the slot past the fetch timeout; - a holder that never serves; - answers that arrive together; - serving between spells; - recovery; - time alone; - a full table. A new end-to-end test has a real holder refuse the requester's fetch worker key after key until it is charged, as its attempt ends, and then clears the holder's count once it serves. The test-utils hook fetch_source_unserved_busy_for_test exposes that count. ADR-0025 records the decision. --- docs/adr/ADR-0025-fetch-busy-reply.md | 117 +++++--- src/replication/audit_metrics.rs | 18 ++ src/replication/fetch_busy.rs | 387 +++++++++++++++++++++----- src/replication/mod.rs | 63 +++-- tests/e2e/fetch_busy_reply.rs | 207 +++++++++++++- 5 files changed, 655 insertions(+), 137 deletions(-) diff --git a/docs/adr/ADR-0025-fetch-busy-reply.md b/docs/adr/ADR-0025-fetch-busy-reply.md index f6f5a8c3..98677709 100644 --- a/docs/adr/ADR-0025-fetch-busy-reply.md +++ b/docs/adr/ADR-0025-fetch-busy-reply.md @@ -43,8 +43,10 @@ as a wire change with its own rollout. - Most peers run older versions for weeks after any release. An old node must decode the same messages it always did and never be sent a `Busy` it cannot decode. -- A delivered `Busy` must not cost an honest responder trust; a refusal that cannot be answered - stays the old silence. +- A delivered `Busy` within its sender's neutral allowance must not cost it trust; a refusal that + cannot be answered stays the old silence. +- A holder must not be able to keep the data, pass its storage audits and answer `Busy` to every + fetch for good at no cost: neutrality has to be bounded per holder, across keys. - The added reply traffic must be bounded: no request storm from requesters retrying on every busy answer, and no echo amplification at responders. - No new requester scheduling state that would collide with #252's per-source scheduler. @@ -67,6 +69,10 @@ as a wire change with its own rollout. busy answer turns each refusal into an immediate new request, so a requester whose holders all refuse asks once per round trip instead of once per fetch timeout; it competes with `tried` state; and charging at budget expiry punishes honest overloaded holders once per key per cycle. +6. **Busy trust-neutral without a limit.** The first version of this change. Rejected in review: a + holder could answer `Busy` forever, putting off every fetch at no trust cost while it kept + answering storage audits. A per-key retry limit or a longer cooldown does not fix that, because + the same holder can be asked about any number of keys. ## Decision @@ -95,10 +101,23 @@ otherwise it refuses silently, exactly as before. The `Fetch response dropped at WARN line and its counters are unchanged, with `capability=asked|not_asked` and `reply=scheduled|none` fields added. -**Requester.** Every fetch request this node's fetch worker sends asks for busy replies. A `Busy` for the requested key carries -no trust event. The attempt then waits 5-10 s (uniform jitter), never past its own fetch timeout, -still holding its fetch slot, and ends as `SourceFailed`, so the queue moves on to the next untried -holder exactly as after a timeout. `SourceBusyTracker` remembers the busy holder: a new key starts, +**Requester.** Every fetch request this node's fetch worker sends asks for busy replies. A `Busy` +for the requested key carries no trust event while its holder is within its neutral allowance +(`FETCH_BUSY_NEUTRAL_ALLOWANCE`): eight busy answers in a row to this requester, counted across all +keys, since the holder last served it. The attempt then waits 5-10 s (uniform jitter), never past +its own fetch timeout, still holding its fetch slot, and ends as `SourceFailed`, so the queue moves +on to the next untried holder exactly as after a timeout. Past the allowance, each further busy +answer is handled as the silent refusal it replaces: the attempt holds its slot until its fetch +timeout, counted from when it began, runs out, never longer however late the answer came, and +then the requester records one `ApplicationFailure` of weight 1.0, which saorsa-core scores +exactly as the `ConnectionTimeout` a silent refusal earned (one failed response of unit weight). +An attempt cancelled by shutdown before then is not charged. Silence's timeout starts once any +dial is done, so after a cold dial the charge can come up to that dial's time sooner than +silence's would have. Only serving this requester once, with a chunk that passes its integrity +checks, restores the allowance; time does not, so a holder asked only rarely still uses it up. +The allowance is per requester: each node judges only its own fetches. + +`SourceBusyTracker` remembers the busy holder: a new key starts, and a failed key retries, on the first holder in preference order that is not cooling down, falling back to the first when all are. It reorders and never withholds, so it lowers how often a busy holder is asked but is not a hard per-holder bound; the hard bound is the busy wait per fetch @@ -106,24 +125,28 @@ slot. The cooldown is 1, 2, 4, then 8 times the full jittered wait drawn for tha the multiplier rising with each further busy answer before the holder next serves this requester, capped at 60 s, and a later busy answer never shortens a running cooldown; once it runs out the holder is no longer put last. Serving once clears the holder's record, so its next -busy answer starts the multiplier again; a record with no busy answer for 10 minutes does the -same. When the 4096-entry table is full, a holder that is no longer cooling is forgotten first, -or, if every holder in it is cooling, the one whose cooldown ends soonest. A `Busy` for any -other key is treated as malformed, like every other key mismatch. +busy answer starts the multiplier again; after 10 minutes without a busy answer, the next one +starts the multiplier again too, but leaves what the holder owes against its allowance as it was. +Records go only when their holder serves: not for age, and not to make room. When the +4096-entry table is full, a holder it does not already track is charged for each busy answer, +as for silence, and is not put last. A `Busy` for any other key is treated as malformed, like +every other key mismatch. **Order.** This is merged after #252. #252's per-source share is what removes most per-peer over-share and bounds how many fetches one requester has open at one answering holder; without it, a requester with many fetch slots still over-shares, and most of those refusals exceed the -per-peer busy allowance and stay silent. On rebase, #252's source choice also puts cooling holders -last. The two changes conflict in `src/replication/mod.rs` and have not yet been built or measured -together; the gates and the real-node measurements below will be rerun on the rebased change. +per-peer busy-reply allowance and stay silent. On rebase, #252's source choice also puts cooling +holders last. The two changes conflict in `src/replication/mod.rs` and have not yet been built or +measured together; the gates and the real-node measurements below will be rerun on the rebased +change. ## Consequences ### Positive -- When a refused requester receives `Busy`, it moves on after 5-10 s, or sooner if its own fetch - timeout runs out first, instead of 30 s, without charging the responder. +- When a refused requester receives `Busy` within the responder's neutral allowance, it moves on + after 5-10 s, or sooner if its own fetch timeout runs out first, instead of 30 s, without + charging the responder. - A holder that just refused is not handed this requester's next keys while another holder that is not cooling exists. - Old nodes decode exactly the messages they always did: a request carries 5 bytes after the @@ -142,16 +165,26 @@ together; the gates and the real-node measurements below will be rerun on the re fetch timeout `T`: at most `C / min(5 s, T)` small requests with `C` hardware-thread fetch slots, against `C / T` today, in the case where every holder refuses everything; with the default 30 s timeout, `C / 5 s` against `C / 30 s`. Capping `C` is a separate decision. -- Refusals beyond the per-peer busy allowance stay silent and are still charged by saorsa-core. +- Refusals beyond the per-peer busy-reply allowance stay silent and are still charged by saorsa-core. Without #252, a requester that over-shares by more than two requests at once still meets some silence. -- Busy is trust-neutral. A holder that tries to answer busy to everything can still be charged when - a request did not ask, a reply was suppressed, or a scheduled send did not complete; otherwise, - when this requester starts or retries a key, it is put behind every holder that is not cooling - (when all are cooling, the first is still tried), for up to 60 s at a time, and remains subject - to storage audits. - Penalising "never serves" would need a service audit with reserved capacity, which is not part - of this decision. +- Busy is trust-neutral only within the allowance. A holder that never serves gets eight free busy + answers in a row per requester, then each further refusal costs it what silence cost, so past + the allowance it does no better than before this change. An honest holder that stays overloaded + past the allowance is likewise charged per refusal, as it was before this change. A holder that + serves one fetch in nine is never charged: bounding selective service more tightly would need a + service audit with reserved capacity, which is not part of this decision. +- The count does not decay with time. An honest holder that, over any length of time, answers one + requester busy nine times without serving it once in between is charged for the ninth, as + silence would have charged every one. +- The count lives in memory, so a requester restart starts every holder afresh: at most eight free + busy answers per holder per run of the requester. Keeping it across restarts would need a new + on-disk format. A holder does not control when a requester restarts. +- A full table keeps every count, so it charges holders it does not track yet from their first + busy answer, as silence did. That takes 4096 holders each answering this requester busy + without serving it since; a node fetches from its neighbourhood, far fewer peers, and the table + empties on restart. As with the global busy-reply allowance, what identities filling it on + purpose win is the old silence for everyone else, not a new failure. - Sixteen identities each spending their per-peer second can use up the global allowance and deny busy replies to everyone else, and stalled peers can hold the 48 send permits for up to 2 s each. What they win is the old silence for those peers, not a new failure. The global @@ -163,7 +196,8 @@ together; the gates and the real-node measurements below will be rerun on the re - New fields on the `replication responder admission summary (cumulative)` line: the cumulative counters `fetch_busy_reply_{not_asked,send_pool_full,rate_limited,peer_in_flight,sent,send_failed,timed_out,cancelled}`, - `fetch_busy_received` and `fetch_busy_cooling_skips`, and the gauge `fetch_busy_sources_tracked`, + `fetch_busy_received`, `fetch_busy_charged` (busy answers charged past the neutral allowance) and + `fetch_busy_cooling_skips`, and the gauge `fetch_busy_sources_tracked`, the current size of the requester's busy-holder table. `not_asked` against the rest is the share of refused requests that did not ask for a busy reply: a proxy, and an upper bound, for how much refused traffic still comes from older versions (a @@ -177,9 +211,16 @@ together; the gates and the real-node measurements below will be rerun on the re - trailer parsing: exact, wrong magic, truncated, unknown bits, reserved bytes, padding; - an older requester cannot decode `Busy`; - limiter: burst, refill, global cap, no spend on a denial, in-flight limit, bounded table; - - the busy wait's range and its cap at the fetch timeout; + - the busy wait's range and its cap at the fetch timeout, and that a busy answer, however late + and whether charged or not, never holds the slot past the attempt's fetch timeout; - tracker: cooldown, escalation, cap, not shortened by a later busy answer, reset, expiry, bounded table; + - neutral allowance: a sole holder that answers busy every 5 s for over an hour without serving + is still chosen each time, and every answer after the eighth is charged; two answers at once + spend two; a holder that serves once between spells of eight is never charged; serving + restores the whole allowance and another holder serving does not; answers an hour apart + still use it up; a full table forgets no count while one more holder than it holds takes + turns with the rest, charges that newcomer instead, and makes room once a holder serves; - a retry passes over a cooling holder but still takes it last; - a busy reply is not counted as served bytes. - End to end on a local network, made deterministic with a test-only fetch serve gate (the same @@ -189,16 +230,22 @@ together; the gates and the real-node measurements below will be rerun on the re - the held requests are served once the gate opens; - the requester's own fetch worker, refused because its two-request share at the holder is full, records the holder as cooling, keeps the attempt for at least the minimum busy wait, - then gives the key up, and the holder's trust at the requester never drops. + then gives the key up, and the holder's trust at the requester never drops; + - a holder that keeps refusing the requester's fetch worker, one new key after another, costs + nothing at the requester for eight busy answers, and is charged for the ninth only as that + attempt's fetch timeout runs out; once it serves the worker a chunk, what it owes is + cleared. - Measured on real nodes against upstream main a6e3388e with the same test-only gate, in - `tests/e2e/fetch_busy_ab.rs` (opt-in): + `tests/e2e/fetch_busy_ab.rs` (opt-in), with the neutral allowance in place: - 20 requesters each within their two-request share, the first of five holders with its - 12-slot pool full for 2 s (26 refusals with this change, 28 on main), one of the refusal - cases left after #252: all 40 chunks in 10.5 s, no holder charged, against 90.3 s with 14, - 9, 9, 4 and 3 requesters charging the five holders (one run each; on the previous main - 8d25f0c3 it was 60.4 s with 14 and 4 requesters charging the first two); - - one requester with 8 fetch slots over-asking three holders, which #252 removes: 38.1 s, 39.2 s - and 37.2 s with only the first holder charged (4 charges), against 90.2 s, 90.2 s and 90.3 s + 12-slot pool full for 2 s (26 to 28 refusals with this change, 28 on main), one of the + refusal cases left after #252: all 40 chunks in 10.1 s, 10.1 s and 9.4 s, no holder charged, + against one run on main at 90.3 s with 14, 9, 9, 4 and 3 requesters charging the five + holders (on the previous main 8d25f0c3 it was 60.4 s with 14 and 4 requesters charging the + first two). Two more runs with this change stopped before measuring, at the test's check that + every key was newly queued (39 of 40); + - one requester with 8 fetch slots over-asking three holders, which #252 removes: 38.3 s, 38.9 s + and 35.4 s with only the first holder charged (4 charges), against 90.2 s, 90.2 s and 90.3 s with all three charged (6 charges on the first, 1 to 4 on the others). - A local, unpublished discrete-event model of the 177-node join wave and 24 more scenarios informed the design, including the 5-10 s wait. Its source and raw output are not part of this @@ -206,7 +253,9 @@ together; the gates and the real-node measurements below will be rerun on the re - In production: - `fetch_busy_reply_sent` against `fetch_admission_dropped_*`; - the fall in `connection_timeout` trust events between fleet nodes; - - `fetch_busy_reply_rate_limited` and `fetch_busy_reply_send_pool_full` staying small. + - `fetch_busy_reply_rate_limited` and `fetch_busy_reply_send_pool_full` staying small; + - `fetch_busy_charged` against `fetch_busy_received`: how often holders stay busy past the + neutral allowance. Revisit the 5-10 s wait and the allowances against measured admission hold times. diff --git a/src/replication/audit_metrics.rs b/src/replication/audit_metrics.rs index 0989d5b2..281476b0 100644 --- a/src/replication/audit_metrics.rs +++ b/src/replication/audit_metrics.rs @@ -408,6 +408,10 @@ static FETCH_BUSY_REPLIES: [AtomicU64; FetchBusyReplyOutcome::COUNT] = /// Busy answers this node received to its own fetches. static FETCH_BUSY_RECEIVED: AtomicU64 = AtomicU64::new(0); +/// Busy answers this node charged, because their source had used up its +/// neutral allowance without serving it. +static FETCH_BUSY_CHARGED: AtomicU64 = AtomicU64::new(0); + /// Times this node, choosing where to start or retry a fetch, passed over a /// holder that had recently answered it busy. static FETCH_BUSY_COOLING_SKIPS: AtomicU64 = AtomicU64::new(0); @@ -614,6 +618,19 @@ pub fn fetch_busy_received_total() -> u64 { FETCH_BUSY_RECEIVED.load(Ordering::Relaxed) } +/// Count a busy answer charged because its source was past its neutral +/// allowance. +pub fn record_fetch_busy_charged() { + FETCH_BUSY_CHARGED.fetch_add(1, Ordering::Relaxed); +} + +/// Busy answers this node charged since process start. +#[must_use] +#[cfg_attr(not(feature = "logging"), allow(dead_code))] +pub fn fetch_busy_charged_total() -> u64 { + FETCH_BUSY_CHARGED.load(Ordering::Relaxed) +} + /// Count a fetch started or retried on a later holder because an earlier one /// had recently answered busy. pub fn record_fetch_busy_cooling_skip() { @@ -759,6 +776,7 @@ pub fn log_responder_admission_summary() { fetch_busy_reply_timed_out = fetch_busy_replies_total(FetchBusyReplyOutcome::TimedOut), fetch_busy_reply_cancelled = fetch_busy_replies_total(FetchBusyReplyOutcome::Cancelled), fetch_busy_received = fetch_busy_received_total(), + fetch_busy_charged = fetch_busy_charged_total(), fetch_busy_cooling_skips = fetch_busy_cooling_skips_total(), fetch_busy_sources_tracked = fetch_busy_sources_tracked(), verification_admission_dropped_global_pool = verification.global_pool, diff --git a/src/replication/fetch_busy.rs b/src/replication/fetch_busy.rs index b651b304..6a9c6b65 100644 --- a/src/replication/fetch_busy.rs +++ b/src/replication/fetch_busy.rs @@ -20,11 +20,21 @@ //! [`busy_wait`] keeps the attempt, and so its fetch slot, for a few seconds //! before the key moves on, so a requester asks at most once per wait per //! slot instead of as fast as answers come back. The attempt then ends as a -//! source failure, exactly as the silent timeout did, minus the trust charge. +//! source failure, exactly as the silent timeout did, minus the trust charge +//! while the source is within its neutral allowance (see below). //! [`SourceBusyTracker`] also puts a busy source last when choosing where to //! start or retry a key, for longer each time it stays busy, so a requester //! does not keep picking a holder that just said it cannot take more. //! +//! A busy answer costs no trust only while it can still be passing overload. +//! A source gets [`FETCH_BUSY_NEUTRAL_ALLOWANCE`] busy answers in a row, across +//! all keys, before it next serves this requester. Past that, until it serves +//! once, each busy answer is handled as the silent refusal it replaces: the +//! attempt holds its slot until its fetch timeout runs out, and the source is +//! then charged the same trust. +//! Without that bound a holder could keep the data, answer its storage audits +//! and say busy to every fetch for good, at no cost. +//! //! [`FetchResponse::Busy`]: crate::replication::protocol::FetchResponse::Busy use std::collections::HashMap; @@ -55,6 +65,27 @@ pub const FETCH_BUSY_WAIT_MAX: Duration = Duration::from_secs(10); pub const FETCH_BUSY_WAIT_SPREAD: Duration = FETCH_BUSY_WAIT_MAX.saturating_sub(FETCH_BUSY_WAIT_MIN); +/// Busy answers in a row that a source may give this requester, before it +/// next serves it, at no trust cost. +/// +/// Honest overload passes: a share frees in seconds. A source that keeps +/// answering busy cannot be told apart from one that keeps the data and will +/// not serve it. So past this many, until the source serves this requester +/// once, each busy answer is handled as the silent refusal it replaces: the +/// attempt holds its slot until its fetch timeout, counted from when it +/// began, runs out, and the source is then charged the same trust. Silence's +/// timeout starts once any dial is done, so after a cold dial the charge can +/// come up to that dial's time sooner. Only serving restores the allowance; +/// time does not. +/// +/// The count lives in memory, so a restart of the requester starts every +/// source afresh: at most this many free busy answers per source per run. +/// +/// With two answers a round, one for each request in a peer's reply burst, +/// eight is four rounds of being put last: minutes, when the requester has +/// other holders to ask, and well under a minute when it has none. +pub const FETCH_BUSY_NEUTRAL_ALLOWANCE: u32 = 8; + /// Longest a responder spends handing one busy reply to the transport. /// /// A reply that cannot be handed over in this long is abandoned: the requester @@ -104,12 +135,18 @@ const MAX_SOURCE_COOLDOWN: Duration = Duration::from_secs(60); /// then two, four and eight times it, capped at [`MAX_SOURCE_COOLDOWN`]. const MAX_COOLDOWN_DOUBLINGS: u32 = 3; -/// How long a source's busy record is kept after its last busy answer. Long -/// enough to remember a holder that stays busy across many cooldowns; short -/// enough that one busy spell is not held against it for good. -const SOURCE_RECORD_TTL: Duration = Duration::from_secs(600); +/// A source that has not answered busy for this long starts a new busy spell +/// with its next busy answer: its cooldown starts again from the wait, so one +/// busy spell is not held against its scheduling for good. What it owes +/// against its neutral allowance does not start again. +const BUSY_SPELL_GAP: Duration = Duration::from_secs(600); /// Most sources whose busy record is kept at once. +/// +/// Records go only when their source serves, so a full table forgets nothing +/// to make room. A source it does not already hold is then charged for every +/// busy answer, as for silence, and is not put last. A node fetches from its +/// neighbourhood, far fewer peers than this, and the table empties on restart. const MAX_TRACKED_SOURCES: usize = 4096; /// How long to wait after a busy answer, given `remaining`, the time left @@ -127,6 +164,25 @@ pub fn busy_wait(remaining: Duration, jitter: Duration) -> Duration { .min(remaining) } +/// When an attempt answered busy at `now` lets go of its fetch slot, given +/// `wait`, the busy wait drawn for it, and `deadline`, when the attempt's fetch +/// timeout runs out. +/// +/// Within the neutral allowance it is once the busy wait is over; past it, at +/// the deadline, as silence would have held it. Either way it is never after +/// `deadline`, so however late a busy answer comes, the slot is held no longer +/// than silence would have held it. A fixed time rather than a length, so +/// nothing between the answer and the sleep can stretch it. +#[must_use] +pub fn busy_wake(answer: BusyAnswer, wait: Duration, now: Instant, deadline: Instant) -> Instant { + if answer.charged { + deadline + } else { + now.checked_add(wait) + .map_or(deadline, |done| done.min(deadline)) + } +} + /// A generic cell rate limiter: `burst` events back to back, then one per /// `interval`. /// @@ -277,11 +333,24 @@ impl Drop for BusyReplyGrant { } } -/// One source's recent busy answers to this requester. +/// What one busy answer means for its source at this requester. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct BusyAnswer { + /// How long the source is now put last. + pub cooling_for: Duration, + /// Whether the answer is past the source's neutral allowance, and so is + /// handled as the silent refusal it replaces. + pub charged: bool, +} + +/// One source's busy answers to this requester since it last served it. #[derive(Debug, Clone, Copy)] struct SourceBusy { - /// Busy answers since it last served this requester. + /// Busy answers in its current busy spell, which set its cooldown. strikes: u32, + /// Busy answers since it last served this requester, however long ago: + /// what spends its neutral allowance. + unserved: u32, /// Until when it is put last. cooling_until: Instant, /// When it last answered busy. @@ -293,23 +362,29 @@ impl SourceBusy { now < self.cooling_until } - fn expired(self, now: Instant) -> bool { - !self.is_cooling(now) && now.saturating_duration_since(self.last_busy) >= SOURCE_RECORD_TTL + fn spell_over(self, now: Instant) -> bool { + !self.is_cooling(now) && now.saturating_duration_since(self.last_busy) >= BUSY_SPELL_GAP } } -/// Which sources recently answered this requester busy, and until when they -/// should be asked last. +/// Which sources recently answered this requester busy, until when they +/// should be asked last, and which have used up their neutral allowance. +/// +/// Mostly local scheduling: a holder that just refused is the worst place to +/// send the next fetch, and one that keeps refusing is put last for longer +/// each time. It orders sources and never withholds one: a key whose every +/// holder is cooling still starts on the first of them, and then waits out +/// that holder's busy answer like any other. So per requester it lowers how +/// often a busy holder is asked; the hard bound is the busy wait per fetch +/// slot. /// -/// A busy answer is not a fault, so nothing here touches trust. It is local -/// scheduling: a holder that just refused is the worst place to send the next -/// fetch, and one that keeps refusing is put last for longer each time. -/// Serving once clears its record. It orders sources and never withholds one: -/// a key whose every holder is cooling still starts on the first of them, and -/// then waits out that holder's busy answer like any other. So per requester -/// it lowers how often a busy holder is asked; the hard bound is the busy wait -/// per fetch slot. Synchronous and short, so it is safe to consult while -/// holding the fetch queues lock. +/// The same record counts a source's busy answers in a row, across all keys, +/// against [`FETCH_BUSY_NEUTRAL_ALLOWANCE`]. It does not touch trust itself; +/// it tells the caller which busy answer is past the allowance. Only serving +/// clears the record, and so restores the allowance: neither time nor a full +/// table does. The table is not saved, so a restart clears it. +/// Synchronous and short, so it is safe to consult while holding the fetch +/// queues lock. #[derive(Debug, Default)] pub struct SourceBusyTracker { sources: Mutex>, @@ -323,40 +398,36 @@ impl SourceBusyTracker { } /// Record a busy answer from `source` at `now`. `wait` is the busy wait - /// before any cap from the attempt's own timeout. Returns how long the - /// source is now put last: `wait` the first time, doubling with each further - /// busy answer before it serves again, up to a minute. A cooldown already - /// running is never shortened, so when it outlasts the new one, what is - /// left of it is returned. - pub fn record_busy(&self, source: &PeerId, now: Instant, wait: Duration) -> Duration { + /// before any cap from the attempt's own timeout. + /// + /// Returns how long the source is now put last: `wait` the first time, + /// doubling with each further busy answer in the same busy spell, up to a + /// minute. A cooldown already running is never shortened, so when it + /// outlasts the new one, what is left of it is returned. Also returns + /// whether this answer is past the source's + /// [`FETCH_BUSY_NEUTRAL_ALLOWANCE`]. When the table is full and holds no + /// record of `source`, the answer is charged and nothing is recorded. + pub fn record_busy(&self, source: &PeerId, now: Instant, wait: Duration) -> BusyAnswer { let mut sources = self.sources.lock(); - if sources - .get(source) - .is_some_and(|record| record.expired(now)) - { - sources.remove(source); - } if !sources.contains_key(source) && sources.len() >= MAX_TRACKED_SOURCES { - sources.retain(|_, record| !record.expired(now)); - if sources.len() >= MAX_TRACKED_SOURCES { - // Prefer to forget a source that is not cooling. Forgetting - // one that is ends its cooldown early; the busy wait still - // bounds how often it is asked. - let victim = sources - .iter() - .min_by_key(|(_, record)| (record.is_cooling(now), record.cooling_until)) - .map(|(peer, _)| *peer); - if let Some(peer) = victim { - sources.remove(&peer); - } - } + // Forgetting any record would restore its source's allowance, so + // with no room this answer gets none. + return BusyAnswer { + cooling_for: Duration::ZERO, + charged: true, + }; } let record = sources.entry(*source).or_insert(SourceBusy { strikes: 0, + unserved: 0, cooling_until: now, last_busy: now, }); + if record.spell_over(now) { + record.strikes = 0; + } record.strikes = record.strikes.saturating_add(1); + record.unserved = record.unserved.saturating_add(1); let doublings = record.strikes.saturating_sub(1).min(MAX_COOLDOWN_DOUBLINGS); let cooldown = wait .saturating_mul(1u32 << doublings) @@ -364,12 +435,16 @@ impl SourceBusyTracker { let until = now.checked_add(cooldown).unwrap_or(now); record.cooling_until = record.cooling_until.max(until); record.last_busy = now; - let cooling_for = record.cooling_until.saturating_duration_since(now); + let answer = BusyAnswer { + cooling_for: record.cooling_until.saturating_duration_since(now), + charged: record.unserved > FETCH_BUSY_NEUTRAL_ALLOWANCE, + }; audit_metrics::set_fetch_busy_sources_tracked(sources.len()); - cooling_for + answer } - /// `source` served this requester: forget its busy record. + /// `source` served this requester: forget its busy record, which restores + /// its neutral allowance. pub fn record_served(&self, source: &PeerId) { let mut sources = self.sources.lock(); if sources.remove(source).is_some() { @@ -410,6 +485,16 @@ impl SourceBusyTracker { first } + /// Busy answers from `source` since it last served this requester, as far + /// as it is remembered. + #[must_use] + pub fn unserved_busy(&self, source: &PeerId) -> u32 { + self.sources + .lock() + .get(source) + .map_or(0, |record| record.unserved) + } + /// When `source` last answered busy, if it is still remembered. #[must_use] pub fn last_busy(&self, source: &PeerId) -> Option { @@ -485,6 +570,43 @@ mod tests { ); } + #[test] + fn a_busy_answer_never_holds_the_slot_past_the_fetch_timeout() { + let neutral = BusyAnswer { + cooling_for: FETCH_BUSY_WAIT_MIN, + charged: false, + }; + let charged = BusyAnswer { + cooling_for: FETCH_BUSY_WAIT_MIN, + charged: true, + }; + let sent = Instant::now(); + let deadline = sent + Duration::from_secs(30); + // A prompt answer: the busy wait within the allowance, the rest of the + // timeout past it. + assert_eq!( + busy_wake(neutral, FETCH_BUSY_WAIT_MAX, sent, deadline), + sent + FETCH_BUSY_WAIT_MAX + ); + assert_eq!( + busy_wake(charged, FETCH_BUSY_WAIT_MAX, sent, deadline), + deadline + ); + // A late answer, a second before the deadline, holds the slot for that + // second and no more; one after it, not at all. + for now in [ + sent + Duration::from_secs(29), + deadline + Duration::from_secs(5), + ] { + for answer in [neutral, charged] { + assert_eq!( + busy_wake(answer, FETCH_BUSY_WAIT_MAX, now, deadline), + deadline + ); + } + } + } + #[test] fn a_peer_gets_its_burst_then_one_reply_per_interval() { let limiter = limiter(); @@ -643,7 +765,7 @@ mod tests { let sources = [a, b, c]; let t0 = Instant::now(); assert_eq!(tracker.choose(&sources, t0), Some(&a)); - assert_eq!(tracker.record_busy(&a, t0, WAIT), WAIT); + assert_eq!(tracker.record_busy(&a, t0, WAIT).cooling_for, WAIT); assert!(tracker.is_cooling(&a, t0)); assert_eq!(tracker.choose(&sources, t0), Some(&b)); // Cooling ends after the wait. @@ -666,7 +788,9 @@ mod tests { let tracker = SourceBusyTracker::new(); let a = peer(1); let t0 = Instant::now(); - let cooldowns: Vec = (0..6).map(|_| tracker.record_busy(&a, t0, WAIT)).collect(); + let cooldowns: Vec = (0..6) + .map(|_| tracker.record_busy(&a, t0, WAIT).cooling_for) + .collect(); assert_eq!( cooldowns, vec![WAIT, WAIT * 2, WAIT * 4, WAIT * 8, WAIT * 8, WAIT * 8] @@ -677,7 +801,10 @@ mod tests { for _ in 0..4 { tracker.record_busy(&b, t0, long); } - assert_eq!(tracker.record_busy(&b, t0, long), MAX_SOURCE_COOLDOWN); + assert_eq!( + tracker.record_busy(&b, t0, long).cooling_for, + MAX_SOURCE_COOLDOWN + ); } #[test] @@ -692,7 +819,9 @@ mod tests { // would end far sooner; the running cooldown stands, and what is left // of it is what the caller is told. assert_eq!( - tracker.record_busy(&a, t0 + Duration::from_secs(1), Duration::from_millis(1)), + tracker + .record_busy(&a, t0 + Duration::from_secs(1), Duration::from_millis(1)) + .cooling_for, Duration::from_secs(59) ); assert!(tracker.is_cooling(&a, t0 + Duration::from_secs(59))); @@ -709,43 +838,157 @@ mod tests { tracker.record_served(&a); assert!(!tracker.is_cooling(&a, t0)); assert!(tracker.is_empty()); - assert_eq!(tracker.record_busy(&a, t0, WAIT), WAIT); + assert_eq!(tracker.record_busy(&a, t0, WAIT).cooling_for, WAIT); } #[test] - fn a_stale_record_starts_over() { + fn a_new_busy_spell_starts_the_cooldown_over() { let tracker = SourceBusyTracker::new(); let a = peer(1); let t0 = Instant::now(); for _ in 0..3 { tracker.record_busy(&a, t0, WAIT); } - // Long after its last busy answer, the next one counts as the first. - let later = t0 + SOURCE_RECORD_TTL + MAX_SOURCE_COOLDOWN; - assert_eq!(tracker.record_busy(&a, later, WAIT), WAIT); + // Long after its last busy answer, the next one starts a new spell: + // its cooldown is the first one's again. + let later = t0 + BUSY_SPELL_GAP + MAX_SOURCE_COOLDOWN; + assert_eq!(tracker.record_busy(&a, later, WAIT).cooling_for, WAIT); + assert_eq!(tracker.unserved_busy(&a), 4); + } + + /// A holder that keeps the data but answers busy to every fetch, one + /// every busy wait for over an hour, as the sole holder of each key. The + /// record is per source, so these answers count together whichever keys + /// they answer. It is never withheld, but only its first + /// `FETCH_BUSY_NEUTRAL_ALLOWANCE` answers are free. + #[test] + fn a_holder_that_never_serves_is_charged_past_its_allowance() { + let tracker = SourceBusyTracker::new(); + let holder = peer(1); + let t0 = Instant::now(); + let neutral = usize::try_from(FETCH_BUSY_NEUTRAL_ALLOWANCE).unwrap(); + let charged: Vec = (0..1_000u32) + .map(|i| { + let now = t0 + WAIT * i; + assert_eq!(tracker.choose(&[holder], now), Some(&holder)); + tracker.record_busy(&holder, now, WAIT).charged + }) + .collect(); + assert!(charged.iter().take(neutral).all(|c| !c)); + assert!(charged.iter().skip(neutral).all(|c| *c)); } + /// Two answers at once, one for each request in the reply burst, spend + /// two of the allowance: it counts answers, not rounds. #[test] - fn the_source_table_stays_bounded_and_spares_cooling_sources() { + fn answers_that_arrive_together_each_spend_the_allowance() { + let tracker = SourceBusyTracker::new(); + let holder = peer(1); + let t0 = Instant::now(); + for _ in 0..FETCH_BUSY_NEUTRAL_ALLOWANCE { + assert!(!tracker.record_busy(&holder, t0, WAIT).charged); + } + assert!(tracker.record_busy(&holder, t0, WAIT).charged); + } + + /// Honest overload comes and goes: a holder that serves this requester + /// once between busy spells, each as long as the allowance, is never + /// charged however many spells there are. + #[test] + fn a_holder_that_serves_between_busy_spells_is_never_charged() { + let tracker = SourceBusyTracker::new(); + let holder = peer(1); + let mut now = Instant::now(); + for _ in 0..100 { + for _ in 0..FETCH_BUSY_NEUTRAL_ALLOWANCE { + assert!(!tracker.record_busy(&holder, now, WAIT).charged); + now += WAIT; + } + tracker.record_served(&holder); + } + } + + /// Serving once restores the whole allowance, and another source serving + /// does not. + #[test] + fn serving_once_restores_the_allowance() { + let tracker = SourceBusyTracker::new(); + let (holder, other) = (peer(1), peer(2)); + let t0 = Instant::now(); + for _ in 0..FETCH_BUSY_NEUTRAL_ALLOWANCE { + tracker.record_busy(&holder, t0, WAIT); + } + // Another holder serving changes nothing for this one. + tracker.record_served(&other); + assert!(tracker.record_busy(&holder, t0, WAIT).charged); + tracker.record_served(&holder); + for _ in 0..FETCH_BUSY_NEUTRAL_ALLOWANCE { + assert!(!tracker.record_busy(&holder, t0, WAIT).charged); + } + assert!(tracker.record_busy(&holder, t0, WAIT).charged); + } + + /// Time does not restore the allowance: a holder asked only rarely, long + /// after each busy spell has ended, still uses it up across spells and is + /// charged until it serves. + #[test] + fn time_alone_never_restores_the_allowance() { + let tracker = SourceBusyTracker::new(); + let holder = peer(1); + let t0 = Instant::now(); + let apart = BUSY_SPELL_GAP * 6; + let charged: Vec = (0..=FETCH_BUSY_NEUTRAL_ALLOWANCE + 2) + .map(|i| { + let answer = tracker.record_busy(&holder, t0 + apart * i, WAIT); + // Every answer starts a new spell, so each cools only the wait. + assert_eq!(answer.cooling_for, WAIT); + answer.charged + }) + .collect(); + let neutral = usize::try_from(FETCH_BUSY_NEUTRAL_ALLOWANCE).unwrap(); + assert!(charged.iter().take(neutral).all(|c| !c)); + assert!(charged.iter().skip(neutral).all(|c| *c)); + } + + /// A full table forgets nothing, so many holders taking turns to answer + /// busy cannot push one another's records out to start again. A newcomer + /// is charged instead, and a tracked holder keeps counting. + #[test] + fn a_full_table_forgets_nothing_and_charges_newcomers() { let tracker = SourceBusyTracker::new(); let t0 = Instant::now(); - // One source whose cooldown has ended, the rest still cooling. - tracker.record_busy(&peer(0), t0, Duration::from_millis(1)); - let t1 = t0 + Duration::from_secs(1); let max = u32::try_from(MAX_TRACKED_SOURCES).unwrap(); - for i in 1..max { - tracker.record_busy(&peer(i), t1, WAIT); + for i in 0..max { + assert!(!tracker.record_busy(&peer(i), t0, WAIT).charged); } assert_eq!(tracker.len(), MAX_TRACKED_SOURCES); - let newcomer = peer(u32::MAX); - tracker.record_busy(&newcomer, t1, WAIT); + // One more holder than the table holds, all taking turns, long apart: + // every later round is charged for the newcomer, and nobody's count + // starts again. + let newcomer = peer(max); + let mut now = t0; + for _ in 0..=FETCH_BUSY_NEUTRAL_ALLOWANCE { + now += BUSY_SPELL_GAP * 2; + assert_eq!( + tracker.record_busy(&newcomer, now, WAIT), + BusyAnswer { + cooling_for: Duration::ZERO, + charged: true, + } + ); + for i in 0..max { + tracker.record_busy(&peer(i), now, WAIT); + } + } assert_eq!(tracker.len(), MAX_TRACKED_SOURCES); - assert!(tracker.is_cooling(&newcomer, t1)); - // The one that had stopped cooling made room; every cooling one stayed. - assert!((1..max).all(|i| tracker.is_cooling(&peer(i), t1))); - // Long after, expired records make room before anything is displaced. - let later = t1 + SOURCE_RECORD_TTL + MAX_SOURCE_COOLDOWN; - tracker.record_busy(&peer(u32::MAX - 1), later, WAIT); - assert_eq!(tracker.len(), 1); + assert!(!tracker.is_cooling(&newcomer, now)); + assert_eq!(tracker.unserved_busy(&newcomer), 0); + assert!( + (0..max).all(|i| tracker.unserved_busy(&peer(i)) == FETCH_BUSY_NEUTRAL_ALLOWANCE + 2) + ); + assert!(tracker.record_busy(&peer(0), now, WAIT).charged); + // Serving makes room again. + tracker.record_served(&peer(0)); + assert!(!tracker.record_busy(&newcomer, now, WAIT).charged); } } diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 3d694c2f..5cde604a 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -92,8 +92,8 @@ use crate::replication::config::{ SUBTREE_ROUND1_WORK_REFILL_BYTES_PER_SEC, SUBTREE_SESSION_TTL, }; use crate::replication::fetch_busy::{ - busy_wait, BusyReplyDenied, BusyReplyGrant, FetchBusyReplyLimiter, SourceBusyTracker, - FETCH_BUSY_REPLY_SEND_TIMEOUT, FETCH_BUSY_WAIT_SPREAD, + busy_wait, busy_wake, BusyReplyDenied, BusyReplyGrant, FetchBusyReplyLimiter, + SourceBusyTracker, FETCH_BUSY_REPLY_SEND_TIMEOUT, FETCH_BUSY_WAIT_SPREAD, }; use crate::replication::paid_list::PaidList; use crate::replication::protocol::{ @@ -2238,6 +2238,14 @@ impl ReplicationEngine { self.fetch_busy_sources.last_busy(source) } + /// Test-only: busy answers `source` has given this engine's fetches since + /// it last served one, which spend its neutral allowance. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn fetch_source_unserved_busy_for_test(&self, source: &PeerId) -> u32 { + self.fetch_busy_sources.unserved_busy(source) + } + /// Test-only: fetch requests this engine has admitted and not yet finished /// serving. #[cfg(any(test, feature = "test-utils"))] @@ -4031,7 +4039,11 @@ impl ReplicationEngine { // not outlive the engine. let handle = tracker.spawn(async move { // Cancel-aware: abort when the engine shuts down. + // Shutdown is checked first, so a fetch that is + // ready to charge a busy source at the same + // moment is cancelled uncharged instead. tokio::select! { + biased; () = token.cancelled() => FetchOutcome { key: fetch_key, result: FetchResult::SourceFailed, @@ -4094,6 +4106,7 @@ impl ReplicationEngine { // initial fetch spawn above. let handle = tracker.spawn(async move { tokio::select! { + biased; () = token.cancelled() => FetchOutcome { key: fetch_key, result: FetchResult::SourceFailed, @@ -9638,11 +9651,11 @@ async fn execute_single_fetch( key: resp_key, }) if resp_key == key => { // The source refused at admission and said so, where it - // used to stay silent. That is an honest, prompt answer, so - // it carries no trust event. Waiting here, still holding - // this fetch slot, before the key moves on is what keeps a - // busy answer from becoming a fast retry loop: this slot asks - // at most once per wait, where silence made it once per fetch + // used to stay silent. A prompt answer to passing overload + // carries no trust event. Waiting here, still holding this + // fetch slot, before the key moves on is what keeps a busy + // answer from becoming a fast retry loop: this slot asks at + // most once per wait, where silence made it once per fetch // timeout. The attempt then ends exactly as that timeout // did, so nothing downstream changes. The worker races this // whole task against shutdown, so the wait does not delay it. @@ -9653,19 +9666,37 @@ async fn execute_single_fetch( // wait, even when this attempt's own deadline cuts the // sleep short. let full_wait = busy_wait(Duration::MAX, jitter); - let remaining = config - .fetch_request_timeout - .saturating_sub(sent_at.elapsed()); - let wait = full_wait.min(remaining); - let cooldown = busy_sources.record_busy(&source, Instant::now(), full_wait); + let answer = busy_sources.record_busy(&source, Instant::now(), full_wait); + // Past its neutral allowance, a source that keeps saying + // busy without serving is handled as the silence it + // replaces: the attempt holds its slot until its fetch + // timeout runs out, never longer, however late the answer + // came, and only then is the source charged the unit-weight + // failure saorsa-core records for a timeout. A fetch + // cancelled by shutdown before then is not charged. + let deadline = sent_at + .checked_add(config.fetch_request_timeout) + .unwrap_or(sent_at); + let now = Instant::now(); + let wake = busy_wake(answer, full_wait, now, deadline); + let charged = answer.charged; debug!( "Fetch: source {source} busy for {}; moving on in {} ms, \ - putting it last for {} ms", + putting it last for {} ms, charged: {charged}", hex::encode(key), - wait.as_millis(), - cooldown.as_millis() + wake.saturating_duration_since(now).as_millis(), + answer.cooling_for.as_millis() ); - tokio::time::sleep(wait).await; + tokio::time::sleep_until(wake.into()).await; + if charged { + audit_metrics::record_fetch_busy_charged(); + p2p_node + .report_trust_event( + &source, + TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), + ) + .await; + } FetchOutcome { key, result: FetchResult::SourceFailed, diff --git a/tests/e2e/fetch_busy_reply.rs b/tests/e2e/fetch_busy_reply.rs index 5419888f..83fc8d97 100644 --- a/tests/e2e/fetch_busy_reply.rs +++ b/tests/e2e/fetch_busy_reply.rs @@ -1,6 +1,7 @@ //! A fetch responder answers `Busy` to a refused request that asked for it, //! stays silent to one that did not, and a requester takes a busy answer -//! without charging the holder. +//! without charging the holder, until the holder has answered busy past its +//! neutral allowance without serving. //! //! The unit tests pin the wire format, the limiter, the wait and the queue //! transitions. What they cannot show is the whole path between two real @@ -17,8 +18,10 @@ use super::TestHarness; use ant_node::client::compute_address; -use ant_node::replication::config::REPLICATION_PROTOCOL_ID; -use ant_node::replication::fetch_busy::{FETCH_BUSY_WAIT_MAX, FETCH_BUSY_WAIT_MIN}; +use ant_node::replication::config::{FETCH_REQUEST_TIMEOUT, REPLICATION_PROTOCOL_ID}; +use ant_node::replication::fetch_busy::{ + FETCH_BUSY_NEUTRAL_ALLOWANCE, FETCH_BUSY_WAIT_MAX, FETCH_BUSY_WAIT_MIN, +}; use ant_node::replication::protocol::{ FetchRequest, FetchResponse, ReplicationMessage, ReplicationMessageBody, }; @@ -26,6 +29,7 @@ use futures::stream::{FuturesUnordered, StreamExt}; use saorsa_core::identity::PeerId; use saorsa_core::P2PNode; use serial_test::serial; +use std::sync::atomic::{AtomicU64, Ordering}; use std::time::{Duration, Instant}; const HOLDER_INDEX: usize = 3; @@ -41,6 +45,9 @@ const SILENCE_TIMEOUT: Duration = Duration::from_secs(3); const PROMPTLY: Duration = Duration::from_secs(10); const POLL: Duration = Duration::from_millis(50); const CHUNK_LEN: usize = 64 * 1024; +/// Just over the holder's per-peer busy reply interval of one a second, so +/// each of the requester's refused fetches is answered busy, not silently. +const BUSY_REPLY_SPACING: Duration = Duration::from_millis(1_100); #[derive(Debug, Clone, PartialEq, Eq)] enum Answer { @@ -86,7 +93,6 @@ async fn fetch( } fn rand_request_id() -> u64 { - use std::sync::atomic::{AtomicU64, Ordering}; static NEXT: AtomicU64 = AtomicU64::new(9_000); NEXT.fetch_add(1, Ordering::Relaxed) } @@ -97,28 +103,40 @@ struct Setup { holder: PeerId, } -async fn setup() -> Setup { +/// A network whose holder has its fetch serve gate closed, and the holder. +async fn gated_holder() -> (TestHarness, PeerId) { let harness = TestHarness::setup_minimal().await.expect("setup"); harness.warmup_dht().await.expect("warmup"); let holder_node = harness.test_node(HOLDER_INDEX).expect("holder"); + let holder = *holder_node.p2p_node.as_ref().expect("holder p2p").peer_id(); + holder_node + .replication_engine + .as_ref() + .expect("holder engine") + .close_fetch_serve_gate_for_test(); + (harness, holder) +} + +/// Put a chunk made from `seed` straight into the holder's store. +async fn put_on_holder(harness: &TestHarness, seed: usize) -> [u8; 32] { let content: Vec = (0..CHUNK_LEN) - .map(|i| u8::try_from(i % 251).unwrap()) + .map(|i| u8::try_from((i + seed * 7) % 251).unwrap()) .collect(); let key = compute_address(&content); - holder_node - .ant_protocol - .as_ref() + harness + .test_node(HOLDER_INDEX) + .and_then(|n| n.ant_protocol.as_ref()) .expect("holder protocol") .storage() .put(&key, &content) .await .expect("put"); - let holder = *holder_node.p2p_node.as_ref().expect("holder p2p").peer_id(); - holder_node - .replication_engine - .as_ref() - .expect("holder engine") - .close_fetch_serve_gate_for_test(); + key +} + +async fn setup() -> Setup { + let (harness, holder) = gated_holder().await; + let key = put_on_holder(&harness, 0).await; Setup { harness, key, @@ -264,3 +282,162 @@ async fn the_requester_takes_a_busy_answer_without_charging_the_holder() { harness.teardown().await.expect("teardown"); } + +/// A holder that keeps answering busy, across many keys and without serving, +/// costs it nothing at the requester's fetch worker for +/// `FETCH_BUSY_NEUTRAL_ALLOWANCE` answers in a row. The next is charged, as its +/// silence would have been, once that attempt's fetch timeout has run out. +/// Serving the requester once then clears what the holder owes. +/// +/// The holder stores nothing until the end. A chunk put straight into its +/// store has no payment behind it, so once replication starts, about a minute +/// in, the holder's replica hint for it is rejected and charged, and this test +/// runs longer than that. +#[tokio::test] +#[serial] +async fn a_holder_that_keeps_answering_busy_is_charged_past_its_allowance() { + let (harness, holder) = gated_holder().await; + let requester_node = harness.test_node(REQUESTER_INDEX).expect("requester"); + let requester = requester_node.p2p_node.as_ref().expect("requester p2p"); + let engine = requester_node + .replication_engine + .as_ref() + .expect("requester engine"); + let trust_before = requester.peer_trust(&holder); + + // Fill this requester's share at the holder, so the holder refuses every + // fetch the worker sends it. Each attempt below runs to its end before the + // next starts, so the held requests must outlast all of them: a timeout + // would charge the holder. They are never answered. The gate stays closed + // for longer than the holder's fetch timeout, so once it opens the holder + // sheds them as stale. + let held_key = compute_address(b"held"); + let held_timeout = (BUSY_REPLY_SPACING + FETCH_BUSY_WAIT_MAX + PROMPTLY) + * (FETCH_BUSY_NEUTRAL_ALLOWANCE + 1) + + FETCH_REQUEST_TIMEOUT; + let mut held: FuturesUnordered<_> = (0..=SHARE) + .map(|_| fetch(requester, &holder, held_key, true, held_timeout)) + .collect(); + let first = tokio::time::timeout(PROMPTLY, held.next()) + .await + .expect("the refused request is answered promptly") + .expect("a request"); + assert_eq!(first, Answer::Busy); + + // Busy answers from the worker, each for a new key, until the holder has + // used up its allowance: the allowance belongs to the holder, not to a key. + // Each attempt runs to its end before the next key is queued, so any + // charge would come from the attempt just made. + let mut next_key = 0u32; + let mut lowest_trust = trust_before; + loop { + let owed = engine.fetch_source_unserved_busy_for_test(&holder); + if owed >= FETCH_BUSY_NEUTRAL_ALLOWANCE { + break; + } + tokio::time::sleep(BUSY_REPLY_SPACING).await; + let other_key = compute_address(&next_key.to_le_bytes()); + next_key += 1; + assert!(engine.enqueue_fetch_for_test(other_key, vec![holder]).await); + let deadline = Instant::now() + FETCH_REQUEST_TIMEOUT + PROMPTLY; + while engine.fetch_queued_or_in_flight_for_test(&other_key).await { + lowest_trust = lowest_trust.min(requester.peer_trust(&holder)); + assert!( + Instant::now() < deadline, + "the attempt after {owed} busy answers never ended" + ); + tokio::time::sleep(POLL).await; + } + lowest_trust = lowest_trust.min(requester.peer_trust(&holder)); + assert!( + engine.fetch_source_unserved_busy_for_test(&holder) > owed, + "the attempt after {owed} busy answers was not answered busy" + ); + assert!( + lowest_trust >= trust_before, + "a busy answer within the allowance cost the holder trust: \ + {trust_before} fell to {lowest_trust}" + ); + } + assert_eq!( + engine.fetch_source_unserved_busy_for_test(&holder), + FETCH_BUSY_NEUTRAL_ALLOWANCE + ); + + // The next answer is past the allowance. Its attempt holds its slot until + // its fetch timeout runs out, costing nothing meanwhile, and the holder is + // charged only as it ends. + tokio::time::sleep(BUSY_REPLY_SPACING).await; + let last_key = compute_address(&next_key.to_le_bytes()); + assert!(engine.enqueue_fetch_for_test(last_key, vec![holder]).await); + let deadline = Instant::now() + PROMPTLY; + while engine.fetch_source_unserved_busy_for_test(&holder) <= FETCH_BUSY_NEUTRAL_ALLOWANCE { + assert!( + Instant::now() < deadline, + "the answer past the allowance never came" + ); + tokio::time::sleep(POLL).await; + } + let answered_at = Instant::now(); + // The answer was seen within `PROMPTLY` of the request, so the attempt + // holds its slot, uncharged, for at least the rest: still well past any + // busy wait. + let not_before = FETCH_REQUEST_TIMEOUT.saturating_sub(PROMPTLY); + let deadline = answered_at + FETCH_REQUEST_TIMEOUT + PROMPTLY; + while engine.fetch_queued_or_in_flight_for_test(&last_key).await { + if answered_at.elapsed() < not_before { + lowest_trust = lowest_trust.min(requester.peer_trust(&holder)); + } + assert!( + Instant::now() < deadline, + "the attempt past the allowance never ended" + ); + tokio::time::sleep(POLL).await; + } + let attempt_lasted = answered_at.elapsed(); + assert!( + attempt_lasted >= not_before, + "the attempt past the allowance ended {attempt_lasted:?} after its answer, \ + before its fetch timeout" + ); + assert!( + lowest_trust >= trust_before, + "the answer past the allowance was charged before its attempt ended: \ + {trust_before} fell to {lowest_trust}" + ); + let trust_after = requester.peer_trust(&holder); + assert!( + trust_after < trust_before, + "the answer past the allowance was not charged: trust stayed {trust_after}" + ); + drop(held); + + // Serving once clears what the holder owes. The chunk is new, so the + // requester cannot already hold it, and it is fetched only after the + // holder has shed the held requests and so has room for it. + let fresh_key = put_on_holder(&harness, 1).await; + open_gate(&harness); + let holder_engine = harness + .test_node(HOLDER_INDEX) + .and_then(|n| n.replication_engine.as_ref()) + .expect("holder engine"); + let deadline = Instant::now() + PROMPTLY; + while holder_engine.fetch_admitted_for_test() > 0 { + assert!( + Instant::now() < deadline, + "the holder never let go of the held requests" + ); + tokio::time::sleep(POLL).await; + } + assert!(engine.enqueue_fetch_for_test(fresh_key, vec![holder]).await); + let deadline = Instant::now() + PROMPTLY; + while engine.fetch_source_unserved_busy_for_test(&holder) > 0 { + assert!( + Instant::now() < deadline, + "serving the requester did not clear what the holder owes" + ); + tokio::time::sleep(POLL).await; + } + + harness.teardown().await.expect("teardown"); +} From 447c000dc7b422dee8f94400035b5821efae6880 Mon Sep 17 00:00:00 2001 From: grumbach Date: Tue, 13 Oct 2026 16:28:51 +0900 Subject: [PATCH 04/10] fix(replication): keep fetches within each answering holder's admission share A fetch responder admits two outstanding requests per requesting peer and drops later arrivals without replying while that share is full. The fetch worker ran one fetch per hardware thread and sent each one to the first verified holder of its key, and the nearest-first queue often groups keys that share holders. When more than two fetch slots were free and several queued keys shared a holder, the worker sent more than that holder's share. Each resulting drop cost the full 30 s fetch timeout, a connection-timeout trust event against a holder that did nothing wrong, and a retry that ignored how busy the next holder was. The fetch worker now keeps at most FETCH_MAX_IN_FLIGHT_PER_SOURCE (the responder's share) of its fetches in flight to any one source that is answering: - Each free slot goes to the nearest queued key that can start now, on the first untried source in its preference order that is answering and below the limit, or else on its first silent source. - A key whose untried sources are all answering and full is parked: it leaves the fetch heap, is indexed exactly under each of those sources, and starts again as soon as one frees a share or falls silent, competing with the heap on distance. - A failed fetch moves on by the same rule. If every remaining source is answering and full, it is parked instead of being sent to a full holder, keeping its tried sources and its verification retry reservation. With no untried source left, or a full fetch queue, it returns to verification as before. - The sources a queued or in-flight key is associated with, tried ones included, are capped at close_group_size + paid_list_close_group_size + MAX_HINT_SOURCES_PER_KEY from the running configuration. A re-advertisement merges into such a key's verification retry entry under MAX_HINT_SOURCES_PER_KEY, as into a pending entry, instead of without a bound. A holder that stops answering has no share to keep within, and holding it to two requests per fetch timeout would drain the keys only it holds four times more slowly than before. Each attempt now records whether its source answered. Requests and answers take places in one process-wide sequence, so they are ordered exactly whatever order their outcomes are applied in. An answer reaches the scheduler as soon as it arrives, before its chunk is checked and stored, so a timeout it outdates cannot mark its source while that chunk is still being written. A source is silent while a request sent to it after its latest answer has gone unanswered, and a silent source is fetched from past the limit, as before this change, until it answers again, or until no newer unanswered request to it has been seen for two fetch timeouts (at least one second), after which it is probed under the limit. The table is bounded at 8,192 sources and drops only entries that hold no mark and guard no in-flight fetch; if that frees nothing, untracked sources count as silent for one mark lifetime, which again is the behaviour before this change. A "replication fetch dispatch summary" line every 5 minutes and at shutdown reports parked keys, index size, oldest parked age, queued and in-flight fetches, sources at the limit, parks, attempts by outcome, stored bytes, attempt latency, cancelled attempts, fill-pass lock wait and hold, parked-index maintenance time, and the silent-source counts and events. The limit removes over-share bursts but cannot rule out every refusal: a request this node has already timed out on may still hold its share at the responder until a worker sheds it. Responder admission, the fetch timeout, fetch concurrency and the wire protocol are unchanged. ADR-0024 records the decision, a local risk simulation and a scale benchmark at the fetch queue's hard bound, which runs when ANT_FETCH_SCALE_BENCH names a sources-per-key count. Test-only hooks, compiled only under cfg(test) or the test-utils feature and scoped to one engine: the fetch concurrency, a gate that holds the engine's admitted fetch requests before serving them and that engine shutdown releases, and a fetch-pipeline snapshot. With concurrency eight and twelve keys directed to one holder, the requester records at most two in-flight fetches to it. With a 5 s fetch timeout and sixteen keys directed to a holder that answers nothing, the requester sends past the share once the first requests time out and drains the keys in three timeouts. --- ...uesters-stay-within-the-responder-share.md | 282 ++ docs/adr/README.md | 1 + src/replication/mod.rs | 884 +++++- src/replication/scheduling.rs | 2732 ++++++++++++++++- tests/e2e/fetch_per_source_share.rs | 328 ++ tests/e2e/mod.rs | 3 + tests/e2e/testnet.rs | 8 + 7 files changed, 4005 insertions(+), 233 deletions(-) create mode 100644 docs/adr/ADR-0024-fetch-requesters-stay-within-the-responder-share.md create mode 100644 tests/e2e/fetch_per_source_share.rs diff --git a/docs/adr/ADR-0024-fetch-requesters-stay-within-the-responder-share.md b/docs/adr/ADR-0024-fetch-requesters-stay-within-the-responder-share.md new file mode 100644 index 00000000..909a67f8 --- /dev/null +++ b/docs/adr/ADR-0024-fetch-requesters-stay-within-the-responder-share.md @@ -0,0 +1,282 @@ +# ADR-0024: Fetch requesters stay within the responder's per-peer share + +- **Status:** Proposed +- **Date:** 2026-10-13 +- **Decision owners:** +- **Reviewers:** +- **Supersedes:** none +- **Superseded by:** none +- **Related:** [ADR-0005](ADR-0005-replication-repair-hardening.md) (the repair fetch pipeline), + [ADR-0017](ADR-0017-bounded-fresh-offers-and-copy-free-sends.md) (send-path bounds) + +## Context + +A replication fetch is a request/response exchange between two nodes. The responder bounds what +one requester can occupy: + +- `FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER = 2` admitted requests per source, checked before the + 12-slot admission pool, in front of a 3-worker serving pool; +- a request past that share is not answered. The responder logs + `Fetch response dropped at admission: reason=per_peer_cap_full` and returns nothing. + +The requesting side never matched that share: + +- the fetch worker ran one fetch per hardware thread; +- it sent each one to the first verified holder of its key, in source-preference order (the + closest holder that answered Present); +- the fetch queue is nearest-first, so the keys at its head usually share the same few holders. + +When more than two fetch slots were available and several queued keys shared a holder, the worker +could send more than that holder's two-request share. A request arriving while two from the same +requester were still admitted was dropped. Each such drop cost the requester: + +- the full fetch timeout (30 s) before moving on; +- a `ConnectionTimeout` trust event, recorded by the transport against the holder that dropped it, + although that holder had done nothing wrong; +- a retry that went to the next holder without regard to how busy that holder was. + +On the production fleet this was common, according to the per-service cumulative counters: + +| period | inbound fetch requests dropped at admission | +|---|---| +| on 0.21.0 | 36.6% | +| on 0.22.0 | 44.3% | +| 25-hour window containing a wave of 177 freshly installed nodes filling their ranges | 65.8% overall by cumulative counters; 3.22M WARN-line drops came from the new nodes, about 2.8 per chunk attributed to them | + +In that window, 68.5% of drops happened with only 2 of the responder's 12 admission slots in use, +both held by the refused peer. + +## Decision Drivers + +- No wire, storage or production public API change, so the fix ships as a behavioural change on + the release train. +- Do not weaken the responder's protection against one peer occupying the whole serving pool. +- A key must never be lost, tracked twice, or released from bootstrap accounting early because of + scheduling. +- Keep the queue's nearest-first priority and each key's source preference whenever they do not + conflict with the share. +- No scenario may drain more slowly than before the change by more than one fetch timeout, in + particular the keys of a holder that has stopped answering. + +## Considered Options + +1. **Raise the responder's per-peer share.** Rejected. It hides the symptom, lets one requester take + all three serving workers, which is what the share exists to prevent, and still drops requesters + with more threads than the new share. +2. **Answer an over-share request with an explicit "busy" reply.** Deferred. It needs a new + response variant, which peers that cannot decode it treat as malformed and charge an application + failure, so it is a wire change with its own rollout. It removes the cost of a drop, not the + cause. +3. **Limit fetch concurrency to 2.** Rejected. It serialises a node's whole repair and bootstrap fill + through one share even when other holders are idle. +4. **Ship the scheduler switched off, behind a per-node switch, for a later canary.** Rejected. It + merges code that no node runs, so its risks are deferred rather than measured. The risks were + simulated instead (below), and the one regression found was fixed before merge. +5. **Keep at most the share in flight per answering holder on the requesting side, and give each + free slot to the nearest queued key that can start now.** Chosen. + +## Decision + +The fetch worker keeps at most `FETCH_MAX_IN_FLIGHT_PER_SOURCE` fetches in flight to any one source +peer that is answering. The constant is defined equal to `FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER`, +the share every deployed responder grants. It is the worker's only scheduling path. + +**Dispatch.** Each free fetch slot goes to the nearest queued key that can start now. In the key's +source-preference order, over its untried sources, it starts on: + +1. the first source that is answering and below the limit; +2. otherwise the first silent source (below), past the limit. + +**Parking.** + +- A key whose untried sources are all answering and at the limit is parked: it leaves the fetch + heap and is indexed under each of those sources. +- It becomes a candidate again as soon as any of them has a free share or falls silent, and it + competes with the heap on distance. +- Each key leaves the heap once, so a startable key further down is reached in the same pass, and + no source ordering can hide a startable key. + +**Retries.** + +- A failed fetch moves to its next untried source by the same rule. If every untried source is + answering and at the limit, it is parked rather than sent to a full holder. +- It keeps the sources it already tried, so none is asked twice in one verification cycle, and it + keeps its verification retry reservation. +- When no untried source is left, or the fetch queue is full, it returns to verification as before, + so the fetch queue's hard bound holds. + +**Silent sources.** The limit protects a holder's admission share, and only a holder that answers +has one. Without an exception, the keys held only by a holder that stopped answering would be tried +two per fetch timeout instead of one per free slot: four times more slowly than before on a node +with eight slots. + +- Each attempt records whether its source answered. Requests and answers take places in one + process-wide sequence, as a request is sent and as an answer returns, so they can be ordered + exactly, whatever order their outcomes are applied in and whatever the clock reads. +- An answer reaches the scheduler as soon as it arrives, before its chunk is checked and stored, + and again with the attempt's outcome, to no further effect. A timeout that an answer already + outdates therefore never marks the source while that answer's chunk is still being written. +- A source is **silent** while a request sent to it after its latest answer has gone without any + response (a timeout or connection failure). Any answer received after that request was sent ends + it. +- A silent source is fetched from past the limit, as before the change, until it answers again, + or until this node has learned of no newer unanswered request to it for two fetch timeouts (at + least one second). A holder that stays dead while this node keeps fetching from it therefore + stays silent, and one this node stopped fetching from is probed under the limit when it is next + used, marked again only if a request sent then also goes unanswered. +- The table of observations is bounded at 8,192 sources. It drops only entries that hold no mark and + whose source has no fetch in flight. If that frees nothing, a source not in the table counts as + silent for one mark lifetime, which is again the behaviour before the change, never a slower one. + +**Source limit.** + +- The sources a queued or in-flight key is associated with, tried ones included, are bounded by + `close_group_size + paid_list_close_group_size + MAX_HINT_SOURCES_PER_KEY` from the running + configuration. +- That never truncates a verification round's own sources, and it bounds the parked index. +- A re-advertisement of a queued or in-flight key merges into the verification entry kept for its + retry under `MAX_HINT_SOURCES_PER_KEY`, the bound a pending entry has, in the same order: replica + claimants first, displacing paid-only advertisers other than the key's capacity owner, then + paid-only advertisers while there is room. Before the change that merge had no bound. + +**Dispatch summary.** The fetch worker logs `replication fetch dispatch summary` every 5 minutes and +at shutdown, with: + +- the parked count, the index size and the oldest parked age; +- queued and in-flight fetches, and sources at the limit; +- parks and retry parks; +- attempts by outcome, bytes stored, and attempt latency; +- the longest fill-pass lock wait and hold, and the longest parked-index maintenance after a + departure; +- silent sources now and at peak, marks, clears, expiries, ignored stale timeouts, starts on silent + sources, unreachable attempts, tracked sources and table overflows. + +Responder admission, the fetch timeout, fetch concurrency and the wire protocol are unchanged. + +## Risk simulation + +A local 12-node network, eight fetch slots per node and a 5 s fetch timeout, timed each scenario on +this change and on a build with the limit removed and no silent-source handling, which schedules +exactly as before the change: each key to its first untried source. + +| scenario | this change | before the change | +|---|---|---| +| fill: 48 keys held by 3 answering holders | 0.36 s | 10.08 s | +| a non-answering holder first among 3 sources, 2 answering: 24 keys | 5.1 s | 20.1 s | +| a non-answering sole holder: 16 keys | 15.0 s | 10.15 s | + +The third row is the accepted penalty: the first two requests to a holder that just went silent take +one fetch timeout to show it, and the rest then go out past the limit. It is paid once each time +this node starts fetching from a silent holder again after two fetch timeouts without trying it. +For `N` keys held only by that holder, `C ≥ 2` fetch slots and fetch timeout `T`, draining takes +`T × (1 + ceil((N − 2) / C))` against `T × ceil(N / C)` before the change. With one fetch slot the +first wave is one request, and the bound is the same with 1 in place of 2. Without the +silent-source exception the same scenario took 40.1 s. + +The scale benchmark at the fetch queue's hard bound (`scale_benchmark_at_the_fetch_queue_bound`, +release build, 131,072 parked keys) stayed within the thresholds declared before it ran, apart from +dropping the queues (below): + +| sources per key | one park pass, over repeated runs | peak process RSS | +|---|---|---| +| 1 | 89 to 99 ms | 197 MiB | +| 8 | 207 to 216 ms | 342 MiB | +| 35 (the limit) | 754 to 960 ms | 1.13 GiB | + +- Waking and the stats read took under 1 ms; a departure took at most 107 ms. +- Dropping the queues at the limit took 274 to 315 ms over repeated runs. The scheduler before the + silent-source handling took 226 to 295 ms, and the same time as this code in alternating runs on + one machine. It was first held to 250 ms, which a loaded machine did not meet. It happens once, + when the engine shuts down, outside any lock, so it is now held to 1 s. +- The silent-source check makes the park pass about 5% slower than before. + +## Consequences + +### Positive + +- A requester on this release keeps its own in-flight fetches to one answering holder within that + holder's share. That removes the ordinary over-share bursts and their timeout and trust cost; it + does not cover stale work the responder still holds, described below. +- A fill or repair can finish sooner, because fetch slots no longer spend 30 s on over-share + requests that will not be answered. +- Responders should log fewer `per_peer_cap_full` drops as requesters upgrade. +- The dispatch summary shows the scheduler's queue state, lock times and silent sources in + production. + +### Negative / Trade-offs + +- It is not a guarantee of zero refusals. A request this node has already timed out on can still + hold its share at the responder until a worker sheds it, and a new request sent in that window + is refused. Closing that window needs option 2. During the 25-hour wave, responders shed 31,172 + admitted requests after waiting 30 s for a worker; the production data do not show how often + that window caused a later refusal. +- The requester counts a fetch as in flight for its whole attempt, local checks and the local write + included, which is longer than the responder holds its share. A holder can therefore sit below + its share while this node waits, and a fetch whose local work stalls holds its holder's share for + as long as it holds its fetch slot. +- A key whose answering holders are all busy waits in the queue, or goes to a lower-ranked holder + with room, instead of being sent to its first holder. +- Keys held only by a holder that just stopped answering take one fetch timeout longer to drain + than before the change, as in the simulation. A holder that is alive but so overloaded that it + times out is fetched from as before the change while its mark lasts. +- Recovering the slots dropped requests used to waste raises a filling node's instantaneous + bandwidth, disk and CPU use. Each responder still serves at most its share per requester with + three workers; the data a joining node needs to move is unchanged. +- At the source limit, the parked index can reach 4.6 million entries and about 1.1 GiB of process + memory; at 8 sources per key, about 340 MiB. +- Requesters on older releases still overshoot until they upgrade, so the network-wide drop rate + falls only as the release spreads, and the traffic pattern changes during that time. +- There is no runtime switch. Undoing the change means a release without it. + +### Neutral / Operational + +- The effect shows in the responders' cumulative counters (`replication responder admission + summary`): `fetch_admission_dropped_per_peer_share` relative to `fetch_request_rx_count` should + fall as the fleet and the community upgrade. +- Before release, a dev testnet run with a join wave of fresh nodes should compare drain time, + stored bytes, admission drops, trust events and host load against the previous release. + +## Validation + +**Unit tests on the queue:** + +- no answering source exceeds the share; +- preference order is kept; +- a startable key behind hundreds of blocked keys starts in the same pass; +- a freed share starts the nearest parked key, whatever its source; +- parked and retried keys keep their payload, tried sources and retry reservation; +- departures and merges keep the parked index exact; +- the source limit holds across park, retry and merge; +- a silent source is fetched from past the share, an answering source with room is preferred, and + parked keys start when a full source falls silent; +- whether a source is silent does not depend on the order its outcomes are applied in; marks are + set, cleared and expired as described, and the table bound keeps every guard and mark; +- an answer reported when it arrives keeps a source within its share when an earlier request to it + times out while the answered fetch is still storing its chunk; known only from the outcome, the + same answer comes too late and the source is sent every waiting key; +- re-advertisements keep a queued or in-flight key's retry entry within the hint bound, with its + capacity owner; +- sixteen keys held only by a silent holder, with eight slots, start in exactly three waves of 2, 8 + and 6; +- a randomised run of queue operations, re-advertisements and source contacts keeps every + invariant, the retry entries' bound included. + +**End-to-end tests:** + +- with eight fetch slots and twelve keys directed to one holder that holds what it admits, the + requester's own pipeline records at most two in-flight fetches to that holder; with the limit + removed it records eight and the test fails; +- with a 5 s fetch timeout and sixteen keys directed to a holder that answers nothing, the requester + sends past the share once the first requests time out and the keys leave its pipeline within four + timeouts (15.1 s measured); with the silent-source exception removed the test fails. + +**Revisit if:** + +- the responder's per-peer share changes; +- a busy reply is introduced; +- the fleet counters keep showing a high dropped share once most requesters run this release; +- the dispatch summary shows sustained table overflows or a large count of silent sources. + +## Notes for AI-assisted work + +AI tools may help draft this ADR, but **must not mark it Accepted without human review**. Accepted ADRs are immutable: create a new superseding ADR rather than editing an Accepted ADR. diff --git a/docs/adr/README.md b/docs/adr/README.md index b58e9a32..bcde1f6f 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -40,3 +40,4 @@ See [`TOOLING.md`](./TOOLING.md) for `adrs`, `adr-kit`, and AI harness setup. - [ADR-0018: Pointer ownership transfer by final redirection](./ADR-0018-pointer-transfer-by-final-redirection.md) - [ADR-0019: Pointer audits serve the record round 1 bound](./ADR-0019-pointer-audits-serve-the-record-round-one-bound.md) - [ADR-0022: Remove the LMDB Chunk Store](./ADR-0022-remove-the-lmdb-chunk-store.md) +- [ADR-0024: Fetch requesters stay within the responder's per-peer share](./ADR-0024-fetch-requesters-stay-within-the-responder-share.md) diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 62937413..2c50bd27 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -42,6 +42,8 @@ use std::fmt; use std::num::NonZeroUsize; use std::ops::ControlFlow; use std::path::{Path, PathBuf}; +#[cfg(any(test, feature = "test-utils"))] +use std::sync::atomic::AtomicBool; use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::Arc; use std::time::{Duration, Instant, SystemTime}; @@ -62,6 +64,7 @@ use tokio_util::sync::CancellationToken; use tokio_util::task::TaskTracker; use crate::ant_protocol::XorName; +use crate::client::xor_distance; use crate::error::{Error, Result}; use crate::payment::{ PaymentVerifier, VerificationContext, MAX_PAYMENT_PROOF_SIZE_BYTES, @@ -95,7 +98,12 @@ use crate::replication::protocol::{ }; use crate::replication::quorum::KeyVerificationOutcome; use crate::replication::recent_provers::RecentProvers; -use crate::replication::scheduling::{CapacityDisplacement, DeferralOutcome, ReplicationQueues}; +#[cfg(any(test, feature = "test-utils"))] +pub use crate::replication::scheduling::FetchPipelineSnapshot; +use crate::replication::scheduling::{ + fetch_source_limit, CapacityDisplacement, Contact, DeferralOutcome, FetchDispatchStats, + FetchRetry, ReplicationQueues, +}; use crate::replication::storage_commitment_audit::PointerBindings; use crate::replication::types::{ AuditFailureReason, BootstrapClaimObservation, BootstrapState, FailureEvidence, @@ -116,6 +124,16 @@ use saorsa_pqc::api::sig::{MlDsaSecretKey, MlDsaVariant}; /// counted: it is not a saturation signal. static FIRST_AUDIT_INGRESS_DROPPED: AtomicU64 = AtomicU64::new(0); +/// One sequence over every fetch request this process sends and every answer +/// it receives, so the fetch scheduler can order a request against an answer +/// even when both read the same clock value. See [`Contact`]. +static FETCH_CONTACT_SEQUENCE: AtomicU64 = AtomicU64::new(0); + +/// The next place in [`FETCH_CONTACT_SEQUENCE`]. +fn next_fetch_contact() -> u64 { + FETCH_CONTACT_SEQUENCE.fetch_add(1, Ordering::Relaxed) +} + /// Record one ingress-full drop. Called by the payment verifier's `try_send` /// sites; read by the drainer's periodic summary. pub(crate) fn note_monetized_ingress_drop() { @@ -1129,6 +1147,24 @@ const FETCH_RESPONDER_MAX_OUTSTANDING: usize = FETCH_RESPONDER_WORKER_LIMIT * 4; /// of the three workers available to other peers under a single-source flood. const FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER: u32 = 2; +/// Maximum fetches this node keeps in flight to any one source peer that is +/// answering. +/// +/// The share every responder grants one requester +/// ([`FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER`], unchanged since it was +/// introduced). A request past that share is dropped without an answer, so +/// sending it costs the full fetch timeout and a timeout trust event against a +/// holder that did nothing wrong. Staying within it removes those bursts; it +/// cannot rule out every refusal, because a request this node has already +/// timed out on may still be holding its share at the responder until a worker +/// sheds it. +/// +/// A source whose request got no answer, and which has not answered since that +/// request was sent, has no share to stay within. It is fetched from without +/// this cap, as before the cap existed, until it answers again or two fetch +/// timeouts pass, so a holder that went silent does not hold its keys back. +const FETCH_MAX_IN_FLIGHT_PER_SOURCE: usize = FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER as usize; + /// Maximum verification batches served concurrently. /// /// Point lookups are fast, but a batch can contain 8,192 of them. Two @@ -1222,7 +1258,9 @@ fn paid_notify_payment_context() -> VerificationContext { } /// Boxed future type for in-flight fetch tasks. -type FetchFuture = Pin)> + Send>>; +/// A fetch task's key, its outcome (`None` when the task panicked), and when +/// the task finished. +type FetchFuture = Pin, Instant)> + Send>>; /// Fresh-offer keys currently being handled, each with the proofs still to try. /// @@ -1667,6 +1705,8 @@ pub struct ReplicationEngine { payment_verifier: Arc, /// Replication pipeline queues. queues: Arc>, + /// Fetches the worker runs at once: one per hardware thread. + fetch_concurrency: usize, /// Neighbor sync cycle state. sync_state: Arc>, /// Per-peer sync history (for `RepairOpportunity`). @@ -1787,6 +1827,9 @@ pub struct ReplicationEngine { fetch_responder_admission_semaphore: Arc, /// Per-source fetch responder counts for flood-fair admission. fetch_responder_inflight: Arc>>, + /// Test-only hold on serving admitted fetch requests. + #[cfg(any(test, feature = "test-utils"))] + fetch_serve_gate: Arc, /// Worker permits for lookup-heavy verification batches. verification_responder_worker_semaphore: Arc, /// Admission permits bounding verification workers and queued waiters. @@ -1902,13 +1945,21 @@ impl ReplicationEngine { let (monetized_pin_tx, monetized_pin_rx) = mpsc::channel(config::FIRST_AUDIT_INGRESS_CAPACITY); + let source_limit = + fetch_source_limit(config.close_group_size, config.paid_list_close_group_size); + let engine = Self { config: Arc::clone(&config), p2p_node, storage, paid_list, payment_verifier, - queues: Arc::new(RwLock::new(ReplicationQueues::new())), + queues: Arc::new(RwLock::new( + ReplicationQueues::new() + .with_fetch_source_limit(source_limit) + .with_fetch_request_timeout(config.fetch_request_timeout), + )), + fetch_concurrency: max_parallel_fetch(), sync_state: Arc::new(RwLock::new(initial_neighbors)), sync_history: Arc::new(RwLock::new(HashMap::new())), audit_on_gossip_cooldown: Arc::new(RwLock::new(HashMap::new())), @@ -1940,6 +1991,8 @@ impl ReplicationEngine { FETCH_RESPONDER_MAX_OUTSTANDING, )), fetch_responder_inflight: Arc::new(RwLock::new(HashMap::new())), + #[cfg(any(test, feature = "test-utils"))] + fetch_serve_gate: Arc::new(FetchServeGate::default()), verification_responder_worker_semaphore: Arc::new(Semaphore::new( VERIFICATION_RESPONDER_WORKER_LIMIT, )), @@ -2194,6 +2247,35 @@ impl ReplicationEngine { self.queues.read().await.fetch_queued_or_in_flight(key) } + /// Test-only: run `concurrency` fetches at once instead of one per hardware + /// thread. Call before [`Self::start`], which reads it. Per engine, so it + /// affects no other node or test. + #[cfg(any(test, feature = "test-utils"))] + pub fn set_fetch_concurrency_for_test(&mut self, concurrency: usize) { + self.fetch_concurrency = concurrency; + } + + /// Test-only: hold this engine's admitted fetch requests before they are + /// served, until [`Self::open_fetch_serve_gate_for_test`]. + #[cfg(any(test, feature = "test-utils"))] + pub fn close_fetch_serve_gate_for_test(&self) { + self.fetch_serve_gate.close(); + } + + /// Test-only: release this engine's held fetch requests and stop holding + /// new ones. + #[cfg(any(test, feature = "test-utils"))] + pub fn open_fetch_serve_gate_for_test(&self) { + self.fetch_serve_gate.open(); + } + + /// Test-only: every key queued for fetch and every in-flight fetch with + /// the source it was sent to, read under one lock. + #[cfg(any(test, feature = "test-utils"))] + pub async fn fetch_pipeline_snapshot_for_test(&self) -> FetchPipelineSnapshot { + self.queues.read().await.fetch_pipeline_snapshot() + } + /// Test-only: place `key` into pending verification as though `hinter` had /// just advertised it with a replica hint. Returns whether it was admitted. /// @@ -2434,6 +2516,12 @@ impl ReplicationEngine { } } + // A test may shut down with the fetch serve gate still closed. The + // requests it holds wait on the gate before they reach a worker pool, + // so release them, or the drain below never finishes. + #[cfg(any(test, feature = "test-utils"))] + self.fetch_serve_gate.open(); + // Close the responder worker pools. Detached tasks still queued for a // worker take the `Err` arm of `acquire_owned()` and exit immediately // rather than waiting for a slot that will never free up — without @@ -3063,6 +3151,8 @@ impl ReplicationEngine { let fetch_responder_admission_semaphore = Arc::clone(&self.fetch_responder_admission_semaphore); let fetch_responder_inflight = Arc::clone(&self.fetch_responder_inflight); + #[cfg(any(test, feature = "test-utils"))] + let fetch_serve_gate = Arc::clone(&self.fetch_serve_gate); let verification_responder_worker_semaphore = Arc::clone(&self.verification_responder_worker_semaphore); let verification_responder_admission_semaphore = @@ -3119,6 +3209,8 @@ impl ReplicationEngine { fetch_responder_worker_semaphore, fetch_responder_admission_semaphore, fetch_responder_inflight, + #[cfg(any(test, feature = "test-utils"))] + fetch_serve_gate, verification_responder_worker_semaphore, verification_responder_admission_semaphore, verification_responder_inflight, @@ -3881,72 +3973,91 @@ impl ReplicationEngine { let is_bootstrapping = Arc::clone(&self.is_bootstrapping); let bootstrap_complete_notify = Arc::clone(&self.bootstrap_complete_notify); let detached_task_tracker = self.detached_task_tracker.clone(); - let concurrency = max_parallel_fetch(); + let concurrency = self.fetch_concurrency; info!("Fetch worker concurrency set to {concurrency} (hardware threads)"); let handle = tokio::spawn(async move { - // Each in-flight future yields (key, Option) so we - // always recover the key — even if the inner task panics. + // Each in-flight future yields (key, Option, when it + // finished), so we always recover the key, even if the inner task + // panics. let mut in_flight = FuturesUnordered::::new(); + let mut dispatch_window = FetchDispatchWindow::new(); + // Each answer as it arrives, ahead of its fetch's outcome. + let (answers, mut early_answers) = mpsc::unbounded_channel(); loop { - // Fill up to `concurrency` slots from the queue. - { - let mut q = queues.write().await; - while in_flight.len() < concurrency { - let Some(candidate) = q.dequeue_fetch() else { - break; - }; - let fetch_key = candidate.key; - let Some(&source) = candidate.sources.first() else { - warn!( - "Fetch candidate {} has no sources; requeueing for verification", - hex::encode(fetch_key) - ); - let _ = q.requeue_candidate_for_verification( - candidate, - config.verification_request_timeout, - ); - continue; - }; - q.start_dequeued_fetch(candidate, source); - - let p2p = Arc::clone(&p2p); - let storage = Arc::clone(&storage); - let config = Arc::clone(&config); - let token = shutdown.clone(); - let tracker = detached_task_tracker.clone(); - in_flight.push(Box::pin(async move { - // Tracked so shutdown() still awaits the task if - // this awaiter is dropped (e.g. the worker is - // aborted): it holds Arc and must - // not outlive the engine. - let handle = tracker.spawn(async move { - // Cancel-aware: abort when the engine shuts down. - tokio::select! { - () = token.cancelled() => FetchOutcome { - key: fetch_key, - result: FetchResult::SourceFailed, - }, - outcome = execute_single_fetch( - p2p, storage, config, fetch_key, source, - ) => outcome, - } - }); - match handle.await { - Ok(outcome) => (outcome.key, Some(outcome)), - Err(e) => { - error!( - "Fetch task for {} panicked: {e}", - hex::encode(fetch_key) - ); - (fetch_key, None) - } + // Fill free slots from the queue, never sending more than + // `FETCH_MAX_IN_FLIGHT_PER_SOURCE` of this node's fetches to any + // one answering source at once. + let lock_requested = Instant::now(); + let mut q = queues.write().await; + let lock_acquired = Instant::now(); + record_early_answers(&mut q, &mut early_answers); + let free_slots = concurrency.saturating_sub(in_flight.len()); + let dispatch = q.dispatch_fetches(free_slots, FETCH_MAX_IN_FLIGHT_PER_SOURCE); + let parked = dispatch.parked; + for candidate in dispatch.sourceless { + warn!( + "Fetch candidate {} has no untried sources; requeueing for verification", + hex::encode(candidate.key) + ); + let _ = q.requeue_candidate_for_verification( + candidate, + config.verification_request_timeout, + ); + } + for (fetch_key, source) in dispatch.started { + let p2p = Arc::clone(&p2p); + let storage = Arc::clone(&storage); + let config = Arc::clone(&config); + let answers = answers.clone(); + let token = shutdown.clone(); + let tracker = detached_task_tracker.clone(); + in_flight.push(Box::pin(async move { + // Tracked so shutdown() still awaits the task if + // this awaiter is dropped (e.g. the worker is + // aborted): it holds Arc and must + // not outlive the engine. + let handle = tracker.spawn(async move { + // Cancel-aware: abort when the engine shuts down. + tokio::select! { + () = token.cancelled() => FetchOutcome { + key: fetch_key, + source, + result: FetchResult::SourceFailed, + contact: Contact::Unknown, + }, + outcome = execute_single_fetch( + p2p, storage, config, fetch_key, source, answers, + ) => outcome, } - })); - } - } // release queues write lock + }); + let joined = handle.await; + let completed_at = Instant::now(); + match joined { + Ok(outcome) => (outcome.key, Some(outcome), completed_at), + Err(e) => { + error!("Fetch task for {} panicked: {e}", hex::encode(fetch_key)); + (fetch_key, None, completed_at) + } + } + })); + } + drop(q); + dispatch_window.parks += parked; + dispatch_window.record_fill( + lock_acquired.duration_since(lock_requested), + lock_acquired.elapsed(), + ); + if dispatch_window.age() >= FETCH_DISPATCH_SUMMARY_INTERVAL { + let stats_requested = Instant::now(); + let stats = queues + .write() + .await + .take_fetch_dispatch_stats(FETCH_MAX_IN_FLIGHT_PER_SOURCE); + dispatch_window.log_and_reset(&stats, stats_requested.elapsed()); + } if in_flight.is_empty() { // No work — wait for new items or shutdown. @@ -3961,22 +4072,32 @@ impl ReplicationEngine { // Wait for the next fetch to complete and process the result. tokio::select! { () = shutdown.cancelled() => break, - Some((key, maybe_outcome)) = in_flight.next() => { + Some((key, maybe_outcome, completed_at)) = in_flight.next() => { let mut q = queues.write().await; + record_early_answers(&mut q, &mut early_answers); + let attempt = attempt_latency(q.in_flight_started_at(&key), completed_at); + dispatch_window.record_outcome(maybe_outcome.as_ref(), attempt); let terminal = if let Some(outcome) = maybe_outcome { + q.record_fetch_contact(outcome.source, outcome.contact); match apply_fetch_result( &mut q, &key, &outcome.result, config.verification_request_timeout, + xor_distance(&key, p2p.peer_id().as_bytes()), ) { FetchFollowUp::Terminal => true, FetchFollowUp::RequeuedForVerification => false, + FetchFollowUp::Parked => { + dispatch_window.retry_parks += 1; + false + } FetchFollowUp::RetryFrom(next_peer) => { // Spawn a new fetch task for the next source. let p2p = Arc::clone(&p2p); let storage = Arc::clone(&storage); let config = Arc::clone(&config); + let answers = answers.clone(); let token = shutdown.clone(); let tracker = detached_task_tracker.clone(); let fetch_key = key; @@ -3987,21 +4108,32 @@ impl ReplicationEngine { tokio::select! { () = token.cancelled() => FetchOutcome { key: fetch_key, + source: next_peer, result: FetchResult::SourceFailed, + contact: Contact::Unknown, }, outcome = execute_single_fetch( - p2p, storage, config, fetch_key, next_peer, + p2p, + storage, + config, + fetch_key, + next_peer, + answers, ) => outcome, } }); - match handle.await { - Ok(outcome) => (outcome.key, Some(outcome)), + let joined = handle.await; + let completed_at = Instant::now(); + match joined { + Ok(outcome) => { + (outcome.key, Some(outcome), completed_at) + } Err(e) => { error!( "Fetch task for {} panicked: {e}", hex::encode(fetch_key) ); - (fetch_key, None) + (fetch_key, None, completed_at) } } })); @@ -4043,7 +4175,22 @@ impl ReplicationEngine { // Cancel and drain remaining in-flight fetches on shutdown. // The CancellationToken is already cancelled by this point, so // spawned tasks will see cancellation via their select! branches. - while in_flight.next().await.is_some() {} + // Their outcomes are counted but not applied, so the final + // summary's in-flight count still includes them. + let mut drained = Vec::new(); + while let Some(resolved) = in_flight.next().await { + drained.push(resolved); + } + let stats_requested = Instant::now(); + let mut q = queues.write().await; + for (key, maybe_outcome, completed_at) in &drained { + let attempt = attempt_latency(q.in_flight_started_at(key), *completed_at); + dispatch_window.record_outcome(maybe_outcome.as_ref(), attempt); + } + let stats = q.take_fetch_dispatch_stats(FETCH_MAX_IN_FLIGHT_PER_SOURCE); + drop(q); + dispatch_window.shutdown = true; + dispatch_window.log_and_reset(&stats, stats_requested.elapsed()); debug!("Fetch worker shut down"); }); self.task_handles.push(handle); @@ -4589,6 +4736,9 @@ struct ReplicationMessageHandlerContext { fetch_responder_worker_semaphore: Arc, fetch_responder_admission_semaphore: Arc, fetch_responder_inflight: Arc>>, + /// Test-only hold on serving admitted fetch requests. + #[cfg(any(test, feature = "test-utils"))] + fetch_serve_gate: Arc, verification_responder_worker_semaphore: Arc, verification_responder_admission_semaphore: Arc, verification_responder_inflight: Arc>>, @@ -7112,6 +7262,45 @@ pub fn verification_requests_for_key_from_for_test(requester: &PeerId, key: &Xor }) } +/// Test-only: holds an engine's admitted fetch requests before they are served. +/// +/// A held request keeps its admission slot and its requester's per-peer share, +/// exactly as a request waiting for a worker does, so a test can read what a +/// requester has outstanding without racing the speed at which the host serves +/// a chunk. One per engine, so closing it affects no other node or test. +#[cfg(any(test, feature = "test-utils"))] +#[derive(Debug, Default)] +struct FetchServeGate { + closed: AtomicBool, + opened: Notify, +} + +#[cfg(any(test, feature = "test-utils"))] +impl FetchServeGate { + fn close(&self) { + self.closed.store(true, Ordering::Release); + } + + fn open(&self) { + self.closed.store(false, Ordering::Release); + self.opened.notify_waiters(); + } + + /// Wait while the gate is closed. The wakeup is registered before the flag + /// is read, so an open that lands between the two is not missed. + async fn wait_open(&self) { + loop { + let opened = self.opened.notified(); + tokio::pin!(opened); + opened.as_mut().enable(); + if !self.closed.load(Ordering::Acquire) { + return; + } + opened.await; + } + } +} + async fn handle_verification_request( source: &PeerId, request: &protocol::VerificationRequest, @@ -7365,8 +7554,12 @@ async fn dispatch_fetch_request( let p2p_node = Arc::clone(&ctx.p2p_node); let request_timeout = ctx.config.fetch_request_timeout; let rr_message_id = rr_message_id.map(ToOwned::to_owned); + #[cfg(any(test, feature = "test-utils"))] + let serve_gate = Arc::clone(&ctx.fetch_serve_gate); ctx.detached_task_tracker.spawn(async move { let _guard = guard; + #[cfg(any(test, feature = "test-utils"))] + serve_gate.wait_open().await; let Ok(_worker_permit) = worker_semaphore.acquire_owned().await else { debug!( responder_class = "fetch", @@ -8931,7 +9124,10 @@ async fn complete_bootstrap( /// Result classification for a single fetch attempt. enum FetchResult { /// Data fetched, integrity-checked, and stored successfully. - Stored, + Stored { + /// Size of the stored chunk, for the per-source scheduler's summary. + bytes: usize, + }, /// Content-address integrity check failed — do not retry. IntegrityFailed, /// Source failed (network error or non-success response) — retryable. @@ -8979,7 +9175,159 @@ enum FetchResult { /// worker loop to update queue state. struct FetchOutcome { key: XorName, + /// The source this attempt was sent to. + source: PeerId, result: FetchResult, + /// Whether `source` answered, for the per-source cap's exemption of + /// sources that stopped answering. + contact: Contact, +} + +/// How often the fetch worker reports its dispatching. +const FETCH_DISPATCH_SUMMARY_INTERVAL: Duration = Duration::from_mins(5); + +/// How long an attempt that started at `started` took, up to when its task +/// finished rather than when the worker got to its outcome. +fn attempt_latency(started: Option, completed_at: Instant) -> Option { + started.map(|started| completed_at.saturating_duration_since(started)) +} + +/// What the fetch worker observed about its own dispatching since the last +/// summary. +#[derive(Default)] +struct FetchDispatchWindow { + /// When this window began. + started: Option, + /// Longest wait for the queues write lock before a fill pass. + fill_lock_wait_max: Duration, + /// Longest hold of the queues write lock by a fill pass, spawning included. + fill_lock_hold_max: Duration, + /// Fill passes in this window. + fills: u64, + /// Queued keys parked by fill passes in this window. + parks: usize, + /// Failed fetches parked because every untried source was full. + retry_parks: u64, + /// Attempts by outcome. + stored: u64, + already_held: u64, + no_longer_responsible: u64, + source_failed: u64, + integrity_failed: u64, + local_write_failed: u64, + panicked: u64, + /// Attempts cancelled by shutdown before they resolved. + cancelled: u64, + /// Bytes of chunks stored by attempts in this window. + stored_bytes: u64, + /// Attempt latency: longest, and total over `timed_attempts`. + attempt_max: Duration, + attempt_total: Duration, + timed_attempts: u64, + /// Set for the summary logged at shutdown, whose in-flight count includes + /// attempts drained without being applied. + shutdown: bool, +} + +impl FetchDispatchWindow { + fn new() -> Self { + Self { + started: Some(Instant::now()), + ..Self::default() + } + } + + /// Time since this window began. + fn age(&self) -> Duration { + self.started + .map_or(Duration::ZERO, |started| started.elapsed()) + } + + fn record_fill(&mut self, wait: Duration, hold: Duration) { + self.fill_lock_wait_max = self.fill_lock_wait_max.max(wait); + self.fill_lock_hold_max = self.fill_lock_hold_max.max(hold); + self.fills += 1; + } + + /// Count one resolved attempt: `None` when its task panicked. A cancelled + /// attempt is counted as cancelled whatever result it carries, and its + /// cut-short latency is left out. + fn record_outcome(&mut self, outcome: Option<&FetchOutcome>, attempt: Option) { + if outcome.is_some_and(|outcome| outcome.contact == Contact::Unknown) { + self.cancelled += 1; + return; + } + match outcome.map(|outcome| &outcome.result) { + Some(FetchResult::Stored { bytes }) => { + self.stored += 1; + self.stored_bytes = self + .stored_bytes + .saturating_add(u64::try_from(*bytes).unwrap_or(u64::MAX)); + } + Some(FetchResult::AlreadyHeld) => self.already_held += 1, + Some(FetchResult::NoLongerResponsible) => self.no_longer_responsible += 1, + Some(FetchResult::SourceFailed) => self.source_failed += 1, + Some(FetchResult::IntegrityFailed) => self.integrity_failed += 1, + Some(FetchResult::LocalWriteFailed) => self.local_write_failed += 1, + None => self.panicked += 1, + } + if let Some(attempt) = attempt { + self.attempt_max = self.attempt_max.max(attempt); + self.attempt_total = self.attempt_total.saturating_add(attempt); + self.timed_attempts += 1; + } + } + + /// Log this window with the queues' state, then start a new window. + /// `stats_lock` is how long reading that state took, lock wait included. + fn log_and_reset(&mut self, stats: &FetchDispatchStats, stats_lock: Duration) { + let attempt_mean_ms = u32::try_from(self.timed_attempts) + .ok() + .filter(|count| *count > 0) + .map_or(0, |count| (self.attempt_total / count).as_millis()); + info!( + per_source_cap = FETCH_MAX_IN_FLIGHT_PER_SOURCE, + window_secs = self.age().as_secs(), + parked = stats.parked, + parked_associations = stats.parked_associations, + oldest_parked_secs = stats.oldest_parked.map_or(0, |age| age.as_secs()), + queued = stats.queued, + in_flight = stats.in_flight, + saturated_sources = stats.saturated_sources, + fills = self.fills, + parks = self.parks, + retry_parks = self.retry_parks, + stored = self.stored, + stored_bytes = self.stored_bytes, + already_held = self.already_held, + no_longer_responsible = self.no_longer_responsible, + source_failed = self.source_failed, + integrity_failed = self.integrity_failed, + local_write_failed = self.local_write_failed, + panicked = self.panicked, + cancelled = self.cancelled, + shutdown = self.shutdown, + attempt_max_ms = self.attempt_max.as_millis(), + attempt_mean_ms, + fill_lock_wait_max_ms = self.fill_lock_wait_max.as_millis(), + fill_lock_hold_max_ms = self.fill_lock_hold_max.as_millis(), + parked_maintenance_max_ms = stats.parked_maintenance_max.as_millis(), + tracked_sources = stats.tracked_sources, + silent_sources = stats.silent_sources, + health_overflow_active = stats.overflow_active, + silent_peak = stats.health.silent_peak, + silent_starts = stats.health.silent_starts, + unreachable = stats.health.unreachable, + silent_marks = stats.health.marks, + silent_clears = stats.health.clears, + silent_expired = stats.health.expired_marks, + silent_ignored_stale = stats.health.ignored_stale, + health_overflows = stats.health.overflows, + stats_lock_ms = stats_lock.as_millis(), + "replication fetch dispatch summary" + ); + *self = Self::new(); + } } /// What the fetch worker must do next for a key whose fetch attempt resolved. @@ -8993,6 +9341,11 @@ enum FetchFollowUp { RequeuedForVerification, /// Retry immediately from this next untried verified source. RetryFrom(PeerId), + /// Every untried source already has this node's per-source share in + /// flight, so the key went back to the fetch queue, parked, keeping its + /// tried sources and retry reservation. Not terminal: the key is still in + /// the pipeline, and the next fill pass starts it once a share frees. + Parked, } /// Apply a resolved [`FetchResult`] to the queues and report the follow-up. @@ -9013,19 +9366,26 @@ fn apply_fetch_result( key: &XorName, result: &FetchResult, verification_retry_after: Duration, + distance: XorName, ) -> FetchFollowUp { match result { - FetchResult::Stored | FetchResult::NoLongerResponsible | FetchResult::AlreadyHeld => { + FetchResult::Stored { .. } + | FetchResult::NoLongerResponsible + | FetchResult::AlreadyHeld => { q.complete_fetch(key); FetchFollowUp::Terminal } FetchResult::IntegrityFailed | FetchResult::SourceFailed => { - if let Some(next_peer) = q.retry_fetch(key) { - FetchFollowUp::RetryFrom(next_peer) - } else if q.requeue_fetch_for_verification(key, verification_retry_after) { - FetchFollowUp::RequeuedForVerification - } else { - FetchFollowUp::Terminal + match q.retry_or_park_fetch(key, distance, FETCH_MAX_IN_FLIGHT_PER_SOURCE) { + FetchRetry::Next(next_peer) => FetchFollowUp::RetryFrom(next_peer), + FetchRetry::Parked => FetchFollowUp::Parked, + FetchRetry::Exhausted => { + if q.requeue_fetch_for_verification(key, verification_retry_after) { + FetchFollowUp::RequeuedForVerification + } else { + FetchFollowUp::Terminal + } + } } } // No `retry_fetch`: the local write is what failed, so every other @@ -9072,7 +9432,6 @@ async fn is_storage_admitted( .await } -#[allow(clippy::too_many_lines)] /// Execute a single fetch request against `source` for `key`. /// /// Handles encoding, network I/O, integrity checking, storage, and trust @@ -9092,17 +9451,64 @@ async fn execute_single_fetch( config: Arc, key: XorName, source: PeerId, + answers: mpsc::UnboundedSender<(PeerId, Contact)>, ) -> FetchOutcome { + let mut contact = Contact::NotAttempted; + let result = fetch_attempt( + p2p_node, + storage, + config, + key, + source, + &mut contact, + &answers, + ) + .await; + FetchOutcome { + key, + source, + result, + contact, + } +} + +/// Record every answer a fetch has reported since the last call. +/// +/// A fetch reports its source's answer as soon as it arrives, before checking +/// and storing the chunk, so the per-source cap never treats an answering +/// source as silent while that work is still under way. The same answer comes +/// again with the fetch's outcome, which records it a second time to no +/// effect. Each is reported while its fetch is in flight, and the worker calls +/// this before applying any outcome, so it is recorded before its fetch ends. +fn record_early_answers( + queues: &mut ReplicationQueues, + early_answers: &mut mpsc::UnboundedReceiver<(PeerId, Contact)>, +) { + while let Ok((source, contact)) = early_answers.try_recv() { + queues.record_fetch_contact(source, contact); + } +} + +/// The body of [`execute_single_fetch`]: the attempt's result, with `contact` +/// set to whether the source answered, stamped when the request returned. An +/// answer is also sent on `answers` at once. +#[allow(clippy::too_many_lines)] +async fn fetch_attempt( + p2p_node: Arc, + storage: Arc, + config: Arc, + key: XorName, + source: PeerId, + contact: &mut Contact, + answers: &mpsc::UnboundedSender<(PeerId, Contact)>, +) -> FetchResult { let self_id = *p2p_node.peer_id(); if !is_storage_admitted(&self_id, &key, &p2p_node, &config).await { debug!( "Skipping fetch for {}: no longer in the storage-admission group", hex::encode(key) ); - return FetchOutcome { - key, - result: FetchResult::NoLongerResponsible, - }; + return FetchResult::NoLongerResponsible; } // Possession, then capacity — both before the dial, and in that order. @@ -9116,10 +9522,7 @@ async fn execute_single_fetch( "Skipping fetch for {}: already held locally", hex::encode(key) ); - return FetchOutcome { - key, - result: FetchResult::AlreadyHeld, - }; + return FetchResult::AlreadyHeld; } // Capacity is checked before the dial rather than after the bytes arrive. @@ -9141,10 +9544,7 @@ async fn execute_single_fetch( "Skipping fetch for {}: local storage cannot accept a write: {e}", hex::encode(key) ); - return FetchOutcome { - key, - result: FetchResult::LocalWriteFailed, - }; + return FetchResult::LocalWriteFailed; } let request = protocol::FetchRequest { key }; @@ -9157,13 +9557,11 @@ async fn execute_single_fetch( Ok(data) => data, Err(e) => { warn!("Failed to encode fetch request: {e}"); - return FetchOutcome { - key, - result: FetchResult::SourceFailed, - }; + return FetchResult::SourceFailed; } }; + let sent = next_fetch_contact(); let result = p2p_node .send_request( &source, @@ -9172,6 +9570,18 @@ async fn execute_single_fetch( config.fetch_request_timeout, ) .await; + *contact = if result.is_ok() { + Contact::Answered { + order: next_fetch_contact(), + } + } else { + Contact::Unreachable { sent } + }; + if matches!(contact, Contact::Answered { .. }) { + // Fails only once the fetch worker has stopped, when no scheduling + // decision is left to inform. + let _ = answers.send((source, *contact)); + } match result { Ok(response) => { @@ -9182,10 +9592,7 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - return FetchOutcome { - key, - result: FetchResult::SourceFailed, - }; + return FetchResult::SourceFailed; }; match resp_msg.body { @@ -9209,10 +9616,7 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - return FetchOutcome { - key, - result: FetchResult::IntegrityFailed, - }; + return FetchResult::IntegrityFailed; } // Enforce chunk size invariant on fetched data. @@ -9231,10 +9635,7 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - return FetchOutcome { - key, - result: FetchResult::IntegrityFailed, - }; + return FetchResult::IntegrityFailed; } // Content-address integrity check. @@ -9251,10 +9652,7 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - return FetchOutcome { - key, - result: FetchResult::IntegrityFailed, - }; + return FetchResult::IntegrityFailed; } // Responsibility can lapse during the network round-trip. @@ -9268,10 +9666,7 @@ async fn execute_single_fetch( "Fetched {} but responsibility lapsed in transit; not storing", hex::encode(key) ); - return FetchOutcome { - key, - result: FetchResult::NoLongerResponsible, - }; + return FetchResult::NoLongerResponsible; } if let Err(e) = storage.put(&resp_key, &data).await { @@ -9285,16 +9680,10 @@ async fn execute_single_fetch( "Failed to store fetched record {}: {e}", hex::encode(resp_key) ); - return FetchOutcome { - key, - result: FetchResult::LocalWriteFailed, - }; + return FetchResult::LocalWriteFailed; } - FetchOutcome { - key, - result: FetchResult::Stored, - } + FetchResult::Stored { bytes: data.len() } } ReplicationMessageBody::FetchResponse( ref response @ (protocol::FetchResponse::NotFound { .. } @@ -9320,10 +9709,7 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - FetchOutcome { - key, - result: FetchResult::SourceFailed, - } + FetchResult::SourceFailed } _ => { // Unexpected message type — treat as malformed. @@ -9333,10 +9719,7 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - FetchOutcome { - key, - result: FetchResult::SourceFailed, - } + FetchResult::SourceFailed } } } @@ -9344,10 +9727,7 @@ async fn execute_single_fetch( debug!("Fetch request to {source} failed: {e}"); // No ApplicationFailure here — P2PNode::send_request() already // reports ConnectionTimeout / ConnectionFailed to the TrustEngine. - FetchOutcome { - key, - result: FetchResult::SourceFailed, - } + FetchResult::SourceFailed } } } @@ -10537,6 +10917,7 @@ mod tests { use crate::replication::audit::AuditTickResult; use crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL; use crate::replication::recent_provers::RecentProvers; + use crate::replication::scheduling::MAX_FETCH_QUEUE; use crate::replication::types::{AuditFailureReason, FailureEvidence}; use lru::LruCache; use saorsa_core::identity::PeerId; @@ -13779,8 +14160,13 @@ mod tests { drive_key_in_flight(&mut q, key, vec![test_peer(0x01), test_peer(0x02)]); assert_eq!(q.retry_reserved_slot_count(), 1); - let follow_up = - apply_fetch_result(&mut q, &key, &FetchResult::NoLongerResponsible, RETRY_AFTER); + let follow_up = apply_fetch_result( + &mut q, + &key, + &FetchResult::NoLongerResponsible, + RETRY_AFTER, + key, + ); assert_eq!( follow_up, @@ -13810,12 +14196,15 @@ mod tests { // Both variants must walk the identical terminal gate so the // battle-tested Stored accounting (retry-slot release + bootstrap // pending-set shrink in the caller) covers lapsed responsibility too. - for result in [FetchResult::Stored, FetchResult::NoLongerResponsible] { + for result in [ + FetchResult::Stored { bytes: 0 }, + FetchResult::NoLongerResponsible, + ] { let mut q = ReplicationQueues::new(); let key = test_key(0xCD); drive_key_in_flight(&mut q, key, vec![test_peer(0x03)]); - let follow_up = apply_fetch_result(&mut q, &key, &result, RETRY_AFTER); + let follow_up = apply_fetch_result(&mut q, &key, &result, RETRY_AFTER, key); assert_eq!(follow_up, FetchFollowUp::Terminal); assert!(!q.contains_key(&key)); @@ -13833,7 +14222,8 @@ mod tests { let second = test_peer(0x05); drive_key_in_flight(&mut q, key, vec![first, second]); - let follow_up = apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER); + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); assert_eq!( follow_up, FetchFollowUp::RetryFrom(second), @@ -13841,7 +14231,8 @@ mod tests { ); assert_eq!(q.in_flight_count(), 1, "retry keeps the key in flight"); - let follow_up = apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER); + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); assert_eq!( follow_up, FetchFollowUp::RequeuedForVerification, @@ -13855,6 +14246,199 @@ mod tests { ); } + /// Put two unrelated fetches in flight to `source`, filling its share. + fn saturate_source(q: &mut ReplicationQueues, source: PeerId, first_key_byte: u8) { + for offset in 0..2u8 { + let key = test_key(first_key_byte + offset); + q.start_fetch(key, source, vec![source]); + } + } + + #[test] + fn source_failure_with_every_untried_source_full_parks_the_key() { + const RETRY_AFTER: Duration = Duration::from_mins(1); + + let mut q = ReplicationQueues::new(); + let key = test_key(0x2A); + let (first, second) = (test_peer(0x01), test_peer(0x02)); + assert_eq!(drive_key_in_flight(&mut q, key, vec![first, second]), first); + saturate_source(&mut q, second, 0xE0); + let reserved = q.retry_reserved_slot_count(); + + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); + + assert_eq!( + follow_up, + FetchFollowUp::Parked, + "the only untried source is at its share, so the key must wait rather \ + than be sent a request it will drop" + ); + assert!( + q.contains_key(&key), + "a parked key is still in the pipeline" + ); + assert!(q.fetch_queued_or_in_flight(&key)); + assert_eq!( + q.in_flight_count(), + 2, + "only the two saturating fetches remain" + ); + assert_eq!( + q.pending_count(), + 0, + "parking is not a verification requeue" + ); + assert_eq!(q.retry_reserved_slot_count(), reserved); + } + + #[test] + fn an_answer_reported_before_its_fetch_ends_keeps_a_timed_out_source_capped() { + const RETRY_AFTER: Duration = Duration::from_mins(1); + const SLOTS: usize = 8; + + let source = test_peer(0x0A); + let (dropped, answered) = (test_key(0x50), test_key(0x51)); + let waiting: Vec = (0x60..0x64).map(test_key).collect(); + // Both of `source`'s shares are in flight and four keys only it holds + // wait behind them. + let setup = || { + let mut q = ReplicationQueues::new(); + q.start_fetch(dropped, source, vec![source]); + q.start_fetch(answered, source, vec![source]); + for key in &waiting { + assert!(q.enqueue_fetch(*key, *key, vec![source])); + } + q + }; + let in_flight_to_source = |q: &ReplicationQueues| { + q.fetch_pipeline_snapshot() + .in_flight + .iter() + .filter(|(_, peer)| *peer == source) + .count() + }; + // The first request is dropped and times out. The second, sent after + // it, has been answered and is still checking and storing its chunk. + let dropped_sent = next_fetch_contact(); + let _answered_sent = next_fetch_contact(); + let answer = Contact::Answered { + order: next_fetch_contact(), + }; + let timeout = Contact::Unreachable { sent: dropped_sent }; + + // Known only from the second fetch's outcome, the answer comes too + // late: the timeout marks `source` silent and the next pass sends it + // every waiting key, past its share. + let mut late = setup(); + late.record_fetch_contact(source, timeout); + apply_fetch_result( + &mut late, + &dropped, + &FetchResult::SourceFailed, + RETRY_AFTER, + dropped, + ); + late.dispatch_fetches(SLOTS, FETCH_MAX_IN_FLIGHT_PER_SOURCE); + assert_eq!(in_flight_to_source(&late), 1 + waiting.len()); + + // Reported when it arrives, the answer is recorded before the timeout, + // which it outdates, so `source` stays within its share. + let (answers, mut early_answers) = mpsc::unbounded_channel(); + let mut q = setup(); + assert!(answers.send((source, answer)).is_ok()); + record_early_answers(&mut q, &mut early_answers); + q.record_fetch_contact(source, timeout); + apply_fetch_result( + &mut q, + &dropped, + &FetchResult::SourceFailed, + RETRY_AFTER, + dropped, + ); + record_early_answers(&mut q, &mut early_answers); + let dispatch = q.dispatch_fetches(SLOTS, FETCH_MAX_IN_FLIGHT_PER_SOURCE); + assert_eq!(dispatch.started.len(), 1, "one share is free"); + assert_eq!(in_flight_to_source(&q), FETCH_MAX_IN_FLIGHT_PER_SOURCE); + + // The same answer again, with the second fetch's outcome, changes + // nothing. + q.record_fetch_contact(source, answer); + assert_eq!( + q.dispatch_fetches(SLOTS, FETCH_MAX_IN_FLIGHT_PER_SOURCE) + .started + .len(), + 0 + ); + assert_eq!(in_flight_to_source(&q), FETCH_MAX_IN_FLIGHT_PER_SOURCE); + } + + #[test] + fn the_dispatch_window_times_resolved_attempts_and_counts_cancelled_ones() { + let (key, source) = (test_key(0x31), test_peer(0x01)); + let outcome = |result: FetchResult, contact: Contact| FetchOutcome { + key, + source, + result, + contact, + }; + let started = Instant::now(); + let at = |millis: u64| started + Duration::from_millis(millis); + let mut window = FetchDispatchWindow::new(); + + // Two attempts resolve at different times; the worker reaches both + // later, which must not count. + let stored = outcome( + FetchResult::Stored { bytes: 10 }, + Contact::Answered { order: 2 }, + ); + window.record_outcome(Some(&stored), attempt_latency(Some(started), at(100))); + let failed = outcome(FetchResult::SourceFailed, Contact::Unreachable { sent: 1 }); + window.record_outcome(Some(&failed), attempt_latency(Some(started), at(300))); + // A cancelled attempt is counted as cancelled, not by its placeholder + // result, and its cut-short time is left out. + let cancelled = outcome(FetchResult::SourceFailed, Contact::Unknown); + window.record_outcome(Some(&cancelled), attempt_latency(Some(started), at(5_000))); + + assert_eq!((window.stored, window.source_failed), (1, 1)); + assert_eq!((window.cancelled, window.timed_attempts), (1, 2)); + assert_eq!(window.attempt_max, Duration::from_millis(300)); + assert_eq!(window.attempt_total, Duration::from_millis(400)); + assert_eq!(window.stored_bytes, 10); + assert_eq!(attempt_latency(None, at(100)), None); + } + + #[test] + fn source_failure_with_a_full_fetch_queue_requeues_for_verification() { + const RETRY_AFTER: Duration = Duration::from_mins(1); + + let mut q = ReplicationQueues::new(); + let key = test_key(0x2B); + let (first, second) = (test_peer(0x01), test_peer(0x02)); + drive_key_in_flight(&mut q, key, vec![first, second]); + saturate_source(&mut q, second, 0xE0); + let filler_source = test_peer(0x09); + let mut filler = 0u32; + while q.fetch_queue_count() < MAX_FETCH_QUEUE { + let mut filler_key = [0xC0u8; 32]; + filler_key[..4].copy_from_slice(&filler.to_be_bytes()); + assert!(q.enqueue_fetch(filler_key, filler_key, vec![filler_source])); + filler += 1; + } + + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); + + assert_eq!( + follow_up, + FetchFollowUp::RequeuedForVerification, + "with no room to park, the key returns to verification on its own \ + reserved slot instead of overfilling the fetch queue" + ); + assert_eq!(q.fetch_queue_count(), MAX_FETCH_QUEUE); + assert_eq!(q.pending_count(), 1); + } + #[test] fn source_failure_without_retry_metadata_is_terminal() { const RETRY_AFTER: Duration = Duration::from_mins(1); @@ -13868,7 +14452,8 @@ mod tests { let candidate = q.dequeue_fetch().expect("enqueued key must dequeue"); q.start_dequeued_fetch(candidate, source); - let follow_up = apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER); + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); assert_eq!(follow_up, FetchFollowUp::Terminal); assert!(!q.contains_key(&key)); @@ -13883,7 +14468,8 @@ mod tests { let source = test_peer(0x0B); drive_key_in_flight(&mut q, key, vec![source, test_peer(0x0C)]); - let follow_up = apply_fetch_result(&mut q, &key, &FetchResult::AlreadyHeld, RETRY_AFTER); + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::AlreadyHeld, RETRY_AFTER, key); assert_eq!( follow_up, @@ -13912,8 +14498,13 @@ mod tests { let third = test_peer(0x09); drive_key_in_flight(&mut q, key, vec![first, second, third]); - let follow_up = - apply_fetch_result(&mut q, &key, &FetchResult::LocalWriteFailed, RETRY_AFTER); + let follow_up = apply_fetch_result( + &mut q, + &key, + &FetchResult::LocalWriteFailed, + RETRY_AFTER, + key, + ); assert_eq!( follow_up, @@ -13946,8 +14537,13 @@ mod tests { let candidate = q.dequeue_fetch().expect("enqueued key must dequeue"); q.start_dequeued_fetch(candidate, source); - let follow_up = - apply_fetch_result(&mut q, &key, &FetchResult::LocalWriteFailed, RETRY_AFTER); + let follow_up = apply_fetch_result( + &mut q, + &key, + &FetchResult::LocalWriteFailed, + RETRY_AFTER, + key, + ); assert_eq!(follow_up, FetchFollowUp::Terminal); assert!(!q.contains_key(&key)); diff --git a/src/replication/scheduling.rs b/src/replication/scheduling.rs index e296606e..95c6b168 100644 --- a/src/replication/scheduling.rs +++ b/src/replication/scheduling.rs @@ -5,13 +5,15 @@ //! time, with strict dedup across all three stages. use std::cmp::Ordering; -use std::collections::{BTreeMap, BinaryHeap, HashMap, HashSet, VecDeque}; +use std::collections::{BTreeMap, BTreeSet, BinaryHeap, HashMap, HashSet, VecDeque}; use std::time::{Duration, Instant}; use crate::logging::debug; -use crate::ant_protocol::XorName; -use crate::replication::config::VERIFICATION_RETRY_BACKOFF_MAX; +use crate::ant_protocol::{XorName, CLOSE_GROUP_SIZE}; +use crate::replication::config::{ + FETCH_REQUEST_TIMEOUT, PAID_LIST_CLOSE_GROUP_SIZE, VERIFICATION_RETRY_BACKOFF_MAX, +}; use crate::replication::types::{ FetchCandidate, FetchOrder, FetchPayload, VerificationEntry, VerificationState, }; @@ -94,6 +96,32 @@ pub const MAX_FETCH_QUEUE: usize = 131_072; /// decides *which* sources survive the cap. pub const MAX_HINT_SOURCES_PER_KEY: usize = 8; +/// Maximum distinct source peers one key may be associated with while it is +/// queued or in flight for fetch, counting the sources it was already fetched +/// from this cycle, for the given close-group widths. +/// +/// A verification round names at most its quorum targets and its paid-list +/// targets as holders, and the replica-hint fallbacks added to them come from a +/// pending entry already bounded by [`MAX_HINT_SOURCES_PER_KEY`], so this never +/// truncates a source list a round produced. It bounds what later +/// advertisements can merge into a queued or in-flight key, and with it the +/// parked-fetch index, which holds one entry per untried source of each parked +/// key. +#[must_use] +pub(crate) const fn fetch_source_limit( + close_group_size: usize, + paid_list_close_group_size: usize, +) -> usize { + close_group_size + .saturating_add(paid_list_close_group_size) + .saturating_add(MAX_HINT_SOURCES_PER_KEY) +} + +/// [`fetch_source_limit`] at the production close-group widths, and the limit +/// [`ReplicationQueues::new`] starts with. +pub(crate) const MAX_FETCH_SOURCES_PER_KEY: usize = + fetch_source_limit(CLOSE_GROUP_SIZE, PAID_LIST_CLOSE_GROUP_SIZE); + /// Outcome of [`ReplicationQueues::add_pending_verify`]. /// /// Distinguishes "the key is already being handled" from "the key was @@ -174,9 +202,13 @@ pub struct InFlightEntry { pub key: XorName, /// The peer we are currently fetching from. pub source: PeerId, - /// When the fetch started. + /// When the current attempt started: the first, or under the per-source + /// scheduler the latest retry. pub started_at: Instant, - /// All verified sources for this key. + /// Verified sources for this key not yet tried before this episode's + /// current attempt. A key that was parked between attempts keeps the + /// sources it already tried only in `tried`, so everything the key is + /// associated with is the union of the two. pub all_sources: Vec, /// Sources already attempted (failed or in progress). pub tried: HashSet, @@ -184,6 +216,409 @@ pub struct InFlightEntry { pub retry_verification: Option, } +// --------------------------------------------------------------------------- +// Per-source fetch dispatch +// --------------------------------------------------------------------------- + +/// A queued key none of whose untried sources had a free share when it was +/// last considered for dispatch. +/// +/// Its payload stays in `fetch_payloads`; only its place in line moves here +/// from the fetch heap. +#[derive(Debug, Clone)] +struct ParkedFetch { + /// XOR distance from self to the key, its priority once startable again. + distance: XorName, + /// Untried sources whose next free share could start the key, in the + /// payload's source-preference order. Each one indexes the key in + /// `parked_by_source`. + wake_sources: Vec, + /// When the key was parked, for the dispatch summary's oldest-parked age. + parked_at: Instant, +} + +/// Fetches started, and candidates found with no untried source, by one +/// [`ReplicationQueues::dispatch_fetches`] pass. +#[derive(Debug, Default)] +pub(crate) struct FetchDispatch { + /// Each started key with the source it was sent to. Already in flight. + pub(crate) started: Vec<(XorName, PeerId)>, + /// Dequeued candidates with no untried source left. The caller decides + /// what to do with them, exactly as for a sourceless dequeue. + pub(crate) sourceless: Vec, + /// Heap keys this pass parked because every untried source was full. + pub(crate) parked: usize, +} + +/// The per-source scheduler's queue state, for its periodic summary. +/// +/// Read only by that summary's log line, which a build without logging +/// compiles out. +#[derive(Debug, Clone, Copy)] +#[cfg_attr(not(feature = "logging"), allow(dead_code))] +pub(crate) struct FetchDispatchStats { + /// Keys parked behind full sources. + pub(crate) parked: usize, + /// How long the longest-parked key has waited, if any is parked. + pub(crate) oldest_parked: Option, + /// Keys queued for fetch, parked ones included. + pub(crate) queued: usize, + /// Fetches in flight. + pub(crate) in_flight: usize, + /// Sources holding at least the per-source cap of this node's fetches. + pub(crate) saturated_sources: usize, + /// Entries in the parked index: one per untried source of each parked key. + pub(crate) parked_associations: usize, + /// Longest parked-index maintenance after a departed source, since the last + /// read. + pub(crate) parked_maintenance_max: Duration, + /// Sources in the health table before the summary's purge. + pub(crate) tracked_sources: usize, + /// Tracked sources silent now. + pub(crate) silent_sources: usize, + /// Whether the health table is overflowing, so every untracked source + /// also counts as silent. + pub(crate) overflow_active: bool, + /// Source-health events since the last read. + pub(crate) health: SourceHealthCounters, +} + +/// What [`ReplicationQueues::retry_or_park_fetch`] did with a failed fetch. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum FetchRetry { + /// Retry now from this untried source, which has a free share. + Next(PeerId), + /// Every untried source is at its share: the key went back to the fetch + /// queue, parked, keeping its tried sources and retry reservation. + Parked, + /// No untried source is left, or the fetch queue is full. The key stays + /// in flight, with its failed source now marked tried, for the caller's + /// verification requeue. + Exhausted, +} + +// --------------------------------------------------------------------------- +// Fetch source health +// --------------------------------------------------------------------------- + +/// How many fetch timeouts a source stays silent after this node last learned +/// of an unanswered request to it. +const SILENCE_LIFETIME_TIMEOUTS: u32 = 2; + +/// Shortest time a mark lasts, whatever the fetch timeout, so a zero or +/// tiny configured timeout cannot turn silent sources off. +const MIN_SILENCE_LIFETIME: Duration = Duration::from_secs(1); + +/// Most fetch sources whose answering this node tracks at once. +/// +/// Far above the sources one node fetches from within a silence lifetime. At +/// the bound, entries that neither hold a mark nor guard an in-flight fetch +/// are dropped. If that frees nothing, a source not in the table counts as +/// silent for one silence lifetime: fetched from as it was before the +/// per-source cap, never more slowly. +const MAX_TRACKED_FETCH_SOURCES: usize = 8_192; + +/// Whether a fetch attempt reached its source. +/// +/// Requests and answers carry their place in one process-wide sequence, taken +/// as the request is sent and as its answer returns, so the table can tell +/// which came first however the fetch worker orders their outcomes and +/// whatever the clock reads. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum Contact { + /// The attempt ended before a request was sent. + NotAttempted, + /// The source answered, whatever the answer then turned out to be. + Answered { + /// Where the answer's arrival falls in the sequence. + order: u64, + }, + /// The request got no answer: a timeout or a connection failure. + Unreachable { + /// Where the request's sending falls in the sequence. + sent: u64, + }, + /// The attempt was cancelled at shutdown, so nothing is known. + Unknown, +} + +/// What this node has observed of one source's answering. +#[derive(Debug, Clone, Copy, Default)] +struct SourceHealth { + /// Sequence place of the latest answer from this source. + last_answer: Option, + /// Sequence place of the latest-sent request to this source that got no + /// answer. + last_unanswered: Option, + /// When this node learned of that request's failure, while the request is + /// newer than the latest answer and the mark has not expired. + marked_since: Option, +} + +impl SourceHealth { + /// Whether a request sent after this source's latest answer went + /// unanswered. It depends only on the two latest events, so it is the same + /// whatever order the outcomes were applied in. + fn unanswered_since_answer(&self) -> bool { + self.last_unanswered + .is_some_and(|sent| self.last_answer.is_none_or(|answer| sent > answer)) + } +} + +/// Source-health events since the dispatch summary last read them. +/// +/// Read only by that summary's log line, which a build without logging +/// compiles out. +#[derive(Debug, Clone, Copy, Default)] +#[cfg_attr(not(feature = "logging"), allow(dead_code))] +pub(crate) struct SourceHealthCounters { + /// Attempts whose request got no answer. + pub(crate) unreachable: u64, + /// Sources that became silent. + pub(crate) marks: u64, + /// Marks cleared by an answer. + pub(crate) clears: u64, + /// Marks that ran out their lifetime. + pub(crate) expired_marks: u64, + /// Unanswered requests sent before their source's latest answer. + pub(crate) ignored_stale: u64, + /// Fetches started or retried on a silent source, outside the cap. + pub(crate) silent_starts: u64, + /// Times the table was full of marks and in-flight guards. + pub(crate) overflows: u64, + /// Most tracked sources silent at once. + pub(crate) silent_peak: usize, +} + +/// What this node has observed of each fetch source's answering. +/// +/// The per-source cap keeps this node within the admission share a responder +/// grants it, and only a responder that answers has a share to keep within. A +/// source is silent while the latest-sent request to it that went unanswered +/// was sent after its latest answer: it is fetched from without the cap, as +/// before the cap existed. Any answer received after that request was sent +/// ends it. So does learning of no newer unanswered request for one silence +/// lifetime, so a source this node has stopped trying is probed again under +/// the cap. +#[derive(Debug)] +struct SourceHealthTable { + entries: HashMap, + /// Every mark as `(marked_since, source)`, so marks expire in order. + marks_by_since: BTreeSet<(Instant, PeerId)>, + /// How long a mark lasts without newer evidence. + silence_lifetime: Duration, + /// When the table last overflowed. For one silence lifetime after, a + /// source not in the table counts as silent. + overflow_since: Option, + counters: SourceHealthCounters, +} + +impl SourceHealthTable { + fn new(fetch_request_timeout: Duration) -> Self { + Self { + entries: HashMap::new(), + marks_by_since: BTreeSet::new(), + silence_lifetime: fetch_request_timeout + .saturating_mul(SILENCE_LIFETIME_TIMEOUTS) + .max(MIN_SILENCE_LIFETIME), + overflow_since: None, + counters: SourceHealthCounters::default(), + } + } + + /// Whether something that began at `since` is still within one silence + /// lifetime at `now`. + fn within_lifetime(&self, since: Instant, now: Instant) -> bool { + now.saturating_duration_since(since) < self.silence_lifetime + } + + fn overflowing(&self, now: Instant) -> bool { + self.overflow_since + .is_some_and(|since| self.within_lifetime(since, now)) + } + + /// Whether `source` counts as silent at `now`. + fn is_silent(&self, source: &PeerId, now: Instant) -> bool { + self.entries.get(source).map_or_else( + || self.overflowing(now), + |health| { + health + .marked_since + .is_some_and(|since| self.within_lifetime(since, now)) + }, + ) + } + + /// Drop the marks that ran out their lifetime by `now`. + fn expire(&mut self, now: Instant) { + while let Some(&(since, source)) = self.marks_by_since.first() { + if self.within_lifetime(since, now) { + break; + } + self.marks_by_since.pop_first(); + if let Some(health) = self.entries.get_mut(&source) { + if health.marked_since == Some(since) { + health.marked_since = None; + } + } + self.counters.expired_marks += 1; + } + } + + /// Drop every entry that neither holds a mark nor guards an in-flight + /// fetch to its source. Every contact, an answer reported ahead of its + /// fetch's outcome included, is applied while its own fetch is still in + /// flight, so none can arrive for a dropped source's past requests: + /// anything later is newer than all of its history. Call after + /// [`Self::expire`]. + fn purge(&mut self, in_flight: &HashMap) { + let guarded: HashSet = in_flight.values().map(|entry| entry.source).collect(); + self.entries + .retain(|source, health| health.marked_since.is_some() || guarded.contains(source)); + } + + /// The entry for `source`, purging the table at the bound. `None` when the + /// table cannot take it, which starts an overflow. + fn entry( + &mut self, + source: PeerId, + now: Instant, + in_flight: &HashMap, + ) -> Option<&mut SourceHealth> { + if !self.entries.contains_key(&source) && self.entries.len() >= MAX_TRACKED_FETCH_SOURCES { + if self.overflowing(now) { + return None; + } + self.purge(in_flight); + if self.entries.len() >= MAX_TRACKED_FETCH_SOURCES { + self.overflow_since = Some(now); + self.counters.overflows += 1; + return None; + } + } + Some(self.entries.entry(source).or_default()) + } + + /// Apply, at `now`, one attempt's contact with `source`. + fn record( + &mut self, + source: PeerId, + contact: Contact, + now: Instant, + in_flight: &HashMap, + ) { + self.expire(now); + match contact { + Contact::NotAttempted | Contact::Unknown => {} + Contact::Answered { order } => { + let Some(health) = self.entry(source, now, in_flight) else { + return; + }; + health.last_answer = health.last_answer.max(Some(order)); + let cleared = if health.unanswered_since_answer() { + None + } else { + health.marked_since.take() + }; + if let Some(since) = cleared { + self.marks_by_since.remove(&(since, source)); + self.counters.clears += 1; + } + } + Contact::Unreachable { sent } => { + self.counters.unreachable += 1; + let Some(health) = self.entry(source, now, in_flight) else { + return; + }; + let stale = health.last_answer.is_some_and(|answer| sent < answer); + let newest = health.last_unanswered.is_none_or(|latest| sent > latest); + if newest { + health.last_unanswered = Some(sent); + } + // The newest unanswered request, if newer than every answer, + // marks the source from now: from when this node learned of it. + let (replaced, marked) = if newest && health.unanswered_since_answer() { + (health.marked_since.replace(now), true) + } else { + (None, false) + }; + if stale { + self.counters.ignored_stale += 1; + } + if marked { + if let Some(since) = replaced { + self.marks_by_since.remove(&(since, source)); + } else { + self.counters.marks += 1; + } + self.marks_by_since.insert((now, source)); + self.counters.silent_peak = + self.counters.silent_peak.max(self.marks_by_since.len()); + } + } + } + } + + /// The counters since the last call, and the table's size before the + /// purge that follows. + fn take_counters( + &mut self, + now: Instant, + in_flight: &HashMap, + ) -> (SourceHealthCounters, usize) { + self.expire(now); + let tracked = self.entries.len(); + self.purge(in_flight); + let counters = std::mem::take(&mut self.counters); + self.counters.silent_peak = self.marks_by_since.len(); + (counters, tracked) + } +} + +/// What lets a parked key start now: an untried source below the per-source +/// cap, or a silent one. +#[derive(Clone, Copy)] +struct Startable<'a> { + in_flight: &'a HashMap, + per_source_cap: usize, + now: Instant, +} + +/// The next queued key, taken off the heap or out of the parked set. +/// +/// Returned and matched at once, never stored, so the size gap between the +/// variants costs nothing; boxing would add an allocation per dispatched fetch. +#[allow(clippy::large_enum_variant)] +enum TakenFetch { + /// The key, rejoined with its payload. + Candidate(FetchCandidate), + /// An inconsistent entry was dropped instead; try again. + Skipped, + /// The heap is empty and no parked key was asked for. + Empty, +} + +/// How [`ReplicationQueues::dispatch_fetches`] may start one candidate. +enum SourceChoice { + /// This untried source has a free share. + Start(PeerId), + /// Untried sources exist, but every one is at its share. + AllBusy, + /// No untried source is left. + NoSource, +} + +/// Test-only: every key queued or parked for fetch, and every in-flight fetch +/// with its source, read under one lock. +#[cfg(any(test, feature = "test-utils"))] +#[derive(Debug, Clone, Default)] +pub struct FetchPipelineSnapshot { + /// Keys queued for fetch, on the heap or parked. + pub queued: Vec, + /// In-flight fetches with the source each one was sent to. + pub in_flight: Vec<(XorName, PeerId)>, +} + // --------------------------------------------------------------------------- // Central queue manager // --------------------------------------------------------------------------- @@ -205,10 +640,11 @@ pub struct ReplicationQueues { /// Capacity-bounded by [`MAX_PENDING_VERIFY`]. At capacity, max-min sender /// accounting reclaims borrowed entries before rejecting new work. pending_verify: HashMap, - /// Nearest-first priority order over the keys in `fetch_payloads`. + /// Nearest-first priority order over the queued keys that are not parked. /// /// Holds ordering data only: every entry has exactly one matching - /// `fetch_payloads` entry, and vice versa. Splitting the mutable half of a + /// `fetch_payloads` entry, and every `fetch_payloads` entry is represented + /// exactly once, here or in `fetch_parked`. Splitting the mutable half of a /// queued fetch out into `fetch_payloads` keeps this heap's ordering /// immutable for as long as a key is queued, so a source merge never has /// to rebuild it. @@ -221,8 +657,30 @@ pub struct ReplicationQueues { /// Capacity-bounded by [`MAX_FETCH_QUEUE`]: enqueues are dropped once /// full, preventing unbounded growth under a network hint flood. fetch_payloads: HashMap, + /// Queued keys taken off `fetch_queue` because every untried source was at + /// its per-source share. Every key in `fetch_payloads` is in exactly one of + /// `fetch_queue` and this map. + fetch_parked: HashMap, + /// Exact reverse index over `fetch_parked`: `(distance, key)` is in the set + /// for `source` if and only if `key` is parked with `source` among its + /// wake sources. Ordered so the nearest parked key per source is `first()`; + /// empty sets are removed. + parked_by_source: HashMap>, + /// Sources a queued key was already fetched from in the current + /// verification cycle. Present only for a key that went back to the queue + /// from flight, and carried into its next in-flight entry. + fetch_tried: HashMap>, /// Active downloads keyed by `XorName`. in_flight_fetch: HashMap, + /// Most distinct sources a queued or in-flight key may be associated with, + /// tried ones included. See [`fetch_source_limit`]. + max_fetch_sources: usize, + /// Longest parked-index maintenance after a departed source since the + /// dispatch summary last read it. + parked_maintenance_max: Duration, + /// Which fetch sources stopped answering, so the per-source cap can leave + /// them out. + source_health: SourceHealthTable, /// Reverse index for removing a departed peer from every pending hint /// without scanning the entire pending table. pending_keys_by_source: HashMap>, @@ -272,7 +730,13 @@ impl ReplicationQueues { pending_verify: HashMap::new(), fetch_queue: BinaryHeap::new(), fetch_payloads: HashMap::new(), + fetch_parked: HashMap::new(), + parked_by_source: HashMap::new(), + fetch_tried: HashMap::new(), in_flight_fetch: HashMap::new(), + max_fetch_sources: MAX_FETCH_SOURCES_PER_KEY, + parked_maintenance_max: Duration::ZERO, + source_health: SourceHealthTable::new(FETCH_REQUEST_TIMEOUT), pending_keys_by_source: HashMap::new(), capacity_owner_by_key: HashMap::new(), pending_keys_by_owner: HashMap::new(), @@ -286,6 +750,39 @@ impl ReplicationQueues { } } + /// Use `limit` as the most distinct sources a queued or in-flight key may + /// be associated with, for close-group widths other than the production + /// ones. See [`fetch_source_limit`]. + #[must_use] + pub(crate) fn with_fetch_source_limit(mut self, limit: usize) -> Self { + self.set_fetch_source_limit(limit); + self + } + + /// Change the source limit. Only for queues no key has entered yet, such + /// as an engine's before it starts. + pub(crate) fn set_fetch_source_limit(&mut self, limit: usize) { + self.max_fetch_sources = limit.max(1); + } + + /// Use `timeout`, the fetch request timeout, to time how long a source + /// that stopped answering stays silent. Only for queues no fetch has + /// resolved in yet. + #[must_use] + pub(crate) fn with_fetch_request_timeout(mut self, timeout: Duration) -> Self { + self.source_health = SourceHealthTable::new(timeout); + self + } + + /// Apply whether a fetch attempt's `source` answered. Called for every + /// resolved attempt, before the attempt's result moves its key on, and for + /// an answer as soon as it arrives, while its attempt is still in flight. + /// Applying the same answer twice changes nothing. + pub(crate) fn record_fetch_contact(&mut self, source: PeerId, contact: Contact) { + self.source_health + .record(source, contact, Instant::now(), &self.in_flight_fetch); + } + // ----------------------------------------------------------------------- // PendingVerify // ----------------------------------------------------------------------- @@ -302,6 +799,7 @@ impl ReplicationQueues { /// source-side retry) MUST treat this as still-outstanding work, not as /// "done". Without this distinction a bootstrap snapshot whose hints /// are capacity-rejected would silently mark itself drained. + #[allow(clippy::too_many_lines)] pub fn add_pending_verify( &mut self, key: XorName, @@ -373,32 +871,66 @@ impl ReplicationQueues { return AdmissionResult::AlreadyPresent; } // Merging a source touches only `FetchPayload`, which no ordering - // reads, so the fetch heap is left completely untouched here. + // reads, so the fetch heap is left completely untouched here. A source + // this key was already fetched from this cycle is not merged back: it + // would be asked twice. A parked key can start from a merged source as + // soon as that source has a free share, so it is indexed there too. if let Some(payload) = self.fetch_payloads.get_mut(&key) { + let tried = self.fetch_tried.get(&key); + let mut merged = Vec::new(); for source in &entry.replica_hint_sources { - if !payload.sources.contains(source) { + // Payload sources and tried sources are disjoint, so this is + // the size of everything the key has been associated with. + if payload.sources.len() + tried.map_or(0, HashSet::len) >= self.max_fetch_sources { + break; + } + if !payload.sources.contains(source) && !tried.is_some_and(|t| t.contains(source)) { payload.sources.push(*source); + merged.push(*source); + } + } + if let Some(parked) = self.fetch_parked.get_mut(&key) { + for source in merged { + parked.wake_sources.push(source); + self.parked_by_source + .entry(source) + .or_default() + .insert((parked.distance, key)); } } if let Some(retry) = &mut payload.retry_verification { - retry - .replica_hint_sources - .extend(entry.replica_hint_sources); - retry.hint_sources.extend(entry.hint_sources); + Self::merge_retry_hints( + retry, + &entry.replica_hint_sources, + &entry.hint_sources, + self.capacity_owner_by_key.get(&key).copied(), + ); } return AdmissionResult::AlreadyPresent; } if let Some(in_flight) = self.in_flight_fetch.get_mut(&key) { + let mut associated = in_flight.all_sources.len() + + in_flight + .tried + .iter() + .filter(|peer| !in_flight.all_sources.contains(peer)) + .count(); for source in &entry.replica_hint_sources { - if !in_flight.all_sources.contains(source) { + if associated >= self.max_fetch_sources { + break; + } + if !in_flight.all_sources.contains(source) && !in_flight.tried.contains(source) { in_flight.all_sources.push(*source); + associated += 1; } } if let Some(retry) = &mut in_flight.retry_verification { - retry - .replica_hint_sources - .extend(entry.replica_hint_sources); - retry.hint_sources.extend(entry.hint_sources); + Self::merge_retry_hints( + retry, + &entry.replica_hint_sources, + &entry.hint_sources, + self.capacity_owner_by_key.get(&key).copied(), + ); } return AdmissionResult::AlreadyPresent; } @@ -652,10 +1184,7 @@ impl ReplicationQueues { key: &XorName, protected_owner: Option, ) -> bool { - let victim = entry.hint_sources.iter().copied().find(|source| { - !entry.replica_hint_sources.contains(source) && protected_owner != Some(*source) - }); - let Some(victim) = victim else { + let Some(victim) = Self::paid_only_victim(entry, protected_owner) else { return false; }; entry.hint_sources.remove(&victim); @@ -668,6 +1197,53 @@ impl ReplicationQueues { true } + /// The paid-only advertiser [`Self::displace_paid_only_source`] would drop + /// from `entry`: never a replica claimant, never `protected_owner`. + fn paid_only_victim( + entry: &VerificationEntry, + protected_owner: Option, + ) -> Option { + entry.hint_sources.iter().copied().find(|source| { + !entry.replica_hint_sources.contains(source) && protected_owner != Some(*source) + }) + } + + /// Merge a re-advertisement into the verification entry a fetch-stage key + /// keeps for a later retry, under the bound and in the order + /// [`Self::add_pending_verify`] applies to a pending entry: replica + /// claimants first, displacing paid-only advertisers other than + /// `protected_owner` when the set is full, then paid-only advertisers + /// while it is below [`MAX_HINT_SOURCES_PER_KEY`]. A fetch-stage key is in + /// no pending index, so only the entry changes. + fn merge_retry_hints( + retry: &mut VerificationEntry, + replica_hint_sources: &HashSet, + hint_sources: &HashSet, + protected_owner: Option, + ) { + for source in replica_hint_sources { + if retry.replica_hint_sources.contains(source) { + continue; + } + if !retry.hint_sources.contains(source) + && retry.hint_sources.len() >= MAX_HINT_SOURCES_PER_KEY + { + let Some(victim) = Self::paid_only_victim(retry, protected_owner) else { + break; + }; + retry.hint_sources.remove(&victim); + } + retry.replica_hint_sources.insert(*source); + retry.hint_sources.insert(*source); + } + for source in hint_sources { + if retry.hint_sources.len() >= MAX_HINT_SOURCES_PER_KEY { + break; + } + retry.hint_sources.insert(*source); + } + } + /// Reserve a pending slot for `key`, charged to its capacity owner. /// /// Must be called while `capacity_owner_by_key` still holds `key`, which is @@ -951,8 +1527,9 @@ impl ReplicationQueues { .retain(|key, _| !orphaned_fetch_keys.contains(key)); self.fetch_queue .retain(|order| !orphaned_fetch_keys.contains(&order.key)); - orphaned.extend(orphaned_fetch_keys); } + self.drop_departed_source_from_parked(source, &orphaned_fetch_keys); + orphaned.extend(orphaned_fetch_keys); for entry in self.in_flight_fetch.values_mut() { entry.all_sources.retain(|peer| peer != source); @@ -1114,7 +1691,7 @@ impl ReplicationQueues { &mut self, key: XorName, distance: XorName, - sources: Vec, + mut sources: Vec, retry_verification: Option, ) -> bool { if self.pending_verify.contains_key(&key) @@ -1130,6 +1707,7 @@ impl ReplicationQueues { ); return false; } + sources.truncate(self.max_fetch_sources); self.fetch_payloads.insert( key, FetchPayload { @@ -1138,6 +1716,10 @@ impl ReplicationQueues { }, ); self.fetch_queue.push(FetchOrder { key, distance }); + // A fresh enqueue starts a new verification cycle's fetch, so nothing + // asked in an earlier cycle is excluded. Cleared only now, after the + // insert succeeded: a refused enqueue leaves an existing entry alone. + self.fetch_tried.remove(&key); true } @@ -1184,39 +1766,84 @@ impl ReplicationQueues { enqueued } - /// Dequeue the nearest fetch candidate. + /// Dequeue the nearest fetch candidate, whether it is on the fetch heap or + /// parked. /// /// Returns `None` when the queue is empty. Silently skips candidates /// that are somehow already in-flight. Concurrency is enforced by the /// fetch worker, not by this method. /// /// A returned candidate may carry a live verification retry-slot - /// reservation. Callers must consume it with - /// [`Self::start_dequeued_fetch`], [`Self::discard_fetch_candidate`], or + /// reservation, and the sources it was already fetched from this cycle. + /// Callers must consume it with [`Self::start_dequeued_fetch`], + /// [`Self::discard_fetch_candidate`], or /// [`Self::requeue_candidate_for_verification`] so that reservation is /// either transferred, released, or restored to `pending_verify`. pub fn dequeue_fetch(&mut self) -> Option { - while let Some(order) = self.fetch_queue.pop() { - let Some(payload) = self.fetch_payloads.remove(&order.key) else { - debug_assert!( - false, - "fetch order without payload for {}", - hex::encode(order.key) - ); - continue; - }; - let candidate = FetchCandidate { - key: order.key, - distance: order.distance, - sources: payload.sources, - retry_verification: payload.retry_verification, + loop { + let parked = self.nearest_parked(None); + let candidate = match self.take_queued(self.parked_if_nearer(parked)) { + TakenFetch::Candidate(candidate) => candidate, + TakenFetch::Skipped => continue, + TakenFetch::Empty => return None, }; if !self.in_flight_fetch.contains_key(&candidate.key) { return Some(candidate); } - self.release_retry_slot_for_candidate(&candidate); + self.drop_duplicate_of_in_flight(&candidate); } - None + } + + /// `parked`'s key when it is nearer than the heap head, or the heap is + /// empty; `None` to take the heap head. + fn parked_if_nearer(&self, parked: Option<(XorName, XorName)>) -> Option { + let heap_head = self + .fetch_queue + .peek() + .map(|order| (order.distance, order.key)); + match (heap_head, parked) { + (Some(head), Some(parked)) if parked < head => Some(parked.1), + (None, Some(parked)) => Some(parked.1), + _ => None, + } + } + + /// Remove a queued key from where it waits, the parked key `parked` or + /// otherwise the heap head, and rejoin it with its payload. + fn take_queued(&mut self, parked: Option) -> TakenFetch { + let (key, distance) = if let Some(key) = parked { + let Some(distance) = self.unpark(&key) else { + self.purge_parked_index_entry(&key); + return TakenFetch::Skipped; + }; + (key, distance) + } else { + let Some(order) = self.fetch_queue.pop() else { + return TakenFetch::Empty; + }; + (order.key, order.distance) + }; + let Some(payload) = self.fetch_payloads.remove(&key) else { + debug_assert!( + false, + "queued fetch without payload for {}", + hex::encode(key) + ); + return TakenFetch::Skipped; + }; + TakenFetch::Candidate(FetchCandidate { + key, + distance, + sources: payload.sources, + retry_verification: payload.retry_verification, + }) + } + + /// Discard a dequeued candidate whose key is somehow already in flight, + /// releasing what it carried. + fn drop_duplicate_of_in_flight(&mut self, candidate: &FetchCandidate) { + self.release_retry_slot_for_candidate(candidate); + self.fetch_tried.remove(&candidate.key); } /// Number of keys waiting in the fetch queue. @@ -1253,8 +1880,41 @@ impl ReplicationQueues { all_sources: Vec, retry_verification: Option, ) { - let mut tried = HashSet::new(); + self.start_fetch_after(key, source, all_sources, retry_verification, HashSet::new()); + } + + /// Start `key` from `source`, carrying the sources it was already fetched + /// from this cycle into the in-flight entry's tried set. + fn start_fetch_after( + &mut self, + key: XorName, + source: PeerId, + mut all_sources: Vec, + retry_verification: Option, + mut tried: HashSet, + ) { tried.insert(source); + // Keep the distinct union of tried and listed sources within the + // limit by dropping untried sources from the end of the list. Every + // dispatched start is already within it; this binds direct starts. + let union = all_sources.len() + + tried + .iter() + .filter(|peer| !all_sources.contains(peer)) + .count(); + let mut excess = union.saturating_sub(self.max_fetch_sources); + if excess > 0 { + let mut kept = Vec::with_capacity(all_sources.len()); + for peer in all_sources.into_iter().rev() { + if excess > 0 && !tried.contains(&peer) { + excess -= 1; + } else { + kept.push(peer); + } + } + kept.reverse(); + all_sources = kept; + } let replaced = self.in_flight_fetch.insert( key, InFlightEntry { @@ -1271,8 +1931,9 @@ impl ReplicationQueues { } } - /// Consume a dequeued fetch candidate and transfer its retry reservation - /// into the in-flight entry. + /// Consume a dequeued fetch candidate and transfer its retry reservation, + /// and the sources it was already fetched from this cycle, into the + /// in-flight entry. pub fn start_dequeued_fetch(&mut self, candidate: FetchCandidate, source: PeerId) { let FetchCandidate { key, @@ -1280,7 +1941,8 @@ impl ReplicationQueues { retry_verification, .. } = candidate; - self.start_fetch_with_retry(key, source, sources, retry_verification); + let tried = self.fetch_tried.remove(&key).unwrap_or_default(); + self.start_fetch_after(key, source, sources, retry_verification, tried); } /// Mark a fetch as completed (success or permanent failure). @@ -1300,6 +1962,7 @@ impl ReplicationQueues { retry_verification, .. } = candidate; + self.fetch_tried.remove(&key); if retry_verification.is_some() { self.release_retry_slot(&key); self.capacity_owner_by_key.remove(&key); @@ -1329,59 +1992,454 @@ impl ReplicationQueues { } } - /// Consume a dequeued candidate and restore its verification entry for a - /// later retry when retry metadata exists. - pub fn requeue_candidate_for_verification( + // ----------------------------------------------------------------------- + // Per-source dispatch + // ----------------------------------------------------------------------- + + /// Start up to `slots` queued fetches without putting more than + /// `per_source_cap` of this node's fetches in flight to any one source + /// that is answering. + /// + /// Each slot goes to the nearest queued key that can start now: one with + /// an untried source below the cap, or an untried silent source. It starts + /// on the first untried source below the cap that is not silent, in its + /// source-preference order, or else on its first untried silent source, + /// past the cap. A heap key that cannot start is parked under each of its + /// untried sources, so it leaves the heap once and a startable key behind + /// it is reached in the same pass; a parked key becomes a candidate again + /// as soon as one of those sources has a free share or falls silent, and + /// competes on distance with the heap. + /// + /// With `per_source_cap = usize::MAX` nothing parks and every key starts on + /// its first untried source that is not silent, in plain nearest-first + /// order. + pub(crate) fn dispatch_fetches( &mut self, - candidate: FetchCandidate, - retry_after: Duration, - ) -> bool { + slots: usize, + per_source_cap: usize, + ) -> FetchDispatch { + let now = Instant::now(); + self.source_health.expire(now); + let mut dispatch = FetchDispatch::default(); + let mut in_flight = self.in_flight_counts_by_source(None); + let mut best_parked = self.nearest_parked(Some(Startable { + in_flight: &in_flight, + per_source_cap, + now, + })); + while dispatch.started.len() < slots { + let from_parked = self.parked_if_nearer(best_parked); + let candidate = match self.take_queued(from_parked) { + TakenFetch::Candidate(candidate) => Some(candidate), + TakenFetch::Skipped => None, + TakenFetch::Empty => break, + }; + let mut started = false; + if let Some(candidate) = candidate { + if self.in_flight_fetch.contains_key(&candidate.key) { + self.drop_duplicate_of_in_flight(&candidate); + } else { + let startable = Startable { + in_flight: &in_flight, + per_source_cap, + now, + }; + match self.first_startable_source(&candidate, startable) { + SourceChoice::Start(source) => { + if self.source_health.is_silent(&source, now) { + self.source_health.counters.silent_starts += 1; + } + *in_flight.entry(source).or_default() += 1; + dispatch.started.push((candidate.key, source)); + self.start_dequeued_fetch(candidate, source); + started = true; + } + SourceChoice::NoSource => dispatch.sourceless.push(candidate), + SourceChoice::AllBusy => { + if from_parked.is_none() { + dispatch.parked += 1; + } + self.park_candidate(candidate); + } + } + } + } + // A start changes the per-source counts, and a key taken from the + // parked set changes the set itself. A heap key that parked is + // indexed only under full sources that are not silent, so it + // cannot change the answer. + if started || from_parked.is_some() { + best_parked = self.nearest_parked(Some(Startable { + in_flight: &in_flight, + per_source_cap, + now, + })); + } + } + dispatch + } + + /// Put a dequeued candidate back in the queue, parked under its untried + /// sources. + fn park_candidate(&mut self, candidate: FetchCandidate) { let FetchCandidate { key, + distance, + sources, retry_verification, - .. } = candidate; - let Some(mut verification) = retry_verification else { - return false; - }; - verification.state = VerificationState::PendingVerify; - verification.verified_sources.clear(); - verification.tried_sources.clear(); - verification.next_verify_at = Instant::now() + retry_after; + self.fetch_payloads.insert( + key, + FetchPayload { + sources, + retry_verification, + }, + ); + self.park(key, distance); + } - let owner = self - .capacity_owner_by_key - .get(&key) + /// Mark the current fetch attempt as failed and move `key` on, without + /// sending to a source that is answering and already has `per_source_cap` + /// of this node's fetches in flight. + /// + /// The next untried source with a free share that is not silent is + /// returned to retry from now, as [`Self::retry_fetch`] would, or else the + /// first untried silent source, past the cap. If every untried source is + /// answering and at its share, the key goes back to the fetch queue + /// parked, with only its untried sources and with its tried set kept, so + /// no source is asked twice this cycle; its retry reservation travels with + /// it. If no untried source is left, or the fetch queue is at + /// [`MAX_FETCH_QUEUE`], the in-flight entry is left for the caller's + /// verification requeue, which keeps `MAX_FETCH_QUEUE` a hard bound. + pub(crate) fn retry_or_park_fetch( + &mut self, + key: &XorName, + distance: XorName, + per_source_cap: usize, + ) -> FetchRetry { + let now = Instant::now(); + self.source_health.expire(now); + let in_flight = self.in_flight_counts_by_source(Some(key)); + let queue_full = self.fetch_payloads.len() >= MAX_FETCH_QUEUE; + let Some(entry) = self.in_flight_fetch.get_mut(key) else { + return FetchRetry::Exhausted; + }; + entry.tried.insert(entry.source); + let untried: Vec = entry + .all_sources + .iter() + .filter(|peer| !entry.tried.contains(peer)) .copied() - .or_else(|| self.least_loaded_source(&verification.hint_sources)); - let Some(owner) = owner else { - self.release_retry_slot(&key); - return false; + .collect(); + if untried.is_empty() { + return FetchRetry::Exhausted; + } + let health = &self.source_health; + let free = untried + .iter() + .find(|peer| { + !health.is_silent(peer, now) + && in_flight.get(peer).copied().unwrap_or(0) < per_source_cap + }) + .copied(); + let silent = || { + untried + .iter() + .find(|peer| health.is_silent(peer, now)) + .copied() }; - // Release before re-inserting: the reservation is charged to the owner - // recorded now, which `insert_pending_owned_unchecked` may replace. - self.release_retry_slot(&key); - self.insert_pending_owned_unchecked(key, verification, owner, true); - true + let next = free.or_else(silent); + if let Some(next) = next { + if free.is_none() { + self.source_health.counters.silent_starts += 1; + } + entry.source = next; + entry.tried.insert(next); + entry.started_at = now; + return FetchRetry::Next(next); + } + if queue_full { + return FetchRetry::Exhausted; + } + let Some(entry) = self.in_flight_fetch.remove(key) else { + return FetchRetry::Exhausted; + }; + self.fetch_payloads.insert( + *key, + FetchPayload { + sources: untried, + retry_verification: entry.retry_verification, + }, + ); + self.fetch_tried.insert(*key, entry.tried); + self.park(*key, distance); + FetchRetry::Parked } - /// Complete an exhausted fetch and restore its verification entry for a - /// later retry when retry metadata exists. - pub fn requeue_fetch_for_verification(&mut self, key: &XorName, retry_after: Duration) -> bool { - let Some(mut entry) = self.in_flight_fetch.remove(key) else { - return false; - }; - let Some(mut verification) = entry.retry_verification.take() else { - self.capacity_owner_by_key.remove(key); - return false; - }; - verification.state = VerificationState::PendingVerify; - verification.verified_sources.clear(); - verification.tried_sources.clear(); - verification.next_verify_at = Instant::now() + retry_after; + /// In-flight fetches per source, optionally leaving out `exclude`'s own + /// entry. + fn in_flight_counts_by_source(&self, exclude: Option<&XorName>) -> HashMap { + let mut counts = HashMap::new(); + for (key, entry) in &self.in_flight_fetch { + if exclude != Some(key) { + *counts.entry(entry.source).or_default() += 1; + } + } + counts + } - let owner = self - .capacity_owner_by_key + /// The first untried source of `candidate`, in its source-preference + /// order, that is not silent and is below the cap, or else its first + /// untried silent source. + fn first_startable_source( + &self, + candidate: &FetchCandidate, + startable: Startable<'_>, + ) -> SourceChoice { + let tried = self.fetch_tried.get(&candidate.key); + let mut any_untried = false; + let mut first_silent = None; + for source in &candidate.sources { + if tried.is_some_and(|t| t.contains(source)) { + continue; + } + any_untried = true; + if self.source_health.is_silent(source, startable.now) { + first_silent = first_silent.or(Some(*source)); + } else if startable.in_flight.get(source).copied().unwrap_or(0) + < startable.per_source_cap + { + return SourceChoice::Start(*source); + } + } + match (first_silent, any_untried) { + (Some(source), _) => SourceChoice::Start(source), + (None, true) => SourceChoice::AllBusy, + (None, false) => SourceChoice::NoSource, + } + } + + /// Park a queued key whose payload is in `fetch_payloads`, indexed under + /// each of its untried sources. A key with no untried source goes back on + /// the heap instead, so it is never parked with nothing to wake it. + fn park(&mut self, key: XorName, distance: XorName) { + self.unpark(&key); + let tried = self.fetch_tried.get(&key); + let wake_sources: Vec = self + .fetch_payloads + .get(&key) + .map(|payload| { + payload + .sources + .iter() + .filter(|source| !tried.is_some_and(|t| t.contains(source))) + .copied() + .collect() + }) + .unwrap_or_default(); + if wake_sources.is_empty() { + self.fetch_queue.push(FetchOrder { key, distance }); + return; + } + for source in &wake_sources { + self.parked_by_source + .entry(*source) + .or_default() + .insert((distance, key)); + } + self.fetch_parked.insert( + key, + ParkedFetch { + distance, + wake_sources, + parked_at: Instant::now(), + }, + ); + } + + /// The per-source scheduler's state for its periodic summary, resetting + /// the maintenance maximum and source-health counters it reports and + /// purging the health table. Walks the parked set, the index's per-source + /// sets, the health table and the in-flight fetches once each. + pub(crate) fn take_fetch_dispatch_stats( + &mut self, + per_source_cap: usize, + ) -> FetchDispatchStats { + let now = Instant::now(); + let parked_maintenance_max = std::mem::take(&mut self.parked_maintenance_max); + let (health, tracked_sources) = + self.source_health.take_counters(now, &self.in_flight_fetch); + FetchDispatchStats { + tracked_sources, + silent_sources: self.source_health.marks_by_since.len(), + overflow_active: self.source_health.overflowing(now), + health, + parked_associations: self.parked_by_source.values().map(BTreeSet::len).sum(), + parked_maintenance_max, + parked: self.fetch_parked.len(), + oldest_parked: self + .fetch_parked + .values() + .map(|parked| now.saturating_duration_since(parked.parked_at)) + .max(), + queued: self.fetch_payloads.len(), + in_flight: self.in_flight_fetch.len(), + saturated_sources: self + .in_flight_counts_by_source(None) + .values() + .filter(|count| **count >= per_source_cap) + .count(), + } + } + + /// The parked-state part of [`Self::remove_hint_source`]: orphaned keys + /// leave it entirely, and a parked key can no longer start from the + /// departed source. One left with no source to wake it goes back on the + /// heap, where dispatch hands it out as sourceless, exactly as a queued key + /// that lost its last source. Timed for the dispatch summary. + fn drop_departed_source_from_parked( + &mut self, + source: &PeerId, + orphaned_fetch_keys: &HashSet, + ) { + if self.fetch_parked.is_empty() && self.fetch_tried.is_empty() { + return; + } + let started = Instant::now(); + for key in orphaned_fetch_keys { + self.unpark(key); + self.fetch_tried.remove(key); + } + if let Some(parked_here) = self.parked_by_source.remove(source) { + for (distance, key) in parked_here { + let Some(parked) = self.fetch_parked.get_mut(&key) else { + continue; + }; + parked.wake_sources.retain(|peer| peer != source); + if parked.wake_sources.is_empty() { + self.fetch_parked.remove(&key); + self.fetch_queue.push(FetchOrder { key, distance }); + } + } + } + self.parked_maintenance_max = self.parked_maintenance_max.max(started.elapsed()); + } + + /// Take `key` out of the parked state and the index, returning its + /// distance. The payload is left in `fetch_payloads`. + fn unpark(&mut self, key: &XorName) -> Option { + let parked = self.fetch_parked.remove(key)?; + for source in &parked.wake_sources { + if let Some(keys) = self.parked_by_source.get_mut(source) { + keys.remove(&(parked.distance, *key)); + if keys.is_empty() { + self.parked_by_source.remove(source); + } + } + } + Some(parked.distance) + } + + /// The nearest parked key, as `(distance, key)`. With `startable`, only + /// keys indexed under a source below the cap or silent count, so the + /// result is the nearest parked key that could start now. + fn nearest_parked(&self, startable: Option>) -> Option<(XorName, XorName)> { + self.parked_by_source + .iter() + .filter(|(source, _)| { + startable.is_none_or(|startable| { + startable.in_flight.get(source).copied().unwrap_or(0) < startable.per_source_cap + || self.source_health.is_silent(source, startable.now) + }) + }) + .filter_map(|(_, keys)| keys.first().copied()) + .min() + } + + /// Defensive: drop every index entry for `key` when the index names a key + /// that is not parked. Unreachable while the index stays exact; without it + /// such an entry would be offered forever. + fn purge_parked_index_entry(&mut self, key: &XorName) { + debug_assert!( + false, + "parked index names an unparked key {}", + hex::encode(key) + ); + self.parked_by_source.retain(|_, keys| { + keys.retain(|(_, parked)| parked != key); + !keys.is_empty() + }); + } + + /// Test-only: every queued or parked key, and every in-flight fetch with + /// its source, read together. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn fetch_pipeline_snapshot(&self) -> FetchPipelineSnapshot { + FetchPipelineSnapshot { + queued: self.fetch_payloads.keys().copied().collect(), + in_flight: self + .in_flight_fetch + .iter() + .map(|(key, entry)| (*key, entry.source)) + .collect(), + } + } + + /// Consume a dequeued candidate and restore its verification entry for a + /// later retry when retry metadata exists. + pub fn requeue_candidate_for_verification( + &mut self, + candidate: FetchCandidate, + retry_after: Duration, + ) -> bool { + let FetchCandidate { + key, + retry_verification, + .. + } = candidate; + self.fetch_tried.remove(&key); + let Some(mut verification) = retry_verification else { + return false; + }; + verification.state = VerificationState::PendingVerify; + verification.verified_sources.clear(); + verification.tried_sources.clear(); + verification.next_verify_at = Instant::now() + retry_after; + + let owner = self + .capacity_owner_by_key + .get(&key) + .copied() + .or_else(|| self.least_loaded_source(&verification.hint_sources)); + let Some(owner) = owner else { + self.release_retry_slot(&key); + return false; + }; + // Release before re-inserting: the reservation is charged to the owner + // recorded now, which `insert_pending_owned_unchecked` may replace. + self.release_retry_slot(&key); + self.insert_pending_owned_unchecked(key, verification, owner, true); + true + } + + /// Complete an exhausted fetch and restore its verification entry for a + /// later retry when retry metadata exists. + pub fn requeue_fetch_for_verification(&mut self, key: &XorName, retry_after: Duration) -> bool { + let Some(mut entry) = self.in_flight_fetch.remove(key) else { + return false; + }; + let Some(mut verification) = entry.retry_verification.take() else { + self.capacity_owner_by_key.remove(key); + return false; + }; + verification.state = VerificationState::PendingVerify; + verification.verified_sources.clear(); + verification.tried_sources.clear(); + verification.next_verify_at = Instant::now() + retry_after; + + let owner = self + .capacity_owner_by_key .get(key) .copied() .or_else(|| self.least_loaded_source(&verification.hint_sources)); @@ -1396,6 +2454,11 @@ impl ReplicationQueues { true } + /// When `key`'s current fetch attempt started, if it is in flight. + pub(crate) fn in_flight_started_at(&self, key: &XorName) -> Option { + self.in_flight_fetch.get(key).map(|entry| entry.started_at) + } + /// Number of in-flight fetches. #[must_use] pub fn in_flight_count(&self) -> usize { @@ -1516,6 +2579,8 @@ fn max_min_allocations( mod tests { use super::*; use crate::replication::config::CAPACITY_BLOCKED_RETRY; + use rand::rngs::StdRng; + use rand::{Rng, SeedableRng}; /// Build a `PeerId` from a single byte (zero-padded to 32 bytes). fn peer_id_from_byte(b: u8) -> PeerId { @@ -3026,4 +4091,1493 @@ mod tests { ); assert_eq!(queues.pending_count(), 1); } + + // -- per-source fetch dispatch --------------------------------------------- + + const CAP: usize = 2; + + fn peer(b: u8) -> PeerId { + peer_id_from_byte(b) + } + + /// A key whose byte order matches the numeric order of `distance`. + fn key_at(distance: u32) -> XorName { + let mut name = [0u8; 32]; + name[..4].copy_from_slice(&distance.to_be_bytes()); + name + } + + /// Queue `key` at `distance` with `sources`, keyed at its own distance so + /// tests can reason about nearest-first order directly. + fn queue(queues: &mut ReplicationQueues, distance: u32, sources: &[PeerId]) -> XorName { + let key = key_at(distance); + assert!(queues.enqueue_fetch(key, key, sources.to_vec())); + key + } + + /// Queue `key` through pending verification and promotion, so it carries a + /// verification retry reservation owned by `hinter`. + fn queue_promoted( + queues: &mut ReplicationQueues, + distance: u32, + hinter: u8, + sources: &[PeerId], + ) -> XorName { + let key = key_at(distance); + assert!(queues + .add_pending_verify(key, test_entry(hinter)) + .admitted()); + assert!(queues.promote_pending_to_fetch(key, key, sources.to_vec())); + key + } + + fn in_flight_to(queues: &ReplicationQueues, source: &PeerId) -> usize { + queues + .in_flight_fetch + .values() + .filter(|entry| entry.source == *source) + .count() + } + + fn heap_keys(queues: &ReplicationQueues) -> HashSet { + queues.fetch_queue.iter().map(|order| order.key).collect() + } + + /// I1 and I2: every queued payload is on the heap or parked, never both, + /// and the reverse index is exactly what `fetch_parked` implies. + fn assert_queue_invariants(queues: &ReplicationQueues) { + let heap = heap_keys(queues); + assert_eq!(heap.len(), queues.fetch_queue.len(), "duplicate heap entry"); + for key in queues.fetch_payloads.keys() { + let on_heap = heap.contains(key); + let parked = queues.fetch_parked.contains_key(key); + assert!( + on_heap != parked, + "queued key must be on exactly one of heap/parked" + ); + } + for key in heap.iter().chain(queues.fetch_parked.keys()) { + assert!( + queues.fetch_payloads.contains_key(key), + "order without payload" + ); + } + let mut expected: HashMap> = HashMap::new(); + for (key, parked) in &queues.fetch_parked { + assert!( + !parked.wake_sources.is_empty(), + "parked key with nothing to wake it" + ); + for source in &parked.wake_sources { + expected + .entry(*source) + .or_default() + .insert((parked.distance, *key)); + } + } + assert_eq!( + queues.parked_by_source, expected, + "parked index is not exact" + ); + for key in queues.fetch_tried.keys() { + assert!( + queues.fetch_payloads.contains_key(key), + "tried sources kept for a key that is not queued" + ); + } + } + + #[test] + fn dispatch_spreads_fetches_across_sources_within_the_cap() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let keys: Vec = (0..8).map(|d| queue(&mut queues, d, &[a, b, c])).collect(); + + let dispatch = queues.dispatch_fetches(8, CAP); + + assert_eq!(dispatch.started.len(), 6, "two per source, three sources"); + assert!(dispatch.sourceless.is_empty()); + for source in [a, b, c] { + assert_eq!(in_flight_to(&queues, &source), CAP); + } + // Preference order is kept: the nearest keys take the first source. + let expected: Vec<(XorName, PeerId)> = keys + .iter() + .take(6) + .zip([a, a, b, b, c, c]) + .map(|(key, source)| (*key, source)) + .collect(); + assert_eq!(dispatch.started, expected); + assert_eq!(queues.fetch_parked.len(), 2); + assert_queue_invariants(&queues); + } + + #[test] + fn uncapped_dispatch_keeps_the_first_source_order() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + for d in 0..5 { + queue(&mut queues, d, &[a, b]); + } + + let dispatch = queues.dispatch_fetches(8, usize::MAX); + + assert_eq!(dispatch.started.len(), 5); + assert!(dispatch.started.iter().all(|(_, source)| *source == a)); + assert!(queues.fetch_parked.is_empty()); + assert_queue_invariants(&queues); + } + + #[test] + fn a_startable_key_behind_blocked_keys_starts_in_the_same_pass() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + queue(&mut queues, 0, &[a]); + queue(&mut queues, 1, &[a]); + assert_eq!(queues.dispatch_fetches(8, CAP).started.len(), 2); + for d in 10..310 { + queue(&mut queues, d, &[a]); + } + let deeper = queue(&mut queues, 1_000, &[b]); + + let dispatch = queues.dispatch_fetches(8, CAP); + + assert_eq!(dispatch.started, vec![(deeper, b)]); + assert_eq!( + queues.fetch_parked.len(), + 300, + "every blocked key left the heap" + ); + assert!(queues.fetch_queue.is_empty()); + assert_eq!( + queues.fetch_queue_count(), + 300, + "parked keys are still queued" + ); + assert_queue_invariants(&queues); + } + + #[test] + fn a_freed_share_starts_the_nearest_parked_key_for_that_source() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + let first = queue(&mut queues, 0, &[a]); + queue(&mut queues, 1, &[a]); + for d in [40, 20, 30] { + queue(&mut queues, d, &[a]); + } + assert_eq!(queues.dispatch_fetches(8, CAP).started.len(), 2); + assert_eq!(queues.fetch_parked.len(), 3); + + queues.complete_fetch(&first); + let dispatch = queues.dispatch_fetches(8, CAP); + + assert_eq!(dispatch.started, vec![(key_at(20), a)]); + assert_eq!(in_flight_to(&queues, &a), CAP); + assert_queue_invariants(&queues); + } + + #[test] + fn the_nearest_startable_parked_key_wins_whatever_its_source() { + let mut queues = ReplicationQueues::new(); + let sources: Vec = (1..=5).map(peer).collect(); + // Saturate every source with two blockers, then park one key per source. + let mut blockers = Vec::new(); + for (index, source) in sources.iter().enumerate() { + let base = u32::try_from(index).unwrap() * 2; + blockers.push(queue(&mut queues, base, &[*source])); + blockers.push(queue(&mut queues, base + 1, &[*source])); + } + assert_eq!(queues.dispatch_fetches(10, CAP).started.len(), 10); + let parked_distances = [500, 100, 400, 200, 300]; + for (source, distance) in sources.iter().zip(parked_distances) { + queue(&mut queues, distance, &[*source]); + } + assert_eq!(queues.dispatch_fetches(5, CAP).started, Vec::new()); + assert_eq!(queues.fetch_parked.len(), 5); + + // Free one share on every source: the two slots go to the two nearest. + for pair in blockers.chunks(2) { + queues.complete_fetch(&pair[0]); + } + let dispatch = queues.dispatch_fetches(2, CAP); + + assert_eq!( + dispatch.started, + vec![(key_at(100), sources[1]), (key_at(200), sources[3])] + ); + assert_queue_invariants(&queues); + } + + #[test] + fn a_parked_key_starts_on_its_preferred_free_source() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + let blockers: Vec = (0..4) + .map(|d| queue(&mut queues, d, &[if d < 2 { a } else { b }])) + .collect(); + assert_eq!(queues.dispatch_fetches(4, CAP).started.len(), 4); + let key = queue(&mut queues, 50, &[a, b]); + assert_eq!(queues.dispatch_fetches(1, CAP).started, Vec::new()); + + // Free b first, then a: both have room when dispatch runs. + queues.complete_fetch(&blockers[2]); + queues.complete_fetch(&blockers[0]); + let dispatch = queues.dispatch_fetches(1, CAP); + + assert_eq!(dispatch.started, vec![(key, a)]); + assert_queue_invariants(&queues); + } + + #[test] + fn retry_moves_to_a_free_untried_source() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + queue(&mut queues, 10, &[b]); + queue(&mut queues, 11, &[b]); + let key = queue(&mut queues, 0, &[a, b, c]); + assert_eq!(queues.dispatch_fetches(3, CAP).started.len(), 3); + assert_eq!(queues.in_flight_fetch[&key].source, a); + + let retry = queues.retry_or_park_fetch(&key, key, CAP); + + assert_eq!(retry, FetchRetry::Next(c), "b is at its share"); + let entry = &queues.in_flight_fetch[&key]; + assert_eq!(entry.source, c); + assert_eq!(entry.tried, HashSet::from([a, c])); + assert_queue_invariants(&queues); + } + + #[test] + fn retry_with_every_untried_source_full_parks_and_keeps_its_history() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = queue_promoted(&mut queues, 0, 9, &[a, b, c]); + let blockers: Vec = (10..14) + .map(|d| queue(&mut queues, d, &[if d < 12 { b } else { c }])) + .collect(); + assert_eq!(queues.dispatch_fetches(5, CAP).started.len(), 5); + assert_eq!(queues.in_flight_fetch[&key].source, a); + let reserved = queues.retry_reserved_slot_count(); + let owner = queues.capacity_owner_by_key.get(&key).copied(); + let reserved_by_owner = queues.retry_reserved_by_owner.clone(); + assert_eq!(reserved_by_owner.get(&peer(9)), Some(&1)); + + let retry = queues.retry_or_park_fetch(&key, key, CAP); + + assert_eq!(retry, FetchRetry::Parked); + assert!(!queues.in_flight_fetch.contains_key(&key)); + assert_eq!(queues.fetch_payloads[&key].sources, vec![b, c]); + assert!(queues.fetch_payloads[&key].retry_verification.is_some()); + assert_eq!(queues.fetch_tried[&key], HashSet::from([a])); + assert_eq!(queues.fetch_parked[&key].wake_sources, vec![b, c]); + assert_eq!(queues.retry_reserved_slot_count(), reserved); + assert_eq!(queues.capacity_owner_by_key.get(&key).copied(), owner); + assert_eq!(queues.retry_reserved_by_owner, reserved_by_owner); + assert!(queues.contains_key(&key)); + assert_queue_invariants(&queues); + + // A share on b frees: the key restarts there, still remembering a. + queues.complete_fetch(&blockers[0]); + let dispatch = queues.dispatch_fetches(1, CAP); + assert_eq!(dispatch.started, vec![(key, b)]); + assert_eq!(queues.in_flight_fetch[&key].tried, HashSet::from([a, b])); + assert!(!queues.fetch_tried.contains_key(&key)); + assert_eq!(queues.retry_reserved_slot_count(), reserved); + assert_eq!(queues.capacity_owner_by_key.get(&key).copied(), owner); + assert_eq!(queues.retry_reserved_by_owner, reserved_by_owner); + assert_queue_invariants(&queues); + } + + #[test] + fn a_parked_key_wakes_through_any_of_its_sources_beyond_the_eighth() { + let mut queues = ReplicationQueues::new(); + let sources: Vec = (1..=9).map(peer).collect(); + let mut blockers = Vec::new(); + for (index, source) in sources.iter().enumerate() { + let base = u32::try_from(index).unwrap() * 2; + blockers.push(queue(&mut queues, base, &[*source])); + blockers.push(queue(&mut queues, base + 1, &[*source])); + } + assert_eq!(queues.dispatch_fetches(18, CAP).started.len(), 18); + let key = queue(&mut queues, 100, &sources); + assert_eq!(queues.dispatch_fetches(1, CAP).started, Vec::new()); + assert_eq!(queues.fetch_parked[&key].wake_sources, sources); + + // Only the ninth source frees a share. + queues.complete_fetch(&blockers[16]); + let dispatch = queues.dispatch_fetches(1, CAP); + + assert_eq!(dispatch.started, vec![(key, sources[8])]); + assert_queue_invariants(&queues); + } + + #[test] + fn retry_with_no_untried_source_is_exhausted_and_left_in_flight() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + let key = queue(&mut queues, 0, &[a]); + assert_eq!(queues.dispatch_fetches(1, CAP).started.len(), 1); + + assert_eq!( + queues.retry_or_park_fetch(&key, key, CAP), + FetchRetry::Exhausted + ); + assert!(queues.in_flight_fetch.contains_key(&key)); + assert_queue_invariants(&queues); + } + + #[test] + fn retry_with_a_full_fetch_queue_is_exhausted_rather_than_overfilling_it() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + let key = queue(&mut queues, 0, &[a, b]); + queue(&mut queues, 1, &[b]); + queue(&mut queues, 2, &[b]); + assert_eq!(queues.dispatch_fetches(3, CAP).started.len(), 3); + let mut filler = 1_000_u32; + while queues.fetch_queue_count() < MAX_FETCH_QUEUE { + queue(&mut queues, filler, &[peer(7)]); + filler += 1; + } + + assert_eq!( + queues.retry_or_park_fetch(&key, key, CAP), + FetchRetry::Exhausted + ); + assert!(queues.in_flight_fetch.contains_key(&key)); + assert_eq!(queues.fetch_queue_count(), MAX_FETCH_QUEUE); + } + + /// Park `key` (sources a, b, c) with a already tried and b, c full. + fn park_after_trying_a( + queues: &mut ReplicationQueues, + a: PeerId, + b: PeerId, + c: PeerId, + ) -> XorName { + let key = queue(queues, 0, &[a, b, c]); + for d in 10..14 { + queue(queues, d, &[if d < 12 { b } else { c }]); + } + assert_eq!(queues.dispatch_fetches(5, CAP).started.len(), 5); + assert_eq!( + queues.retry_or_park_fetch(&key, key, CAP), + FetchRetry::Parked + ); + key + } + + #[test] + fn a_tried_source_readvertised_for_a_parked_key_is_not_merged_back() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + + let result = queues.add_pending_verify(key, test_entry(1)); + + assert_eq!(result, AdmissionResult::AlreadyPresent); + assert_eq!(queues.fetch_payloads[&key].sources, vec![b, c]); + assert_eq!(queues.fetch_parked[&key].wake_sources, vec![b, c]); + assert_queue_invariants(&queues); + } + + #[test] + fn a_new_source_merged_into_a_parked_key_can_start_it() { + let mut queues = ReplicationQueues::new(); + let (a, b, c, d) = (peer(1), peer(2), peer(3), peer(4)); + let key = park_after_trying_a(&mut queues, a, b, c); + + assert_eq!( + queues.add_pending_verify(key, test_entry(4)), + AdmissionResult::AlreadyPresent + ); + assert_eq!(queues.fetch_parked[&key].wake_sources, vec![b, c, d]); + assert_queue_invariants(&queues); + + let dispatch = queues.dispatch_fetches(1, CAP); + assert_eq!(dispatch.started, vec![(key, d)]); + assert_eq!(queues.in_flight_fetch[&key].tried, HashSet::from([a, d])); + assert_queue_invariants(&queues); + } + + #[test] + fn an_in_flight_merge_skips_tried_sources_and_respects_the_limit() { + let mut queues = ReplicationQueues::new().with_fetch_source_limit(4); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + // Restart it on b, carrying a as tried: associated sources are a, b, c. + let blocker = queues + .in_flight_fetch + .iter() + .find(|(_, entry)| entry.source == b) + .map(|(key, _)| *key) + .unwrap(); + queues.complete_fetch(&blocker); + assert_eq!(queues.dispatch_fetches(1, CAP).started, vec![(key, b)]); + + // a was tried: not merged. 5 is new and fits (4 associated). 6 does not. + for hinter in [1, 5, 6] { + queues.add_pending_verify(key, test_entry(hinter)); + } + + assert_eq!( + queues.in_flight_fetch[&key].all_sources, + vec![b, c, peer(5)] + ); + assert_queue_invariants(&queues); + } + + /// A re-advertisement from `sender` that claims no replica. + fn paid_only_entry(sender: u8) -> VerificationEntry { + let mut entry = test_entry(sender); + entry.replica_hint_sources.clear(); + entry + } + + /// Every fetch-stage key's retry entry is within the hint bound, as a + /// pending entry is, and still holds the key's capacity owner. + fn assert_retry_entries_bounded(queues: &ReplicationQueues) { + let retries = queues + .fetch_payloads + .iter() + .filter_map(|(key, payload)| Some((key, payload.retry_verification.as_ref()?))) + .chain( + queues + .in_flight_fetch + .iter() + .filter_map(|(key, entry)| Some((key, entry.retry_verification.as_ref()?))), + ); + for (key, retry) in retries { + assert!( + retry.hint_sources.len() <= MAX_HINT_SOURCES_PER_KEY, + "{} retry entry over the hint bound", + hex::encode(key) + ); + assert!(retry.replica_hint_sources.is_subset(&retry.hint_sources)); + if let Some(owner) = queues.capacity_owner_by_key.get(key) { + assert!( + retry.hint_sources.contains(owner), + "{} retry entry lost its capacity owner", + hex::encode(key) + ); + } + } + } + + #[test] + fn readvertisements_keep_a_fetch_stage_retry_entry_within_the_hint_bound() { + let mut queues = ReplicationQueues::new(); + let (owner, holder) = (peer(1), peer(2)); + let key = key_at(0); + // The owner advertised the key without claiming a replica, so only + // the capacity accounting keeps it in the set. + assert!(queues + .add_pending_verify(key, paid_only_entry(1)) + .admitted()); + assert!(queues.promote_pending_to_fetch(key, key, vec![holder])); + + // Queued: paid-only advertisers fill the set to the bound, then replica + // claimants displace every one of them except the owner. + for sender in 20..40 { + queues.add_pending_verify(key, paid_only_entry(sender)); + } + for sender in 40..60 { + queues.add_pending_verify(key, test_entry(sender)); + } + assert_retry_entries_bounded(&queues); + let retry_sizes = queues + .fetch_payloads + .get(&key) + .and_then(|payload| payload.retry_verification.as_ref()) + .map(|retry| (retry.hint_sources.len(), retry.replica_hint_sources.len())); + assert_eq!( + retry_sizes, + Some((MAX_HINT_SOURCES_PER_KEY, MAX_HINT_SOURCES_PER_KEY - 1)), + "a full set keeps the owner and fills the rest with replica claimants" + ); + + // In flight: the same bound. + assert_eq!(queues.dispatch_fetches(1, CAP).started, vec![(key, holder)]); + for sender in 60..80 { + queues.add_pending_verify(key, test_entry(sender)); + } + assert_retry_entries_bounded(&queues); + + // Back in pending verification, the restored entry keeps the bound and + // the owner its reservation was charged to. + assert!(queues.requeue_fetch_for_verification(&key, Duration::ZERO)); + let pending = queues.get_pending(&key).map(|entry| { + ( + entry.hint_sources.len(), + entry.hint_sources.contains(&owner), + ) + }); + assert_eq!(pending, Some((MAX_HINT_SOURCES_PER_KEY, true))); + assert_eq!(queues.capacity_owner_by_key.get(&key), Some(&owner)); + } + + #[test] + fn source_lists_are_capped_at_the_limit_including_tried_sources() { + let mut queues = ReplicationQueues::new().with_fetch_source_limit(3); + let sources: Vec = (1..=6).map(peer).collect(); + let key = queue(&mut queues, 0, &sources); + assert_eq!(queues.fetch_payloads[&key].sources, sources[..3].to_vec()); + + // Park it with one tried source: payload b, c plus tried a is the limit. + let mut queues = ReplicationQueues::new().with_fetch_source_limit(3); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + queues.add_pending_verify(key, test_entry(4)); + assert_eq!(queues.fetch_payloads[&key].sources, vec![b, c]); + assert_queue_invariants(&queues); + + // A direct start with an oversized list is cut to the limit too. + let mut queues = ReplicationQueues::new().with_fetch_source_limit(2); + let direct = key_at(9); + queues.start_fetch(direct, sources[0], sources.clone()); + assert_eq!( + queues.in_flight_fetch[&direct].all_sources, + sources[..2].to_vec() + ); + + // A start from a source outside the list counts that source too. + let mut queues = ReplicationQueues::new().with_fetch_source_limit(3); + let outside = peer(9); + queues.start_fetch(direct, outside, sources[..4].to_vec()); + let entry = &queues.in_flight_fetch[&direct]; + assert_eq!(entry.all_sources, sources[..2].to_vec()); + let union: HashSet<&PeerId> = entry.all_sources.iter().chain(&entry.tried).collect(); + assert_eq!(union.len(), 3); + } + + #[test] + fn a_parked_key_that_loses_its_last_wake_source_returns_to_the_heap() { + let mut queues = ReplicationQueues::new(); + let (b, c) = (peer(2), peer(3)); + queue(&mut queues, 0, &[b]); + queue(&mut queues, 1, &[b]); + queue(&mut queues, 2, &[c]); + // Hinted by 9, so the retry metadata survives b leaving. + let key = queue_promoted(&mut queues, 10, 9, &[b]); + let started = queues.dispatch_fetches(4, CAP).started; + assert!(started.iter().all(|(started_key, _)| *started_key != key)); + assert!(queues.fetch_parked.contains_key(&key)); + + let orphaned = queues.remove_hint_source(&b); + + assert!(!orphaned.contains(&key)); + assert!(!queues.fetch_parked.contains_key(&key)); + assert!(heap_keys(&queues).contains(&key)); + assert_queue_invariants(&queues); + let dispatch = queues.dispatch_fetches(4, CAP); + assert!(dispatch + .sourceless + .iter() + .any(|candidate| candidate.key == key)); + } + + #[test] + fn an_orphaned_parked_key_leaves_every_structure() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + queues.remove_hint_source(&b); + assert!( + queues.fetch_parked.contains_key(&key), + "c can still wake it" + ); + + let orphaned = queues.remove_hint_source(&c); + + assert!(orphaned.contains(&key)); + assert!(!queues.fetch_payloads.contains_key(&key)); + assert!(!queues.fetch_parked.contains_key(&key)); + assert!(!queues.fetch_tried.contains_key(&key)); + assert!(!queues.contains_key(&key)); + assert_queue_invariants(&queues); + } + + #[test] + fn tried_sources_survive_dequeue_and_move_into_the_next_start() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + + let candidate = queues.dequeue_fetch().unwrap(); + assert_eq!( + candidate.key, key, + "the parked key is the nearest queued key" + ); + assert!(queues.fetch_tried.contains_key(&key)); + queues.start_dequeued_fetch(candidate, c); + + assert_eq!(queues.in_flight_fetch[&key].tried, HashSet::from([a, c])); + assert!(!queues.fetch_tried.contains_key(&key)); + assert_queue_invariants(&queues); + } + + #[test] + fn tried_sources_are_dropped_with_a_discarded_or_requeued_candidate() { + for requeue in [false, true] { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + let candidate = queues.dequeue_fetch().unwrap(); + assert_eq!(candidate.key, key); + if requeue { + queues.requeue_candidate_for_verification(candidate, Duration::from_secs(1)); + } else { + queues.discard_fetch_candidate(candidate); + } + assert!(!queues.fetch_tried.contains_key(&key)); + assert_queue_invariants(&queues); + } + } + + #[test] + fn tried_sources_are_kept_by_a_refused_enqueue_and_cleared_by_a_fresh_one() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + assert!(!queues.enqueue_fetch(key, key, vec![a]), "already queued"); + assert_eq!(queues.fetch_tried[&key], HashSet::from([a])); + + let stray = key_at(77); + queues.fetch_tried.insert(stray, HashSet::from([a])); + assert!(queues.enqueue_fetch(stray, stray, vec![a])); + assert!(!queues.fetch_tried.contains_key(&stray)); + } + + #[test] + fn a_duplicate_of_an_in_flight_key_is_dropped_with_its_tried_sources() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + let key = key_at(3); + queues.start_fetch(key, a, vec![a]); + // Force the impossible state the defensive skip exists for. + queues.fetch_payloads.insert( + key, + FetchPayload { + sources: vec![a], + retry_verification: None, + }, + ); + queues.fetch_queue.push(FetchOrder { key, distance: key }); + queues.fetch_tried.insert(key, HashSet::from([a])); + + assert!(queues.dequeue_fetch().is_none()); + assert!(!queues.fetch_tried.contains_key(&key)); + assert!(queues.in_flight_fetch.contains_key(&key)); + } + + #[test] + fn dequeue_returns_a_parked_key_nearer_than_the_heap_head() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + queue(&mut queues, 0, &[a]); + queue(&mut queues, 1, &[a]); + let parked = queue(&mut queues, 5, &[a]); + assert_eq!(queues.dispatch_fetches(3, CAP).started.len(), 2); + let farther = queue(&mut queues, 9, &[peer(2)]); + + assert_eq!(queues.dequeue_fetch().map(|c| c.key), Some(parked)); + assert_eq!(queues.dequeue_fetch().map(|c| c.key), Some(farther)); + } + + #[test] + fn a_parked_key_holds_bootstrap_drain_open() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + queue(&mut queues, 0, &[a]); + queue(&mut queues, 1, &[a]); + let parked = queue(&mut queues, 2, &[a]); + queues.dispatch_fetches(3, CAP); + assert!(queues.fetch_parked.contains_key(&parked)); + + assert!(!queues.is_bootstrap_work_empty(&HashSet::from([parked]))); + } + + // -- silent sources --------------------------------------------------------- + + /// A fetch request timeout for the table tests: a mark lasts twice this. + const TEST_TIMEOUT: Duration = Duration::from_secs(10); + + /// A peer distinct from every `peer(b)` and every other index. + fn distinct_peer(index: u32) -> PeerId { + let mut bytes = [0xA5_u8; 32]; + bytes[..4].copy_from_slice(&index.to_be_bytes()); + PeerId::from_bytes(bytes) + } + + /// `base` moved forward by `millis`. + fn at(base: Instant, millis: u64) -> Instant { + base + Duration::from_millis(millis) + } + + fn unreachable(sent: u64) -> Contact { + Contact::Unreachable { sent } + } + + fn answered(order: u64) -> Contact { + Contact::Answered { order } + } + + /// Mark `source` silent from now with a request at sequence place `sent`. + fn silence(queues: &mut ReplicationQueues, source: PeerId, sent: u64) { + queues.record_fetch_contact(source, unreachable(sent)); + } + + fn in_flight_entry(key: XorName, source: PeerId, started_at: Instant) -> InFlightEntry { + InFlightEntry { + key, + source, + started_at, + all_sources: vec![source], + tried: HashSet::from([source]), + retry_verification: None, + } + } + + /// The health table's mark index matches its entries exactly, and only a + /// source with an unanswered request since its latest answer is marked. + fn assert_health_invariants(table: &SourceHealthTable) { + assert!(table.entries.len() <= MAX_TRACKED_FETCH_SOURCES); + let marked: BTreeSet<(Instant, PeerId)> = table + .entries + .iter() + .filter_map(|(source, health)| health.marked_since.map(|since| (since, *source))) + .collect(); + assert_eq!(table.marks_by_since, marked, "mark index is not exact"); + for health in table.entries.values() { + assert!( + health.marked_since.is_none() || health.unanswered_since_answer(), + "a mark outlived the evidence for it" + ); + } + } + + #[test] + fn an_unanswered_request_marks_its_source_until_the_mark_expires() { + let none = HashMap::new(); + let a = peer(1); + let t0 = Instant::now(); + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + + table.record(a, unreachable(1), at(t0, 100), &none); + assert!(table.is_silent(&a, at(t0, 100))); + assert!(table.is_silent(&a, at(t0, 20_099))); + assert!( + !table.is_silent(&a, at(t0, 20_100)), + "a mark lasts two fetch timeouts from when the failure was learned" + ); + + table.expire(at(t0, 20_100)); + assert_eq!(table.counters.expired_marks, 1); + assert!(table.marks_by_since.is_empty()); + + // An older unanswered request learned after the expiry is not new + // evidence: the source stays under the cap for its probe. + table.record(a, unreachable(0), at(t0, 20_200), &none); + assert!(!table.is_silent(&a, at(t0, 20_200))); + + // A request sent later that also goes unanswered marks it again. + table.record(a, unreachable(2), at(t0, 30_200), &none); + assert!(table.is_silent(&a, at(t0, 30_200))); + assert_eq!(table.counters.marks, 2); + assert_eq!(table.counters.unreachable, 3); + assert_health_invariants(&table); + } + + #[test] + fn a_source_that_keeps_failing_stays_silent_past_one_lifetime() { + let none = HashMap::new(); + let a = peer(1); + let t0 = Instant::now(); + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + + for (sent, millis) in [(1, 0), (2, 15_000), (3, 30_000)] { + table.record(a, unreachable(sent), at(t0, millis), &none); + } + + assert!(table.is_silent(&a, at(t0, 49_999))); + assert!(!table.is_silent(&a, at(t0, 50_000))); + assert_eq!( + table.counters.marks, 1, + "one mark, refreshed by newer failures" + ); + assert_eq!(table.marks_by_since.len(), 1); + assert_health_invariants(&table); + } + + #[test] + fn a_failure_learned_late_still_marks_for_a_whole_lifetime() { + let none = HashMap::new(); + let a = peer(1); + let learned = at(Instant::now(), 600_000); + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + + table.record(a, unreachable(1), learned, &none); + + assert!(table.is_silent(&a, learned)); + assert!(table.is_silent(&a, at(learned, 19_999))); + assert!(!table.is_silent(&a, at(learned, 20_000))); + } + + /// Every order of applying `events` to a fresh table, all learned at + /// `now`, so only the sequence places can decide. + fn apply_in_every_order(events: &[Contact], now: Instant) -> Vec { + let none = HashMap::new(); + let a = peer(1); + let mut orders: Vec> = vec![Vec::new()]; + for _ in events { + orders = orders + .into_iter() + .flat_map(|prefix| { + (0..events.len()) + .filter(|index| !prefix.contains(index)) + .map(|index| { + let mut next = prefix.clone(); + next.push(index); + next + }) + .collect::>() + }) + .collect(); + } + orders + .iter() + .map(|order| { + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + for event in order.iter().filter_map(|index| events.get(*index)) { + table.record(a, *event, now, &none); + } + assert_health_invariants(&table); + table + }) + .collect() + } + + #[test] + fn whether_a_source_is_silent_does_not_depend_on_the_order_outcomes_arrive() { + let a = peer(1); + let now = Instant::now(); + let lifetime_ms = + u64::try_from((TEST_TIMEOUT * SILENCE_LIFETIME_TIMEOUTS).as_millis()).unwrap(); + + // Unanswered at 1, an answer at 2, unanswered again at 3: silent, as + // the request sent after the answer went unanswered, for one lifetime. + let silent = [unreachable(1), answered(2), unreachable(3)]; + let tables = apply_in_every_order(&silent, now); + assert_eq!(tables.len(), 6); + for table in &tables { + assert!(table.is_silent(&a, now)); + assert!(table.is_silent(&a, at(now, lifetime_ms - 1))); + assert!(!table.is_silent(&a, at(now, lifetime_ms))); + } + + // Unanswered at 1 and 2, then an answer at 3: answering. + let answering = [unreachable(1), unreachable(2), answered(3)]; + for table in &apply_in_every_order(&answering, now) { + assert!(!table.is_silent(&a, now)); + } + } + + #[test] + fn an_answer_older_than_the_unanswered_request_neither_clears_nor_blocks_its_mark() { + let none = HashMap::new(); + let a = peer(1); + let t0 = Instant::now(); + + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + table.record(a, unreachable(5), t0, &none); + table.record(a, answered(4), t0, &none); + assert!(table.is_silent(&a, t0)); + assert_eq!(table.counters.clears, 0); + + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + table.record(a, answered(4), t0, &none); + table.record(a, unreachable(5), t0, &none); + assert!(table.is_silent(&a, t0)); + assert_eq!(table.counters.ignored_stale, 0); + + // A request sent before the latest answer is stale and cannot mark. + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + table.record(a, answered(6), t0, &none); + table.record(a, unreachable(5), t0, &none); + assert!(!table.is_silent(&a, t0)); + assert_eq!((table.counters.marks, table.counters.ignored_stale), (0, 1)); + assert_health_invariants(&table); + } + + #[test] + fn an_expired_mark_is_kept_only_while_its_source_has_a_fetch_in_flight() { + let t0 = Instant::now(); + let (guarded, idle) = (peer(1), peer(2)); + let key = key_at(0); + let in_flight = HashMap::from([(key, in_flight_entry(key, guarded, t0))]); + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + table.record(guarded, unreachable(1), t0, &in_flight); + table.record(idle, unreachable(2), t0, &in_flight); + + let later = at(t0, 20_000); + let (counters, tracked) = table.take_counters(later, &in_flight); + + assert_eq!((counters.marks, counters.expired_marks, tracked), (2, 2, 2)); + assert!(table.entries.contains_key(&guarded)); + assert!(!table.entries.contains_key(&idle)); + // The guarded source keeps its history: its older request is no new + // evidence, its newer one marks it again. + table.record(guarded, unreachable(0), later, &in_flight); + assert!(!table.is_silent(&guarded, later)); + table.record(guarded, unreachable(3), later, &in_flight); + assert!(table.is_silent(&guarded, later)); + assert_health_invariants(&table); + } + + #[test] + fn attempts_that_sent_nothing_or_were_cancelled_change_nothing() { + let none = HashMap::new(); + let a = peer(1); + let t0 = Instant::now(); + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + + table.record(a, Contact::NotAttempted, t0, &none); + table.record(a, Contact::Unknown, t0, &none); + + assert!(table.entries.is_empty()); + assert!(!table.is_silent(&a, t0)); + assert_eq!(table.counters.unreachable, 0); + } + + #[test] + fn a_zero_fetch_timeout_still_leaves_silent_sources_a_lifetime() { + let none = HashMap::new(); + let a = peer(1); + let t0 = Instant::now(); + let mut table = SourceHealthTable::new(Duration::ZERO); + + table.record(a, unreachable(1), t0, &none); + + assert!(table.is_silent(&a, t0)); + assert!(table.is_silent(&a, at(t0, 999))); + assert!(!table.is_silent(&a, at(t0, 1_000))); + } + + /// A table at its bound: `guarded` answered and has a fetch in flight, + /// `idle` answered and has none, and every other entry is marked. + fn full_table( + t0: Instant, + marked: usize, + ) -> ( + SourceHealthTable, + HashMap, + PeerId, + PeerId, + ) { + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + let guarded = distinct_peer(0); + let idle = distinct_peer(1); + let key = key_at(0); + let in_flight = HashMap::from([(key, in_flight_entry(key, guarded, t0))]); + table.record(guarded, answered(1), t0, &in_flight); + if marked < MAX_TRACKED_FETCH_SOURCES - 1 { + table.record(idle, answered(1), t0, &in_flight); + } + for index in 0..marked { + let source = distinct_peer(u32::try_from(index).unwrap() + 2); + table.record(source, unreachable(1), t0, &in_flight); + } + assert_eq!(table.entries.len(), MAX_TRACKED_FETCH_SOURCES); + (table, in_flight, guarded, idle) + } + + #[test] + fn a_full_table_makes_room_by_dropping_answers_that_guard_nothing() { + let t0 = Instant::now(); + let (mut table, in_flight, guarded, idle) = full_table(t0, MAX_TRACKED_FETCH_SOURCES - 2); + let newcomer = distinct_peer(u32::MAX); + + table.record(newcomer, unreachable(1), t0, &in_flight); + + assert!(table.is_silent(&newcomer, t0)); + assert!( + table.entries.contains_key(&guarded), + "an in-flight guard stays" + ); + assert!(!table.entries.contains_key(&idle)); + assert_eq!(table.counters.overflows, 0); + assert_health_invariants(&table); + } + + #[test] + fn a_table_full_of_marks_and_guards_counts_untracked_sources_as_silent() { + let t0 = Instant::now(); + let (mut table, in_flight, guarded, _) = full_table(t0, MAX_TRACKED_FETCH_SOURCES - 1); + let newcomer = distinct_peer(u32::MAX); + let bystander = distinct_peer(u32::MAX - 1); + + table.record(newcomer, unreachable(1), t0, &in_flight); + + assert_eq!(table.counters.overflows, 1); + assert!(!table.entries.contains_key(&newcomer)); + assert!( + table.is_silent(&newcomer, t0), + "an untracked source that stopped answering is fetched from as before the cap" + ); + assert!(table.is_silent(&bystander, t0)); + assert!( + table.entries.contains_key(&guarded), + "an in-flight guard is never dropped" + ); + assert!( + !table.is_silent(&guarded, t0), + "a tracked source keeps its own state" + ); + let lifetime = TEST_TIMEOUT * SILENCE_LIFETIME_TIMEOUTS; + assert!( + !table.is_silent(&bystander, t0 + lifetime), + "the overflow lasts one silence lifetime" + ); + assert_health_invariants(&table); + } + + #[test] + fn a_silent_source_is_fetched_from_past_the_cap() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + for d in 0..6 { + queue(&mut queues, d, &[a]); + } + assert_eq!(queues.dispatch_fetches(8, CAP).started.len(), CAP); + assert_eq!(queues.fetch_parked.len(), 4); + + silence(&mut queues, a, 1); + let dispatch = queues.dispatch_fetches(8, CAP); + + assert_eq!( + dispatch.started.len(), + 4, + "every parked key starts on the silent source" + ); + assert_eq!(in_flight_to(&queues, &a), 6); + assert!(queues.fetch_parked.is_empty()); + assert_eq!(queues.source_health.counters.silent_starts, 4); + assert_queue_invariants(&queues); + } + + #[test] + fn a_sole_silent_holder_drains_one_timeout_after_it_is_noticed() { + // Sixteen keys held only by `a`, eight fetch slots, and every request + // to `a` unanswered: one wave of two at the share, then the rest past + // it, eight per wave, as many waves as before the cap plus one. + let mut queues = ReplicationQueues::new(); + let a = peer(1); + for d in 0..16 { + queue(&mut queues, d, &[a]); + } + let mut sequence = 0; + let mut waves = Vec::new(); + loop { + let started = queues.dispatch_fetches(8, CAP).started; + if started.is_empty() { + break; + } + waves.push(started.len()); + // Every request in the wave times out. + for (key, source) in started { + sequence += 1; + queues.record_fetch_contact(source, unreachable(sequence)); + assert_eq!( + queues.retry_or_park_fetch(&key, key, CAP), + FetchRetry::Exhausted + ); + queues.complete_fetch(&key); + } + assert_queue_invariants(&queues); + } + assert_eq!(waves, vec![2, 8, 6]); + assert_eq!(queues.fetch_queue_count(), 0); + } + + #[test] + fn an_answering_source_with_room_is_preferred_over_a_silent_one() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + silence(&mut queues, a, 1); + let keys: Vec = (0..3).map(|d| queue(&mut queues, d, &[a, b])).collect(); + + let dispatch = queues.dispatch_fetches(8, CAP); + + let expected: Vec<(XorName, PeerId)> = keys.iter().copied().zip([b, b, a]).collect(); + assert_eq!( + dispatch.started, expected, + "b until its share is full, then the silent a rather than parking" + ); + assert!(queues.fetch_parked.is_empty()); + assert_queue_invariants(&queues); + } + + #[test] + fn retry_prefers_an_answering_source_with_room_then_a_silent_one() { + let (a, b, c) = (peer(1), peer(2), peer(3)); + for c_full in [false, true] { + let mut queues = ReplicationQueues::new(); + let key = queue(&mut queues, 0, &[a, b, c]); + if c_full { + queue(&mut queues, 10, &[c]); + queue(&mut queues, 11, &[c]); + } + let started = if c_full { 3 } else { 1 }; + assert_eq!(queues.dispatch_fetches(3, CAP).started.len(), started); + assert_eq!( + queues.in_flight_fetch.get(&key).map(|entry| entry.source), + Some(a) + ); + silence(&mut queues, b, 1); + + let retry = queues.retry_or_park_fetch(&key, key, CAP); + + let expected = if c_full { b } else { c }; + assert_eq!(retry, FetchRetry::Next(expected)); + assert_eq!( + queues.source_health.counters.silent_starts, + u64::from(c_full) + ); + assert_queue_invariants(&queues); + } + } + + #[test] + fn parked_keys_start_on_a_source_that_falls_silent_while_the_others_stay_full() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + for d in 0..4 { + queue(&mut queues, d, &[if d < 2 { a } else { b }]); + } + assert_eq!(queues.dispatch_fetches(4, CAP).started.len(), 4); + let parked: Vec = (10..20).map(|d| queue(&mut queues, d, &[a, b])).collect(); + assert_eq!(queues.dispatch_fetches(8, CAP).started, Vec::new()); + assert_eq!(queues.fetch_parked.len(), 10); + + silence(&mut queues, b, 1); + let dispatch = queues.dispatch_fetches(8, CAP); + + // a stays full and answering: every slot goes to b, and the pass ends. + let expected: Vec<(XorName, PeerId)> = parked.iter().take(8).map(|key| (*key, b)).collect(); + assert_eq!(dispatch.started, expected); + assert_eq!(queues.fetch_parked.len(), 2); + assert_queue_invariants(&queues); + } + + #[test] + fn an_answer_puts_a_silent_source_back_under_the_cap() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + for d in 0..6 { + queue(&mut queues, d, &[a]); + } + silence(&mut queues, a, 1); + assert_eq!(queues.dispatch_fetches(3, CAP).started.len(), 3); + + queues.record_fetch_contact(a, answered(2)); + let dispatch = queues.dispatch_fetches(8, CAP); + + assert_eq!( + dispatch.started, + Vec::new(), + "a answers again and already has three fetches in flight" + ); + assert_eq!(queues.fetch_parked.len(), 3); + assert_queue_invariants(&queues); + } + + #[test] + fn a_departure_keeps_what_was_observed_of_the_source() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + let key = queue(&mut queues, 0, &[a]); + assert_eq!(queues.dispatch_fetches(1, CAP).started, vec![(key, a)]); + silence(&mut queues, a, 1); + + queues.remove_hint_source(&a); + + assert!(queues.source_health.is_silent(&a, Instant::now())); + // The attempt still in flight answers after the departure: applied + // like any other outcome. + queues.record_fetch_contact(a, answered(2)); + assert!(!queues.source_health.is_silent(&a, Instant::now())); + // A late unanswered request sent before that answer cannot mark it. + queues.record_fetch_contact(a, unreachable(1)); + assert!(!queues.source_health.is_silent(&a, Instant::now())); + assert_health_invariants(&queues.source_health); + } + + #[test] + fn the_summary_drops_answers_that_guard_nothing_and_keeps_marks() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = queue(&mut queues, 0, &[a]); + assert_eq!(queues.dispatch_fetches(1, CAP).started, vec![(key, a)]); + queues.record_fetch_contact(a, answered(1)); + queues.record_fetch_contact(b, answered(2)); + silence(&mut queues, c, 3); + + let stats = queues.take_fetch_dispatch_stats(CAP); + + assert_eq!(stats.tracked_sources, 3); + assert_eq!(stats.silent_sources, 1); + assert!(!stats.overflow_active); + assert_eq!(stats.health.marks, 1); + assert_eq!(stats.health.silent_peak, 1); + let kept: HashSet = queues.source_health.entries.keys().copied().collect(); + assert_eq!( + kept, + HashSet::from([a, c]), + "a guards a fetch in flight and c is marked" + ); + assert_eq!(queues.source_health.counters.marks, 0, "counters restart"); + assert_eq!( + queues.source_health.counters.silent_peak, 1, + "the peak restarts at the sources silent now" + ); + assert_health_invariants(&queues.source_health); + } + + #[test] + #[allow(clippy::too_many_lines)] + fn queue_invariants_hold_under_random_operations() { + let mut rng = StdRng::seed_from_u64(0x1514); + let mut queues = ReplicationQueues::new().with_fetch_source_limit(5); + let sources: Vec = (1..=6).map(peer).collect(); + let mut sequence = 0_u64; + for step in 0..4_000_u32 { + match rng.gen_range(0..10) { + 0 | 1 => { + let count = rng.gen_range(1..=4); + let mut chosen: Vec = Vec::new(); + while chosen.len() < count { + let source = sources[rng.gen_range(0..sources.len())]; + if !chosen.contains(&source) { + chosen.push(source); + } + } + let key = key_at(rng.gen_range(0..400)); + if rng.gen_bool(0.25) { + // Through verification, so it carries a retry entry. + let hinter = rng.gen_range(1..=30); + if queues + .add_pending_verify(key, test_entry(hinter)) + .admitted() + { + queues.promote_pending_to_fetch(key, key, chosen); + } + } else { + queues.enqueue_fetch(key, key, chosen); + } + } + 2 => { + // As the fetch worker does, a candidate with no untried + // source goes back to verification. + let dispatch = queues.dispatch_fetches(rng.gen_range(0..6), CAP); + for candidate in dispatch.sourceless { + queues.requeue_candidate_for_verification(candidate, Duration::ZERO); + } + } + 3 => { + if let Some(key) = queues.in_flight_fetch.keys().min().copied() { + queues.complete_fetch(&key); + } + } + 4 => { + let mut in_flight: Vec = + queues.in_flight_fetch.keys().copied().collect(); + in_flight.sort_unstable(); + if !in_flight.is_empty() { + let key = in_flight[rng.gen_range(0..in_flight.len())]; + if queues.retry_or_park_fetch(&key, key, CAP) == FetchRetry::Exhausted { + queues.complete_fetch(&key); + } + } + } + 5 => { + let mut waiting: Vec = queues.fetch_payloads.keys().copied().collect(); + waiting.sort_unstable(); + if !waiting.is_empty() { + let key = waiting[rng.gen_range(0..waiting.len())]; + let hinter = rng.gen_range(1..=30); + let entry = if rng.gen_bool(0.5) { + test_entry(hinter) + } else { + paid_only_entry(hinter) + }; + queues.add_pending_verify(key, entry); + } + } + 6 => { + if rng.gen_range(0..10) == 0 { + queues.remove_hint_source(&sources[rng.gen_range(0..sources.len())]); + } + } + 7 => { + if let Some(candidate) = queues.dequeue_fetch() { + match candidate.sources.first().copied() { + Some(source) if rng.gen_bool(0.5) => { + queues.start_dequeued_fetch(candidate, source); + } + _ => queues.discard_fetch_candidate(candidate), + } + } + } + _ => { + // A source answers, or a request to it goes unanswered, + // applied a little out of sequence order. + if let Some(source) = sources.get(rng.gen_range(0..sources.len())).copied() { + sequence += 1; + let place = sequence.saturating_sub(rng.gen_range(0..4)); + let contact = if rng.gen_bool(0.5) { + Contact::Answered { order: place } + } else { + Contact::Unreachable { sent: place } + }; + queues.record_fetch_contact(source, contact); + } + } + } + assert_queue_invariants(&queues); + assert_health_invariants(&queues.source_health); + assert_retry_entries_bounded(&queues); + for (key, entry) in &queues.in_flight_fetch { + let associated: HashSet<&PeerId> = + entry.all_sources.iter().chain(entry.tried.iter()).collect(); + assert!( + associated.len() <= 5, + "step {step}: {} over the source limit", + hex::encode(key) + ); + } + for (key, payload) in &queues.fetch_payloads { + let tried = queues.fetch_tried.get(key).map_or(0, HashSet::len); + assert!( + payload.sources.len() + tried <= 5, + "step {step}: queued over the limit" + ); + } + } + } + + /// Scale benchmark for the per-source scheduler at the fetch queue's hard + /// bound. Runs only when `ANT_FETCH_SCALE_BENCH` names a sources-per-key + /// count (it allocates hundreds of MiB and takes seconds); otherwise it + /// returns at once. One count per process, so `/usr/bin/time -l` reports + /// that scenario's peak memory: + /// + /// `ANT_FETCH_SCALE_BENCH=8 cargo test --release --features test-utils --lib + /// scale_benchmark -- --nocapture` + /// + /// Thresholds, declared before the release runs: one pass parking every key + /// holds the queue for at most 1 s at up to 8 sources per key and at most + /// 5 s at the 35-source limit; waking, the stats read and a departure each + /// stay under 250 ms; the process's peak RSS is at most 1 GiB at 8 sources + /// per key and 3 GiB at 35. Dropping the queues was first held to 250 ms + /// too, but measured 274 to 315 ms at 35 sources per key, and 226 to 295 ms + /// with the scheduler before the silent-source handling. It happens once, + /// when the engine shuts down, outside any lock, so it is held to 1 s. + #[test] + fn scale_benchmark_at_the_fetch_queue_bound() { + let Some(sources_per_key) = std::env::var("ANT_FETCH_SCALE_BENCH") + .ok() + .and_then(|raw| raw.parse::().ok()) + else { + return; + }; + let park_limit = if sources_per_key <= 8 { + Duration::from_secs(1) + } else { + Duration::from_secs(5) + }; + { + let other_limit = Duration::from_millis(250); + let teardown_limit = Duration::from_secs(1); + let sources: Vec = (0..sources_per_key) + .map(|index| { + let mut bytes = [0u8; 32]; + bytes[..8].copy_from_slice(&(index as u64 + 1).to_be_bytes()); + PeerId::from_bytes(bytes) + }) + .collect(); + let mut queues = ReplicationQueues::new(); + // Saturate every source with this node's own fetches first. + let mut blockers = Vec::new(); + for (index, source) in sources.iter().enumerate() { + for offset in 0..CAP { + let key = key_at(u32::MAX - u32::try_from(index * CAP + offset).unwrap()); + queues.start_fetch(key, *source, vec![*source]); + blockers.push(key); + } + } + let mut distance = 0u32; + while queues.fetch_queue_count() < MAX_FETCH_QUEUE { + let key = key_at(distance); + assert!(queues.enqueue_fetch(key, key, sources.clone())); + distance += 1; + } + + let started = Instant::now(); + let dispatch = queues.dispatch_fetches(8, CAP); + let park_pass = started.elapsed(); + assert_eq!(dispatch.started, Vec::new()); + assert_eq!(queues.fetch_parked.len(), MAX_FETCH_QUEUE); + + let associations: usize = queues.parked_by_source.values().map(BTreeSet::len).sum(); + + let started = Instant::now(); + let stats = queues.take_fetch_dispatch_stats(CAP); + let stats_pass = started.elapsed(); + assert_eq!(stats.parked, MAX_FETCH_QUEUE); + + let started = Instant::now(); + queues.complete_fetch(&blockers[0]); + let wake = queues.dispatch_fetches(8, CAP); + let wake_pass = started.elapsed(); + assert_eq!(wake.started.len(), 1); + + let started = Instant::now(); + queues.remove_hint_source(&sources[0]); + let departure = started.elapsed(); + + let started = Instant::now(); + drop(queues); + let teardown = started.elapsed(); + + println!( + "FETCH-SCALE sources_per_key={sources_per_key} keys={MAX_FETCH_QUEUE} \ + associations={associations} park_pass_ms={} stats_ms={} wake_ms={} \ + departure_ms={} teardown_ms={}", + park_pass.as_millis(), + stats_pass.as_millis(), + wake_pass.as_millis(), + departure.as_millis(), + teardown.as_millis() + ); + assert!( + park_pass <= park_limit, + "park pass {park_pass:?} over {park_limit:?}" + ); + for (name, took) in [ + ("stats", stats_pass), + ("wake", wake_pass), + ("departure", departure), + ] { + assert!( + took <= other_limit, + "{name} took {took:?}, over {other_limit:?}" + ); + } + assert!( + teardown <= teardown_limit, + "teardown took {teardown:?}, over {teardown_limit:?}" + ); + } + } } diff --git a/tests/e2e/fetch_per_source_share.rs b/tests/e2e/fetch_per_source_share.rs new file mode 100644 index 00000000..7b9b87ef --- /dev/null +++ b/tests/e2e/fetch_per_source_share.rs @@ -0,0 +1,328 @@ +//! A fetching node keeps no more of its fetches in flight to one answering +//! holder than that holder will admit from it. +//! +//! A fetch responder admits two outstanding requests per requesting peer and +//! answers a third with nothing at all, so a requester that sends more than two +//! at once to one holder loses the rest to its own fetch timeout. The fetch +//! worker used to do exactly that: it ran one fetch per slot and sent every one +//! to the first verified holder of its key, so a node whose queued keys shared +//! one holder had all of them outstanding there at once. +//! +//! The observation is made on the requester, which is what the scheduler +//! governs, and made deterministic by holding the holder's work in place: the +//! holder's test-only serve gate holds every fetch request it admits before +//! serving it, so whatever the requester sends stays outstanding for as long as +//! the test looks at it, however fast the host serves a chunk. Every node runs +//! eight fetch slots, more than one holder's share on any host. With the +//! per-source limit removed the requester holds all eight in flight to the one +//! holder and this test fails. Both hooks are per engine, so no other node or +//! test is affected. +//! +//! The limit is a share of a holder that answers. A holder whose requests go +//! unanswered has no share to keep within, and keeping its keys at two per +//! fetch timeout would drain them four times more slowly than before the limit. +//! The second test closes the holder's gate for good, with a short fetch +//! timeout, and checks that once the first requests time out the requester +//! sends past the share and the keys leave its fetch pipeline in a bounded time. +//! +//! Both tests tear the network down with the holder's gate still closed and +//! requests held at it, which the holder's engine shutdown must release. + +#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] + +use super::testnet::TestNetworkConfig; +use super::verify_storage_gate::find_key_for_target; +use super::TestHarness; +use ant_node::client::XorName; +use ant_node::replication::config::storage_admission_width; +use ant_node::{ReplicationConfig, ReplicationEngine}; +use saorsa_core::identity::PeerId; +use serial_test::serial; +use std::collections::HashSet; +use std::time::{Duration, Instant}; + +/// Above `storage_admission_width` (9), so the network has a real close group. +const NODE_COUNT: usize = 12; +/// Production close-group size. +const CLOSE_GROUP_SIZE: usize = 7; +/// Node whose fetch worker is observed. +const REQUESTER_INDEX: usize = 5; +/// The one node hosting every chunk. +const HOLDER_INDEX: usize = 6; +/// More keys than fetch slots, all held by the one holder. +const KEY_COUNT: usize = 12; +/// Fetch slots per node, above the per-source share on every host. +const FETCH_CONCURRENCY: usize = 8; +/// The responder's per-peer fetch share. +const PER_SOURCE_SHARE: usize = 2; +/// How long the requester's fetch pipeline must stay empty before the test +/// starts, so earlier traffic cannot occupy its slots. +const QUIET_FOR: Duration = Duration::from_secs(1); +/// Upper bound on waiting for that quiet period. +const QUIET_DEADLINE: Duration = Duration::from_secs(30); +/// How long the requester is observed with the gate closed. Far below the 30 s +/// fetch timeout, so no held request times out and frees its slot early. +const OBSERVATION_WINDOW: Duration = Duration::from_secs(3); +const OBSERVATION_POLL: Duration = Duration::from_millis(50); +/// The silent-holder test's fetch timeout, short so silence is seen in seconds. +const SILENT_FETCH_TIMEOUT: Duration = Duration::from_secs(5); +/// Keys held only by the silent holder: two fetch slots' worth. +const SILENT_KEY_COUNT: usize = 16; +/// The longest the silent holder's keys may take to leave the requester's +/// fetch pipeline: four fetch timeouts. This scheduler takes three, one more +/// than fetching as before the limit, to see the silence; keeping the holder +/// at its share would take eight (sixteen keys, two per timeout). The exact +/// three waves are checked without timing in the scheduler's unit tests. +const SILENT_DRAIN_LIMIT: Duration = Duration::from_secs(20); +/// The longest tearing the network down may take with requests still held at +/// the holder's closed serve gate. Without the release at engine shutdown it +/// waits for them for ever. +const TEARDOWN_LIMIT: Duration = Duration::from_secs(120); + +#[tokio::test] +#[serial] +async fn requester_keeps_at_most_its_share_in_flight_to_one_holder() { + let max_in_flight = max_in_flight_to_one_holder().await; + assert!( + max_in_flight <= PER_SOURCE_SHARE, + "the requester had {max_in_flight} fetches outstanding at one holder, which \ + admits {PER_SOURCE_SHARE} from it and answers the rest with nothing" + ); + assert_eq!( + max_in_flight, PER_SOURCE_SHARE, + "the requester never filled the holder's share, so the run says nothing \ + about the limit: the worker did not dispatch" + ); +} + +#[tokio::test] +#[serial] +async fn a_holder_that_stops_answering_is_fetched_from_past_its_share() { + let harness = start_network(SILENT_FETCH_TIMEOUT).await; + let chunks = chunks_for_requester(&harness, "silent-holder", SILENT_KEY_COUNT).await; + let keys: HashSet = chunks.iter().map(|(_, key)| *key).collect(); + + let requester = harness.test_node(REQUESTER_INDEX).expect("requester"); + let requester_engine = requester + .replication_engine + .as_ref() + .expect("requester engine"); + let holder = harness.test_node(HOLDER_INDEX).expect("holder"); + let holder_peer = *holder.p2p_node.as_ref().expect("holder p2p").peer_id(); + let holder_engine = holder.replication_engine.as_ref().expect("holder engine"); + + wait_until_fetch_pipeline_is_quiet(requester_engine).await; + + // The holder answers none of these fetches, ever. + holder_engine.close_fetch_serve_gate_for_test(); + let started = Instant::now(); + for key in &keys { + assert!( + requester_engine + .enqueue_fetch_for_test(*key, vec![holder_peer]) + .await, + "every key must be queued for the silent holder alone" + ); + } + + let mut max_in_flight_to_holder = 0; + let drained_after = loop { + let snapshot = requester_engine.fetch_pipeline_snapshot_for_test().await; + let in_flight_to_holder = snapshot + .in_flight + .iter() + .filter(|(key, source)| keys.contains(key) && *source == holder_peer) + .count(); + max_in_flight_to_holder = max_in_flight_to_holder.max(in_flight_to_holder); + let pending = snapshot + .queued + .iter() + .chain(snapshot.in_flight.iter().map(|(key, _)| key)) + .filter(|key| keys.contains(*key)) + .count(); + if pending == 0 { + break started.elapsed(); + } + assert!( + started.elapsed() < SILENT_DRAIN_LIMIT, + "{pending} of {SILENT_KEY_COUNT} keys held only by a silent holder were still \ + in the fetch pipeline after {SILENT_DRAIN_LIMIT:?}" + ); + tokio::time::sleep(OBSERVATION_POLL).await; + }; + println!( + "SILENT-HOLDER-RESULT drained_after={drained_after:?} \ + max_in_flight_to_holder={max_in_flight_to_holder} timeout={SILENT_FETCH_TIMEOUT:?}" + ); + + teardown_with_the_gate_closed(harness).await; + assert!( + drained_after < SILENT_DRAIN_LIMIT, + "the silent holder's keys took {drained_after:?} to leave the fetch pipeline" + ); + assert!( + max_in_flight_to_holder > PER_SOURCE_SHARE, + "the requester kept a holder that answers nothing at its share of \ + {PER_SOURCE_SHARE}, which drains that holder's keys at two per timeout" + ); +} + +/// Build the network with `fetch_request_timeout` and warm its DHT. +async fn start_network(fetch_request_timeout: Duration) -> TestHarness { + let config = TestNetworkConfig { + node_count: NODE_COUNT, + replication_config: Some(ReplicationConfig { + close_group_size: CLOSE_GROUP_SIZE, + fetch_request_timeout, + ..ReplicationConfig::default() + }), + fetch_concurrency: Some(FETCH_CONCURRENCY), + ..TestNetworkConfig::default() + }; + let harness = TestHarness::setup_with_config(config) + .await + .expect("setup per-source share network"); + harness.warmup_dht().await.expect("warmup"); + harness +} + +/// `count` distinct chunks the requester is responsible for storing. +async fn chunks_for_requester( + harness: &TestHarness, + label: &str, + count: usize, +) -> Vec<(Vec, XorName)> { + let width = storage_admission_width(CLOSE_GROUP_SIZE); + let mut chunks = Vec::with_capacity(count); + for index in 0..count { + let label = format!("{label}-{index}"); + chunks.push(find_key_for_target(harness, REQUESTER_INDEX, width, true, &label).await); + } + let keys: HashSet<&XorName> = chunks.iter().map(|(_, key)| key).collect(); + assert_eq!(keys.len(), count, "every key must be distinct"); + chunks +} + +/// Build the network, direct twelve keys at one holder while that holder holds +/// what it admits, and return the most fetches the requester had in flight to +/// it. +async fn max_in_flight_to_one_holder() -> usize { + let harness = start_network(ReplicationConfig::default().fetch_request_timeout).await; + let chunks = chunks_for_requester(&harness, "per-source-share", KEY_COUNT).await; + let keys: HashSet = chunks.iter().map(|(_, key)| *key).collect(); + + let requester = harness.test_node(REQUESTER_INDEX).expect("requester"); + let requester_engine = requester + .replication_engine + .as_ref() + .expect("requester engine"); + let holder = harness.test_node(HOLDER_INDEX).expect("holder"); + let holder_peer = *holder.p2p_node.as_ref().expect("holder p2p").peer_id(); + let holder_engine = holder.replication_engine.as_ref().expect("holder engine"); + let holder_storage = holder + .ant_protocol + .as_ref() + .expect("holder protocol") + .storage(); + + // Start from an idle requester, so its fetch slots are all free. + wait_until_fetch_pipeline_is_quiet(requester_engine).await; + + // Hold every fetch the holder admits, then make the chunks available. + holder_engine.close_fetch_serve_gate_for_test(); + for (content, key) in &chunks { + holder_storage.put(key, content).await.expect("host chunk"); + } + let before = requester_engine.fetch_pipeline_snapshot_for_test().await; + let unrelated: Vec = before + .queued + .iter() + .chain(before.in_flight.iter().map(|(key, _)| key)) + .filter(|key| !keys.contains(*key)) + .copied() + .collect(); + assert!( + unrelated.is_empty(), + "unrelated fetch work could occupy the requester's slots: {unrelated:?}" + ); + for key in &keys { + // A replica hint may already have queued the key; either way it is + // headed to the one holder. + let _ = requester_engine + .enqueue_fetch_for_test(*key, vec![holder_peer]) + .await; + } + + let max_in_flight_to_holder = + observe_max_in_flight_to(requester_engine, &keys, &holder_peer).await; + println!( + "PER-SOURCE-SHARE-RESULT max_in_flight_to_holder={max_in_flight_to_holder} \ + concurrency={FETCH_CONCURRENCY} keys={KEY_COUNT}" + ); + + teardown_with_the_gate_closed(harness).await; + max_in_flight_to_holder +} + +/// Tear the network down while the holder's serve gate is still closed. +async fn teardown_with_the_gate_closed(harness: TestHarness) { + let torn_down = tokio::time::timeout(TEARDOWN_LIMIT, harness.teardown()).await; + assert!( + matches!(torn_down, Ok(Ok(()))), + "teardown with requests held at a closed serve gate did not finish within \ + {TEARDOWN_LIMIT:?}: {torn_down:?}" + ); +} + +/// Wait until `engine` has nothing queued or in flight for fetch, continuously +/// for [`QUIET_FOR`]. +async fn wait_until_fetch_pipeline_is_quiet(engine: &ReplicationEngine) { + let deadline = tokio::time::Instant::now() + QUIET_DEADLINE; + let mut quiet_since = tokio::time::Instant::now(); + loop { + let snapshot = engine.fetch_pipeline_snapshot_for_test().await; + let now = tokio::time::Instant::now(); + if !snapshot.queued.is_empty() || !snapshot.in_flight.is_empty() { + quiet_since = now; + } else if now.duration_since(quiet_since) >= QUIET_FOR { + return; + } + assert!( + now < deadline, + "the requester's fetch pipeline never went quiet: {snapshot:?}" + ); + tokio::time::sleep(OBSERVATION_POLL).await; + } +} + +/// The most of `keys` that `engine` had in flight to `holder` at once over +/// [`OBSERVATION_WINDOW`], failing if any other fetch work appears. +async fn observe_max_in_flight_to( + engine: &ReplicationEngine, + keys: &HashSet, + holder: &PeerId, +) -> usize { + let deadline = tokio::time::Instant::now() + OBSERVATION_WINDOW; + let mut max_in_flight = 0; + while tokio::time::Instant::now() < deadline { + let snapshot = engine.fetch_pipeline_snapshot_for_test().await; + let unrelated: Vec<&XorName> = snapshot + .queued + .iter() + .chain(snapshot.in_flight.iter().map(|(key, _)| key)) + .filter(|key| !keys.contains(*key)) + .collect(); + assert!( + unrelated.is_empty(), + "unrelated fetch work appeared during the observation: {unrelated:?}" + ); + let to_holder = snapshot + .in_flight + .iter() + .filter(|(key, source)| keys.contains(key) && source == holder) + .count(); + max_in_flight = max_in_flight.max(to_holder); + tokio::time::sleep(OBSERVATION_POLL).await; + } + max_in_flight +} diff --git a/tests/e2e/mod.rs b/tests/e2e/mod.rs index ec2a74c7..295b022d 100644 --- a/tests/e2e/mod.rs +++ b/tests/e2e/mod.rs @@ -75,6 +75,9 @@ mod fetch_local_write_guard; #[cfg(test)] mod fetch_responsibility_recheck; +#[cfg(test)] +mod fetch_per_source_share; + #[cfg(test)] mod first_audit_ab; diff --git a/tests/e2e/testnet.rs b/tests/e2e/testnet.rs index ca33a901..051e42b4 100644 --- a/tests/e2e/testnet.rs +++ b/tests/e2e/testnet.rs @@ -230,6 +230,10 @@ pub struct TestNetworkConfig { /// use this to shorten timers — e.g. the ADR-0003 possession-check delay. pub replication_config: Option, + /// Fetches each node's worker runs at once. `None` uses one per hardware + /// thread. + pub fetch_concurrency: Option, + /// Optional per-node storage disk-reserve overrides. /// /// Tests can set a node's reserve above available disk space to make its @@ -283,6 +287,7 @@ impl Default for TestNetworkConfig { payment_enforcement: false, evm_network: None, replication_config: None, + fetch_concurrency: None, storage_disk_reserve_overrides: HashMap::new(), } } @@ -1388,6 +1393,9 @@ impl TestNetwork { .await { Ok(mut engine) => { + if let Some(concurrency) = self.config.fetch_concurrency { + engine.set_fetch_concurrency_for_test(concurrency); + } // Pointers replicate through the same engine (ADR-0016), // which the service also asks before a final state // (ADR-0018). From 267cf9748584377666b83fd368ac8d7f6dca81a0 Mon Sep 17 00:00:00 2001 From: grumbach Date: Tue, 20 Oct 2026 17:41:30 +0900 Subject: [PATCH 05/10] style(replication): build the busy durations from whole minutes and hours Clippy's duration_suboptimal_units lint, as of Rust 1.96, flags Duration::from_secs(60), (600) and (3600). Build those durations with from_mins and from_hours instead, as the crate already does elsewhere. Every value is unchanged. --- src/replication/fetch_busy.rs | 8 ++++---- tests/e2e/fetch_busy_ab.rs | 2 +- tests/e2e/fetch_busy_reply.rs | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/src/replication/fetch_busy.rs b/src/replication/fetch_busy.rs index 6a9c6b65..1a606f27 100644 --- a/src/replication/fetch_busy.rs +++ b/src/replication/fetch_busy.rs @@ -129,7 +129,7 @@ const GLOBAL_INTERVAL: Duration = Duration::from_micros(62_500); const MAX_TRACKED_PEERS: usize = 4096; /// Longest a source that keeps answering busy is put last. -const MAX_SOURCE_COOLDOWN: Duration = Duration::from_secs(60); +const MAX_SOURCE_COOLDOWN: Duration = Duration::from_mins(1); /// Busy answers after which a source's cooldown stops doubling: the wait, /// then two, four and eight times it, capped at [`MAX_SOURCE_COOLDOWN`]. @@ -139,7 +139,7 @@ const MAX_COOLDOWN_DOUBLINGS: u32 = 3; /// with its next busy answer: its cooldown starts again from the wait, so one /// busy spell is not held against its scheduling for good. What it owes /// against its neutral allowance does not start again. -const BUSY_SPELL_GAP: Duration = Duration::from_secs(600); +const BUSY_SPELL_GAP: Duration = Duration::from_mins(10); /// Most sources whose busy record is kept at once. /// @@ -553,7 +553,7 @@ mod tests { ); // Jitter past the spread is clamped, not added. assert_eq!( - busy_wait(remaining, Duration::from_secs(3600)), + busy_wait(remaining, Duration::from_hours(1)), FETCH_BUSY_WAIT_MAX ); } @@ -746,7 +746,7 @@ mod tests { } // Long after, everyone else is idle and a full table sheds them, but // never the peer whose reply is still in flight. - let later = now + Duration::from_secs(3600); + let later = now + Duration::from_hours(1); assert!(grant(&limiter, &peer(u32::MAX), later)); assert_eq!( limiter.try_acquire(&peer(0), later).err(), diff --git a/tests/e2e/fetch_busy_ab.rs b/tests/e2e/fetch_busy_ab.rs index 739bbb10..b5988848 100644 --- a/tests/e2e/fetch_busy_ab.rs +++ b/tests/e2e/fetch_busy_ab.rs @@ -136,7 +136,7 @@ const POOL_GATE_HOLD: Duration = Duration::from_secs(2); /// A holder's fetch admission pool: `FETCH_RESPONDER_MAX_OUTSTANDING`. const POOL_SLOTS: usize = 12; /// Bound on reaching the precondition before measuring. -const POOL_SETUP_DEADLINE: Duration = Duration::from_secs(60); +const POOL_SETUP_DEADLINE: Duration = Duration::from_mins(1); #[tokio::test] #[serial] diff --git a/tests/e2e/fetch_busy_reply.rs b/tests/e2e/fetch_busy_reply.rs index 83fc8d97..b4bbcabc 100644 --- a/tests/e2e/fetch_busy_reply.rs +++ b/tests/e2e/fetch_busy_reply.rs @@ -38,7 +38,7 @@ const REQUESTER_INDEX: usize = 4; const SHARE: usize = 2; /// Long enough that a held request is never timed out before the test opens /// the gate; a timeout would charge the holder and spoil the trust checks. -const HELD_TIMEOUT: Duration = Duration::from_secs(60); +const HELD_TIMEOUT: Duration = Duration::from_mins(1); /// How long a request that should be refused silently is given. const SILENCE_TIMEOUT: Duration = Duration::from_secs(3); /// Bound on how long anything that must happen promptly may take. From e35236b85978e9e86837e534c681c2baa77105ea Mon Sep 17 00:00:00 2001 From: grumbach Date: Tue, 20 Oct 2026 18:35:19 +0900 Subject: [PATCH 06/10] test(replication): keep only the pool-full A/B measurement With #252's per-source scheduler, one requester no longer sends more than a holder's two-request share, so the over-share measurement can no longer produce a busy reply. Keep the measurement of the refusals that #252 leaves, a holder whose admission pool is full of other peers' requests, and say so in the file's description. --- tests/e2e/fetch_busy_ab.rs | 118 ++++--------------------------------- 1 file changed, 12 insertions(+), 106 deletions(-) diff --git a/tests/e2e/fetch_busy_ab.rs b/tests/e2e/fetch_busy_ab.rs index b5988848..4b570da3 100644 --- a/tests/e2e/fetch_busy_ab.rs +++ b/tests/e2e/fetch_busy_ab.rs @@ -1,22 +1,18 @@ -//! Opt-in measurement, not a pass/fail gate: how long one node takes to fetch -//! a batch of chunks whose holders it over-asks, and what that costs the -//! holders in the requester's trust table. +//! Opt-in measurement, not a pass/fail gate: how long requesters take to +//! fetch a batch of chunks while the first of their holders keeps its whole +//! fetch admission pool full, and what that costs the holders in the +//! requesters' trust tables. //! -//! The requester fetches every chunk with the same three holders in the same -//! order, so its fetch worker, one slot per hardware thread, sends more than a -//! holder's two-request share to the first holder at once. That is the -//! over-share the busy reply answers. The file uses only test hooks that exist -//! on `main` too, so the same file measures a build without the busy reply and -//! a build with it. -//! -//! A second measurement, [`fetch_busy_ab_pool`], has every requester stay -//! within its share, as #252 makes them, and fills the first holder's -//! twelve-slot pool instead: the refusals that remain after #252. +//! Every requester stays within its two-request share at each holder, as +//! #252's per-source scheduler keeps it, so the refusals measured are the ones +//! left after #252: a holder whose twelve-slot pool is full of other peers' +//! requests. The file uses only the test-only fetch serve gate, which #252 +//! has too, so the same file measures #252 without the busy reply and with +//! it. //! //! Run with `ANT_FETCH_BUSY_AB=1 cargo test --test e2e --features test-utils -//! fetch_busy_ab -- --nocapture --test-threads=1`. Without the variable both -//! return at once. The pool measurement uses the test-only fetch serve gate, -//! which a build without the busy reply needs patched in to compare. +//! fetch_busy_ab -- --nocapture --test-threads=1`. Without the variable it +//! returns at once. #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] @@ -27,99 +23,9 @@ use ant_node::ReplicationConfig; use serial_test::serial; use std::time::{Duration, Instant}; -const REQUESTER_INDEX: usize = 4; -const HOLDER_INDICES: [usize; 3] = [1, 2, 3]; -const CHUNKS: usize = 48; -const CHUNK_LEN: usize = 256 * 1024; const FILL_DEADLINE: Duration = Duration::from_mins(15); const POLL: Duration = Duration::from_millis(100); -#[tokio::test] -#[serial] -async fn fetch_busy_ab() { - if std::env::var("ANT_FETCH_BUSY_AB").is_err() { - return; - } - let harness = TestHarness::setup_minimal().await.expect("setup"); - harness.warmup_dht().await.expect("warmup"); - - let holders: Vec<_> = HOLDER_INDICES - .iter() - .map(|&i| { - *harness - .test_node(i) - .and_then(|n| n.p2p_node.as_ref()) - .expect("holder p2p") - .peer_id() - }) - .collect(); - - let mut keys = Vec::with_capacity(CHUNKS); - for c in 0..CHUNKS { - let content: Vec = (0..CHUNK_LEN) - .map(|i| u8::try_from((i * 31 + c * 7) % 251).unwrap()) - .collect(); - let key = compute_address(&content); - for &h in &HOLDER_INDICES { - harness - .test_node(h) - .and_then(|n| n.ant_protocol.as_ref()) - .expect("holder protocol") - .storage() - .put(&key, &content) - .await - .expect("put"); - } - keys.push(key); - } - - let requester = harness.test_node(REQUESTER_INDEX).expect("requester"); - let p2p = requester.p2p_node.as_ref().expect("requester p2p"); - let engine = requester - .replication_engine - .as_ref() - .expect("requester engine"); - let storage = requester - .ant_protocol - .as_ref() - .expect("requester protocol") - .storage(); - - let trust_before: Vec = holders.iter().map(|h| p2p.peer_trust(h)).collect(); - let started = Instant::now(); - let mut enqueued = 0usize; - for key in &keys { - if engine.enqueue_fetch_for_test(*key, holders.clone()).await { - enqueued += 1; - } - } - - let mut stored = 0usize; - while started.elapsed() < FILL_DEADLINE { - stored = 0; - for key in &keys { - if storage.exists(key).unwrap_or(false) { - stored += 1; - } - } - if stored == CHUNKS { - break; - } - tokio::time::sleep(POLL).await; - } - let elapsed = started.elapsed(); - let trust_after: Vec = holders.iter().map(|h| p2p.peer_trust(h)).collect(); - - println!( - "FETCH_BUSY_AB slots={} chunks={CHUNKS} enqueued={enqueued} stored={stored} \ - elapsed_ms={} trust_before={trust_before:?} trust_after={trust_after:?}", - std::thread::available_parallelism().map_or(0, std::num::NonZero::get), - elapsed.as_millis(), - ); - - harness.teardown().await.expect("teardown"); -} - /// Nodes in the pool measurement's network. const POOL_NODES: usize = 25; /// A close group this wide makes every node responsible for every key, so From 277893a5638b10d477a835b7d7bca38f782c8e56 Mon Sep 17 00:00:00 2001 From: grumbach Date: Tue, 20 Oct 2026 19:12:08 +0900 Subject: [PATCH 07/10] test(replication): keep #252's silent holder silent now that refusals answer busy #252's a_holder_that_stops_answering_is_fetched_from_past_its_share closes the holder's fetch serve gate so that the holder answers nothing. With busy replies, that holder still answers busy to the requests it refuses. It is then not silent, and #252's per-source cap rightly applies to it. Add a test-only switch, turn_off_fetch_busy_replies_for_test, that makes a holder refuse silently. Use it there, so that the test again models a holder that has stopped answering. Also correct the A/B file's description of the test hooks it needs. --- src/replication/mod.rs | 24 ++++++++++++++++++++++++ tests/e2e/fetch_busy_ab.rs | 7 ++++--- tests/e2e/fetch_per_source_share.rs | 10 +++++++--- 3 files changed, 35 insertions(+), 6 deletions(-) diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 2d591610..a18084b8 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -1846,6 +1846,10 @@ pub struct ReplicationEngine { /// pool was full. #[cfg(any(test, feature = "test-utils"))] fetch_pool_full_refusals: Arc, + /// Test-only: refuse every fetch past admission silently, without a busy + /// reply, as a holder that has stopped answering would. + #[cfg(any(test, feature = "test-utils"))] + fetch_busy_replies_off: Arc, /// Permits for busy replies being sent to fetch requests refused at /// admission. fetch_busy_reply_semaphore: Arc, @@ -2022,6 +2026,8 @@ impl ReplicationEngine { fetch_serve_gate: Arc::new(FetchServeGate::default()), #[cfg(any(test, feature = "test-utils"))] fetch_pool_full_refusals: Arc::new(AtomicU64::new(0)), + #[cfg(any(test, feature = "test-utils"))] + fetch_busy_replies_off: Arc::new(AtomicBool::new(false)), fetch_busy_reply_semaphore: Arc::new(Semaphore::new(FETCH_BUSY_REPLY_MAX_OUTSTANDING)), fetch_busy_reply_limiter: Arc::new(FetchBusyReplyLimiter::new()), fetch_busy_sources, @@ -2298,6 +2304,13 @@ impl ReplicationEngine { self.fetch_pool_full_refusals.load(Ordering::Relaxed) } + /// Test-only: from now on, refuse fetches past admission silently, with no + /// busy reply, as a holder that has stopped answering would. + #[cfg(any(test, feature = "test-utils"))] + pub fn turn_off_fetch_busy_replies_for_test(&self) { + self.fetch_busy_replies_off.store(true, Ordering::Relaxed); + } + /// Test-only: whether `key` is still tracked in any fetch-pipeline stage /// (pending verification, fetch queue, or in-flight fetch). #[cfg(any(test, feature = "test-utils"))] @@ -3228,6 +3241,8 @@ impl ReplicationEngine { let fetch_serve_gate = Arc::clone(&self.fetch_serve_gate); #[cfg(any(test, feature = "test-utils"))] let fetch_pool_full_refusals = Arc::clone(&self.fetch_pool_full_refusals); + #[cfg(any(test, feature = "test-utils"))] + let fetch_busy_replies_off = Arc::clone(&self.fetch_busy_replies_off); let fetch_busy_reply_semaphore = Arc::clone(&self.fetch_busy_reply_semaphore); let fetch_busy_reply_limiter = Arc::clone(&self.fetch_busy_reply_limiter); let verification_responder_worker_semaphore = @@ -3290,6 +3305,8 @@ impl ReplicationEngine { fetch_serve_gate, #[cfg(any(test, feature = "test-utils"))] fetch_pool_full_refusals, + #[cfg(any(test, feature = "test-utils"))] + fetch_busy_replies_off, fetch_busy_reply_semaphore, fetch_busy_reply_limiter, verification_responder_worker_semaphore, @@ -4839,6 +4856,9 @@ struct ReplicationMessageHandlerContext { /// pool was full. #[cfg(any(test, feature = "test-utils"))] fetch_pool_full_refusals: Arc, + /// Test-only: refuse silently, without busy replies. + #[cfg(any(test, feature = "test-utils"))] + fetch_busy_replies_off: Arc, fetch_busy_reply_semaphore: Arc, fetch_busy_reply_limiter: Arc, verification_responder_worker_semaphore: Arc, @@ -7746,6 +7766,10 @@ fn fetch_busy_reply_permit( audit_metrics::record_fetch_busy_reply(FetchBusyReplyOutcome::NotAsked); return None; } + #[cfg(any(test, feature = "test-utils"))] + if ctx.fetch_busy_replies_off.load(Ordering::Relaxed) { + return None; + } // The send permit first: if none is free, the reply would not be sent, so // the rate allowance is left unspent. let Ok(permit) = Arc::clone(&ctx.fetch_busy_reply_semaphore).try_acquire_owned() else { diff --git a/tests/e2e/fetch_busy_ab.rs b/tests/e2e/fetch_busy_ab.rs index 4b570da3..36e537c2 100644 --- a/tests/e2e/fetch_busy_ab.rs +++ b/tests/e2e/fetch_busy_ab.rs @@ -6,9 +6,10 @@ //! Every requester stays within its two-request share at each holder, as //! #252's per-source scheduler keeps it, so the refusals measured are the ones //! left after #252: a holder whose twelve-slot pool is full of other peers' -//! requests. The file uses only the test-only fetch serve gate, which #252 -//! has too, so the same file measures #252 without the busy reply and with -//! it. +//! requests. Besides the test-only fetch serve gate, which #252 has too, it +//! reads two test-only counters, `fetch_admitted_for_test` and +//! `fetch_pool_full_refusals_for_test`. With those two patched in, the same +//! file measures #252 without the busy reply and with it. //! //! Run with `ANT_FETCH_BUSY_AB=1 cargo test --test e2e --features test-utils //! fetch_busy_ab -- --nocapture --test-threads=1`. Without the variable it diff --git a/tests/e2e/fetch_per_source_share.rs b/tests/e2e/fetch_per_source_share.rs index 7b9b87ef..b4c32d2a 100644 --- a/tests/e2e/fetch_per_source_share.rs +++ b/tests/e2e/fetch_per_source_share.rs @@ -2,8 +2,10 @@ //! holder than that holder will admit from it. //! //! A fetch responder admits two outstanding requests per requesting peer and -//! answers a third with nothing at all, so a requester that sends more than two -//! at once to one holder loses the rest to its own fetch timeout. The fetch +//! refuses a third. Before busy replies it answered that third with nothing +//! at all, and it still does when it cannot send a busy reply, so a requester +//! that sends more than two at once to one holder can lose the rest to its own +//! fetch timeout. The fetch //! worker used to do exactly that: it ran one fetch per slot and sent every one //! to the first verified holder of its key, so a node whose queued keys shared //! one holder had all of them outstanding there at once. @@ -113,8 +115,10 @@ async fn a_holder_that_stops_answering_is_fetched_from_past_its_share() { wait_until_fetch_pipeline_is_quiet(requester_engine).await; - // The holder answers none of these fetches, ever. + // The holder answers none of these fetches, ever: it holds the ones it + // admits, and refuses the rest without even a busy reply. holder_engine.close_fetch_serve_gate_for_test(); + holder_engine.turn_off_fetch_busy_replies_for_test(); let started = Instant::now(); for key in &keys { assert!( From 20eddd04913eeb73aa68eb3e0ec83a9081b2bd52 Mon Sep 17 00:00:00 2001 From: grumbach Date: Tue, 20 Oct 2026 19:45:24 +0900 Subject: [PATCH 08/10] docs(adr): ADR-0025 on top of #252, with the combined-build measurements Record how the busy reply works with #252's per-source scheduler: a cooling holder goes after every other free holder, a busy answer counts as an answer, and a busy wait counts against the holder's share. Replace the measurements with runs on the combined build and on #252 alone. Keep the earlier runs without #252 as history. --- docs/adr/ADR-0025-fetch-busy-reply.md | 33 ++++++++++++++++----------- 1 file changed, 20 insertions(+), 13 deletions(-) diff --git a/docs/adr/ADR-0025-fetch-busy-reply.md b/docs/adr/ADR-0025-fetch-busy-reply.md index 041ff0f2..9f57ba04 100644 --- a/docs/adr/ADR-0025-fetch-busy-reply.md +++ b/docs/adr/ADR-0025-fetch-busy-reply.md @@ -238,19 +238,26 @@ busy-reply allowance and stay silent. In the combined build: - a holder that keeps refusing the requester's fetch worker, one new key after another, costs nothing at the requester for eight busy answers, and is charged for the ninth only as that attempt's fetch timeout runs out; once it serves the worker a chunk, what it owes is - cleared. -- Measured on real nodes against upstream main a6e3388e with the same test-only gate, in - `tests/e2e/fetch_busy_ab.rs` (opt-in), with the neutral allowance in place: - - 20 requesters each within their two-request share, the first of five holders with its - 12-slot pool full for 2 s (26 to 28 refusals with this change, 28 on main), one of the - refusal cases left after #252: all 40 chunks in 10.1 s, 10.1 s and 9.4 s, no holder charged, - against one run on main at 90.3 s with 14, 9, 9, 4 and 3 requesters charging the five - holders (on the previous main 8d25f0c3 it was 60.4 s with 14 and 4 requesters charging the - first two). Two more runs with this change stopped before measuring, at the test's check that - every key was newly queued (39 of 40); - - one requester with 8 fetch slots over-asking three holders, which #252 removes: 38.3 s, 38.9 s - and 35.4 s with only the first holder charged (4 charges), against 90.2 s, 90.2 s and 90.3 s - with all three charged (6 charges on the first, 1 to 4 on the others). + cleared; + - #252's two per-source share tests pass on the combined build. Its silent-holder test turns + off that holder's busy replies with a test-only switch, so it still models a holder that + answers nothing: one that answers busy is answering, and stays under #252's cap. +- Measured on real nodes with the test-only gate, in `tests/e2e/fetch_busy_ab.rs` (opt-in): 20 + requesters each within their two-request share, and the first of five holders with its 12-slot + pool full for 2 s, the refusal case left after #252, with 28 refusals in every run below: + - the combined #252 + #253 build, on main b0e29c32: all 40 chunks in 10.1 s, 10.2 s and + 10.2 s, no holder charged; + - #252 alone (447c000d, with the file's two test-only counters added): 60.4 s, 60.5 s and + 60.8 s, with 14 requesters charging the first holder and 8 the second each time, and 0 to 7 + the others. + + One more run of each stopped before measuring, at the test's check that every key was newly + queued (38 of 40). +- Earlier, without #252, on main a6e3388e: the same pool case took 90.3 s on that main + (2026-10-16), with 14, 9, 9, 4 and 3 requesters charging the five holders, and 9.4 s to 10.5 s + with no holder charged with this change alone (2026-10-16 and 2026-10-20). A single requester + over-asking three holders, the case #252 removes, took 90.2 s on that main and 35 s to 39 s + with this change alone; that measurement left the file once #252 made the case impossible. - A local, unpublished discrete-event model of the 177-node join wave and 24 more scenarios informed the design, including the 5-10 s wait. Its source and raw output are not part of this change, so its results are not offered as validation. From 1a9a0f1f7834d71e03e49dd71958e63d7a674fb8 Mon Sep 17 00:00:00 2001 From: grumbach Date: Tue, 20 Oct 2026 19:56:46 +0900 Subject: [PATCH 09/10] docs(adr): ADR-0025 combined-build runs at 20eddd04, and exact earlier runs Quote the four pool runs made on the combined build after the last code change, and every earlier run in place of rounded ranges. --- docs/adr/ADR-0025-fetch-busy-reply.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/docs/adr/ADR-0025-fetch-busy-reply.md b/docs/adr/ADR-0025-fetch-busy-reply.md index 9f57ba04..fb3d01eb 100644 --- a/docs/adr/ADR-0025-fetch-busy-reply.md +++ b/docs/adr/ADR-0025-fetch-busy-reply.md @@ -245,19 +245,20 @@ busy-reply allowance and stay silent. In the combined build: - Measured on real nodes with the test-only gate, in `tests/e2e/fetch_busy_ab.rs` (opt-in): 20 requesters each within their two-request share, and the first of five holders with its 12-slot pool full for 2 s, the refusal case left after #252, with 28 refusals in every run below: - - the combined #252 + #253 build, on main b0e29c32: all 40 chunks in 10.1 s, 10.2 s and - 10.2 s, no holder charged; + - the combined #252 + #253 build, on main b0e29c32: all 40 chunks in 10.2 s, 10.0 s, 10.0 s + and 10.0 s, no holder charged; - #252 alone (447c000d, with the file's two test-only counters added): 60.4 s, 60.5 s and 60.8 s, with 14 requesters charging the first holder and 8 the second each time, and 0 to 7 the others. - One more run of each stopped before measuring, at the test's check that every key was newly - queued (38 of 40). -- Earlier, without #252, on main a6e3388e: the same pool case took 90.3 s on that main - (2026-10-16), with 14, 9, 9, 4 and 3 requesters charging the five holders, and 9.4 s to 10.5 s - with no holder charged with this change alone (2026-10-16 and 2026-10-20). A single requester - over-asking three holders, the case #252 removes, took 90.2 s on that main and 35 s to 39 s - with this change alone; that measurement left the file once #252 made the case impossible. + One more run of #252 alone stopped before measuring, at the test's check that every key was + newly queued (38 of 40). +- Earlier, without #252, on main a6e3388e: the same pool case took 90.3 s in one run on that + main (2026-10-16), with 14, 9, 9, 4 and 3 requesters charging the five holders, and 9.4 s to + 10.5 s in four runs, no holder charged, with this change alone (2026-10-16 and 2026-10-20). A + single requester over-asking three holders, the case #252 removes, took 90.2 s, 90.2 s and + 90.3 s on that main, and 35.4 s to 39.2 s in six runs with this change alone; that measurement + left the file once #252 made the case impossible. - A local, unpublished discrete-event model of the 177-node join wave and 24 more scenarios informed the design, including the 5-10 s wait. Its source and raw output are not part of this change, so its results are not offered as validation. From 7e9a2aa1f86a9ce31ac41bc14b9a2e2491136be8 Mon Sep 17 00:00:00 2001 From: grumbach Date: Tue, 20 Oct 2026 20:04:00 +0900 Subject: [PATCH 10/10] docs(adr): ADR-0025 says a cooling holder yields only to a free one Under #252's per-source share, a holder that just answered busy is passed over only for another free holder of the same key. --- docs/adr/ADR-0025-fetch-busy-reply.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/adr/ADR-0025-fetch-busy-reply.md b/docs/adr/ADR-0025-fetch-busy-reply.md index fb3d01eb..ff78d275 100644 --- a/docs/adr/ADR-0025-fetch-busy-reply.md +++ b/docs/adr/ADR-0025-fetch-busy-reply.md @@ -150,8 +150,8 @@ busy-reply allowance and stay silent. In the combined build: - When a refused requester receives `Busy` within the responder's neutral allowance, it moves on after 5-10 s, or sooner if its own fetch timeout runs out first, instead of 30 s, without charging the responder. -- A holder that just refused is not handed this requester's next keys while another holder that is - not cooling exists. +- A holder that just refused is not chosen for a key while another free holder for that key is + not cooling. - Old nodes decode exactly the messages they always did: a request carries 5 bytes after the message that they never read, and they are never sent `Busy`. Rollback is a release without the change.