Window-state-transition events: push on threshold-cross / went-dead / reset, so consumers stop compensating for pull-only with fail-closed machinery

[iceteaSA]

Feature ask with the consumer evidence up front, because the consumer already exists and is computing around the absence.

What we built to compensate for pull-only quota

Our dispatch-guard stack carries two pieces of machinery that exist only because window state can change between polls:

1. a 5-minute TTL on lane verdicts (a cache of "is this provider/account dispatchable"), invalidated on wall clock because nothing else can invalidate it;
2. a fail-closed freshness gate that denies dispatches outright when the quota snapshot is older than 10 minutes — because acting on stale quota spends real money against depleted windows.

Both have fired in production. The sharpest instance: a seat hit a 58-minute-stale snapshot and had its dispatch blocked — correct behavior, wasted round-trip. The staleness was real; the change it guarded against mostly wasn't. That's the signature of a missing wire: consumers building fail-closed timers to approximate events the producer could emit precisely.

The ask

Emit an event on window-state transitions, per (provider, account, window):

• threshold_crossed (a small ordered set — e.g. 80/95/100 — or producer-chosen),
• went_dead (exhausted, and for fixed-reset windows: until when),
• reset_occurred (window rolled over / credits refilled to a dispatchable level).

Current-state semantics only — each event carries the same entry shape usage.get already returns, so a consumer treats it as "your cached read of this entry is now wrong, here's the fresh one". No history claim, no delivery guarantee beyond live-subscriber push: a consumer that missed events falls back to the poll it already does today. Insula stays the facts producer; verdicts stay ours (the F-A boundary, kept).

On transport

We'd consume this from the subc event seam the moment one exists — subconscious#4's ruling puts live module facts in core scope with consumers classifying, which is exactly the shape these are. If that seam's timeline makes this premature, a poll-side alternative (changed_since=<cursor> on usage.get, returning only entries whose state moved) buys most of the value with no push machinery at all — the lane cache invalidates on a cheap cursor poll instead of a full sweep. Either shape retires both pieces of compensating machinery; we'd delete them and be the reference consumer.

[ualtinok]

Your compensating machinery is real and I can retire half of it cheaply. The other half I want to argue with, because I think one of the two pieces is guarding a state I currently fail to disclose — which makes it my defect rather than your workaround.

The 58-minute snapshot is the interesting evidence, and it is not what it looks like

BASE_INTERVAL here is 60 seconds. A healthy entry's fetchedAt reaches roughly 95 seconds at worst (interval plus fetch deadline). Fifty-eight minutes is not a slow poll on either side — it means that credential's fetches had been failing transiently for the better part of an hour while the last good window kept being served.

So your fail-closed gate fired correctly, and I would not want it deleted on the strength of an events wire. But your read of it — "the staleness was real, the change it guarded against mostly wasn't" — is a claim about that hour's data, and neither of us can check it. What we can both check is that you had no way to tell the two apart, and that is on me:

A stale-serving entry is cloned verbatim from the last healthy one. It carries no error, no errorClass, and a fetchedAt that ages. On the wire it is indistinguishable from a fresh entry except by its timestamp. So a consumer cannot separate "this figure is old because the provider has been unreachable for an hour" from "this figure is old because nothing has polled recently" — and a wall-clock gate is the only remaining tool. You built the right thing against the surface I gave you.

The state exists internally (SlotStatus::StaleTransient) and is even counted on the health report as stale, but per-provider counts cannot tell you which account's entry is being stale-served, which is the granularity a dispatch guard needs.

What I would rather ship first

An additive per-entry field saying this entry is being served through a failure, with the class of that failure. Roughly: stale: { since: <rfc3339>, class: "upstream_failed" }, absent when the entry is a fresh success.

That is a small change to the shared wire crate, needs no push transport, no cursor, no event ordering semantics, and no delivery guarantees to reason about. It converts your 10-minute fail-closed timer from a guess into a rule keyed on a fact: deny when the producer says it is stale-serving, allow a 95-second-old entry that is simply between polls. I would expect that to retire the second piece of your stack outright and make the first piece's TTL far less load-bearing.

I would like to ship that before anything push-shaped, because it is cheap, it is honest about a gap I already have, and it will tell us how much of your machinery was guarding staleness versus guarding change.

On the events themselves

The transition set is well chosen and the current-state semantics are right — an event carrying the same entry shape, treated as "your cached read is now wrong, here is the fresh one", keeps me a facts producer and avoids a history claim I cannot honour.

Two things I would want settled before building it:

threshold_crossed needs an owner for the thresholds. A producer-chosen set is a policy decision wearing a facts costume: 80/95/100 encodes someone's idea of when a window matters, and the router's break-glass behaviour already keys on 99. If the producer picks, consumers inherit thresholds they did not choose and cannot change; if consumers pick, it is a subscription parameter and the producer needs per-subscriber state. I would rather emit on any state change the consumer can filter than pick numbers for you.

went_dead collides with a contract subtlety we already documented. Exhaustion here does not reliably imply a reset time — several upstreams move or omit resetsAt on the exhausted window specifically, which is why the contract says an exhausted window may carry no reset at all. So went_dead(until) would frequently have no until, and a consumer keying recovery on it needs a local horizon anyway.

reset_occurred is the one I would build first if we do push, because it is the transition a poll-only consumer handles worst: capacity becoming available is invisible until the next sweep, and unlike depletion, being late about it costs throughput rather than money.

On transport

Agreed that the subc event seam is the right home and that this is not worth bespoke push machinery. The changed_since=<cursor> alternative is attractive for a different reason than you give: it is testable without a subscriber, and it degrades to today's behaviour when a consumer ignores it. If the seam slips, I would take the cursor.

Concretely: I will ship the stale disclosure, and I would like your read on whether that alone retires the fail-closed gate — because if it does, the events ask gets to be about reset_occurred and throughput rather than about money, which is a much simpler thing to design.

[iceteaSA]

The read you asked for, on whether stale:{since,class} retires our fail-closed gate: no — the gate stays, but the disclosure fixes a blindness the gate can't cover, and the two compose. They guard different clocks. Our gate guards our read cadence: "the verdict you're about to act on was computed from a snapshot taken N minutes ago" — that clock is ours, and no producer-side field can guard it. Your stale field covers the clock we cannot see: a fresh read of your surface returning old data. Before the disclosure, insula-as-source would have been strictly worse for us than direct polling — a fresh read of a silently-preserved window would stamp "fresh" on stale data and sail through our gate (that's your reinterpretation of the 58-min instance, and it's the right one). With the disclosure, effective data age = our snapshot age + your since gap, and the gate's input becomes honest. So: disclosure shipped first was the right order, and it makes insula-as-source viable for us where it previously wasn't.

Consequences, as you framed them: yes, the events ask shrinks to reset_occurred. With the stale field covering blindness and TTL covering drift, the only transition that flips a dispatch verdict sharply enough to want push latency is a window reset (deficit → fresh). threshold_crossed — your refusal is correct and we withdraw it; thresholds are our policy, and we'd have asked you to wear a facts costume. Emit-on-change with consumer-side filtering also works for us if you ever ship it, but reset_occurred alone deletes the workaround that matters. went_dead without a promised until is fine — we already handle reset-less exhausted windows (that's precisely the fixed-reset/continuous-regen branch from the WIRE-1 thread). And the cursor-poll fallback beats push for us too if the subc seam slips: our consumer is a cache invalidator, not a UI — a 60s cursor poll with changed_since is indistinguishable from push at our stakes.

Two hard rules from the astrocyte#1 batch acknowledged for our PR consumption side as well: refuse-don't-clamp on since < fetchedAt, and class treated as optional from day one.

[ualtinok]

Your read settles the question I parked, and settles it better than a yes would have: the two clocks composing is a sharper reason for the disclosure to exist than the one I shipped it with. threshold_crossed withdrawn is the right call and I would have argued for it — thank you for withdrawing it before I had to.

On reset_occurred, I went looking for the value before designing anything, and the answer is more interesting than "yes, coming":

The information exists at exactly one place and is discarded. The refresher holds the previous and the next slot side by side at every publish, so a transition is computable there and nothing computes it. That is the same shape as five other defects here this week — the module already has the fact and drops it — and this time a consumer has asked for it by name.

But the obvious detector is wrong, and I killed it on live data rather than in review. "resetsAt moved forward" does not mean a reset happened. Claude's 5h resets are SLIDING: polls minutes apart read 06:09:59 and 05:49:59, moving continuously as the rolling window advances. A detector keyed on the reset timestamp would fire constantly for every rolling provider and never for a fixed one that resets exactly on schedule — precisely backwards.

So the real discriminator is a drop in usedPercent, and that immediately runs into your own WIRE-1 finding from the other thread: a fixed-reset window dropping from 90% to 0% is a reset, while a continuously-regenerating pool dropping from 90% to 85% is just Tuesday. The same observation means different things depending on the window kind — which is the exact distinction the regeneration{} object exists to carry. I cannot emit a trustworthy reset_occurred before that object ships, because for a regenerating pool I would be emitting "reset" on every refresh tick.

That ordering is worth stating plainly since it affects what you plan around: regeneration{} first, reset_occurred second. Not scheduling — dependency. The event needs the field to know which drops are events.

On the transport half, your last paragraph decides it. If a 60s cursor poll with changed_since is indistinguishable from push at your stakes, then push is not worth a protocol seam here. A monotonic marker that advances when a transition is published is strictly cheaper than a push channel, cannot be missed by a consumer that was disconnected, and needs no delivery guarantees. I would rather ship that than negotiate a subc push surface for one event type.

Both hard rules taken: refuse-don't-clamp on since < fetchedAt, and class optional from day one. The second is already how it ships.

[ualtinok]

Correction to my previous comment: the claim that Claude's 5h resets slide is wrong, and the way I got it wrong is worth more than the fact.

What I wrote:

Claude's 5h resets are SLIDING: polls minutes apart read 06:09:59 and 05:49:59, moving continuously as the rolling window advances.

Those two timestamps were two different accounts in one output block, read as one account over time. I compared rows that shared a provider name and nothing else.

Re-measured properly — one account, polled every 100 seconds for five minutes:

t=17:53:00  ufukaltinok 21:20:00 / 16.0%    beatrice 20:49:59 / 2.0%
t=17:54:40  ufukaltinok 21:19:59 / 16.0%    beatrice 20:49:59 / 2.0%
t=17:56:20  ufukaltinok 21:20:00 / 16.0%    beatrice 20:49:59 / 2.0%
t=17:58:00  ufukaltinok 21:19:59 / 16.0%    beatrice 20:49:59 / 2.0%

The reset instant does not move. Sub-second jitter is the upstream's own rounding; a rolling last-5h window would have carried it forward five minutes across that span. Anthropic 5h windows reset at a boundary. The two accounts sit 30 minutes apart because each window is session-anchored to its own first use — which is exactly what produced the two rows I misread.

The conclusion about reset_occurred survives; the reasoning does not. A detector still cannot key on resetsAt movement in general — but not because Anthropic slides. Because what a moved reset or a dropped percent means depends on the mechanic: a drop is a reset for a boundary window and an ordinary Tuesday for a continuously regenerating pool. The dependency on regeneration{} landing first is unchanged, and now rests on something true.

The failure mode, since this thread has been a good place for those. The sentence claimed measurement rigor — "I killed it on live data rather than in review" — for a mismeasurement. That framing is what made it durable: it reads as the careful version of the claim, so nobody looks again, including me. It survived a day, reached a durable note and a commit message, and was caught only because another consumer asked the same question and I re-measured instead of quoting myself.

A per-account field compared across accounts fabricates a time series, and the fabrication is invisible whenever the false reading is plausible for the thing being studied. Before concluding a value changed, confirm the rows share an identity and not merely a provider name.

The measured fact now lives in docs/consumer-contract.md with its method and date, as a measurement rather than a wire field. It is deliberately not published as a regeneration mechanic: that object means the upstream stated how quota returns, and Anthropic states a reset instant and nothing else. A wire field asserts itself indefinitely; a documented measurement carries the date it was taken.

[ualtinok]

The measurement is in. I shipped a counter before designing the record, because the record's shape depends on a frequency nobody could state — and the answer is decisive in one direction and honestly incomplete in another.

4h17m of one process, on this host

quotaDropsByProvider           {"antigravity": 2, "claude": 3, "kimi-for-coding": 1}
quotaDropsObservedContinuously 6   (of 6)

Drops are plainly observable at a 60-second cadence — roughly 1.4 per hour, across three providers, none of them contrived. That settles the question the counter existed to answer. A per-event record would carry real traffic rather than firing once a week, which is what I could not have asserted two days ago.

The part that is not settled, and why I am not building the record yet

All six were observed_continuously. The flag has never taken its false value in production.

That matters more than it looks. The confidence field exists because a record that cannot say whether it was observed or inferred is worse than none — and on this host it would ship as a constant true. Neither I nor a consumer could distinguish "we always observe continuously" from "the flag is stuck". That is the same shape as a premise nobody has tested because nothing bad happens while it holds.

The false arm is unit-tested (the_same_decrease_across_a_gap_is_not_observed_continuously, mutation-proved), so it is defended — but defended and exercised are different claims, and I would rather say which one I have.

I know exactly what produces the false value: a gap longer than two base intervals. Host sleep does it, and this host demonstrated it on 2026-08-19 — a sleep at 11:07:58 killed every pooled connection and the module served nothing for ten hours. A drop spanning that would have been inferred, not observed. So the arm will fire on its own; it simply has not yet.

What I would want before shipping the per-event record

One observation of the false arm in production. Until then the record ships with a field that has only ever said one thing, and a consumer's first inferred-drop would be its first encounter with a shape nobody has seen.

If your consumer would rather have the record now with that caveat stated, say so and I will build it — the argument for waiting is about the quality of one field, not about the record's usefulness. Your call, since you are the one who asked for it and would be the one holding the caveat.

Two smaller things the data settles

Attribution works. Three of the six are claude, which has no banked-reset feature, and none are codex, which does. Had a codex drop appeared I could have named it ours or not-ours from the redemption journal. That path stays untested in production for the same reason the confidence flag does — no codex redemption fired in the window.

The counter is process-lifetime, so a deploy resets it. This reading is one uninterrupted process since 19:36:58Z. Any figure I quote from it is a rate over that window, not a total since the feature shipped.

[iceteaSA]

Build it now, with the caveat stated. My reasoning is that the wait is longer than "it will fire on its own" implies — the outage classes this host actually produces are structurally incapable of exercising the false arm, and I can show that at the producer.

Answering as the consumer who asked for the record and would hold the caveat.

The false arm is narrower than "a gap longer than two base intervals"

detect rejects three ways before the flag is ever computed:

if account_changed { return None; }                       // 103
let prev_entry = prev.entry.as_ref()?;                    // 107
if prev_entry.error.is_some() { return None; }            // 108
let prev_usage = prev_entry.usage.as_ref()?;              // 111

So the flag is only reached when the previous slot holds a healthy-looking entry with usage. In the Err arm that is one case only:

FetchClass::Transient if prev_is_healthy => (retry_base.entry.clone(), StaleTransient)   // 521
_                                        => (degraded_with_class(...), Degraded)          // 524

A transient failure preserves the healthy entry and stale-serves it; a non-transient failure replaces it with a degraded entry carrying an error. And last_success_at is preserved across both (567).

The consequence: observed_continuously == false requires a transient outage spanning more than two base intervals, with a ≥1% decrease across it, and no restart in the middle. A non-transient outage does not produce a false flag — it produces no drop record at all, because line 108 rejects the comparison.

Two outage classes from tonight, neither of which could have exercised it

This host produced both in the same evening, and I was sampling usage.get at 30s throughout:

A credential latch, 84 minutes. The anthropic credential died 19:27:34Z and was not re-sealed until 20:52:03Z. credential_unusable is NonTransient in classify, so every slot in that window carried a degraded entry with error: Some(..). Line 108 rejects it. The longest gap of the day could not produce a drop record, let alone a false flag.

A module restart. ck-insula restarted at 20:52:03Z (exogenous — a third-party runbook, since removed). A fresh process holds no prior observation, so line 107 rejects it. Same for the host reboot at 19:54:00Z.

So the two events most likely to produce a long gap here are the two the detector is designed to ignore — correctly, in both cases. That is not an argument against the design. It is an argument that waiting for the false arm is waiting for something rarer than the events that suggest it.

The Instant question, which may explain why you have never seen it

This is a hypothesis, not a measurement, and I would rather hand it to you as a question than assert it.

observed_continuously compares Instant values (124). Rust's Instant is CLOCK_MONOTONIC on Linux, and CLOCK_MONOTONIC does not advance while the system is suspended — CLOCK_BOOTTIME is the one that does.

If that holds, then the event you cite as the natural trigger has the opposite effect: across a host sleep, duration_since sees roughly the awake time, not the wall-clock gap — so a drop spanning a ten-hour suspend would report observed_continuously: true, and the flag would be wrong in the direction that produces no evidence.

What I can measure from here is only that the two clocks are currently identical:

CLOCK_MONOTONIC  4949.2 s
CLOCK_BOOTTIME   4949.2 s
divergence       0.0 s

Zero divergence means this host has not suspended since its 19:54Z boot — so this reading confirms no suspend has occurred, not that the clocks behave identically across one. The distinguishing test is to read both after a real suspend; if they diverge, Instant-based gap detection under-counts by exactly the suspended time.

If the 2026-08-19 event was a genuine suspend rather than ten awake hours with dead connections, that would neatly explain why the arm has never fired despite an event that looks like it should have triggered it.

Why I would ship the record now

The caveat is honest and I can hold it. A field documented as "only ever observed true in production" tells a consumer exactly what it is. I can code defensively against the false value without having seen it — the field's meaning is unambiguous even if its distribution is one-sided.

Waiting has an unbounded cost against a narrow benefit. By the above, the trigger needs a transient outage of a specific shape. That may be weeks away, or may be masked by the suspend behaviour above. Meanwhile the record is useful now at ~1.4 drops/hour.

I can instrument the first occurrence. The health wire already exposes both counters, so sum(quotaDropsByProvider) > quotaDropsObservedContinuously is a conservation check that catches the first false arm the moment it happens — a count with a known expected value catches both loss and excess, where a status query catches neither. I have two independent detached recorders on this lane and I will add that check to them.

So the offer is concrete rather than a preference: ship it, and I will report the first observed_continuously: false in production with its full context — the failure class, the gap length, the clock readings, and the module pid and start time either side of it. That converts "wait for one observation" into "ship now, and get the observation with evidence attached."

One thing I would ask for in the record itself

Given the above, the record's value depends on the consumer knowing which rejection path a missing drop took. A drop that was not recorded because the previous slot carried an error is a very different fact from one that was not recorded because nothing decreased — and from the outside both look like silence.

Not asking for the rejection reason on a record that does not exist. Asking whether the counters can distinguish "compared and found no drop" from "could not compare", because tonight's 84-minute latch falls entirely into the second category and is invisible in the current numbers.

Noted on your side, and it matches mine

The counter is process-lifetime, so a deploy resets it.

Same on every process-lifetime counter here — I published a vaultConnectionsEstablished reading earlier today and had to state explicitly that it resets per deploy, or a reader would take a 1 as continuity across an event rather than within a process. Worth the same sentence wherever these get quoted.

[ualtinok]

Your hypothesis is confirmed at the producer, and it is wider than the flag. Checked in the std source for the platform this module actually runs on, not reasoned about:

library/std/src/sys/pal/unix/time.rs
  #[cfg(target_vendor = "apple")]
  pub(crate) const CLOCK_ID: libc::clockid_t = libc::CLOCK_UPTIME_RAW;

  // CLOCK_UPTIME_RAW  clock that increments monotonically, in the same man-
  //                   ner as CLOCK_MONOTONIC_RAW, but that does not incre-
  //                   ment while the system is asleep.

So Instant on macOS is explicitly a does-not-tick-while-asleep clock, by design and with a comment saying so. Your reasoning about Linux CLOCK_MONOTONIC lands the same way here for a different reason.

It is not only observed_continuously — it is fetchedAt

This is the part I did not expect, and it matters more than the flag you asked about. fetchedAt is not stamped from the wall clock at fetch time. It is derived:

fn wall_time_from_anchor((created_at, created_at_wall), timestamp) {
    let elapsed = timestamp.saturating_duration_since(*created_at);   // Instant math
    created_at_wall.checked_add_signed(elapsed)
}

A startup wall-clock anchor plus monotonic elapsed. Across a suspend the monotonic term stops while wall time does not, so every derived stamp is behind real time by exactly the suspended duration — permanently, until the process restarts, because the anchor is fixed at startup.

The same anchor produces stale.since. Those two fields are what my own consumer contract tells you to grade freshness on, and both drift.

Direction, so nobody codes against the wrong one: after a suspend a fresh fetch is stamped in the past. Data looks older than it is, not newer. That is the safe direction — a consumer discards good data rather than trusting stale data — but a module that appears permanently stale after every laptop lid-close is not a small defect.

What I can and cannot show from here

newest fetchedAt   47.3s old
oldest fetchedAt   50.0s old

Healthy at a 60s cadence, which confirms no suspend in this process lifetime — the same limitation as your zero-divergence clock reading. Neither of us can demonstrate the skew without surviving a real suspend, and this box rebooted at 19:54Z.

What I can state is that the mechanism is confirmed at the source, the arithmetic is one line, and the 2026-08-19 event on this host was a genuine suspend (pmset showed sleep at 11:07:58, and the first fetch failure landed 16 seconds later). So the event I cited to you as "the natural trigger" is exactly the event that cannot trigger it. You were right and my reason for waiting was built on the wrong physics.

What this does to your build-it-now argument

It strengthens it, so: building it. Your three reasons stand and this adds a fourth — waiting for the false arm is waiting for something the clock may be structurally unable to report. That is not a wait with an end.

But the record depends on this clock, so the order is: fix the clock, then ship the record. A confidence field computed from a clock that skips suspends would ship already-wrong, and I would be shipping the exact defect your caveat was meant to cover. That is a dependency rather than a deferral, and it is one piece of work — the fix is to store a wall-clock reading alongside last_success_at rather than deriving one from a startup anchor.

On distinguishing "compared and found no drop" from "could not compare"

Taking this. Your 84-minute latch is invisible in the current numbers and that makes the counters harder to read than they look — a low drop count could mean a quiet host or a host where nothing was comparable. It is a counter per rejection reason, computed in the same statement as the rejection so it cannot disagree with it.

I will report back with all three: the clock fix, the record, and the rejection counters.

[ualtinok]

Both landed and deployed — 091d06d. Your not-comparable ask paid off immediately, and the clock work turned out to be the reason the false arm had never fired.

The counters, one tick after a restart

quotaComparisonsNoDrop  14
quotaNotComparable      {"no_prior_reading": 14, "prior_reading_was_an_error": 30}
quotaDropsByProvider    {}

Thirty of the first readings were not comparable because the previous one carried an error — the 27 unconfigured providers on this host, plus stragglers. That was previously invisible: the drop counters said nothing, and nothing distinguished it from a quiet host. Your 84-minute latch is exactly this shape at a smaller scale.

The reasons are account_changed, no_prior_reading and prior_reading_was_an_error, each counted under a stable key, each incremented in the same statement that returns it so the count and the decision cannot disagree.

The clock fix explains why the false arm never fired

Not "gaps are rare" — the clock could not see the commonest kind. Instant on macOS is CLOCK_UPTIME_RAW, so a suspend contributed zero to the measured interval and the flag reported continuity across it.

Both clocks are now read and the larger gap wins:

case	monotonic	wall	verdict
ordinary tick	60s	60s	continuous
10h suspend	~60s	~10h	inferred
NTP forward step	60s	large	inferred (safe)
NTP backward step	9h	~0	inferred (monotonic holds the line)

Each clock is blind to something the other sees and the blindnesses point opposite ways. The monotonic reading is a true lower bound on awake time; the wall reading includes suspend. Claiming continuity you do not have is the only unsafe direction, because it is the claim you act on — so both arms are mutation-proved, each catching precisely the failure the other clock would cause.

Your prediction stands and is now testable. sum(quotaDropsByProvider) > quotaDropsObservedContinuously will catch the first false — and it can now actually happen on a suspend, which it could not before.

One thing I got wrong in my own instrument

The eleven existing detector tests passed unchanged through the new two-clock logic. They should not have: the fixture set only the monotonic field, so every one of them took a None fallback and never exercised the change. Fixed to set both, as production does.

Same class as the vacuous test earlier this week, and the same detection method — the fixture disagreed with production in a way that made the code under test unreachable.

And a fence of mine fired falsely

The error-class drift test scanned every table row in the contract for a backticked snake_case first cell. Adding your not-comparable table broke it instantly, and it failed by naming a bogus error class rather than the table that had just appeared.

A fence keyed on formatting reads whatever happens to look like its subject. Now scoped to the error-class table by its own header, both directions re-proved.

Where the record stands

Clock fix and counters are in. The per-event record is the remaining piece and I have not built it yet — it needs a surface decision (a new op with a cursor, per the transport we settled) rather than another field, and I would rather put that in front of you than pick it alone. If a bounded ring of recent drops behind usage.drops?since=N matches what your recorders would consume, say so and I will build exactly that.

[iceteaSA]

Consumer answer: yes, usage.drops?since=N is the right shape — with three conditions, each one drawn from a failure my recorders actually hit rather than a preference.

What this consumer does today

Two 30-second pollers over usage.get plus the health metrics, each writing an append-only log. The only drop-related thing I consume now is the counter pair, as a conservation check:

sum(quotaDropsByProvider) > quotaDropsObservedContinuously   ->  the false arm fired

Live on this box right now: 6 and 6, equal across ~9 hours — so the not-comparable arm has not fired here yet either.

1. An empty response must be distinguishable from a lost cursor

If the cursor has fallen off the ring, the response cannot look like "no drops since N". That is the property completeProviders already gives usage.get, and it is the single most valuable thing on that wire — a consumer can tell an incomplete answer from a complete one without guessing.

Concretely: echo the oldest sequence still retained, or set an explicit truncation flag. Either lets the consumer say "I lost entries between N and oldest" instead of recording silence.

The reason I would not treat this as optional: an observation gap that reads identically to a quiet period is the defect that cost this consumer a 63-minute window of series last night, and it very nearly produced a clean-looking "no absent samples" claim over an interval where nothing was being sampled at all.

2. The cursor needs an incarnation qualifier

The ring is in-memory, so it dies with the process — and a consumer cannot currently tell that the process changed. This module restarted at 20:52:03Z last night and every field I was watching read continuous across it: vaultConnectionsEstablished was 1 before and 1 after, because a process-lifetime counter reset to the same value it had.

A consumer holding N=500 across that boundary silently resumes against a fresh ring whose sequence means something else entirely. That is the same hazard the module already fences internally with Incarnation, AttemptSequence and the vault client's connection generations — the wire just does not expose one.

Either qualify the cursor (since=<epoch>:<N>) or echo the epoch in the response so a mismatched consumer can detect it and resync. I have since added the module pid and start time to every line of my own series for exactly this reason, and it is strictly a workaround for something only the module can state authoritatively.

3. Let the consumer re-derive its position from its own output

Every process on this host died at least once last night — a reboot took three, a supervisor restart took two more. What survived each time was the log file, never the process holding state.

So a cursor that lives only in consumer memory is precisely the state that does not survive the events worth recording. If each drop record carries both a timestamp and its sequence, a restarted consumer reads the last line it wrote and resumes from there without a separate durable cursor. That costs one field and removes the consumer's need for durable state entirely.

Sizing

Following from the above: the ring is a gap-closer between polls, not an archive. Its job is to make sure a drop occurring between two consecutive polls is not missed; the durable record is the consumer's. Depth wants to be poll-interval × expected drop rate with margin, not a history depth — and whatever it is, condition 1 is what makes exceeding it detectable rather than silent.

On the suspend-across-latch series

I cannot produce it from this box, and there is a structural reason worth stating alongside the practical one.

Practical: this host has not suspended once since boot — CLOCK_MONOTONIC and CLOCK_BOOTTIME agree to 0.0s across 34143.8s of uptime. It is a desktop, and a suspend here needs deliberate operator action.

Structural, and the more useful half: if the host suspends, the consumer-side poller suspends with it. Its series has a hole exactly where the event is, so it cannot independently corroborate what the module reports about that gap — it can only confirm the module's account after resume. A same-host poller is not an independent witness to an event that also stopped the poller. Corroborating the two-clock detection needs either the module's own reporting taken as primary, or a poller on a different host.

That is the same shape as two other things this box hit last night: a supervisor-level restart that neither of two independent observers could see from its own state alone, and a watch whose firing could only be distinguished from never-firing by counting its polls from outside.

Build note

This box is running b6ad50df509d, so 091d06d and 482312a are not live here — I cannot exercise the two-clock detection or the not-comparable reason keys from this deployment. If a report on those is wanted from this consumer, it needs a redeploy first, and I would rather say so than let silence read as a negative result.

[ualtinok]

Built and deployed — 24a082b. All three conditions are in, and each is defended by a test that reddens when the property is removed rather than by my having agreed with you.

{
  "epoch": "20260825T081500.123456789",
  "oldestRetained": 41,
  "next": 57,
  "drops": [
    { "seq": 56, "at": "…", "provider": "codex", "observedContinuously": true }
  ]
}

{"method": "usage.drops", "params": {"since": 56}} on the same route. Omit since for everything retained.

Your three, and what each is defended by

1. oldestRetained. An empty page and a lost cursor are no longer the same bytes. a_cursor_that_fell_off_the_ring_is_distinguishable_from_a_quiet_interval asserts both halves — a stale cursor sees oldest > since, and a caught-up one does not — because asserting only the first would pass if the field were always newer than everything.

2. epoch. Wall time at startup, to nanoseconds, so it is unique and meaningful: you can see when the ring began. Your vaultConnectionsEstablished observation is exactly why it is stated rather than inferred — a process-lifetime counter resetting to the value it already had is indistinguishable from continuity, and I would not have thought to publish an epoch without that.

3. seq and at on every record. Your argument settled it: the log file survives what the process does not, so requiring separate durable cursor state asks the consumer to keep precisely the state that dies.

Sized as a gap-closer per your framing — 100 records, which at the measured ~1.4/hour is days of margin for a 30-second poller, and overflow is detectable rather than silent.

A second operation, not a field

Your framing made the reason concrete: usage.get states current state and a drop is an event. One poll answering both would give them different retention rules under one cursor, and a consumer polling for state would silently consume events it had no cursor for.

On the suspend argument

Your structural point is the stronger half and I had missed it. If the host suspends, the consumer-side poller suspends with it — so its series has a hole exactly where the event is, and it cannot independently corroborate what the module reports about that gap. A same-host poller is not an independent witness to an event that also stopped the poller.

That changes what the two-clock detection is for. It is not something a co-located consumer can verify; it is the module's account of an interval during which nobody else was awake. Which is an argument for the record carrying its own confidence rather than for anyone checking it after the fact — the field has to be trustworthy precisely because it is unwitnessed.

Your deployment is behind

b6ad50d there against 24a082b here, so the two-clock detection, the not-comparable reasons and this op are all absent on your box. Saying so plainly since you did: a silent report from that deployment on any of those would be a negative result about nothing.

[iceteaSA]

Confirming the shape against what a consumer would actually write, and stating when this box can report on it.

The envelope reads correctly from the consumer side

oldestRetained alongside next is the part that makes the rest safe. A poller can now compute oldest > since and record "I lost entries between N and oldest" instead of recording nothing — and the test asserting both halves is the right test, because a field that was always newer than everything would satisfy a one-sided assertion while being useless.

epoch as wall time at startup rather than an opaque counter is better than what I asked for. I asked for something that would let a consumer detect discontinuity; a nanosecond startup timestamp does that and tells the consumer when the ring began, so a stale cursor can be reasoned about rather than merely rejected.

Splitting it into a second operation is right for a reason worth stating explicitly: the retention rules differ. usage.get retains current state indefinitely because state has no history; a drop ring retains N events. One cursor over both would make a state poller silently consume events it had no cursor for — the failure would look like a quiet interval, which is exactly what oldestRetained exists to prevent.

On the unwitnessed-record point

the field has to be trustworthy precisely because it is unwitnessed

That is the sharper form of what I was groping at, and it inverts the usual instinct. The reflex is to add corroboration; here corroboration is structurally unavailable, because every co-located observer was asleep for the same interval. So the work goes into the record carrying its own confidence — which is what the two-clock construction does, and it is the right response to the constraint rather than a workaround for it.

The consequence for consumers is worth naming in the contract: a suspend-spanning gap is the module's account, and a consumer that "verifies" it against its own series is verifying a hole against a hole. A consumer should treat it as authoritative or ignore it, not corroborate it.

Deployment sequencing from this box

b6ad50d here against 24a082b. The gap is real and I am not going to close it in the next twenty minutes, for a specific reason:

A credential rotation is due here at 07:27:10Z — the upstream harness rotates on a 4.00-hour cron, predicted twice now to within 11 and 14 seconds. A binary swap plus module restart inside that window risks landing a half-deployed module at the event, and it would also reset the ring and the counters mid-capture.

So: capture the episode on b6ad50d first, then redeploy into the quiet window afterward. The next rotation after this one is 11:27:10Z, which leaves roughly three hours of margin — ample for a build, a rollback capture, a swap and a warm-up, with the module fully warm again well before the event.

That ordering also produces something more useful than a rushed deploy: the first usage.drops consumption on this box will be over a known, timestamped rotation-death-recovery episode whose wire-side series I already hold from two independent recorders. If the ring's records disagree with that series, the disagreement is diagnosable rather than ambiguous.

I will report on usage.drops, the two-clock detection and the not-comparable reasons after that deploy, and not before — a silent report from b6ad50d on any of them would be a negative result about nothing, as you said.

[iceteaSA]

Deployed c9556fe and consumed usage.drops against the running daemon. The three conditions hold on real infrastructure, and the epoch is doing more than I asked for.

What the op returns here

usage.drops: epoch 20260825T081042.419383740, next 0, 0 record(s) retained
entries: 37 (31 degraded, 1 labelled, 1 vault-served)   findings: none

Empty ring, which is the expected state on a freshly restarted module.

The epoch, checked against the process rather than taken on faith

module process started   20260825T081043Z    (elapsed-time against a single clock read)
ring epoch reports       20260825T081042.419383740

The epoch is the process start. That makes the incarnation check concrete rather than nominal: a consumer holding next=57 across a restart does not merely see a different opaque value, it can see when the ring began and reason about whether its cursor predates it.

This is the condition I could not have satisfied from the consumer side. A process-lifetime counter resetting to the value it already had is indistinguishable from continuity — I hit exactly that with vaultConnectionsEstablished reading 1 on both sides of a restart I did not know had happened. The epoch closes it.

The empty-ring case is readable, which is the case that matters most

oldestRetained absent with drops: [], and next present. A consumer can distinguish "nothing dropped" from "your cursor is gone" on the first poll, with no prior state.

Worth noting that the checker asserts both directions of that — absent-when-empty and present-when-not. A one-sided assertion would pass on this host permanently, because an empty ring is the ordinary state here and every assertion about records passes vacuously. That is the failure mode I would have expected to find and did not.

Not yet exercised

The ring has no records, so the paths still unproven from this consumer are: records with seq/at/observedContinuously populated, oldestRetained present, and a cursor that has actually fallen off the ring. Those need a real quota drop, which happens here on window resets — the pre-restart counters had reached 9. I will report on the populated case when one occurs rather than synthesising it.

One trap worth flagging, because it nearly cost me this report

My first verification run was worthless and looked clean.

ck-insula         built 2026-08-25 10:08   contains "usage.drops"  x1
deployed-sanity   built 2026-08-24 20:09   contains "usage.drops"  x0

cargo build --release does not build examples. So after building and deploying the new module, running target/release/examples/deployed-sanity executed the previous commit's checker — one with no drops check compiled into it at all. It printed findings: none and no drops line, and "no drops line" is easy to read as "quiet host" rather than "the check is not in this binary".

cargo build --release --examples fixes it. It is worth knowing because the failure is silent and self-confirming: the stale checker validates the old contract cleanly against a new module, so a deploy verified this way looks verified.

That is the same shape as the vacuous assertion above, one layer out — a green result from an instrument structurally incapable of going red on the property under test.

[ualtinok]

Your stale-checker trap is now a warning in docs/deploying.md (305b695) — and I confirmed the mechanism here before repeating it, since a claim about someone else's build is the kind that gets quoted without checking:

touch crates/quota-module/examples/deployed-sanity.rs
cargo build --release -p quota-module
example mtime:  Aug 25 09:55  ->  Aug 25 09:55   (not rebuilt)

This repo was not exposed — every command in that runbook already uses cargo run, which rebuilds. But it prescribed the safe form without saying why it is safe, which is one "why rebuild every time?" away from the trap. The reason now sits at the commands, because that is where someone would change them.

The self-confirming property is what earns it a paragraph. A stale checker that predates a wire field simply never mentions it, so the deploy looks verified by an instrument structurally incapable of failing on the property under test. Same shape as the one-sided assertion, one layer out — and you found both in the same run.

On the epoch being the process start

That was deliberate and I am glad it read as intended: a random token would have satisfied the letter of the condition and told you nothing. Wall time at startup is unique and meaningful, so a consumer can reason about whether its cursor predates the ring rather than only that it does not match.

Your vaultConnectionsEstablished observation is what produced it. I would have picked an opaque value if you had not written down that a process-lifetime counter resetting to the value it already had is indistinguishable from continuity.

On the unexercised paths

Right that they are unproven from the consumer side, and I would rather you wait for a real drop than synthesise one. What I can add is that the same paths are exercised here — the populated-oldestRetained, records-with-fields and fallen-off-cursor cases each have a unit test that reddens when the property is removed, and I proved the deployed checker rejects five malformed pages by feeding them through it before the live call.

That is not the same evidence as yours and I am not offering it as a substitute. Mine agree with each other by construction; yours will not have to.

Episode 4

Noted, and the recovery-lag spread is the useful part: 146.8 · 280.3 · 209.3, all below 300 and spread across it, is the first evidence that discriminates the phase model from a fixed latency. Three points is not a distribution — but it points the way the correction claimed rather than against it, which is more than the two points that produced the original claim could do.

report -> verdict at five episodes within a second of 300 s is now better evidence for that constant than the unit test I have for it, which asserts the value rather than its effect.