From 570e01a0ddaae5d342d6e02a7f485b55f37252c1 Mon Sep 17 00:00:00 2001 From: Jeremi Joslin Date: Tue, 13 Oct 2026 02:53:34 +0700 Subject: [PATCH 1/6] docs(platform): tell operators how a stalled stdout audit stream behaves A stdout audit destination whose collector stops reading now refuses audited requests after a fixed 5 second write deadline, reports not ready, and resumes by itself once the collector takes the held entry. The operator pages only described the stopped writer that needs a restart, so an operator reading them would restart a process that recovers on its own, and during a stall a restart can lose the entries queued behind the held one. Describe the stall, its two log messages, and the recovery on the retention overview and on the Base Registry Engine, Evidence, Casework, and Registry Render operate pages, with the troubleshooting rows that point at it. Signed-off-by: Jeremi Joslin --- .../content/docs/operate/breg-retention.mdx | 17 ++++++++++++++++- .../docs/operate/casework-retention.mdx | 12 ++++++++++-- .../src/content/docs/operate/casework.mdx | 14 ++++++++++++-- .../content/docs/operate/evidence-audit.mdx | 19 ++++++++++++++++++- .../content/docs/operate/registry-render.mdx | 2 +- .../retention-and-persistent-state.mdx | 15 +++++++++++++++ 6 files changed, 72 insertions(+), 7 deletions(-) diff --git a/docs/site/src/content/docs/operate/breg-retention.mdx b/docs/site/src/content/docs/operate/breg-retention.mdx index 0b1e46529d..d818e0b9ac 100644 --- a/docs/site/src/content/docs/operate/breg-retention.mdx +++ b/docs/site/src/content/docs/operate/breg-retention.mdx @@ -404,6 +404,21 @@ entry. A runtime read or mutation answers with a `response` entry whose record ` when it is refused, or when it stops with nothing to do, such as a `history rebaseline` with no history to rebaseline, so an `unfinished` outcome does not always mean the outcome was unknown. +A `stdout` destination stalls when its collector stops reading the stream. +Once the writer has held one entry for 5 seconds, the request waiting on that entry is refused as +audit unavailable, later audited requests are refused at once, and readiness fails. +The runtime log carries `audit stream has held one entry past the write deadline`. +When the collector takes the entry, the log carries `audit stream took the entry it was holding` +and the writer accepts entries again with no restart. +The stream still receives the held entry, so it can carry an entry for a request that was refused. + +{/* Evidence: crates/registry-platform-audit/src/writer.rs, STREAM_WRITE_DEADLINE, + AuditUnavailableReason::Stalled, StreamEvent::StallStarted, StreamEvent::StallEnded, + StreamEvent::WrittenAfterRefusal, + a_stalled_stream_refuses_the_append_at_the_deadline_and_later_ones_at_once, and + readiness_follows_a_stall_and_returns_once_the_stream_drains; + crates/registry-breg/src/startup.rs, audit_writer. */} + This pairing is a property of a running process, and a request may be answered more than once, so read every response under a correlation. A process that crashes, is killed, exits, or shuts its runtime down while a write is in flight, or a destination that already stopped accepting entries, @@ -502,7 +517,7 @@ outside the new writer's retention directory. | `request-retention erase` is refused | The request entity's retention mode is `retain`, or the version is pinned. Run the dry run for the exact version first. | | `evidence-retention erase-expired` removes nothing | No retained assertion has passed its 24-hour window yet, or the cutoff precedes every expiry. The count is the whole report. | | A retry answers `410 idempotency.expired` | The key was spent by a successful write whose receipt horizon has passed. The write committed; read the record or its history instead of retrying, and send a fresh key only for a new write. | -| Readiness or an operation reports audit unavailable | Check destination permissions, free space, and the single-writer lock. Repair the destination and restart a stopped writer. For a mutation whose response failed after commit, inspect its receipt or domain history before retrying. | +| Readiness or an operation reports audit unavailable | Check destination permissions, free space, and the single-writer lock. Repair the destination and restart a stopped writer. When the log carries `audit stream has held one entry past the write deadline`, the `stdout` collector stopped reading: restore it, and the writer resumes with no restart. For a mutation whose response failed after commit, inspect its receipt or domain history before retrying. | | An operator command cannot reach PostgreSQL | Check the migration URL secret, the role names, and `SSL_CERT_FILE` for a private authority. | {/* Evidence: crates/registry-breg/src/history_maintenance.rs, lock_registry; diff --git a/docs/site/src/content/docs/operate/casework-retention.mdx b/docs/site/src/content/docs/operate/casework-retention.mdx index 091c20de82..d27707da51 100644 --- a/docs/site/src/content/docs/operate/casework-retention.mdx +++ b/docs/site/src/content/docs/operate/casework-retention.mdx @@ -371,8 +371,16 @@ call is refused while the process keeps running. The runtime log carries `audit with the cause. Treat it as a disk, permission, or lock problem at the audit path rather than a database outage, fix it, and restart the process. -{/* Evidence: crates/registry-platform-audit/src/writer.rs, AuditWriter::append() and - AuditWriter::ready(); +A `stdout` destination whose collector stops reading is a stall, not a failed write. +After the writer has held one entry for 5 seconds, audited calls fail with `service.unavailable` +and `GET /ready` answers `503` until the collector takes the entry; the writer then accepts +entries again with no restart. +The runtime log carries `audit stream has held one entry past the write deadline` when the stall +starts. + +{/* Evidence: crates/registry-platform-audit/src/writer.rs, AuditWriter::append(), + AuditWriter::ready(), STREAM_WRITE_DEADLINE, AuditUnavailableReason::Stalled, + StreamEvent::StallStarted, and readiness_follows_a_stall_and_returns_once_the_stream_drains; crates/registry-casework/src/service.rs, ready(); crates/registry-casework/src/http.rs, ready(). */} diff --git a/docs/site/src/content/docs/operate/casework.mdx b/docs/site/src/content/docs/operate/casework.mdx index 529a1dceab..a072753e60 100644 --- a/docs/site/src/content/docs/operate/casework.mdx +++ b/docs/site/src/content/docs/operate/casework.mdx @@ -859,6 +859,14 @@ the process's life: every later audited call is refused and `/ready` answers `50 process keeps running, so restart it once the cause is fixed. `/ready` also answers `503` when something else replaces or changes the audit file or its lock. +A `stdout` destination whose collector stops reading stalls instead, and a stall ends without a +restart. +Once the writer has held one entry for 5 seconds, the call waiting on that entry fails with +`service.unavailable`, later audited calls are refused at once, and `/ready` answers `503`. +The runtime log carries `audit stream has held one entry past the write deadline`. +When the collector takes the entry, the log carries `audit stream took the entry it was holding` +and the writer accepts entries again. + A maintenance pass runs every two seconds and a source reconciliation pass per bound source at that binding's `reconciliationIntervalMilliseconds`, every 60 seconds unless set. A source outage stays visible @@ -893,7 +901,9 @@ is among the items it reads. {/* Evidence: crates/registry-casework/src/runtime.rs, check_activation(), serve_from_path(), serve_until_worker_stops(), supervise(), operational_log_level(), and open_audit(); crates/registry-casework/src/audit.rs, CASEWORK_AUDIT_SCHEMA and AuditOperation::commit(); - crates/registry-platform-audit/src/writer.rs, AuditWriter::ready(); + crates/registry-platform-audit/src/writer.rs, AuditWriter::ready(), STREAM_WRITE_DEADLINE, + AuditUnavailableReason::Stalled, StreamEvent::StallStarted, StreamEvent::StallEnded, and + readiness_follows_a_stall_and_returns_once_the_stream_drains; crates/registry-casework/src/main.rs, initialize_logging(); crates/registry-casework/src/store.rs, migrate(); crates/registry-casework/src/http.rs, health() and ready(); @@ -1063,7 +1073,7 @@ bound to the actor and selected profile for 15 minutes; an expired preview retur | `the Casework audit destination could not be opened: another process holds the single-writer lock beside the audit file` | Another `casework` process writes this file. Stop it, or give this process its own `audit.path`. | | `the Casework audit destination could not be opened: the audit file could not be opened: ...` | The rest of the message names the rule the audit file or its directory broke and the fix, for example `chown it to the service user and chmod it 0700`, or `archive it and restart with a fresh path` for a file whose last entry is incomplete. Apply it and start again. | | `caseworkctl doctor` stops at `the Casework audit destination is not writable by this user` | Run `doctor` as the runtime user, and fix the audit directory's owner and mode, or the audit file's, as the message names. | -| `GET /ready` returns `503` while `GET /health` returns `200` | The audit writer stopped, PostgreSQL is failing, or a source's reconciliation is failing. Run `caseworkctl doctor`, which names the check, then read the runtime log for `audit file write failed` or the reconciliation entry, and check the audit file, its lock, the free space, and the database. | +| `GET /ready` returns `503` while `GET /health` returns `200` | The audit writer stopped, PostgreSQL is failing, or a source's reconciliation is failing. Run `caseworkctl doctor`, which names the check, then read the runtime log for `audit file write failed` or the reconciliation entry, and check the audit file, its lock, the free space, and the database. When the log carries `audit stream has held one entry past the write deadline`, restore the `stdout` collector; the writer resumes with no restart. | | `doctor` refuses at `doctor:/checks/reconciliation` | The named source's last passes failed. `source-unavailable` means the source did not answer; `source-refused` means it answered with a refusal or a response Casework cannot use; `store` and `configuration` point at the database and the binding. Repair the cause; the next pass that succeeds restores readiness. | | `doctor` refuses at `doctor:/checks/pinnedWork`, or `caseworkctl apply` refuses with `would strand work pinned under an earlier package` | The package removes a queue, profile, review producer, or source that in-flight work still needs, edits a pinned review kind without a new version, or changes what a source read discloses. Keep the earlier package until that work finishes, or set `package.acknowledgeStrandedWork` to the digest the refusal names. | | `BReg source ID runs engine version X and this Casework runs Y`, or `BReg source ID does not report its engine version` | Casework and that BReg source are on different releases, or a proxy removes the `Registry-Engine-Version` header. Upgrade the one that is behind as [the lock-step upgrade](#upgrade-casework-and-breg-in-lock-step) describes, or let the header through; source reads resume on the next matching read. | diff --git a/docs/site/src/content/docs/operate/evidence-audit.mdx b/docs/site/src/content/docs/operate/evidence-audit.mdx index 162a4ef90c..0e46f0e90d 100644 --- a/docs/site/src/content/docs/operate/evidence-audit.mdx +++ b/docs/site/src/content/docs/operate/evidence-audit.mdx @@ -103,9 +103,26 @@ Never rename, edit, or truncate the active file while the service runs: the writ tampering, and readiness and every later audited request fail closed until the process restarts. Copying a sealed file is safe at any time. +A `stdout` destination stalls when its collector stops reading the stream. +Once the writer has held one entry for 5 seconds, the request waiting on that entry is refused, +later audited requests are refused at once, and readiness fails. +The log on standard error carries `audit stream has held one entry past the write deadline`. +When the collector reads again, the log carries `audit stream took the entry it was holding`, and +the writer accepts entries with no restart. +The stream still receives the entry the collector was holding, so it can carry an entry for a +request that was refused. +Restore the collector rather than restarting Evidence Gateway: a process that exits during a stall +can lose the entries queued behind the held one. +A write the stream refuses is a different fault and stops the writer until the process restarts. + {/* Evidence: products/evidence/contracts/runtime.schema.yaml, audit; crates/registry-platform-audit/src/writer.rs, DEFAULT_AUDIT_ROTATE_BYTES, - DEFAULT_AUDIT_RETAIN_DAYS, and AuditDestination. */} + DEFAULT_AUDIT_RETAIN_DAYS, AuditDestination, STREAM_WRITE_DEADLINE, + AuditUnavailableReason::Stalled, StreamEvent::StallStarted, StreamEvent::StallEnded, + StreamEvent::WrittenAfterRefusal, StreamEvent::AbandonedAtDrop, + a_stalled_stream_refuses_the_append_at_the_deadline_and_later_ones_at_once, and + readiness_follows_a_stall_and_returns_once_the_stream_drains; + crates/registry-evidence/src/runtime.rs, ready(). */} ## Ship audit to append-only storage diff --git a/docs/site/src/content/docs/operate/registry-render.mdx b/docs/site/src/content/docs/operate/registry-render.mdx index a30da1236b..5bf2f479a7 100644 --- a/docs/site/src/content/docs/operate/registry-render.mdx +++ b/docs/site/src/content/docs/operate/registry-render.mdx @@ -200,7 +200,7 @@ curl -s http://127.0.0.1:8080/ready `GET /health` answers with the bundle version and package hash and the renderer version, so you can reconcile the running service against what you deployed. `GET /ready` includes the audit writer: -if it has stopped, readiness fails. Both are value-free. +if it has stopped, or its `stdout` destination has stalled, readiness fails. Both are value-free. `registry-render healthcheck --runtime-config ` performs the same probe for a supervisor script. {/* Evidence: crates/registry-render/src/server.rs, health and healthcheck; diff --git a/docs/site/src/content/docs/operate/retention-and-persistent-state.mdx b/docs/site/src/content/docs/operate/retention-and-persistent-state.mdx index 20c8d42f72..500bb379f3 100644 --- a/docs/site/src/content/docs/operate/retention-and-persistent-state.mdx +++ b/docs/site/src/content/docs/operate/retention-and-persistent-state.mdx @@ -60,6 +60,21 @@ Never truncate, rewrite, or reserialize the active audit file as part of cleanup treats any external change to it as a fault and stops, and audited requests fail closed until a restart. Copy sealed files to append-only storage before the audit retention period deletes them. +A `stdout` destination can also stall, and a stall ends without a restart. +When the log collector stops reading the stream and the writer has held one entry for 5 seconds, +the writer refuses the request waiting on that entry, refuses later audited requests at once, and +reports not ready. +When the collector takes the entry, the writer accepts entries again. +The 5 seconds are fixed, with no configuration key. +A write the stream refuses is a different fault: it stops the writer until a restart, as a failed +file write does. + +{/* Evidence: crates/registry-platform-audit/src/writer.rs, STREAM_WRITE_DEADLINE, + AuditUnavailableReason::Stalled, + a_stalled_stream_refuses_the_append_at_the_deadline_and_later_ones_at_once, + readiness_follows_a_stall_and_returns_once_the_stream_drains, and + a_write_error_still_stops_the_stream_after_its_sink_recovers. */} + A crash, a killed process, or a full disk can leave the active file's final line incomplete (any byte other than a newline at end of file). The next start recovers it without operator action: the writer copies the bytes after the last complete line to the owner-only (`0600`) side file From 183f521a19d30770a3a21f4c8d4045c164681877 Mon Sep 17 00:00:00 2001 From: Jeremi Joslin Date: Tue, 13 Oct 2026 16:32:46 +0700 Subject: [PATCH 2/6] docs(platform): describe a stalled audit stream by what BReg and Casework operators can observe The Base Registry Engine and Casework log filters admit only their own targets, so neither runtime log carries the audit writer's stall records. The pages now point operators at readiness and the refused requests. Both runtimes log to the standard output a stdout audit destination writes. The pages say what that means while the collector is not reading: a Base Registry Engine request or probe can go unanswered, because one log record is written before each answer leaves, and a Casework record written meanwhile holds a runtime thread. The Base Registry Engine page also says that a request timeout shorter than the write deadline answers 504 first. Signed-off-by: Jeremi Joslin --- .../content/docs/operate/breg-retention.mdx | 31 ++++++++++++++----- .../docs/operate/casework-retention.mdx | 10 +++--- .../src/content/docs/operate/casework.mdx | 16 ++++++---- 3 files changed, 39 insertions(+), 18 deletions(-) diff --git a/docs/site/src/content/docs/operate/breg-retention.mdx b/docs/site/src/content/docs/operate/breg-retention.mdx index d818e0b9ac..bb416dc1f8 100644 --- a/docs/site/src/content/docs/operate/breg-retention.mdx +++ b/docs/site/src/content/docs/operate/breg-retention.mdx @@ -407,17 +407,34 @@ history to rebaseline, so an `unfinished` outcome does not always mean the outco A `stdout` destination stalls when its collector stops reading the stream. Once the writer has held one entry for 5 seconds, the request waiting on that entry is refused as audit unavailable, later audited requests are refused at once, and readiness fails. -The runtime log carries `audit stream has held one entry past the write deadline`. -When the collector takes the entry, the log carries `audit stream took the entry it was holding` -and the writer accepts entries again with no restart. +A request that reaches `operationalTimeouts.httpRequestMilliseconds` before those 5 seconds pass +answers `504` with `request.timeout` instead. +When the collector takes the entry, the writer accepts entries again with no restart. The stream still receives the held entry, so it can carry an entry for a request that was refused. +The runtime log does not name a stall, and it shares standard output with a `stdout` audit +destination. +At the default `BREG_LOG` level of `info`, the runtime writes one log record for each request and +each probe before the answer leaves, and while the collector is not reading, that record waits +with no deadline. +A refused request or a probe, `/health` included, can therefore get no answer at all until the +collector reads again. +The refusal holds either way: protected I/O and disclosure stay blocked as for any failed entry. +Set a timeout on every probe and treat a probe that gets no answer as failed. +The `file` destination accepts entries whatever the collector does. +The runtime log waits on the collector under either destination, so run the runtime under a +collector that does not block standard output when it stops reading. + {/* Evidence: crates/registry-platform-audit/src/writer.rs, STREAM_WRITE_DEADLINE, - AuditUnavailableReason::Stalled, StreamEvent::StallStarted, StreamEvent::StallEnded, - StreamEvent::WrittenAfterRefusal, + AuditUnavailableReason::Stalled, StreamEvent::WrittenAfterRefusal, a_stalled_stream_refuses_the_append_at_the_deadline_and_later_ones_at_once, and readiness_follows_a_stall_and_returns_once_the_stream_drains; - crates/registry-breg/src/startup.rs, audit_writer. */} + crates/registry-breg/src/startup.rs, audit_writer, request_timeout(), timeout_problem(), and + operational_log_level(); + crates/registry-breg/src/runtime_config.rs, RawOperationalTimeouts; + crates/registry-breg/src/correlation.rs, finish_response(); + crates/registry-breg/src/api/mod.rs, authenticated_router() and probe_routes(); + crates/registry-breg/src/main.rs, initialize_logging_filter(). */} This pairing is a property of a running process, and a request may be answered more than once, so read every response under a correlation. A process that crashes, is killed, exits, or shuts its @@ -517,7 +534,7 @@ outside the new writer's retention directory. | `request-retention erase` is refused | The request entity's retention mode is `retain`, or the version is pinned. Run the dry run for the exact version first. | | `evidence-retention erase-expired` removes nothing | No retained assertion has passed its 24-hour window yet, or the cutoff precedes every expiry. The count is the whole report. | | A retry answers `410 idempotency.expired` | The key was spent by a successful write whose receipt horizon has passed. The write committed; read the record or its history instead of retrying, and send a fresh key only for a new write. | -| Readiness or an operation reports audit unavailable | Check destination permissions, free space, and the single-writer lock. Repair the destination and restart a stopped writer. When the log carries `audit stream has held one entry past the write deadline`, the `stdout` collector stopped reading: restore it, and the writer resumes with no restart. For a mutation whose response failed after commit, inspect its receipt or domain history before retrying. | +| Readiness or an operation reports audit unavailable | Check destination permissions, free space, and the single-writer lock. Repair the destination and restart a stopped writer. With a `stdout` destination, check that its collector is reading the stream: a stalled writer resumes with no restart once it is, and until then a request or probe can go unanswered instead of refused. For a mutation whose response failed after commit, inspect its receipt or domain history before retrying. | | An operator command cannot reach PostgreSQL | Check the migration URL secret, the role names, and `SSL_CERT_FILE` for a private authority. | {/* Evidence: crates/registry-breg/src/history_maintenance.rs, lock_registry; diff --git a/docs/site/src/content/docs/operate/casework-retention.mdx b/docs/site/src/content/docs/operate/casework-retention.mdx index d27707da51..0b55cb0dda 100644 --- a/docs/site/src/content/docs/operate/casework-retention.mdx +++ b/docs/site/src/content/docs/operate/casework-retention.mdx @@ -375,14 +375,14 @@ A `stdout` destination whose collector stops reading is a stall, not a failed wr After the writer has held one entry for 5 seconds, audited calls fail with `service.unavailable` and `GET /ready` answers `503` until the collector takes the entry; the writer then accepts entries again with no restart. -The runtime log carries `audit stream has held one entry past the write deadline` when the stall -starts. +The runtime log does not name a stall, so `GET /ready` and the refused calls are the signal. {/* Evidence: crates/registry-platform-audit/src/writer.rs, AuditWriter::append(), - AuditWriter::ready(), STREAM_WRITE_DEADLINE, AuditUnavailableReason::Stalled, - StreamEvent::StallStarted, and readiness_follows_a_stall_and_returns_once_the_stream_drains; + AuditWriter::ready(), STREAM_WRITE_DEADLINE, AuditUnavailableReason::Stalled, and + readiness_follows_a_stall_and_returns_once_the_stream_drains; crates/registry-casework/src/service.rs, ready(); - crates/registry-casework/src/http.rs, ready(). */} + crates/registry-casework/src/http.rs, ready(); + crates/registry-casework/src/main.rs, initialize_logging(). */} ## Troubleshooting diff --git a/docs/site/src/content/docs/operate/casework.mdx b/docs/site/src/content/docs/operate/casework.mdx index a072753e60..ba2b778d0b 100644 --- a/docs/site/src/content/docs/operate/casework.mdx +++ b/docs/site/src/content/docs/operate/casework.mdx @@ -863,9 +863,13 @@ A `stdout` destination whose collector stops reading stalls instead, and a stall restart. Once the writer has held one entry for 5 seconds, the call waiting on that entry fails with `service.unavailable`, later audited calls are refused at once, and `/ready` answers `503`. -The runtime log carries `audit stream has held one entry past the write deadline`. -When the collector takes the entry, the log carries `audit stream took the entry it was holding` -and the writer accepts entries again. +When the collector takes the entry, the writer accepts entries again. +The runtime log does not name a stall, so `/ready` and the refused calls are the signal. +That log shares standard output with a `stdout` audit destination. +A refused call and `/ready` write no log record, so their answers do not wait on the stream. +A record the runtime does write while the collector is not reading, such as a background pass that +did not complete, waits with no deadline and holds a runtime thread until the collector reads +again, so set a timeout on the `/ready` probe and treat a probe that gets no answer as failed. A maintenance pass runs every two seconds and a source reconciliation pass per bound source at that binding's @@ -902,11 +906,11 @@ is among the items it reads. serve_until_worker_stops(), supervise(), operational_log_level(), and open_audit(); crates/registry-casework/src/audit.rs, CASEWORK_AUDIT_SCHEMA and AuditOperation::commit(); crates/registry-platform-audit/src/writer.rs, AuditWriter::ready(), STREAM_WRITE_DEADLINE, - AuditUnavailableReason::Stalled, StreamEvent::StallStarted, StreamEvent::StallEnded, and + AuditUnavailableReason::Stalled, and readiness_follows_a_stall_and_returns_once_the_stream_drains; crates/registry-casework/src/main.rs, initialize_logging(); crates/registry-casework/src/store.rs, migrate(); - crates/registry-casework/src/http.rs, health() and ready(); + crates/registry-casework/src/http.rs, health(), ready(), and StoreError::AuditUnavailable; crates/registry-casework/src/service.rs, ready() and assemble_caller_visible_item(); crates/registry-casework/src/assignment.rs, assignment_visible_item_for_origin(); crates/registry-casework/src/task_grants.rs, preview_tasks(), approve_task(), and list_tasks(); @@ -1073,7 +1077,7 @@ bound to the actor and selected profile for 15 minutes; an expired preview retur | `the Casework audit destination could not be opened: another process holds the single-writer lock beside the audit file` | Another `casework` process writes this file. Stop it, or give this process its own `audit.path`. | | `the Casework audit destination could not be opened: the audit file could not be opened: ...` | The rest of the message names the rule the audit file or its directory broke and the fix, for example `chown it to the service user and chmod it 0700`, or `archive it and restart with a fresh path` for a file whose last entry is incomplete. Apply it and start again. | | `caseworkctl doctor` stops at `the Casework audit destination is not writable by this user` | Run `doctor` as the runtime user, and fix the audit directory's owner and mode, or the audit file's, as the message names. | -| `GET /ready` returns `503` while `GET /health` returns `200` | The audit writer stopped, PostgreSQL is failing, or a source's reconciliation is failing. Run `caseworkctl doctor`, which names the check, then read the runtime log for `audit file write failed` or the reconciliation entry, and check the audit file, its lock, the free space, and the database. When the log carries `audit stream has held one entry past the write deadline`, restore the `stdout` collector; the writer resumes with no restart. | +| `GET /ready` returns `503` while `GET /health` returns `200` | The audit writer stopped, PostgreSQL is failing, or a source's reconciliation is failing. Run `caseworkctl doctor`, which names the check, then read the runtime log for `audit file write failed` or the reconciliation entry, and check the audit file, its lock, the free space, and the database. With a `stdout` audit destination, also check that its collector is reading the stream: a stalled writer resumes with no restart once it is. | | `doctor` refuses at `doctor:/checks/reconciliation` | The named source's last passes failed. `source-unavailable` means the source did not answer; `source-refused` means it answered with a refusal or a response Casework cannot use; `store` and `configuration` point at the database and the binding. Repair the cause; the next pass that succeeds restores readiness. | | `doctor` refuses at `doctor:/checks/pinnedWork`, or `caseworkctl apply` refuses with `would strand work pinned under an earlier package` | The package removes a queue, profile, review producer, or source that in-flight work still needs, edits a pinned review kind without a new version, or changes what a source read discloses. Keep the earlier package until that work finishes, or set `package.acknowledgeStrandedWork` to the digest the refusal names. | | `BReg source ID runs engine version X and this Casework runs Y`, or `BReg source ID does not report its engine version` | Casework and that BReg source are on different releases, or a proxy removes the `Registry-Engine-Version` header. Upgrade the one that is behind as [the lock-step upgrade](#upgrade-casework-and-breg-in-lock-step) describes, or let the header through; source reads resume on the next matching read. | From babd7c640c126736514b13f8b9d24b8dd405b87e Mon Sep 17 00:00:00 2001 From: Jeremi Joslin Date: Tue, 13 Oct 2026 16:32:46 +0700 Subject: [PATCH 3/6] docs(platform): cover a full audit stream queue and link stall recovery from Render A request that finds the stream queue full is refused while the writer can still report ready, so the shared page says readiness alone does not show it. The Render page links the recovery guidance for a stalled destination and a stopped writer, and its evidence names the readiness probe it describes. Signed-off-by: Jeremi Joslin --- docs/site/src/content/docs/operate/registry-render.mdx | 5 ++++- .../docs/operate/retention-and-persistent-state.mdx | 8 ++++++-- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/site/src/content/docs/operate/registry-render.mdx b/docs/site/src/content/docs/operate/registry-render.mdx index 5bf2f479a7..72462efb48 100644 --- a/docs/site/src/content/docs/operate/registry-render.mdx +++ b/docs/site/src/content/docs/operate/registry-render.mdx @@ -202,8 +202,11 @@ curl -s http://127.0.0.1:8080/ready reconcile the running service against what you deployed. `GET /ready` includes the audit writer: if it has stopped, or its `stdout` destination has stalled, readiness fails. Both are value-free. `registry-render healthcheck --runtime-config ` performs the same probe for a supervisor script. +A stalled destination recovers when its collector reads the stream again, and a stopped writer needs +a restart; [audit retention](../retention-and-persistent-state/#audit-retention) describes both. -{/* Evidence: crates/registry-render/src/server.rs, health and healthcheck; +{/* Evidence: crates/registry-render/src/server.rs, health, ready(), and healthcheck; + crates/registry-platform-audit/src/writer.rs, AuditWriter::ready(); crates/registry-render/src/openapi.rs, OPENAPI_JSON. */} ## The rendering call diff --git a/docs/site/src/content/docs/operate/retention-and-persistent-state.mdx b/docs/site/src/content/docs/operate/retention-and-persistent-state.mdx index 500bb379f3..d03102f958 100644 --- a/docs/site/src/content/docs/operate/retention-and-persistent-state.mdx +++ b/docs/site/src/content/docs/operate/retention-and-persistent-state.mdx @@ -66,13 +66,17 @@ the writer refuses the request waiting on that entry, refuses later audited requ reports not ready. When the collector takes the entry, the writer accepts entries again. The 5 seconds are fixed, with no configuration key. +The writer also bounds the entries waiting behind the one it is writing: a request that finds that +queue full is refused at once, with no stall logged and with the writer still able to report +ready, so readiness alone does not show it. A write the stream refuses is a different fault: it stops the writer until a restart, as a failed file write does. {/* Evidence: crates/registry-platform-audit/src/writer.rs, STREAM_WRITE_DEADLINE, - AuditUnavailableReason::Stalled, + STREAM_QUEUE_LINES, AuditUnavailableReason::Stalled, a_stalled_stream_refuses_the_append_at_the_deadline_and_later_ones_at_once, - readiness_follows_a_stall_and_returns_once_the_stream_drains, and + readiness_follows_a_stall_and_returns_once_the_stream_drains, + a_full_stream_queue_refuses_a_line_instead_of_growing, and a_write_error_still_stops_the_stream_after_its_sink_recovers. */} A crash, a killed process, or a full disk can leave the active file's final line incomplete (any From c47bf24466714949927872e9edee3b69740179af Mon Sep 17 00:00:00 2001 From: Jeremi Joslin Date: Tue, 13 Oct 2026 16:38:54 +0700 Subject: [PATCH 4/6] docs(casework): limit the no-log-record claim to a call refused before its work starts Signed-off-by: Jeremi Joslin --- docs/site/src/content/docs/operate/casework.mdx | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/site/src/content/docs/operate/casework.mdx b/docs/site/src/content/docs/operate/casework.mdx index ba2b778d0b..1fee6846e3 100644 --- a/docs/site/src/content/docs/operate/casework.mdx +++ b/docs/site/src/content/docs/operate/casework.mdx @@ -866,7 +866,8 @@ Once the writer has held one entry for 5 seconds, the call waiting on that entry When the collector takes the entry, the writer accepts entries again. The runtime log does not name a stall, so `/ready` and the refused calls are the signal. That log shares standard output with a `stdout` audit destination. -A refused call and `/ready` write no log record, so their answers do not wait on the stream. +A call refused before its work starts and `/ready` write no log record, so their answers do not +wait on the stream. A record the runtime does write while the collector is not reading, such as a background pass that did not complete, waits with no deadline and holds a runtime thread until the collector reads again, so set a timeout on the `/ready` probe and treat a probe that gets no answer as failed. From 297390e1a855270a88d0c02a2123cb86d5b725e2 Mon Sep 17 00:00:00 2001 From: Jeremi Joslin Date: Tue, 13 Oct 2026 16:50:39 +0700 Subject: [PATCH 5/6] docs(platform): qualify the Evidence stall records and cover a slow audit stream reader The two stall records reach an Evidence operator only where the EVIDENCE_LOG filter admits their target and levels, and a collector that reads slowly without stopping refuses requests with no stall and with readiness still true. Signed-off-by: Jeremi Joslin --- docs/site/src/content/docs/operate/evidence-audit.mdx | 9 +++++++-- .../docs/operate/retention-and-persistent-state.mdx | 5 ++++- 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/docs/site/src/content/docs/operate/evidence-audit.mdx b/docs/site/src/content/docs/operate/evidence-audit.mdx index 0e46f0e90d..b4706e1014 100644 --- a/docs/site/src/content/docs/operate/evidence-audit.mdx +++ b/docs/site/src/content/docs/operate/evidence-audit.mdx @@ -106,9 +106,13 @@ Copying a sealed file is safe at any time. A `stdout` destination stalls when its collector stops reading the stream. Once the writer has held one entry for 5 seconds, the request waiting on that entry is refused, later audited requests are refused at once, and readiness fails. -The log on standard error carries `audit stream has held one entry past the write deadline`. +With `EVIDENCE_LOG` at its default of `info`, the log on standard error carries +`audit stream has held one entry past the write deadline`. When the collector reads again, the log carries `audit stream took the entry it was holding`, and the writer accepts entries with no restart. +The first record is logged at `error` and the second at `warn`, both under the +`registry_platform_audit` target, so an `EVIDENCE_LOG` filter that leaves out that target or those +levels hides them. The stream still receives the entry the collector was holding, so it can carry an entry for a request that was refused. Restore the collector rather than restarting Evidence Gateway: a process that exits during a stall @@ -122,7 +126,8 @@ A write the stream refuses is a different fault and stops the writer until the p StreamEvent::WrittenAfterRefusal, StreamEvent::AbandonedAtDrop, a_stalled_stream_refuses_the_append_at_the_deadline_and_later_ones_at_once, and readiness_follows_a_stall_and_returns_once_the_stream_drains; - crates/registry-evidence/src/runtime.rs, ready(). */} + crates/registry-evidence/src/runtime.rs, ready(); + crates/registry-evidence/src/main.rs, install_operational_logging. */} ## Ship audit to append-only storage diff --git a/docs/site/src/content/docs/operate/retention-and-persistent-state.mdx b/docs/site/src/content/docs/operate/retention-and-persistent-state.mdx index d03102f958..b0a9999761 100644 --- a/docs/site/src/content/docs/operate/retention-and-persistent-state.mdx +++ b/docs/site/src/content/docs/operate/retention-and-persistent-state.mdx @@ -69,11 +69,14 @@ The 5 seconds are fixed, with no configuration key. The writer also bounds the entries waiting behind the one it is writing: a request that finds that queue full is refused at once, with no stall logged and with the writer still able to report ready, so readiness alone does not show it. +A collector that reads slowly without stopping has the same effect: a request whose entry waits 5 +seconds behind earlier entries is refused and its entry is not written, although the writer held +no single entry for 5 seconds, logs no stall, and still reports ready. A write the stream refuses is a different fault: it stops the writer until a restart, as a failed file write does. {/* Evidence: crates/registry-platform-audit/src/writer.rs, STREAM_WRITE_DEADLINE, - STREAM_QUEUE_LINES, AuditUnavailableReason::Stalled, + STREAM_QUEUE_LINES, AuditUnavailableReason::Stalled, LineStream::append, StreamState::run, a_stalled_stream_refuses_the_append_at_the_deadline_and_later_ones_at_once, readiness_follows_a_stall_and_returns_once_the_stream_drains, a_full_stream_queue_refuses_a_line_instead_of_growing, and From 01d8849be20834893bf4285017d064d7ed059916 Mon Sep 17 00:00:00 2001 From: Jeremi Joslin Date: Tue, 13 Oct 2026 19:07:40 +0700 Subject: [PATCH 6/6] docs(evidence): say that a stalled audit stream also needs standard error read The two stall records go to standard error from the request, readiness check, or writer that noticed the change, and that write blocks when standard error is full. A collector that stops reading both streams can leave a request waiting rather than refused. Signed-off-by: Jeremi Joslin --- docs/site/src/content/docs/operate/evidence-audit.mdx | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/site/src/content/docs/operate/evidence-audit.mdx b/docs/site/src/content/docs/operate/evidence-audit.mdx index b4706e1014..aba50d61bc 100644 --- a/docs/site/src/content/docs/operate/evidence-audit.mdx +++ b/docs/site/src/content/docs/operate/evidence-audit.mdx @@ -113,6 +113,12 @@ the writer accepts entries with no restart. The first record is logged at `error` and the second at `warn`, both under the `registry_platform_audit` target, so an `EVIDENCE_LOG` filter that leaves out that target or those levels hides them. +Keep standard error read as well. +Evidence Gateway writes both records to standard error from the request, readiness check, or +writer that noticed the change, and that write waits when standard error is full. +A collector that stops reading both streams can therefore leave a request or a readiness check +waiting rather than refused, until standard error is read again, so give callers and probes their +own deadlines. The stream still receives the entry the collector was holding, so it can carry an entry for a request that was refused. Restore the collector rather than restarting Evidence Gateway: a process that exits during a stall