docs(platform): tell operators how a stalled stdout audit stream behaves

docs/site/src/content/docs/operate/breg-retention.mdx

@@ -404,6 +404,38 @@ entry. A runtime read or mutation answers with a `response` entry whose record `
 when it is refused, or when it stops with nothing to do, such as a `history rebaseline` with no
 history to rebaseline, so an `unfinished` outcome does not always mean the outcome was unknown.
 
+A `stdout` destination stalls when its collector stops reading the stream.
+Once the writer has held one entry for 5 seconds, the request waiting on that entry is refused as
+audit unavailable, later audited requests are refused at once, and readiness fails.
+A request that reaches `operationalTimeouts.httpRequestMilliseconds` before those 5 seconds pass
+answers `504` with `request.timeout` instead.
+When the collector takes the entry, the writer accepts entries again with no restart.
+The stream still receives the held entry, so it can carry an entry for a request that was refused.
+
+The runtime log does not name a stall, and it shares standard output with a `stdout` audit
+destination.
+At the default `BREG_LOG` level of `info`, the runtime writes one log record for each request and
+each probe before the answer leaves, and while the collector is not reading, that record waits
+with no deadline.
+A refused request or a probe, `/health` included, can therefore get no answer at all until the
+collector reads again.
+The refusal holds either way: protected I/O and disclosure stay blocked as for any failed entry.
+Set a timeout on every probe and treat a probe that gets no answer as failed.
+The `file` destination accepts entries whatever the collector does.
+The runtime log waits on the collector under either destination, so run the runtime under a
+collector that does not block standard output when it stops reading.
+
+{/* Evidence: crates/registry-platform-audit/src/writer.rs, STREAM_WRITE_DEADLINE,
+    AuditUnavailableReason::Stalled, StreamEvent::WrittenAfterRefusal,
+    a_stalled_stream_refuses_the_append_at_the_deadline_and_later_ones_at_once, and
+    readiness_follows_a_stall_and_returns_once_the_stream_drains;
+    crates/registry-breg/src/startup.rs, audit_writer, request_timeout(), timeout_problem(), and
+    operational_log_level();
+    crates/registry-breg/src/runtime_config.rs, RawOperationalTimeouts;
+    crates/registry-breg/src/correlation.rs, finish_response();
+    crates/registry-breg/src/api/mod.rs, authenticated_router() and probe_routes();
+    crates/registry-breg/src/main.rs, initialize_logging_filter(). */}
+
 This pairing is a property of a running process, and a request may be answered more than once, so
 read every response under a correlation. A process that crashes, is killed, exits, or shuts its
 runtime down while a write is in flight, or a destination that already stopped accepting entries,
@@ -502,7 +534,7 @@ outside the new writer's retention directory.
 | `request-retention erase` is refused | The request entity's retention mode is `retain`, or the version is pinned. Run the dry run for the exact version first. |
 | `evidence-retention erase-expired` removes nothing | No retained assertion has passed its 24-hour window yet, or the cutoff precedes every expiry. The count is the whole report. |
 | A retry answers `410 idempotency.expired` | The key was spent by a successful write whose receipt horizon has passed. The write committed; read the record or its history instead of retrying, and send a fresh key only for a new write. |
-| Readiness or an operation reports audit unavailable | Check destination permissions, free space, and the single-writer lock. Repair the destination and restart a stopped writer. For a mutation whose response failed after commit, inspect its receipt or domain history before retrying. |
+| Readiness or an operation reports audit unavailable | Check destination permissions, free space, and the single-writer lock. Repair the destination and restart a stopped writer. With a `stdout` destination, check that its collector is reading the stream: a stalled writer resumes with no restart once it is, and until then a request or probe can go unanswered instead of refused. For a mutation whose response failed after commit, inspect its receipt or domain history before retrying. |
 | An operator command cannot reach PostgreSQL | Check the migration URL secret, the role names, and `SSL_CERT_FILE` for a private authority. |
 
 {/* Evidence: crates/registry-breg/src/history_maintenance.rs, lock_registry;

docs/site/src/content/docs/operate/casework-retention.mdx

@@ -371,10 +371,18 @@ call is refused while the process keeps running. The runtime log carries `audit
 with the cause. Treat it as a disk, permission, or lock problem at the audit path rather than a
 database outage, fix it, and restart the process.
 
-{/* Evidence: crates/registry-platform-audit/src/writer.rs, AuditWriter::append() and
-    AuditWriter::ready();
+A `stdout` destination whose collector stops reading is a stall, not a failed write.
+After the writer has held one entry for 5 seconds, audited calls fail with `service.unavailable`
+and `GET /ready` answers `503` until the collector takes the entry; the writer then accepts
+entries again with no restart.
+The runtime log does not name a stall, so `GET /ready` and the refused calls are the signal.
+
+{/* Evidence: crates/registry-platform-audit/src/writer.rs, AuditWriter::append(),
+    AuditWriter::ready(), STREAM_WRITE_DEADLINE, AuditUnavailableReason::Stalled, and
+    readiness_follows_a_stall_and_returns_once_the_stream_drains;
     crates/registry-casework/src/service.rs, ready();
-    crates/registry-casework/src/http.rs, ready(). */}
+    crates/registry-casework/src/http.rs, ready();
+    crates/registry-casework/src/main.rs, initialize_logging(). */}
 
 ## Troubleshooting
 

docs/site/src/content/docs/operate/casework.mdx

@@ -859,6 +859,19 @@ the process's life: every later audited call is refused and `/ready` answers `50
 process keeps running, so restart it once the cause is fixed. `/ready` also answers `503` when
 something else replaces or changes the audit file or its lock.
 
+A `stdout` destination whose collector stops reading stalls instead, and a stall ends without a
+restart.
+Once the writer has held one entry for 5 seconds, the call waiting on that entry fails with
+`service.unavailable`, later audited calls are refused at once, and `/ready` answers `503`.
+When the collector takes the entry, the writer accepts entries again.
+The runtime log does not name a stall, so `/ready` and the refused calls are the signal.
+That log shares standard output with a `stdout` audit destination.
+A call refused before its work starts and `/ready` write no log record, so their answers do not
+wait on the stream.
+A record the runtime does write while the collector is not reading, such as a background pass that
+did not complete, waits with no deadline and holds a runtime thread until the collector reads
+again, so set a timeout on the `/ready` probe and treat a probe that gets no answer as failed.
+
 A maintenance pass runs every two seconds and a source reconciliation pass per bound source at that
 binding's
 `reconciliationIntervalMilliseconds`, every 60 seconds unless set. A source outage stays visible
@@ -893,10 +906,12 @@ is among the items it reads.
 {/* Evidence: crates/registry-casework/src/runtime.rs, check_activation(), serve_from_path(),
     serve_until_worker_stops(), supervise(), operational_log_level(), and open_audit();
     crates/registry-casework/src/audit.rs, CASEWORK_AUDIT_SCHEMA and AuditOperation::commit();
-    crates/registry-platform-audit/src/writer.rs, AuditWriter::ready();
+    crates/registry-platform-audit/src/writer.rs, AuditWriter::ready(), STREAM_WRITE_DEADLINE,
+    AuditUnavailableReason::Stalled, and
+    readiness_follows_a_stall_and_returns_once_the_stream_drains;
     crates/registry-casework/src/main.rs, initialize_logging();
     crates/registry-casework/src/store.rs, migrate();
-    crates/registry-casework/src/http.rs, health() and ready();
+    crates/registry-casework/src/http.rs, health(), ready(), and StoreError::AuditUnavailable;
     crates/registry-casework/src/service.rs, ready() and assemble_caller_visible_item();
     crates/registry-casework/src/assignment.rs, assignment_visible_item_for_origin();
     crates/registry-casework/src/task_grants.rs, preview_tasks(), approve_task(), and list_tasks();
@@ -1063,7 +1078,7 @@ bound to the actor and selected profile for 15 minutes; an expired preview retur
 | `the Casework audit destination could not be opened: another process holds the single-writer lock beside the audit file` | Another `casework` process writes this file. Stop it, or give this process its own `audit.path`. |
 | `the Casework audit destination could not be opened: the audit file could not be opened: ...` | The rest of the message names the rule the audit file or its directory broke and the fix, for example `chown it to the service user and chmod it 0700`, or `archive it and restart with a fresh path` for a file whose last entry is incomplete. Apply it and start again. |
 | `caseworkctl doctor` stops at `the Casework audit destination is not writable by this user` | Run `doctor` as the runtime user, and fix the audit directory's owner and mode, or the audit file's, as the message names. |
-| `GET /ready` returns `503` while `GET /health` returns `200` | The audit writer stopped, PostgreSQL is failing, or a source's reconciliation is failing. Run `caseworkctl doctor`, which names the check, then read the runtime log for `audit file write failed` or the reconciliation entry, and check the audit file, its lock, the free space, and the database. |
+| `GET /ready` returns `503` while `GET /health` returns `200` | The audit writer stopped, PostgreSQL is failing, or a source's reconciliation is failing. Run `caseworkctl doctor`, which names the check, then read the runtime log for `audit file write failed` or the reconciliation entry, and check the audit file, its lock, the free space, and the database. With a `stdout` audit destination, also check that its collector is reading the stream: a stalled writer resumes with no restart once it is. |
 | `doctor` refuses at `doctor:/checks/reconciliation` | The named source's last passes failed. `source-unavailable` means the source did not answer; `source-refused` means it answered with a refusal or a response Casework cannot use; `store` and `configuration` point at the database and the binding. Repair the cause; the next pass that succeeds restores readiness. |
 | `doctor` refuses at `doctor:/checks/pinnedWork`, or `caseworkctl apply` refuses with `would strand work pinned under an earlier package` | The package removes a queue, profile, review producer, or source that in-flight work still needs, edits a pinned review kind without a new version, or changes what a source read discloses. Keep the earlier package until that work finishes, or set `package.acknowledgeStrandedWork` to the digest the refusal names. |
 | `BReg source ID runs engine version X and this Casework runs Y`, or `BReg source ID does not report its engine version` | Casework and that BReg source are on different releases, or a proxy removes the `Registry-Engine-Version` header. Upgrade the one that is behind as [the lock-step upgrade](#upgrade-casework-and-breg-in-lock-step) describes, or let the header through; source reads resume on the next matching read. |

docs/site/src/content/docs/operate/evidence-audit.mdx

@@ -103,9 +103,37 @@ Never rename, edit, or truncate the active file while the service runs: the writ
 tampering, and readiness and every later audited request fail closed until the process restarts.
 Copying a sealed file is safe at any time.
 
+A `stdout` destination stalls when its collector stops reading the stream.
+Once the writer has held one entry for 5 seconds, the request waiting on that entry is refused,
+later audited requests are refused at once, and readiness fails.
+With `EVIDENCE_LOG` at its default of `info`, the log on standard error carries
+`audit stream has held one entry past the write deadline`.
+When the collector reads again, the log carries `audit stream took the entry it was holding`, and
+the writer accepts entries with no restart.
+The first record is logged at `error` and the second at `warn`, both under the
+`registry_platform_audit` target, so an `EVIDENCE_LOG` filter that leaves out that target or those
+levels hides them.
+Keep standard error read as well.
+Evidence Gateway writes both records to standard error from the request, readiness check, or
+writer that noticed the change, and that write waits when standard error is full.
+A collector that stops reading both streams can therefore leave a request or a readiness check
+waiting rather than refused, until standard error is read again, so give callers and probes their
+own deadlines.
+The stream still receives the entry the collector was holding, so it can carry an entry for a
+request that was refused.
+Restore the collector rather than restarting Evidence Gateway: a process that exits during a stall
+can lose the entries queued behind the held one.
+A write the stream refuses is a different fault and stops the writer until the process restarts.
+
 {/* Evidence: products/evidence/contracts/runtime.schema.yaml, audit;
     crates/registry-platform-audit/src/writer.rs, DEFAULT_AUDIT_ROTATE_BYTES,
-    DEFAULT_AUDIT_RETAIN_DAYS, and AuditDestination. */}
+    DEFAULT_AUDIT_RETAIN_DAYS, AuditDestination, STREAM_WRITE_DEADLINE,
+    AuditUnavailableReason::Stalled, StreamEvent::StallStarted, StreamEvent::StallEnded,
+    StreamEvent::WrittenAfterRefusal, StreamEvent::AbandonedAtDrop,
+    a_stalled_stream_refuses_the_append_at_the_deadline_and_later_ones_at_once, and
+    readiness_follows_a_stall_and_returns_once_the_stream_drains;
+    crates/registry-evidence/src/runtime.rs, ready();
+    crates/registry-evidence/src/main.rs, install_operational_logging. */}
 
 ## Ship audit to append-only storage
 

docs/site/src/content/docs/operate/registry-render.mdx

@@ -200,10 +200,13 @@ curl -s http://127.0.0.1:8080/ready
 
 `GET /health` answers with the bundle version and package hash and the renderer version, so you can
 reconcile the running service against what you deployed. `GET /ready` includes the audit writer:
-if it has stopped, readiness fails. Both are value-free.
+if it has stopped, or its `stdout` destination has stalled, readiness fails. Both are value-free.
 `registry-render healthcheck --runtime-config <file>` performs the same probe for a supervisor script.
+A stalled destination recovers when its collector reads the stream again, and a stopped writer needs
+a restart; [audit retention](../retention-and-persistent-state/#audit-retention) describes both.
 
-{/* Evidence: crates/registry-render/src/server.rs, health and healthcheck;
+{/* Evidence: crates/registry-render/src/server.rs, health, ready(), and healthcheck;
+    crates/registry-platform-audit/src/writer.rs, AuditWriter::ready();
     crates/registry-render/src/openapi.rs, OPENAPI_JSON. */}
 
 ## The rendering call

docs/site/src/content/docs/operate/retention-and-persistent-state.mdx

@@ -60,6 +60,28 @@ Never truncate, rewrite, or reserialize the active audit file as part of cleanup
 treats any external change to it as a fault and stops, and audited requests fail closed until a
 restart. Copy sealed files to append-only storage before the audit retention period deletes them.
 
+A `stdout` destination can also stall, and a stall ends without a restart.
+When the log collector stops reading the stream and the writer has held one entry for 5 seconds,
+the writer refuses the request waiting on that entry, refuses later audited requests at once, and
+reports not ready.
+When the collector takes the entry, the writer accepts entries again.
+The 5 seconds are fixed, with no configuration key.
+The writer also bounds the entries waiting behind the one it is writing: a request that finds that
+queue full is refused at once, with no stall logged and with the writer still able to report
+ready, so readiness alone does not show it.
+A collector that reads slowly without stopping has the same effect: a request whose entry waits 5
+seconds behind earlier entries is refused and its entry is not written, although the writer held
+no single entry for 5 seconds, logs no stall, and still reports ready.
+A write the stream refuses is a different fault: it stops the writer until a restart, as a failed
+file write does.
+
+{/* Evidence: crates/registry-platform-audit/src/writer.rs, STREAM_WRITE_DEADLINE,
+    STREAM_QUEUE_LINES, AuditUnavailableReason::Stalled, LineStream::append, StreamState::run,
+    a_stalled_stream_refuses_the_append_at_the_deadline_and_later_ones_at_once,
+    readiness_follows_a_stall_and_returns_once_the_stream_drains,
+    a_full_stream_queue_refuses_a_line_instead_of_growing, and
+    a_write_error_still_stops_the_stream_after_its_sink_recovers. */}
+
 A crash, a killed process, or a full disk can leave the active file's final line incomplete (any
 byte other than a newline at end of file). The next start recovers it without operator action: the
 writer copies the bytes after the last complete line to the owner-only (`0600`) side file