diff --git a/docs/site/src/content/docs/operate/breg-retention.mdx b/docs/site/src/content/docs/operate/breg-retention.mdx index 0b1e46529..bb416dc1f 100644 --- a/docs/site/src/content/docs/operate/breg-retention.mdx +++ b/docs/site/src/content/docs/operate/breg-retention.mdx @@ -404,6 +404,38 @@ entry. A runtime read or mutation answers with a `response` entry whose record ` when it is refused, or when it stops with nothing to do, such as a `history rebaseline` with no history to rebaseline, so an `unfinished` outcome does not always mean the outcome was unknown. +A `stdout` destination stalls when its collector stops reading the stream. +Once the writer has held one entry for 5 seconds, the request waiting on that entry is refused as +audit unavailable, later audited requests are refused at once, and readiness fails. +A request that reaches `operationalTimeouts.httpRequestMilliseconds` before those 5 seconds pass +answers `504` with `request.timeout` instead. +When the collector takes the entry, the writer accepts entries again with no restart. +The stream still receives the held entry, so it can carry an entry for a request that was refused. + +The runtime log does not name a stall, and it shares standard output with a `stdout` audit +destination. +At the default `BREG_LOG` level of `info`, the runtime writes one log record for each request and +each probe before the answer leaves, and while the collector is not reading, that record waits +with no deadline. +A refused request or a probe, `/health` included, can therefore get no answer at all until the +collector reads again. +The refusal holds either way: protected I/O and disclosure stay blocked as for any failed entry. +Set a timeout on every probe and treat a probe that gets no answer as failed. +The `file` destination accepts entries whatever the collector does. +The runtime log waits on the collector under either destination, so run the runtime under a +collector that does not block standard output when it stops reading. + +{/* Evidence: crates/registry-platform-audit/src/writer.rs, STREAM_WRITE_DEADLINE, + AuditUnavailableReason::Stalled, StreamEvent::WrittenAfterRefusal, + a_stalled_stream_refuses_the_append_at_the_deadline_and_later_ones_at_once, and + readiness_follows_a_stall_and_returns_once_the_stream_drains; + crates/registry-breg/src/startup.rs, audit_writer, request_timeout(), timeout_problem(), and + operational_log_level(); + crates/registry-breg/src/runtime_config.rs, RawOperationalTimeouts; + crates/registry-breg/src/correlation.rs, finish_response(); + crates/registry-breg/src/api/mod.rs, authenticated_router() and probe_routes(); + crates/registry-breg/src/main.rs, initialize_logging_filter(). */} + This pairing is a property of a running process, and a request may be answered more than once, so read every response under a correlation. A process that crashes, is killed, exits, or shuts its runtime down while a write is in flight, or a destination that already stopped accepting entries, @@ -502,7 +534,7 @@ outside the new writer's retention directory. | `request-retention erase` is refused | The request entity's retention mode is `retain`, or the version is pinned. Run the dry run for the exact version first. | | `evidence-retention erase-expired` removes nothing | No retained assertion has passed its 24-hour window yet, or the cutoff precedes every expiry. The count is the whole report. | | A retry answers `410 idempotency.expired` | The key was spent by a successful write whose receipt horizon has passed. The write committed; read the record or its history instead of retrying, and send a fresh key only for a new write. | -| Readiness or an operation reports audit unavailable | Check destination permissions, free space, and the single-writer lock. Repair the destination and restart a stopped writer. For a mutation whose response failed after commit, inspect its receipt or domain history before retrying. | +| Readiness or an operation reports audit unavailable | Check destination permissions, free space, and the single-writer lock. Repair the destination and restart a stopped writer. With a `stdout` destination, check that its collector is reading the stream: a stalled writer resumes with no restart once it is, and until then a request or probe can go unanswered instead of refused. For a mutation whose response failed after commit, inspect its receipt or domain history before retrying. | | An operator command cannot reach PostgreSQL | Check the migration URL secret, the role names, and `SSL_CERT_FILE` for a private authority. | {/* Evidence: crates/registry-breg/src/history_maintenance.rs, lock_registry; diff --git a/docs/site/src/content/docs/operate/casework-retention.mdx b/docs/site/src/content/docs/operate/casework-retention.mdx index 091c20de8..0b55cb0dd 100644 --- a/docs/site/src/content/docs/operate/casework-retention.mdx +++ b/docs/site/src/content/docs/operate/casework-retention.mdx @@ -371,10 +371,18 @@ call is refused while the process keeps running. The runtime log carries `audit with the cause. Treat it as a disk, permission, or lock problem at the audit path rather than a database outage, fix it, and restart the process. -{/* Evidence: crates/registry-platform-audit/src/writer.rs, AuditWriter::append() and - AuditWriter::ready(); +A `stdout` destination whose collector stops reading is a stall, not a failed write. +After the writer has held one entry for 5 seconds, audited calls fail with `service.unavailable` +and `GET /ready` answers `503` until the collector takes the entry; the writer then accepts +entries again with no restart. +The runtime log does not name a stall, so `GET /ready` and the refused calls are the signal. + +{/* Evidence: crates/registry-platform-audit/src/writer.rs, AuditWriter::append(), + AuditWriter::ready(), STREAM_WRITE_DEADLINE, AuditUnavailableReason::Stalled, and + readiness_follows_a_stall_and_returns_once_the_stream_drains; crates/registry-casework/src/service.rs, ready(); - crates/registry-casework/src/http.rs, ready(). */} + crates/registry-casework/src/http.rs, ready(); + crates/registry-casework/src/main.rs, initialize_logging(). */} ## Troubleshooting diff --git a/docs/site/src/content/docs/operate/casework.mdx b/docs/site/src/content/docs/operate/casework.mdx index 529a1dcea..1fee6846e 100644 --- a/docs/site/src/content/docs/operate/casework.mdx +++ b/docs/site/src/content/docs/operate/casework.mdx @@ -859,6 +859,19 @@ the process's life: every later audited call is refused and `/ready` answers `50 process keeps running, so restart it once the cause is fixed. `/ready` also answers `503` when something else replaces or changes the audit file or its lock. +A `stdout` destination whose collector stops reading stalls instead, and a stall ends without a +restart. +Once the writer has held one entry for 5 seconds, the call waiting on that entry fails with +`service.unavailable`, later audited calls are refused at once, and `/ready` answers `503`. +When the collector takes the entry, the writer accepts entries again. +The runtime log does not name a stall, so `/ready` and the refused calls are the signal. +That log shares standard output with a `stdout` audit destination. +A call refused before its work starts and `/ready` write no log record, so their answers do not +wait on the stream. +A record the runtime does write while the collector is not reading, such as a background pass that +did not complete, waits with no deadline and holds a runtime thread until the collector reads +again, so set a timeout on the `/ready` probe and treat a probe that gets no answer as failed. + A maintenance pass runs every two seconds and a source reconciliation pass per bound source at that binding's `reconciliationIntervalMilliseconds`, every 60 seconds unless set. A source outage stays visible @@ -893,10 +906,12 @@ is among the items it reads. {/* Evidence: crates/registry-casework/src/runtime.rs, check_activation(), serve_from_path(), serve_until_worker_stops(), supervise(), operational_log_level(), and open_audit(); crates/registry-casework/src/audit.rs, CASEWORK_AUDIT_SCHEMA and AuditOperation::commit(); - crates/registry-platform-audit/src/writer.rs, AuditWriter::ready(); + crates/registry-platform-audit/src/writer.rs, AuditWriter::ready(), STREAM_WRITE_DEADLINE, + AuditUnavailableReason::Stalled, and + readiness_follows_a_stall_and_returns_once_the_stream_drains; crates/registry-casework/src/main.rs, initialize_logging(); crates/registry-casework/src/store.rs, migrate(); - crates/registry-casework/src/http.rs, health() and ready(); + crates/registry-casework/src/http.rs, health(), ready(), and StoreError::AuditUnavailable; crates/registry-casework/src/service.rs, ready() and assemble_caller_visible_item(); crates/registry-casework/src/assignment.rs, assignment_visible_item_for_origin(); crates/registry-casework/src/task_grants.rs, preview_tasks(), approve_task(), and list_tasks(); @@ -1063,7 +1078,7 @@ bound to the actor and selected profile for 15 minutes; an expired preview retur | `the Casework audit destination could not be opened: another process holds the single-writer lock beside the audit file` | Another `casework` process writes this file. Stop it, or give this process its own `audit.path`. | | `the Casework audit destination could not be opened: the audit file could not be opened: ...` | The rest of the message names the rule the audit file or its directory broke and the fix, for example `chown it to the service user and chmod it 0700`, or `archive it and restart with a fresh path` for a file whose last entry is incomplete. Apply it and start again. | | `caseworkctl doctor` stops at `the Casework audit destination is not writable by this user` | Run `doctor` as the runtime user, and fix the audit directory's owner and mode, or the audit file's, as the message names. | -| `GET /ready` returns `503` while `GET /health` returns `200` | The audit writer stopped, PostgreSQL is failing, or a source's reconciliation is failing. Run `caseworkctl doctor`, which names the check, then read the runtime log for `audit file write failed` or the reconciliation entry, and check the audit file, its lock, the free space, and the database. | +| `GET /ready` returns `503` while `GET /health` returns `200` | The audit writer stopped, PostgreSQL is failing, or a source's reconciliation is failing. Run `caseworkctl doctor`, which names the check, then read the runtime log for `audit file write failed` or the reconciliation entry, and check the audit file, its lock, the free space, and the database. With a `stdout` audit destination, also check that its collector is reading the stream: a stalled writer resumes with no restart once it is. | | `doctor` refuses at `doctor:/checks/reconciliation` | The named source's last passes failed. `source-unavailable` means the source did not answer; `source-refused` means it answered with a refusal or a response Casework cannot use; `store` and `configuration` point at the database and the binding. Repair the cause; the next pass that succeeds restores readiness. | | `doctor` refuses at `doctor:/checks/pinnedWork`, or `caseworkctl apply` refuses with `would strand work pinned under an earlier package` | The package removes a queue, profile, review producer, or source that in-flight work still needs, edits a pinned review kind without a new version, or changes what a source read discloses. Keep the earlier package until that work finishes, or set `package.acknowledgeStrandedWork` to the digest the refusal names. | | `BReg source ID runs engine version X and this Casework runs Y`, or `BReg source ID does not report its engine version` | Casework and that BReg source are on different releases, or a proxy removes the `Registry-Engine-Version` header. Upgrade the one that is behind as [the lock-step upgrade](#upgrade-casework-and-breg-in-lock-step) describes, or let the header through; source reads resume on the next matching read. | diff --git a/docs/site/src/content/docs/operate/evidence-audit.mdx b/docs/site/src/content/docs/operate/evidence-audit.mdx index 162a4ef90..aba50d61b 100644 --- a/docs/site/src/content/docs/operate/evidence-audit.mdx +++ b/docs/site/src/content/docs/operate/evidence-audit.mdx @@ -103,9 +103,37 @@ Never rename, edit, or truncate the active file while the service runs: the writ tampering, and readiness and every later audited request fail closed until the process restarts. Copying a sealed file is safe at any time. +A `stdout` destination stalls when its collector stops reading the stream. +Once the writer has held one entry for 5 seconds, the request waiting on that entry is refused, +later audited requests are refused at once, and readiness fails. +With `EVIDENCE_LOG` at its default of `info`, the log on standard error carries +`audit stream has held one entry past the write deadline`. +When the collector reads again, the log carries `audit stream took the entry it was holding`, and +the writer accepts entries with no restart. +The first record is logged at `error` and the second at `warn`, both under the +`registry_platform_audit` target, so an `EVIDENCE_LOG` filter that leaves out that target or those +levels hides them. +Keep standard error read as well. +Evidence Gateway writes both records to standard error from the request, readiness check, or +writer that noticed the change, and that write waits when standard error is full. +A collector that stops reading both streams can therefore leave a request or a readiness check +waiting rather than refused, until standard error is read again, so give callers and probes their +own deadlines. +The stream still receives the entry the collector was holding, so it can carry an entry for a +request that was refused. +Restore the collector rather than restarting Evidence Gateway: a process that exits during a stall +can lose the entries queued behind the held one. +A write the stream refuses is a different fault and stops the writer until the process restarts. + {/* Evidence: products/evidence/contracts/runtime.schema.yaml, audit; crates/registry-platform-audit/src/writer.rs, DEFAULT_AUDIT_ROTATE_BYTES, - DEFAULT_AUDIT_RETAIN_DAYS, and AuditDestination. */} + DEFAULT_AUDIT_RETAIN_DAYS, AuditDestination, STREAM_WRITE_DEADLINE, + AuditUnavailableReason::Stalled, StreamEvent::StallStarted, StreamEvent::StallEnded, + StreamEvent::WrittenAfterRefusal, StreamEvent::AbandonedAtDrop, + a_stalled_stream_refuses_the_append_at_the_deadline_and_later_ones_at_once, and + readiness_follows_a_stall_and_returns_once_the_stream_drains; + crates/registry-evidence/src/runtime.rs, ready(); + crates/registry-evidence/src/main.rs, install_operational_logging. */} ## Ship audit to append-only storage diff --git a/docs/site/src/content/docs/operate/registry-render.mdx b/docs/site/src/content/docs/operate/registry-render.mdx index a30da1236..72462efb4 100644 --- a/docs/site/src/content/docs/operate/registry-render.mdx +++ b/docs/site/src/content/docs/operate/registry-render.mdx @@ -200,10 +200,13 @@ curl -s http://127.0.0.1:8080/ready `GET /health` answers with the bundle version and package hash and the renderer version, so you can reconcile the running service against what you deployed. `GET /ready` includes the audit writer: -if it has stopped, readiness fails. Both are value-free. +if it has stopped, or its `stdout` destination has stalled, readiness fails. Both are value-free. `registry-render healthcheck --runtime-config ` performs the same probe for a supervisor script. +A stalled destination recovers when its collector reads the stream again, and a stopped writer needs +a restart; [audit retention](../retention-and-persistent-state/#audit-retention) describes both. -{/* Evidence: crates/registry-render/src/server.rs, health and healthcheck; +{/* Evidence: crates/registry-render/src/server.rs, health, ready(), and healthcheck; + crates/registry-platform-audit/src/writer.rs, AuditWriter::ready(); crates/registry-render/src/openapi.rs, OPENAPI_JSON. */} ## The rendering call diff --git a/docs/site/src/content/docs/operate/retention-and-persistent-state.mdx b/docs/site/src/content/docs/operate/retention-and-persistent-state.mdx index 20c8d42f7..b0a999976 100644 --- a/docs/site/src/content/docs/operate/retention-and-persistent-state.mdx +++ b/docs/site/src/content/docs/operate/retention-and-persistent-state.mdx @@ -60,6 +60,28 @@ Never truncate, rewrite, or reserialize the active audit file as part of cleanup treats any external change to it as a fault and stops, and audited requests fail closed until a restart. Copy sealed files to append-only storage before the audit retention period deletes them. +A `stdout` destination can also stall, and a stall ends without a restart. +When the log collector stops reading the stream and the writer has held one entry for 5 seconds, +the writer refuses the request waiting on that entry, refuses later audited requests at once, and +reports not ready. +When the collector takes the entry, the writer accepts entries again. +The 5 seconds are fixed, with no configuration key. +The writer also bounds the entries waiting behind the one it is writing: a request that finds that +queue full is refused at once, with no stall logged and with the writer still able to report +ready, so readiness alone does not show it. +A collector that reads slowly without stopping has the same effect: a request whose entry waits 5 +seconds behind earlier entries is refused and its entry is not written, although the writer held +no single entry for 5 seconds, logs no stall, and still reports ready. +A write the stream refuses is a different fault: it stops the writer until a restart, as a failed +file write does. + +{/* Evidence: crates/registry-platform-audit/src/writer.rs, STREAM_WRITE_DEADLINE, + STREAM_QUEUE_LINES, AuditUnavailableReason::Stalled, LineStream::append, StreamState::run, + a_stalled_stream_refuses_the_append_at_the_deadline_and_later_ones_at_once, + readiness_follows_a_stall_and_returns_once_the_stream_drains, + a_full_stream_queue_refuses_a_line_instead_of_growing, and + a_write_error_still_stops_the_stream_after_its_sink_recovers. */} + A crash, a killed process, or a full disk can leave the active file's final line incomplete (any byte other than a newline at end of file). The next start recovers it without operator action: the writer copies the bytes after the last complete line to the owner-only (`0600`) side file