Repository navigation
docs(platform): tell operators how a stalled stdout audit stream behaves - #2027
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 69533254f1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 77aad27fe8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2b6034f55a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
aaaeda0 to
34182ef
Compare
A stdout audit destination whose collector stops reading now refuses audited requests after a fixed 5 second write deadline, reports not ready, and resumes by itself once the collector takes the held entry. The operator pages only described the stopped writer that needs a restart, so an operator reading them would restart a process that recovers on its own, and during a stall a restart can lose the entries queued behind the held one. Describe the stall, its two log messages, and the recovery on the retention overview and on the Base Registry Engine, Evidence, Casework, and Registry Render operate pages, with the troubleshooting rows that point at it. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…work operators can observe The Base Registry Engine and Casework log filters admit only their own targets, so neither runtime log carries the audit writer's stall records. The pages now point operators at readiness and the refused requests. Both runtimes log to the standard output a stdout audit destination writes. The pages say what that means while the collector is not reading: a Base Registry Engine request or probe can go unanswered, because one log record is written before each answer leaves, and a Casework record written meanwhile holds a runtime thread. The Base Registry Engine page also says that a request timeout shorter than the write deadline answers 504 first. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…ry from Render A request that finds the stream queue full is refused while the writer can still report ready, so the shared page says readiness alone does not show it. The Render page links the recovery guidance for a stalled destination and a stopped writer, and its evidence names the readiness probe it describes. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…e its work starts Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…udit stream reader The two stall records reach an Evidence operator only where the EVIDENCE_LOG filter admits their target and levels, and a collector that reads slowly without stopping refuses requests with no stall and with readiness still true. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
2b6034f to
297390e
Compare
…rror read The two stall records go to standard error from the request, readiness check, or writer that noticed the change, and that write blocks when standard error is full. A collector that stops reading both streams can leave a request waiting rather than refused. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
1 participant
Stacked on #2019. Review and merge that one first; this branch targets it.
Why
#2019 gives a
stdoutaudit destination a fixed 5 second write deadline: when the log collector stops reading, audited requests are refused, readiness fails, and the writer resumes by itself once the collector takes the held entry. The operator pages only describe the stopped writer that needs a restart. An operator following them would restart a process that recovers on its own, and a process that exits during a stall can lose the entries queued behind the held one.What changes
Operator documentation only, under
docs/site/src/content/docs/operate/:retention-and-persistent-state.mdx: the stall, the fixed deadline with no configuration key, and how it differs from a refused write; the full queue and the slow reader, two cases in which requests are refused with no stall logged and with readiness still true.breg-retention.mdx: the stall, what an operator can observe of it, the shared standard output limit, and the troubleshooting row for "audit unavailable".evidence-audit.mdx: the stall and its two log messages, with theEVIDENCE_LOGdefault, target, and levels they depend on; that the held entry is still written; and the instruction to restore the collector instead of restarting.casework.mdxandcasework-retention.mdx: the stall,service.unavailable,/ready503, and the troubleshooting row.registry-render.mdx: readiness also fails on a stalledstdoutdestination, with a link to the recovery guidance for a stalled destination and a stopped writer.Each paragraph carries an evidence anchor to
crates/registry-platform-audit/src/writer.rs(STREAM_WRITE_DEADLINE,AuditUnavailableReason::Stalled, the stream events, and the tests that hold the behaviour).What each operator can observe
The pages claim only what each operator can observe. The Evidence page keeps the two stall log lines, because Evidence logs the audit writer's records on standard error. The Base Registry Engine and Casework pages do not promise those lines, because those runtimes' log filters drop them: the signal there is readiness and the refused requests.
Both pages also state the shared standard output limit. With a
stdoutaudit destination and a collector that is not reading, a Base Registry Engine request or probe (/healthincluded) can go unanswered, because the runtime writes one log record before each answer leaves; the refusal itself still holds. In Casework a call refused before its work starts and/readywrite no log record, but any record written meanwhile holds a runtime thread. Both pages ask for a timeout on probes.The Base Registry Engine page adds that a request reaching
operationalTimeouts.httpRequestMillisecondsbefore the 5 second deadline answers504request.timeout. The shared retention page adds that a full stream queue refuses requests while the writer can still report ready.The shared standard output behavior and the log filtering are confirmed by reading the code, not by running a stalled collector against each binary.
Verification
From
docs/site, on the branch head:npm run checkpassed (evidence anchors, markdown, style, content, build, 253693 built links) andnpm testpassed, 690 tests.Not in this change
breg-mcp, orbreg-review. Left as it is.messaging.mdxsays a failed writer stays unhealthy until restart. That stays true, but the page does not mention the stall. I did not verify how Messaging readiness reports a stall, so that page,breg-mcp.mdx, and the two requirements pages are unchanged.breg-mcp, andbreg-reviewlogs, and bounding a runtime log write to a stalled standard output, are product changes and are not made here.audit file write failed, which the Casework log filter also drops. That text predates this change and is left as it is.