Skip to content

feat(cp): land Agent Control Plane v1 on main - #1528

Closed
chaodu-agent wants to merge 5 commits into
mainfrom
feat/cp-runtime-client-main
Closed

chaodu-agent wants to merge 5 commits into
mainfrom
feat/cp-runtime-client-main

Conversation

@chaodu-agent

@chaodu-agent chaodu-agent commented Sep 8, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Land the complete Agent Control Plane v1 stack on current main:

  • CP protocol, policy, registry, routing, lifecycle, and observer surface
  • runtime registration, heartbeat/reconnect, worker delegation serving, and headless worker mode
  • primary-side delegation state with reconnect-safe, count-and-byte-bounded terminal history
  • owner-only local Unix socket with opaque handles shared by MCP and CLI
  • four session-gated direct MCP tools: spawn_agent, check_delegation, list_agents, cancel_delegation
  • openab agent spawn|list|status|cancel
  • automatic MCP facade/session injection for adapter-less headless primaries

This PR began as the recovery replay for #1471 after that PR was merged into an obsolete stacked base rather than main. Stacked PR #1529 has now merged into this branch, so #1528 is the final roll-up that makes the complete reviewed stack reachable from main.

Discord Discussion URL: https://discord.com/channels/1491295327620169908/1491365158868619404/1532377620241449040

Review Contract

Goal

Make Agent Control Plane v1 reachable from main: Unix-hosted OpenAB runtimes can register as primaries/workers, serve and initiate delegations over one CP WebSocket, and expose the initiating surface through session-gated MCP tools and the equivalent owner-only local CLI API without exposing CP credentials to agents or operators.

Non-goals

  • CP-side DAG/pipeline scheduling, durable inboxes, offline delivery, or multi-replica HA.
  • Human relay (cp/attach), session streaming, or lobby UI.
  • Standalone openab-cp image/release packaging and production canary deployment.
  • Sweeper supervision/readiness and container-grade shutdown/drain; tracked separately in fix(cp): sweeper task has no panic supervision or liveness signal #1474 and feat(cp): container-grade shutdown — SIGTERM handling and connection drain #1475.
  • Local CP API, CLI, or MCP transport on Windows/non-Unix. This surface is Unix-only by design; Windows/non-Unix runtime support is not planned, and fail-loudly stubs exist only for build compatibility.
  • Per-session MCP facade credential injection for non-acp: sessions. In v1 the facade session token (and the agent's facade entry) is minted only for sessions opened through the ACP-over-WebSocket gateway; chat-adapter sessions and adapter-less primaries initiate through the openab agent CLI. The docs, ADR, and startup log state this reach explicitly (round-4 F2).

Accepted Residual Risks

  • Opaque local-handle storage is bounded at 4096 tokens with deterministic oldest-first eviction. An aged-out handle returns unknown handle; eviction does not cancel live CP work.
  • Terminal history is independently bounded at 4096 entries and 16 MiB. It evicts oldest completions only and never live work.
  • Blocking spawn uses the requested deadline plus a five-second local response margin; async mode is available for long-running work.
  • An automatic headless-primary facade binds loopback port 8848 when [mcp] is absent. A bind conflict fails visibly. Its four tools are reachable only from acp: sessions (see Non-goals); the startup report names the mode and that reach.
  • Primary-side pending state is bounded: at most 256 requests outstanding to the CP (Overloaded beyond), unanswered requests failed Timeout after 60s, and a delegation still non-terminal 60s past its own deadline dropped from tracking with waiters answered Timeout. A terminal frame arriving after that is acked and ignored.
  • The local socket server serves at most 64 concurrent connections (further connections receive one error line and are closed), idles connections out after 5 minutes, and bounds every CP round trip; handlers are aborted on shutdown.
  • One runtime per socket path: a second runtime under the same uid refuses to start over a socket that answers a connect probe and must be given a distinct OPENAB_AGENT_SOCKET.
  • Every runtime WebSocket write is bounded at 30s (mirroring the CP's write_timeout_secs); a stall is treated as a disconnect and recovered by reconnect.

Acceptance Criteria

  • Runtime config, registration, heartbeat/reconnect, lease handling, and worker execution are present and headless-compatible.
  • Primary requests share the single WebSocket writer/id space and correlate by (delegation_id, admission) with first-terminal-wins semantics.
  • Primary state survives reconnects; terminal retention is bounded by count and bytes, completion-ordered, and never evicts live work.
  • Local handles are random opaque 256-bit tokens; no CP key/address or protocol admission token is exposed to MCP/CLI callers.
  • UDS parent/socket permissions are 0700/0600; existing parents are validated without chmod and stale unsafe entries are refused.
  • JSON-lines framing preserves pipelined requests and strictly bounds each line at 1 MiB.
  • Delegate ack ids are validated and oversized outbound spawn frames fail locally without dropping the CP connection.
  • The four direct MCP tools are session-token gated and invisible/unreachable to anonymous loopback clients.
  • MCP and CLI use the same local API, delegation-id generator, validation, and admission-inclusive timeout path.
  • Real integration tests drive primary UDS -> primary runtime -> real CP -> worker runtime for list/spawn/await/cancel and connection-survival cases.
  • No CP key, prompt, or raw tool arguments enter child env or logs.
  • Current-main merge-ref CI, Docker smoke matrix, review-contract check, and final regression review all reach green terminal states.

Follow-ups

  1. fix(cp): sweeper task has no panic supervision or liveness signal #1474: supervise sweeper health/readiness.
  2. feat(cp): container-grade shutdown — SIGTERM handling and connection drain #1475: SIGTERM, connection close, and delegation drain.
  3. Standalone openab-cp image/release workflow and single-replica canary.
  4. cp/attach, session streaming, lobby UI, and durable event history.
  5. Per-session facade credential injection for non-acp: sessions (chat adapters, adapter-less primaries), including rotation/revocation for long-lived chat sessions.
  6. refactor(cp): post-v1 hardening follow-ups from #1528 self-review #1530: remaining post-v1 hardening (managed-parent dir verification, deadline-range single source of truth, max_frame_bytes bail string, heartbeat test margin).

Architecture

agent MCP call                         operator CLI
      |                                     |
      v                                     v
session-token-gated facade        openab agent <verb>
      |                                     |
      +----------- JSON over UDS -----------+
                    0600 socket
                         |
                         v
             primary command/state machine
                         |
                  registered CP WebSocket
                         |
                         v
                    openab-cp
                         |
                         v
                 worker ACP session

Review History

Verification

Validated on macmini before the stacked merge:

  • workspace --all-features: passed
  • openab-core and root --no-default-features: passed
  • production clippy (openab-core, openab-mcp, root): passed with -D warnings
  • openab-cp: 142 unit + 9 integration passed
  • openab-mcp: 245 passed
  • openab-core: 848 unit + 11 real-CP integration passed (one known unrelated macOS secret test, secrets::tests::resolve_exec_nonzero_exit, fails identically on main)
  • root binary: 65 passed
  • cargo check --workspace --all-features --locked, cargo check -p openab-core --no-default-features --locked, cargo check --no-default-features --locked: passed at c061158; 91c0a89 re-verified on macOS and ext4 Linux (850/850 core lib on Linux)
  • rustfmt --check on crates/openab-core/src/control_plane/*.rs: clean
  • independent Sol/Fable fix and final-regression audits: PASS / PASS

Current-main GitHub merge-ref CI is running on 91c0a89; the final checkbox above will be updated only after all required checks are terminal and green.

…ing, headless mode (PR 3/4) (#1471)

* feat(cp): runtime control-plane client — [control_plane] config, worker delegation serving, headless mode (PR 3/4)

PR 3/4 of the Agent Control Plane stack: the OAB runtime side. The runtime
dials out to the CP, registers, serves `cp/delegate` through the same ACP
turn driver every chat platform uses, and reports each turn as
`cp/delegate_result`.

Config surface (`[control_plane]`, `crates/openab-core/src/config.rs`)
- `url`, `auth_key`, `namespace`, `name`, `type` (`primary` | `worker`),
  optional `labels`, `max_delegated_sessions` (default 1).
- Presence is the opt-in — no cargo feature — following the `[mcp]`
  precedent. `deny_unknown_fields`, so a mistyped key fails startup instead
  of defaulting silently. `${ENV}` expansion comes from the shared loader.
- `CpAgentType` has only Primary and Worker: `observer` is a read-only lobby
  role, so a runtime claiming it is unrepresentable rather than merely
  validated against.
- Validation: non-empty url/auth_key/namespace/name, and
  `max_delegated_sessions > 0`. Each of these would otherwise fail at
  registration and reconnect forever.

State machine (`control_plane/client.rs`)
- One process-lifetime instance id (uuid), reused across reconnects: it
  identifies the replica, not the connection.
- Connect ws/wss with `Authorization: Bearer` (never in the URL), send
  `cp/register` as the first frame, adopt `effective_max_delegated_sessions`
  from the ack, then one loop over inbound frames / heartbeat tick /
  delegation completions / shutdown watch.
- On close or error: cancel every local in-flight delegation (each stops its
  agent and drops its session; the results are deliberately dropped, since
  the CP synthesizes `target_disconnected`), then 1/2/4/8/16/30s
  shutdown-aware backoff, reconnect, re-register.
- CP-issued requests get JSON-RPC result acks; a delegation's outcome never
  rides in that ack.

Delegation serving (`control_plane/executor.rs`)
- Admission never executes: over-cap or duplicate `delegation_id` returns
  `status = failed` without touching the session pool.
- Bounded by the nearer of the CP deadline and
  `[pool].prompt_hard_timeout_secs`. Outcomes map to
  completed / failed (agent error, broker error, silent failure) /
  timeout / cancelled.
- One fresh session per delegation, keyed
  `control-plane:<sha256(instance_id + delegation_id)>`, discarded on every
  terminal outcome. Adds `SessionPool::discard_session` for the non-cancel
  teardown (`reset_session` would emit a spurious `session/cancel` and log an
  error for the benign already-gone case).

Prompt seam refactor safety
- `AdapterRouter::stream_prompt_blocks` now returns
  `Result<PromptExecution { final_text, terminal_error, silent_failure }>`.
  Nothing else changed inside it: two values are captured before the existing
  composition consumes them, and the delivery paths are byte-for-byte the
  same. `Err` still means what it meant (incomplete user-visible delivery).
- All platform callers map back with `.map(|_| ())`, and `DispatchTarget`
  stays at `Result<()>`, so Discord/Slack/gateway/ambient — and the existing
  dispatch mock — are untouched. Every pre-existing adapter test passes
  unmodified.
- The executor reuses that same driver with a private sink adapter (send-once,
  no placeholder, no reactions) and reads the reply from the return value.

Feature split (Dockerfile-neutral)
- `openab-cp` gains a default `server` feature gating config/registry/policy/
  router/events/server and the binary (`required-features`); `proto` — the
  wire contract — stays unconditional, as do serde/serde_json/chrono.
- `openab-core` depends on `openab-cp` with `default-features = false` for
  wire types only. Verified: `cargo tree -p openab-core -e normal` shows only
  chrono/serde/serde_json under openab-cp, an inverted axum query finds
  nothing, and `cargo build -p openab-cp` still produces the binary.
- No new crate, no new runtime cargo feature.

Headless mode (`src/main.rs`)
- `[agent]` + `[control_plane] type = "worker"` with no adapter is a valid
  boot; a `primary` alone is not (its prompts come from a chat platform).
  `[mcp]`-only is unchanged, and mcp + worker runs both.
- Extracted as `headless_run_mode` so the matrix is unit-tested.
- Client spawned after router construction, wired to the existing shutdown
  watch, and stopped before `pool.shutdown()` so the CP sees a clean
  disconnect rather than a lease timeout.

Security
- `auth_key` is used only for the upgrade header (marked sensitive), is never
  logged, and never enters the agent child env — `[agent].env` plumbing is
  untouched.

Tests
- Config: absent section, full section incl. `type` rename and `${ENV}`,
  defaults, observer refusal, unknown-key refusal, every validation failure.
- Run modes: the full headless matrix.
- Executor: cap enforcement, duplicate id, all outcome mappings, and
  cancel/deadline cleanup asserting the session is gone.
- `crates/openab-core/tests/cp_client.rs`: seven tests against the REAL
  in-process CP (ephemeral loopback port + `run_sweeper`), with a raw-WS
  primary as initiator and a scripted `PromptRunner` at the trait seam —
  register/ack, heartbeats holding the lease, delegate→result round trip,
  local deadline → `timeout`, cancel mid-flight, zero-lease sweep →
  reconnect + re-register (same instance id, and the new registration
  serves), and clean deregistration on shutdown.

Deviations from the design
1. `openab-cp`'s `config.rs` is behind `server` too. The design said "proto.rs
   (+ config types needed by proto) stays unconditional"; proto needs nothing
   from config (the dependency runs the other way), so gating it keeps anyhow
   and toml out of the runtime build.
2. Completed delegations report through an mpsc channel rather than a
   `JoinSet` polled in the `select!`. The inbound branch must spawn while the
   completion branch is armed, and one `select!` cannot lend the same
   `JoinSet` to both — the borrow checker rejects it. Same for the socket,
   which is split into sink/stream halves. Behaviour is unchanged; handles are
   still tracked for the drain-then-abort teardown.
3. `axum` added as a dev-dependency of `openab-core`: `openab_cp::server::app`
   returns an `axum::Router`, and the integration test needs `axum::serve` to
   bind it. Test-only — the runtime links no HTTP server.

Audit round-1 fixes (all four blocking findings):

1. [mcp] + [control_plane type=primary] now takes the full boot path so the
   facade and the CP client both run; facade-only had foreclosed the client.
   Run-mode test updated to pin both-run.
2. Reconnect backoff no longer resets on every clean close: a session must
   live STABLE_SESSION_SECS (60s) before a close resets backoff to 1s, so a
   CP that accepts-then-closes (lease misconfig, crash loop) faces
   1/2/4/8/16/30 escalation instead of a 1s reconnect storm.
3. The executor's best-effort session/cancel before discard is now bounded
   at 5s (the pool's own cleanup bound): a wedged agent stdin can no longer
   block discard and leak the delegation slot. Both sites (cancel, deadline).
4. Shutdown is storm-proof: the dial/register phase races the shutdown
   signal via select (a hung connect cannot stall shutdown), and main aborts
   the client task if it misses the 10s deadline instead of detaching it
   over pool teardown.

* fix(cp): cap delegate_result bodies below the CP transport frame limit

Self-review finding. The worker sent the agent's full final text in
cp/delegate_result (SinkAdapter deliberately does no chunking), but the CP
enforces max_frame_bytes (default 1 MiB) at the WS transport BEFORE parsing —
its router-level max_result_bytes truncation can never save an oversized
frame. A single >1 MiB result therefore dropped the whole connection, and
every in-flight delegation on that worker died as target_disconnected.

The executor now caps Completed results at 512 KiB (half the default frame
budget, leaving JSON-envelope headroom) with a UTF-8-boundary-safe truncation
marker; the CP still applies its own, typically smaller, max_result_bytes on
arrival. Regression test covers the oversized, multibyte-straddling, and
under-cap cases.

* fix(cp): free the delegation slot via RAII so task abort cannot leak it

Self-review round-2 finding. serve() released its inflight slot with a plain
call after the execute().await — but the client ABORTS serving tasks that
outlive the 5s drain window on disconnect, and an aborted task never runs
code after its await point. The leaked entry then survives reconnection
(executor state is process-lifetime), permanently consuming capacity: with
the default max_delegated_sessions = 1, one abort turns the worker into a
zombie that refuses every delegation while heartbeating happily.

The window is realistic, not theoretical: the bounded (5s) session/cancel
added in the previous fix means a wedged agent stdin makes the task overrun
the equally-sized drain window by construction.

release() now lives in a Drop guard taken at admission, so the abort's
unwind frees the slot. Regression test aborts a serving task mid-run and
asserts active() == 0; mutation-verified (plain-release pattern fails it
5/5, the guard passes).

* fix(cp): echo the CP admission token in every delegate_result and match it on cancel

Round-8 made the admission token protocol-visible on the result path;
round-9 extended it to cancellation. The runtime side of both:

- The executor carries forward.admission into all five terminal frame
  kinds (completed, failed, refused-at-admission, timeout, cancelled).
- The worker-side cancel handler matches (delegation_id, admission):
  a stale cancel naming a superseded admission of a reused id is ignored
  instead of aborting the live delegation — the worker-side half of the
  misdelivery the wire token closes. Regression pins both directions.
- The initiator-side integration test echoes the ack's admission in its
  cp/cancel, exercising the CP's required-field validation end-to-end.

The real-server integration suite is the first live client of the
token-enforcing contract on both the result and cancel paths.

* fix(review): cap error payloads, admission-scope session keys, bound teardown

Round-7 review fixes:

- F1: failed() now caps every error string (64 KiB, UTF-8-safe) inside the
  constructor so no call site can bypass it; oversized-error regression test
  mirrors the success-path one
- F34: delegation_session_key mixes in the CP admission token, so a
  re-admission of a reusable id can never resume an earlier admission's
  session (e.g. one orphaned by a drain-timeout abort); false reconnect
  claim in the doc corrected; key-property test pins admission scoping
- F22/F31: duplicated cancel+discard blocks extracted into bounded helpers;
  discard now shares the 5s bound instead of being unbounded on every
  terminal path
- F15: client WS now sets max_message_size/max_frame_size (1 MiB),
  mirroring the CP's accept-side max_frame_bytes instead of tungstenite's
  64 MiB default
- F5: the cp/register ack wait is bounded (10s, mirroring the CP's
  register_timeout_secs) so a stalled CP enters backoff instead of hanging
- F27: config/docs no longer claim a local min() the code does not
  implement — the runtime enforces the ack value
- F37: ADR section 7 facade label corrected from PR 3/4 to PR 4/4

* fix(review): bound escaped result payloads, share teardown bound, defer success discard

Round-8 review fixes:

- F46: cap_text and cap_result now evaluate JSON-escaped byte length
  instead of raw UTF-8 length so control-character and quote/slash escape
  inflation cannot inflate a 512 KiB result past the CP's pre-parse 1 MiB
  max_frame_bytes limit; regression test pins escape-heavy and quote-heavy
  inflation safety; client::send() adds an outbound transport size guard
- F52: cancel_and_discard() now shares a single 5s deadline across both
  the cancel and discard phases (discard uses whatever budget remains,
  or is skipped if cancel consumed the bound), keeping total teardown
  strictly <= DRAIN_TIMEOUT (5s); regression test pins stalled cancel +
  stalled discard bound
- F53: success path now spawns session discard in the background off the
  critical path, preventing a slow or wedged discard from delaying result
  delivery to the initiator; safe because session keys are admission-scoped
- F54: executor module-level invariant doc updated to state the
  three-parameter session key (instance_id, delegation_id, admission)
- F55: truncation marker uses ASCII "..." instead of U+2026 ellipsis
- F51: documented MAX_INBOUND_FRAME_BYTES relation to CP-side configurable
  max_frame_bytes and future RegisterAck negotiation direction

* fix(clippy): remove useless format! invocations

* fix(review): route register through send() guard, clarify primary headless doc (review nits)

* fix(cp): resolve runtime client review findings

* fix(cp): address runtime client regression findings

* docs(cp): document delegation context schema

---------

Co-authored-by: chaodu-agent <chaodu-agent@users.noreply.github.com>
@chaodu-agent

This comment has been minimized.

* feat(cp): add primary delegation tools and local API

* fix(control-plane): address PR4 review findings

* fix(control-plane): order terminal eviction by completion

---------

Co-authored-by: chaodu-agent <chaodu-agent@users.noreply.github.com>
Co-authored-by: chaodu-agent <274062505+chaodu-agent@users.noreply.github.com>
@chaodu-agent chaodu-agent changed the title feat(cp): land runtime control-plane client on main feat(cp): land Agent Control Plane v1 on main Sep 9, 2026
@chaodu-obk

This comment has been minimized.

- Cargo.lock: #1529 added the `reqwest013` dev-dependency to the root
  crate but the lockfile's `openab` block was never regenerated, so
  `cargo check --workspace --all-features --locked` failed and any plain
  `cargo check` rewrote the lock. Regenerated; `--locked` now passes.
- main.rs: `report_facade_status` was keyed on `cfg.mcp.is_some()`, so a
  headless control-plane primary without `[mcp]` logged "unconfigured —
  nothing was started" while the automatic loopback facade on
  127.0.0.1:8848 was in fact serving. Key it on `facade_serving`, the
  same bool that decides whether the listener is spawned.

Self-review round 3 findings F1/F2 on #1528.
@chaodu-agent

This comment has been minimized.

@chaodu-obk

This comment has been minimized.

@chaodu-obk chaodu-obk Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

CHANGES REQUESTED ⚠️ - Two contract-level correctness failures and four liveness/resource defects must be fixed before merge.

Consolidated review: #1528 (comment)

Comment on lines +470 to +484
match self.classify_inbound(&text, primary) {
FrameAction::Serve { ack, forward } => {
let executor = Arc::clone(&self.executor);
let tx = result_tx.clone();
serving.push(tokio::spawn(async move {
let result = executor.serve(forward).await;
// A closed channel means the connection
// that would carry this result is gone;
// the CP fails it as target_disconnected.
let _ = tx.send(result).await;
}));
// Prune finished tasks so a long-lived
// connection does not accumulate handles.
serving.retain(|h| !h.is_finished());
if sink.send(Message::Text(ack)).await.is_err() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 F1 - Preserve cancellation before worker admission

This task is spawned before executor.serve is polled and inserts the admission. If the next ordered frame is cp/cancel, executor.cancel() sees no inflight entry, the cancel is still acknowledged, and this task can later admit and run cancelled work.

Requested change: reserve admission synchronously before spawn/ack, or retain admission-scoped pre-cancel tombstones. Add a deterministic test that handles cancel before the serving future's first poll and asserts the runner never starts.

Comment thread src/main.rs Outdated
Comment on lines +803 to +817
let facade_sessions = openab_mcp::mcp::sources::SessionTokens::new();
// Only read under the acp feature (pool facade wiring below).
#[cfg(feature = "acp")]
let facade_serving = cfg.mcp.is_some();
let facade_serving = facade_listen.is_some();
// Startup, not per-session: report whether the facade is serving, so an operator learns it
// here rather than by inferring it from tools that never appear. This is NOT the
// `OPENAB_BROWSER_MODE` migration notice — that was removed (see `acp_mcp`), and nothing
// reports the variable now.
// Gated on `acp` (the root feature that pulls in core's `acp-mcp`), not on `acp-mcp` itself —
// that is a core feature and naming it here is an unknown-cfg error.
//
// Keyed on `facade_serving`, NOT `cfg.mcp.is_some()`: a headless control-plane primary
// without `[mcp]` still gets the automatic loopback facade (see `facade_listen` above), and
// reporting "nothing was started — add [mcp]" while that listener is up would contradict
// the running process.
#[cfg(feature = "acp")]
openab_core::acp_mcp::report_facade_status(cfg.mcp.is_some(), &cfg.agent.working_dir);
if let Some(mcp_cfg) = cfg.mcp.clone() {
let listen = mcp_cfg.listen.clone();
openab_core::acp_mcp::report_facade_status(facade_serving, &cfg.agent.working_dir);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 F2 - Make automatic headless-primary tools reachable

This automatic path starts with an empty SessionTokens registry, while ControlPlaneTools requires a resolved session. The only production mint path is creation of an acp: session, but an adapter-less primary creates no chat session and does not serve worker delegations. The four advertised tools are therefore hidden/unreachable. This status call also runs before bind and selects text claiming [mcp] is configured and an entry was written, which is false here.

Requested change: implement a real bootstrap session/token plus agent configuration or environment injection for this mode, test an authenticated tool call end to end, and report readiness only after bind/injection succeeds.

Comment on lines +351 to +360
pub struct PrimaryState {
/// JSON-RPC id → the request the loop originated and awaits a reply for.
pending: HashMap<u64, PendingRequest>,
/// (delegation_id, admission) → its lifecycle.
tracked: HashMap<DelegationKey, TrackedDelegation>,
/// Insertion order for deterministic oldest-terminal history eviction.
tracked_order: VecDeque<DelegationKey>,
/// Callers awaiting a terminal result, keyed by the same composite key.
/// A caller can await before OR after the terminal arrives.
awaiters: HashMap<DelegationKey, Vec<AwaitReply>>,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 F3 - Bound and expire parked primary requests

The mpsc ingress is capped, but entries moved into pending and awaiters have no count cap or expiry. A connected CP can accept writes while withholding replies or terminal frames, allowing these maps and the Unix-socket tasks waiting on their oneshots to grow for process life. Caller-side spawn timeout does not remove the parked server-side entry.

Requested change: add per-request expiry, caller-drop cleanup, fixed pending/handler caps, and bounded local I/O/response waits. Test a connected-but-withholding peer and assert state returns to a fixed bound.

Comment on lines +860 to +872
Ok(frame)
}

async fn send(sink: &mut WsSink, frame: &JsonRpcRequest) -> anyhow::Result<()> {
let text = serde_json::to_string(frame)?;
if text.len() > MAX_FRAME_BYTES {
anyhow::bail!(
"outbound frame ({} bytes) exceeds transport limit ({} bytes)",
text.len(),
MAX_FRAME_BYTES
);
}
sink.send(Message::Text(text)).await?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 F4 - Put a deadline on every runtime WebSocket write

This helper awaits the sink without a timeout, and several ack/reply/pong writes bypass even this helper. Because writes run inside the sole serve loop, one backpressured peer stops inbound processing, heartbeats, completions, command traffic, and graceful shutdown.

Requested change: route all writes through one bounded helper raced against shutdown; on timeout, disconnect, fail/cancel live state, and reconnect. Add a stalled-sink regression test.

Comment on lines +43 to +52
pub fn delegation_session_key(
instance_id: &str,
delegation_id: &str,
admission: AdmissionToken,
) -> String {
let mut hasher = Sha256::new();
hasher.update(instance_id.as_bytes());
hasher.update(delegation_id.as_bytes());
hasher.update(admission.to_be_bytes());
format!("control-plane:{:x}", hasher.finalize())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 F5 - Reclaim creation gates for one-shot admission keys

A unique session key per admission is correct for isolation, but SessionPool::get_or_create retains a strong creating gate per key and discard_session deliberately does not remove it. Every completed delegation therefore leaves a permanent string, Arc, and mutex; max_sessions does not bound this map.

Requested change: preserve same-key serialization with reclaimable gates (for example Weak entries plus identity-checked final-holder cleanup) and test many unique admissions while asserting bounded gate cardinality.

mut shutdown: watch::Receiver<bool>,
) -> anyhow::Result<()> {
prepare_socket_dir(&path)?;
remove_stale_socket(&path)?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 F6 - Do not unlink a live same-UID runtime socket

remove_stale_socket checks type and UID but not liveness or path generation. A second runtime can unlink the first live pathname and bind a successor; the older runtime's unconditional shutdown cleanup can then unlink that successor.

Requested change: hold an exclusive lifetime lock, refuse a live listener, and make cleanup generation/inode-aware. Add two-live-runtime and predecessor-cleanup tests.

F1 cancel-before-admission: the connection loop spawned `executor.serve`,
so admission happened on the task's first poll and a `cp/cancel` that was
the very next frame found nothing in flight, was acked as unknown, and
the task then admitted and ran cancelled work. `serve` is split into a
synchronous `admit_delegation` (called on the loop's own task before it
yields) and `Admitted::run`; the reservation is an owned RAII value so an
ack-write failure or task abort still frees the slot. Refusals are
answered inline after the ack.

F4 unbounded writes: every WebSocket write in the single-owner loop goes
through one bounded helper (30s, mirroring the CP's write_timeout) raced
against shutdown. A stall classifies the connection as disconnected and
reconnects; shutdown never waits on a stalled peer.

F3 unbounded pending state: PrimaryState caps outstanding CP requests at
256 (`Overloaded` beyond), ages unanswered requests out after 60s
(`Timeout`), drops awaiters whose callers left, and drops tracking for a
delegation still non-terminal 60s past its deadline, answering its
waiters `Timeout`. `prune` runs on every heartbeat tick. The local socket
server bounds concurrent handlers at 64 (one error line, then close),
idles connections out after 5 minutes, bounds response writes and every
CP round trip, and aborts remaining handlers on shutdown.

F5 creating-gate leak: `purge_session_entries` now drops a key's creating
gate when the map holds the last reference (checked under the state
write lock), so single-use control-plane session keys no longer leave a
permanent String + Arc<Mutex> behind.

F6 live-socket unlink: `remove_stale_socket` connect-probes an owned
socket node and refuses to replace a listener that answers, pointing at
OPENAB_AGENT_SOCKET; only ECONNREFUSED is unlinked. The bound node's
(dev, ino) is recorded and shutdown removes the path only if it still
names that node, so an older runtime cannot delete a successor's socket.

F2 unreachable MCP surface: the only production mint of a facade session
token is an `acp:` session, so chat-adapter and adapter-less primaries
never see the four tools. v1 is narrowed honestly rather than widened:
docs/control-plane.md, the ADR, and config-reference.md now state that
reach and name the CLI as the initiating surface for those deployments;
per-adapter injection is recorded as explicitly deferred. The startup
report takes a three-way FacadeMode so the automatic facade is neither
reported as "[mcp] configured" nor as "nothing was started".

Also from #1530: `admit()` reads the ceiling under the inflight lock, and
a `cp/delegate` ack that repeats an already-tracked admission is
rejected instead of overwriting the entry.

Tests: cancel handled before first poll never starts the runner; a
dropped Admitted releases its slot; stalled write -> Disconnected at the
bound; shutdown wins over a stalled write; pending cap/TTL/orphan/overdue
pruning; terminal history untouched by prune; duplicate ack rejected;
gate map bounded across 1000 single-use keys; live same-uid socket
refused (helper and full boot path); predecessor cleanup leaves a
successor's node; connections past the cap get a diagnostic and a freed
slot is served.
@chaodu-agent

chaodu-agent commented Sep 10, 2026 •

Copy link
Copy Markdown
Collaborator Author

Note

Round 4 findings addressed in c061158 + 91c0a89 — F1, F3, F4, F5, F6 fixed in code with regression tests; F2 resolved by narrowing the v1 contract to what the code actually delivers (MCP tools reachable from acp: sessions; CLI otherwise) rather than adding a bootstrap identity model. Full matrix green on clean worktrees of the head on macOS and ext4 Linux; merge-ref CI running on 91c0a89.

Addressing Round-4 Review Feedback

🔴 F1 — Cancel-before-admission is lost

FrameAction::Serve spawns executor.serve(forward) before sending the delegate acknowledgement. Admission does not happen until that spawned future is polled.

✅ Addressed in c061158. DelegationExecutor::serve is split into a synchronous admit_delegation(&Arc<Self>, DelegateForward) -> Result<Admitted, DelegateResultParams> and Admitted::run(). The serve loop now calls admit_delegation on its own task before spawning or acking (client.rs, FrameAction::Serve arm), so any later frame — including an immediately following cp/cancel — observes the reservation. Admitted owns a SlotReservation RAII value: an ack-write failure after admission, or a task abort, still frees the slot. Refusals are answered inline after the ack so the initiator is not left on its deadline.

Regression tests (client.rs): a_cancel_handled_before_the_serving_task_is_polled_never_starts_the_runner reproduces the loop's exact ordering — classify cp/delegate (admits), classify cp/cancel as the next frame (acked ok:true because it finds the reservation), then first-poll the serving future — and asserts status == Cancelled, the prompt runner never started, and the slot is released. an_admitted_delegation_dropped_without_running_releases_its_slot covers the ack-failure path.

🔴 F2 — Automatic headless-primary tools cannot authenticate

The only production mint path is inside SessionPool::get_or_create for acp: sessions. … Report readiness only after bind and configuration/injection succeed, with text that distinguishes automatic from configured facade mode.

✅ Addressed in c061158 by narrowing the contract, not the code. Confirmed: pool.rs mints a facade token only for thread_id.strip_prefix("acp:"). Rather than invent a bootstrap credential for adapter-less primaries in this PR, v1 now states its real reach everywhere it is described:

  • PR body: new Non-goal ("per-session MCP facade credential injection for non-acp: sessions"), residual-risk line on the 8848 facade, and Follow-up 5.
  • docs/control-plane.md: a "Who can reach the MCP tools in v1" paragraph; the headless sentence now says the CLI is the initiating surface.
  • docs/adr/agent-control-plane.md: v1-reach callout under headless mode, the "Injected per-session via session/new" sentence rewritten to describe the real token path, and a new bullet under "Explicitly deferred from v1".
  • docs/config-reference.md: headless section and operational note corrected.
  • Startup report: report_facade_status now takes FacadeMode { Off, Configured, AutomaticControlPlane { listen } }. The automatic case has its own wording — names the listen address, the four tools, that they are session-gated and reachable from acp: sessions only, and that a bind failure is fatal and logged as such — so it is reported neither as "[mcp] configured" nor as "nothing was started". main.rs derives the mode from cfg.mcp.is_some() / facade_serving and shares the 127.0.0.1:8848 constant with the listener.

The ask to "define and implement the real bootstrap session/token … for an adapter-less primary" is recorded as Follow-up 5: it needs a decision on rotation/revocation for long-lived chat sessions and is out of this PR's frozen Goal. What ships is honest about it.

🟡 F3 — Pending request state and local waits are unbounded

PrimaryState.pending and awaiters have neither a count cap nor expiry. Each accepted Unix-socket connection is also served in an untracked task, and spawn/await/cancel/list handlers wait on oneshots without a timeout.

✅ Addressed in c061158. primary.rs: MAX_PENDING_REQUESTS = 256 (further begin_spawn/begin_cancel/begin_list_agents answer CommandError::Overloaded and send nothing), parked requests carry since and are failed CommandError::Timeout after PENDING_REQUEST_TTL = 60s, parked requests and awaiters whose callers dropped their receiver are removed silently, and a non-terminal delegation past deadline + LATE_TERMINAL_GRACE (60s) is dropped from tracking with its awaiters answered Timeout (a later terminal is acked and ignored, like any untracked one). PrimaryState::prune(now, wall) runs on every heartbeat tick in the serve loop. local.rs: handlers live in a JoinSet capped at MAX_LOCAL_CONNECTIONS = 64 (over-cap connections get one Error line and are closed), LOCAL_IDLE_TIMEOUT = 300s between requests, LOCAL_IO_TIMEOUT = 10s on response writes, LOCAL_REPLY_TIMEOUT = TTL + 15s on spawn/check/cancel/list, LOCAL_AWAIT_TIMEOUT = 1800 + 120s as the hard backstop on Await, and abort_all() on shutdown.

Tests: pending_requests_are_capped_and_the_overflow_is_answered_overloaded, prune_times_out_stale_pending_and_forgets_callers_that_left, prune_drops_orphaned_awaiters_and_ages_out_overdue_live_delegations, prune_never_touches_terminal_history (primary.rs); connections_past_the_cap_are_refused_with_a_diagnostic (local.rs — fills all 64 slots with idle connections, asserts the 65th gets the diagnostic line, frees one, asserts the next is served, and that shutdown removes the node).

🟡 F4 — A stalled runtime WebSocket write blocks all progress

The shared send helper checks only frame size; it has no time limit.

✅ Addressed in c061158. All writes in client.rs go through write_bounded (WRITE_TIMEOUT = 30s, mirroring the CP's write_timeout_secs default) and, inside the serve loop, write_or_shutdown, which races the bounded write against the shutdown watch. send(sink, frame, shutdown) -> Result<(), WriteEnd> with WriteEnd::{Shutdown, Disconnected} -> Outcome; heartbeat, delegate ack, reply, pong, result, and all handle_command writes use it; cp/register uses write_bounded (the caller already races registration against shutdown); the close handshake is bounded too. A timeout logs and ends the session as Disconnected, which runs the existing cancel/fail/drain/reconnect path.

Tests (client.rs, paused time, a StalledSink that is never ready): a_stalled_write_is_classified_as_disconnected_after_the_write_timeout (pending at WRITE_TIMEOUT − 1ms, Disconnected after) and shutdown_wins_over_a_stalled_write.

🟡 F5 — Admission-scoped creation gates accumulate forever

purge_session_entries explicitly preserves that gate and discard_session uses the purge path.

✅ Addressed in c061158. purge_session_entries now calls drop_idle_gate, which removes the key's gate only when Arc::strong_count == 1. The count is authoritative because every clone is taken under the same state write lock the purge holds (get_or_insert_gate), so a count of one means no creation is in progress — the serialization guarantee the old comment protected is preserved for held gates and the entry is dropped for unheld ones. Test purge_drops_a_gate_nobody_holds_so_single_use_keys_stay_bounded runs 1000 unique control-plane: keys through mint → release → purge and asserts the map is empty, then asserts a held clone keeps its gate until released. The existing purge test now holds a clone and asserts ptr_eq.

🟡 F6 — Live same-UID Unix sockets are treated as stale

That helper checks type and UID but never probes liveness … When the older runtime shuts down, its unconditional cleanup removes the successor's pathname.

✅ Addressed in c061158. remove_stale_socket now connect-probes an owned socket node: a listener that answers → refuse with "another runtime is already serving … point OPENAB_AGENT_SOCKET at a distinct path"; ECONNREFUSED → unlink (nobody listening); any other error → refuse (not evidence of staleness). serve_local records the bound node's (dev, ino) right after bind and shutdown calls remove_if_same_node, so an older runtime never unlinks a successor's node. Documented under "One runtime per socket path" in docs/control-plane.md and docs/config-reference.md.

Tests (local.rs): a_live_same_uid_socket_is_refused_not_unlinked, serve_local_refuses_to_replace_a_live_listener (full boot path), cleanup_leaves_a_live_successors_node_alone, cleanup_leaves_a_foreign_dead_node_alone.

Follow-through 91c0a89: the first CI run on c061158 failed cleanup_leaves_a_successors_node_alone on Linux — ext4 hands the next create in a directory the inode number just freed by unlink, so a successor's node can carry the predecessor's exact (dev, ino) and identity alone cannot protect it. Shutdown now closes our listener first and unlinks only if the node matches our identity and a connect probe is refused (anyone answering after we stopped listening is a successor). Reproduced on an ext4 host (p1), fixed test passes there (850/850 core lib) and on macOS.

Also closed from #1530

  • admit() reads effective_max() under the inflight lock.
  • A cp/delegate ack repeating an already-tracked (delegation_id, admission) is rejected (Internal) instead of overwriting the entry — a_repeated_admission_in_a_delegate_ack_is_rejected_not_overwritten.

Verified

Clean git worktree of c061158 on macmini (working tree 0 changes), plus 91c0a89's local.rs re-run on macmini and on an ext4 Linux host (cargo test -p openab-core --all-features --lib: 850 passed, 0 failed on Linux; control_plane:: 111 passed on both):

  • cargo check --workspace --all-features --locked ✅; cargo check -p openab-core --no-default-features --locked ✅; cargo check --no-default-features --locked ✅
  • cargo clippy -p openab-core -p openab-mcp -p openab-cp --all-features -- -D warnings ✅; cargo clippy --all-features -- -D warnings (root) ✅
  • rustfmt --check crates/openab-core/src/control_plane/*.rs ✅ clean
  • cargo test -p openab-core --all-features --lib: 848 passed (+14 new), 1 failed = known pre-existing macOS-only secrets::tests::resolve_exec_nonzero_exit (fails identically on main)
  • cargo test -p openab-core --all-features --test cp_client: 11/11 real-CP integration ✅ (round-trip, cancel, drain, oversized spawn, reconnect, deadline, heartbeat — all exercise the rewired serve loop)
  • openab-cp 142 + 9 ✅; openab-mcp 245 ✅; root --bin openab 65 ✅

Diff 644024f..c061158: 11 files, +1402 / −225 (control_plane/{client,executor,local,primary,mod}.rs, acp/pool.rs, acp_mcp.rs, src/main.rs, three docs).

Lineage

F1 REGRESSION-class correctness within the frozen scope (cancel semantics) — fixed. F2 NEW EVIDENCE against the Goal's MCP clause — resolved by contract revision (owner-authorized) and documentation, deferred implementation tracked as Follow-up 5. F3–F6 correctness/liveness within scope — fixed. #1530 keeps the remaining non-blocking items (managed-parent verification, deadline-range constant, bail string, heartbeat test margin).

CI (Linux) failed `cleanup_leaves_a_successors_node_alone`: ext4 hands
the very next create in a directory the inode number just freed by
unlink, so a successor's socket node can carry the predecessor's exact
(dev, ino). Identity alone therefore cannot protect the successor there.

Shutdown now closes our own listener first and unlinks only if the node
still matches our identity AND a connect probe is refused — anyone who
answers after we stopped listening is a successor. The identity check
still catches replaced nodes on filesystems with fresh inode numbers;
the probe catches ext4's reuse. Reproduced on an ext4 host, fixed test
passes there (850/850) and on macOS; a second test covers a foreign
dead node being left alone.
@chaodu-obk

chaodu-obk Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

> [!IMPORTANT]
> CHANGES REQUESTED ⚠️ — All six round-4 findings are genuinely fixed with regression tests, but the required CI check job is red on this exact head, and the group review surfaced ten additional findings (one CI blocker, nine mostly small and mechanical items).

What This PR Does

Lands the complete Agent Control Plane v1 stack on main: runtime registration/heartbeat/reconnect, worker delegation serving, primary-side delegation state, an owner-only Unix-socket local API, openab agent spawn|list|status|cancel, and four session-gated MCP tools with an automatic loopback facade for adapter-less headless primaries.

How It Works

  • One registered CP WebSocket carries heartbeats, worker results, and primary spawn/cancel/list commands; replies correlate on (delegation_id, admission) with first-terminal-wins semantics.
  • Round-4 remediation (c061158 + 91c0a89): admission is now synchronous on the serve loop's own task before spawn/ack (F1); pending state is capped at 256 with a 60s TTL and heartbeat-driven pruning (F3); every WebSocket write is bounded at 30s and raced against shutdown (F4); idle session-creation gates are dropped at last reference (F5); socket cleanup is identity- and liveness-checked, ext4-safe (F6).
  • Round-4 F2 (auto-facade tools have no production credential path) was resolved by narrowing the v1 contract: docs, ADR reach callout, config reference, and startup log now state that only acp: sessions mint the facade token, and the CLI is the initiating surface for all other deployments.

Findings

# Severity Lineage Finding Location
1 🔴 NEW Required CI check job fails on this exact head: cargo test exits 101, all later test/build steps skipped; failing test unidentified (logs need authenticated access) CI run 34434639420, job check
2 🟡 DELTA F2 narrowing is not synced everywhere: README (en/zh-TW) line 61 still unconditionally advertises "primaries get four direct MCP tools"; ADR section 6 heading still says "primary interface" and the alternatives table still rejects a CLI-only interface; PR body Summary bullet 6 contradicts the body's own Non-goals README.md:61, docs/adr/agent-control-plane.md:651,806, PR body
3 🟡 DELTA The 60s pending-state TTL is enforced only on the CP heartbeat tick, whose interval is CP-controlled with no ceiling; a valid long heartbeat delays capacity reclamation far past the documented bound client.rs:412-441, primary.rs:523-555
4 🟡 DELTA refuse_conn tasks are spawned into the same JoinSet counted against MAX_LOCAL_CONNECTIONS, so refusals occupy cap slots and a connect flood grows the set beyond the promised bound of 64 local.rs:295-301,338-350
5 🟡 DELTA F6 residual TOCTOU: shutdown cleanup has no barrier between the ECONNREFUSED probe and remove_file; a successor that unlinks-and-binds in that window loses its live pathname local.rs:547-561
6 🟡 DELTA The two blocking UnixStream::connect probes (startup and shutdown) are the last unbounded waits; a live-but-wedged peer with a full accept backlog stalls boot or shutdown indefinitely local.rs:491,558
7 🟡 PRE-EXISTING SpawnTarget doc comment claims constructors prevent empty selectors, but by_name("") and by_labels(BTreeMap::new()) are representable; an empty label map is structurally match-all if IPC validation is bypassed primary.rs:84
8 🟡 PRE-EXISTING Auto-facade binds fixed 127.0.0.1:8848 and exits fatally on conflict, including for deployments that can never mint its token post-narrowing; second same-host primary crashes, unlike the UDS path which has an override plus guided refusal src/main.rs:766-771,834-862
9 🟡 DELTA LocalClient::request has no client-side transport timeout; a wedged server or full backlog can hang callers past the 75s server-side reply bound local.rs:966
10 🟡 DELTA spawn_discard task panics are silently swallowed; an error log would aid diagnosis executor.rs:421-434
11 🟡 DELTA Readability nits: wait_reply returns Option&lt;Result&lt;T, ()&gt;&gt; forcing triple-nested match arms at six call sites; the Serve arm's admit/spawn/ack/refuse sequence would benefit from a one-line comment local.rs:724, client.rs:484-510
12 🟢 — All six round-4 findings verified genuinely fixed with regression tests reproducing the original scenarios; cross-confirmed by multiple independent reviewers client.rs, primary.rs, local.rs, executor.rs, pool.rs
13 🟢 — Security posture clean: credentials out of URLs/logs/child env, constant-time token compare, session gate enforced in both tools/list and call_direct_tool, owner-only UDS with bounded framing —
14 🟢 — Nit: reject_command rationale comment says "no tracking survives a disconnect" while terminal history intentionally survives fail_all_live; a Check during reconnect backoff could be answered from surviving history client.rs:1037-1059
Finding Details

🔴 F1: Required CI is red on the reviewed head

The main CI workflow (pull_request merge-ref) on 91c0a89 fails at the cargo test step with exit 101; cargo check, both clippy steps pass, and the four later test steps plus the unified build are skipped. The PR body reports 850/850 core lib green on macOS and ext4 Linux for the same head, so the failure is unexplained: possibly an environment-specific socket/timing issue or a flaky test. Job logs require authenticated access which was unavailable during this review; the failing test has not been identified.

Requested change: identify the failing test from the CI logs, fix or justify it, and get the required check green. Note the PR's own final acceptance criterion is still unchecked and mergeable_state is blocked.

🟡 F2: Narrowed contract not applied to all claims

The core reach documentation is consistent and honest (control-plane.md "Who can reach the MCP tools in v1", config-reference.md, ADR reach callout, startup log). But the outer claims were not synced: README.md:61 and README.zh-TW.md:61 still advertise four MCP tools for primaries unconditionally; ADR section 6 still titles the MCP facade "primary interface" while most primary deployments cannot reach it in v1; the ADR alternatives table still rejects a CLI-only interface even though CLI-only is exactly what v1 delivers for chat-adapter and adapter-less primaries; and the PR body Summary bullet still says "automatic MCP facade/session injection", contradicting the body's own Non-goals.

Requested change: qualify the README bullets with the v1 reach, align the ADR heading/alternatives wording with the narrowed scope, and fix PR body Summary bullet 6.

🟡 F3: Prune cadence is CP-controlled

PrimaryState::prune runs only in the heartbeat arm; the tick interval comes verbatim from the register ack (heartbeat_interval_secs.max(1), floor but no ceiling), and live tracked entries are never evicted by make_history_room (terminals only). CP config validation only requires lease_expiry_secs &gt; heartbeat_interval_secs, so a valid large interval delays the documented 60s reclamation arbitrarily.

Requested change: clamp the prune cadence (e.g. heartbeat_interval_secs.clamp(1, 60)) or run prune on an independent bounded timer; add a regression with heartbeat_interval_secs &gt; PENDING_REQUEST_TTL.

🟡 F4: Refusal tasks break the local connection bound

At the cap, handlers.spawn(refuse_conn(stream)) adds the refusal to the same JoinSet whose len() is the cap check. Refusals (up to 10s each under LOCAL_IO_TIMEOUT) occupy capacity that should belong to real handlers, and a same-uid connect flood grows the set bounded only by rate times timeout, not by the constant 64. The existing cap test does not cover flood behavior.

Requested change: do not count refusals against the live-handler cap (drop the stream, or bound refusals separately); add a flood test asserting the set stays bounded.

🟡 F5: Shutdown unlink TOCTOU (F6 residual)

remove_if_ours_and_dead checks identity, probes, then unlinks with no barrier between probe and remove_file. Sequence: predecessor probes (refused), successor unlinks the dead node and binds, predecessor removes the successor's live pathname. The ext4 identity-reuse case makes a post-probe identity recheck insufficient. Related to F6 below: both are in the same cleanup path.

Requested change: preferred fix is to not unlink on shutdown at all — startup's probe-plus-ECONNREFUSED unlink already handles stale nodes, and a stale node is safe while deleting a live path is not. Otherwise use a rename/unlinkat-based scheme; add a probe-then-bind race regression test.

🟡 F6: Blocking connect probes are unbounded

Both probes use std blocking UnixStream::connect. On Linux, connecting to a listener with a saturated accept backlog blocks until accept. A wedged sibling stalls boot inside serve_local (before the shutdown watch is armed); a wedged successor stalls the predecessor's shutdown cleanup. These are the only unbounded waits left in a delta whose theme is "every wait is bounded". Related to F9: same "bound every local wait" theme on the client side.

Requested change: bound the probe (nonblocking connect with a short poll deadline, or spawn_blocking plus timeout, treating expiry as "cannot tell; refuse to unlink").

🟡 F7: SpawnTarget constructors do not enforce the documented invariant

The type's doc comment claims construction prevents empty selectors, but only the one-of-two invariant is enforced by the enum; emptiness is checked in the local IPC adapter, which an in-process caller can bypass. An empty label map is semantically match-any unless every downstream independently rejects it.

Requested change: validating constructors (return Result) or non-empty wrapper types, plus CP-side selector rejection as defense in depth, with tests for both empty forms.

🟡 F8: Auto-facade fixed-port fatal bind for deployments that cannot use it

Post-narrowing, every [control_plane] primary without [mcp] still binds 127.0.0.1:8848 and std::process::exit(1) on conflict — including chat-adapter and CLI-driven primaries whose models can never mint the token, so the listener serves no legitimate client yet still carries fatal-bind risk. Two primaries on one host: the second dies; the UDS side of the same scenario got OPENAB_AGENT_SOCKET plus live-refusal guidance. Lineage: pre-existing geometry, not introduced by this delta; fails the late-blocker gate, so tracked as a follow-up rather than blocking here.

Requested change (follow-up): gate auto-start on deployments that can actually mint a token, or provide a facade listen override/port fallback; at minimum document the multi-primary constraint next to the socket-path rule.

🟡 F9–F11: Minor hardening and readability

  • F9: LocalClient::request relies entirely on server-side bounds; add a modest client-side transport timeout (see F6, same theme).
  • F10: log spawn_discard task panics instead of swallowing them.
  • F11: consider a small enum to flatten wait_reply's Option&lt;Result&lt;T, ()&gt;&gt;, and a one-line comment on the Serve arm's refuse dance. Both defensible as-is.

🟢 F12–F14: What was verified good

  • F12: every round-4 fix was line-verified against its regression test by multiple reviewers independently: synchronous admission with RAII SlotReservation/Admitted (including abort and ack-failure paths), pending caps/TTL/orphan/overdue pruning that never touches terminal history, write_bounded/write_or_shutdown on every write including the close handshake, drop_idle_gate with lock-protected strong-count reasoning, probe-based socket safety with real ext4 CI evidence, plus the two ride-along refactor(cp): post-v1 hardening follow-ups from #1528 self-review #1530 closures (admit lock ordering, duplicate-ack rejection).
  • F13: security posture is clean across the surface; the F2 gate is real enforcement in both discovery and invocation paths, not documentation cover.
  • F14: cosmetic comment/behavior mismatch in reject_command; a disconnect-window Check could legitimately be answered from surviving terminal history.
Baseline Check
Verification
  • Head SHA revalidated immediately before publishing this review; unchanged at 91c0a89f.
  • CI on this head: 41 of 42 check runs green or expected-skipped; the one failure is the required CI / check job (F1), independently confirmed via the checks API: cargo test step failed with exit 101, later steps skipped. Job logs were not retrievable without authenticated CLI access; the failing test is unidentified.
  • Findings F3–F7 were verified against the local checkout of the exact head (source lines quoted in the details above); F4 and F5 were re-verified line-by-line by the coordinator before inclusion.
  • No reviewer environment had a Rust toolchain, so tests could not be executed locally this round; executable evidence is CI (currently red) plus the documented clean-worktree matrix on macOS and ext4 Linux.

Addressing Prior Review Feedback

Round 4 (six findings at 644024f)

All six were re-verified against this head by multiple reviewers:

  • ✅ F1 cancel-before-admission, F3 unbounded pending state, F4 unbounded writes, F5 gate leak, F6 live-socket unlink: genuinely fixed in code with regression tests reproducing the original scenarios (round-6 finding 12). New smaller gaps found at the edges of F3/F6 are this round's findings 3–6.
  • ℹ️ F2 (no production credential path for auto-facade tools): resolved by an owner-authorized contract narrowing rather than a bootstrap identity model. The group review assessed this as a legitimate closure on three conditions, all satisfied: (1) the reach is stated consistently in the core docs/ADR callout/startup log; (2) the excluded deployments have an equally tested alternative surface (the CLI shares the same local API, validation, and bounds); (3) the deferred work is explicitly tracked (Follow-up 5). The remaining gap is unsynced outer claims — this round's finding 2.

Round 5 (fix-response comment on this head)

Verified accurate: the described fixes and regression tests exist as claimed; the ext4 follow-through in 91c0a89 matches its description. Superseded by this consolidated round.

No new external (non-team) reviewer feedback was present on this head at review time.

Review Coverage

Reviewer Scope Outcome
Reviewer A Architecture, API contract CHANGES REQUESTED: findings 2, 7
Reviewer B Performance, resource lifecycle CHANGES REQUESTED: finding 3
Reviewer C Correctness, cancellation semantics finding 10; round-4 fixes confirmed
Reviewer D Readability plus general findings 11; round-4 fixes confirmed
Reviewer E Safety, concurrency CHANGES REQUESTED: findings 4, 5
Reviewer F Docs/UX, config accuracy LGTM in lane; praise findings 12–13
Reviewer G Security, CI CHANGES REQUESTED: finding 1; security posture clean
Reviewer H Decision boundary (F2 narrowing) Endorsed narrowing; findings 1 (confirm), 2, 3 (confirm), 8, 14
Reviewer I Operability, shutdown/drain CHANGES REQUESTED: findings 6, 8, 9
Reviewer J Simplicity, scope LGTM in lane; review-lineage note folded into Baseline

Three Reasons We Might Not Need This PR

  1. v1 utility is narrower than the headline — after the F2 narrowing, the most common deployment shape (chat-driven primaries) cannot use the MCP delegation surface at all; v1 delivers CP transport, worker execution, and operator/ACP-session initiation. If per-session injection (Follow-up 5) stalls, the flagship agent-to-agent story stays unshipped. Rebuttal: the CLI surface is fully functional and shares the same enforcement path, and landing the reviewed transport unblocks incremental adoption.
  2. A 10k-line roll-up before any production canary — sweeper supervision (fix(cp): sweeper task has no panic supervision or liveness signal #1474), drain (feat(cp): container-grade shutdown — SIGTERM handling and connection drain #1475), and packaging are all deferred, so maintainers carry transitional failure modes. Rebuttal: the code is opt-in behind [control_plane], and carrying a 10k-line branch out-of-tree rots faster than landing it under CI.
  3. The auto-facade serves deployments that cannot use it — a fixed-port listener with fatal bind for clients that can never authenticate is surface without demand (finding 8). Rebuttal: loopback-only and session-gated, so the risk is operational rather than security; a follow-up can gate auto-start.

@github-actions github-actions Bot added the closing-soon PR missing Discord Discussion URL — will auto-close in 24 hours. label Sep 12, 2026
@github-actions

Copy link
Copy Markdown

Caution

This PR has been waiting on the author for more than 2 days (labeled pending-contributor since 2026-09-10).
It will be automatically closed in 24 hours if there is no update.

@chaodu-agent — You must add a new comment on this PR to remove the closing-soon label and keep it open. Pushing commits alone is not sufficient. Feel free to reopen a new PR later if it gets closed and you want to pick it back up.

@github-actions

Copy link
Copy Markdown

🔒 Auto-closing: this PR has had the closing-soon label for more than 1 days without activity from the author.

If you'd like to continue working on this, please submit a new PR and link to this one if necessary.

@github-actions github-actions Bot closed this Sep 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

closing-soon PR missing Discord Discussion URL — will auto-close in 24 hours. pending-contributor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant