Skip to content

fix(llm): stop CLI backends re-reading the repo and blowing the argv limit - #19

Merged
jamesaphoenix merged 1 commit into
jamesaphoenix:mainfrom
jakob1379:fix/cli-prompt-stdin-and-token-reduction
Sep 8, 2026
Merged

fix(llm): stop CLI backends re-reading the repo and blowing the argv limit#19
jamesaphoenix merged 1 commit into
jamesaphoenix:mainfrom
jakob1379:fix/cli-prompt-stdin-and-token-reduction

Conversation

@jakob1379

Copy link
Copy Markdown
Contributor

Two independent problems in the claude and codex CLI backends, both hit in normal use.

1. analyze this flow crashed with Argument list too long

The prompt was passed as a single argv entry. Linux caps one argument at MAX_ARG_STRLEN (128 KiB — this is separate from ARG_MAX, which is 2 MB and looked fine), and a group's diff plus per-file excerpts routinely exceeds it. The spawn failed with E2BIG before the model was reached:

ERROR activity: LLM error: CLI invocation failed: Failed to launch claude: Argument list too long (os error 7)

Prompts now go on stdin, which has no such limit — claude --print reads stdin as the prompt, and codex exec documents - for the same. Note pass2_user_prompt never truncates file.diff at all (only new_content gets a 2000-token budget), so a single large changed file can reach the cap on its own.

2. Every pass paid to re-read context it had already been given

AGENT_ADDENDUM told each spawn to "use your built-in read/search tools ... to inspect files, plans, and git state" — while the prompt already carries the diffs and file excerpts. Each pass re-derived what it had just been handed. Separately, the user's MCP servers, plugins and slash commands were loaded into every spawn's prefix, none of which are reachable from a --print run.

Measured against the real binary on an identical prompt:

tool calls turns prefix tokens
before 2 3 57,475
after 1 2 26,191

−54% prefix tokens per pass. The addendum now states the prompt is self-contained and forbids inspection; --strict-mcp-config --mcp-config '{"mcpServers":{}}' --disable-slash-commands drop the unreachable prefix.

Also included

  • Bounds the previously unbounded metadata batch fan-out (Semaphore(3)). Note this needs >60 groups to trigger at the default batch_size of 20, so it was likely never a rate-limit contributor — included as correctness, not as the fix.
  • kill_on_drop(true) so a cancelled command stops burning quota instead of orphaning the child.
  • Debug logging of each spawn (arg count, cwd, prompt size) on the claude_cli / codex_cli targets — the prompt byte count is what makes this class of failure obvious immediately.
  • rustfmt and clippy added to the nix devshell; they weren't available, which is why cargo clippy -p diffcore-core currently fails on 8 pre-existing errors nobody could see. Those are untouched here.

Verification

  • 1763 lib tests + all integration suites pass.
  • Clippy clean on all changed files.
  • rustfmt reports exactly the pre-existing baseline — no new drift.
  • New test a_prompt_over_the_kernel_arg_limit_reaches_the_child spawns a real child with a 200 KiB prompt and asserts arrival; it fails if the prompt returns to argv.
  • The stdin and flag behaviour was confirmed against the real claude binary, not assumed.

Known limitations, not addressed here

  • run_metadata_pass has no aggregate deadline; with INTERACTIVE_CLI_TIMEOUT_SECS = 3600 a stalled pass can run long. Pre-existing.
  • detect_status() still forks claude auth status once per call. Pre-existing.
  • Forbidding repository inspection is a deliberate behavioural trade: the prompt is assumed self-contained. If any pass genuinely needed to read beyond what it is sent, that would show up as a quality regression in the eval suite rather than an error.

…limit

Two independent problems in the `claude` and `codex` CLI backends.

The prompt was passed as a single argv entry. Linux caps one argument at
MAX_ARG_STRLEN (128 KiB), and a group's diff plus file excerpts routinely
exceeds that, so "analyze this flow" failed the spawn outright with
`Argument list too long` (E2BIG) before the model was ever reached.
Prompts now go on stdin, which has no such limit.

Separately, the agent addendum told every spawn to "use your built-in
read/search tools to inspect files, plans, and git state" even though the
prompt already carries the diffs and file excerpts. Each pass therefore
paid to re-derive context it had just been handed. Measured on an
identical prompt, dropping that instruction and the unreachable MCP,
plugin and slash-command prefix took a pass from 57,475 to 26,191 prefix
tokens (-54%), with one fewer tool call and turn.

Also bounds the previously unbounded metadata batch fan-out, kills child
processes when their future is dropped, logs each spawn at debug level,
and adds rustfmt and clippy to the devshell.
@jamesaphoenix
jamesaphoenix merged commit fdc081f into jamesaphoenix:main Sep 8, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants