fix(analyzer): keep checking command lines inside a word that spans lines

[elliottwaves-20]

Fixes #813.

Problem

A double quote in a heredoc body, a shell comment or Markdown prose can pair with a quote on a later command line. _has_shell_command_word_exhaustion then parses one word across those lines and moves parsed_through past it. Every candidate inside the word is skipped, including a runtime-selected command such as $CMD"" -rf /, and the file is reported as completely inspected. The reproducers are in #813.

Change

The fix does not try to prove shell quote state. It only stops a multi-line word from hiding later command lines.

• A parsed word that spans a line break (LF or lone CR) opens a shadow region (shadow_through) instead of moving parsed_through.
• Candidates inside the region are still parsed and checked:
  • A candidate that opens a runtime word ($…, "$…, or a single backtick, bare or after "; helper _opens_runtime_word) is checked wherever main checks it, because such a command can follow any prefix word: sudo -u root $CMD, then $CMD, { $CMD, env A=1 $CMD.
  • Any other candidate is checked only at a command position: line start, or after ;, |, & or ( (helper _is_shell_command_position).
  • A run of backticks (a Markdown fence) never opens a substitution, so it is not treated as a runtime word. test_json_owned_runtime_bounds.py still passes.
• Shadowed candidates never own quotes and never move parsed_through or shadow_through. An unresolved parse inside the region is not charged to the command-word budget, because the spanning word already closed that region's quotes.
• Python sources (python_source is not None) keep the current frontier, because the AST already proves their string and comment bounds.

The candidates after the region are therefore the same as on main, and the region's candidates can only add checks. No file can move from partial to complete.

Tests

tests/nodes/analyzers/test_multiline_word_shadow.py:

• test_quote_spanning_lines_cannot_hide_a_later_command: comment, heredoc with plain and quoted delimiter, ${X}, ${X}" ", $()"e", a backtick variant, lone-CR line endings, and Markdown prose into a fence, and runtime commands after the prefix words sudo, sudo -u root, then, {, !, env A=1 and xargs -0 (with $CMD, "$CMD" and backtick forms). Each runs short and padded. All fail on main.
• test_heredoc_quote_cannot_hide_a_runtime_rm_at_scan_level: the run.sh from Quote in a heredoc body, comment or prose hides a later command line from the command-word scan #813 through run_static_patterns_with_ledger, expected to be not COMPLETED.
• Pins for unchanged behavior: a multi-line echo "…$HOME" stays complete, a dynamic multi-line $(…) word behaves as on main, and Python sources keep their frontier.

Verification against main 5edc347

• Full suite: the set of failing tests is identical on main and on this branch. Those failures are Windows/environment-specific on my machine (release, git and CLI tests).
  The corpus and fuzzing figures below were measured on the first head (011c792). The review fix only lets more shadowed candidates be checked, so it can only add partial results.

• Corpus of about 3,600 skill files: no file moves from partial to complete, and no file loses a TM1 finding. 4 files move from complete to partial (3 distinct). All are Markdown where a prose quote pairs across lines into a later block, so main skipped those lines.

• Differential fuzzing, 100k generated shell snippets with heredoc, comment and adjacent-quote shapes: 0 cases where main is partial and this branch is complete.

🤖 Generated with Claude Code

[rng1995]

[SkillSpector Review]

@elliottwaves-20 thank you for the clear reproducers in #813, and for designing the shadow region so that it can only add checks. The "no file moves from partial to complete" property holds up when I read the code.

Value and readiness

This is a real scan-integrity bug. On main, a quote in a heredoc body, a comment or prose can pair across lines and hide $CMD"" -rf /, and the file is reported COMPLETED. I confirmed the #813 reproducers on main. The design is sound and monotonic:

• The spanning word now sets shadow_through instead of parsed_through.
• Shadowed candidates never change ownership or the frontier.
• Candidates after the region are exactly the ones main checks.

However, the new command-position predicate still lets the same hiding happen whenever the runtime command follows a common prefix word, so the PR only partly achieves its stated outcome.

Material findings

1. Blocker: src/skillspector/nodes/analyzers/static_patterns_tool_misuse.py:522 (_is_shell_command_position). Inside a shadow region, only line starts and positions after ;, |, & or ( are checked.

   The candidates that matter here are $-led runtime commands, and those often follow a prefix word: sudo $CMD …, then $CMD …, do/else $CMD …, { $CMD …; }, ! $CMD, exec $CMD, time $CMD. Outside a region, main checks them, because _is_shell_command_word_start accepts any candidate preceded by whitespace.

   On main, each of sudo $CMD"" -rf / # ", if true; then $CMD"" -rf / # " and { $CMD"" -rf / # " is partial on its own. Each becomes complete when preceded by cat <<E / he said "hi / E. In this PR, the $ after sudo , then or { is not a command position, so the shadowed candidate is skipped and the result stays COMPLETE. Prefixing the #813 reproducer with sudo brings the hiding back.

   Expected: also count a position as a command position when the preceding word is one of:

   • the clause prefixes this module already defines (_SHELL_CLAUSE_PREFIX_WORDS), plus if/while/until
   • { or a backtick
   • the wrappers it models (sudo, env, command, nohup, nice, timeout, xargs, exec)

   Alternatively, check $-led shadowed candidates with _is_shell_command_word_start. Please add the sudo, then and { variants to test_quote_spanning_lines_cannot_hide_a_later_command.

PIC tradeoffs

• The corpus run moves 4 Markdown files from complete to partial, where a prose quote pairs into a later block. That is the intended conservative cost. Widening the predicate per finding 1 may add a few more.

Verification and gaps

• Read the diff against _has_shell_command_word_exhaustion, _is_shell_command_word_start and _parse_shell_command_word on main. I checked that the candidates and owned positions after a region match main. Shadowed candidates can only return True (through the limited, printf or runtime-operand paths), so the change cannot make a file complete.
• The baseline probes above ran on trusted main code only. I did not run the PR's code or tests.
• static_patterns_tool_misuse.py on main has not changed since this PR's base (5edc347), so the green CI (6/6) on 011c792 is current. The PR merges cleanly with main and with #779, #783, #790 and #801.
• I checked the parse-work bound only by reading. The widened predicate should still keep test_json_owned_runtime_bounds.py passing.

[rng1995]

Inside a shadow region, a runtime command after a prefix word is still hidden: sudo $CMD"" -rf / # ", if true; then $CMD"" -rf / # " and { $CMD"" -rf / # " are each partial on main when standalone. Each becomes complete after cat <<E / he said "hi / E, and here the $ after sudo /then /{ is not a command position, so it stays complete. Please also accept positions after _SHELL_CLAUSE_PREFIX_WORDS, if/while/until, {, a backtick, and the wrappers this module models (sudo, env, command, nohup, nice, timeout, xargs, exec). Alternatively, check $-led shadowed candidates with _is_shell_command_word_start. Please add these variants to the test.

[elliottwaves-20]

Fixed in 8dd0d6a. Shadowed candidates that open a runtime word ($, "$, or a single backtick) keep main's word-start check, which also covers sudo -u root $CMD, env A=1 $CMD and xargs -0 $CMD. The variants are in test_quote_spanning_lines_cannot_hide_a_later_command. Details in the PR comment.

[elliottwaves-20]

@rng1995 I reproduced the prefix-word cases on this branch and fixed them in 8dd0d6a.

I took your second option. A candidate inside a shadow region that opens a runtime word now skips the command-position filter and keeps main's _is_shell_command_word_start check (_opens_runtime_word). A runtime word is $…, "$…, or a single backtick, bare or after ". I chose this over the list of prefix words because the word right before $CMD is not always a prefix word: sudo -u root $CMD, env A=1 $CMD and xargs -0 $CMD would have stayed hidden with the list.

Backticks are included because sudo `echo x`"" -rf / # " had the same gap. A run of backticks (a Markdown fence) is excluded, since it never opens a substitution. Without that exclusion, test_dense_owned_json_strings_have_bounded_total_parse_spans exceeds its call budget by the fence markers.

test_quote_spanning_lines_cannot_hide_a_later_command has ten new cases. Each is partial on main when the line stands alone, and complete on the previous head after the heredoc prefix:
sudo, sudo -u root, then, {, !, env A=1, xargs -0, sudo "$CMD", sudo `…` and then "`…`".

Verification (Windows):

• test_multiline_word_shadow.py and test_json_owned_runtime_bounds.py: 42 passed.
• Full suite: the set of failing tests is identical to the previous head (Windows/environment-specific failures, as described in the PR body).
• ruff check and ruff format --check on both files: clean.

🤖 Generated with Claude Code