Repository navigation
fix(report): keep missing references as non-blocking caveats - #805
Open
efegokdemir wants to merge 3 commits into
Open
efegokdemir wants to merge 3 commits into
efegokdemir wants to merge 3 commits into
Conversation
Signed-off-by: Efe <efe@rexcode.co.uk>
rng1995
requested changes
Oct 14, 2026
rng1995
left a comment
Collaborator
There was a problem hiding this comment.
[SkillSpector Review]
@efegokdemir thank you for taking on #804, centralizing the missing-reference predicate, and keeping ambiguous references blocking.
Value and readiness
The noise #804 describes is real, and aligning CLI/SARIF with the MCP gate is a reasonable goal. However, reference_missing does not only mean "absent". Referenced content that exists but was never inspected gets the same reason, and this PR turns those scans into complete/SAFE results that pass --fail-on-incomplete. Not ready yet.
Material findings
- Blocker,
src/skillspector/inspection_ledger.py:1056: the caveat-only rule also covers present but uninspected targets.- Resolution checks only the discovery inventory (
references.py:508-529). Any existing target outside that inventory therefore becomesmissing. - Repro A, in-bundle symlink: SKILL.md says "Follow the steps in
docs/setup.md", anddocs/setup.mdis a symlink to a file outside the scan root. On mainb07f181dwith--no-llm --fail-on-incomplete, the only ledger exception isreference_missing, and the symlink appears only inscope_exclusionsasnot_regular_file. There are 0 partial and 0 uninspected files and no findings; the result is CAUTION with exit 1. - Repro B, single-file target:
skillspector scan skill/SKILL.md, where SKILL.md says "Runscripts/setup.shfirst" andskill/scripts/setup.shexists and pipescurltosh. Main gives the same result: onlyreference_missing, CAUTION, exit 1. - With this PR, both match
reference_caveat_onlyand becomecomplete_with_caveats,is_complete=true, SAFE, exit 0. The uninspected instruction or script is indistinguishable from a workspace path. The MCP gate already allows these cases; its comment atmcp_server.py:196assumes the path is not carried by the bundle. - Expected: establish that the target is absent before treating it as a caveat. One option is an
lstat(no read) of the normalized target relative to the skill directory or the selected file's parent, with a distinct blocking reason for present-but-uninspected targets. At minimum, exclude targets that match a scope exclusion. Please add regressions where A and B stay partial and a truly absent workspace path becomescomplete_with_caveats.
- Resolution checks only the discovery inventory (
- Blocker: the new status is not handled where existing code compares against
"complete".src/skillspector/nodes/report.py:1736: transitive truncation setsis_complete=Falsebut rewrites the status only when it equals"complete". JSON can then showstatus: complete_with_caveatstogether withis_complete: false, the kind of contradiction downstream validators reject (#800).src/skillspector/nodes/finalize_inspection_ledger.py:821: the coverage-finding downgrade likewise checks only"complete".- Expected: handle both complete statuses at these sites, or derive the status from
is_complete, and test the truncation path.
PIC tradeoffs
- Policy: this changes the default CLI/SARIF verdict and
--fail-on-incomplete, not only MCP. An opt-in flag is the alternative. - Schema:
complete_with_caveatsis a new public status. Consumers that allow onlycomplete|partial|failedmay reject it. The alternative isstatus: completeplus a caveats flag. Either way it needs a release note. - The recommendation for these scans changes from CAUTION to SAFE.
Verification and gaps
- I read the full diff against
3c8e4b95and tracedfinalize_ledger→report()→_build_sarif→ the CLI gate (cli.py:866, which readsis_complete) and the MCP gate. - I ran trusted
origin/main(b07f181d) in a scratch worktree for both repros (results above). I did not run PR code; the PR's behavior comes from the predicate, since there are no other exceptions, no limitations and zero partial/uninspected counts. - User exclusions, excluded directories and executable symlinks are in
known_paths, resolve with partial dispositions, and stay blocking. The hole covers non-executable symlinks and special files, plus files outside a single-file target. - Main is 49 commits ahead, including #796 and #806, which touch
report.pyandmcp_server.py;git merge-treeis clean and those sites are unchanged. - CI: all 6 checks are green on
f6bdc08.
Signed-off-by: Efe <efe@rexcode.co.uk>
Signed-off-by: Efe Gökdemir <gokdemirefe1903@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Treat scans with only missing-reference ledger caveats as complete when every discovered file was inspected and no other limitations exist. Preserve the reference and exception details in reports, and keep
--fail-on-incompletefrom rejecting these scans.Why
The MCP safety gate already allowed a missing-reference-only result, but CLI and SARIF completeness still reported it as partial and raised the recommendation to
CAUTION. This made the same fully inspected workspace appear incomplete depending on the output path. Ambiguous references and actual coverage gaps still need to block completeness.How
complete_with_caveats, withis_complete=true, only when there are no limitations or uninspected files.Testing
main: a no-findings scan with only missing workspace references returnedpartial,CAUTION, and exit code 1 with--fail-on-incomplete.pytest -q tests/test_reference_destinations.py tests/test_reference_destination_kinds.py tests/nodes/test_finalize_inspection_ledger.py tests/nodes/test_analysis_completeness.py tests/nodes/test_security_remediation.py— 755 passed.make test-unit— 10,540 passed, 17 skipped, 133 deselected, 4 xfailed.make lint— passed.make format-check— passed.uv build --out-dir /tmp/skillspector-805-audit-dist— passed.Compatibility
Single-key and normal complete-scan behavior is unchanged. Missing references remain visible; only their effect on completeness/recommendation changes when there is no actual inspection gap. Ambiguous references and incomplete coverage remain blocking.
Checklist
Fixes #804