Skip to content

feat(cli): --strict fails a run whose analysis degraded - #242

Merged
rahlk merged 1 commit into
mainfrom
fix/no-build-precondition-error
Sep 8, 2026
Merged

feat(cli): --strict fails a run whose analysis degraded#242
rahlk merged 1 commit into
mainfrom
fix/no-build-precondition-error

Conversation

@rahlk

@rahlk rahlk commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

A run that loses the RTA overlay or the L4 semantic ddg exits 0 and reports the loss only as a WARN on stderr. Interactively that is fine. In a pipeline it is not — -a 2 --no-build on an unbuilt project returns a call graph of declared edges only, and nothing in the exit code or the payload distinguishes it from a complete run.

--strict turns any such degradation into a non-zero exit that names what was lost, and fails before writing analysis.json so a caller cannot pick up a thin payload believing it is whole.

$ codeanalyzer -i ./app -a 2 --no-build --strict
error: analysis degraded and --strict was requested:
  - RTA overlay: no entrypoints; call graph is declared edges only
Build the project (or drop --no-build) to get these overlays, or rerun without --strict to
accept the degraded output.

Why opt-in, and not a fix to --no-build

I started by making --no-build reject the combinations that need WALA. That was wrong, and the tests said so:

  • CodeAnalyzerV2CliTest:344 compiles a fixture, then runs --l3-engine wala --no-build and expects full overlays. That is the workflow --no-build exists for — I already built it — and the analyzer cannot know at parse time whether you did.
  • CodeAnalyzerV2CliTest:519 pins -a 4 --no-build at exit 0, because the L4 vertices, param_in/param_out and summaries are engine-free and still run.
  • CodeAnalyzerIntegrationTest:172 runs --analysis-level=2 --no-build against mvnw-corrupt-test, a fixture that exists to prove the analyzer survives a project whose build is broken.

Nine test call sites pass --no-build. Degrading is a supported mode, not an oversight: you still get the tree, the declared call graph, the syntactic CFG/CDG/DDG and, at -a 4, the SDG vertices, param edges and summaries. Everything except the WALA-derived overlays. Defaulting to failure would delete a working mode to fix a reporting problem.

So the flag is opt-in, default behaviour is byte-identical to today, and all nine sites stay green untouched.

Implementation

No changes to ScopeUtils, RtaCallGraph or WalaAnalysis. All three degradations are already visible in CodeAnalyzer where they are warned about, so the flag only had to collect them and check once — after every overlay pass has had its chance, before anything is written.

strict is an instance field, not another static, for the reason --schema already records in this class: the pre-existing static options leak between CommandLine instances in one JVM.

Verification

Full suite 550 tests, 1 failure — CodeAnalyzerIntegrationTest, no Docker daemon here, fails identically on a clean main
Default path -a 2 --no-build on an unbuilt project → exit 0, payload written, unchanged
Strict path same run --strict → exit 1, no analysis.json, message names the lost overlay
-a 1 --no-build --strict exit 0 — nothing WALA-dependent was requested, so there is nothing to complain about

Four new tests cover exactly those four cases.

Known blemish

The error surfaces wrapped in a RuntimeException stack trace, because CodeAnalyzer.run() wraps every exception from analyze(). That is how all existing parameter errors already surface (--l3-engine, --emit neo4j + -a). Unwrapping would change their exit code from 1 to picocli's 2, which is a behaviour change beyond this fix, so I left it.

Propagation verdict

Corrected. An earlier draft of this section said python-sdk vendors a 2.4.1 jar and therefore consumes no 3.x release. That was read off origin/main, which is still the 1.x line (1.5.0) — the wrong branch.

The live 2.0 line, origin/release/2.0 at v2.0.0-rc.3, already consumes this analyzer properly:

  • java = ["codeanalyzer-java==3.1.0"] — a real PyPI extra, installed via pip install "cldk[java]"
  • [tool.backend-versions] codeanalyzer-java = "3.1.0"
  • no vendored jarcodeanalyzer.py:213-227 calls codeanalyzer_java.command(), so the wheel is the single source of both the jar and the JVM

So the 3.1.0 released tonight is what rc.3 consumes. Two real consequences follow:

  1. The pin is exact (==3.1.0), so a 3.1.1 will not reach SDK users without a bump. Shipping --strict means python-sdk's release/2.0 needs codeanalyzer-java==3.1.1 in both the extra and [tool.backend-versions]. That is the follow-on item, and it is a lockstep bump, not an optional one.
  2. The JRE gap lands squarely on SDK users. Because the SDK now runs the analyzer through the wheel's bundled jdk4py JVM (codeanalyzer.py:215), every python-sdk Java user at -a 2 or above hits the javac-absent path documented in docs: the PyPI install needs a real JDK from -a 2 onwards #241 — and silently, which is exactly what --strict exists to surface. That raises this flag's value for the SDK above what it was worth for the CLI alone.

Sibling analyzers: no equivalent bug class — neither codeanalyzer-python nor codeanalyzer-typescript has a build-dependent overlay to lose. Docs: README FAQ updated here.

A run that loses the RTA overlay or the L4 semantic ddg exits 0 and says so
only as a WARN on stderr. Interactively that is fine. In a pipeline it is not:
`-a 2 --no-build` on an unbuilt project returns a call graph of declared edges
only, and nothing in the exit code or the payload distinguishes it from a
complete run.

`--strict` turns any such degradation into a non-zero exit that names what was
lost, and it fails before writing `analysis.json` so a caller cannot pick up a
thin payload believing it is whole.

Opt-in, and deliberately so. Degrading is a supported mode, not an oversight:
`--no-build` on an unbuilt project still yields the tree, the declared call
graph, the syntactic CFG/CDG/DDG and, at -a 4, the SDG vertices, param edges and
summaries — everything except the WALA-derived overlays. Nine test call sites
depend on that, including the container integration test, whose fixture
(`mvnw-corrupt-test`) exists precisely to prove the analyzer survives a project
whose build is broken. Defaulting to failure would delete a working mode to fix
a reporting problem.

Detection needs no changes to ScopeUtils, RtaCallGraph or WalaAnalysis: all
three degradations are already visible where they are warned about, so the flag
only had to collect them and check once, after every overlay pass has had its
chance.

The flag is an instance field, not another static, for the reason the `--schema`
option already records: the pre-existing static options leak between CommandLine
instances in one JVM.

README gains the flag under a new FAQ entry, and says plainly that degrading is
deliberate and exactly which overlays it costs.
@rahlk
rahlk merged commit d0123f3 into main Sep 8, 2026
@rahlk rahlk mentioned this pull request Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant