feat(java): leg 3a — Java on codeanalyzer-java's schema v2, from the analyzer wheel - #347
Closed
rahlk wants to merge 34 commits into
Closed
feat(java): leg 3a — Java on codeanalyzer-java's schema v2, from the analyzer wheel#347rahlk wants to merge 34 commits into
rahlk wants to merge 34 commits into
Conversation
…elpers out of the Python backend A pure move. The validation, keyset paging, edge ordering, bounded-subgraph, path-assembly and module-key helpers leg 1.5/1.6 built contained nothing Python-specific except three things, now parameters: the PY_ relationship prefix, the .py extension, and the VIA map. Python re-imports every name, so its suite is the proof that nothing changed. TypeScript (leg 2.5) and Java (leg 2) inherit the same rulings instead of forking them. The one test that read DEFAULT_DEPTH's rule out of the Python backend's source text now reads it from commons.bounds, where the constant and its rule live.
…om the wire-schema inventory
Rewrite cldk/models/typescript/ as a pydantic v2 mirror of cants'
src/schema/schema.ts at 1.2.0, validated level by level against real
`-a 1..4` output of the released wheel (the four fixtures under
tests/resources/typescript/analysis_json/v2/, which replace the 1.x
slim/analysis.json).
- TSAnalysis envelope, TSApplication with the app-scope overlays
(call_graph, param_in/out, artifacts, dependencies, unresolved_imports,
config_uses/config_reads, id-keyed external_symbols and
synthesized_callables), TSModule with `source` and a unified `types{}`.
- `types{}` is a discriminated union on `kind` over the five existing
classes; the 1.x `classes/interfaces/enums/type_aliases/namespaces`
maps (module and namespace) are read-only properties over it.
- TSSpan modelled faithfully; start_line/end_line/start_column/end_column
are properties over `span`; `code` slices the owning module's `source`
(threaded in by a TSModule after-validator, private, not dumped) as
characters — UTF-16 caveat documented (cants #174).
- TSCallGraphEdge{src,dst,prov,weight} replaces the 1.x edge shape;
TSExternalNode / TSSynthesizedNode replace the slim externals; TSField
is the one shape for attributes, properties, variables and enum members.
1.x names stay importable as aliases.
- extra="forbid" kept; 1.3.0's known additive fields declared Optional now.
- Not modelled because the analyzer never emits them: abs_path, call_sites,
config_accesses, last_modified, file_size, path, code (as a field),
accessed_symbols, local_variables, code_start_line.
- TSCallableOverview.from_callable takes the module key as `path=`.
The TypeScript backends and facade still load the 1.x shape and go red
here; they are re-pointed in the next change.
…he fixture app id corrected; T3 seek decision measured; TS live writers gated first
…zer-typescript 1.2.0 TSAnalysisBackend now parameterises AnalysisBackend[TSApplication, TSModule, TSType, TSCallable, TSField, str] (P = N = "TS"); the TypeScript-only remainder stays declared on it. TSCodeanalyzer implements the generic twelve: the artifact/config five return the shared PyArtifact / PyDependency / PyConfigKey / PyConfigUseEdge / PyConfigRead models (ecosystem "npm"; a non-string config value rendered as its JSON text), and get_application_view replaces get_application. The backend reads the 1.2.0 envelope (TSAnalysis; kept as .analysis for max_level and analyzer.version) and drives the 1.2.0 CLI: -i <project> --app-name <project.name> -a <1..4> -o <cache> --cache-dir <cache> --skip-tests [--eager] [-t <file>]... Every AnalysisLevel is sent as its integer through commons/levels.py, lifted out of the Python backend (which re-binds the same objects); the old cap at level 2 and --tsc-only are gone, and TSCodeAnalyzerConfig.tsc_only is a deprecated no-op that warns naming the analyzer release that removed it. The call graph (TS-10/TS-11) keys nodes as every other accessor does — module file key, signature, "<module>.<name>" for an external — with id and kind node attributes and CALL_DEP/weight/provenance edge attributes as Python's; can:// endpoints resolve through an id index built once over modules, types, callables, externals and the synthesized index (keyed by the older anonymous id). Module callers and class callees are kept. An unhomed endpoint raises rather than being skipped. Call sites, calling lines, call targets and callsites_for read body nodes of kind "call". The two accessors that only ever raised NotImplementedError are removed from the facade; a frozen public-surface test pins the remaining 46. A new e2e runs the pinned binary on the sample application at all four levels; the app's package.json / tsconfig.json (the fixture's own artifacts, byte-identical by sha256) are tracked so the fixtures are regenerable from the tree. The Neo4j backend is untouched and now abstract until it implements the six generic methods it lacks (T3).
- A call node whose callee id is unindexed now raises the unhomed-endpoint error through _node_of instead of leaking the raw can:// id into TSCallsite.callee_signature / get_call_targets / get_calling_lines (E6/E7); a null callee stays None. - A synthesized entry is keyed by the tree callable its id resolves to, or by its own name when it is a named residual; a residual without a name, or a non-residual whose id is unindexed, raises at load rather than minting a node keyed by a raw id. - The call-graph `kind` vocabulary is the full set the index holds (module, the five type kinds, callable, external), stated once as CALL_GRAPH_NODE_KINDS and pinned by the e2e against the documented literal. - The "dangling-free" assertions were tautologies (nx creates endpoints); both now assert every node is a key an accessor returns and never a can:// id. - Unit tests for each unhomed case on a hand-built payload; the tsc_only DeprecationWarning points at the CLDK.typescript(...) call; cldk.core's docstring no longer says tsc_only passes --tsc-only.
…er gating as Task 3 Step 0
…vocabulary, scoped by can:// prefix
TSNeo4jBackend is re-pointed statement by statement from the 0.4.3 graph
(:Symbol/:CallSite/CALLS/HAS_CALLSITE, scoped on _module) to the 1.2.0
projection: :Application {id: can://typescript/<app>}, TSModule (name holds
the file key), the five TS type labels, TSCallable (anonymous ones also
TSAnonymousCallable), TSField, TSBodyNode {kind:'call'} under TS_HAS_BODY_NODE
resolving over TS_RESOLVES_TO, TS_CALLS {weight, prov}, TSDecorator over
TS_DECORATED_BY, TSExternal, and the unprefixed artifact layer.
Scope (TS-3) is the two id prefixes can://typescript/<app>/ and
can://javascript/<app>/, spelled once by _scoped(var) as
`x.id STARTS WITH $p1 OR x.id STARTS WITH $p2` and bound from _scope_params;
there is no _module to fall back on. Seek labels follow the measurements on
the superset graph: signature/prefix statements anchor on the bare label
(:TSCallable 7.9 ms vs 43.9 ms with :CanNode, which turns the predicate into
a range-seek union); id-equality point lookups anchor on :CanNode:<Label>
(1.5 ms unique-index seek vs a 9.1 ms label scan).
The containment tree is fetched as one variable-length subtree per accessor
(two statements for the whole symbol table: 1.9 s for 1,841 modules) and
assembled by parent id, children keyed by the id segment under the parent
(#get/#set for accessors). The probe (G5) requires TS_HAS_MODULE /
TS_HAS_METHOD / TS_HAS_BODY_NODE / TS_CALLS and refuses an application whose
analyzer_version is absent, unparsable or below 1.2.0, naming what it found.
reconstruct.py rebuilds the T1 models from the rows. What the projection does
not carry stays at the model's empty default and is stated: module source
(each node's own code is threaded in as its text; spans are line-only),
parameters, comments, type parameters, overloads, bodies and the L3/L4 graphs,
enum member values, imports and exports; call sites keep lines and the
resolved callee. get_unresolved_config_reads raises naming the gap: the
projection has no config_reads, and an empty list would read as "every read
resolved". The anonymous index is keyed by the tree node's own id. A value
and a type of the same name share one id in the emitter (declaration
merging), so MERGE collapses them; three nodes on the superset graph.
The generic six the ABC left abstract are implemented; the artifact four
reuse the Python reconstruct helpers, the shared layer being identical.
Tests: the schema-probe matrix (0.4.3, Python, empty, 1.1.0, absent, garbage,
1.2.0+); the multi-application audit in the leg-1.6 shape, harvesting every
class-level and inline statement plus every _fetch anchor, over a fake
two-application 1.2.0 graph with no _module evaluated by a small in-test
Cypher interpreter; the stub tests re-pointed; a read-only live suite for the
superset graph (21 tests, exact against raw Cypher; gated on
CLDK_TEST_NEO4J_URI with no default). The two live writer files are gated on
CLDK_TEST_NEO4J_WRITE_URI / _WRITE_USER / _WRITE_PASSWORD with no defaults, a
skip reason saying they write, and a teardown scoped to the application's two
prefixes.
- A declaration-merged node (one id, two labels, the last writer's kind) is no longer a TSNamespace by fall-through: _type dispatches on kind with an explicit namespace branch and raises otherwise; _declared rebuilds a TS_DECLARES child whose kind is neither a type nor a callable kind as the one type facet its labels name and raises when they name none or several; the type accessors and the method-owner statements carry a kind predicate alongside the label. Rows carry labels(n) for that dispatch and the message, which names signature/name, labels and kind, never the id. - get_method_parameters raises for a found method (the projection has no parameters), [] for a missing one; get_imports/get_all_exports raise naming the gap instead of returning empty maps. - _graph_key raises for an endpoint without a signature, or a package-level external without a member name, rather than keying by the raw id. - get_external_symbols returns the external symbols (module/name) only: the 2,653 nameless package-level externals were keyed "<module>." before. - The probe distinguishes "no :Application node" from "node present, no analyzer_version"; reconstruct reads signature/kind/name without defaults. - Audit: the CanNode rule covers every prefix-scoped statement without an id equality; the fake driver asserts every bound prefix is application A's; application B's external has its own name; only _run may call .run(; the fake graph carries the merged nodes; the live suite asserts element types and what each merged node is through the facade.
A pure move, deferred from the Task 3 review. The version-floor parser both Neo4j probes compare against their _ANALYZER_FLOOR is commons.backend.semver; the three reconstructors of the shared artifact layer (artifact / config_key / dependency -- the one part of the graph every analyzer projects identically, into the same Py* models the generic ABC promises) are commons/artifacts.py. The Python backend and its reconstruct module re-import them under the old names, pinned by identity in tests/analysis/commons/test_lifted_helpers.py; the TypeScript Neo4j backend imports from commons instead of from the Python package. The TS backend's private _module(props, children) helper is renamed _build_module so a grep for the retired _module node property finds prose only.
One reconciled [Unreleased] CHANGELOG block instead of two: the pin
0.4.3 -> 1.2.0, JavaScript modules in scope on both id prefixes, the generic
ABC inheritance and the shared artifact layer, the call-graph keying with its
kind vocabulary (module|class|interface|enum|type_alias|namespace|callable|
external), TSCallGraphEdge{src,dst,prov,weight} as BREAKING-by-value, the
1.2.0 CLI, and the Neo4j vocabulary migration with the refusals it implies:
a graph below the 1.2.0 floor, or one holding no :Application with the
requested id, now raises GraphSchemaMismatch at attach instead of answering
every query with zero rows.
The projection's lossiness is stated plainly rather than papered over: the
four accessors that RAISE naming the gap (get_imports, get_exports,
get_unresolved_config_reads, get_method_parameters for a found method) and
the ones that return the model's documented empty; the get_call_targets ""
parity note; the declaration-merge collapse (three nodes on superset) as a
known emitter limitation. The commons/ lifts are listed as internal.
docs/agent-api-reference.md gains a TypeScript section: what attaches today,
that the rest of the reference's query surface arrives in 2.5b, the version
floor table, JavaScript scope, the call-graph endpoints TypeScript keeps, and
the lossiness table. CLAUDE.md's TypeScript row carries the pin and the graph
floor. The spec's §4 records the measured lossiness and §5 the per-label seek
rule as it ended up (bare label for signature/prefix statements,
:CanNode:<Label> for id-equality point lookups).
Also: a live, read-only refusal test for attaching to a codeanalyzer-python
graph, gated on its own CLDK_TEST_NEO4J_PYTHON_* variables with no defaults;
and GraphSchemaMismatch's two generic fallback sentences no longer name
codeanalyzer-python when the mismatch is a TypeScript one.
…line Task 2's checkboxes were never ticked though bf187c4 and 35b400f did the work. The Interfaces block still prescribed any(p IN $prefixes WHERE ...), which the measurements in the same task rejected: it plans as a label scan where the OR form seeks. The shipped code uses the OR form through one helper; the plan now says so.
…ked application view, honest extends/implements, doc corrections Seventeen review findings. Three change behaviour: - `get_application_view()` on `TSNeo4jBackend` carried an empty artifact layer while the dedicated accessors answered 832 artifacts, 3,601 dependencies and 526 config keys on the same attach, so `app.dependencies` read as "this project declares none". It now builds `artifacts` (config keys included), `dependencies` and `config_uses` from those same accessors, retyped onto the wire's `TS*` models. `param_in`/`param_out` stay empty because 2.5a reads no dataflow overlay at all — stated in the docstring and the lossiness tables as a 2.5b item, with `config_reads` and `unresolved_imports` beside it. - `get_extended_classes`/`get_implemented_interfaces` read `implements_types`, which the projection never writes: 0 of 207 classes and 0 of 687 interfaces on the superset-frontend graph carry it, so `base_classes` minus nothing would have returned a class's implemented interfaces as its extended classes — the one review finding that could return a wrong value rather than an empty one. The split is now read off the relationships that draw it, `TS_EXTENDS` (154 edges on that graph) and `TS_IMPLEMENTS` (not a relationship type there at all), which covers resolved in-repo bases only; a graph declaring neither cannot be split at that seam, so each raises naming the gap as the four other projection gaps do. `_probe_schema` records the graph's relationship types for that check. - `is_entrypoint`/`entrypoints` were declared only on `TSClass`, so under `extra="forbid"` a 1.3.0 payload stamping them on an interface, enum, type alias or namespace would fail validation — the pin-bump breakage TS-8 exists to prevent. Both move onto `_Type`, with a test constructing all five kinds. Honesty and diagnostics: - The `TSAnalysisBackend` ABC said `get_synthesized_callables` keys by the analyzer's older anonymous id; that is the in-memory backend's behaviour, not the contract both backends sign. The ABC now says the keying is backend-dependent and names both. `get_all_nested_classes` no longer promises classes it can never return: on schema v2 a class holds only `callables` and `fields`, so it is permanently empty on both backends, with `TSCallable.inner_classes` as the surviving case — a lossiness row and a docstring on the facade accessor. - `get_config_uses` returns `[]` both for a corpus with no config read and for a database declaring no `TS_USES_CONFIG`. Adding the type to the required set would refuse a valid graph, so the empty stays and the docstring and the lossiness tables say the two states are indistinguishable here. - `docs/agent-api-reference.md` claimed every accessor was exercised against the live graph. Measured: that graph has zero `TSDecorator` nodes and no `TS_DECORATED_BY`, `TS_IMPLEMENTS` or `TS_USES_CONFIG` relationship types, so the five decorator accessors and `get_config_uses` are verified only to return their empty and `get_implemented_interfaces` only to raise. Named as untested-by-corpus. - `2,184 .ts/.tsx` becomes what the graph holds — 1,841 modules, 1,557 TypeScript and 284 JavaScript — in the reference, the spec and the CHANGELOG. - `_module_key`'s failure named nothing; it now names the node whose key missed. Nothing else identifies it at that point: a call-graph module endpoint reaches the helper with no signature, and the missing module key is the failure itself. - `GraphSchemaMismatch` cited only the Python backend as its raiser and its `PY_HAS_CALLSITE` branch told any caller to re-ingest with codeanalyzer-python; both are language-neutral now, and the branch states the fact instead of suggesting a fix. - `module_dotted`'s unconditional `/__init__` strip is documented as Python-specific. - The CHANGELOG's BREAKING list gains `TSCallableOverview.from_callable`'s required keyword-only `path`. - The spec's G2 records that TS-4 assigns the non-Python-shaped query surface to 2.5b, so a reader does not score G2 as unmet; the plan's TS-11 line records the shipped eight-value `kind` vocabulary in place of the drafted four; the plan records the post-review counts. - `test_no_statement_names_retired_vocabulary` treated `TSCanNode`/`JSCanNode` as retired. They are the newer labels the live graph carries and cants#95 is expected to land them, so they are now asserted as labels this backend deliberately does not target yet, separately from the retired 0.4.3 set. - One `E741` at the declaration-merge facet comprehension.
…r-typescript 1.3.0
TS-1. `cldk/analysis/commons/results.py` declared a language-neutral result
module and then imported one language's schema to type it:
`LocateResult.node` was codeanalyzer-python's `BodyNode` and `LocateResult.span`
its `Span`. That is what kept `locate` off the cross-language ABC, and what
TypeScript would have had to satisfy as-typed in 2.5b.
`BodyRef{id, kind, span, callee}` replaces the field (`LocateResult.node` ->
`LocateResult.body`) and carries only what both analyzers emit for a body node.
Both Python backends map their node onto it where they build the result; the
graph backend's `callee` is projection-lossy and says so, since callee
resolution there is a `PY_RESOLVES_TO` edge, not a node property.
`Span` is *added* to commons, not moved. It cannot be moved: `cldk/models/python`
re-exports codeanalyzer-python's schema, and every `Py*` model's `span` field is
annotated on that class, so putting a commons class in its place turns
`BodyNode(span=Span(...))` into a ValidationError at six call sites and silently
breaks `isinstance(callable.span, Span)` for every caller. The commons `Span`
sets `from_attributes=True` instead, so codeanalyzer-python's `Span` and
`TSSpan` both validate into it field for field and no backend converts by hand.
`cldk.models.python.Span` is unchanged, pinned by an is-identity test.
commons/ now imports no language package except the five repository-artifact
models its own ABC documents as the cross-language layer; a parsing test pins
that allow-list.
Pin moves 1.2.0 -> 1.3.0 and the graph floor with it. 1.3.0 is the first release
whose L4 port lattice is wired to the statement DDG, whose body nodes and
parameters carry ids, and which retired `_module` -- a 1.2.0 graph has the whole
relationship vocabulary and would answer the query surface with empties, so it
is refused on its version stamp (verified: 7692 attaches at 1.3.0, 7690 is
refused naming both). The four v2 fixtures are regenerated with the 1.3.0 wheel;
no model needed a field, which is 2.5a's pre-declaration doing its job.
Seven accessors on both TypeScript backends and the facade, mirroring PythonAnalysis's signatures exactly: locate, locate_many, resolve_callable, resolve_value, get_source, describe, has_resolution_edges. Two rulings the corpus forced. An anonymous callable is addressed by its signature, never by its name. cants names all 7,044 of them "(anonymous)" and gives each a unique signature ending in <anon@line:column>, so the signature is the address and the display name is not one at all: the resolver matches signatures, none carries that string, and the spelling misses outright rather than becoming a 7,044-way ambiguity. A declaration-merged name resolves to the facet its kind names, or to nothing. The emitter mints one id for a value and a type of the same name (cants#177; three collisions on superset-frontend), collapsing them onto one node with both labels and one kind. resolve_callable's domain is the kind, not the label, so a merged node can never come back described as something it is not. Seek labels re-measured on the reference graph rather than assumed: a per-module prefix seek wants :CanNode:TSCallable (346-358 ms bare against 97-102 ms, same rows), a per-application one wants the bare label (24-28 ms against 44-51 ms). :TSCanNode is refused on correctness — it is the per-namespace marker and drops every can://javascript/<app>/ module. The numbers are in the leg plan. module_dotted's /__init__ strip becomes a parameter: nothing stops a TypeScript project having src/foo/__init__.ts, which is a module in its own right. body_key_column moves to commons — both languages' body nodes share its key grammar — and is re-exported where Python already read it.
…predicates Fourteen accessors on both TypeScript backends, mirroring PythonAnalysis's signatures keyword-for-keyword: the three per-callable EdgePages, the three slices, reaches, callers_of/callees_of, the two path queries and the two flow predicates. The bounds stay asymmetric on purpose. Slices and backward_cone default depth to a finite value and cap max_nodes; reaches, paths_between, call_paths_between, flows_to_call and flows_to_argument are unbounded by default, because a bounded predicate returns a wrong answer rather than a small one. Both halves are asserted, offline and live: a known flow is True unbounded and False at a depth that cuts it. TypeScript's DDG has exactly one provenance tier -- all 119,384 TS_DDG edges on the reference application carry ["reaching-defs"] -- so Python's three-way certainty ranking collapses to a single value. The field and the ranking helper stay; no tier is invented. The per-callable page is anchored on a body-node id prefix rather than on the containment pattern the Python backend uses. Measured on the reference graph it is both faster (42.8/11.7/10.8/9.2 ms against 55.5/19.4/17.6/16.8 ms across four callables, with :CanNode:TSBodyNode earning its seek on the narrowest prefix on this surface) and correct: the containment pattern binds two TS_HAS_BODY_NODE relationships in one MATCH, which Cypher's relationship-uniqueness rule forbids from being the same one, so it silently drops every self-loop edge -- 173 TS_DDG and 2,511 TS_SUMMARY of them here. The call graph keeps TypeScript's own vertices (TS-11): a module is the caller of its own top-level code, so callers_of, backward_cone and the call paths report it with kind="module" rather than answering "nothing reaches this". _shortest_walks is lifted out of PyCodeanalyzer into commons/graphs.py with the via table as its one parameter -- it is a graph algorithm over strings and knows no language, and both local backends now bind it. Three findings that are not this change's code but block the harness: codeanalyzer-typescript 1.3.0 refuses -a alongside --emit neo4j; its live Bolt push silently drops every type node and containment edge while the same run's graph.cypher carries them; and it projects :TSCallable.code one line short. The write-gated parity harness now applies the emitter's own projection, and runs 17 of 22 -- with all 11 of this change's dataflow parity items passing. The five failures are pre-existing source-text accessors and are left failing rather than softened.
2 tasks
…ecord the 1.3.0 query surface Leg 2.5b Task 3. Nine accessors on `TypeScriptAnalysis`, each with `PythonAnalysis`'s signature and semantics: `get_entrypoints`, `get_entrypoint_classes`, `get_entrypoint_coverage`, `get_config_readers`, and the five repository-artifact getters, which had been on both backends since 2.5a and now reach the caller by delegation. The frozen public surface grows 46 -> 74 methods plus one property; a new test asserts the parameter mirror against `PythonAnalysis` for all 54 shared accessors, with a five-entry ledger of pre-existing 1.x divergences that must be deleted rather than left behind. The graph carries the entrypoint report as a JSON string property `entrypoint_report_json` on the `:Application` anchor, so it is parsed rather than rebuilt from nodes -- there are no per-entrypoint nodes to rebuild it from. `is_entrypoint` is projected onto `:TSCallable` and `:TSClass` only, so `get_entrypoint_classes` is classes-only on both backends; widening it would make them disagree. A source carrying no report answers with the shared `entrypoint_report_unavailable` diagnostic rather than empty-but-clean-looking fields. The multi-application audit now judges scope **per bound variable** and pins the driver surface to an allow-list, both ported from Java's leg-3a review. The per-variable net found seven real leaks in statements written by earlier tasks -- `TS_CALLS` scoped on one endpoint of two (matching the other by a signature two applications can both declare), a quantified path's far end and trailing node, a slice's reached body nodes, and the extends/implements target -- and each is fixed rather than excused. The harness also could not read a Cypher boolean literal, so it silently ignored `is_entrypoint = true`; fixed. Release gate 1059 passed / 285 skipped at 80.14% (was 1012/274 at 79.66%). TypeScript live on the 1.3.0 reference graph 491/49 (was 440/42); offline 419/121 (was 372/110); Python live 588/6, unchanged. The write-gated dual-backend harness is 24 passed / 5 failed of 29, with all seven new parity tests green; the five failures are the pre-existing source-text ones that trace to `:TSCallable.code` being projected one line short. `get_config_keys`'s `can://` dict key is deliberately left alone: it is a cross-language break tracked by #346, and the reference now says so at the point of use.
…test (#337) The inject step downloaded codeanalyzer-java's latest release regardless of the pin in pyproject.toml, so 2.0.0rc2 shipped both codeanalyzer-2.4.1.jar (checked in) and codeanalyzer-3.0.1.jar (injected); JCodeanalyzer._locate_jar picked 2.4.1 only by lexical order. Read the pin, fetch codeanalyzer-<pin>.jar from releases/tags/v<pin>, replace whatever is under jar/, and make the bundle check require exactly that one jar in the wheel and the sdist. A pin without a matching asset now fails the release instead of silently bundling something else. Closes #336
Decisions J-1..J-12: string call-graph node keys, path key + package dotted vocabulary, name-or-signature callable resolution with overloads raising, CRUD raising on v2 until codeanalyzer-java #187, annotation markers, addressability of initializers/implicit/anonymous callables, leaf accessors, rich v1 fields as computed views, floor 3.0.1 with a generation-aware probe, source_code removal, two work items (#310 = 3a, #311 = 3b) in 2.0.0-rc.4.
Generated by the released codeanalyzer-3.0.1.jar on Temurin 21.0.5+11, never hand-edited. a1 is the whole application at -a 1 (138 units). a4 is -a 4 for the two beans, TradeDirect and TradeServletAction; --schema v2 rejects -t, so the restriction is a source-pruned copy of the tree analyzed with --no-build. The README carries the exact command lines and the measured contents.
….3.7 ones analysis_json now yields the v2/a1 (L1, whole application) text and a new analysis_json_a4 fixture yields v2/a4 (L4, four units). The two 2.3.7 fixtures go with them. Expected red until T1 lands the v2 models: 83 failed, 52 passed, 14 skipped -- every one the v1 JApplication refusing the v2 envelope (symbol_table Field required). Plan Task 0 ticked with the measured floors and the deviations (no -t at --schema v2; fixture sizes).
The analyzer pretty-prints; the raw a1 + a4 pair is 18.7 MB. gzip -9 brings them to 1.3 MB with no loss, and conftest reads them with gzip.open.
…es kept as views cldk/models/java is now an extra="forbid" pydantic mirror of the 3.0.1 envelope (JAnalysis -> JApplication -> JCompilationUnit -> JType -> JCallable, with body nodes, cfg/cdg/ddg/summary edges, param edges, artifacts and dependencies). The v1 field names callers read stay as computed views: code is the span slice of the unit source, call_sites are built from the call body nodes, thrown_exceptions/cyclomatic_complexity/ variable_declarations/referenced_types/accessed_fields read error_channel/ metrics/local_variables/refs, the JType predicates read kind and the owner chain threaded in model_post_init. JGraphEdges is now the wire JCallGraphEdge; the module-global callable lookup table is gone. Both 3.0.1 fixtures round-trip byte-equal through model_dump(exclude_unset, by_alias). tests/analysis/java is red at the backend sites until Task 2.
- code slices by UTF-8 byte offsets (span.bytes are getBytes(UTF_8) prefix sums), decoded; plain-index fast path for ASCII units - JCallable.code is the v1 value: the body block (body_span), falling back to span only without a body - node identity is the id on JType/JCallable/JField/JCompilationUnit so parses compare without walking the owner back-references - nested_type_declarations returns qualified names; annotations return the v1 source spelling (@name(args)) - a spanned node never threaded into a unit raises instead of returning ""; file_path raises when no JApplication stamped it - JBodyNode is spanned; one helper each for decorator names / first comment
…3.0.1, string call-graph keys, source_code removed JavaAnalysisBackend now parameterises AnalysisBackend[JApplication, JCompilationUnit, JType, JCallable, JField, JCallableParameter] with P = N = "J", so a query added to the shared shape has to land in Java too or the backends stop instantiating. JCodeanalyzer drives 3.0.1 at L1-L4 (the level reaches the analyzer as -a via analyzer_level, so 3 and 4 are newly requestable for Java), validates the schema-v2 envelope with JAnalysis.model_validate_json, and keeps it on .analysis alongside .application. Two sanctioned public shifts (spec leg 3): - J-1: get_call_graph() nodes are "<type fqn>.<signature>" strings, not (signature, klass) tuples. Node attrs method_detail/kind; edge attrs type/weight/calling_lines unchanged. Nested and local classes are spelled in source form off JType.qualified_name (Outer.Inner, Outer.$anon$0). @external/ endpoints are dropped, keeping 1.x's callable-only graph; get_external_symbols arrives in 3b. An endpoint homed on no callable raises naming it rather than becoming a node keyed by a raw id. - J-10: source_code single-file mode is removed from JavaAnalysis, CLDK.java, the compat shim's Java branch and JCodeanalyzer. get_test_methods reads every unit's source; remove_all_comments raises, which is the honest v2 behaviour for the one accessor that only ever worked in source mode. The ten skip-gated witnesses retire. Also: CRUD accessors raise from one module-level constant on both backends, naming codeanalyzer-java#187 (J-4); a cached analysis.json without schema_version is refused with a re-run message and one below the requested max_level triggers a re-run (J-9, local half); the artifact/config five return the shared Py* models, with get_config_uses and get_unresolved_config_reads empty because the Java wire carries neither; JGraphEdgesST is deleted with its importers and get_system_dependency_graph returns the wire call graph; get_method_parameters is annotated List[JCallableParameter], which is what it always returned. The public surface is frozen by test, derived from 1375b55 minus source_code. The pin moves to 3.0.1 but no jar is committed: python-sdk#339 removes committed jars in this leg, so _locate_jar gains a documented test/dev seam, $CLDK_CODEANALYZER_JAVA_JAR, and the e2e runs the release jar through it. The jar the e2e exercised is codeanalyzer-3.0.1.jar, sha256 c4e7d4b0b3aa4750d6783b370b6b2aea011df6607d9ee4dd6319cd776ab7a4dd. JNeo4jBackend stays abstract until Task 3 implements the artifact five; its 17 red tests are that one cause, as TSNeo4jBackend's were at the equivalent TypeScript step.
…scoped by can:// prefix, with a schema probe
Re-points every statement `JNeo4jBackend` issues at what 3.0.1 projects, and
refuses at attach anything else. The anchor is `:JApplication {name}`; scope is
the single id prefix `can://java/<app>/` from one helper, `_scoped`, so the
spelling cannot drift; `_module`, `J_HAS_UNIT`, `:JCompilationUnit`,
`J_HAS_CALLABLE`, `:JParameter`, `:JCallSite`, `:JComment` and the CRUD labels
are gone (the last six still have index definitions on a 3.0.1 database and
hold no nodes, which is why reading them was a silent empty, not an error).
Eight statements rebuild the canonical JApplication: one anchored module fetch,
one `*0..` containment traversal that replaces every per-parent fetch, one
prefix-scoped call-site fetch, one anchored import fetch, one doubly-scoped
J_CALLS fetch and the anchored artifact/dependency pair. Every accessor then
answers from the same models the in-memory backend walks, so the two agree by
construction rather than by two implementations of the same query.
Seek labels are measured, not assumed (table in the plan): every keyed label
already owns a uniqueness constraint on `id`, so each statement anchors on the
bare specific label and `:JCanNode` is used nowhere -- it loses the
whole-application prefix (118 ms against 24) and the point lookup (1.40 against
1.13 on `:JSymbol`), and wins only a per-module prefix, which no statement here
issues.
Two accessors raise instead of answering, because the projection cannot serve
them and an empty list would read as an answer: get_all_comments and
get_comment_in_file (there are no comment nodes at all -- a declaration keeps
only its javadoc, in a `docstring` property, and a file-level comment is not
projected). get_method_parameters does *not* raise: `parameters_json` is the
analyzer's own serialisation of the list, so parameters round-trip exactly.
Live parity on daytrader8 with ThingsBoard in the same database: identical
symbol-table keys (138), classes (149), JType/JField/JCallable fields,
parameters, call sites (4,006), wire call graph (1,862 edges, prov and weight
included), nx node/edge sets, callers/callees and class call graphs in both
modes, entrypoints (66/133), artifacts (235), dependencies (4), config keys
(336). The projection forces external calls -- 10,705 J_CALLS out of daytrader8
callables, 8,843 to `:JExternal` -- and 3a keeps the 1.x callable-only graph, so
1,862 remain, exactly the JSON's count. Documented tolerances, each measured:
a module's `source` is not projected; `code` is the declaration slice, not the
body block, so it ends with the reference's on all 1,117 callables with a body
and `calling_lines` shift on 563 of 1,862 edges; order within one source line is
not recoverable for fields (1 class), local variables (12 callables) or
annotations (30 declarations); imports are aggregated per target; docstrings are
declaration-level, not file-level; `param_in`/`param_out` and the L3/L4 graphs
are not rebuilt in 3a.
…est span sentinels, and review corrections A local or anonymous class's `JType.qualified_name` now carries the signature of the callable that declares it -- `p.Outer.m(int).$anon$0`, mirroring the id grammar the analyzer already writes -- and with it every `get_class(...)`, `get_all_classes()` and `get_call_graph()` node key on both backends. `$anon$N` is numbered per declaring callable, so without that segment two sibling callables of one type both spell `$anon$0`: on ThingsBoard 97 colliding names shadowing 381 of 5,102 type declarations, and on `JNeo4jBackend` the duplicate made the first index-routed accessor raise -- the whole backend was dead on the scale corpus, while daytrader8's four non-colliding local classes hid it. Fixed once on the model, so both indexes inherit it and neither is patched. Member (nested) types are unchanged: no collision in the 97 involves only type-declared anons, because those share a per-type counter. Recorded as the J-1 erratum in the spec and in CHANGELOG.md with a migration; witnessed offline by a hand-built payload driven through both backends, and live by a new ThingsBoard suite. Spans stop reporting a plausible wrong number. The projection carries `start_line`/`end_line` and nothing else, so every reconstructed column and both byte offsets are the model's own -1 on types, callables, fields, local variables, call sites and units -- the `0` columns and the `bytes=(0, len(code))` offset into a `source` that is `""` are gone. Frozen by assertion in the type, callable and field parity tests. J-16, a new spec decision that overrides Task 3's ruling: the two *file-keyed* comment accessors keep the raise (the graph holds nothing file-level, so any answer is fabricated), the three *declaration-keyed* ones keep the javadoc-only subset -- and the split is now stated where a caller reads it, with a `Raises:` clause on the ABC and the narrowing documented on the ABC and the facade both. `get_all_docstrings` also harvests enum-constant and record-component javadoc, which `reconstruct` had always rebuilt and the loop had always skipped (ThingsBoard: 1,248 constants, 29 documented; daytrader8 has none, so the parity test could not see it). The Neo4j module docstring's seek rationale was factually wrong -- `:JCallable` owns no id index at all, only a range index on `name` and a fulltext, so bare `:JCallable` plans a label scan. The conclusion survives re-measurement and stands; the argument is now the one the measurements support: the two prefix-scoped statements are traversal-dominated (under 1% wall clock either way, `:JCanNode` a quarter again the db hits) and everything else walks out from the `:JApplication` anchor. The multi-application audit was accepting a statement that scopes one endpoint of two, and was blind to any statement not passed to `self._run(`. It now parses the node variables out of each pattern and requires every one to be prefixed or reachable from the application anchor over a containment or shared-vocabulary relationship (`J_CALLS` is deliberately neither), with three negative and three positive cases as its own net; and it allow-lists every attribute touched on `self._driver` / `self._session_obj`, so `execute_query` cannot slip past. Also: `application` is a read-only property over a private `_application` cache, because rebinding it left `_idx` and `_call_graph` stale; `reconstruct` reads `props["source"]`, `edge["spec"]` and `edge["kind"]` rather than defaulting (`"runtime"` would have silently reclassified every dependency); `_facet` is deleted (its result was discarded, its `"callable"` branch unreachable, and `JType.kind` is already a `Literal`); the two backends' twin defect messages are one shared id-free text each and `_node_of` names the endpoint by signature and module key; the `code`/`body` divergence is documented on the model, the ABC and the facade; the probe's substring assertions are replaced by behaviour, the seek test renamed to say it greps a ruling rather than measuring one, and the dead `_MATCHES_BY_ID` branch removed. Five measured numbers corrected in the plan and the docstrings that quoted them: body-less callables are 99, not 129 (99 + 1,117 = 1,216); local-variable order differs on 6 callables, not 12; the body-key column delta reaches 110 with a mode of 4, not "up to 12"; attach plus first use is nine round trips (3 + 6), not eight statements. `calling_lines` re-measures at 1,299 of 1,862 -- the plan was right -- and the shared-contract line's `26 passed` was the correct total of the three files it named; both are now labelled rather than dropped. Java offline: 304 passed, 29 skipped (from 294/25 -- +10 tests, +4 live-gated skips). Live parity on 7691 with ThingsBoard in the same database: 19 passed. New ThingsBoard suite: 4 passed.
…anctioned shifts CHANGELOG [Unreleased] gains the Java block: the 2.4.1 -> 3.0.1 pin with the extra="forbid" v2 model mirror and the refusal of a v1 cache; analysis levels 3 and 4 becoming reachable; the two sanctioned breaks (string call-graph node keys with a migration that survives dotted parameter types and local classes, and the removal of source_code single-file mode); JGraphEdges becoming JCallGraphEdge; the CRUD raise (codeanalyzer-java#187); the J-16 comment split; the declaration vs body-block divergence of JCallable.code over Neo4j (codeanalyzer-java#176); the 3.0.1 graph vocabulary and probe floor; the generic-ABC artifact accessors; and the get_method_parameters annotation fix. docs/agent-api-reference.md gains a Java section: what attaches today, the 3.0.1 floor as an attach table, the accessors that still raise, and the note that the leg-1.5/1.6 query surface reaches Java in 3b. CLAUDE.md's Java row records the level range, the probed graph and the jar that #339 still has to move. The leg-3 spec gains a J-8 erratum: 3.0.1 emits declaration for every callable that has one (1,115 of daytrader8's 1,216); it is None only on the 99 implicit callables and the two <clinit>$N() initializers, and codeanalyzer-java#215 is about storing the text once per module, not about a missing field. The plan's Task 4 is ticked with the three runs, the grep and the refusal matrix. Also here, because the verification found them: a live refusal test attaching the Java Neo4j backend to a real codeanalyzer-python graph, and the two prose mentions of the retired 2.4.1 pin reworded to name schema v1 instead.
…d drop the bundled jar and JDK download The Java backend now gets both the analyzer and the JVM it runs on from the codeanalyzer-java PyPI wheel, pinned at 3.0.2 behind a new optional `java` extra. 3.0.2 reads its primordial scope from `jrt:/` in the running JVM, so nothing needs `JAVA_HOME` or a `jmods/` directory any more: * `JCodeanalyzer._get_codeanalyzer_exec` returns `codeanalyzer_java.command()`. The import is lazy, so `import cldk` and `import cldk.analysis.java` work without the extra; running the backend without it raises CodeanalyzerExecutionException naming the distribution and `pip install "cldk[java]"`. * Deleted `cldk/analysis/java/codeanalyzer/_jdk.py` (`ensure_jdk`, the pinned Temurin download, the ~200 MB per-project JDK cache), the checked-in `codeanalyzer-2.4.1.jar` and its directory, the `[tool.hatch.build]` force-include, the root `.gitignore` `*.jar` rule and the `CLDK_CODEANALYZER_JAVA_JAR` seam the e2e grew while the tree's jar was stale. * release.yml no longer downloads the pinned jar from the codeanalyzer-java GitHub release and no longer verifies a jar is bundled in the wheel and sdist. This retires the mechanism #284, #336 and #337 patched: the wheel pin is now the single source of the analyzer version, and nothing is fetched at build time. * Extras take the thin-base shape 2.0 is heading for: `neo4j`, `java`, and `all = ["cldk[java]", "cldk[neo4j]"]`. codeanalyzer-python and codeanalyzer-typescript stay hard dependencies until #340. * Fixtures regenerated at 3.0.2 with the command lines the v2 README records; the only content delta against the 3.0.1 pair is `analyzer.version` (verified by recursive parsed-JSON compare: one differing leaf per file). Verified: the wheel path with `JAVA_HOME` unset and no JDK on PATH returns the identical daytrader8 `-a 4` call-graph edge set the 3.0.1 jar returned on the provisioned Temurin -- 1862 edges, 1378 `declared+rta` / 375 `rta` / 109 `declared`, symmetric difference 0 after normalising the app name. A wheel built from this tree contains no `.jar`, and neither does the sdist. Closes #339
…urning a declared-only graph silently codeanalyzer-java degrades rather than failing: when the build it needs for L3/L4 cannot run, it emits a declared-only call graph and an SDG with no points-to provenance, exits 0, and still stamps max_level with the level it was asked for. Measured on daytrader8 at -a 4, the only difference between Maven on PATH and not is 1862 edges (declared+rta) versus 1391 (all declared) and 1220 points-to ddg edges versus none -- the envelope, the exit code and the application keys are identical. The analyzer declares this itself, but only on its log, at WARN. So the log is the signal. The SDK now passes -v (only alongside -o, since without it the payload occupies the same stdout the log would), matches WARN lines on the shape the analyzer uses -- a capability "unavailable", "emitting ... only" -- rather than on the exception text inside the parentheses, strips the ANSI colour, and keeps the analyzer's own sentence as the message. The verdict is persisted beside analysis.json as <cache>/java/analyzer_diagnostics.json, never as a field inside the payload, so a cache hit -- which by design never re-invokes the analyzer, and therefore has no log -- still knows. Three states, distinguishable, on JCodeanalyzer.analyzer_diagnostics: a non-empty list of level_too_low Diagnostics (the analyzer declared a degradation), [] (it declared none), and None (no verdict is recorded, so it is unknown -- a pre-#341 cache, an analysis.json from elsewhere, or stdout-pipe mode). None is reported as unknown, never as clean. Each state is logged once per analysis at WARNING, and nothing raises: a declared-only call graph is still a real answer, and callers content with it keep working. The payload's proxies (no rta, no points-to) are not consulted at all -- a small project can legitimately have neither. No new accessor on JavaAnalysis; a public reader is a 3b candidate. Closes #341
…g, and close the leg review
`get_test_methods()` re-parsed each compilation unit's `source` with Tree-sitter, and the
Neo4j projection carries no module source, so it returned `{}` on every graph-backed
analysis — `{}` on ThingsBoard, where 3,354 callables carry `@Test`/`@ParameterizedTest`.
It now reads the analyzer's own annotations off the model, so both backends answer the
same, and is keyed by `"<type fqn>.<signature>"` rather than a bare method name that
collapses same-named tests across classes.
`CHANGELOG.md`'s `[Unreleased]` block was structurally corrupt and the release workflow
lifts it verbatim: a `### Removed` heading split the `### Changed` list, two bullets
appeared twice (the second copy superseded), the Java pin read 3.0.1 where the pin is
3.0.2, and the preamble named only leg 2.5a. One heading of each kind now, no bullet
repeated, and both legs and both release targets named.
Also from the review: `get_config_keys()` is keyed by the artifact-relative key rather
than a `can://` id stamped with `--app-name` (the two backends shared zero keys);
`calling_lines` are sorted absolute file lines rather than offsets into a `JCallable.code`
that differs by backend, which removes the divergence on 560 of daytrader8's 1,862 edges;
`get_call_graph()` parses each body once instead of once per edge (ThingsBoard 145.7s →
41.0s); the degradation sidecar is bound to its payload's sha256 and never raises; the
`Makefile` no longer wgets a jar into a deleted directory; and the docstrings that
described behaviour neither backend has now name the backend arm.
The rebase left two Changed and two Fixed headings and six duplicated bullets, at 417 lines. Same content, one section of each, ordered by what a reader needs first: what breaks and how to migrate, what is new, what changed, what is fixed, what is still missing.
rahlk
force-pushed
the
feat/issue-339-codeanalyzer-java-wheel
branch
from
September 7, 2026 03:39
4b259e9 to
bb47baf
Compare
Collaborator
Author
|
Closing as superseded rather than merging: every one of this branch's 34 commits is already contained in #358, rebased. When the Java query surface (#358) was rebased onto leg 2.5b so the stack would be linear, this branch's commits were replayed with new hashes. That leaves the branch itself an older tip of content that has since moved on. Merging it now conflicts on Verified before closing: with this branch dropped, the stack merges cleanly in order, and all 34 subjects from here are present in the result. The merged tree is byte-for-byte identical to the stack tip that passed the gate at 1537 passed / 352 skipped, 85.11%. So leg 3a ships in #358 alongside 3b, and issues #310 and #339 are closed by that merge rather than this one. No work is lost; only this branch pointer is. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #310. Closes #339. Closes #341. Epic: codellm-devkit/.github#55. Spec:
docs/design/specs/2026-09-06-leg-3-java.md(J-1…J-17, with errata on J-1, J-8 and J-9). Plan:docs/design/plans/2026-09-06-leg-3a-java-schema-v2.md.Stacked on #342 (TypeScript leg 2.5a) and reviewable after it merges; the range that belongs to this PR is
0170f67..HEAD, which touches no TypeScript, Python or shared code.Three things ship together because they are interdependent: the migration pins an analyzer, the wheel changes where that analyzer comes from, and the degradation reporting only makes sense once levels 3 and 4 are reachable.
The migration (#310)
The Java models become a mirror of what codeanalyzer-java 3.0.1 emits, with every v1 field name kept as a computed view, so
JCallable.code,.call_sites,.thrown_exceptionsand the rest still answer.JavaAnalysisBackendinherits the generic backend, and both backends move onto the 3.0.1 graph vocabulary withcan://java/<app>/prefix scoping and a schema probe that refuses anything older at attach rather than serving silent empties. Levels 3 and 4 are reachable for the first time.Four sanctioned breaks, each with a migration line in the CHANGELOG: call-graph node keys are
<type fqn>.<signature>strings rather than(signature, klass)tuples; single-filesource_codemode is gone;JGraphEdgesis now a four-field edge; and a local class's qualified name carries its declaring callable. That last one is an erratum to J-1 rather than a plan item, because the original rule was not unique:$anon$Nis numbered per declaring callable, so on ThingsBoard 97 names collided and shadowed 381 of 5,102 type declarations, which killed every index-routed accessor on the Neo4j backend. daytrader8 has four local classes and no collision, which is exactly why 19 live parity tests passed over it.The analyzer wheel (#339)
The analyzer now comes from the
codeanalyzer-javaPyPI wheel, which carries the jar and its own JVM, as the newcldk[java]extra. The checked-in 35 MB jar,_jdk.py's Temurin download, and the release workflow's jar injection and verification steps are all deleted, which retires the mechanism #336 and #337 patched.pip install cldkno longer pulls a JVM at all;pip install "cldk[all]"reproduces today's behaviour. The wheel drops from about 35 MB to 320 KB, and neither it nor the sdist contains a.jar.Verified that the wheel path returns the identical edge set to the old jar path on the same sources: 1,862 edges, symmetric difference zero.
Honest degradation (#341)
Levels 3 and 4 need compiled classes. When the build cannot run, the analyzer degrades rather than failing: exit zero, the level still reported, but a declared-only call graph. That was silent. The SDK now keeps the analyzer's own warning sentences, logs them, and persists them beside the payload so a cache hit still knows. Three states stay distinguishable: degraded, clean, and no verdict recorded, which reads as unknown rather than clean. Nothing is raised and nothing withheld, because a declared-only graph is still a real answer.
Verification
Live against a graph holding daytrader8 and ThingsBoard together, so scoping is exercised rather than assumed.
Parity is asserted as identical values, not matching counts: every type, callable, field, parameter, all 4,006 call sites, the wire call graph byte for byte, and now every call-line list. Both backends raise byte-identical diagnostics on all eight comparable defect paths.
Two defects the corpus hid, found by review
get_test_methods()returned an empty dict on every Neo4j-backed analysis, because it read module source text the projection does not store. daytrader8 contains zero@Testmethods, so the parity gate agreed by coincidence. It now answers off the annotation relationships: 3,354 test methods on ThingsBoard, instantly.get_config_keys()keyed its dict by an id stamped with the application name, so the two backends could return the same 336 keys under disjoint key sets. It now keys by the artifact-relative key. The same defect remains on Python and TypeScript and is filed as #346.Also fixed while in range:
get_call_graph()re-parsed each method body once per edge, 145.7 seconds on ThingsBoard, now 41.0.Known and filed, not fixed here
codeanalyzer-java#187 (CRUD absent from v2, so those five accessors raise naming it), #176 (the graph's
codeis the declaration slice where the JSON's is the body block, and the graph cannot recover the body block), #215, plus python-sdk #345 and #346 in shared code.