diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 009a8deb..721fd3c6 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -17,6 +17,10 @@ jobs: - name: Check out code uses: actions/checkout@v4 + # Load-bearing for **Maven**, not for the analyzer. The analyzer jar and the JVM it runs + # on come from the `codeanalyzer-java` wheel (#339), but codeanalyzer-java's own auto-build + # shells out to `mvn`, which needs a JDK on PATH; without one the Java e2e degrades to a + # declared-only call graph and CI goes green on a partial answer. Do not delete this step. - name: Set up GraalVM CE Java 11 uses: graalvm/setup-graalvm@v1 with: @@ -65,50 +69,9 @@ jobs: git push --delete origin ${GITHUB_REF#refs/tags/} exit 1 - - name: Inject the latest Code Analyzer JAR - run: | - # The release has multiple .jar assets (the versioned codeanalyzer-.jar and an - # unversioned codeanalyzer.jar) — select only the versioned one so $CODE_ANALYZER_URL - # is a single URL. - CODE_ANALYZER_URL=$(curl -s https://api.github.com/repos/codellm-devkit/codeanalyzer-java/releases/latest | jq -r '.assets[] | select(.name | test("^codeanalyzer-[0-9].*\\.jar$")) | .browser_download_url') - echo "Downloading: $CODE_ANALYZER_URL" - wget -q "$CODE_ANALYZER_URL" - mkdir -p ${{ github.workspace }}/cldk/analysis/java/codeanalyzer/jar/ - mv codeanalyzer-*.jar ${{ github.workspace }}/cldk/analysis/java/codeanalyzer/jar/ - - name: Build Package run: uv build - - name: Verify the codeanalyzer JAR is bundled - # Guard against the hatchling/.gitignore regression (issue #284): a jarless wheel - # installs fine but fails at runtime with "codeanalyzer jar not found". Fail the - # release here rather than publish a broken artifact to PyPI. - # - # The listing is captured before grepping: piping `tar tzf` (which decompresses the - # whole 32MB sdist) straight into `grep -q` lets grep close the pipe on first match, - # SIGPIPE-killing tar and — under `pipefail` — reporting a false "missing JAR". - run: | - set -euo pipefail - jar_re='codeanalyzer/jar/codeanalyzer-[0-9][^/]*\.jar$' - fail=0 - for f in dist/*.whl dist/*.tar.gz; do - case "$f" in - *.whl) listing=$(unzip -l "$f") ;; - *.tar.gz) listing=$(tar tzf "$f") ;; - esac - if grep -qE "$jar_re" <<<"$listing"; then - echo " ✓ $f" - else - echo "::error::$f is missing the codeanalyzer JAR" - grep -i '\.jar' <<<"$listing" || echo " (no .jar entries at all)" - fail=1 - fi - done - if [ "$fail" -ne 0 ]; then - echo "Refusing to publish a jarless release."; exit 1 - fi - echo "codeanalyzer JAR present in wheel and sdist ✓" - - name: Extract release notes from CHANGELOG.md id: notes # Source the release body from the hand-written CHANGELOG.md section for this tag — diff --git a/.gitignore b/.gitignore index 9214263b..589a4cf7 100644 --- a/.gitignore +++ b/.gitignore @@ -25,7 +25,6 @@ .mtj.tmp/ # Package Files # -*.jar *.war *.nar *.ear @@ -53,11 +52,11 @@ scratch* *.json !devcontainer.json -# Blessed TS unit-test fixture: hand-built from a sample app whose source (incl. src/external.ts) -# was never committed, so it cannot be regenerated by running codeanalyzer-typescript again. The -# bulk-accessor tests assert exact-set constants (signature counts, ownerless sets) against this -# exact file -- losing it breaks the suite for every fresh clone (#298). -!tests/resources/typescript/analysis_json/slim/analysis.json +# Blessed TS unit-test fixtures: analysis.json at each level, generated by the pinned +# codeanalyzer-typescript from tests/resources/typescript/application (see the README there), and +# the sample app's own manifests, which the analyzer's artifact layer reads. +!tests/resources/typescript/analysis_json/v2/*/analysis.json +!tests/resources/typescript/application/*.json # Python compiled files and env diff --git a/CHANGELOG.md b/CHANGELOG.md index 000be40c..ac0410a1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,91 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +Three legs of the 2.0 line: **TypeScript on schema v2** (2.5a) and **its query surface** (2.5b), and +**Java on schema v2** (3a, with the analyzer wheel and honest degradation reporting). Design records: +`docs/design/specs/2026-09-06-leg-2.5-typescript.md` and `docs/design/specs/2026-09-06-leg-3-java.md`. + +### Breaking + +- **Java: `get_call_graph()` nodes are `"."` strings, not `(signature, klass)` tuples.** + Take the owning class from `cg.nodes[key]["method_detail"].klass` rather than splitting the key. +- **Java: single-file `source_code` mode is gone.** Analyze the project directory instead. +- **Java: a local or anonymous class's qualified name carries its declaring callable** (`p.Outer.m(int).$anon$0`). + Without it, sibling callables collide. Take class keys from `get_all_classes()` rather than composing them. +- **A graph emitted below the analyzer floor is refused at attach** with `GraphSchemaMismatch` instead of + answering every query with zero rows: codeanalyzer-java 3.0.1 and codeanalyzer-typescript 1.3.0. Re-emit; + there is no in-place upgrade. `TSNeo4jBackend` also speaks a graph vocabulary that shares nothing with 0.4.3's. +- **Values that changed shape:** `JGraphEdges` and `TSCallEdge` are now `{src, dst, prov, weight}`; Java's + `calling_lines` is a sorted list of absolute file lines; Java's `get_config_keys()` is keyed by the + artifact-relative key; Java's `get_test_methods()` reads the analyzer's annotations rather than re-parsing + source; `TSCallable` lost `path`/`call_sites`/`accessed_symbols`/`local_variables`/`code_start_line` and + `TSModule` lost `file_path`/`module_name`, since v2 keys modules by path and stores source once; + `TSCallableOverview.from_callable` takes a required keyword-only `path`. +- **Removed:** `TypeScriptAnalysis.get_entry_point_methods` and `get_service_entry_point_methods`, which only + ever raised — the working entrypoint accessors below replace them. + +### Added + +- **The agent-facing query surface on `TypeScriptAnalysis` — 29 accessors, each with `PythonAnalysis`'s + signature and semantics.** Addressing (`locate`, `locate_many`, `resolve_callable`, `resolve_value`, + `get_source`, `describe`, `has_resolution_edges`); per-callable graphs and dataflow (`get_cfg`/`get_cdg`/ + `get_ddg`, `slice_backward`/`slice_forward`/`backward_cone`, `reaches`, `callers_of`/`callees_of`, + `paths_between`/`call_paths_between`, `flows_to_call`/`flows_to_argument`); entrypoints + (`get_entrypoints`, `get_entrypoint_classes`, `get_entrypoint_coverage`, `get_config_readers`); and the five + repository-artifact getters. Both backends answer identically, including on the miss paths. +- **Java reaches analysis levels 3 and 4** — control flow, control and data dependence, and the interprocedural + graph. The level now reaches the analyzer, which it never did before. +- **A `java` install extra.** `pip install "cldk[java]"` brings the analyzer and its bundled JVM; + `pip install "cldk[all]"` reproduces the previous behaviour. Bare `cldk` no longer carries either. The Python + and TypeScript analyzers are still installed unconditionally; #340 moves them into extras of their own. +- **JavaScript modules are in scope** for TypeScript analysis, under their own id prefix. +- **A degraded Java level-3 or level-4 run is reported rather than silent.** Those levels need compiled classes; + without them the analyzer emits a declared-only graph and still reports the level. The SDK now surfaces the + analyzer's own warnings and records them beside the payload, so a cached run still knows. The graph is + returned either way. +- `cldk.models.typescript.TSClassOverview`, the class-level projection `get_entrypoint_classes` returns. + +### Changed + +- **Pins:** `codeanalyzer-java` 2.4.1 → 3.0.2, `codeanalyzer-typescript` 0.4.3 → 1.3.0. +- **The Java analyzer ships as a wheel, not a jar in this repo.** The 35 MB checked-in jar, the Temurin download + in `_jdk.py`, and the release workflow's jar injection are gone; no `JAVA_HOME` is read or set, and no JDK is + downloaded. The published wheel drops from about 35 MB to 320 KB. +- **Both languages' backends inherit the generic `AnalysisBackend`**, so each answers the shared artifact, + dependency and configuration accessors. +- **Where a backend cannot answer, it says so instead of returning an empty value.** On Neo4j: Java's file-keyed + comment accessors, and TypeScript's `get_imports`, `get_all_exports`, `get_unresolved_config_reads`, + `get_method_parameters` for a found method, and `get_extended_classes`/`get_implemented_interfaces` when the + relationship type is absent. The remaining documented gaps, and where the two backends legitimately differ, + are listed in `docs/agent-api-reference.md`. +- **Every Cypher statement is scoped per bound variable, not per statement** — both endpoints of a call edge, a + quantified path's far end, and a slice's reached body nodes. A statement that matched one endpoint by a + signature two applications both declare could previously return the other application's node. +- **`LocateResult.body` is a language-neutral `BodyRef`**, so `cldk/analysis/commons/` no longer imports a + language package for it. +- Internal: the language-neutral query helpers moved from `cldk/analysis/python/` to `cldk/analysis/commons/`; + Python re-imports every name unchanged. + +### Fixed + +- **`JavaAnalysis.get_method_parameters()`** is annotated `List[JCallableParameter]`, which is what it has always + returned. +- **`get_call_graph()` on Java no longer re-parses each method body once per edge** — 145.7s to 41.0s on a + 4,100-file project. +- Two documentation errors: `EntrypointCoverage.unresolved` is a `dict[str, int]`, not a `list[str]`; and the + `get_config_keys()` example used a key form that never worked. + +### Known limitations + +- Java CRUD accessors raise: schema v2 does not carry CRUD yet (codeanalyzer-java#187). +- The Java graph's `JCallable.code` is the declaration slice where the JSON's is the body block, and the graph + cannot recover the body block (codeanalyzer-java#176). +- TypeScript's DDG has one provenance tier; Java's has two; Python's has three. +- codeanalyzer-typescript mints one id for a value and a type of the same name under declaration merging + (codeanalyzer-typescript#177); such a node resolves to the facet its kind names, or not at all. +- `get_config_keys()` is still keyed by a `can://` id on Python and TypeScript, where Java now uses the + artifact-relative key (#346). + ## [v2.0.0-rc.2] - 2026-09-06 Python legs 1, 1.5 and 1.6 of the CLDK 2.0 agent-facing query facade (see `docs/design/specs/2026-09-03-agent-facing-query-facade.md`, diff --git a/CLAUDE.md b/CLAUDE.md index e36c713c..16c0ce9d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,9 +10,14 @@ an optional read-only Neo4j backend — selected by the *type* of the `backend=` | Language | Entry point | Local backend | Neo4j backend | Models | |----------|-------------|---------------|---------------|--------| -| Java | `CLDK.java(...)` | `JCodeanalyzer` (bundled JAR, subprocess) | `JNeo4jBackend` | `cldk/models/java/` | +| Java | `CLDK.java(...)` (needs the `cldk[java]` extra) | `JCodeanalyzer` (the `codeanalyzer-java` 3.0.2 wheel's jar on its bundled JVM, subprocess, `-a 1..4` — no jar in this repo, no JDK download) | `JNeo4jBackend` (3.0.1 graph, probed at attach) | `cldk/models/java/` (schema v2 mirror) | | Python | `CLDK.python(...)` | `PyCodeanalyzer` (in-process `codeanalyzer-python`) | `PyNeo4jBackend` | re-exported from `codeanalyzer-python` | -| TypeScript | `CLDK.typescript(...)` | `TSCodeanalyzer` (`codeanalyzer-typescript` binary, subprocess) | `TSNeo4jBackend` | `cldk/models/typescript/` | +| TypeScript (+ JavaScript modules) | `CLDK.typescript(...)` | `TSCodeanalyzer` (`codeanalyzer-typescript` 1.3.0 binary from the wheel, subprocess; `-a 1..4`, but `--emit neo4j` takes no `-a` and is always full depth) | `TSNeo4jBackend` (graphs emitted by ≥ 1.3.0; older refused at attach) | `cldk/models/typescript/` (schema v2 mirror) | + +**Java, since leg 3a (#310):** the models are an `extra="forbid"` mirror of canonical schema v2, so +a 1.x `analysis.json` (and a pre-3.0.1 Neo4j graph) is refused, not parsed; `get_call_graph()` keys +nodes by the string `"."`; the `source_code` single-file mode is gone; the CRUD +accessors raise (codeanalyzer-java#187). The leg-1.5/1.6 query surface reaches Java in 3b (#311). The legacy `CLDK(language="").analysis(...)` entry still works as a compat shim. Adding a language means a new factory method + facade + backend ABC/impl(s) + models + tests — **update this diff --git a/Makefile b/Makefile index 58cb540e..b4fcc620 100644 --- a/Makefile +++ b/Makefile @@ -39,20 +39,9 @@ clean: ## Cleans up from previous compiles $(info Cleaning up compile artifacts...) rm -fr dist -.PHONY: refresh -refresh: ## Refresh code analyzer - $(info Refreshing CodeAnalyzer...) - wget $(curl -s https://api.github.com/repos/IBM/codenet-minerva-code-analyzer/releases/latest | grep "browser_download_url" | grep codeanalyzer.jar | cut -d '"' -f 4) - mv codeanalyzer.jar cldk/analysis/java/codeanalyzer/jar/codeanalyzer.jar - .PHONY: build build: ## Builds a new Python wheel $(info Building artifacts...) - - # Inject the latest Code Analyzer JAR - wget -q $(shell curl -s https://api.github.com/repos/IBM/codenet-minerva-code-analyzer/releases/latest | jq -r '.assets[] | .browser_download_url') - mkdir -p cldk/analysis/java/codeanalyzer/jar/ - mv codeanalyzer-*.jar cldk/analysis/java/codeanalyzer/jar/ - - # Build the package + # No jar is fetched or injected: the Java analyzer is the `codeanalyzer-java` wheel + # (the `java` extra), a normal locked dependency. Nothing is downloaded at build time. uv build diff --git a/cldk/analysis/commons/artifacts.py b/cldk/analysis/commons/artifacts.py new file mode 100644 index 00000000..c5e1e7ac --- /dev/null +++ b/cldk/analysis/commons/artifacts.py @@ -0,0 +1,105 @@ +################################################################################ +# Copyright IBM Corporation 2026 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + + +"""The shared repository-artifact layer, rebuilt from the Neo4j projection: property maps → +``PyArtifact`` / ``PyConfigKey`` / ``PyDependency``. + +Every codeanalyzer projects this layer identically and unprefixed (``:Artifact``, ``:ConfigKey``, +``:Package``; ``HAS_ARTIFACT``, ``DEFINES_CONFIG``, ``DECLARES_DEPENDENCY``, ``LOCKS``), and the +generic ABC (:mod:`cldk.analysis.commons.backend`) promises the same four ``Py*`` models from every +language, so the reconstructors live here once. Lifted verbatim from +``cldk/analysis/python/neo4j/reconstruct.py`` (leg 2.5a), which re-exports them. +""" + +from __future__ import annotations + +from typing import Any, List, Mapping + +from cldk.models.python import PyArtifact, PyConfigKey, PyDependency + +Props = Mapping[str, Any] + + +def config_key(props: Props) -> PyConfigKey: + """Rebuild a :class:`PyConfigKey` from a ``:ConfigKey`` node's properties. + + Line-only ``span`` (see :func:`body_node`): the projection writes ``start_line``/``end_line`` + and nothing finer, so the columns and byte offsets rehydrate as ``0``. ``span`` stays ``None`` + when the node carries no lines at all (best-effort extraction never located the key in the + artifact's source). It is spelled as a mapping rather than built from + ``cldk.models.python.Span``: ``PyConfigKey.span`` is annotated on *that* class, so only it + validates -- and importing it here would put one language's declaration schema back into + ``commons/`` (TS-1), which the shared artifact layer is exempt from only for the five ``Py*`` + models this module rebuilds. + """ + lines = (props.get("start_line"), props.get("end_line")) + return PyConfigKey( + id=props.get("id", ""), + key=props.get("key", ""), + namespace=props.get("namespace", ""), + value=props.get("value"), + span={"start": (lines[0], 0), "end": (lines[1], 0), "bytes": (0, 0)} if None not in lines else None, + references=list(props.get("references", []) or []), + ) + + +def artifact(props: Props, *, config_keys: List[PyConfigKey] | None = None) -> PyArtifact: + """Rebuild a :class:`PyArtifact` from an ``:Artifact`` node's properties plus its fetched + :class:`PyConfigKey` children (``[:DEFINES_CONFIG]``). + + ``kind`` is not a projected property — every ``PyArtifact`` the analyzer emits carries the + model's own default (``"artifact"``; see ``codeanalyzer/artifacts/discovery.py``), so it is + supplied here rather than queried for. + """ + return PyArtifact( + id=props.get("id", ""), + kind="artifact", + path=props.get("path", ""), + format=props.get("format", ""), + roles=list(props.get("roles", []) or []), + size_bytes=props.get("size_bytes", 0), + sha256=props.get("sha256", ""), + source=props.get("source", ""), + extraction=props.get("extraction", "none"), + config_keys=config_keys or [], + ) + + +def dependency(props: Props, *, name: str, ecosystem: str, declared_in: str) -> PyDependency: + """Rebuild a :class:`PyDependency` from a ``[:DECLARES_DEPENDENCY]`` edge's properties plus its + endpoints (``name``/``ecosystem`` off the ``:Package`` node, ``declared_in`` off the + ``:Artifact`` node). ``ecosystem`` is a real ``Package`` property (``neo4j/schema.py``'s + ``Package`` node type carries it); ``"pypi"`` is only ever what the analyzer happens to write + there today (its only ecosystem, per ``PyDependency.ecosystem``'s own docstring) — read off the + node rather than hardcoded, so this doesn't silently go stale the day a second ecosystem ships. + + ``locked_version``/``provides_imports`` are projection-lossy here: the graph carries them on + the separate ``[:LOCKS]``/``[:PY_PROVIDES]`` edges (per-package facts, not per-declaration), and + no caller of this reconstruction chases those yet, so they come back at the model's own empty + defaults — the same class of gap :func:`callsite` documents for ``argument_types``. + """ + return PyDependency( + name=name, + ecosystem=ecosystem, + spec=props.get("spec", ""), + kind=props.get("kind", "runtime"), + extras=list(props.get("extras", []) or []), + declared_in=declared_in, + direct=props.get("direct", True), + provides_imports=[], + prov=list(props.get("prov", []) or []), + ) diff --git a/cldk/analysis/commons/backend.py b/cldk/analysis/commons/backend.py index b2f6d8c1..f689dcc7 100644 --- a/cldk/analysis/commons/backend.py +++ b/cldk/analysis/commons/backend.py @@ -28,16 +28,21 @@ discussion. What does *not* belong here is a query whose return type is one language's models. ``locate`` / -``locate_many`` (the v2 query-facade spec's D3) is declared on -:class:`~cldk.analysis.python.backend.PythonAnalysisBackend` instead, because -:class:`~cldk.analysis.commons.results.LocateResult` carries ``codeanalyzer-python``'s ``BodyNode`` -and ``Span``: hoisted here it would be a shared contract Java and TypeScript cannot satisfy as -typed. It is hoisted when a second language implements it and the language-neutral shape of a body -node and a span is known from two examples rather than guessed from one. +``locate_many`` (the v2 query-facade spec's D3) was declared on +:class:`~cldk.analysis.python.backend.PythonAnalysisBackend` for exactly that reason: +:class:`~cldk.analysis.commons.results.LocateResult` carried ``codeanalyzer-python``'s ``BodyNode`` +and ``Span``, so hoisted here it would have been a shared contract Java and TypeScript could not +satisfy as typed. TS-1 (leg 2.5b) settled the language-neutral shape from two examples rather than +one -- :class:`~cldk.analysis.commons.results.BodyRef` and the commons +:class:`~cldk.analysis.commons.results.Span` -- so the type blocker is gone; the declaration itself +is hoisted in the same leg's task that gives TypeScript an implementation, since it is a second +implementation, not a neutral type, that earns a method its place here. The repository-artifact getters below (``get_artifacts`` / ``get_dependencies`` / ``get_config_keys`` / ``get_config_uses`` / ``get_unresolved_config_reads``) are the opposite case, -even though they are typed on ``cldk.models.python``'s ``Py*`` classes today. Unlike +even though they are typed on ``cldk.models.python``'s ``Py*`` classes today -- the one language +import ``cldk/analysis/commons/`` still makes, pinned as such by +``tests/analysis/commons/test_lifted_helpers.py``. Unlike ``PyModule``/``PyClass``/``BodyNode``, those models are not Python-specific shapes wearing a ``Py`` prefix out of habit: ``codeanalyzer-python``'s own schema module documents the ``Py`` on ``PyArtifact`` / ``PyConfigKey`` @@ -52,13 +57,25 @@ from __future__ import annotations +import re from abc import ABC, abstractmethod -from typing import ClassVar, Dict, Generic, List, TypeVar +from typing import Any, ClassVar, Dict, Generic, List, Tuple, TypeVar import networkx as nx from cldk.models.python import PyArtifact, PyConfigKey, PyConfigRead, PyConfigUseEdge, PyDependency + +def semver(raw: Any) -> Tuple[int, int, int] | None: + """``"1.4.1"`` (or ``"1.4.1.post0"``) as ``(1, 4, 1)``; ``None`` for anything that does not + start with three dotted integers, so an unparsable version is *unknown*, never silently zero. + + The parser behind every Neo4j backend's analyzer-version floor: the schema probe compares its + result against the backend's ``_ANALYZER_FLOOR`` and refuses the graph below it.""" + m = re.match(r"(\d+)\.(\d+)\.(\d+)", raw) if isinstance(raw, str) else None + return (int(m[1]), int(m[2]), int(m[3])) if m else None + + AppT = TypeVar("AppT") ModuleT = TypeVar("ModuleT") TypeT = TypeVar("TypeT") diff --git a/cldk/analysis/commons/backend_config.py b/cldk/analysis/commons/backend_config.py index 66c1787c..911b3582 100644 --- a/cldk/analysis/commons/backend_config.py +++ b/cldk/analysis/commons/backend_config.py @@ -77,13 +77,14 @@ class PyCodeAnalyzerConfig(CodeAnalyzerConfig): class TSCodeAnalyzerConfig(CodeAnalyzerConfig): """Select the in-process codeanalyzer backend for TypeScript. - Adds the TypeScript-only call-graph knob on top of :class:`CodeAnalyzerConfig`. + Kept distinct from :class:`CodeAnalyzerConfig` for the one knob it used to add; that knob is + now a no-op, and the class stays so existing call sites keep constructing it. Attributes: - tsc_only: If ``True``, restrict the analyzer to the tsc resolver call graph by passing - ``--tsc-only`` (codeanalyzer-typescript >= 0.4.2). Defaults to ``False`` (let the - binary choose its default). This is the supported replacement for the obsolete - ``--call-graph-provider both``. + tsc_only: **Deprecated, no-op.** codeanalyzer-typescript removed ``--tsc-only`` in 1.0.0: + the call graph always carries both resolvers, each edge tagged with its provenance + (``tsc`` / ``defuse`` / ``import``). Passing ``True`` emits a :class:`DeprecationWarning` + and changes nothing; filter edges by ``provenance`` instead. """ tsc_only: bool = False diff --git a/cldk/analysis/commons/bounds.py b/cldk/analysis/commons/bounds.py new file mode 100644 index 00000000..f32ded56 --- /dev/null +++ b/cldk/analysis/commons/bounds.py @@ -0,0 +1,357 @@ +################################################################################ +# Copyright IBM Corporation 2026 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +"""Bounds, selection and keyset paging: the language-neutral rulings every backend shares. + +Lifted out of the Python backend (leg 2.5a, G4) unchanged. Nothing here knows a language: a +``depth`` is a hop budget, a ``page_size`` is a count, a selector that names nothing is refused +the same way whether the names are Python paths or TypeScript ones. The per-language backends +import these rather than re-deriving them, which is what keeps two backends of one language -- +and the backends of two languages -- from drifting on what a keyword means. +""" + +from __future__ import annotations + +import base64 +import json +from bisect import bisect_right +from typing import Callable, Dict, List, NamedTuple, Sequence, Tuple + +from cldk.analysis.commons.results import EdgePage, SliceNode +from cldk.utils.exceptions import SelectorNotInGraph + + +def reject_bare_string(kind: str, values: object) -> None: + """Refuse a single string where a sequence of names is required. + + ``paths='pkg/mod.py'`` is not a type error to Python — a string *is* a sequence, of ten + characters — so it used to reach :func:`check_selector` as ten requested paths and come back as + ``10 of 10 paths not in graph: 'p', 'k', 'g', '/', …``. The mistake is the likely one because + the sibling keyword ``module=`` genuinely is single-valued, so both spellings look plausible. + + Raises: + TypeError: ``values`` is a ``str``. + """ + if isinstance(values, str): + raise TypeError(f"{kind}= takes a sequence of names, not a string; pass [{values!r}] to select just that one") + + +def check_selector(kind: str, requested: Sequence[str], missing: Sequence[str]) -> None: + """The one place a scoping keyword's *selection* is judged, for both backends. + + Every scoped accessor — ``get_symbol_table(paths=)``, ``get_classes(module=)``, + ``get_call_graph(roots=)`` — narrows a whole-application enumeration to what the caller named. + Two ways of naming nothing must not both come back as an empty result: + + * **an empty sequence** (``paths=[]``, ``roots=[]``) selected nothing while missing nothing. It + is a caller bug — the argument to omit is the argument that means "everything" — and it + raises the same :class:`ValueError` ``depth=`` without ``roots=`` already does. + * **values that match nothing** are the ambiguous empty the parent spec's D7 calls a defect: + a mistyped path and a module that genuinely declares no classes were the same ``{}``. They + raise :class:`~cldk.utils.exceptions.SelectorNotInGraph`, which names them and stops. It + offers no near-miss candidates on purpose — leg 1.5's E8 puts typo-tolerant matching out of + scope "not in the resolver, not in the error path". + + A **partial** miss raises too. Returning the values that did match would make a result whose + size the caller cannot check against what it asked for, which is the same silence one step + quieter. + + Args: + kind: The keyword's name, as it appears in the caller's own call — ``"paths"``, + ``"module"`` or ``"roots"``. + requested: Everything the keyword named, in the caller's spelling. + missing: The subset of ``requested`` that matched nothing. Callers with no membership + information to bring (``call_graph_scope``, which has not seen the graph yet) pass an + empty sequence and get only the empty-selection check. + + Raises: + ValueError: ``requested`` is empty. + SelectorNotInGraph: ``missing`` is non-empty. + """ + if not requested: + raise ValueError(f"{kind}= selected nothing; omit it to enumerate the whole application") + if missing: + # ``roots=`` is an exact filter, unlike every name-taking accessor on this surface, so a + # correct short name and a typo miss the same way -- the message has to say which + # vocabulary it wanted (see the assessment on PythonAnalysisBackend.get_call_graph). + detail = ( + "roots= takes full signatures (as get_callables_overview() reports them) or @external ids, not bare names; " + "to address a callable by name use resolve_callable(name).callable, or backward_cone / callers_of / call_paths_between" + if kind == "roots" + else None + ) + raise SelectorNotInGraph(kind, list(missing), len(requested), detail=detail) + + +def check_depth(depth: int | None) -> int | None: + """``depth`` is a hop budget: ``None`` for unbounded, otherwise an ``int`` of at least 1. + + Type-checked and not merely range-checked, because the two ways of getting it wrong are silent + otherwise: ``depth="2"`` raised ``TypeError`` from somewhere further in, and ``depth=2.5`` was + accepted and truncated to 2 by the Cypher/ego-graph radius. ``bool`` is rejected for the same + reason — ``depth=True`` is ``1`` by accident. + + One function, so ``get_call_graph``, the slices and the reachability accessors cannot come to + disagree about what a hop budget is. + """ + if depth is not None and (not isinstance(depth, int) or isinstance(depth, bool) or depth < 1): + raise ValueError(f"depth must be an int >= 1, got {depth!r}") + return depth + + +# ---------------------------------------------------------------------------------------------- +# Paging the per-callable graphs (E5). +# +# THE CANONICAL ORDER, defined once here because it is the only thing that makes a page mean the +# same thing on both backends. Neo4j returns rows in no order unless told to, and the local +# backend returns the analyzer's emission order; without one stated sort, page two on Neo4j is a +# different set of edges from page two locally. Each backend uses these functions -- the local one +# sorts and slices with them directly, the Neo4j one writes the same components into its ORDER BY +# and rebuilds the cursor from them -- so a change here moves both at once. +# +# The order is over the edge's OWN fields, in the order a reader would name them: source, then +# target, then whatever else the edge carries. Nothing positional and nothing backend-specific +# (no relationship element id, no row number), because a key one backend cannot compute is not a +# shared order. +# +# TOTALITY. A keyset cursor resumes strictly *after* a key, so a repeated key would drop its twin. +# The full field tuple is unique on real data: measured across odoo-slim-19's 5,134,655 PY_DDG, +# 247,906 PY_CFG_NEXT and 139,065 PY_CDG edges, zero (src, dst, ...) tuples repeat -- and for CFG +# the endpoints alone are *not* enough (13,310 node pairs carry two edges of different ``kind``), +# which is why ``kind`` is in the key. On the graph side the emitter MERGEs these relationships on +# exactly these properties, so uniqueness is structural there rather than incidental. Two edges +# equal in every field would be equal as values -- the models carry nothing else -- so their +# relative order is unobservable, and the page boundary is the same either way. +# +# ``or ""`` / ``or []`` is not cosmetic: ``DdgEdge.var`` is ``Optional[str]``, and a ``None`` in a +# sort key raises in Python and silently drops the row in Cypher (``null > x`` is null). The +# Cypher spells the same normalisation with ``coalesce``. + + +#: Edges per page when the caller does not say. 10,000 is where the measured distribution +#: splits: on odoo-slim-19, 15,520 of the 15,549 callables have fewer than 10,000 DDG edges, so +#: this default answers 99.8% of callables completely in one page and no caller of a normal +#: callable ever writes a loop -- while the 29 that are larger, up to 1,386,918 edges, are held to +#: a response a caller can actually hold. CFG and CDG max out at 402 and 314 edges on the same +#: application, so for them it is never reached. +DEFAULT_PAGE_SIZE = 10_000 + + +class EdgeOrder(NamedTuple): + """One edge kind's canonical order, in both spellings that have to agree. + + The Python sort key and the Cypher expressions are the same components said twice, in two + languages, and the whole point of the order is that the two never disagree — so they are + written down once, together, and each backend takes the half it can run. ``len(exprs)`` is + also the order's arity, which is how a cursor from one accessor is refused by another + (:func:`decode_cursor`): the three arities are 3, 2 and 4. + + ``coalesce`` in the expressions is ``or ""`` / ``or []`` in the key: ``DdgEdge.var`` is + optional, and a ``None`` in a sort key raises in Python and silently drops the row in Cypher. + """ + + key: Callable[[object], Tuple] + exprs: Tuple[str, ...] + + +def encode_cursor(scope: str, key: Tuple) -> str: + """An opaque, round-trippable spelling of a sort key, stamped with the callable it came from. + + Opaque on purpose: the caller passes it back and never reads it, so the components of the + order stay an implementation detail rather than joining the caller's vocabulary. Base64 of + JSON, because the key holds strings and a list of strings, and both survive that unchanged. + + ``scope`` is the resolved callable signature, carried so that :func:`decode_cursor` can refuse + a cursor minted for a different callable. Without it, an agent looping over callables and + reusing the wrong ``next_cursor`` would get a plausible page of the *right* callable's edges + resumed from a position in the *wrong* one — silently, since body-node ids sort by callable id + and the filter would simply skip everything or nothing. + """ + return base64.urlsafe_b64encode(json.dumps([scope, list(key)]).encode("utf-8")).decode("ascii") + + +def decode_cursor(cursor: str, scope: str, arity: int) -> Tuple: + """Inverse of :func:`encode_cursor`, checked against the caller it is being used for. + + Three ways a cursor can be wrong, all of them raising rather than being read as "start from + the beginning" — which would silently hand back page one when page nine was asked for: + it does not decode; it was minted for another callable; or it has the wrong number of + components, which is what a cursor from a *different accessor* looks like (the three orders + have arities 3, 2 and 4, so no cursor is silently valid for the wrong graph). + """ + try: + got_scope, key = json.loads(base64.urlsafe_b64decode(cursor.encode("ascii")).decode("utf-8")) + except Exception as exc: # noqa: BLE001 -- any decode failure is the same caller error + raise ValueError(f"not a cursor from a previous page: {cursor!r}") from exc + if got_scope != scope: + raise ValueError(f"this cursor is from a page of {got_scope!r}, not {scope!r}") + if len(key) != arity: + raise ValueError(f"cursor has {len(key)} components, this accessor's order has {arity}: {cursor!r}") + return tuple(key) + + +def check_page_size(page_size: int) -> int: + """``page_size`` must ask for at least one edge. + + Zero is refused rather than treated as "no limit": a page of nothing whose ``next_cursor`` can + never advance is an infinite loop dressed as an empty answer. + """ + if page_size < 1: + raise ValueError(f"page_size must be at least 1, got {page_size}") + return page_size + + +def keyset_where(exprs: Sequence[str]) -> str: + """The Cypher for "strictly after the cursor", written out because Cypher has no tuple + comparison: ``(a, b) > ($c0, $c1)`` has to become + ``a > $c0 OR (a = $c0 AND (b > $c1))``. + + Keyset rather than ``SKIP``: measured on ``Website.configurator_apply`` (1,386,918 DDG edges, + 10,000 per page, query alone), ``SKIP`` costs 2.6s for the first page, 9.0s for the middle one + and 4.3s for the last -- it re-sorts a prefix that grows with the offset -- while this filter + is flat at 3.1s / 2.9s / 2.4s. The offset form is not wrong, it just gets worse the further in + the caller reads, which is the one direction pagination exists to make cheap. + """ + clause = "" + for i in reversed(range(len(exprs))): + expr, param = exprs[i], f"$c{i}" + clause = f"{expr} > {param}" + (f" OR ({expr} = {param} AND ({clause}))" if clause else "") + return clause + + +def cursor_params(cursor: str, scope: str, arity: int) -> Dict[str, object]: + """The ``$c0…$cN`` bindings :func:`keyset_where` reads, from an opaque cursor.""" + return {f"c{i}": v for i, v in enumerate(decode_cursor(cursor, scope, arity))} + + +def edge_page(model, scope: str, edges: List, order: EdgeOrder, page_size: int, cursor: str | None) -> EdgePage: + """One page of an edge set already held in memory. + + The local backend has every edge in hand, so it sorts by ``key`` and slices. The cursor is + resolved by binary search over the sorted keys -- ``bisect_right``, i.e. the first edge + strictly after it -- so it means exactly what :func:`keyset_where` makes it mean on the graph, + rather than an independently-invented position that happens to line up. + """ + check_page_size(page_size) + key = order.key + rows = sorted(edges, key=key) + start = bisect_right([key(e) for e in rows], decode_cursor(cursor, scope, len(order.exprs))) if cursor is not None else 0 + window = rows[start : start + page_size] + more = start + len(window) < len(rows) + return EdgePage[model](edges=window, total=len(rows), next_cursor=encode_cursor(scope, key(window[-1])) if more and window else None) + + +#: Paths per query when the caller does not say. A path list is a set of *witnesses* for a flow, +#: not the flow's extent, and ten worked examples is already more than a reader will follow; the +#: extent question is ``slice_forward``, which reports a ``total``. +DEFAULT_MAX_PATHS = 10 + + +def check_max_paths(max_paths: int) -> int: + """``max_paths`` must admit at least one path. Zero is refused for :func:`check_max_nodes`'s + reason: an empty list whose ``truncated`` says "there were more" answers nothing, and it is + indistinguishable at a glance from "there is no flow".""" + if max_paths < 1: + raise ValueError(f"max_paths must be at least 1, got {max_paths}") + return max_paths + + +def check_distinct_endpoints(src: SliceNode, dst: SliceNode) -> None: + """A path query must have two different endpoints. + + Neo4j's shortest-path search *refuses* a self-question outright ("the shortest path algorithm + does not work when the start and end nodes are the same"), which would otherwise surface as a + raw driver error from one backend and an empty list from the other. Both raise here instead, + and neither answers ``[]``: for a node that genuinely sits on a cycle, ``[]`` would be + indistinguishable from a proved absence of one, which is the ambiguous empty in another + costume. ``reaches(x, x)`` is the accessor that answers the existence question, and it does + terminate (measured: 0.03s, where the obvious ``EXISTS`` spelling never finished). + + Takes the *resolved* endpoints rather than their refs so the message speaks the caller's + vocabulary (E6/E7): a value is named ``'kwargs' within '….configurator_apply'``, a callable + by its signature, and the advice is a call that actually runs -- ``reaches`` takes callable + names, so for a value the cycle question is asked of its enclosing callable. + """ + if src.ref != dst.ref: + return + if src.kind == "callable": + raise ValueError(f"paths from {src.callable!r} to itself are not answered; ask reaches({src.callable!r}, {src.callable!r}) whether a cycle exists") + raise ValueError( + f"paths from {src.name!r} to itself (within {src.callable!r}) are not answered; a value reaches itself only through " + f"recursion, so ask reaches({src.callable!r}, {src.callable!r}) whether the callable is on a call cycle" + ) + + +#: Nodes per slice when the caller does not say. The same 10,000 as :data:`DEFAULT_PAGE_SIZE`, and +#: for a different reason: there, it is where 99.8% of callables fit in one page; here, nothing +#: fits, because the measured distribution has no middle (see +#: :class:`~cldk.analysis.commons.results.Slice`). 10,000 is the largest result that stays +#: readable, and every slice above it is one a caller should be re-asking with ``depth=``. +DEFAULT_MAX_NODES = 10_000 + +#: Hops from the seed when the caller does not say. **Finite, and that is the whole point.** +#: +#: The measured distribution has no middle (see :class:`~cldk.analysis.commons.results.Slice`), so +#: an unbounded default hands a connected seed 10,000 arbitrary nodes of a 195,819-node closure -- +#: an unprincipled 5%, honestly flagged ``truncated`` and useless either way. A finite default +#: answers a *narrower* question *completely* instead, and ``depth=None`` is how a caller asks for +#: the whole cone. +#: +#: 5 is the largest bound at which no measured slice needs ``max_nodes`` at all. Over 120 random +#: ``formal_in`` seeds with callers on odoo-slim-19, node counts by depth: +#: +#: ========= ===== ======= ======= ======= ======== +#: direction depth median p75 max > 10,000 +#: ========= ===== ======= ======= ======= ======== +#: backward 3 14 70 846 0 +#: backward 5 33 188 1,539 0 +#: backward 6 56 324 2,818 0 +#: backward 8 464 2,044 16,028 1 +#: backward None 195,786 195,787 198,306 79 +#: forward 3 12 34 440 0 +#: forward 5 24 63 1,053 0 +#: forward 6 35 166 14,260 1 +#: forward 8 48 402 37,326 2 +#: forward None 71 440,269 440,645 52 +#: ========= ===== ======= ======= ======= ======== +#: +#: 3 is informative but thin; 6 is where a forward slice first exceeds the cap and the default +#: would start truncating again. 5 is the last depth that never does, in either direction. +#: +#: **Which accessors take it, and which deliberately do not.** The three *slices* +#: (``slice_backward``, ``slice_forward``, ``backward_cone``) default to it: a bounded slice is a +#: *complete* answer to a narrower question, and ``total`` says so. The two *predicates* +#: (``reaches``, ``flows_to_call``, ``flows_to_argument``) and the two *path* queries +#: (``paths_between``, ``call_paths_between``) default to ``None`` -- unbounded -- because a hop +#: budget on a boolean or a path list is not a smaller answer but a **wrong** one: "no flow" and +#: "no flow within five hops" collapse into the same ``False`` / ``[]`` with nothing in the result +#: to tell them apart. Measured on odoo-slim-19: ``flows_to_call("kwargs", "Website.create", +#: within="Website.configurator_apply")`` is ``False`` at five hops and ``True`` unbounded, and +#: the matching ``paths_between`` is ``[]`` at five hops and ten paths at eight. ``depth=`` stays +#: on all five as an explicit narrowing a caller can name; it is only the *default* that differs. +DEFAULT_DEPTH = 5 + + +def check_max_nodes(max_nodes: int) -> int: + """``max_nodes`` must admit at least one node — the seed, if nothing else. + + Zero is refused rather than read as "no limit": a slice of nothing whose ``total`` says + 195,784 is a result no caller can act on, and "unbounded" is what ``max_nodes=None`` would + have to mean if it ever meant anything. + """ + if max_nodes < 1: + raise ValueError(f"max_nodes must be at least 1, got {max_nodes}") + return max_nodes diff --git a/cldk/analysis/commons/graphs.py b/cldk/analysis/commons/graphs.py new file mode 100644 index 00000000..8b6d392c --- /dev/null +++ b/cldk/analysis/commons/graphs.py @@ -0,0 +1,311 @@ +################################################################################ +# Copyright IBM Corporation 2026 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +"""Graph walks and path assembly: the language-neutral half of slicing and reachability. + +Lifted out of the Python backend (leg 2.5a, G4) unchanged except for three parameters: the +relationship-type prefix (``sdg_rels(P)`` / ``via_table(P)`` where the Python backend had +``PY_``-spelled tables) and the ``via`` map :func:`flow_path` translates through. The per-language +backend binds each once and hands the bound object down. +""" + +from __future__ import annotations + +from typing import Callable, Iterable, List, Literal, Mapping, Sequence, Tuple + +import networkx as nx + +from cldk.analysis.commons.bounds import check_selector, reject_bare_string +from cldk.analysis.commons.results import FlowPath, PathHop, SliceNode + + +def bounded_subgraph(graph: nx.DiGraph, roots: List[str], depth: int | None, declared: Iterable[str]) -> nx.DiGraph: + """The sub-call-graph reachable from ``roots``, within ``depth`` hops when given. + + **Induced**, not path-only: every edge between two reached nodes is kept, including one + pointing back towards a root. A path-only answer would let ``graph.predecessors(n)`` lie about + a node the caller can see, which is a worse defect than the extra edges are a cost. The Neo4j + backend's Cypher is written to produce the same induced shape rather than the cheaper + edges-along-the-path shape, for exactly this reason. + + **The domain a root is judged against — stated here because both backends must judge against + the same one — is the callable inventory, not this graph.** ``graph`` is built from call + *edges* alone, so a callable that neither calls nor is called by anything is not a node in it: + 444 of the live odoo application's 15,549 in-scope callables, 2.9%. Checking membership of + ``graph`` therefore raised for a callable that plainly exists, while the Neo4j backend — whose + Cypher matches a root by node *label*, not by edge participation — returned the one-node graph + it is. ``declared`` closes that gap: it carries every callable the application declares, and a + root is valid when it is **in the inventory or is a node of the graph**. The second disjunct is + not redundant — an ``@external`` ghost is a legitimate root, is a graph node, and is not a + declared callable — and the union is exactly what the Neo4j root match accepts (a + ``:PyCallable`` of this application, or a ``:PyExternal``). + + A root outside that domain raises (:func:`check_selector`) rather than contributing nothing: + "no such callable" and "a callable that calls nothing" are different answers, and before this + they were the same empty graph. + + The returned graph stays **edge-induced**. An isolated root is added back as a lone node — + which is the answer, and the one Neo4j gives — but nothing else the inventory knows about is + seeded into it. Seeding all declared callables would make the unbounded local graph disagree + with Neo4j's node-for-node, trading one parity defect for a larger one. + """ + inventory = set(declared) + check_selector("roots", roots, [r for r in roots if r not in graph and r not in inventory]) + nodes: set = set() + isolated: set = set() + for root in roots: + if root not in graph: + isolated.add(root) # declared, but in no call edge: its own one-node graph + elif depth is None: + nodes |= nx.descendants(graph, root) | {root} + else: + nodes |= set(nx.ego_graph(graph, root, radius=depth).nodes) + sub = graph.subgraph(nodes).copy() + sub.add_nodes_from(isolated) + return sub + + +# The structural half of the per-callable graph orders. The components and their sequence are +# what make a page mean the same thing on both backends (see the paging block in ``bounds``); the +# per-language backend binds each to its own edge model -- ``cfg_sort_key(edge: CfgEdge)`` in the +# Python backend -- so the typed name a reader greps for stays where the type lives. +_EDGE_KEYS: dict[str, Callable[[object], Tuple]] = { + "cfg": lambda e: (e.src, e.dst, e.kind or ""), + "cdg": lambda e: (e.src, e.dst), + "ddg": lambda e: (e.src, e.dst, e.var or "", list(e.prov or [])), +} + + +def edge_sort_key(kind: Literal["cfg", "cdg", "ddg"]) -> Callable[[object], Tuple]: + """The canonical sort key for one per-callable graph kind, over the edge's own fields. + + ``cfg``: source, target, kind. ``cdg``: source, target. ``ddg``: source, target, variable, + provenance. ``or ""`` / ``or []`` because an optional field's ``None`` in a sort key raises in + Python and silently drops the row in Cypher; the Cypher spells it ``coalesce``. + """ + return _EDGE_KEYS[kind] + + +# ---------------------------------------------------------------------------------------------- +# Slicing and reachability (E2, E3, E5). +# +# THE FIVE RELATIONSHIP TYPES A SLICE FOLLOWS, verified against codeanalyzer's own +# ``neo4j/schema.py`` REL_TYPES and against ``CALL db.relationshipTypes()`` on odoo-slim-19 rather +# than copied from a plan -- the names in this leg's plan have been wrong before (PY_CFG_NEXT is +# not PY_CFG). All five exist, with these edge counts on that application: +# +# PY_DDG 5,134,655 data dependence, within a callable (var, prov) +# PY_CDG 139,065 control dependence, within a callable +# PY_PARAM_IN 229,035 actual_in -> formal_in : an argument entering a callee +# PY_PARAM_OUT 133,267 formal_out -> actual_out : a value coming back to the caller +# PY_SUMMARY 453,398 actual_in -> actual_out : a callee's pass-through, at the call site +# +# All five point WITH the flow -- verified on the live graph, where every PY_PARAM_IN runs +# actual_in -> formal_in and every PY_PARAM_OUT runs formal_out -> actual_out, with no exceptions +# in 362,302 edges. So a forward slice follows them and a backward slice follows them reversed; +# there is no per-type direction table to keep straight, which is why they can share one match. +# +# PY_CFG_NEXT is deliberately NOT here. Control *flow* says what runs next; a slice is about what +# a value or a decision depends on, and following successor edges would pull in every later +# statement whether or not it depends on anything -- the "returns the whole callable" bug that a +# non-emptiness assertion cannot catch. +# +# The table is a function of the backend's relationship prefix (``AnalysisBackend.P``): the five +# kinds and their meaning are the analyzer family's, the ``PY_`` / ``TS_`` spelling is one language's. + + +def sdg_rels(P: str) -> tuple[str, ...]: + """The five relationship types a slice follows, spelled with the language's prefix ``P``.""" + return (f"{P}_DDG", f"{P}_CDG", f"{P}_PARAM_IN", f"{P}_PARAM_OUT", f"{P}_SUMMARY") + + +def sdg_rel_pattern(P: str) -> str: + """The Cypher spelling of :func:`sdg_rels` for a relationship-type disjunction.""" + return "|".join(sdg_rels(P)) + + +#: The caller's word for each relationship a path hop can be justified by (E6). The graph's own +#: ``PY_DDG``/``PY_PARAM_IN`` spelling never leaves the backend; both backends translate through +#: this one table so a hop cannot be labelled ``data`` over Neo4j and ``ddg`` locally. +#: +#: ``argument`` and ``return`` are the two interprocedural edges, and they are deliberately not +#: both called "parameter": ``PY_PARAM_IN`` binds a caller's argument to a callee's formal, and +#: ``PY_PARAM_OUT`` binds a callee's result back into the caller. A reader following a path needs +#: to know which way it just crossed a call boundary. +def via_table(P: str) -> dict[str, str]: + """The relationship-type -> caller's-word table above, for the language whose prefix is ``P``.""" + return { + f"{P}_DDG": "data", + f"{P}_CDG": "control", + f"{P}_PARAM_IN": "argument", + f"{P}_PARAM_OUT": "return", + f"{P}_SUMMARY": "summary", + f"{P}_CALLS": "call", + } + + +def hop_sort_key(hops: Sequence[PathHop]) -> Tuple: + """The order two paths are compared in, in the caller's *own* vocabulary. + + E2 makes a path a sequence, which only means something if the *list* of paths is stable too: + ``max_paths`` truncates, and a truncation of a non-deterministic order is not reproducible. + So paths are ordered shortest first, then hop by hop on ``(via, var, to.ref)`` — every term of + which the caller can see in the result it gets back. + + Two hops that are indistinguishable in that vocabulary (parallel edges of the same kind, on + the same variable, between the same two nodes) are left to a backend-local tie-break: the + Neo4j backend appends the relationship's ``elementId``, the local backend keeps the order the + analyzer emitted them in. Either is stable for repeated calls against one graph; neither is + meaningful to a caller, which is why it is last and why nothing above depends on it. + """ + return (len(hops), tuple((h.via, h.var or "", h.to.ref) for h in hops)) + + +def flow_path(nodes: Sequence[SliceNode], edges: Sequence[Tuple[str, "str | None", "Sequence[str] | None"]], *, via: Mapping[str, str]) -> FlowPath: + """Join a walk's ``n`` nodes and its ``n - 1`` edges into a :class:`FlowPath`. + + Both backends build paths through here, which is what makes the joining invariant + (``hops[i].to is hops[i + 1].frm``) a property of the construction rather than something each + backend has to be trusted to preserve. ``edges`` are the graph's own relationship types; they + are translated to the caller's word through ``via`` (the backend's :func:`via_table`) exactly once, here. + + Raises: + KeyError: A relationship type with no word in ``via`` — a new edge kind from a future + analyzer generation, which must be named before it can be reported rather than passed + through in the graph's spelling. + """ + return FlowPath(hops=[PathHop(frm=nodes[i], to=nodes[i + 1], via=via[rel], var=var, prov=list(prov or [])) for i, (rel, var, prov) in enumerate(edges)]) + + +def shortest_walks(edges: Mapping[str, Mapping[str, Sequence[tuple]]], src: str, dst: str, depth: int | None, limit: int, *, via: Mapping[str, str]) -> List[list]: + """Up to ``limit`` shortest ``src``->``dst`` walks over ``edges``, in the documented order. + + The local backends' twin of the graph's ``allShortestPaths``, and only shortest walks for its + reason: enumerating every walk does not terminate on a real dependence graph. + + ``edges`` is the ``{src: {dst: [label]}}`` adjacency each local backend builds, where a label is + ``(relationship type, var, prov)``. A **list** per ``(src, dst)`` pair because parallel edges are + ordinary -- one statement feeding one argument on several variables is several distinct paths -- + and collapsing them would merge several pieces of evidence into one. + + Two passes. The first is a breadth-first level walk keeping the hop count each node was *first* + reached at; the second is a depth-first replay that only ever steps to a node whose recorded + distance is exactly one more than the walk so far, so it visits shortest walks and nothing else. + + The replay's branch order is ``(via, var, to)`` -- exactly the per-hop key + :func:`hop_sort_key` documents -- and every walk found has the same length, so a pre-order + depth-first traversal emits them already sorted. That is what makes ``limit`` a *prefix* of a + total order rather than whichever ``limit`` walks the recursion happened to find first. ``via`` + is the backend's :func:`via_table`, so the branch order is the caller's vocabulary and not the + graph's relationship-type spelling. + + Lifted out of ``PyCodeanalyzer`` (leg 2.5b) unchanged except for the ``via`` parameter: it is + a graph algorithm over an adjacency of strings and knows no language, and a second copy would be + a second place for the two backends of a language -- or the backends of two languages -- to + drift on what "shortest, in order" means. + """ + dist, frontier, hops = {src: 0}, [src], 0 + while frontier and dst not in dist and (depth is None or hops < depth): + hops += 1 + nxt = [] + for s in frontier: + for d in edges.get(s, ()): + if d not in dist: + dist[d] = hops + nxt.append(d) + frontier = nxt + if dst not in dist or dist[dst] == 0: + return [] + target, out = dist[dst], [] + + def walk(node: str, walked: list) -> None: + if len(walked) == target: + if node == dst: + out.append(list(walked)) + return + options = sorted((via[rel], var or "", d, (rel, var, prov)) for d, labels in edges.get(node, {}).items() if dist.get(d) == len(walked) + 1 for rel, var, prov in labels) + for _, _, d, label in options: + walked.append((d, label)) + walk(d, walked) + walked.pop() + if len(out) >= limit: + return + + walk(src, []) + return out + + +def as_slice_node(node: object) -> SliceNode: + """The :class:`~cldk.analysis.commons.results.SliceNode` for anything carrying an address. + + :meth:`PythonAnalysisBackend.describe` takes "anything with a ``ref``" — slice nodes, the + endpoints of a :class:`~cldk.analysis.commons.results.PathHop`, a + :class:`~cldk.analysis.commons.results.LocateResult` — because the addressing layer hands a + caller three shapes and asking them to convert between shapes to hydrate one is the kind of + friction that gets worked around with string surgery. + + A ``SliceNode`` passes through untouched. A ``LocateResult`` is re-expressed as one, keeping + the vocabulary it already speaks: ``module.path`` is the file, ``callable.signature`` the + enclosing callable, ``body.kind`` the position's kind. + + Raises: + TypeError: ``node`` carries neither a ``ref`` nor a ``node_id``, so there is nothing to + look up. Guessing an address from a file and a line is what ``locate`` is for. + """ + if isinstance(node, SliceNode): + return node + ref = getattr(node, "node_id", None) + if ref is None: + raise TypeError(f"describe() needs something carrying a ref (a SliceNode, a path hop endpoint, a locate() result); got {type(node).__name__}") + module, callable_ref, body = node.module, node.callable, getattr(node, "body", None) + return SliceNode( + file=module.path, + line=node.span.start[0], + callable=callable_ref.signature if callable_ref else "", + kind=body.kind if body else "callable", + name=callable_ref.name if callable_ref else None, + source=node.source or None, + ref=ref, + ) + + +def cone_sinks(resolve: Callable[[str], SliceNode], sinks: Sequence[str]) -> List[SliceNode]: + """Resolve ``backward_cone``'s sinks, refusing the two ways of naming nothing. + + The same discipline :func:`check_selector` applies to ``roots=`` and ``paths=``: a bare string + is ten one-character sinks and is refused as a type error, and an empty sequence is refused + because "everything" is the argument omitted, not the argument emptied — and there is no + "everything" here to fall back to. Each surviving name goes through ``resolve``, so an + ambiguous sink raises listing candidates instead of one of them being picked. + + Duplicates are collapsed by resolved signature, not by the string the caller wrote: naming the + same callable twice, once bare and once qualified, is one sink. + """ + reject_bare_string("sinks", sinks) + if not sinks: + raise ValueError("sinks= names nothing to walk back from; pass at least one callable") + resolved = {node.callable: node for node in (resolve(s) for s in sinks)} + return list(resolved.values()) + + +def slice_resolved(roots: List[SliceNode]) -> str: + """The audit line on a :class:`~cldk.analysis.commons.results.Slice`: what the caller's names + matched, in the caller's vocabulary. + + Both backends build it here rather than each formatting its own, so a caller comparing two + results is comparing answers and not two spellings of one. + """ + return ", ".join(f"{r.callable} {r.kind} {r.name!r}" if r.kind != "callable" else r.callable for r in roots) diff --git a/cldk/analysis/commons/keys.py b/cldk/analysis/commons/keys.py new file mode 100644 index 00000000..518616a6 --- /dev/null +++ b/cldk/analysis/commons/keys.py @@ -0,0 +1,184 @@ +################################################################################ +# Copyright IBM Corporation 2026 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +"""Module keys and scoping: how a caller's path names a module, in any language. + +Lifted out of the Python backend and its Neo4j reconstruction (leg 2.5a, G4) unchanged except +that :func:`module_dotted` takes the language's source extensions as a parameter. Keys are the +repo-relative paths the analyzer saw; every ruling here is about matching a caller's spelling of +one to the key that exists, or refusing. +""" + +from __future__ import annotations + +import os +import posixpath +from typing import Collection, Iterable, List, Sequence + +from cldk.analysis.commons.bounds import check_depth, check_selector, reject_bare_string + + +def resolve_module_key(path: str, keys: Iterable[str]) -> str: + """The symbol-table / graph ``file_key`` naming ``path``, or ``path`` unchanged if none does. + + A caller of :meth:`PythonAnalysisBackend.locate` hands over whatever its scanner printed — + ``./src/app.py``, ``src/../src/app.py``, or an absolute path from the machine the scan ran on — + while both backends are keyed by the project-relative path the analyzer saw. Exact key first, + then the normalised form, then the longest known key the normalised path *ends on a segment + boundary* of (which is what an absolute path is). Returning ``path`` unchanged when nothing + matches is deliberate: the caller then gets ``file_not_in_graph`` naming the path it asked + about, not a silently substituted neighbour. + """ + keys = list(keys) + if path in keys: + return path + norm = posixpath.normpath(str(path).replace(os.sep, "/")) + if norm in keys: + return norm + suffix_matches = [k for k in keys if norm.endswith("/" + k)] + return max(suffix_matches, key=len) if suffix_matches else path + + +def scope_paths(paths: Sequence[str] | None, keys: Iterable[str], kind: str = "paths") -> List[str] | None: + """Resolve requested module paths to symbol-table keys, or ``None`` for "the whole application". + + Both backends route their ``paths=`` / ``module=`` keywords through here, so the lenient + resolution (:func:`resolve_module_key` — an absolute path or one with native separators finds + its module) and the strictness (:func:`check_selector` — a path naming no module raises) cannot + drift apart between them. + + Args: + paths: What the caller named, or ``None`` for the unscoped call. + keys: The symbol-table keys that exist — ``symbol_table.keys()`` locally, the + application's module ``file_key``s over Neo4j. + kind: The keyword's name for the error message; ``"module"`` for ``get_classes``, whose + single-valued keyword routes through here as a one-element sequence. + + **Resolution is many-to-one, and the result is de-duplicated.** Leniency is the whole point of + :func:`resolve_module_key` — ``"pkg/a.py"`` and ``"/abs/pkg/a.py"`` are two spellings a scanner + may plausibly hand over for the *same* module — so two requested paths legitimately collapse to + one key and the caller gets one entry back. Raising on the collapse would punish the very + caller the leniency exists for; de-duplicating explicitly is what keeps the returned list from + naming the same module twice and asking both backends to fetch it twice. + + Raises: + TypeError: ``paths`` is a bare string (see :func:`reject_bare_string`). + ValueError: ``paths`` is an empty sequence. + SelectorNotInGraph: a path names no module in this application. + """ + reject_bare_string(kind, paths) + if paths is None: + return None + known = list(keys) + resolved = [resolve_module_key(p, known) for p in paths] + check_selector(kind, list(paths), [p for p, r in zip(paths, resolved) if r not in known]) + return list(dict.fromkeys(resolved)) + + +def call_graph_scope(roots: Sequence[str] | None, depth: int | None) -> List[str] | None: + """Normalise :meth:`PythonAnalysisBackend.get_call_graph`'s scoping keywords. + + Returns the roots as a list, or ``None`` for "the whole application" — the unscoped call, + which must keep behaving exactly as it did before the keywords existed. + + Both backends route through this so the two cannot drift apart on what a keyword combination + means (the failure mode Fix 1 of leg 1.5 had to go back and repair on the child-fetch paths). + Whether each root *exists* is checked later, by whichever backend has the graph in hand, but + through the same :func:`check_selector` — see :func:`bounded_subgraph`. + + Raises: + TypeError: ``roots`` is a bare string (see :func:`reject_bare_string`). + ValueError: ``depth`` that is not a positive ``int``, ``depth`` without ``roots``, or an + empty ``roots``. A hop budget with no origin to count from has no meaning, and quietly + returning all 364,752 edges would be the worst of the available answers — the caller + asked for a bounded graph and would be handed an unbounded one with no signal. + ``depth`` is type-checked rather than merely range-checked because the two ways of + getting it wrong are silent otherwise: ``depth="2"`` raised ``TypeError`` from the + comparison, and ``depth=2.5`` was accepted and truncated to 2 by the Cypher/ego-graph + radius. ``bool`` is rejected for the same reason — ``depth=True`` is ``1`` by accident. + """ + check_depth(depth) + reject_bare_string("roots", roots) + if roots is None: + if depth is not None: + raise ValueError("depth= requires roots=; a hop budget needs an origin to count from") + return None + check_selector("roots", list(roots), ()) + return list(roots) + + +def module_key_of(node_id: str, prefix: str, known: Collection[str]) -> str: + """The repo-relative module key embedded in a ``can://`` id (F4). + + Ids are ``/`` (or exactly ```` for a module), and a + file key can itself contain ``.py/`` as a directory name, so the key is never recovered by + splitting: every ``/``-boundary prefix of the id is tried longest first and the first that is + a member of ``known`` -- the application's verified module keys -- wins. A miss raises: a key + we cannot verify is a defect, not a guess. ``known`` should be a set; this runs once per row. + """ + if not node_id.startswith(prefix): + raise KeyError(node_id) + parts = node_id[len(prefix) :].split("/") + for n in range(len(parts), 0, -1): + candidate = "/".join(parts[:n]) + if candidate in known: + return candidate + raise KeyError(node_id) + + +def module_dotted(path: str, *, extensions: Sequence[str] = (".py",), package_index: str | None = "__init__") -> str: + """The dotted module name a repo-relative path spells: ``"odoo/tools/mail.py"`` → + ``"odoo.tools.mail"``, ``"pkg/__init__.py"`` → ``"pkg"``. The same derivation the analyzer's + signatures embody, so ``in_module=`` can be written the way a signature reads. ``extensions`` is + the language's source suffixes; the Python default keeps every existing call site as it was. + + ``package_index`` is the file name a language addresses by its *package* name -- Python's + ``__init__``, stripped from the stem's tail. It is a parameter rather than an unconditional + step because it is a language convention and nothing stops another language having a file of + that name: a TypeScript ``src/foo/__init__.ts`` is a module in its own right and must dot to + ``src.foo.__init__``, so :mod:`cldk.analysis.typescript` passes ``package_index=None``. Passing + a name TypeScript *does* use (``"index"``) would be a different ruling and is deliberately not + made here -- ``src/foo/index.ts`` dots to ``src.foo.index``, because that is the module key it + is addressed by everywhere else on the surface.""" + stem = next((path[: -len(ext)] for ext in extensions if path.endswith(ext)), path) + if package_index and stem.endswith("/" + package_index): + stem = stem[: -len(package_index) - 1] + return stem.replace("/", ".") + + +def body_key_column(key: str) -> int: + """The start column encoded in a body node's local key (``"21:12"`` -> ``12``), or ``-1``. + + Both backends of a language need one tie-break for two body nodes that span the *same* line -- + ``if x: return x`` emits an ``if`` and a ``return`` each spanning one line -- and line numbers + are the only positional data a Neo4j projection carries, so the span cannot break it. The local + *key* can: it is ``:`` (sometimes suffixed, as in ``"22:8/actual_in:0"``), it exists + on both sides (locally the ``body`` dict key, over Neo4j the trailing segment of + ``@``), and a larger column is the more deeply nested statement. Comparing the + keys as *strings* instead would order ``"29:10"`` before ``"29:4"`` and pick the outer node, so + the column is parsed as an int. + + ``-1`` for a key with no column (the synthetic ``@entry`` / ``@exit`` vertices). In Python those + carry no span and are filtered out before ranking; in TypeScript they span the whole callable, + so they lose on width to anything nested inside them and reach this only in a one-line callable, + where ranking last is the right answer -- a statement inside the callable is the more precise + position. + + Lifted out of ``cldk/analysis/python/backend.py`` unchanged (leg 2.5b): it reads a key grammar + both analyzers emit, and a second copy is a second thing to keep in step. + """ + _, _, col = key.split("/", 1)[0].partition(":") + return int(col) if col.isdigit() else -1 diff --git a/cldk/analysis/commons/levels.py b/cldk/analysis/commons/levels.py new file mode 100644 index 00000000..0eb4de40 --- /dev/null +++ b/cldk/analysis/commons/levels.py @@ -0,0 +1,56 @@ +################################################################################ +# Copyright IBM Corporation 2026 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + + +"""The SDK's :class:`~cldk.analysis.AnalysisLevel` names mapped to the analyzers' integer levels. + +Every codeanalyzer takes ``-a 1..4`` (symbol table, call graph, intraprocedural dataflow, +interprocedural SDG); the SDK names those levels once, here, so no backend sends the enum's +display string or caps the level on the caller's behalf. +""" + +from __future__ import annotations + +from cldk.analysis import AnalysisLevel + +#: The analyzer's ``-a`` integer for each SDK level. +ANALYZER_LEVELS = { + AnalysisLevel.symbol_table: 1, + AnalysisLevel.call_graph: 2, + AnalysisLevel.program_dependency_graph: 3, + AnalysisLevel.system_dependency_graph: 4, +} + +#: The inverse, by the member name a caller writes (``"call_graph"``, not ``"call graph"``) — so +#: an error about the level in use names it the way it was asked for. +LEVEL_NAMES = {n: lvl.name for lvl, n in ANALYZER_LEVELS.items()} + + +def analyzer_level(level: "AnalysisLevel | str") -> int: + """The analyzer's integer level for one of the SDK's :class:`~cldk.analysis.AnalysisLevel` + names. + + Accepts the enum, its value (``"call graph"``) and its member name (``"call_graph"``): the + facade's parameter is typed ``str``, and the underscore spelling is what a caller writing + ``analysis_level="system_dependency_graph"`` produces. An unrecognised name raises rather than + falling back to a default — a level that silently becomes 1 is the defect this function exists + to close. + """ + key = str(getattr(level, "value", level)).replace("_", " ") + try: + return ANALYZER_LEVELS[AnalysisLevel(key)] + except ValueError: + raise ValueError(f"unknown analysis_level {level!r}; expected one of {[lvl.name for lvl in AnalysisLevel]}") from None diff --git a/cldk/analysis/commons/resolve.py b/cldk/analysis/commons/resolve.py index 34e68cd9..ed50ef57 100644 --- a/cldk/analysis/commons/resolve.py +++ b/cldk/analysis/commons/resolve.py @@ -49,6 +49,7 @@ from typing import Callable, List, NamedTuple, Optional, Sequence, TypeVar +from cldk.analysis.commons.keys import module_dotted from cldk.utils.exceptions import AmbiguousName, SelectorNotInGraph T = TypeVar("T") @@ -188,6 +189,7 @@ def resolve_callable_signature( *, in_class: Optional[str] = None, in_module: Optional[str] = None, + dotted: Callable[[str], str] = module_dotted, ) -> str: """The signature of the one callable ``name`` names, narrowed by ``in_class`` / ``in_module``. @@ -219,6 +221,13 @@ def resolve_callable_signature( in_class: Keep only callables whose owning class this names. A callable with no owning class is excluded outright, not silently kept. in_module: Keep only callables whose module this names, by path or by dotted name. + dotted: How this language spells a module path as a dotted name -- the Python default + strips a trailing ``/__init__`` and knows only ``.py``. TypeScript passes a + :func:`~cldk.analysis.commons.keys.module_dotted` bound to its six source extensions + and ``package_index=None``, because ``__init__.ts`` is a module in its own right there. + One injected function rather than two forwarded keywords: the caller already has to + know its own convention, and threading each knob separately is how the two backends of + one language start disagreeing about it. Raises: AmbiguousName: More than one callable matched. @@ -229,7 +238,7 @@ class is excluded outright, not silently kept. """ filters = { "in_class": (in_class, lambda c: bool(c.class_signature) and segment_match(in_class, c.class_signature)), - "in_module": (in_module, lambda c: segment_match(in_module, c.path, sep="/") or segment_match(in_module, module_dotted(c.path))), + "in_module": (in_module, lambda c: segment_match(in_module, c.path, sep="/") or segment_match(in_module, dotted(c.path))), } by_name = set(_narrow(name, [c.signature for c in candidates])) matched = [c for c in candidates if c.signature in by_name] @@ -246,15 +255,6 @@ class is excluded outright, not silently kept. return resolve_name(name, [c.signature for c in candidates], kind="callable", narrow_with=narrow_with) -def module_dotted(path: str) -> str: - """The dotted module name a repo-relative path spells: ``"odoo/tools/mail.py"`` → - ``"odoo.tools.mail"``, ``"pkg/__init__.py"`` → ``"pkg"``. The same derivation the analyzer's - signatures embody, so ``in_module=`` can be written the way a signature reads.""" - stem = path[:-3] if path.endswith(".py") else path - if stem.endswith("/__init__"): - stem = stem[: -len("/__init__")] - return stem.replace("/", ".") - def resolve_within(resolve_callable: Callable[[str], "T"], within: str) -> "T": """Resolve a ``within=`` argument, re-raising an ambiguity in terms the caller can act on. diff --git a/cldk/analysis/commons/results.py b/cldk/analysis/commons/results.py index ddcc946e..e1babf00 100644 --- a/cldk/analysis/commons/results.py +++ b/cldk/analysis/commons/results.py @@ -24,11 +24,11 @@ :class:`LocateResult` (and the ``CallableRef`` / ``TypeRef`` / ``ModuleRef`` handles it carries) answers the single most-needed query: a scanner alert arrives as ``file:line`` and the caller needs the enclosing callable *and its source* in one round trip (see -:meth:`~cldk.analysis.python.backend.PythonAnalysisBackend.locate`). ``node``/``span`` are typed -against ``codeanalyzer-python``'s models for this leg, so ``locate`` is declared on the *Python* -backend ABC rather than the generic cross-language one — a shared declaration typed on one -language's models is a contract no other language can satisfy. A later leg generalises ``node`` / -``span`` and hoists the declaration once Java or TypeScript needs the same shape. +:meth:`~cldk.analysis.python.backend.PythonAnalysisBackend.locate`). Its body-node field is +:class:`BodyRef` and its span is this module's :class:`Span` (TS-1, leg 2.5b): both were typed on +``codeanalyzer-python``'s ``BodyNode`` / ``Span``, which made the language-neutral result module +import one language's schema — and made ``locate`` undeclarable on the cross-language ABC, since a +shared declaration typed on one language's models is a contract no other language can satisfy. :class:`EntrypointCoverage` is the same "absence is never null" discipline applied to ``get_entrypoints()``: that accessor's ``List[PyCallableOverview]`` return is frozen and cannot @@ -50,9 +50,7 @@ from typing import ClassVar, Generic, Iterator, Literal, TypeVar -from pydantic import BaseModel, computed_field - -from cldk.models.python import BodyNode, Span +from pydantic import BaseModel, ConfigDict, computed_field class Diagnostic(BaseModel): @@ -89,6 +87,66 @@ class Diagnostic(BaseModel): suggestions: list[str] = [] +class Span(BaseModel): + """Where something lives in source, in the one spelling this whole surface speaks. + + ``start`` / ``end`` are ``[line, column]`` (1-based line, 0-based column) and ``bytes`` are + UTF-8 offsets into the owning module's source — the shape *both* analyzers already emit + (``codeanalyzer-python``'s ``Span``, ``codeanalyzer-typescript``'s ``TSSpan``). It is declared + here rather than borrowed from one of them because these results are language-neutral, and a + span is the one attribute every language's nodes carry (TS-1). + + ``from_attributes`` is load-bearing, not decoration: it is what lets a backend hand its own + analyzer's span object straight to a :class:`BodyRef` or a :class:`LocateResult` and have it + validate field by field, so no backend converts by hand and no field goes missing in the + handover. It does **not** make this class interchangeable with either analyzer's — the reverse + direction (this class into a ``Py*`` model's ``span`` field) is a ``ValidationError``, which is + why ``cldk.models.python.Span`` stays codeanalyzer-python's own class rather than being + replaced by this one. + + Which fields are *meaningful* depends on the backend, and they say so rather than fabricating: + a Neo4j projection carrying only ``start_line`` / ``end_line`` rehydrates the columns and + ``bytes`` as ``0`` placeholders, never offsets to slice with (see :class:`LocateResult`). + """ + + model_config = ConfigDict(from_attributes=True) + + start: tuple[int, int] + end: tuple[int, int] + bytes: tuple[int, int] + + +class BodyRef(BaseModel): + """A handle on one node inside a callable's body — the statement, call or branch a position + landed on, in the vocabulary every analyzer shares (TS-1). + + Carries only what both analyzers emit for such a node. The Python-schema ``BodyNode`` this + replaced on :class:`LocateResult` also carried Python-shaped dataflow detail (``of``, + ``parent``, ``arguments``, the call-site facets), none of which a TypeScript body node speaks + in the same spelling; a caller that needs it addresses the node by :attr:`id` and asks the + language's own accessors. + + Attributes: + id: The analyzer's own id for the node (``"@"``) — **opaque**. + The same value :attr:`LocateResult.node_id` carries, and the handle + ``get_source`` takes. Pass it back; do not parse it and do not build one (E6). + kind: The node's kind in the analyzer's already-English spelling — ``statement``, ``call``, + ``branch``, ``loop``, ``raise``, ``handler`` and the synthetic ``entry`` / ``exit`` + bookends (the vocabulary :attr:`SliceNode.KINDS` pins). + span: The node's own source region, or ``None`` when it has none: the synthetic analysis + vertices (``@entry`` / ``@exit`` / a formal parameter) are dataflow positions, not + regions in the file, and a fabricated zero span would read as "line 0". + callee: On a ``call`` node, the id of what it resolves to; ``None`` everywhere else and on + a call that never resolved. This is the sanctioned null-to-id slot, not an absence to + paper over — measured across real applications, one call site in four resolves nothing. + """ + + id: str + kind: str + span: Span | None = None + callee: str | None = None + + class ModuleRef(BaseModel): """A lightweight handle on a module — enough to name it and fetch it again. @@ -125,12 +183,14 @@ class LocateResult(BaseModel): to the nearest callable) apart from "this file was never analysed". Attributes: - node: The innermost body node containing the position, if the graph has one that precise. - ``None`` does not mean "not found" — see ``callable``/``diagnostics`` for that. - node_id: ``node``'s identifier for :meth:`~cldk.analysis.python.backend.PythonAnalysisBackend.get_source`, - or ``None`` exactly when ``node`` is ``None``. It is the analyzer's own id for that - node (``"@"``), read off the graph where the backend can - and composed the emitter's way where it cannot — **an opaque handle, not a string to + body: The innermost body node containing the position, as a :class:`BodyRef`, if the graph + has one that precise. ``None`` does not mean "not found" — see + ``callable``/``diagnostics`` for that. + node_id: ``body``'s identifier for :meth:`~cldk.analysis.python.backend.PythonAnalysisBackend.get_source`, + or ``None`` exactly when ``body`` is ``None`` (it is the same value as ``body.id``, kept + as a field of its own because it is published contract). It is the analyzer's own id for + that node (``"@"``), read off the graph where the backend + can and composed the emitter's way where it cannot — **an opaque handle, not a string to parse or build**. Treat it as something to pass back, and address a callable by its ``callable.signature`` instead, the same key :meth:`get_method_bodies` uses. callable: The enclosing callable, or ``None`` if the position is not inside one (module @@ -148,7 +208,7 @@ class LocateResult(BaseModel): ``module_source_unavailable`` diagnostic. Callable text is available on both. span: The span the ``source`` slice covers. Which of its fields are meaningful depends on the backend, because they carry different data: the local backend returns the - analyzer's real :class:`~cldk.models.python.Span` (1-based line, 0-based column, and + analyzer's real span (1-based line, 0-based column, and UTF-8 byte offsets into the module source), while the Neo4j graph projects only ``start_line`` / ``end_line`` on ``:PyCallable`` and ``:PyBodyNode`` — so over Neo4j the line components are real and the columns and ``bytes`` are ``0`` placeholders, @@ -159,7 +219,7 @@ class LocateResult(BaseModel): no analysed module. """ - node: BodyNode | None + body: BodyRef | None node_id: str | None = None callable: CallableRef | None type: TypeRef | None diff --git a/cldk/analysis/java/backend.py b/cldk/analysis/java/backend.py index defe61ac..35f72012 100644 --- a/cldk/analysis/java/backend.py +++ b/cldk/analysis/java/backend.py @@ -16,26 +16,43 @@ """The Java analysis backend contract. -:class:`JavaAnalysis` is a (mostly) thin façade that delegates its static-analysis queries to a -*backend*. Today the only backend is :class:`~cldk.analysis.java.codeanalyzer.JCodeanalyzer` -(in-memory pydantic / NetworkX over the codeanalyzer JSON); this ABC formalizes the surface the -façade depends on so an alternative backend (e.g. a forthcoming Neo4j/Cypher backend, mirroring -the TypeScript :class:`~cldk.analysis.typescript.neo4j.TSNeo4jBackend`) can be dropped in and -selected without touching the façade. - -The contract is enforced by the type system and at instantiation time rather than matching only by -convention. Note the façade also calls Tree-sitter directly for a few parsing/sanitization helpers -(e.g. ``is_parsable``, ``get_raw_ast``); those are not part of the backend contract — only the -analysis queries the façade routes through ``self.backend`` are. +:class:`JavaAnalysis` is a thin façade that delegates its static-analysis queries to a *backend*. +Two interchangeable backends exist: + +* :class:`~cldk.analysis.java.codeanalyzer.JCodeanalyzer` — walks the in-memory pydantic + ``JApplication`` / a NetworkX call graph built from the ``analysis.json`` codeanalyzer-java emits; +* :class:`~cldk.analysis.java.neo4j.JNeo4jBackend` — answers the *same* queries with Cypher over + the graph codeanalyzer-java emits with ``--emit neo4j``. + +The shape shared with every other language — application view, symbol table, call graph, the +class/method/field lookups and the repository-artifact layer — is inherited from the generic +:class:`~cldk.analysis.commons.backend.AnalysisBackend`; what is declared here is the Java-native +remainder (compilation units, the 1.x caller/callee and class-call-graph accessors, constructors, +sub/nested classes, entry points, CRUD, comments). Both backends subclass it; the façade is typed +against it. Note the façade also calls Tree-sitter directly for a few parsing helpers +(``is_parsable``, ``get_raw_ast``); those are not part of the backend contract. + +``get_call_graph()`` on both backends is keyed by ``"."`` strings (spec J-1), +with a ``method_detail`` (:class:`~cldk.models.java.models.JMethodDetail`) and ``kind`` node +attribute; edges carry ``type``, ``weight`` and ``calling_lines``. A local or anonymous class's +segment of that key carries the signature of the callable that declares it +(``p.Outer.m(int).$anon$0``): ``$anon$N`` is numbered per declaring callable (J-1 erratum). + +**Two fields of a returned** :class:`~cldk.models.java.models.JCallable` **depend on the backend.** +Off ``analysis.json``, ``code`` is the body block and ``body`` is every body node. Off the Neo4j +projection, ``code`` is the whole *declaration* (which ends with the body block) and ``body`` holds +the ``call`` nodes only — about 30% of the graph's body nodes, enough for ``call_sites`` and +nothing else. """ from __future__ import annotations -from abc import ABC, abstractmethod -from typing import Dict, List, Tuple, Union - -import networkx as nx +from abc import abstractmethod +from typing import ClassVar, Dict, List, Tuple, Union +from cldk.analysis.commons.backend import AnalysisBackend +from cldk.analysis.commons.treesitter import TreesitterJava +from cldk.analysis.commons.treesitter.models import Captures from cldk.models.java.models import ( JApplication, JCallable, @@ -51,43 +68,118 @@ # A CRUD query row: the owning type + callable and the operations found within it. CRUDRow = Dict[str, Union[JType, JCallable, List[JCRUDOperation]]] +#: J-4: the CRUD accessors keep their names and raise this on schema v2, on both backends. +CRUD_UNAVAILABLE = "CRUD operations are not emitted by codeanalyzer-java 3.0.1 or newer (schema v2); tracked upstream as codeanalyzer-java#187" + + +#: Every call-shaped site in a callable body, under **one** capture name. One name matters: the +#: query result is a ``{capture name: [node]}`` mapping, so several names would put the nodes in +#: per-name groups and lose their source order between the groups. +_CALL_SITES = ( + "(object_creation_expression (type_identifier) @call) " + "(object_creation_expression type: (scoped_type_identifier (type_identifier) @call)) " + "(method_invocation name: (identifier) @call)" +) -class JavaAnalysisBackend(ABC): - """Abstract base every Java analysis backend implements. - A backend owns all indexing and query logic for a Java application (symbol table, call graph, - class/method/field navigation, entry points, CRUD operations, comments/docstrings); the - :class:`JavaAnalysis` façade delegates to it. Implementations must return the canonical - ``cldk.models.java`` pydantic objects (or the documented NetworkX / dict / list shapes) so - backends are behaviorally interchangeable. +class CallingLines: + """The ``calling_lines`` edge attribute of ``get_call_graph()``: **absolute file lines**, sorted, + of the calls a source callable makes to a target — parsed once per source callable. + + **Absolute, not offsets into** ``code``. The 1.x value was the 0-based line offset into + ``JCallable.code``, and ``code`` is the body block off ``analysis.json`` and the whole + declaration off the Neo4j projection, so the same call reported two different numbers on the + two backends (560 of daytrader8's 1,862 edges, measured). ``code_start_line`` is on both, so + ``code_start_line + offset`` is the file line both agree on — and it is the number a caller + wants anyway, since it indexes the file rather than a string they would have to fetch first. + + Sorted, because source order within one callable is not otherwise guaranteed: 18 of those 560 + differed only in order, and 24 of the local backend's own lists were not ascending. + + Parsed once per source callable, because the naive form re-parsed the body for every outgoing + edge — 3.6 parses per callable on ThingsBoard, where tree-sitter was 99.5% of the time + ``get_call_graph`` spent. Measured on that corpus (21,269 nodes / 53,938 edges): + **145.7 s → 41.0 s**. """ - # -----[ application / whole-program ]----- - @abstractmethod - def get_application_view(self) -> JApplication: - """The whole application view.""" + def __init__(self) -> None: + self._tsu = TreesitterJava() + self._by_callable: Dict[str, Dict[str, List[int]]] = {} + + def of(self, source: JCallable, target: JCallable) -> List[int]: + index = self._by_callable.get(source.id) + if index is None: + index = self._by_callable[source.id] = self._index(source) + return index.get(target.signature.partition("(")[0], []) + + def _index(self, source: JCallable) -> Dict[str, List[int]]: + """``{callee simple name: sorted absolute file lines}`` for one callable's ``code``.""" + code = source.code + if not code: + return {} + try: + captures: Captures = self._tsu.frame_query_and_capture_output(_CALL_SITES, code) + except Exception: # noqa: BLE001 — an unparsable body costs its lines, not the call graph + return {} + first_line = source.code_start_line + index: Dict[str, List[int]] = {} + for capture in captures: + index.setdefault(capture.node.text.decode(), []).append(first_line + capture.node.start_point[0]) + for lines in index.values(): + lines.sort() + return index + + +def duplicate_type_name(qualified_name: str) -> str: + """The defect message for two declarations that spell one qualified name — which would make a + ``get_call_graph()`` node key and a ``get_class()`` key ambiguous, so it is surfaced rather than + letting the second silently shadow the first. Both backends raise this text, identically, and + it names only the qualified name: a ``can://`` id must not appear in a message (E6).""" + return f"type qualified name {qualified_name!r} is declared twice: codeanalyzer-java emitted two declarations that spell one name" + + +def unhomed_endpoint(node_id: str) -> str: + """The defect message for a call-graph endpoint that is not one of the application's callables. + Both backends raise this text, identically, and it names the endpoint by the signature and + module key its id *spells* rather than by the id itself (E6).""" + module, sep, rest = node_id.partition(".java/") + signature = (rest or node_id).rpartition("/")[2] + where = f"{module.split('/', 4)[4]}.java" if sep and module.count("/") >= 4 else "no module of this application" + return f"call-graph endpoint {signature!r} in {where!r} is not one of its callables: codeanalyzer-java emitted an unhomed endpoint" + + +class JavaAnalysisBackend(AnalysisBackend[JApplication, JCompilationUnit, JType, JCallable, JField, JCallableParameter]): + """Abstract base every Java analysis backend implements. - @abstractmethod - def get_symbol_table(self) -> Dict[str, JCompilationUnit]: - """The per-file symbol table, keyed by file path.""" + A backend owns all indexing and query logic for a Java application; the :class:`JavaAnalysis` + façade delegates to it. Implementations must return the canonical ``cldk.models.java`` pydantic + objects (or the documented NetworkX / dict / list shapes) so backends are behaviorally + interchangeable. + + Inherited abstract (see :class:`~cldk.analysis.commons.backend.AnalysisBackend`): + ``get_application_view``, ``get_symbol_table``, ``get_call_graph``, ``get_all_classes``, + ``get_class``, ``get_all_methods_in_class``, ``get_method``, ``get_all_fields``, + ``get_method_parameters``, ``get_artifacts``, ``get_dependencies``, ``get_config_keys``, + ``get_config_uses``, ``get_unresolved_config_reads``. + """ + + P: ClassVar[str] = "J" + N: ClassVar[str] = "J" + # -----[ application / whole-program ]----- @abstractmethod def get_compilation_units(self) -> List[JCompilationUnit]: """All compilation units.""" @abstractmethod def get_java_file(self, qualified_class_name: str) -> str | None: - """The file path declaring a class. ``None`` if the class is not found.""" + """The (repo-relative) file path declaring a class. ``None`` if the class is not found.""" @abstractmethod def get_java_compilation_unit(self, file_path: str) -> JCompilationUnit: """The compilation unit for a file path.""" # -----[ call graph ]----- - @abstractmethod - def get_call_graph(self) -> nx.DiGraph: - """NetworkX DiGraph of the application's call edges.""" - @abstractmethod def get_call_graph_json(self) -> str: """The call graph serialized as JSON.""" @@ -102,21 +194,13 @@ def get_all_callees(self, source_class_name: str, source_method_signature: str, @abstractmethod def get_class_call_graph(self, qualified_class_name: str, method_name: str | None = None) -> List[Tuple[JMethodDetail, JMethodDetail]]: - """Call-graph edges reachable from a class (or one of its methods).""" + """Call-graph edges out of a class (or one of its methods).""" @abstractmethod def get_class_call_graph_using_symbol_table(self, qualified_class_name: str, method_signature: str | None = None) -> List[Tuple[JMethodDetail, JMethodDetail]]: - """Call-graph edges reachable from a class, computed from the symbol table only.""" + """Call-graph edges out of a class, computed from the symbol table's call sites only.""" # -----[ classes / methods / fields ]----- - @abstractmethod - def get_all_classes(self) -> Dict[str, JType]: - """Every class, keyed by qualified name.""" - - @abstractmethod - def get_class(self, qualified_class_name: str) -> JType | None: - """A single class by qualified name. ``None`` if not found.""" - @abstractmethod def get_all_sub_classes(self, qualified_class_name: str) -> Dict[str, JType]: """Classes that extend/implement the given class.""" @@ -137,26 +221,10 @@ def get_implemented_interfaces(self, qualified_class_name: str) -> List[str]: def get_all_methods_in_application(self) -> Dict[str, Dict[str, JCallable]]: """All methods grouped by their owning class qualified name.""" - @abstractmethod - def get_all_methods_in_class(self, qualified_class_name: str) -> Dict[str, JCallable]: - """The methods of a class.""" - - @abstractmethod - def get_method(self, qualified_class_name: str, method_signature: str) -> JCallable | None: - """A single method of a class. ``None`` if not found.""" - - @abstractmethod - def get_method_parameters(self, qualified_class_name: str, method_signature: str) -> List[JCallableParameter]: - """The parameters of a method. Empty list if the method is not found.""" - @abstractmethod def get_all_constructors(self, qualified_class_name: str) -> Dict[str, JCallable]: """The constructors of a class.""" - @abstractmethod - def get_all_fields(self, qualified_class_name: str) -> List[JField]: - """The fields of a class.""" - # -----[ entry points ]----- @abstractmethod def get_all_entry_point_methods(self) -> Dict[str, Dict[str, JCallable]]: @@ -166,7 +234,7 @@ def get_all_entry_point_methods(self) -> Dict[str, Dict[str, JCallable]]: def get_all_entry_point_classes(self) -> Dict[str, JType]: """Classes identified as application entry points.""" - # -----[ CRUD operations ]----- + # -----[ CRUD operations — J-4: raise CRUD_UNAVAILABLE on schema v2 ]----- @abstractmethod def get_all_crud_operations(self) -> List[CRUDRow]: """All CRUD operations across the application.""" @@ -188,25 +256,66 @@ def get_all_delete_operations(self) -> List[CRUDRow]: """All delete operations.""" # -----[ comments / docstrings ]----- + # J-16, the one rule, split by whether a *smaller* answer is still an answer under the name. + # A backend that keeps only per-declaration javadoc (the Neo4j projection) can answer the three + # **declaration-keyed** accessors with a javadoc-only subset — narrower than "every comment in + # this class", but a real answer about a real declaration. It cannot answer the two + # **file-keyed** ones at all: it holds nothing file-level, so every answer would be an empty + # list reading as "this file has no comments" (D7). Those two therefore raise. @abstractmethod def get_all_comments(self) -> Dict[str, List[JComment]]: - """All comments across the application, keyed by file.""" + """All comments across the application, keyed by file. + + Raises: + CodeanalyzerExecutionException: If the backend's source carries no file-level comments + at all (the Neo4j projection does not), naming what is missing and what to read + instead. Returning the per-declaration javadoc under this name would be a silent + partial (J-16). + """ @abstractmethod def get_comment_in_file(self, file_path: str) -> List[JComment]: - """The comments in a file.""" + """The comments in a file. + + Raises: + CodeanalyzerExecutionException: As :meth:`get_all_comments` does, and for the same + reason (J-16). + """ @abstractmethod def get_comments_in_a_class(self, qualified_class_name: str) -> List[JComment]: - """The comments in a class. Returns an empty list if the class is not found.""" + """The class declaration's **own** comment. Returns an empty list if the class is not found. + + Not the comments inside the class body: on both backends this is the type's own comment + list — the comment immediately above ``class Foo``. A method's is on + :meth:`get_comments_in_a_method`; an inline comment in a body is on neither, and reaches + the SDK only through :meth:`get_comment_in_file`. + + A backend whose source keeps only per-declaration javadoc narrows further, to **just the + javadoc** — a real answer rather than a refusal (J-16). + :class:`~cldk.analysis.java.neo4j.JNeo4jBackend` is such a backend. + """ @abstractmethod def get_comments_in_a_method(self, qualified_class_name: str, method_signature: str) -> List[JComment]: - """The comments in a method. Returns an empty list if the method is not found.""" + """The method declaration's **own** comment (at most one). Returns an empty list if the + method is not found. + + Not every comment inside the body — see :meth:`get_comments_in_a_class`. + + Narrows to javadoc only on a javadoc-only backend, exactly as + :meth:`get_comments_in_a_class` does (J-16). + """ @abstractmethod - def get_all_docstrings(self) -> List[Tuple[str, JComment]]: - """All docstring-style comments across the application.""" + def get_all_docstrings(self) -> Dict[str, List[JComment]]: + """All Javadoc comments across the application, keyed by file. + + Which javadoc depends on what the backend's source keeps: the in-memory backend reports + each compilation unit's own comment list (holding the *file-level* javadoc), the Neo4j + backend the javadoc of each *declaration* in the file. Both are javadoc keyed by file, and + they are different sets for the same file (J-16). + """ @abstractmethod def remove_all_comments(self, src_code: str) -> str: diff --git a/cldk/analysis/java/codeanalyzer/__init__.py b/cldk/analysis/java/codeanalyzer/__init__.py index c8063662..c18a6eaf 100644 --- a/cldk/analysis/java/codeanalyzer/__init__.py +++ b/cldk/analysis/java/codeanalyzer/__init__.py @@ -20,10 +20,4 @@ from .codeanalyzer import JCodeanalyzer - -""" -Download the codeanalyzer.jar file from the latest release on the codeanalyzer repository. -""" - - __all__ = ["JCodeanalyzer"] diff --git a/cldk/analysis/java/codeanalyzer/_jdk.py b/cldk/analysis/java/codeanalyzer/_jdk.py deleted file mode 100644 index 230a0f5e..00000000 --- a/cldk/analysis/java/codeanalyzer/_jdk.py +++ /dev/null @@ -1,183 +0,0 @@ -################################################################################ -# Copyright IBM Corporation 2024 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -################################################################################ - -"""Fetch + cache a self-contained Temurin JDK to run codeanalyzer.jar. - -Follows a platform-binary loader pattern (download a platform archive from a -release, extract restoring exec bits, locate the binary), adapted for a JDK: - - * pinned to an exact Temurin release (reproducible) instead of "latest", - * SHA256-verified, - * downloads the **JDK** (not a JRE) so ``jmods/`` is present -- WALA needs it for - call-graph (``-a 2``) analysis (``ScopeUtils`` walks ``$JAVA_HOME/jmods``), - * cached under the backend's existing per-language cache dir - (``/java/jdk/``; ``cache_dir`` defaults to ``/.codeanalyzer`` - -- the same root every cldk backend uses via ``cache_subdir``). It is **not** a - new cache location; share it across projects by passing a common ``cache_dir``. - -A bundled JDK is too large for the PyPI wheel (~180 MB compressed, per platform), -so it is fetched once on first use rather than shipped. Running on a real HotSpot -JVM gives full analysis fidelity (unlike the GraalVM native image). -""" -from __future__ import annotations - -import hashlib -import logging -import os -import platform -import stat -import tarfile -import urllib.error -import urllib.parse -import urllib.request -import zipfile -from pathlib import Path - -logger = logging.getLogger(__name__) - -# Pinned Temurin release. Bump deliberately; lives in code so it is available at -# runtime (pyproject.toml is not installed into site-packages). -JDK_RELEASE = "jdk-21.0.5+11" - - -class JdkLoader: - """Resolve a Temurin JDK from the Adoptium API.""" - - _API = "https://api.adoptium.net/v3" - - @classmethod - def _os_arch(cls) -> tuple[str, str]: - system = {"Linux": "linux", "Darwin": "mac", "Windows": "windows"}.get(platform.system()) - arch = {"x86_64": "x64", "amd64": "x64", "arm64": "aarch64", "aarch64": "aarch64"}.get( - platform.machine().lower() - ) - if not system or not arch: - raise RuntimeError(f"Unsupported platform: {platform.system()} / {platform.machine()}") - return system, arch - - @classmethod - def _resolve_asset(cls) -> tuple[str, str]: - """Return ``(download_url, sha256)`` for the pinned JDK binary. - - Resolves via the Adoptium ``/binary/version`` endpoint, which 307-redirects to the - GitHub release asset; the checksum comes from the asset's adjacent ``.sha256.txt``. The - older ``/assets/version/{release}`` query endpoint is not used: it returns 404 for pinned - releases (e.g. ``jdk-21.0.5+11``), even though the release exists. - """ - os_, arch = cls._os_arch() - release = urllib.parse.quote(JDK_RELEASE, safe="") # encode the '+' in the path - binary_url = f"{cls._API}/binary/version/{release}/{os_}/{arch}/jdk/hotspot/normal/eclipse" - - # Capture the redirect target (the GitHub asset URL) without downloading the binary. - class _NoRedirect(urllib.request.HTTPRedirectHandler): - def redirect_request(self, *args, **kwargs): - return None - - opener = urllib.request.build_opener(_NoRedirect) - req = urllib.request.Request(binary_url, headers={"User-Agent": "cldk"}) - try: - opener.open(req, timeout=30) - raise RuntimeError(f"Expected a redirect to the Temurin {JDK_RELEASE} asset from {binary_url}") - except urllib.error.HTTPError as exc: - if exc.code not in (301, 302, 303, 307, 308) or not exc.headers.get("Location"): - raise RuntimeError(f"No Temurin {JDK_RELEASE} build for {os_}/{arch} (HTTP {exc.code})") from exc - asset_url = exc.headers["Location"] - - sha_req = urllib.request.Request(asset_url + ".sha256.txt", headers={"User-Agent": "cldk"}) - with urllib.request.urlopen(sha_req, timeout=30) as resp: - sha = resp.read().decode().split()[0] - return asset_url, sha - - @classmethod - def _java_home(cls, root: Path) -> Path: - """The extracted dir with both ``bin/java`` and ``jmods`` (mac: ``.../Contents/Home``).""" - exe = "java.exe" if os.name == "nt" else "java" - for java in root.rglob(exe): - home = java.parent.parent - if java.parent.name == "bin" and (home / "jmods").is_dir(): - return home.resolve() - raise FileNotFoundError("no JDK-with-jmods found in the extracted archive") - - @classmethod - def download_and_extract(cls, dest: Path) -> Path: - url, sha = cls._resolve_asset() - dest.mkdir(parents=True, exist_ok=True) - archive = dest / url.split("/")[-1] - - logger.info(f"Downloading Temurin {JDK_RELEASE} from {url}") - digest = hashlib.sha256() - req = urllib.request.Request(url, headers={"User-Agent": "cldk"}) - with urllib.request.urlopen(req, timeout=120) as resp, open(archive, "wb") as f: - for chunk in iter(lambda: resp.read(1 << 16), b""): - f.write(chunk) - digest.update(chunk) - if digest.hexdigest() != sha: - archive.unlink(missing_ok=True) - raise RuntimeError(f"JDK checksum mismatch: {digest.hexdigest()} != {sha}") - - logger.info(f"Extracting JDK to {dest}") - if archive.name.endswith(".zip"): - # zipfile.extractall drops the executable bit; copy each stored mode back. - with zipfile.ZipFile(archive) as zf: - for info in zf.infolist(): - out = zf.extract(info, dest) - mode = info.external_attr >> 16 - if mode: - os.chmod(out, mode) - else: - with tarfile.open(archive) as tf: # tar preserves modes - tf.extractall(dest) - archive.unlink(missing_ok=True) - - java_home = cls._java_home(dest) - java = java_home / "bin" / ("java.exe" if os.name == "nt" else "java") - st = java.stat() - java.chmod(st.st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH) - return java_home - - -def ensure_jdk(java_cache_dir: Path) -> Path: - """Return ``JAVA_HOME`` for a JDK with ``jmods``. - - Args: - java_cache_dir: the backend's java cache dir (e.g. ``/.codeanalyzer/java``, - from :func:`cldk.analysis.commons.backend_config.cache_subdir`). The JDK is - cached at ``/jdk//`` -- the existing per-language - cache root, not a new location. - - Resolution order: - 1. the cached JDK under ``/jdk//`` -- reused across runs; - 2. a system ``$JAVA_HOME`` that actually has ``jmods`` -- honored verbatim; - 3. otherwise download + extract the pinned Temurin JDK into the cache. - """ - home = Path(java_cache_dir) / "jdk" / JDK_RELEASE - # The archive extracts nested (//bin, or .../Contents/Home/bin on mac), so look - # for the JDK the same way download_and_extract did rather than at /bin. - try: - cached = JdkLoader._java_home(home) - except FileNotFoundError: - pass - else: - logger.debug(f"Reusing cached JDK at {cached}") - return cached - - sys_home = os.environ.get("JAVA_HOME") - if sys_home and (Path(sys_home) / "jmods").is_dir(): - logger.debug(f"Using system JDK (has jmods) at {sys_home}") - return Path(sys_home) - - logger.info(f"JDK with jmods not found; downloading into {home}.") - return JdkLoader.download_and_extract(home) diff --git a/cldk/analysis/java/codeanalyzer/codeanalyzer.py b/cldk/analysis/java/codeanalyzer/codeanalyzer.py index 19945e1b..d376654f 100644 --- a/cldk/analysis/java/codeanalyzer/codeanalyzer.py +++ b/cldk/analysis/java/codeanalyzer/codeanalyzer.py @@ -13,1139 +13,631 @@ # See the License for the specific language governing permissions and # limitations under the License. ################################################################################ + +"""Java Codeanalyzer backend. + +Subprocess wrapper around the analyzer the ``codeanalyzer-java`` wheel carries (the ``java`` +extra; pinned in ``pyproject.toml``, mirrored in ``[tool.backend-versions]``), run on the JVM that +same wheel bundles -- the SDK downloads no JDK and touches no JDK environment variable. +Reads the schema-v2 ``analysis.json`` envelope (:class:`JAnalysis`), keeps its ``application`` as +the queried :class:`JApplication`, and owns all query/indexing logic; the :class:`JavaAnalysis` +facade is a thin delegating shell over it. +""" + +from __future__ import annotations + +import hashlib import json import logging -import os import re -import shlex import subprocess -from itertools import chain, groupby from pathlib import Path from subprocess import CompletedProcess -from typing import Dict, List, Tuple -from typing import Union +from typing import Dict, Iterable, List, Tuple, Union import networkx as nx +from pydantic import ValidationError -from cldk.analysis import AnalysisLevel -from cldk.analysis.java.backend import JavaAnalysisBackend -from cldk.analysis.commons.treesitter import TreesitterJava +from cldk.analysis.commons.levels import LEVEL_NAMES, analyzer_level +from cldk.analysis.commons.results import Diagnostic +from cldk.analysis.java.backend import CRUD_UNAVAILABLE, CallingLines, CRUDRow, JavaAnalysisBackend, duplicate_type_name, unhomed_endpoint from cldk.models.java import JGraphEdges -from cldk.models.java.enums import CRUDOperationType -from cldk.models.java.models import JApplication, JCRUDOperation, JCallable, JCallableParameter, JComment, JField, JMethodDetail, JType, JCompilationUnit, JGraphEdgesST +from cldk.models.java.models import JAnalysis, JApplication, JCallable, JCallableParameter, JCallSite, JComment, JCompilationUnit, JField, JMethodDetail, JType +from cldk.models.python import PyArtifact, PyConfigKey, PyConfigRead, PyConfigUseEdge, PyDependency from cldk.utils.exceptions.exceptions import CodeanalyzerExecutionException -from cldk.analysis.java.codeanalyzer._jdk import ensure_jdk -from cldk.analysis.commons.backend_config import cache_subdir logger = logging.getLogger(__name__) +#: The SDK's record of what the analyzer said about its own run, written **beside** +#: ``analysis.json`` in the cache directory the SDK owns. Never a field inside the payload: that +#: file's shape is codeanalyzer-java's schema, not ours. +VERDICT_FILE = "analyzer_diagnostics.json" + +#: ANSI colour, which the analyzer's console appender writes around the timestamp and the level. +#: Stripped before matching so it is never read as content and never lands in a message. +_ANSI = re.compile(r"\x1b\[[0-9;]*m") + +#: The shape codeanalyzer-java uses to declare that something it was asked for did not run: a WARN +#: line naming the capability that is ``unavailable`` and what it is ``emitting … only`` instead +#: (``RTA call graph unavailable (NullPointerException: null); emitting declared edges only``, +#: ``L4 semantic ddg unavailable (WALA build failed); emitting the derived SDG vertices and param +#: edges only``). Matched on that shape, not on the cause inside the parentheses, which differs per +#: failure and is the analyzer's to word. +_DEGRADED_LINE = re.compile(r"\[WARN\]\s*(?P\S.*?\bunavailable\b.*\bemitting\b.*\bonly\b)\s*$") + + +def _degradations(log: str) -> List[Diagnostic]: + """The analyzer's own degradation sentences, as :class:`Diagnostic`s, in the order it logged them. + + The analyzer degrades rather than failing — a build it cannot run costs it the RTA call graph + and the points-to half of the SDG, and it still exits 0 and stamps ``max_level`` with the level + it was asked for — so its log is the only authoritative signal (#341). The sentence is kept + verbatim as the message: it names the cause (``NullPointerException: null``, ``WALA build + failed``) better than a paraphrase would. + + ``code`` is ``level_too_low`` because that is what happened: the level actually computed is + below the level the envelope reports. The payload's own proxies — no ``rta`` provenance on any + call edge, no ``points-to`` provenance on any ddg edge — are **not** consulted, here or + anywhere: a small project can legitimately have neither, so their absence is corroboration for + a verdict that was recorded, never a substitute for one. + """ + messages: Dict[str, None] = {} + for line in log.splitlines(): + match = _DEGRADED_LINE.search(_ANSI.sub("", line)) + if match: + messages[match.group("message")] = None + return [Diagnostic(code="level_too_low", message=message) for message in messages] + + +def _payload_digest(analysis_json_file: Path) -> str | None: + """The sha256 of the payload the verdict describes, or ``None`` when it cannot be read.""" + try: + return hashlib.sha256(analysis_json_file.read_bytes()).hexdigest() + except OSError: + return None + + +def _recorded_verdict(verdict_file: Path, analysis_json_file: Path) -> List[Diagnostic] | None: + """The verdict recorded beside a cached ``analysis.json``, or ``None`` when there is none *for + this payload*. + + ``None`` is a third state and not a synonym for "the analyzer reported no degradation": a cache + written before this file existed, or an ``analysis.json`` dropped into the cache directory by + something other than this backend, carries no verdict at all. Reporting that as clean is the + ambiguous-empty defect (#341), so it stays distinguishable from ``[]``. + + **The verdict is bound to the payload it was written for** by that payload's sha256. A verdict + file has no other relation to the ``analysis.json`` beside it, so pairing it with a payload it + did not describe — the analyzer re-run by hand, an ``analysis.json`` copied in over one this + backend wrote — would report a stale verdict as current. A digest that does not match is no + verdict for this payload, which is ``None``. + + Never raises: every way this file can be wrong (missing, unreadable, not JSON, not the shape + this backend writes, a ``Diagnostic`` that no longer validates) is a verdict that cannot be + read, and turning a working cache hit into a constructor failure over the SDK's own sidecar is + not a trade this makes. + """ + try: + recorded = json.loads(verdict_file.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + return None + if not isinstance(recorded, dict) or not isinstance(recorded.get("diagnostics"), list): + return None + if recorded.get("payload_sha256") != _payload_digest(analysis_json_file): + return None + try: + return [Diagnostic.model_validate(entry) for entry in recorded["diagnostics"]] + except ValidationError: + return None + class JCodeanalyzer(JavaAnalysisBackend): - """A class for building the application view of a Java application using Codeanalyzer. + """Build and query the application view of a Java project by invoking codeanalyzer-java. Args: - project_dir (str or Path): The path to the root of the Java project. - source_code (str, optional): The source code of a single Java file to analyze. Defaults to None. - analysis_json_path (str or Path, optional): The path to save the intermediate code analysis outputs. - If None, the analysis will be read from the pipe. - analysis_level (str): The level of analysis ('symbol_table' or 'call_graph'). - eager_analysis (bool): If True, the analysis will be performed every time the object is created. + project_dir: Path to the root of the Java project. + analysis_json_path: Directory to persist ``analysis.json`` (the language-keyed cache dir, + ``/java``). If None, the envelope is read from the subprocess stdout pipe. + analysis_level: Any :class:`~cldk.analysis.AnalysisLevel` (or its name); sent to the + analyzer as ``-a 1..4`` — the backend requests what the caller asked for. + eager_analysis: If True, re-run the analyzer even if a compatible ``analysis.json`` is cached. + target_files: Restrict analysis to these files (``-t``); always re-runs. + + Attributes: + analysis: The whole ``analysis.json`` envelope — ``schema_version``, ``max_level``, + ``analyzer.version`` — for callers that need to know what produced the view. + application: ``analysis.application``, the queried view. + analyzer_diagnostics: What the analyzer said about its own run, in three distinguishable + states (#341): a **non-empty** list — it declared a degradation, one + :class:`~cldk.analysis.commons.results.Diagnostic` per sentence it logged; + **empty** — it declared none; **``None``** — no verdict is recorded, so whether the + requested level was fully computed is *unknown*. ``None`` happens on a cache written + before this backend recorded verdicts, on an ``analysis.json`` put in the cache + directory from elsewhere, and in stdout-pipe mode (no ``analysis_json_path``), where the + payload occupies the same pipe the log would. """ def __init__( self, - project_dir: Union[str, Path], - source_code: str | None, + project_dir: Union[str, Path, None], analysis_json_path: Union[str, Path, None], analysis_level: str, eager_analysis: bool, target_files: List[str] | None, ) -> None: self.project_dir = project_dir - self.source_code = source_code self.analysis_json_path = analysis_json_path - self.eager_analysis = eager_analysis self.analysis_level = analysis_level + self.eager_analysis = eager_analysis self.target_files = target_files - if self.source_code is None: - self.application = self._init_codeanalyzer(analysis_level=1 if analysis_level == AnalysisLevel.symbol_table else 2) - else: - self.application = self._codeanalyzer_single_file() - # Attributes related the Java code analysis... - if analysis_level == AnalysisLevel.call_graph: - self.call_graph: nx.DiGraph = self._generate_call_graph(using_symbol_table=False) - else: - self.call_graph: nx.DiGraph | None = None - - def _get_application(self) -> JApplication: - """Should return the application view of the Java code. - - Returns: - JApplication: The application view of the Java code. - """ - if self.application is None: - self.application = self._init_codeanalyzer() - return self.application - - def _locate_jar(self) -> Path: - """The bundled codeanalyzer jar (placed under ``codeanalyzer/jar/`` at build time).""" - jar_dir = Path(__file__).resolve().parent / "jar" - jar = next(iter(sorted(jar_dir.glob("codeanalyzer*.jar"))), None) - if jar is None: - raise CodeanalyzerExecutionException(f"codeanalyzer jar not found in {jar_dir}") - return jar - + self.analyzer_diagnostics: List[Diagnostic] | None = None + self.analysis: JAnalysis = self._init_codeanalyzer(analysis_level=analyzer_level(analysis_level)) + self._report_analyzer_diagnostics(analyzer_level(analysis_level)) + self.application: JApplication = self.analysis.application + self._call_graph: nx.DiGraph | None = None + self._index() + + # -----[ driving the analyzer ]----- def _get_codeanalyzer_exec(self) -> List[str]: - """Return the command that runs codeanalyzer.jar on a bundled-fidelity JVM. - - Resolves (and on first use downloads + caches) a Temurin JDK with ``jmods`` - under the backend's existing java cache dir (``/java/jdk/``; - ``cache_dir`` defaults to ``/.codeanalyzer``) and runs the bundled - jar with it. ``JAVA_HOME`` is exported so the analyzer's WALA scope (call - graph / ``-a 2``) can read ``$JAVA_HOME/jmods``. Running on a real HotSpot JVM - gives full analysis fidelity (unlike the GraalVM native image). + """``codeanalyzer_java.command()`` — ``[, -jar, ]``. - Returns: - List[str]: ``[/bin/java, -jar, ]``. + The ``codeanalyzer-java`` wheel is the single source of both the jar and the JVM it runs + on; 3.0.x reads its primordial scope from ``jrt:/`` inside that JVM, so the SDK has nothing + to provision and no environment to point the analyzer at -- whatever JDK the machine has (or + has not) is left alone. Imported here rather than at module import so ``import cldk`` (and + ``cldk.analysis.java``) work without the ``java`` extra. """ - # analysis_json_path IS the java cache subdir (cache_subdir(cache_dir, project, "java")); - # fall back to the same helper in source/pipe mode where it is None. - java_cache = Path(self.analysis_json_path) if self.analysis_json_path else cache_subdir(None, self.project_dir, "java") - if java_cache is None: + try: + import codeanalyzer_java + except ImportError as exc: raise CodeanalyzerExecutionException( - "Cannot resolve a JDK cache directory: no cache directory and no project directory " - "-- single-file source mode cannot host a JDK (unsupported, see #256)." - ) - java_home = ensure_jdk(java_cache) - # ScopeUtils reads the JAVA_HOME env var (not java.home); child procs inherit os.environ. - os.environ["JAVA_HOME"] = str(java_home) - java_bin = java_home / "bin" / ("java.exe" if os.name == "nt" else "java") - return [str(java_bin), "-jar", str(self._locate_jar())] - - @staticmethod - def _init_japplication(data: str) -> JApplication: - """Should return JApplication giving the stringified JSON as input. - Returns - ------- - JApplication - The application view of the Java code with the analysis results. - """ - # from ipdb import set_trace - - # set_trace() - return JApplication(**json.loads(data)) + 'the Java analyzer is not installed: the codeanalyzer-java distribution (module "codeanalyzer_java") carries the analyzer jar and the JVM it runs on. Install it with: pip install "cldk[java]"' + ) from exc + return codeanalyzer_java.command() + + def _argv(self, analysis_level: int, output_dir: Path | None) -> List[str]: + """The 3.0.x command line: ``-i -a <1..4> [-o -c /cache -v] --app-name + [-t ]...``. The application name is what the analyzer stamps into every + ``can://java//...`` id; without ``-o`` the analyzer prints the JSON to stdout. + + ``-v`` ("print logs to console") rides along with ``-o`` because the analyzer's log is the + only place it declares that a capability it was asked for did not run (#341) — and only + with ``-o``, since without it the payload occupies the same stdout the log would. + """ + if self.project_dir is None: + raise CodeanalyzerExecutionException("Cannot run codeanalyzer-java: no project directory.") + args = self._get_codeanalyzer_exec() + project = Path(self.project_dir) + args += ["-i", str(project), "-a", str(analysis_level)] + if output_dir is not None: + args += ["-o", str(output_dir), "-c", str(output_dir / "cache"), "-v"] + args += ["--app-name", project.name] + for tf in self.target_files or []: + args += ["-t", str(tf).strip()] + return args @staticmethod def check_exisiting_analysis_file_level(analysis_json_path_file: Path, analysis_level: int) -> bool: - """Validate whether a cached analysis file is compatible with the current model. - - Args: - analysis_json_path_file (Path): Path to the cached ``analysis.json`` file. - analysis_level (int): Requested analysis level (1=symbol table, 2=call graph). + """Whether a cached ``analysis.json`` can serve a request at ``analysis_level``. - Returns: - bool: True if the cached file is compatible; otherwise False. + ``False`` (re-run) when the file is missing, unparsable, or was computed at a lower + ``max_level`` than requested. A file without ``schema_version`` is a pre-v2 (2.x) artifact + and is refused outright (J-9): the v2 models cannot read it and a silent re-run would hide + that the cache directory holds a stale generation. """ - analysis_file_compatible = True if not analysis_json_path_file.exists(): - analysis_file_compatible = False - else: - try: - with open(analysis_json_path_file) as f: - data = json.load(f) - if analysis_level == 2 and "call_graph" not in data: - analysis_file_compatible = False - elif analysis_level == 1 and "symbol_table" not in data: - analysis_file_compatible = False - except (json.JSONDecodeError, OSError): - analysis_file_compatible = False - return analysis_file_compatible - - def _init_codeanalyzer(self, analysis_level=1) -> JApplication: - """Should initialize the Codeanalyzer. - - Args: - analysis_level (int): The level of analysis to be performed (1 for symbol table, 2 for call graph). - - Returns: - JApplication: The application view of the Java code with the analysis results. - - Raises: - CodeanalyzerExecutionException: If there is an error running Codeanalyzer. - """ - codeanalyzer_exec = self._get_codeanalyzer_exec() - codeanalyzer_args = "" + return False + try: + data = json.loads(analysis_json_path_file.read_text(encoding="utf-8")) + except (json.JSONDecodeError, OSError): + return False + if not isinstance(data, dict): + return False + if "schema_version" not in data: + raise CodeanalyzerExecutionException(f"cached analysis.json at {analysis_json_path_file} predates schema v2 (no schema_version); delete it or pass eager_analysis=True") + return int(data.get("max_level", 0)) >= analysis_level + + def _init_codeanalyzer(self, analysis_level: int) -> JAnalysis: + """Run the analyzer (or reuse a compatible cache) and return the validated envelope.""" if self.analysis_json_path is None: - logger.info("Reading analysis from the pipe.") - # If target file is provided, the input is merged into a single string and passed to codeanalyzer - if self.target_files: - target_file_options = " -t ".join([s.strip() for s in self.target_files]) - codeanalyzer_args = codeanalyzer_exec + shlex.split(f"-i {Path(self.project_dir)} --analysis-level={analysis_level} -t {target_file_options}") - else: - codeanalyzer_args = codeanalyzer_exec + shlex.split(f"-i {Path(self.project_dir)} --analysis-level={analysis_level}") + args = self._argv(analysis_level, None) try: - logger.info(f"Running codeanalyzer: {' '.join(codeanalyzer_args)}") - console_out: CompletedProcess[str] = subprocess.run( - codeanalyzer_args, - capture_output=True, - text=True, - check=True, - ) - return self._init_japplication(console_out.stdout) - except Exception as e: + logger.info(f"Running codeanalyzer-java: {' '.join(args)}") + console_out: CompletedProcess[str] = subprocess.run(args, capture_output=True, text=True, check=True) + return JAnalysis.model_validate_json(console_out.stdout) + except Exception as e: # noqa: BLE001 raise CodeanalyzerExecutionException(str(e)) from e + + output_dir = Path(self.analysis_json_path) + analysis_json_file = output_dir / "analysis.json" + verdict_file = output_dir / VERDICT_FILE + needs_run = self.eager_analysis or bool(self.target_files) or not self.check_exisiting_analysis_file_level(analysis_json_file, analysis_level) + if needs_run: + args = self._argv(analysis_level, output_dir) + try: + logger.info(f"Running codeanalyzer-java: {' '.join(args)}") + console_out: CompletedProcess[str] = subprocess.run(args, capture_output=True, text=True, check=True) + if not analysis_json_file.exists(): + raise CodeanalyzerExecutionException("codeanalyzer-java did not generate analysis.json.") + except Exception as e: # noqa: BLE001 + raise CodeanalyzerExecutionException(str(e)) from e + # Persisted beside the payload, because the point of the cache is that the next run does + # not invoke the analyzer — and a cached payload has no log to read the verdict off. + self.analyzer_diagnostics = _degradations(console_out.stdout) + verdict = {"payload_sha256": _payload_digest(analysis_json_file), "diagnostics": [d.model_dump() for d in self.analyzer_diagnostics]} + verdict_file.write_text(json.dumps(verdict), encoding="utf-8") else: - # Check if the code analyzer needs to be run - is_run_code_analyzer = False - analysis_json_path_file = Path(self.analysis_json_path).joinpath("analysis.json") - # If target file is provided, the input is merged into a single string and passed to codeanalyzer - if self.target_files: - target_file_options = " -t ".join([s.strip() for s in self.target_files]) - codeanalyzer_args = codeanalyzer_exec + shlex.split( - f"-i {Path(self.project_dir)} --analysis-level={analysis_level}" f" -o {self.analysis_json_path} -t {target_file_options}" - ) - is_run_code_analyzer = True - else: - if not self.check_exisiting_analysis_file_level(analysis_json_path_file, analysis_level) or self.eager_analysis: - # If the analysis file does not exist, we'll run the analysis. Alternately, if the eager_analysis - # flag is set, we'll run the analysis every time the object is created. This will happen regradless - # of the existence of the analysis file. - # Create the executable command for codeanalyzer. - codeanalyzer_args = codeanalyzer_exec + shlex.split(f"-i {Path(self.project_dir)} --analysis-level={analysis_level} -o {self.analysis_json_path} -v") - is_run_code_analyzer = True - - if is_run_code_analyzer: - try: - logger.info(f"Running codeanalyzer subprocess with args {codeanalyzer_args}") - subprocess.run( - codeanalyzer_args, - capture_output=True, - text=True, - check=True, - ) - if not analysis_json_path_file.exists(): - raise CodeanalyzerExecutionException("Codeanalyzer did not generate the analysis file.") - - except Exception as e: - raise CodeanalyzerExecutionException(str(e)) from e - with open(analysis_json_path_file) as f: - data = json.load(f) - return self._init_japplication(json.dumps(data)) - - def _codeanalyzer_single_file(self) -> JApplication: - """Invokes codeanalyzer in a single file mode. - - Returns: - JApplication: The application view of the Java code with the analysis results. - """ - codeanalyzer_exec = self._get_codeanalyzer_exec() - codeanalyzer_args = ["--source-analysis", self.source_code] - codeanalyzer_cmd = codeanalyzer_exec + codeanalyzer_args - try: - logger.info(f"Running {' '.join(codeanalyzer_cmd)}") - console_out: CompletedProcess[str] = subprocess.run(codeanalyzer_cmd, capture_output=True, text=True, check=True) - if console_out.returncode != 0: - raise CodeanalyzerExecutionException(console_out.stderr) - return self._init_japplication(console_out.stdout) - except Exception as e: - raise CodeanalyzerExecutionException(str(e)) from e + self.analyzer_diagnostics = _recorded_verdict(verdict_file, analysis_json_file) + return JAnalysis.model_validate_json(analysis_json_file.read_text(encoding="utf-8")) + + def _report_analyzer_diagnostics(self, analysis_level: int) -> None: + """Say, once per analysis and at ``WARNING``, what the analyzer said about its own run — or + that nothing is recorded, which is *unknown* and never "fine". + + Never raises. A declared-only call graph is still the call graph, and every caller content + with that answer keeps working; the caller who is not needs to be told, not stopped. + + Silent below the call graph: nothing the analyzer can degrade runs at ``-a 1``, so a + verdict-less symbol table is not an unknown worth a warning. + """ + if analysis_level < analyzer_level("call_graph"): + return + level_name = LEVEL_NAMES[analysis_level] + if self.analyzer_diagnostics is None: + logger.warning(f"codeanalyzer-java recorded no degradation verdict for this analysis: whether analysis_level={level_name} was fully computed is unknown") + return + for diagnostic in self.analyzer_diagnostics: + logger.warning(f"codeanalyzer-java did not fully compute analysis_level={level_name}: {diagnostic.message}") + + # -----[ indexing ]----- + def _index(self) -> None: + """Flatten the containment tree once: every type (top-level, nested, local/anonymous) by + its source-spelled qualified name, its file, and every callable by its ``can://`` id — the + join that turns a wire call-graph endpoint into the ``"."`` node key.""" + self._types: Dict[str, JType] = {} + self._file_of: Dict[str, str] = {} + self._callables: Dict[str, Tuple[JType, JCallable]] = {} + for path, unit in self.application.symbol_table.items(): + for t in unit.types.values(): + self._add_type(t, path) + + def _add_type(self, t: JType, path: str) -> None: + name = t.qualified_name + if name in self._types: + raise CodeanalyzerExecutionException(duplicate_type_name(name)) + self._types[name] = t + self._file_of[name] = path + for c in t.callables.values(): + self._callables[c.id] = (t, c) + for lt in c.types.values(): + self._add_type(lt, path) + for nt in t.types.values(): + self._add_type(nt, path) - def get_symbol_table(self) -> Dict[str, JCompilationUnit]: - """Should return the symbol table of the Java code. + @staticmethod + def _detail(klass: str, c: JCallable) -> JMethodDetail: + return JMethodDetail(method_declaration=c.declaration, klass=klass, method=c) + + def _node_of(self, node_id: str) -> Tuple[str, JMethodDetail]: + """The (node key, method detail) a call-graph endpoint id resolves to. Every endpoint the + analyzer emits is homed on the tree; one that is not is the analyzer's defect, surfaced + rather than skipped — named by the signature and module key its id spells, never by the id + (E6), in the same words the Neo4j backend uses.""" + try: + t, c = self._callables[node_id] + except KeyError: + raise CodeanalyzerExecutionException(unhomed_endpoint(node_id)) from None + return f"{t.qualified_name}.{c.signature}", self._detail(t.qualified_name, c) - Returns: - Dict[str, JCompilationUnit]: The symbol table of the Java code. - """ - if self.application is None: - self.application = self._init_codeanalyzer() - return self.application.symbol_table + def _is_external(self, node_id: str) -> bool: + """An ``@external/…`` endpoint (a call target outside the project). 3a keeps the 1.x + callable-only graph and drops edges to them; ``get_external_symbols`` arrives in 3b.""" + return "@external/" in node_id or node_id in (self.application.external_symbols or {}) + # -----[ application / whole-program ]----- def get_application_view(self) -> JApplication: - """Should return the application view of the Java code. - - Returns: - JApplication: The application view of the Java code. - """ - if self.source_code: - # This branch is triggered when a single file is being analyzed. - self.application = self._codeanalyzer_single_file() - return self.application - else: - if self.application is None: - self.application = self._init_codeanalyzer() - return self.application + return self.application - def get_system_dependency_graph(self) -> list[JGraphEdges]: - """Runs the codeanalyzer to get the system dependency graph. + def get_symbol_table(self) -> Dict[str, JCompilationUnit]: + return self.application.symbol_table - Returns: - list[JGraphEdges]: The system dependency graph. - """ - if self.application.system_dependency_graph is None or self.application.call_graph is None: - self.application = self._init_codeanalyzer(analysis_level=2) + def get_compilation_units(self) -> List[JCompilationUnit]: + return list(self.application.symbol_table.values()) - logger.warning("System dependency graph is not yet implemented. Returning the call graph instead.") - return self.application.call_graph + def get_java_file(self, qualified_class_name: str) -> str | None: + return self._file_of.get(qualified_class_name) - def _generate_call_graph(self, using_symbol_table) -> nx.DiGraph: - """Generates the call graph of the Java code. + def get_java_compilation_unit(self, file_path: str) -> JCompilationUnit: + return self.application.symbol_table[file_path] - Args: - using_symbol_table (bool): Whether to use the symbol table for generating the call graph. + def get_system_dependency_graph(self) -> list[JGraphEdges]: + """The wire call graph (``JApplication.call_graph``), one :class:`JCallGraphEdge` per edge.""" + return self.application.call_graph - Returns: - nx.DiGraph: The call graph of the Java code. - """ + # -----[ call graph ]----- + def get_call_graph(self) -> nx.DiGraph: + """Build (and cache) the call graph keyed by ``"."`` (J-1): node attrs + ``method_detail`` / ``kind="callable"``; edge attrs ``type="CALL_DEP"``, ``weight``, + ``calling_lines``. Empty below level 2 (the wire carries no ``call_graph`` there).""" + if self._call_graph is not None: + return self._call_graph cg = nx.DiGraph() - if using_symbol_table: - NotImplementedError("Call graph generation using symbol table is not implemented yet.") - else: - sdg = self.get_system_dependency_graph() - tsu = TreesitterJava() - edge_list = [ - ( - (jge.source.method.signature, jge.source.klass), - (jge.target.method.signature, jge.target.klass), - { - "type": jge.type, - "weight": jge.weight, - "calling_lines": ( - tsu.get_calling_lines(jge.source.method.code, jge.target.method.signature) - if not jge.source.method.is_implicit or not jge.target.method.is_implicit - else [] - ), - }, - ) - for jge in sdg - if jge.type == "CALL_DEP" # or jge.type == "CONTROL_DEP" - ] - for jge in sdg: - cg.add_node( - (jge.source.method.signature, jge.source.klass), - method_detail=jge.source, - ) - cg.add_node( - (jge.target.method.signature, jge.target.klass), - method_detail=jge.target, - ) - cg.add_edges_from(edge_list) + lines = CallingLines() + for edge in self.application.call_graph: + if self._is_external(edge.src) or self._is_external(edge.dst): + continue + src, src_detail = self._node_of(edge.src) + dst, dst_detail = self._node_of(edge.dst) + cg.add_node(src, method_detail=src_detail, kind="callable") + cg.add_node(dst, method_detail=dst_detail, kind="callable") + cg.add_edge(src, dst, type="CALL_DEP", weight=edge.weight, calling_lines=lines.of(src_detail.method, dst_detail.method)) + self._call_graph = cg return cg - def get_class_hierarchy(self) -> nx.DiGraph: - """Should return the class hierarchy of the Java code. - - Returns: - nx.DiGraph: The class hierarchy of the Java code. - """ - - def get_call_graph(self) -> nx.DiGraph: - """Should return the call graph of the Java code. - - Returns: - nx.DiGraph: The call graph of the Java code. - """ - if self.analysis_level == "symbol_table": - self.call_graph = self._generate_call_graph(using_symbol_table=True) - if self.call_graph is None: - self.call_graph = self._generate_call_graph(using_symbol_table=False) - return self.call_graph - def get_call_graph_json(self) -> str: - """Get call graph in serialized json format. - - Returns: - str: Call graph in json. - """ - callgraph_list = [] - edges = list(self.call_graph.edges.data("calling_lines")) - for edge in edges: - callgraph_dict = {} - callgraph_dict["source_method_signature"] = edge[0][0] - callgraph_dict["source_method_body"] = self.call_graph.nodes[edge[0]]["method_detail"].method.code - callgraph_dict["source_class"] = edge[0][1] - callgraph_dict["target_method_signature"] = edge[1][0] - callgraph_dict["target_method_body"] = self.call_graph.nodes[edge[1]]["method_detail"].method.code - callgraph_dict["target_class"] = edge[1][1] - callgraph_dict["calling_lines"] = edge[2] - callgraph_list.append(callgraph_dict) - return json.dumps(callgraph_list) + cg = self.get_call_graph() + rows = [] + for source, target, calling_lines in cg.edges.data("calling_lines"): + s: JMethodDetail = cg.nodes[source]["method_detail"] + t: JMethodDetail = cg.nodes[target]["method_detail"] + rows.append( + { + "source_method_signature": s.method.signature, + "source_method_body": s.method.code, + "source_class": s.klass, + "target_method_signature": t.method.signature, + "target_method_body": t.method.code, + "target_class": t.klass, + "calling_lines": calling_lines, + } + ) + return json.dumps(rows) def get_all_callers(self, target_class_name: str, target_method_signature: str, using_symbol_table: bool) -> Dict: - """Get all the caller details for a given Java method. - - Args: - target_class_name (str): The qualified class name of the target method. - target_method_signature (str): The signature of the target method. - using_symbol_table (bool): Whether to use the symbol table to generate the call graph. - - Returns: - Dict: A dictionary containing caller details. - """ - - caller_detail_dict = {} - call_graph = None - if using_symbol_table: - call_graph = self.__call_graph_using_symbol_table(qualified_class_name=target_class_name, method_signature=target_method_signature, is_target_method=True) - else: - call_graph = self.call_graph - if (target_method_signature, target_class_name) not in call_graph.nodes(): - return caller_detail_dict - - in_edge_view = call_graph.in_edges( - nbunch=( - target_method_signature, - target_class_name, - ), - data=True, - ) - caller_detail_dict["caller_details"] = [] - caller_detail_dict["target_method"] = call_graph.nodes[(target_method_signature, target_class_name)]["method_detail"] - - for source, target, data in in_edge_view: - cm = {"caller_method": call_graph.nodes[source]["method_detail"], "calling_lines": data["calling_lines"]} - caller_detail_dict["caller_details"].append(cm) - return caller_detail_dict - - def get_all_callees(self, source_class_name: str, source_method_signature: str, using_symbol_table: bool) -> Dict: - """Get all the callee details for a given Java method. - - Args: - source_class_name (str): The qualified class name of the source method. - source_method_signature (str): The signature of the source method. - using_symbol_table (bool): Whether to use the symbol table to generate the call graph. - - Returns: - Dict: A dictionary containing callee details. - """ - callee_detail_dict = {} - call_graph = None - if using_symbol_table: - call_graph = self.__call_graph_using_symbol_table(qualified_class_name=source_class_name, method_signature=source_method_signature) - else: - call_graph = self.call_graph - if (source_method_signature, source_class_name) not in call_graph.nodes(): - return callee_detail_dict - - out_edge_view = call_graph.out_edges(nbunch=(source_method_signature, source_class_name), data=True) - - callee_detail_dict["callee_details"] = [] - callee_detail_dict["source_method"] = call_graph.nodes[(source_method_signature, source_class_name)]["method_detail"] - for source, target, data in out_edge_view: - cm = {"callee_method": call_graph.nodes[target]["method_detail"]} - cm["calling_lines"] = data["calling_lines"] - callee_detail_dict["callee_details"].append(cm) - return callee_detail_dict - - def get_all_methods_in_application(self) -> Dict[str, Dict[str, JCallable]]: - """Should return a dictionary of all methods in the Java code with qualified class name as the key - and a dictionary of methods in that class as the value. - - Returns: - Dict[str, Dict[str, JCallable]]: A dictionary of dictionaries of all methods in the Java code. - """ - - class_method_dict = {} - class_dict = self.get_all_classes() - for k, v in class_dict.items(): - class_method_dict[k] = v.callable_declarations - return class_method_dict - - def get_all_classes(self) -> Dict[str, JType]: - """Should return a dictionary of all classes in the Java code. - - Returns: - Dict[str, JType]: A dictionary of all classes in the Java code, with qualified class names as keys. - """ - - class_dict = {} - symtab = self.get_symbol_table() - for v in symtab.values(): - class_dict.update(v.type_declarations) - return class_dict - - def get_class(self, qualified_class_name) -> JType | None: - """Should return a class given the qualified class name. - - Args: - qualified_class_name (str): The qualified name of the class. - - Returns: - JType | None: A class for the given qualified class name, or None if not found. - """ - symtab = self.get_symbol_table() - for _, v in symtab.items(): - if qualified_class_name in v.type_declarations.keys(): - return v.type_declarations.get(qualified_class_name) - return None - - def get_method(self, qualified_class_name, method_signature) -> JCallable | None: - """Should return a method given the qualified method name. - - Args: - qualified_class_name (str): The qualified name of the class. - method_signature (str): The signature of the method. - - Returns: - JCallable | None: A method for the given qualified method name, or None if not found. - """ - symtab = self.get_symbol_table() - for v in symtab.values(): - if qualified_class_name in v.type_declarations.keys(): - ci = v.type_declarations[qualified_class_name] - for cd in ci.callable_declarations.keys(): - if cd == method_signature: - return ci.callable_declarations[cd] - return None - - def get_method_parameters(self, qualified_class_name, method_signature) -> List[JCallableParameter]: - """Should return a dictionary of method parameters given the qualified class name and method signature. - - Args: - qualified_class_name (str): The qualified name of the class. - method_signature (str): The signature of the method. - - Returns: - List[JCallableParameter]: The method parameters for the given qualified class name and method - signature. Empty list if the method is not found. - """ - method = self.get_method(qualified_class_name, method_signature) - return method.parameters if method is not None else [] - - def get_parameters_from_callable(self, callable: JCallable) -> List[JCallableParameter]: - """Should return a dictionary of method parameters given the callable. - - Args: - callable (JCallable): The callable object. - - Returns: - Dict[str, str]: A dictionary of method parameters for the given callable. - """ - return callable.parameters - - def get_java_file(self, qualified_class_name) -> str | None: - """Should return java file name given the qualified class name. - - Args: - qualified_class_name (str): The qualified name of the class. - - Returns: - str | None: Java file name containing the given qualified class, or None if not found. - """ - symtab = self.get_symbol_table() - for k, v in symtab.items(): - if (qualified_class_name) in v.type_declarations.keys(): - return k - return None - - def get_compilation_units(self) -> List[JCompilationUnit]: - """Get all the compilation units in the symbol table. - - Returns: - List[JCompilationUnit]: A list of compilation units. - """ - if self.application is None: - self.application = self._init_codeanalyzer() - return self.get_symbol_table().values() - - def get_java_compilation_unit(self, file_path: str) -> JCompilationUnit: - """Given the path of a Java source file, returns the compilation unit object from the symbol table. - - Args: - file_path (str): Absolute path to the Java source file. - - Returns: - JCompilationUnit: Compilation unit object for the Java source file. - """ - - if self.application is None: - self.application = self._init_codeanalyzer() - return self.application.symbol_table[file_path] - - def get_all_methods_in_class(self, qualified_class_name) -> Dict[str, JCallable]: - """Should return a dictionary of all methods in the given class. - - Args: - qualified_class_name (str): The qualified name of the class. - - Returns: - Dict[str, JCallable]: A dictionary of all methods in the given class. - """ - ci = self.get_class(qualified_class_name) - if ci is None: + cg = self._symbol_table_call_graph(target_class_name, target_method_signature, is_target=True) if using_symbol_table else self.get_call_graph() + key = f"{target_class_name}.{target_method_signature}" + if key not in cg: return {} - methods = {k: v for (k, v) in ci.callable_declarations.items() if v.is_constructor is False} - return methods - - def get_all_constructors(self, qualified_class_name) -> Dict[str, JCallable]: - """Should return a dictionary of all constructors of the given class. + return { + "caller_details": [{"caller_method": cg.nodes[s]["method_detail"], "calling_lines": d["calling_lines"]} for s, _, d in cg.in_edges(key, data=True)], + "target_method": cg.nodes[key]["method_detail"], + } - Args: - qualified_class_name (str): The qualified name of the class. - - Returns: - Dict[str, JCallable]: A dictionary of all constructors of the given class. - """ - ci = self.get_class(qualified_class_name) - if ci is None: + def get_all_callees(self, source_class_name: str, source_method_signature: str, using_symbol_table: bool) -> Dict: + cg = self._symbol_table_call_graph(source_class_name, source_method_signature) if using_symbol_table else self.get_call_graph() + key = f"{source_class_name}.{source_method_signature}" + if key not in cg: return {} - constructors = {k: v for (k, v) in ci.callable_declarations.items() if v.is_constructor is True} - return constructors - - def get_all_sub_classes(self, qualified_class_name) -> Dict[str, JType]: - """Should return a dictionary of all sub-classes of the given class. + return { + "callee_details": [{"callee_method": cg.nodes[t]["method_detail"], "calling_lines": d["calling_lines"]} for _, t, d in cg.out_edges(key, data=True)], + "source_method": cg.nodes[key]["method_detail"], + } - Args: - qualified_class_name (str): The qualified name of the class. - - Returns: - Dict[str, JType]: A dictionary of all sub-classes of the given class, and class details. - """ - - all_classes = self.get_all_classes() - sub_classes = {} - for cls in all_classes: - if qualified_class_name in all_classes[cls].implements_list or qualified_class_name in all_classes[cls].extends_list: - sub_classes[cls] = all_classes[cls] - return sub_classes - - def get_all_fields(self, qualified_class_name) -> List[JField]: - """Should return a list of all fields of the given class. - - Args: - qualified_class_name (str): The qualified name of the class. - - Returns: - List[JField]: A list of all fields of the given class. - """ - ci = self.get_class(qualified_class_name) - if ci is None: - logging.warning(f"Class {qualified_class_name} not found in the application view.") - return list() - return ci.field_declarations - - def get_all_nested_classes(self, qualified_class_name) -> List[JType]: - """Should return a list of all nested classes for the given class. - - Args: - qualified_class_name (str): The qualified name of the class. - - Returns: - List[JType]: A list of nested classes for the given class. - """ - ci = self.get_class(qualified_class_name) - if ci is None: - logging.warning(f"Class {qualified_class_name} not found in the application view.") - return list() - nested_classes = ci.nested_type_declarations - return [self.get_class(c) for c in nested_classes] # Assuming qualified nested class names - - def get_extended_classes(self, qualified_class_name) -> List[str]: - """Should return a list of all extended classes for the given class. - - Args: - qualified_class_name (str): The qualified name of the class. - - Returns: - List[str]: A list of extended classes for the given class. - """ - ci = self.get_class(qualified_class_name) - if ci is None: - logging.warning(f"Class {qualified_class_name} not found in the application view.") - return list() - return ci.extends_list - - def get_implemented_interfaces(self, qualified_class_name) -> List[str]: - """Should return a list of all implemented interfaces for the given class. - - Args: - qualified_class_name (str): The qualified name of the class. - - Returns: - List[str]: A list of implemented interfaces for the given class. - """ - ci = self.get_class(qualified_class_name) - if ci is None: - logging.warning(f"Class {qualified_class_name} not found in the application view.") - return list() - return ci.implements_list - - def get_class_call_graph_using_symbol_table(self, qualified_class_name: str, method_signature: str | None = None) -> (List)[Tuple[JMethodDetail, JMethodDetail]]: - """Should return call graph using symbol table. The analysis will not be - complete as symbol table has known limitation of resolving types - Args: - qualified_class_name: qualified name of the class - method_signature: method signature of the starting point of the call graph - - Returns: List[Tuple[JMethodDetail, JMethodDetail]] - List of edges - """ - call_graph = self.__call_graph_using_symbol_table(qualified_class_name, method_signature) + @staticmethod + def _edges_out_of(cg: nx.DiGraph, qualified_class_name: str, method_signature: str | None) -> List[Tuple[JMethodDetail, JMethodDetail]]: if method_signature is None: - filter_criteria = {node for node in call_graph.nodes if node[1] == qualified_class_name} - else: - filter_criteria = {node for node in call_graph.nodes if tuple(node) == (method_signature, qualified_class_name)} - - graph_edges: List[Tuple[JMethodDetail, JMethodDetail]] = list() - for edge in call_graph.edges(nbunch=filter_criteria): - source: JMethodDetail = call_graph.nodes[edge[0]]["method_detail"] - target: JMethodDetail = call_graph.nodes[edge[1]]["method_detail"] - graph_edges.append((source, target)) - return graph_edges - - def __call_graph_using_symbol_table(self, qualified_class_name: str, method_signature: str, is_target_method: bool = False) -> nx.DiGraph: - """Should generate call graph using symbol table - Args: - qualified_class_name: qualified class name - method_signature: method signature - is_target_method: is the input method is a target method. By default, it is the source method - - Returns: - nx.DiGraph: call graph - """ - cg = nx.DiGraph() - sdg = None - if is_target_method: - sdg = self.__raw_call_graph_using_symbol_table_target_method(target_class_name=qualified_class_name, target_method_signature=method_signature) + seeds = [n for n, a in cg.nodes(data=True) if a["method_detail"].klass == qualified_class_name] else: - sdg = self.__raw_call_graph_using_symbol_table(qualified_class_name=qualified_class_name, method_signature=method_signature) - tsu = TreesitterJava() - edge_list = [ - ( - (jge.source.method.signature, jge.source.klass), - (jge.target.method.signature, jge.target.klass), - { - "type": jge.type, - "weight": jge.weight, - "calling_lines": tsu.get_calling_lines(jge.source.method.code, jge.target.method.signature), - }, - ) - for jge in sdg - ] - for jge in sdg: - cg.add_node( - (jge.source.method.signature, jge.source.klass), - method_detail=jge.source, - ) - cg.add_node( - (jge.target.method.signature, jge.target.klass), - method_detail=jge.target, - ) - cg.add_edges_from(edge_list) - return cg - - def __raw_call_graph_using_symbol_table_target_method(self, target_class_name: str, target_method_signature: str, cg=None) -> list[JGraphEdgesST]: - """Generates call graph using symbol table information given the target method and target class - Args: - qualified_class_name: qualified class name - method_signature: source method signature - cg: call graph + key = f"{qualified_class_name}.{method_signature}" + seeds = [key] if key in cg else [] + return [(cg.nodes[s]["method_detail"], cg.nodes[t]["method_detail"]) for s, t in cg.edges(seeds)] - Returns: - list[JGraphEdgesST]: list of call edges - """ - if cg is None: - cg = [] - target_method_details = self.get_method(qualified_class_name=target_class_name, method_signature=target_method_signature) - if target_method_details is None: - # The target method doesn't exist, so no edges into it can be constructed. - return cg - for class_name in self.get_all_classes(): - for method in self.get_all_methods_in_class(qualified_class_name=class_name): - method_details = self.get_method(qualified_class_name=class_name, method_signature=method) - if method_details is None: - # The symbol table momentarily disagreed with itself; skip this entry. - continue - for call_site in method_details.call_sites: - source_method_details = None - source_class = "" - callee_signature = "" - if call_site.callee_signature != "": - # pattern = r"\b(?:[a-zA-Z_][\w\.]*\.)+([a-zA-Z_][\w]*)\b|<[^>]*>" - # - # # Find the part within the parentheses - # start = call_site.callee_signature.find("(") + 1 - # end = call_site.callee_signature.rfind(")") - # - # # Extract the elements inside the parentheses - # elements = call_site.callee_signature[start:end].split(",") - # - # # Apply the regex to each element - # simplified_elements = [re.sub(pattern, r"\1", element.strip()) for element in elements] - # - # # Reconstruct the string with simplified elements - # callee_signature = f"{call_site.callee_signature[:start]}{', '.join(simplified_elements)}{call_site.callee_signature[end:]}" - callee_signature = call_site.callee_signature - - if call_site.receiver_type != "": - # call to any class - check if the target method exists in receiver type hierarchy - if self.get_class(qualified_class_name=call_site.receiver_type): - # Use hierarchy search to find the method (including inherited methods) - found_method, found_class = self.__find_method_in_hierarchy(call_site.receiver_type, callee_signature) - if found_method is not None and callee_signature == target_method_signature and found_class == target_class_name: - source_method_details = self.get_method(method_signature=method, qualified_class_name=class_name) - source_class = class_name - else: - # check if any method exists with the signature in the class (including inherited) even if the receiver type is blank - found_method, found_class = self.__find_method_in_hierarchy(class_name, callee_signature) - if found_method is not None and callee_signature == target_method_signature and found_class == target_class_name: - source_method_details = self.get_method(method_signature=method, qualified_class_name=class_name) - source_class = class_name - - if source_class != "" and source_method_details is not None: - source: JMethodDetail - target: JMethodDetail - type: str - weight: str - call_edge = JGraphEdgesST( - source=JMethodDetail(method_declaration=source_method_details.declaration, klass=source_class, method=source_method_details), - target=JMethodDetail(method_declaration=target_method_details.declaration, klass=target_class_name, method=target_method_details), - type="CALL_DEP", - weight="1", - ) - if call_edge not in cg: - cg.append(call_edge) - return cg + def get_class_call_graph(self, qualified_class_name: str, method_name: str | None = None) -> List[Tuple[JMethodDetail, JMethodDetail]]: + return self._edges_out_of(self.get_call_graph(), qualified_class_name, method_name) - def __find_method_in_hierarchy(self, qualified_class_name: str, method_signature: str) -> Tuple[JCallable | None, str]: - """Finds a method in the class hierarchy (including inherited methods). - - Ignores interface methods and only returns concrete implementations. + def get_class_call_graph_using_symbol_table(self, qualified_class_name: str, method_signature: str | None = None) -> List[Tuple[JMethodDetail, JMethodDetail]]: + """Edges out of a class (or one method) resolved from its call sites through the symbol + table alone — incomplete by construction: only receivers the symbol table can see, only + concrete implementations up the ``extends`` chain.""" + return self._edges_out_of(self._symbol_table_call_graph(qualified_class_name, method_signature), qualified_class_name, method_signature) - Args: - qualified_class_name (str): The qualified class name to start searching from. - method_signature (str): The method signature to find. + # -----[ symbol-table call graph (call sites → declarations) ]----- + def _symbol_table_call_graph(self, qualified_class_name: str, method_signature: str | None, is_target: bool = False) -> nx.DiGraph: + cg = nx.DiGraph() + lines = CallingLines() + edges = self._st_edges_into(qualified_class_name, method_signature) if is_target else self._st_edges_from(qualified_class_name, method_signature) + for source, target in edges: + src, dst = f"{source.klass}.{source.method.signature}", f"{target.klass}.{target.method.signature}" + cg.add_node(src, method_detail=source, kind="callable") + cg.add_node(dst, method_detail=target, kind="callable") + cg.add_edge(src, dst, type="CALL_DEP", weight=1, calling_lines=lines.of(source.method, target.method)) + return cg - Returns: - Tuple[JCallable | None, str]: A tuple of (method_details, declaring_class). - Returns (None, "") if the method is not found. - """ - # First, check if the method exists in the current class - klass = self.get_class(qualified_class_name=qualified_class_name) - method_details = self.get_method(method_signature=method_signature, qualified_class_name=qualified_class_name) - - # If found and it's not an interface, return it (concrete implementation) - if method_details is not None and klass is not None and not klass.is_interface: - return method_details, qualified_class_name - - # If not found or is an interface, check parent classes (extends) first - # This ensures we find concrete implementations before interface methods + def _st_edges_from(self, qualified_class_name: str, method_signature: str | None) -> Iterable[Tuple[JMethodDetail, JMethodDetail]]: + klass = self.get_class(qualified_class_name) + if klass is None: + return + if method_signature is None: + sources = list(klass.callables.values()) + else: + source = self.get_method(qualified_class_name, method_signature) + sources = [source] if source is not None else [] + for source in sources: + for call_site in source.call_sites: + target, target_class = self._resolve_call_site(qualified_class_name, call_site) + if target is not None: + yield self._detail(qualified_class_name, source), self._detail(target_class, target) + + def _st_edges_into(self, target_class_name: str, target_method_signature: str) -> Iterable[Tuple[JMethodDetail, JMethodDetail]]: + target = self.get_method(target_class_name, target_method_signature) + if target is None: + return + for owner, source in self._callables.values(): + for call_site in source.call_sites: + found, found_class = self._resolve_call_site(owner.qualified_name, call_site) + if found is not None and found_class == target_class_name and call_site.callee_signature == target_method_signature: + yield self._detail(owner.qualified_name, source), self._detail(target_class_name, target) + + def _resolve_call_site(self, owner_class_name: str, call_site: JCallSite) -> Tuple[JCallable | None, str]: + """The (declaration, declaring class) a call site names, or ``(None, "")``: an explicit + receiver type is followed only when it is a project class; an implicit receiver means the + owning class (and its ``extends`` chain).""" + if not call_site.callee_signature: + return None, "" + if call_site.receiver_type: + if self.get_class(call_site.receiver_type) is None: + return None, "" + return self._find_in_hierarchy(call_site.receiver_type, call_site.callee_signature) + return self._find_in_hierarchy(owner_class_name, call_site.callee_signature) + + def _find_in_hierarchy(self, qualified_class_name: str, method_signature: str) -> Tuple[JCallable | None, str]: + """The concrete declaration of ``method_signature`` on the class or up its ``extends`` + chain; interface declarations are not call-graph targets and are skipped.""" + klass = self.get_class(qualified_class_name) + method = self.get_method(qualified_class_name, method_signature) + if method is not None and klass is not None and not klass.is_interface: + return method, qualified_class_name if klass is not None: - # Check extended classes (these are more likely to have concrete implementations) - for parent_class in klass.extends_list: - parent_method, found_class = self.__find_method_in_hierarchy(parent_class, method_signature) - if parent_method is not None: - return parent_method, found_class - - # Only check implemented interfaces if no concrete implementation was found - # This is a fallback for cases where only the interface method exists - # for interface in klass.implements_list: - # interface_method, found_class = self.__find_method_in_hierarchy(interface, method_signature) - # if interface_method is not None: - # return interface_method, found_class - - # Do not return interface methods - only concrete implementations are included in call graph + for parent in klass.extends_list: + found, found_class = self._find_in_hierarchy(parent, method_signature) + if found is not None: + return found, found_class return None, "" - def __raw_call_graph_using_symbol_table(self, qualified_class_name: str, method_signature: str, cg=None) -> list[JGraphEdgesST]: - """Generates a call graph using symbol table information. - - Args: - qualified_class_name (str): The qualified class name. - method_signature (str): The source method signature. - cg (list[JGraphEdgesST], optional): Existing call graph edges. Defaults to None. + # -----[ classes / methods / fields ]----- + def get_all_classes(self) -> Dict[str, JType]: + return dict(self._types) - Returns: - list[JGraphEdgesST]: A list of call edges. - """ - if cg is None: - cg = [] - source_method_details = self.get_method(qualified_class_name=qualified_class_name, method_signature=method_signature) - # If the provided classname and method signature combination do not exist - if source_method_details is None: - return cg - for call_site in source_method_details.call_sites: - target_method_details = None - target_class = "" - callee_signature = "" - if call_site.callee_signature != "": - # Currently the callee signature returns the fully qualified type, whereas - # the key for JCallable does not. The below logic converts the fully qualified signature - # to the desider format. Only limitation is the nested generic type. - # pattern = r"\b(?:[a-zA-Z_][\w\.]*\.)+([a-zA-Z_][\w]*)\b|<[^>]*>" - # - # # Find the part within the parentheses - # start = call_site.callee_signature.find("(") + 1 - # end = call_site.callee_signature.rfind(")") - # - # # Extract the elements inside the parentheses - # elements = call_site.callee_signature[start:end].split(",") - # - # # Apply the regex to each element - # simplified_elements = [re.sub(pattern, r"\1", element.strip()) for element in elements] - # - # # Reconstruct the string with simplified elements - # callee_signature = f"{call_site.callee_signature[:start]}{', '.join(simplified_elements)}{call_site.callee_signature[end:]}" - callee_signature = call_site.callee_signature - - if call_site.receiver_type != "": - # call to any class - if self.get_class(qualified_class_name=call_site.receiver_type): - # Check for method in the receiver type and its hierarchy - tmd, found_class = self.__find_method_in_hierarchy(call_site.receiver_type, callee_signature) - if tmd is not None: - target_method_details = tmd - target_class = found_class - else: - # check if any method exists with the signature in the class (including inherited) even if the receiver type is blank - tmd, found_class = self.__find_method_in_hierarchy(qualified_class_name, callee_signature) - if tmd is not None: - target_method_details = tmd - target_class = found_class - - if target_class != "" and target_method_details is not None: - source: JMethodDetail - target: JMethodDetail - type: str - weight: str - call_edge = JGraphEdgesST( - source=JMethodDetail(method_declaration=source_method_details.declaration, klass=qualified_class_name, method=source_method_details), - target=JMethodDetail(method_declaration=target_method_details.declaration, klass=target_class, method=target_method_details), - type="CALL_DEP", - weight="1", - ) - if call_edge not in cg: - cg.append(call_edge) - # cg = self.__raw_call_graph_using_symbol_table(qualified_class_name=target_class, method_signature=target_method_details.signature, cg=cg) - return cg + def get_class(self, qualified_class_name: str) -> JType | None: + return self._types.get(qualified_class_name) - def get_class_call_graph(self, qualified_class_name: str, method_name: str | None = None) -> List[Tuple[JMethodDetail, JMethodDetail]]: - """Generates a call graph for a given class and (optionally) filters by a given method. + def get_all_methods_in_application(self) -> Dict[str, Dict[str, JCallable]]: + return {name: t.callable_declarations for name, t in self._types.items()} - Args: - qualified_class_name (str): The qualified name of the class. - method_name (str, optional): The name of the method in the class. + def get_all_methods_in_class(self, qualified_class_name: str) -> Dict[str, JCallable]: + klass = self.get_class(qualified_class_name) + if klass is None: + return {} + return {sig: c for sig, c in klass.callables.items() if not c.is_constructor} - Returns: - List[Tuple[JMethodDetail, JMethodDetail]]: An edge list of the call graph - for the given class and method. + def get_all_constructors(self, qualified_class_name: str) -> Dict[str, JCallable]: + klass = self.get_class(qualified_class_name) + if klass is None: + return {} + return {sig: c for sig, c in klass.callables.items() if c.is_constructor} - Notes: - The class name must be fully qualified, e.g., "org.example.MyClass" and not "MyClass". - """ - # If the method name is not provided, we'll get the call graph for the entire class. + def get_method(self, qualified_class_name: str, qualified_method_name: str) -> JCallable | None: + klass = self.get_class(qualified_class_name) + return klass.callables.get(qualified_method_name) if klass is not None else None - if method_name is None: - filter_criteria = {node for node in self.call_graph.nodes if node[1] == qualified_class_name} - else: - filter_criteria = {node for node in self.call_graph.nodes if tuple(node) == (method_name, qualified_class_name)} + def get_method_parameters(self, qualified_class_name: str, qualified_method_name: str) -> List[JCallableParameter]: + method = self.get_method(qualified_class_name, qualified_method_name) + return method.parameters if method is not None else [] - graph_edges: List[Tuple[JMethodDetail, JMethodDetail]] = list() - for edge in self.call_graph.edges(nbunch=filter_criteria): - source: JMethodDetail = self.call_graph.nodes[edge[0]]["method_detail"] - target: JMethodDetail = self.call_graph.nodes[edge[1]]["method_detail"] - graph_edges.append((source, target)) + def get_all_sub_classes(self, qualified_class_name: str) -> Dict[str, JType]: + return {name: t for name, t in self._types.items() if qualified_class_name in t.extends_list or qualified_class_name in t.implements_list} - return graph_edges + def get_all_fields(self, qualified_class_name: str) -> List[JField]: + klass = self.get_class(qualified_class_name) + return klass.field_declarations if klass is not None else [] - def remove_all_comments(self, src_code: str) -> str: - """Remove all comments in the source code. + def get_all_nested_classes(self, qualified_class_name: str) -> List[JType]: + klass = self.get_class(qualified_class_name) + return list(klass.types.values()) if klass is not None else [] - Args: - src_code (str): Original source code. + def get_extended_classes(self, qualified_class_name: str) -> List[str]: + klass = self.get_class(qualified_class_name) + return klass.extends_list if klass is not None else [] - Returns: - str: The same source code without comments. - """ - raise NotImplementedError("This function is not implemented yet.") + def get_implemented_interfaces(self, qualified_class_name: str) -> List[str]: + klass = self.get_class(qualified_class_name) + return klass.implements_list if klass is not None else [] + # -----[ entry points ]----- def get_all_entry_point_methods(self) -> Dict[str, Dict[str, JCallable]]: - """Should return a dictionary of all entry point methods in the Java code. - - Returns: - Dict[str, Dict[str, JCallable]]: A dictionary of all entry point methods in the Java code. - """ - methods = chain.from_iterable( - ((typename, method, callable) for method, callable in methods.items() if callable.is_entrypoint) for typename, methods in self.get_all_methods_in_application().items() - ) - return {typename: {method: callable for _, method, callable in group} for typename, group in groupby(methods, key=lambda x: x[0])} + result: Dict[str, Dict[str, JCallable]] = {} + for name, methods in self.get_all_methods_in_application().items(): + entrypoints = {sig: c for sig, c in methods.items() if c.is_entrypoint} + if entrypoints: + result[name] = entrypoints + return result def get_all_entry_point_classes(self) -> Dict[str, JType]: - """Should return a dictionary of all entry point classes in the Java code. - - Returns: - Dict[str, JType]: A dictionary of all entry point classes in the Java code, - with qualified class names as keys. - """ - - return {typename: klass for typename, klass in self.get_all_classes().items() if klass.is_entrypoint_class} - - def get_all_crud_operations(self) -> List[Dict[str, Union[JType, JCallable, List[JCRUDOperation]]]]: - """Should return a dictionary of all CRUD operations in the source code. - - Raises: - NotImplementedError: Raised when we do not support this function. - - Returns: - Dict[str, List[str]]: A dictionary of all CRUD operations in the source code. - """ - - crud_operations = [] - for class_name, class_details in self.get_all_classes().items(): - for method_name, method_details in class_details.callable_declarations.items(): - if len(method_details.crud_operations) > 0: - crud_operations.append({class_name: class_details, method_name: method_details, "crud_operations": method_details.crud_operations}) - return crud_operations - - def get_all_read_operations(self) -> List[Dict[str, Union[JType, JCallable, List[JCRUDOperation]]]]: - """Should return a list of all read operations in the source code. - - Raises: - NotImplementedError: Raised when we do not support this function. - - Returns: - List[Dict[str, Union[str, JCallable, List[CRUDOperation]]]]:: A list of all read operations in the source code. - """ - crud_read_operations = [] - for class_name, class_details in self.get_all_classes().items(): - for method_name, method_details in class_details.callable_declarations.items(): - if len(method_details.crud_operations) > 0: - crud_read_operations.append( - { - class_name: class_details, - method_name: method_details, - "crud_operations": [crud_op for crud_op in method_details.crud_operations if crud_op.operation_type == CRUDOperationType.READ], - } - ) - return crud_read_operations - - def get_all_create_operations(self) -> List[Dict[str, Union[JType, JCallable, List[JCRUDOperation]]]]: - """Should return a list of all create operations in the source code. - - Raises: - NotImplementedError: Raised when we do not support this function. - - Returns: - List[Dict[str, Union[str, JCallable, List[CRUDOperation]]]]: A list of all create operations in the source code. - """ - crud_create_operations = [] - for class_name, class_details in self.get_all_classes().items(): - for method_name, method_details in class_details.callable_declarations.items(): - if len(method_details.crud_operations) > 0: - crud_create_operations.append( - { - class_name: class_details, - method_name: method_details, - "crud_operations": [crud_op for crud_op in method_details.crud_operations if crud_op.operation_type == CRUDOperationType.CREATE], - } - ) - return crud_create_operations - - def get_all_update_operations(self) -> List[Dict[str, Union[JType, JCallable, List[JCRUDOperation]]]]: - """Should return a list of all update operations in the source code. - - Raises: - NotImplementedError: Raised when we do not support this function. - - Returns: - List[Dict[str, Union[str, JCallable, List[CRUDOperation]]]]: A list of all update operations in the source code. - """ - crud_update_operations = [] - for class_name, class_details in self.get_all_classes().items(): - for method_name, method_details in class_details.callable_declarations.items(): - if len(method_details.crud_operations) > 0: - crud_update_operations.append( - { - class_name: class_details, - method_name: method_details, - "crud_operations": [crud_op for crud_op in method_details.crud_operations if crud_op.operation_type == CRUDOperationType.UPDATE], - } - ) - - return crud_update_operations - - def get_all_delete_operations(self) -> List[Dict[str, Union[JType, JCallable, List[JCRUDOperation]]]]: - """Should return a list of all delete operations in the source code. - - Raises: - NotImplementedError: Raised when we do not support this function. - - Returns: - List[Dict[str, Union[str, JCallable, List[CRUDOperation]]]]: A list of all delete operations in the source code. - """ - crud_delete_operations = [] - for class_name, class_details in self.get_all_classes().items(): - for method_name, method_details in class_details.callable_declarations.items(): - if len(method_details.crud_operations) > 0: - crud_delete_operations.append( - { - class_name: class_details, - method_name: method_details, - "crud_operations": [crud_op for crud_op in method_details.crud_operations if crud_op.operation_type == CRUDOperationType.DELETE], - } - ) - return crud_delete_operations - - # Some APIs to process comments + return {name: t for name, t in self._types.items() if t.is_entrypoint_class} + + # -----[ CRUD (J-4) ]----- + def get_all_crud_operations(self) -> List[CRUDRow]: + raise CodeanalyzerExecutionException(CRUD_UNAVAILABLE) + + def get_all_create_operations(self) -> List[CRUDRow]: + raise CodeanalyzerExecutionException(CRUD_UNAVAILABLE) + + def get_all_read_operations(self) -> List[CRUDRow]: + raise CodeanalyzerExecutionException(CRUD_UNAVAILABLE) + + def get_all_update_operations(self) -> List[CRUDRow]: + raise CodeanalyzerExecutionException(CRUD_UNAVAILABLE) + + def get_all_delete_operations(self) -> List[CRUDRow]: + raise CodeanalyzerExecutionException(CRUD_UNAVAILABLE) + + # -----[ repository artifacts — the shared Py* models, as the generic ABC promises ]----- + def get_artifacts(self) -> Dict[str, PyArtifact]: + """Every non-code artifact (see :meth:`AnalysisBackend.get_artifacts`), keyed by repo-relative + path as the wire keys them. ``JArtifact.text_truncated`` has no home on the shared model and + is not carried; read it off ``JApplication.artifacts`` when it matters.""" + return {path: PyArtifact(**a.model_dump(exclude={"config_keys", "text_truncated"}), config_keys=[PyConfigKey(**ck.model_dump()) for ck in a.config_keys]) for path, a in self.application.artifacts.items()} + + def get_dependencies(self, *, direct_only: bool = False, ecosystem: str | None = None, declared_in: str | None = None) -> List[PyDependency]: + """Every declared dependency, optionally filtered (see :meth:`AnalysisBackend.get_dependencies`). + The Maven ``group`` coordinate has no home on the shared model and is not carried; read it off + ``JApplication.dependencies`` when ``name`` alone is ambiguous.""" + deps = [PyDependency(**d.model_dump(exclude={"group"})) for d in self.application.dependencies] + if direct_only: + deps = [d for d in deps if d.direct] + if ecosystem is not None: + deps = [d for d in deps if d.ecosystem == ecosystem] + if declared_in is not None: + deps = [d for d in deps if d.declared_in == declared_in] + return deps + + def get_config_keys(self) -> Dict[str, PyConfigKey]: + """Every configuration key flattened out of the config-bearing artifacts, keyed + ``"@key/"`` (``pom.xml@key/project.artifactId``). + + That key is the analyzer's own id with its ``can://artifact//`` prefix dropped: the + application name belongs to the run, not to the key, so keying by the raw id made the two + backends share **zero** keys whenever the graph was emitted under a different ``--app-name`` + than the local run passes (the SDK passes the project directory's name). ``can://`` ids also + stay off the public surface (E6); the id is still on ``PyConfigKey.id``. + """ + return {f"{path}@key/{ck.key}": PyConfigKey(**ck.model_dump()) for path, a in self.application.artifacts.items() for ck in a.config_keys} + + def get_config_uses(self, key: str | None = None) -> List[PyConfigUseEdge]: + """Always empty: codeanalyzer-java 3.0.1 emits no code-to-config edges (there is no + ``config_uses`` on the Java wire), so there is nothing to filter by ``key``.""" + return [] + + def get_unresolved_config_reads(self) -> List[PyConfigRead]: + """Always empty: codeanalyzer-java 3.0.1 has no config-read detector (no ``config_reads`` + on the Java wire).""" + return [] + + # -----[ comments ]----- def get_comments_in_a_method(self, qualified_class_name: str, method_signature: str) -> List[JComment]: - """Get all comments in a method. - - Args: - qualified_class_name (str): Qualified name of the class. - method_signature (str): Signature of the method. - - Returns: - List[str]: List of comments in the method. Empty list if the method is not found. - """ - callable = self.get_method(qualified_class_name, method_signature) - return callable.comments if callable is not None else [] + method = self.get_method(qualified_class_name, method_signature) + return method.comments if method is not None else [] def get_comments_in_a_class(self, qualified_class_name: str) -> List[JComment]: - """Get all comments in a class. - - Args: - qualified_class_name (str): Qualified name of the class. - - Returns: - List[str]: List of comments in the class. Empty list if the class is not found. - """ klass = self.get_class(qualified_class_name) return klass.comments if klass is not None else [] def get_comment_in_file(self, file_path: str) -> List[JComment]: - """Get all comments in a file. - - Args: - file_path (str): Path to the file. - - Returns: - List[str]: List of comments in the file. - """ - compilation_unit = self.get_symbol_table().get(file_path, None) - if compilation_unit is None: + unit = self.application.symbol_table.get(file_path) + if unit is None: raise CodeanalyzerExecutionException(f"File {file_path} not found in the symbol table.") - return compilation_unit.comments + return unit.comments def get_all_comments(self) -> Dict[str, List[JComment]]: - """Get all comments in the Java application. + return {path: unit.comments for path, unit in self.application.symbol_table.items()} - Returns: - Dict[str, List[str]]: Dictionary of file paths and their corresponding comments. - """ - comments = {} - for file_path, _ in self.get_symbol_table().items(): - comments[file_path] = self.get_comment_in_file(file_path) - return comments - - def get_all_docstrings(self) -> List[Tuple[str, JComment]]: - """Get all docstrings in the Java application. - - Returns: - Dict[str, List[str]]: Dictionary of file paths and their corresponding docstrings. - """ + def get_all_docstrings(self) -> Dict[str, List[JComment]]: docstrings = {} - for file_path, list_of_comments in self.get_all_comments().items(): - javadoc_comments = [docstring for docstring in list_of_comments if docstring.is_javadoc] - if javadoc_comments: - docstrings[file_path] = javadoc_comments - + for path, comments in self.get_all_comments().items(): + javadoc = [c for c in comments if c.is_javadoc] + if javadoc: + docstrings[path] = javadoc return docstrings + + def remove_all_comments(self, src_code: str) -> str: + raise NotImplementedError("This function is not implemented yet.") diff --git a/cldk/analysis/java/codeanalyzer/jar/.gitignore b/cldk/analysis/java/codeanalyzer/jar/.gitignore deleted file mode 100644 index 9400ddc7..00000000 --- a/cldk/analysis/java/codeanalyzer/jar/.gitignore +++ /dev/null @@ -1 +0,0 @@ -!codeanalyzer-*.jar diff --git a/cldk/analysis/java/codeanalyzer/jar/codeanalyzer-2.4.1.jar b/cldk/analysis/java/codeanalyzer/jar/codeanalyzer-2.4.1.jar deleted file mode 100644 index 3495cdce..00000000 Binary files a/cldk/analysis/java/codeanalyzer/jar/codeanalyzer-2.4.1.jar and /dev/null differ diff --git a/cldk/analysis/java/java_analysis.py b/cldk/analysis/java/java_analysis.py index 230a6f9b..c5302118 100644 --- a/cldk/analysis/java/java_analysis.py +++ b/cldk/analysis/java/java_analysis.py @@ -17,16 +17,14 @@ """Java analysis facade module. This module provides the :class:`JavaAnalysis` class, which serves as the -primary high-level interface for performing static analysis on Java projects -and source files. It combines Tree-sitter-based parsing with the CodeAnalyzer -backend to provide comprehensive code analysis capabilities. +primary high-level interface for performing static analysis on Java projects. +It combines Tree-sitter-based parsing with the CodeAnalyzer backend to provide +comprehensive code analysis capabilities. -The analysis supports two modes of operation: - - **Project mode**: Analyze an entire Java project directory, providing - access to cross-file analysis features like call graphs and class - hierarchies. - - **Source code mode**: Analyze a single Java source code string, useful - for quick syntactic analysis without a full project structure. +The analysis operates on a project directory (cross-file call graphs, class +hierarchies, the symbol table). The 1.x single-file ``source_code`` mode was +removed in 2.0 (spec leg 3, J-10): pass the project directory, or hand a source +string to :class:`~cldk.analysis.commons.treesitter.TreesitterJava` directly. Key capabilities include: - Symbol table extraction (classes, methods, fields, imports) @@ -46,6 +44,8 @@ - :class:`~cldk.analysis.java.codeanalyzer.JCodeanalyzer`: Backend implementation. """ +from __future__ import annotations + from pathlib import Path from typing import Dict, List, Tuple, Set, Union import networkx as nx @@ -56,11 +56,16 @@ from cldk.analysis.commons.treesitter import TreesitterJava from cldk.models.java import JCallable from cldk.models.java import JApplication -from cldk.models.java.models import JCRUDOperation, JComment, JCompilationUnit, JMethodDetail, JType, JField +from cldk.models.java.models import JCallableParameter, JCRUDOperation, JComment, JCompilationUnit, JMethodDetail, JType, JField from cldk.analysis.java.codeanalyzer import JCodeanalyzer from cldk.analysis.java.neo4j import JNeo4jBackend from cldk.analysis.java.backend import JavaAnalysisBackend +#: The annotations that *declare* a test method, by simple name — JUnit 4/5 and TestNG. A lifecycle +#: annotation (``@BeforeEach``, ``@AfterAll``) is deliberately not here: it marks a fixture, not a +#: test. Read by :meth:`JavaAnalysis.get_test_methods`. +_TEST_ANNOTATIONS = frozenset({"Test", "ParameterizedTest", "RepeatedTest", "TestFactory", "TestTemplate"}) + class JavaAnalysis: """Analysis facade for Java code. @@ -69,12 +74,9 @@ class JavaAnalysis: on Java projects and source files. It combines Tree-sitter-based parsing for syntactic analysis with the CodeAnalyzer backend for semantic analysis. - The facade supports two modes of operation: - - **Project mode**: When initialized with ``project_dir``, provides full - analysis capabilities including cross-file call graphs, class hierarchies, - and symbol tables. - - **Source code mode**: When initialized with ``source_code``, provides - syntactic analysis capabilities like parsing and AST extraction. + The facade is initialized with ``project_dir`` and provides full analysis + capabilities including cross-file call graphs, class hierarchies, and symbol + tables; the single-file ``source_code`` mode was removed in 2.0. Key features: - Symbol table access with classes, methods, and fields @@ -87,7 +89,6 @@ class JavaAnalysis: Attributes: project_dir (str | Path | None): Path to the Java project directory. - source_code (str | None): Java source code string for single-file mode (deprecated). analysis_level (str): The depth of analysis performed. eager_analysis (bool): Whether to force regeneration of analysis. target_files (List[str] | None): Specific files to analyze. @@ -103,7 +104,6 @@ class JavaAnalysis: def __init__( self, project_dir: str | Path | None, - source_code: str | None, analysis_level: str, target_files: List[str] | None, eager_analysis: bool, @@ -111,22 +111,16 @@ def __init__( ) -> None: """Initialize the Java analysis facade. - Creates a new analysis facade for Java code. Either ``project_dir`` or - ``source_code`` must be provided, but not both. + Creates a new analysis facade for Java code. Args: project_dir: Absolute or relative path to the Java project directory. The directory should contain Java source files (``.java``). - When provided, enables full analysis including call graphs. - Mutually exclusive with ``source_code``. - source_code: Java source code string for single-file analysis. - Useful for quick syntactic analysis without a project structure. - Mutually exclusive with ``project_dir``. Deprecated; will be - removed in a future release. - analysis_level: The depth of analysis to perform. Common values: - - ``"symbol_table"``: Extract symbols only (faster) - - ``"call_graph"``: Full call graph analysis (comprehensive) - See :class:`~cldk.analysis.AnalysisLevel` for all options. + Optional only for the read-only Neo4j backend. + analysis_level: The depth of analysis to perform — any + :class:`~cldk.analysis.AnalysisLevel`: ``"symbol_table"`` (1), + ``"call_graph"`` (2), ``"program_dependency_graph"`` (3), + ``"system_dependency_graph"`` (4); the level reaches the analyzer as ``-a``. target_files: Optional list of specific file paths (relative to ``project_dir``) to include in the analysis. When provided, only these files are analyzed, improving performance for @@ -145,7 +139,6 @@ def __init__( """ self.project_dir = project_dir - self.source_code = source_code self.analysis_level = analysis_level self.eager_analysis = eager_analysis self.target_files = target_files @@ -164,14 +157,12 @@ def __init__( application_name=application_name, ) else: - # The config only carries the cache root. analysis.json is cached under /java - # (None in source_code mode, where the analyzer streams results over a pipe). + # The config only carries the cache root. analysis.json is cached under /java. cache_path = cache_subdir(self.backend_config.cache_dir, project_dir, "java") if cache_path is not None: cache_path.mkdir(parents=True, exist_ok=True) self.backend = JCodeanalyzer( project_dir=self.project_dir, - source_code=self.source_code, eager_analysis=self.eager_analysis, analysis_level=self.analysis_level, analysis_json_path=cache_path, @@ -279,16 +270,10 @@ def get_application_view(self) -> JApplication: - Project-level metadata - Aggregated statistics about the codebase - Raises: - NotImplementedError: If called in single-file mode (``source_code`` - was provided instead of ``project_dir``). - See Also: :meth:`get_symbol_table`: For direct access to the symbol table. :meth:`get_compilation_units`: For a list of compilation units. """ - if self.source_code: - raise NotImplementedError("Support for this functionality has not been implemented yet.") return self.backend.get_application_view() def get_symbol_table(self) -> Dict[str, JCompilationUnit]: @@ -405,16 +390,17 @@ def get_call_graph(self) -> nx.DiGraph: relationships across the entire project. Each node represents a method, and each edge represents a call from one method to another. - The call graph requires ``analysis_level`` to be set to ``"call_graph"`` - during initialization for accurate results. + The call graph requires ``analysis_level`` of at least ``"call_graph"``; + below it the graph is empty. Returns: A ``networkx.DiGraph`` where: - - Nodes represent methods with attributes containing method - metadata (class name, signature, etc.) + - Nodes are keyed by the string ``"."`` (e.g. + ``"com.acme.Svc.run(java.lang.String)"``), with a + :class:`~cldk.models.java.JMethodDetail` under ``method_detail`` + and ``kind="callable"`` - Edges represent call relationships, directed from caller - to callee - - Edge attributes may include call site information + to callee, with ``type``, ``weight`` and ``calling_lines`` See Also: :meth:`get_callers`: For finding callers of a specific method. @@ -435,15 +421,9 @@ def get_call_graph_json(self) -> str: including compilation units, classes, methods, and call relationships. - Raises: - NotImplementedError: If called in single-file mode (``source_code`` - was provided instead of ``project_dir``). - See Also: :meth:`get_call_graph`: For the graph object directly. """ - if self.source_code: - raise NotImplementedError("Producing a call graph over a single file is not implemented yet.") return self.backend.get_call_graph_json() def get_callers(self, target_class_name: str, target_method_declaration: str, using_symbol_table: bool = False) -> Dict: @@ -468,17 +448,10 @@ def get_callers(self, target_class_name: str, target_method_declaration: str, us - Call site locations (file and line) - Caller class information - Raises: - NotImplementedError: If called in single-file mode (``source_code`` - was provided instead of ``project_dir``). - See Also: :meth:`get_callees`: For the reverse direction (what a method calls). :meth:`get_call_graph`: For the complete call relationship graph. """ - - if self.source_code: - raise NotImplementedError("Generating all callers over a single file is not implemented yet.") return self.backend.get_all_callers(target_class_name, target_method_declaration, using_symbol_table) def get_callees(self, source_class_name: str, source_method_declaration: str, using_symbol_table: bool = False) -> Dict: @@ -503,16 +476,10 @@ def get_callees(self, source_class_name: str, source_method_declaration: str, us - Target class information - Call site locations within the source method - Raises: - NotImplementedError: If called in single-file mode (``source_code`` - was provided instead of ``project_dir``). - See Also: :meth:`get_callers`: For the reverse direction (who calls a method). :meth:`get_call_graph`: For the complete call relationship graph. """ - if self.source_code: - raise NotImplementedError("Generating all callees over a single file is not implemented yet.") return self.backend.get_all_callees(source_class_name, source_method_declaration, using_symbol_table) def get_methods(self) -> Dict[str, Dict[str, JCallable]]: @@ -649,17 +616,24 @@ def get_method(self, qualified_class_name: str, qualified_method_name: str) -> J analyzed information about the method. Returns ``None`` if the method is not found. + Note: + Two fields depend on which backend answered. On the + ``analysis.json`` backend ``code`` is the **body block** and + ``body`` holds every body node. On the Neo4j backend ``code`` is + the whole **declaration** (it *ends with* the body block, because + the graph projects one line range per callable and no + ``body_span``) and ``body`` holds the ``call`` nodes only — about + 30% of the graph's body nodes, which is what ``call_sites`` needs + and all it needs. + See Also: :meth:`get_methods_in_class`: For all methods of a class. :meth:`get_method_parameters`: For just the parameter list. """ return self.backend.get_method(qualified_class_name, qualified_method_name) - def get_method_parameters(self, qualified_class_name: str, qualified_method_name: str) -> List[str]: - """Return the parameter types for a specific method. - - Retrieves the list of parameter type names defined in the method - signature. + def get_method_parameters(self, qualified_class_name: str, qualified_method_name: str) -> List[JCallableParameter]: + """Return the parameters of a specific method. Args: qualified_class_name: The fully qualified name of the class @@ -667,9 +641,10 @@ def get_method_parameters(self, qualified_class_name: str, qualified_method_name qualified_method_name: The method signature to get parameters for. Returns: - A list of parameter type names as strings, in the order they - appear in the method signature. Returns an empty list if the - method is not found or has no parameters. + The :class:`~cldk.models.java.models.JCallableParameter` objects + (name, type, annotations, position), in signature order. Returns an + empty list if the method is not found or has no parameters. (1.x + annotated this ``List[str]``; it always returned the objects.) See Also: :meth:`get_method`: For complete method information. @@ -960,19 +935,16 @@ def remove_all_comments(self) -> str: from the source code, including Javadoc comments. This is useful for code analysis that should ignore comment content. - Returns: - A string containing the source code with all comments removed. - Whitespace where comments were removed may be preserved or - collapsed depending on the implementation. - - Note: - This method operates on the ``source_code`` provided during - initialization. It requires single-file mode. + Raises: + NotImplementedError: always. This accessor only ever operated on the + ``source_code`` given to the 1.x constructor, and that single-file mode + was removed in 2.0 (J-10). Pass the source to + :meth:`TreesitterJava.remove_all_comments` directly. See Also: :meth:`get_all_comments`: For extracting comments instead. """ - return self.backend.remove_all_comments(self.source_code) + raise NotImplementedError("single-file source mode was removed in 2.0; pass the source to TreesitterJava.remove_all_comments directly") def get_methods_with_annotations(self, annotations: List[str]) -> Dict[str, List[Dict]]: """Return methods decorated with specific annotations. @@ -1003,24 +975,34 @@ def get_methods_with_annotations(self, annotations: List[str]) -> Dict[str, List def get_test_methods(self) -> Dict[str, str]: """Return methods identified as test methods. - Finds all test methods in the source code by looking for methods - annotated with common test framework annotations (e.g., ``@Test`` - from JUnit). + A callable is a test method when one of its own annotations is a test-declaring one: + ``@Test`` (JUnit 4/5, TestNG), ``@ParameterizedTest``, ``@RepeatedTest``, ``@TestFactory`` + or ``@TestTemplate``. The annotation is matched by simple name, so a fully qualified + spelling (``@org.junit.Test``) matches too, and its arguments are ignored — the same + marker rule the spec's J-5 gives ``get_decorated_callables``. - Returns: - A dictionary mapping test method signatures to their source - code bodies. + This reads the **analyzer's own** annotations off the model rather than re-parsing a + module's ``source``, so it answers identically on both backends: a Neo4j-backed analysis + carries no module ``source`` at all (``JCompilationUnit.source`` is ``""``), and the + source-parsing version returned ``{}`` there — an empty reading as "this application has + no tests" on an application with thousands. - Note: - This method operates on the ``source_code`` provided during - initialization. It requires single-file mode. + Returns: + A dictionary mapping ``"."`` — the call-graph node key of J-1, + unique application-wide — to the callable's ``code``. Note that ``code`` is the body + block off ``analysis.json`` and the whole declaration off the Neo4j projection + (:attr:`~cldk.models.java.models.JCallable.code`). See Also: :meth:`get_methods_with_annotations`: For finding methods with any annotation. """ - - return self.treesitter_java.get_test_methods(source_class_code=self.source_code) + return { + f"{klass}.{signature}": callable_.code + for klass, methods in self.get_methods().items() + for signature, callable_ in methods.items() + if any(d.name.rsplit(".", 1)[-1] in _TEST_ANNOTATIONS for d in callable_.decorators) + } def get_calling_lines(self, target_method_name: str) -> List[int]: """Return line numbers where a method is called. @@ -1151,10 +1133,14 @@ def get_all_delete_operations(self) -> List[Dict[str, Union[JType, JCallable, Li # Some APIs to process comments def get_comments_in_a_method(self, qualified_class_name: str, method_signature: str) -> List[JComment]: - """Return all comments contained within a specific method. + """Return the method's own comment. - Retrieves all comment nodes (single-line, multi-line, and Javadoc) - that appear within the body of the specified method. + **Not** every comment inside the body: on both backends this is the + analyzer's per-declaration comment list, which holds the comment + immediately above the declaration and nothing else (at most one; 70 of + the 128 callables in the committed ``-a 4`` fixture have one, 65 of + them javadoc). Comments *inside* a method body reach the SDK only + through :meth:`get_comment_in_file`, which reports the whole file's. Args: qualified_class_name: The fully qualified name of the class @@ -1165,6 +1151,14 @@ def get_comments_in_a_method(self, qualified_class_name: str, method_signature: A list of :class:`~cldk.models.java.JComment` objects found within the method body. Returns empty list if method not found. + Note: + On a backend whose source keeps only per-declaration javadoc — the + Neo4j backend — this narrows to **the method's javadoc alone**: a + strictly smaller set than every comment in the body, and still a + real answer about a real declaration, which is why this accessor + narrows where :meth:`get_all_comments` and + :meth:`get_comment_in_file` refuse (J-16). + See Also: :meth:`get_comments_in_a_class`: For class-level comments. :meth:`get_all_comments`: For all comments in the project. @@ -1172,11 +1166,13 @@ def get_comments_in_a_method(self, qualified_class_name: str, method_signature: return self.backend.get_comments_in_a_method(qualified_class_name, method_signature) def get_comments_in_a_class(self, qualified_class_name: str) -> List[JComment]: - """Return all comments contained within a specific class. + """Return the class's own comment. - Retrieves all comment nodes that appear within the class body, - including Javadoc comments, method-level comments, and inline - comments. + **Not** the comments inside the class body: on both backends this is + the type declaration's own comment list — the comment immediately + above ``class Foo``. A method's comment is on + :meth:`get_comments_in_a_method`, and an inline comment in a body is + on neither; :meth:`get_comment_in_file` reports the whole file's. Args: qualified_class_name: The fully qualified name of the class. @@ -1185,6 +1181,10 @@ def get_comments_in_a_class(self, qualified_class_name: str) -> List[JComment]: A list of :class:`~cldk.models.java.JComment` objects found within the class. Returns empty list if class not found. + Note: + Narrows to the class's javadoc alone on a javadoc-only backend, in + exactly the way :meth:`get_comments_in_a_method` does (J-16). + See Also: :meth:`get_comments_in_a_method`: For method-specific comments. :meth:`get_comment_in_file`: For file-level comments. @@ -1204,6 +1204,12 @@ def get_comment_in_file(self, file_path: str) -> List[JComment]: A list of :class:`~cldk.models.java.JComment` objects found in the file. Returns empty list if file not found. + Raises: + CodeanalyzerExecutionException: If the backend's source carries no + file-level comments at all — the Neo4j projection does not — + naming what is missing and what to read instead. An empty list + would read as "this file has no comments" (J-16). + See Also: :meth:`get_all_comments`: For comments across all files. """ @@ -1219,6 +1225,10 @@ def get_all_comments(self) -> Dict[str, List[JComment]]: A dictionary mapping file paths (strings) to lists of :class:`~cldk.models.java.JComment` objects. + Raises: + CodeanalyzerExecutionException: As :meth:`get_comment_in_file` + does, and for the same reason (J-16). + See Also: :meth:`get_all_docstrings`: For Javadoc comments only. """ @@ -1236,6 +1246,14 @@ def get_all_docstrings(self) -> Dict[str, List[JComment]]: :class:`~cldk.models.java.JComment` objects where ``is_javadoc`` is ``True``. + Note: + *Which* javadoc depends on the backend: the ``analysis.json`` + backend reports each compilation unit's own comment list, holding + the **file-level** javadoc; the Neo4j backend reports the javadoc of + each **declaration** in the file (type, callable, field, enum + constant, record component). Both are javadoc keyed by file, and + they are different sets for the same file (J-16). + See Also: :meth:`get_all_comments`: For all comment types. """ diff --git a/cldk/analysis/java/neo4j/neo4j_backend.py b/cldk/analysis/java/neo4j/neo4j_backend.py index d8ce3b38..0e5f9ff1 100644 --- a/cldk/analysis/java/neo4j/neo4j_backend.py +++ b/cldk/analysis/java/neo4j/neo4j_backend.py @@ -14,71 +14,133 @@ # limitations under the License. ################################################################################ -"""Neo4j-backed Java analysis backend (read-only Cypher client). - -A drop-in alternative to :class:`~cldk.analysis.java.codeanalyzer.JCodeanalyzer`: it exposes the -**same query method surface** (the 36 methods of :class:`JavaAnalysisBackend`) so the -:class:`~cldk.analysis.java.JavaAnalysis` facade can delegate to either one, but instead of running -the analyzer JAR it **reconstructs the canonical ``JApplication`` from a Neo4j graph** (the one -``codeanalyzer-java`` >= 2.4.0 emits with ``--emit neo4j``) and then answers every query with the -*identical* logic the in-memory backend uses. Mirrors the Python / TypeScript Neo4j backends. - -It is purely a **query client**: it never builds the graph and has no dependency on the analyzer JAR, -a JDK, or the project sources. The graph is populated out of band — e.g. a job running -``codeanalyzer-java --emit neo4j`` — and the SDK only polls it. - -Reconstruction strategy (see :mod:`reconstruct`): the backend bulk-fetches every node + relationship -for the application in a handful of Cypher queries, groups children by parent, builds an -``analysis.json``-shaped dict, and hands it to ``JApplication(**payload)`` — the same constructor -path as ``JCodeanalyzer._init_japplication``. With ``self.application`` and ``self.call_graph`` -populated, the 36 query methods are the same code the in-memory backend runs. - -Identity / scoping model (must match the emitter; see ``codeanalyzer-java/schema.neo4j.json``): -``:JType`` (id = fqn) and ``:JCallable`` (id = ``#``) share a ``:JSymbol`` label; -compilation units are ``:JCompilationUnit`` keyed by ``file_key`` (== file path == symbol-table key); -call edges are ``(:JCallable)-[:J_CALLS {type, weight, source_kind, destination_kind}]->(:JCallable)``; -every project-owned node carries a ``_module`` provenance prop, so one DB can host several apps, all -scoped under ``(:JApplication {name})-[:J_HAS_UNIT]->(:JCompilationUnit)``. - -Parity: this backend reconstructs everything the graph actually contains identically to the -in-memory ``JCodeanalyzer`` (verified on the daytrader8 sample — 97% of checks, the rest being the -caveats below). The ``codeanalyzer-java`` **2.4.0** emitter had three projection gaps — fields all -collapsing to one ``#field#null`` node, imports reduced to ``:JPackage``, and ``J_CALLS`` -materializing only a fraction of the call graph — all **fixed in 2.4.1** -(codeanalyzer-java#156/#157/#158), the version the SDK now bundles (its release workflow fetches the -latest codeanalyzer-java jar). So a graph emitted by a current analyzer is a complete projection. - -Inherent caveats (present even on a complete graph, NOT query-layer bugs): - -* ``J_CALLS`` only links resolved app callables, so call edges to external/library targets (which the - in-memory backend keeps as synthetic nodes) are absent; -* the call graph is built by a separate analyzer run from the in-memory backend's ``analysis.json``, - so the two can differ by run-to-run WALA variance; -* a ``:JType``'s ``is_class_or_interface_declaration`` / ``is_concrete_class`` flags are not - projected (only the ``kind`` discriminator is); an absent singular ``comment`` rehydrates to - ``None``. +"""Neo4j-backed Java analysis backend (read-only Cypher client) on the codeanalyzer-java 3.0.1 +graph vocabulary. + +A drop-in alternative to :class:`~cldk.analysis.java.codeanalyzer.JCodeanalyzer`: the same query +surface, answered over a live graph that ``codeanalyzer-java --emit neo4j`` populated out of band. +This class never writes and needs neither the analyzer JAR, a JDK, nor the project sources. + +**The graph it reads** (``schema.neo4j.json`` at the 3.0.1 tag, contract ``2.0.0``, verified +against the reference graph): ``:JApplication`` is keyed by **``name``** and stamps +``analyzer_version``; every project-owned node carries a ``can://java//…`` ``id`` and the +marker label ``:JCanNode``. ``:JModule`` holds the repo-relative path in ``file_key``; +``:JType``/``:JCallable``/``:JExternal`` share the merge label ``:JSymbol`` and are told apart by +their own label plus ``kind``; ``:JField``, ``:JVariable``, ``:JEnumConstant``, +``:JRecordComponent`` and ``:JBodyNode`` are keyed by ``id``. Containment is ``J_HAS_MODULE`` / +``J_DECLARES`` / ``J_HAS_METHOD`` / ``J_HAS_FIELD`` / ``J_DECLARES_VAR`` / +``J_HAS_ENUM_CONSTANT`` / ``J_HAS_RECORD_COMPONENT``; annotations are ``J_ANNOTATED_BY``; a call +site is a ``:JBodyNode {kind:'call'}`` under ``J_HAS_BODY_NODE`` resolving over ``J_RESOLVES_TO``; +calls are ``J_CALLS {weight, prov}``. **There is no ``_module`` property anywhere**, and none of the +pre-3.0.1 (schema v1) vocabulary this backend used to read (``:JCompilationUnit``, ``J_HAS_UNIT``, +``J_HAS_CALLABLE``, ``:JParameter``, ``:JCallSite``, ``:JComment``, the CRUD labels) exists — a +graph that still speaks it is refused at attach by :meth:`_probe_schema` (J-9). + +**Scope.** Java has exactly one id namespace, so the application scope is the single prefix +``can://java//`` that :func:`_scoped` spells, or the ``:JApplication {name: $app}`` anchor a +statement walks out from. Nothing else distinguishes two applications in one database: a module +``file_key``, a qualified class name and a method signature are all shared vocabulary. + +**Seek labels.** Every statement anchors on the bare specific label; ``:JCanNode`` is used +nowhere. Not because the bare label always seeks — ``:JCallable`` owns no id index at all (only a +range index on ``name`` and the ``code``/``docstring`` fulltext), so bare ``:JCallable`` plans a +label scan — but because of what the statements here actually are. The two prefix-scoped ones both +fan out over relationships from every matched callable, and measured on ThingsBoard the traversal +dominates: swapping the anchor moves the wall clock by under 1% while ``:JCanNode`` adds a quarter +again as many db hits (5.65M against 4.55M on the call sites, 1.89M against 0.73M on the call +edges). Everything else is anchored on ``(:JApplication {name: $app})`` and never scans at all. +``:JCanNode``'s own index is not a constraint and spans 615,329 nodes, so where it *is* the only +seek it still loses — 118 ms against 24 on a whole-application prefix; it wins only a per-module +prefix, which no statement here issues. See ``test_no_statement_anchors_on_the_marker_label`` and +the table in Task 3 of the leg-3a plan. + +**Strategy.** Unlike the Python and TypeScript Neo4j backends, which answer each accessor with its +own statement, this one rebuilds the canonical :class:`JApplication` from the graph and then answers +every query with the *same* logic the in-memory backend runs over the same models. The application +is built on first use, not at attach, and cached — **nine round trips in all**: three at attach (the +relationship-type fingerprint, the version probe, the module fetch) and six on first use (one +containment-subtree traversal instead of one query per parent, then call sites, imports, call edges, +artifacts and dependencies). + +**Lossiness** relative to the in-memory backend (the projection's, not this client's; see +:mod:`reconstruct` for the per-node detail): a module carries no ``source`` and no span, so +``JCompilationUnit.code`` is ``""`` and only a *callable's* text survives — as its whole +declaration, where the local backend's ``code`` is the body block; comments exist only as one +``docstring`` per declaration, so file-level comments are not projected at all +(:meth:`get_all_comments` and :meth:`get_comment_in_file` raise rather than answer with a smaller +set claiming to be every comment); ``JCallable.body`` holds the ``call`` nodes only, without their +``arguments`` or end columns; ``cfg``/``cdg``/``ddg``/``summary``, ``param_in``/``param_out`` and +``type_parameters`` are not rebuilt in 3a. Parameters, by contrast, round-trip exactly: +``JCallable.parameters_json`` is the analyzer's own serialisation of the list. + +``--emit neo4j`` always runs at level 4 with external calls forced, so this graph carries ``J_CALLS`` +edges to ``:JExternal`` targets that no ``analysis.json`` holds. :meth:`get_call_graph` keeps the 1.x +callable-only graph and drops them (``get_external_symbols`` arrives in 3b). """ from __future__ import annotations import json import logging -from itertools import chain, groupby -from typing import Any, Dict, List, Tuple, Union +import re +from collections import defaultdict +from functools import cached_property +from typing import Any, Dict, FrozenSet, Iterable, List, Tuple import networkx as nx -from cldk.analysis.commons.treesitter import TreesitterJava -from cldk.analysis.java.backend import JavaAnalysisBackend +from cldk.analysis.java.backend import CRUD_UNAVAILABLE, CallingLines, CRUDRow, JavaAnalysisBackend, duplicate_type_name, unhomed_endpoint from cldk.analysis.java.neo4j import reconstruct as R from cldk.models.java import JGraphEdges -from cldk.models.java.enums import CRUDOperationType -from cldk.models.java.models import JApplication, JCRUDOperation, JCallable, JCallableParameter, JComment, JField, JMethodDetail, JType, JCompilationUnit, JGraphEdgesST -from cldk.utils.exceptions.exceptions import CodeanalyzerExecutionException +from cldk.models.java.models import ( + JApplication, + JBodyNode, + JCallable, + JCallableParameter, + JCallGraphEdge, + JCallSite, + JComment, + JCompilationUnit, + JDecorator, + JField, + JMethodDetail, + JType, +) +from cldk.models.python import PyArtifact, PyConfigKey, PyConfigRead, PyConfigUseEdge, PyDependency +from cldk.utils.exceptions.exceptions import CodeanalyzerExecutionException, GraphSchemaMismatch logger = logging.getLogger(__name__) +def _scoped(var: str) -> str: + """The application-scope predicate for node variable ``var``, spelled once here so it cannot + drift: Java has a single id namespace, so it is one ``STARTS WITH`` against the prefix bound + from :attr:`JNeo4jBackend._scope_prefix` — never ``any(p IN $prefixes …)``, which would plan as + a label scan.""" + return f"{var}.id STARTS WITH $prefix" + + +def _semver(raw: Any) -> Tuple[int, int, int] | None: + """``"3.0.1"`` (or ``"3.0.1-rc1"``) as ``(3, 0, 1)``; ``None`` for anything that does not start + with three dotted integers, so an unparsable version is *unknown*, never silently zero.""" + m = re.match(r"(\d+)\.(\d+)\.(\d+)", raw) if isinstance(raw, str) else None + return (int(m[1]), int(m[2]), int(m[3])) if m else None + + +#: A child row of the containment subtree: (relationship type, child properties, edge properties). +_Child = Tuple[str, Dict[str, Any], Dict[str, Any]] + +#: The message for a comment accessor the projection cannot serve (D7): there are no ``:JComment`` +#: nodes, so "every comment in this file" has no answer, and the docstrings that *are* projected +#: are a strictly smaller set that must not be returned as if it were the whole one. +_COMMENTS_UNAVAILABLE = ( + "The codeanalyzer-java Neo4j projection carries no comment nodes for application {app!r}: a type, " + "callable or field keeps only its javadoc, in a docstring property, and a file-level comment is not " + "projected at all. Read the declarations' javadoc with get_all_docstrings(), or the full comment set " + "from analysis.json." +) + + class JNeo4jBackend(JavaAnalysisBackend): """Query the application view of a Java project over Neo4j (Cypher), read-only. @@ -86,10 +148,21 @@ class JNeo4jBackend(JavaAnalysisBackend): neo4j_uri: Bolt URI of the Neo4j server (e.g. ``bolt://localhost:7687``). neo4j_username / neo4j_password: Credentials (read-only is sufficient). neo4j_database: Database name (None ⇒ server default). - application_name: The ``:JApplication`` anchor name to scope every query to. Matches the - ``--app-name`` the graph was loaded with (defaults to the project directory name). + application_name: The ``--app-name`` the graph was emitted with; the anchor is + ``:JApplication {name: }`` and the id prefix is + ``can://java//``. """ + #: Relationship types every supported graph has; a graph missing any was emitted by another + #: generation (a schema-v1 graph shares only ``J_CALLS``) and is refused at attach. + _REQUIRED_RELATIONSHIP_TYPES: FrozenSet[str] = frozenset({"J_HAS_MODULE", "J_HAS_METHOD", "J_HAS_BODY_NODE", "J_CALLS"}) + #: The oldest codeanalyzer-java whose graph this backend serves: 3.0.0 stamped contract 2.2.0, + #: 3.0.1 holds 2.0.0 — the ``can://`` id grammar and body-node shape every statement here reads. + _ANALYZER_FLOOR = (3, 0, 1) + #: Set by :meth:`_probe_schema`; the class-level ``None`` is for the ``object.__new__`` seam. + _analyzer_version: Tuple[int, int, int] | None = None + _call_graph: nx.DiGraph | None = None + def __init__( self, neo4j_uri: str, @@ -101,25 +174,43 @@ def __init__( try: from neo4j import GraphDatabase except ModuleNotFoundError as e: # pragma: no cover - import guard - raise CodeanalyzerExecutionException( - "The Neo4j backend requires the 'neo4j' driver. Install it with " - "`pip install neo4j` (or `pip install cldk[neo4j]`)." - ) from e + raise CodeanalyzerExecutionException("The Neo4j backend requires the 'neo4j' driver. Install it with `pip install neo4j` (or `pip install cldk[neo4j]`).") from e + self._init_with_driver(GraphDatabase.driver(neo4j_uri, auth=(neo4j_username, neo4j_password)), application_name=application_name, neo4j_database=neo4j_database) + + @classmethod + def _from_driver(cls, driver: Any, *, application_name: str | None = None, neo4j_database: str | None = None) -> "JNeo4jBackend": + """Construct from an already-built driver — the seam tests inject a fake driver through.""" + self = cls.__new__(cls) + self._init_with_driver(driver, application_name=application_name, neo4j_database=neo4j_database) + return self + def _init_with_driver(self, driver: Any, *, application_name: str | None, neo4j_database: str | None) -> None: if not application_name: raise CodeanalyzerExecutionException("application_name is required to scope queries to an application.") self.application_name = application_name self._database = neo4j_database - self._driver = GraphDatabase.driver(neo4j_uri, auth=(neo4j_username, neo4j_password)) - - self._units: List[str] = self._load_unit_keys() - self.application: JApplication = self._reconstruct_application() - self.analysis_level = "call_graph" if self.application.call_graph else "symbol_table" - self.call_graph: nx.DiGraph | None = self._generate_call_graph(using_symbol_table=False) if self.application.call_graph else None + self._driver = driver + self._session_obj: Any | None = None + self._probe_schema() + self._module_props: Dict[str, Dict[str, Any]] = self._load_modules() + self._modules: List[str] = list(self._module_props) + self._call_graph = None + + # -----[ scope ]----- + @property + def _scope_prefix(self) -> str: + """``can://java//`` — the trailing slash keeps ``app`` from matching ``app-b``.""" + return f"can://java/{self.application_name}/" # -----[ lifecycle ]----- def close(self) -> None: """Close the underlying Neo4j driver.""" + if self._session_obj is not None: + try: + self._session_obj.close() + except Exception: # noqa: BLE001 - best-effort cleanup + pass + self._session_obj = None self._driver.close() def __enter__(self) -> "JNeo4jBackend": @@ -129,595 +220,629 @@ def __exit__(self, *exc: Any) -> None: self.close() def _run(self, query: str, **params: Any) -> List[Dict[str, Any]]: - with self._driver.session(database=self._database) as session: - return [record.data() for record in session.run(query, **params)] + """Run one read statement over a reused session; drop the session on failure.""" + if self._session_obj is None: + self._session_obj = self._driver.session(database=self._database) + try: + return [record.data() for record in self._session_obj.run(query, **params)] + except Exception: + self._session_obj = None + raise + + # -----[ attach ]----- + def _probe_schema(self) -> None: + """J-9: the relationship-type fingerprint, then the analyzer generation the + ``:JApplication`` anchor stamps, against :attr:`_ANALYZER_FLOOR`. + + A graph built by another codeanalyzer-java generation answers every statement here with + zero rows and no error — indistinguishable from "this application has no callables". Below + the floor, absent, or unreadable is refused, naming what was found. + """ + found = {r["relationshipType"] for r in self._run("CALL db.relationshipTypes()")} + missing = self._REQUIRED_RELATIONSHIP_TYPES - found + if missing: + raise GraphSchemaMismatch(expected=set(self._REQUIRED_RELATIONSHIP_TYPES), found=found, missing=missing) + rows = self._run("OPTIONAL MATCH (a:JApplication {name: $app}) RETURN count(a) AS n, a.analyzer_version AS v", app=self.application_name) + present = bool(rows and rows[0].get("n")) + raw = rows[0].get("v") if rows else None + version = _semver(raw) + floor = ".".join(map(str, self._ANALYZER_FLOOR)) + if version is None or version < self._ANALYZER_FLOOR: + if not present: + what = "has no :JApplication node" + elif version: + what = f"was emitted by codeanalyzer-java {raw}" + elif raw: + what = f"reports analyzer_version {raw!r}" + else: + what = "has a :JApplication node that carries no analyzer_version" + raise GraphSchemaMismatch( + expected=set(self._REQUIRED_RELATIONSHIP_TYPES), + found=found, + missing=set(), + message=f"The graph for application {self.application_name!r} {what}; this backend needs a graph emitted by codeanalyzer-java {floor} or newer.", + ) + self._analyzer_version = version - def _load_unit_keys(self) -> List[str]: + def _load_modules(self) -> Dict[str, Dict[str, Any]]: + """``file_key -> module properties`` for the application's modules.""" rows = self._run( - "MATCH (:JApplication {name: $app})-[:J_HAS_UNIT]->(u:JCompilationUnit) RETURN u.file_key AS k", - app=self.application_name, + "MATCH (:JApplication {name: $app})-[:J_HAS_MODULE]->(m:JModule) RETURN m.file_key AS k, properties(m) AS p ORDER BY m.file_key", app=self.application_name ) - return [r["k"] for r in rows] + return {r["k"]: r["p"] for r in rows} # ===================================================================================== - # Reconstruction: bulk-fetch the graph and rebuild the canonical JApplication. + # Reconstruction: eight statements, then the canonical JApplication. # ===================================================================================== - def _nodes(self, label: str) -> Dict[str, Dict[str, Any]]: - """All nodes of a label owned by this app, keyed by id/file_key/name.""" + #: The whole containment subtree beneath the application's modules, in one statement: the + #: ``*0..`` walk reaches every module, type (nested and local), callable and field, and the last + #: hop yields each one's children as ``(parent id, relationship, child)`` rows. Anchored on the + #: application, so it cannot leave it. ``J_HAS_FIELD`` is in the walk as well as in the child + #: hop because a field is itself an annotation target (``J_ANNOTATED_BY`` runs from a type, a + #: callable *or* a field), so it has to be reachable as a parent. + _SUBTREE = ( + "MATCH (:JApplication {name: $app})-[:J_HAS_MODULE]->(root:JModule) " + "MATCH (root)-[:J_DECLARES|J_HAS_METHOD|J_HAS_FIELD*0..]->(par)" + "-[r:J_DECLARES|J_HAS_METHOD|J_HAS_FIELD|J_DECLARES_VAR|J_HAS_ENUM_CONSTANT|J_HAS_RECORD_COMPONENT|J_ANNOTATED_BY]->(n) " + "RETURN par.id AS pk, type(r) AS rel, properties(n) AS p, properties(r) AS e, labels(n) AS labels " + "ORDER BY n.start_line, n.name" + ) + + def _subtree_rows(self) -> Dict[str, List[_Child]]: + rows = self._run(self._SUBTREE, app=self.application_name) + children: Dict[str, List[_Child]] = defaultdict(list) + for r in rows: + children[r["pk"]].append((r["rel"], {**r["p"], "_labels": r["labels"]}, r["e"] or {})) + return children + + def _call_site_rows(self) -> Dict[str, List[Tuple[Dict[str, Any], str | None]]]: + """Each callable's ``call`` body nodes with the signature its ``J_RESOLVES_TO`` edge names + (a project callable's or an external's), grouped by owning callable id.""" rows = self._run( - f"MATCH (n:{label}) WHERE n._module IN $u RETURN coalesce(n.id, n.file_key, n.name) AS k, properties(n) AS p", - u=self._units, + f"MATCH (c:JCallable)-[:J_HAS_BODY_NODE]->(b:JBodyNode {{kind: 'call'}}) WHERE {_scoped('c')} " + "OPTIONAL MATCH (b)-[:J_RESOLVES_TO]->(t) " + "RETURN c.id AS owner, properties(b) AS p, t.signature AS callee ORDER BY b.start_line, b.id", + prefix=self._scope_prefix, ) - return {r["k"]: r["p"] for r in rows} + out: Dict[str, List[Tuple[Dict[str, Any], str | None]]] = defaultdict(list) + for r in rows: + out[r["owner"]].append((r["p"], r["callee"])) + return out - def _adj(self, rtype: str, scope_child: bool = True) -> Dict[str, List[str]]: - """Adjacency parent_key → [child_keys] for a relationship, scoped to this app.""" - where = "b._module IN $u" if scope_child else "a._module IN $u" + def _import_rows(self) -> Dict[str, List[Dict[str, Any]]]: rows = self._run( - f"MATCH (a)-[:{rtype}]->(b) WHERE {where} " - "RETURN coalesce(a.id, a.file_key, a.name) AS a, coalesce(b.id, b.file_key, b.name) AS b", - u=self._units, + "MATCH (:JApplication {name: $app})-[:J_HAS_MODULE]->(m:JModule)-[r:J_IMPORTS]->() RETURN m.file_key AS k, properties(r) AS e ORDER BY m.file_key", + app=self.application_name, ) - out: Dict[str, List[str]] = {} + out: Dict[str, List[Dict[str, Any]]] = defaultdict(list) for r in rows: - out.setdefault(r["a"], []).append(r["b"]) + out[r["k"]].append(r["e"]) return out - def _reconstruct_application(self) -> JApplication: - units = self._units - # ---- node prop maps ---- - cu_nodes = { - r["k"]: r["p"] - for r in self._run( - "MATCH (:JApplication {name: $app})-[:J_HAS_UNIT]->(u:JCompilationUnit) RETURN u.file_key AS k, properties(u) AS p", - app=self.application_name, - ) - } - types = self._nodes("JType") - callables = self._nodes("JCallable") - fields = self._nodes("JField") - params = self._nodes("JParameter") - callsites = self._nodes("JCallSite") - variables = self._nodes("JVariable") - enums = self._nodes("JEnumConstant") - records = self._nodes("JRecordComponent") - initblocks = self._nodes("JInitializationBlock") - crudops = self._nodes("JCrudOperation") - crudqs = self._nodes("JCrudQuery") - comments = self._nodes("JComment") - - # ---- adjacencies ---- - a_callable = self._adj("J_HAS_CALLABLE") - a_field = self._adj("J_HAS_FIELD") - a_enum = self._adj("J_HAS_ENUM_CONSTANT") - a_record = self._adj("J_HAS_RECORD_COMPONENT") - a_init = self._adj("J_HAS_INIT_BLOCK") - a_param = self._adj("J_HAS_PARAMETER") - a_callsite = self._adj("J_HAS_CALLSITE") - a_var = self._adj("J_DECLARES_VAR") - a_crudop = self._adj("J_HAS_CRUD_OPERATION") - a_crudq = self._adj("J_HAS_CRUD_QUERY") - a_comment = self._adj("J_HAS_COMMENT") - a_import = self._run( - "MATCH (u:JCompilationUnit)-[r:J_IMPORTS]->(t) WHERE u._module IN $u " - "RETURN u.file_key AS cu, coalesce(t.fqn, t.name) AS path, properties(r) AS p", - u=units, + def _call_edge_rows(self) -> List[Dict[str, Any]]: + """Every ``J_CALLS`` edge between two of this application's callables. Both endpoints carry + the scope: an edge is only this application's when both ends are.""" + return self._run( + f"MATCH (s:JCallable)-[r:J_CALLS]->(t:JCallable) WHERE {_scoped('s')} AND {_scoped('t')} " "RETURN s.id AS src, t.id AS dst, r.weight AS weight, r.prov AS prov", + prefix=self._scope_prefix, ) - # ---- ordered helpers ---- - def _comments_of(owner_id: str) -> List[dict]: - ids = a_comment.get(owner_id, []) - built = [R.comment(comments[i]) for i in ids if i in comments] - return sorted(built, key=lambda c: (c["start_line"], c["start_column"])) - - def _first_comment(owner_id: str) -> dict | None: - cs = _comments_of(owner_id) - return cs[0] if cs else None - - def _param_index(pid: str) -> int: - try: - return int(pid.rsplit("#param#", 1)[1]) - except (IndexError, ValueError): - return 0 - - def _build_callsite(cs_id: str) -> dict: - p = callsites[cs_id] - op_ids = a_crudop.get(cs_id, []) - q_ids = a_crudq.get(cs_id, []) - crud_op = R.crud_operation(crudops[op_ids[0]]) if op_ids and op_ids[0] in crudops else None - crud_q = R.crud_query(crudqs[q_ids[0]]) if q_ids and q_ids[0] in crudqs else None - return R.callsite(p, comment_node=_first_comment(cs_id), crud_op=crud_op, crud_q=crud_q) - - def _callsites_of(owner_id: str) -> List[dict]: - ids = a_callsite.get(owner_id, []) - built = [(callsites[i], _build_callsite(i)) for i in ids if i in callsites] - return [cs for _, cs in sorted(built, key=lambda t: (t[0].get("start_line", -1), t[0].get("start_column", -1)))] - - def _vars_of(owner_id: str) -> List[dict]: - ids = a_var.get(owner_id, []) - built = [(variables[i], R.variable(variables[i], comment_node=_first_comment(i))) for i in ids if i in variables] - return [v for _, v in sorted(built, key=lambda t: (t[0].get("start_line", -1), t[0].get("name", "")))] - - # ---- callables ---- - def _build_callable(cid: str) -> dict: - p = callables[cid] - pids = sorted(a_param.get(cid, []), key=_param_index) - parameters = [R.parameter(params[i]) for i in pids if i in params] - op_ids = a_crudop.get(cid, []) - q_ids = a_crudq.get(cid, []) - crud_ops = [R.crud_operation(crudops[i]) for i in op_ids if i in crudops] - crud_qs = [R.crud_query(crudqs[i]) for i in q_ids if i in crudqs] - return R.callable_( - p, - comments=_comments_of(cid), - parameters=parameters, - call_sites=_callsites_of(cid), - variable_declarations=_vars_of(cid), - crud_operations=crud_ops, - crud_queries=crud_qs, - ) - - def _build_initblock(ib_id: str) -> dict: - p = initblocks[ib_id] - return R.init_block(p, comments=_comments_of(ib_id), call_sites=_callsites_of(ib_id), variable_declarations=_vars_of(ib_id)) - - # ---- types ---- - def _build_type(tid: str) -> dict: - p = types[tid] - cdecls = {} - for cid in a_callable.get(tid, []): - if cid in callables: - cdecls[callables[cid].get("signature", cid)] = _build_callable(cid) - fdecls = [R.field(fields[i], comment_node=_first_comment(i)) for i in a_field.get(tid, []) if i in fields] - econsts = [R.enum_constant(enums[i]) for i in a_enum.get(tid, []) if i in enums] - rcomps = [R.record_component(records[i], comment_node=_first_comment(i)) for i in a_record.get(tid, []) if i in records] - iblocks = [_build_initblock(i) for i in a_init.get(tid, []) if i in initblocks] - return R.type_( - p, - comments=_comments_of(tid), - callable_declarations=cdecls, - field_declarations=fdecls, - enum_constants=econsts, - record_components=rcomps, - initialization_blocks=iblocks, - ) + def _artifact_rows(self) -> List[Dict[str, Any]]: + return self._run( + "MATCH (:JApplication {name: $app})-[:HAS_ARTIFACT]->(a:Artifact) " + "OPTIONAL MATCH (a)-[:DEFINES_CONFIG]->(ck:ConfigKey) " + "RETURN properties(a) AS p, collect(properties(ck)) AS cks", + app=self.application_name, + ) - # group types by owning module (file_key); type_declarations is a flat per-CU map - types_by_unit: Dict[str, Dict[str, dict]] = {} - for tid, tp in types.items(): - fkey = tp.get("_module") - fqn = tp.get("fqn", tid) - types_by_unit.setdefault(fkey, {})[fqn] = _build_type(tid) - - # imports by unit - imports_by_unit: Dict[str, List[dict]] = {} - for r in a_import: - imports_by_unit.setdefault(r["cu"], []).append( - {"path": r["path"], "is_static": r["p"].get("is_static", False), "is_wildcard": r["p"].get("is_wildcard", False)} - ) + def _dependency_rows(self) -> List[Dict[str, Any]]: + return self._run( + "MATCH (:JApplication {name: $app})-[:HAS_ARTIFACT]->(a:Artifact)-[r:DECLARES_DEPENDENCY]->(p:Package) " + "RETURN properties(r) AS rel, properties(p) AS pkg, a.id AS declared_in ORDER BY p.name", + app=self.application_name, + ) - # ---- compilation units / symbol table ---- - symbol_table: Dict[str, dict] = {} - for fkey, cp in cu_nodes.items(): - symbol_table[fkey] = R.compilation_unit( - cp, - comments=_comments_of(fkey), - import_declarations=imports_by_unit.get(fkey, []), - type_declarations=types_by_unit.get(fkey, {}), + # -----[ the containment tree ]----- + @staticmethod + def _child_key(parent_id: str, props: Dict[str, Any]) -> str: + """A declared type's container key: the id segment under its parent, which is its simple + name. A child id is minted under its parent's by construction, so a mismatch is an emitter + defect, named by the declaration rather than by either id (E6).""" + node_id = props["id"] + if not node_id.startswith(parent_id + "/"): + raise CodeanalyzerExecutionException( + f"declaration {props.get('name') or props.get('signature')!r} is reached from a parent that did not mint its id: " + f"codeanalyzer-java emitted a containment edge this backend cannot key" ) + return node_id[len(parent_id) + 1 :] + + def _decorators(self, node_id: str, children: Dict[str, List[_Child]]) -> List[JDecorator]: + return [R.decorator(p, e) for rel, p, e in children.get(node_id, []) if rel == "J_ANNOTATED_BY"] + + def _body(self, callable_id: str, sites: Dict[str, List[Tuple[Dict[str, Any], str | None]]]) -> Dict[str, JBodyNode]: + """The ``call`` entries of a callable's ``body`` map, keyed the analyzer's way: the ``L:C`` + the node id's ``@`` suffix spells (which is a key, not a position -- see + :func:`reconstruct.body_node`).""" + return {props["id"][len(callable_id) + 1 :]: R.body_node(props, callee) for props, callee in sites.get(callable_id, [])} + + def _callable(self, props: Dict[str, Any], children: Dict[str, List[_Child]], sites: Dict[str, List[Tuple[Dict[str, Any], str | None]]]) -> JCallable: + node_id = props["id"] + rows = children.get(node_id, []) + return R.callable_( + props, + decorators=self._decorators(node_id, children), + body=self._body(node_id, sites), + local_variables=[R.variable(p) for rel, p, _ in rows if rel == "J_DECLARES_VAR"], + types={self._child_key(node_id, p): self._type(p, children, sites) for rel, p, _ in rows if rel == "J_DECLARES"}, + ) - # ---- call graph edges ---- - call_edges: List[dict] = [] - for r in self._run( - "MATCH (s:JCallable)-[c:J_CALLS]->(t:JCallable) WHERE s._module IN $u " - "RETURN s.id AS src, t.id AS tgt, properties(c) AS p", - u=units, - ): - src = self._endpoint(r["src"], callables) - tgt = self._endpoint(r["tgt"], callables) - if src and tgt: - call_edges.append(R.call_edge(src, tgt, r["p"])) + def _type(self, props: Dict[str, Any], children: Dict[str, List[_Child]], sites: Dict[str, List[Tuple[Dict[str, Any], str | None]]]) -> JType: + node_id = props["id"] + rows = children.get(node_id, []) + callables: Dict[str, JCallable] = {} + types: Dict[str, JType] = {} + for rel, p, _ in rows: + if rel == "J_HAS_METHOD": + callables[p["signature"]] = self._callable(p, children, sites) + elif rel == "J_DECLARES": + types[self._child_key(node_id, p)] = self._type(p, children, sites) + return R.type_( + props, + decorators=self._decorators(node_id, children), + fields={p["name"]: R.field(p, self._decorators(p["id"], children)) for rel, p, _ in rows if rel == "J_HAS_FIELD"}, + callables=callables, + types=types, + enum_constants=[R.enum_constant(p) for rel, p, _ in rows if rel == "J_HAS_ENUM_CONSTANT"], + record_components=[R.record_component(p) for rel, p, _ in rows if rel == "J_HAS_RECORD_COMPONENT"], + ) - return JApplication(symbol_table=symbol_table, call_graph=call_edges) + def _reconstruct(self) -> JApplication: + """The canonical :class:`JApplication` for this application, rebuilt from the graph.""" + children = self._subtree_rows() + sites = self._call_site_rows() + imports = self._import_rows() + symbol_table: Dict[str, JCompilationUnit] = {} + for key, props in self._module_props.items(): + module_id = props["id"] + types: Dict[str, JType] = {} + for rel, p, _ in children.get(module_id, []): + if rel != "J_DECLARES": + continue + # A module declares types only; ``kind`` is a ``Literal`` on :class:`JType`, so a + # row that is not one is refused by the model. + types[self._child_key(module_id, p)] = self._type(p, children, sites) + unit = R.compilation_unit(props, import_declarations=[i for e in imports.get(key, []) for i in R.imports(e)], types=types) + R.thread_code(unit, self._projected_code(children, module_id)) + symbol_table[key] = unit + return JApplication( + id=f"can://java/{self.application_name}", + symbol_table=symbol_table, + call_graph=[JCallGraphEdge(src=r["src"], dst=r["dst"], prov=list(r["prov"] or []), weight=r["weight"] or 1) for r in self._call_edge_rows()], + artifacts={ + a.path: a + for a in ( + R.artifact(r["p"], config_keys=[R.config_key(p) for p in sorted((c for c in r["cks"] if c), key=lambda c: c["id"])]) + for r in sorted(self._artifact_rows(), key=lambda r: r["p"]["path"]) + ) + }, + dependencies=[R.dependency(r["rel"], r["pkg"], r["declared_in"]) for r in self._dependency_rows()], + ) @staticmethod - def _endpoint(node_id: str, callables: Dict[str, Dict[str, Any]]) -> dict | None: - """A J_CALLS endpoint id (``#``) → a JGraphEdges source/target dict.""" - if "#" not in node_id: - return None - fqn, signature = node_id.split("#", 1) - props = callables.get(node_id, {}) - declaration = props.get("declaration") or signature - if "(" not in declaration: - declaration = signature - return {"file_path": props.get("file_path", ""), "type_declaration": fqn, "signature": signature, "callable_declaration": declaration} + def _projected_code(children: Dict[str, List[_Child]], module_id: str) -> Dict[str, str]: + """``callable id -> code`` for one module's subtree — what :func:`reconstruct.thread_code` + threads onto the callables so their ``code`` view reads the graph's text.""" + out: Dict[str, str] = {} + stack = [module_id] + while stack: + for rel, p, _ in children.get(stack.pop(), []): + if rel in ("J_DECLARES", "J_HAS_METHOD"): + stack.append(p["id"]) + if rel == "J_HAS_METHOD": + out[p["id"]] = p.get("code") or "" + return out # ===================================================================================== - # JavaAnalysisBackend — leaf accessors (served from the reconstructed application) + # The reconstructed view and its index (both built on first use) # ===================================================================================== + @cached_property + def _application(self) -> JApplication: + """The application view, rebuilt from the graph on first use and cached. Private because + :attr:`_idx` and :attr:`_call_graph` are derived from it and cached beside it: rebinding it + would leave them answering from the object it replaced. Tests that need a seeded view + without a server write ``backend.__dict__["_application"]``, which is exactly what this + ``cached_property`` would have stored.""" + return self._reconstruct() + + @property + def application(self) -> JApplication: + """The application view (read-only; see :attr:`_application`).""" + return self._application + + @cached_property + def _idx(self) -> Tuple[Dict[str, JType], Dict[str, str], Dict[str, Tuple[JType, JCallable]]]: + """The containment tree flattened once: every type (top-level, nested, local/anonymous) by + its source-spelled qualified name, its file, and every callable by its ``can://`` id — the + join that turns a call-graph endpoint into the ``"."`` node key. + Mirrors :meth:`JCodeanalyzer._index`.""" + types: Dict[str, JType] = {} + file_of: Dict[str, str] = {} + callables: Dict[str, Tuple[JType, JCallable]] = {} + + def add(t: JType, path: str) -> None: + name = t.qualified_name + if name in types: + raise CodeanalyzerExecutionException(duplicate_type_name(name)) + types[name] = t + file_of[name] = path + for c in t.callables.values(): + callables[c.id] = (t, c) + for local in c.types.values(): + add(local, path) + for nested in t.types.values(): + add(nested, path) + + for path, unit in self._application.symbol_table.items(): + for t in unit.types.values(): + add(t, path) + return types, file_of, callables + + @property + def _types(self) -> Dict[str, JType]: + return self._idx[0] + + # -----[ application / whole-program ]----- def get_application_view(self) -> JApplication: return self.application def get_symbol_table(self) -> Dict[str, JCompilationUnit]: return self.application.symbol_table - def get_system_dependency_graph(self) -> list[JGraphEdges]: - return self.application.call_graph or [] - def get_compilation_units(self) -> List[JCompilationUnit]: return list(self.application.symbol_table.values()) + def get_java_file(self, qualified_class_name: str) -> str | None: + return self._idx[1].get(qualified_class_name) + def get_java_compilation_unit(self, file_path: str) -> JCompilationUnit: return self.application.symbol_table[file_path] - # ===================================================================================== - # Call graph (logic mirrors JCodeanalyzer; calling_lines recomputed from JCallable.code) - # ===================================================================================== - def _generate_call_graph(self, using_symbol_table) -> nx.DiGraph: - cg = nx.DiGraph() - if using_symbol_table: - NotImplementedError("Call graph generation using symbol table is not implemented yet.") - else: - sdg = self.get_system_dependency_graph() - tsu = TreesitterJava() - edge_list = [ - ( - (jge.source.method.signature, jge.source.klass), - (jge.target.method.signature, jge.target.klass), - { - "type": jge.type, - "weight": jge.weight, - "calling_lines": ( - tsu.get_calling_lines(jge.source.method.code, jge.target.method.signature) - if not jge.source.method.is_implicit or not jge.target.method.is_implicit - else [] - ), - }, - ) - for jge in sdg - if jge.type == "CALL_DEP" - ] - for jge in sdg: - cg.add_node((jge.source.method.signature, jge.source.klass), method_detail=jge.source) - cg.add_node((jge.target.method.signature, jge.target.klass), method_detail=jge.target) - cg.add_edges_from(edge_list) - return cg + def get_system_dependency_graph(self) -> list[JGraphEdges]: + """The wire call graph (``JApplication.call_graph``), one :class:`JCallGraphEdge` per edge.""" + return self.application.call_graph + + # -----[ call graph ]----- + @staticmethod + def _detail(klass: str, c: JCallable) -> JMethodDetail: + return JMethodDetail(method_declaration=c.declaration, klass=klass, method=c) + + def _node_of(self, node_id: str) -> Tuple[str, JMethodDetail]: + """The (node key, method detail) a call-graph endpoint id resolves to. Every endpoint the + projection writes is homed on the tree; one that is not is a defect, surfaced rather than + skipped — named by the signature and module key its id spells, never by the id (E6), in the + same words the in-memory backend uses.""" + try: + t, c = self._idx[2][node_id] + except KeyError: + raise CodeanalyzerExecutionException(unhomed_endpoint(node_id)) from None + return f"{t.qualified_name}.{c.signature}", self._detail(t.qualified_name, c) def get_call_graph(self) -> nx.DiGraph: - if self.analysis_level == "symbol_table": - self.call_graph = self._generate_call_graph(using_symbol_table=True) - if self.call_graph is None: - self.call_graph = self._generate_call_graph(using_symbol_table=False) - return self.call_graph + """Build (and cache) the call graph keyed by ``"."`` (J-1): node attrs + ``method_detail`` / ``kind="callable"``; edge attrs ``type="CALL_DEP"``, ``weight``, + ``calling_lines``. Edges to external targets are dropped (see the module docstring).""" + if self._call_graph is not None: + return self._call_graph + cg = nx.DiGraph() + lines = CallingLines() + for edge in self.application.call_graph: + src, src_detail = self._node_of(edge.src) + dst, dst_detail = self._node_of(edge.dst) + cg.add_node(src, method_detail=src_detail, kind="callable") + cg.add_node(dst, method_detail=dst_detail, kind="callable") + cg.add_edge(src, dst, type="CALL_DEP", weight=edge.weight, calling_lines=lines.of(src_detail.method, dst_detail.method)) + self._call_graph = cg + return cg def get_call_graph_json(self) -> str: - callgraph_list = [] - edges = list(self.call_graph.edges.data("calling_lines")) - for edge in edges: - callgraph_dict = {} - callgraph_dict["source_method_signature"] = edge[0][0] - callgraph_dict["source_method_body"] = self.call_graph.nodes[edge[0]]["method_detail"].method.code - callgraph_dict["source_class"] = edge[0][1] - callgraph_dict["target_method_signature"] = edge[1][0] - callgraph_dict["target_method_body"] = self.call_graph.nodes[edge[1]]["method_detail"].method.code - callgraph_dict["target_class"] = edge[1][1] - callgraph_dict["calling_lines"] = edge[2] - callgraph_list.append(callgraph_dict) - return json.dumps(callgraph_list) + cg = self.get_call_graph() + rows = [] + for source, target, calling_lines in cg.edges.data("calling_lines"): + s: JMethodDetail = cg.nodes[source]["method_detail"] + t: JMethodDetail = cg.nodes[target]["method_detail"] + rows.append( + { + "source_method_signature": s.method.signature, + "source_method_body": s.method.code, + "source_class": s.klass, + "target_method_signature": t.method.signature, + "target_method_body": t.method.code, + "target_class": t.klass, + "calling_lines": calling_lines, + } + ) + return json.dumps(rows) def get_all_callers(self, target_class_name: str, target_method_signature: str, using_symbol_table: bool) -> Dict: - caller_detail_dict = {} - if using_symbol_table: - call_graph = self.__call_graph_using_symbol_table(qualified_class_name=target_class_name, method_signature=target_method_signature, is_target_method=True) - else: - call_graph = self.call_graph - if (target_method_signature, target_class_name) not in call_graph.nodes(): - return caller_detail_dict - in_edge_view = call_graph.in_edges(nbunch=(target_method_signature, target_class_name), data=True) - caller_detail_dict["caller_details"] = [] - caller_detail_dict["target_method"] = call_graph.nodes[(target_method_signature, target_class_name)]["method_detail"] - for source, target, data in in_edge_view: - cm = {"caller_method": call_graph.nodes[source]["method_detail"], "calling_lines": data["calling_lines"]} - caller_detail_dict["caller_details"].append(cm) - return caller_detail_dict + cg = self._symbol_table_call_graph(target_class_name, target_method_signature, is_target=True) if using_symbol_table else self.get_call_graph() + key = f"{target_class_name}.{target_method_signature}" + if key not in cg: + return {} + return { + "caller_details": [{"caller_method": cg.nodes[s]["method_detail"], "calling_lines": d["calling_lines"]} for s, _, d in cg.in_edges(key, data=True)], + "target_method": cg.nodes[key]["method_detail"], + } def get_all_callees(self, source_class_name: str, source_method_signature: str, using_symbol_table: bool) -> Dict: - callee_detail_dict = {} - if using_symbol_table: - call_graph = self.__call_graph_using_symbol_table(qualified_class_name=source_class_name, method_signature=source_method_signature) + cg = self._symbol_table_call_graph(source_class_name, source_method_signature) if using_symbol_table else self.get_call_graph() + key = f"{source_class_name}.{source_method_signature}" + if key not in cg: + return {} + return { + "callee_details": [{"callee_method": cg.nodes[t]["method_detail"], "calling_lines": d["calling_lines"]} for _, t, d in cg.out_edges(key, data=True)], + "source_method": cg.nodes[key]["method_detail"], + } + + @staticmethod + def _edges_out_of(cg: nx.DiGraph, qualified_class_name: str, method_signature: str | None) -> List[Tuple[JMethodDetail, JMethodDetail]]: + if method_signature is None: + seeds = [n for n, a in cg.nodes(data=True) if a["method_detail"].klass == qualified_class_name] else: - call_graph = self.call_graph - if (source_method_signature, source_class_name) not in call_graph.nodes(): - return callee_detail_dict - out_edge_view = call_graph.out_edges(nbunch=(source_method_signature, source_class_name), data=True) - callee_detail_dict["callee_details"] = [] - callee_detail_dict["source_method"] = call_graph.nodes[(source_method_signature, source_class_name)]["method_detail"] - for source, target, data in out_edge_view: - cm = {"callee_method": call_graph.nodes[target]["method_detail"], "calling_lines": data["calling_lines"]} - callee_detail_dict["callee_details"].append(cm) - return callee_detail_dict + key = f"{qualified_class_name}.{method_signature}" + seeds = [key] if key in cg else [] + return [(cg.nodes[s]["method_detail"], cg.nodes[t]["method_detail"]) for s, t in cg.edges(seeds)] - # ===================================================================================== - # Classes / methods / fields (operate on the reconstructed symbol table) - # ===================================================================================== - def get_all_methods_in_application(self) -> Dict[str, Dict[str, JCallable]]: - class_method_dict = {} - class_dict = self.get_all_classes() - for k, v in class_dict.items(): - class_method_dict[k] = v.callable_declarations - return class_method_dict + def get_class_call_graph(self, qualified_class_name: str, method_name: str | None = None) -> List[Tuple[JMethodDetail, JMethodDetail]]: + return self._edges_out_of(self.get_call_graph(), qualified_class_name, method_name) - def get_all_classes(self) -> Dict[str, JType]: - class_dict = {} - for v in self.get_symbol_table().values(): - class_dict.update(v.type_declarations) - return class_dict - - def get_class(self, qualified_class_name) -> JType | None: - for v in self.get_symbol_table().values(): - if qualified_class_name in v.type_declarations.keys(): - return v.type_declarations.get(qualified_class_name) - return None - - def get_method(self, qualified_class_name, method_signature) -> JCallable | None: - for v in self.get_symbol_table().values(): - if qualified_class_name in v.type_declarations.keys(): - ci = v.type_declarations[qualified_class_name] - for cd in ci.callable_declarations.keys(): - if cd == method_signature: - return ci.callable_declarations[cd] - return None - - def get_method_parameters(self, qualified_class_name, method_signature) -> List[JCallableParameter]: + def get_class_call_graph_using_symbol_table(self, qualified_class_name: str, method_signature: str | None = None) -> List[Tuple[JMethodDetail, JMethodDetail]]: + """Edges out of a class (or one method) resolved from its call sites through the symbol + table alone — incomplete by construction: only receivers the symbol table can see, only + concrete implementations up the ``extends`` chain.""" + return self._edges_out_of(self._symbol_table_call_graph(qualified_class_name, method_signature), qualified_class_name, method_signature) + + # -----[ symbol-table call graph (call sites → declarations) ]----- + # The same resolution the in-memory backend runs (``JCodeanalyzer._symbol_table_call_graph`` + # and friends), over the same models: it reads nothing but ``get_class`` / ``get_method`` and + # the callable index, so the two must agree edge for edge on the same symbol table. + def _symbol_table_call_graph(self, qualified_class_name: str, method_signature: str | None, is_target: bool = False) -> nx.DiGraph: + cg = nx.DiGraph() + lines = CallingLines() + edges = self._st_edges_into(qualified_class_name, method_signature) if is_target else self._st_edges_from(qualified_class_name, method_signature) + for source, target in edges: + src, dst = f"{source.klass}.{source.method.signature}", f"{target.klass}.{target.method.signature}" + cg.add_node(src, method_detail=source, kind="callable") + cg.add_node(dst, method_detail=target, kind="callable") + cg.add_edge(src, dst, type="CALL_DEP", weight=1, calling_lines=lines.of(source.method, target.method)) + return cg + + def _st_edges_from(self, qualified_class_name: str, method_signature: str | None) -> Iterable[Tuple[JMethodDetail, JMethodDetail]]: + klass = self.get_class(qualified_class_name) + if klass is None: + return + if method_signature is None: + sources = list(klass.callables.values()) + else: + source = self.get_method(qualified_class_name, method_signature) + sources = [source] if source is not None else [] + for source in sources: + for call_site in source.call_sites: + target, target_class = self._resolve_call_site(qualified_class_name, call_site) + if target is not None: + yield self._detail(qualified_class_name, source), self._detail(target_class, target) + + def _st_edges_into(self, target_class_name: str, target_method_signature: str) -> Iterable[Tuple[JMethodDetail, JMethodDetail]]: + target = self.get_method(target_class_name, target_method_signature) + if target is None: + return + for owner, source in self._idx[2].values(): + for call_site in source.call_sites: + found, found_class = self._resolve_call_site(owner.qualified_name, call_site) + if found is not None and found_class == target_class_name and call_site.callee_signature == target_method_signature: + yield self._detail(owner.qualified_name, source), self._detail(target_class_name, target) + + def _resolve_call_site(self, owner_class_name: str, call_site: JCallSite) -> Tuple[JCallable | None, str]: + """The (declaration, declaring class) a call site names, or ``(None, "")``: an explicit + receiver type is followed only when it is a project class; an implicit receiver means the + owning class (and its ``extends`` chain).""" + if not call_site.callee_signature: + return None, "" + if call_site.receiver_type: + if self.get_class(call_site.receiver_type) is None: + return None, "" + return self._find_in_hierarchy(call_site.receiver_type, call_site.callee_signature) + return self._find_in_hierarchy(owner_class_name, call_site.callee_signature) + + def _find_in_hierarchy(self, qualified_class_name: str, method_signature: str) -> Tuple[JCallable | None, str]: + """The concrete declaration of ``method_signature`` on the class or up its ``extends`` + chain; interface declarations are not call-graph targets and are skipped.""" + klass = self.get_class(qualified_class_name) method = self.get_method(qualified_class_name, method_signature) - return method.parameters if method is not None else [] + if method is not None and klass is not None and not klass.is_interface: + return method, qualified_class_name + if klass is not None: + for parent in klass.extends_list: + found, found_class = self._find_in_hierarchy(parent, method_signature) + if found is not None: + return found, found_class + return None, "" + + # -----[ classes / methods / fields ]----- + def get_all_classes(self) -> Dict[str, JType]: + return dict(self._types) + + def get_class(self, qualified_class_name: str) -> JType | None: + return self._types.get(qualified_class_name) - def get_java_file(self, qualified_class_name) -> str | None: - for k, v in self.get_symbol_table().items(): - if qualified_class_name in v.type_declarations.keys(): - return k - return None + def get_all_methods_in_application(self) -> Dict[str, Dict[str, JCallable]]: + return {name: t.callable_declarations for name, t in self._types.items()} - def get_all_methods_in_class(self, qualified_class_name) -> Dict[str, JCallable]: - ci = self.get_class(qualified_class_name) - if ci is None: + def get_all_methods_in_class(self, qualified_class_name: str) -> Dict[str, JCallable]: + klass = self.get_class(qualified_class_name) + if klass is None: return {} - return {k: v for (k, v) in ci.callable_declarations.items() if v.is_constructor is False} + return {sig: c for sig, c in klass.callables.items() if not c.is_constructor} - def get_all_constructors(self, qualified_class_name) -> Dict[str, JCallable]: - ci = self.get_class(qualified_class_name) - if ci is None: + def get_all_constructors(self, qualified_class_name: str) -> Dict[str, JCallable]: + klass = self.get_class(qualified_class_name) + if klass is None: return {} - return {k: v for (k, v) in ci.callable_declarations.items() if v.is_constructor is True} - - def get_all_sub_classes(self, qualified_class_name) -> Dict[str, JType]: - all_classes = self.get_all_classes() - sub_classes = {} - for cls in all_classes: - if qualified_class_name in all_classes[cls].implements_list or qualified_class_name in all_classes[cls].extends_list: - sub_classes[cls] = all_classes[cls] - return sub_classes - - def get_all_fields(self, qualified_class_name) -> List[JField]: - ci = self.get_class(qualified_class_name) - if ci is None: - logging.warning(f"Class {qualified_class_name} not found in the application view.") - return list() - return ci.field_declarations - - def get_all_nested_classes(self, qualified_class_name) -> List[JType]: - ci = self.get_class(qualified_class_name) - if ci is None: - logging.warning(f"Class {qualified_class_name} not found in the application view.") - return list() - return [self.get_class(c) for c in ci.nested_type_declarations] - - def get_extended_classes(self, qualified_class_name) -> List[str]: - ci = self.get_class(qualified_class_name) - if ci is None: - logging.warning(f"Class {qualified_class_name} not found in the application view.") - return list() - return ci.extends_list - - def get_implemented_interfaces(self, qualified_class_name) -> List[str]: - ci = self.get_class(qualified_class_name) - if ci is None: - logging.warning(f"Class {qualified_class_name} not found in the application view.") - return list() - return ci.implements_list + return {sig: c for sig, c in klass.callables.items() if c.is_constructor} + + def get_method(self, qualified_class_name: str, qualified_method_name: str) -> JCallable | None: + """The callable, or ``None``. Two fields differ from the in-memory backend's, because the + projection differs: ``code`` is the whole **declaration** (the graph carries one line range + per callable and no ``body_span``), where the in-memory backend's is the body block; and + ``body`` holds the ``call`` nodes **only** — about 30% of the graph's body nodes (4,006 of + daytrader8's 13,436) — which is what ``call_sites`` is a view over.""" + klass = self.get_class(qualified_class_name) + return klass.callables.get(qualified_method_name) if klass is not None else None - # ===================================================================================== - # Symbol-table call graph (pure-Python over call sites; mirrors JCodeanalyzer) - # ===================================================================================== - def get_class_call_graph_using_symbol_table(self, qualified_class_name: str, method_signature: str | None = None) -> List[Tuple[JMethodDetail, JMethodDetail]]: - call_graph = self.__call_graph_using_symbol_table(qualified_class_name, method_signature) - if method_signature is None: - filter_criteria = {node for node in call_graph.nodes if node[1] == qualified_class_name} - else: - filter_criteria = {node for node in call_graph.nodes if tuple(node) == (method_signature, qualified_class_name)} - graph_edges: List[Tuple[JMethodDetail, JMethodDetail]] = list() - for edge in call_graph.edges(nbunch=filter_criteria): - source: JMethodDetail = call_graph.nodes[edge[0]]["method_detail"] - target: JMethodDetail = call_graph.nodes[edge[1]]["method_detail"] - graph_edges.append((source, target)) - return graph_edges - - def __call_graph_using_symbol_table(self, qualified_class_name: str, method_signature: str, is_target_method: bool = False) -> nx.DiGraph: - cg = nx.DiGraph() - if is_target_method: - sdg = self.__raw_call_graph_using_symbol_table_target_method(target_class_name=qualified_class_name, target_method_signature=method_signature) - else: - sdg = self.__raw_call_graph_using_symbol_table(qualified_class_name=qualified_class_name, method_signature=method_signature) - tsu = TreesitterJava() - edge_list = [ - ( - (jge.source.method.signature, jge.source.klass), - (jge.target.method.signature, jge.target.klass), - {"type": jge.type, "weight": jge.weight, "calling_lines": tsu.get_calling_lines(jge.source.method.code, jge.target.method.signature)}, - ) - for jge in sdg - ] - for jge in sdg: - cg.add_node((jge.source.method.signature, jge.source.klass), method_detail=jge.source) - cg.add_node((jge.target.method.signature, jge.target.klass), method_detail=jge.target) - cg.add_edges_from(edge_list) - return cg + def get_method_parameters(self, qualified_class_name: str, qualified_method_name: str) -> List[JCallableParameter]: + """The parameters the callable's ``parameters_json`` carries — the analyzer's own + serialisation, so these round-trip exactly (there is no ``:JParameter`` node in 3.0.1, and + none is needed).""" + method = self.get_method(qualified_class_name, qualified_method_name) + return method.parameters if method is not None else [] - def __raw_call_graph_using_symbol_table_target_method(self, target_class_name: str, target_method_signature: str, cg=None) -> list[JGraphEdgesST]: - if cg is None: - cg = [] - target_method_details = self.get_method(qualified_class_name=target_class_name, method_signature=target_method_signature) - if target_method_details is None: - # The target method doesn't exist, so no edges into it can be constructed. - return cg - for class_name in self.get_all_classes(): - for method in self.get_all_methods_in_class(qualified_class_name=class_name): - method_details = self.get_method(qualified_class_name=class_name, method_signature=method) - if method_details is None: - # The symbol table momentarily disagreed with itself; skip this entry. - continue - for call_site in method_details.call_sites: - source_method_details = None - source_class = "" - callee_signature = call_site.callee_signature if call_site.callee_signature != "" else "" - if call_site.receiver_type != "": - if self.get_class(qualified_class_name=call_site.receiver_type): - found_method, found_class = self.__find_method_in_hierarchy(call_site.receiver_type, callee_signature) - if found_method is not None and callee_signature == target_method_signature and found_class == target_class_name: - source_method_details = self.get_method(method_signature=method, qualified_class_name=class_name) - source_class = class_name - else: - found_method, found_class = self.__find_method_in_hierarchy(class_name, callee_signature) - if found_method is not None and callee_signature == target_method_signature and found_class == target_class_name: - source_method_details = self.get_method(method_signature=method, qualified_class_name=class_name) - source_class = class_name - if source_class != "" and source_method_details is not None: - call_edge = JGraphEdgesST( - source=JMethodDetail(method_declaration=source_method_details.declaration, klass=source_class, method=source_method_details), - target=JMethodDetail(method_declaration=target_method_details.declaration, klass=target_class_name, method=target_method_details), - type="CALL_DEP", - weight="1", - ) - if call_edge not in cg: - cg.append(call_edge) - return cg + def get_all_sub_classes(self, qualified_class_name: str) -> Dict[str, JType]: + return {name: t for name, t in self._types.items() if qualified_class_name in t.extends_list or qualified_class_name in t.implements_list} - def __find_method_in_hierarchy(self, qualified_class_name: str, method_signature: str) -> Tuple[JCallable | None, str]: - klass = self.get_class(qualified_class_name=qualified_class_name) - method_details = self.get_method(method_signature=method_signature, qualified_class_name=qualified_class_name) - if method_details is not None and klass is not None and not klass.is_interface: - return method_details, qualified_class_name - if klass is not None: - for parent_class in klass.extends_list: - parent_method, found_class = self.__find_method_in_hierarchy(parent_class, method_signature) - if parent_method is not None: - return parent_method, found_class - return None, "" + def get_all_fields(self, qualified_class_name: str) -> List[JField]: + klass = self.get_class(qualified_class_name) + return klass.field_declarations if klass is not None else [] - def __raw_call_graph_using_symbol_table(self, qualified_class_name: str, method_signature: str, cg=None) -> list[JGraphEdgesST]: - if cg is None: - cg = [] - source_method_details = self.get_method(qualified_class_name=qualified_class_name, method_signature=method_signature) - if source_method_details is None: - return cg - for call_site in source_method_details.call_sites: - target_method_details = None - target_class = "" - callee_signature = call_site.callee_signature if call_site.callee_signature != "" else "" - if call_site.receiver_type != "": - if self.get_class(qualified_class_name=call_site.receiver_type): - tmd, found_class = self.__find_method_in_hierarchy(call_site.receiver_type, callee_signature) - if tmd is not None: - target_method_details = tmd - target_class = found_class - else: - tmd, found_class = self.__find_method_in_hierarchy(qualified_class_name, callee_signature) - if tmd is not None: - target_method_details = tmd - target_class = found_class - if target_class != "" and target_method_details is not None: - call_edge = JGraphEdgesST( - source=JMethodDetail(method_declaration=source_method_details.declaration, klass=qualified_class_name, method=source_method_details), - target=JMethodDetail(method_declaration=target_method_details.declaration, klass=target_class, method=target_method_details), - type="CALL_DEP", - weight="1", - ) - if call_edge not in cg: - cg.append(call_edge) - return cg + def get_all_nested_classes(self, qualified_class_name: str) -> List[JType]: + klass = self.get_class(qualified_class_name) + return list(klass.types.values()) if klass is not None else [] - def get_class_call_graph(self, qualified_class_name: str, method_name: str | None = None) -> List[Tuple[JMethodDetail, JMethodDetail]]: - if method_name is None: - filter_criteria = {node for node in self.call_graph.nodes if node[1] == qualified_class_name} - else: - filter_criteria = {node for node in self.call_graph.nodes if tuple(node) == (method_name, qualified_class_name)} - graph_edges: List[Tuple[JMethodDetail, JMethodDetail]] = list() - for edge in self.call_graph.edges(nbunch=filter_criteria): - source: JMethodDetail = self.call_graph.nodes[edge[0]]["method_detail"] - target: JMethodDetail = self.call_graph.nodes[edge[1]]["method_detail"] - graph_edges.append((source, target)) - return graph_edges + def get_extended_classes(self, qualified_class_name: str) -> List[str]: + klass = self.get_class(qualified_class_name) + return klass.extends_list if klass is not None else [] - def remove_all_comments(self, src_code: str) -> str: - raise NotImplementedError("This function is not implemented yet.") + def get_implemented_interfaces(self, qualified_class_name: str) -> List[str]: + klass = self.get_class(qualified_class_name) + return klass.implements_list if klass is not None else [] - # ===================================================================================== - # Entry points / CRUD / comments (operate on the reconstructed symbol table) - # ===================================================================================== + # -----[ entry points ]----- def get_all_entry_point_methods(self) -> Dict[str, Dict[str, JCallable]]: - methods = chain.from_iterable( - ((typename, method, callable) for method, callable in methods.items() if callable.is_entrypoint) for typename, methods in self.get_all_methods_in_application().items() - ) - return {typename: {method: callable for _, method, callable in group} for typename, group in groupby(methods, key=lambda x: x[0])} + result: Dict[str, Dict[str, JCallable]] = {} + for name, methods in self.get_all_methods_in_application().items(): + entrypoints = {sig: c for sig, c in methods.items() if c.is_entrypoint} + if entrypoints: + result[name] = entrypoints + return result def get_all_entry_point_classes(self) -> Dict[str, JType]: - return {typename: klass for typename, klass in self.get_all_classes().items() if klass.is_entrypoint_class} + return {name: t for name, t in self._types.items() if t.is_entrypoint_class} - def _crud(self, op_filter: CRUDOperationType | None) -> List[Dict[str, Union[JType, JCallable, List[JCRUDOperation]]]]: - rows = [] - for class_name, class_details in self.get_all_classes().items(): - for method_name, method_details in class_details.callable_declarations.items(): - if method_details.crud_operations and len(method_details.crud_operations) > 0: - ops = method_details.crud_operations if op_filter is None else [o for o in method_details.crud_operations if o.operation_type == op_filter] - rows.append({class_name: class_details, method_name: method_details, "crud_operations": ops}) - return rows + # -----[ CRUD (J-4) ]----- + def get_all_crud_operations(self) -> List[CRUDRow]: + raise CodeanalyzerExecutionException(CRUD_UNAVAILABLE) - def get_all_crud_operations(self) -> List[Dict[str, Union[JType, JCallable, List[JCRUDOperation]]]]: - return self._crud(None) + def get_all_create_operations(self) -> List[CRUDRow]: + raise CodeanalyzerExecutionException(CRUD_UNAVAILABLE) - def get_all_read_operations(self) -> List[Dict[str, Union[JType, JCallable, List[JCRUDOperation]]]]: - return self._crud(CRUDOperationType.READ) + def get_all_read_operations(self) -> List[CRUDRow]: + raise CodeanalyzerExecutionException(CRUD_UNAVAILABLE) - def get_all_create_operations(self) -> List[Dict[str, Union[JType, JCallable, List[JCRUDOperation]]]]: - return self._crud(CRUDOperationType.CREATE) + def get_all_update_operations(self) -> List[CRUDRow]: + raise CodeanalyzerExecutionException(CRUD_UNAVAILABLE) - def get_all_update_operations(self) -> List[Dict[str, Union[JType, JCallable, List[JCRUDOperation]]]]: - return self._crud(CRUDOperationType.UPDATE) + def get_all_delete_operations(self) -> List[CRUDRow]: + raise CodeanalyzerExecutionException(CRUD_UNAVAILABLE) - def get_all_delete_operations(self) -> List[Dict[str, Union[JType, JCallable, List[JCRUDOperation]]]]: - return self._crud(CRUDOperationType.DELETE) + # -----[ repository artifacts — the shared Py* models, as the generic ABC promises ]----- + def get_artifacts(self) -> Dict[str, PyArtifact]: + """Every non-code artifact, keyed by repo-relative path. ``JArtifact.text_truncated`` has no + home on the shared model and is not carried; read it off ``JApplication.artifacts``.""" + return { + path: PyArtifact(**a.model_dump(exclude={"config_keys", "text_truncated"}), config_keys=[PyConfigKey(**ck.model_dump()) for ck in a.config_keys]) + for path, a in self.application.artifacts.items() + } + def get_dependencies(self, *, direct_only: bool = False, ecosystem: str | None = None, declared_in: str | None = None) -> List[PyDependency]: + """Every declared dependency, optionally filtered. The Maven ``group`` coordinate has no home + on the shared model and is not carried; read it off ``JApplication.dependencies``.""" + deps = [PyDependency(**d.model_dump(exclude={"group"})) for d in self.application.dependencies] + if direct_only: + deps = [d for d in deps if d.direct] + if ecosystem is not None: + deps = [d for d in deps if d.ecosystem == ecosystem] + if declared_in is not None: + deps = [d for d in deps if d.declared_in == declared_in] + return deps + + def get_config_keys(self) -> Dict[str, PyConfigKey]: + """Every configuration key flattened out of the config-bearing artifacts, keyed + ``"@key/"`` (``pom.xml@key/project.artifactId``). + + That key is the analyzer's own id with its ``can://artifact//`` prefix dropped: the + application name belongs to the run, not to the key, so keying by the raw id made the two + backends share **zero** keys whenever the graph was emitted under a different ``--app-name`` + than the local run passes (the SDK passes the project directory's name). ``can://`` ids also + stay off the public surface (E6); the id is still on ``PyConfigKey.id``. + """ + return {f"{path}@key/{ck.key}": PyConfigKey(**ck.model_dump()) for path, a in self.application.artifacts.items() for ck in a.config_keys} + + def get_config_uses(self, key: str | None = None) -> List[PyConfigUseEdge]: + """Always empty, and not a projection gap: codeanalyzer-java 3.0.1 emits no code-to-config + edges at all (there is no such relationship type in the Java graph, and no ``config_uses`` + on the Java wire), so the in-memory backend answers the same way.""" + return [] + + def get_unresolved_config_reads(self) -> List[PyConfigRead]: + """Always empty, as on the in-memory backend: codeanalyzer-java 3.0.1 has no config-read + detector.""" + return [] + + # -----[ comments ]----- def get_comments_in_a_method(self, qualified_class_name: str, method_signature: str) -> List[JComment]: - callable = self.get_method(qualified_class_name, method_signature) - return callable.comments if callable is not None else [] + """The method's javadoc — **narrower than the ABC's "the comments in a method"**: the graph + keeps one ``docstring`` per declaration and no other comment, so a non-javadoc comment in + the body is not here (see the module docstring). A javadoc-only subset is still a real + answer under this name, which is why this one narrows where the two file-keyed accessors + refuse (J-16). ``[]`` both for a method with no javadoc and for a missing one, as on the + in-memory backend.""" + method = self.get_method(qualified_class_name, method_signature) + return method.comments if method is not None else [] def get_comments_in_a_class(self, qualified_class_name: str) -> List[JComment]: + """The class's javadoc, narrower than the ABC's "the comments in a class" in exactly the + way :meth:`get_comments_in_a_method` is (J-16).""" klass = self.get_class(qualified_class_name) return klass.comments if klass is not None else [] def get_comment_in_file(self, file_path: str) -> List[JComment]: - compilation_unit = self.get_symbol_table().get(file_path, None) - if compilation_unit is None: - raise CodeanalyzerExecutionException(f"File {file_path} not found in the symbol table.") - return compilation_unit.comments + """Raises: the projection carries no file-level comments at all, so every answer would be + an empty list reading as "this file has no comments" (D7).""" + raise CodeanalyzerExecutionException(_COMMENTS_UNAVAILABLE.format(app=self.application_name)) def get_all_comments(self) -> Dict[str, List[JComment]]: - return {file_path: self.get_comment_in_file(file_path) for file_path in self.get_symbol_table()} - - def get_all_docstrings(self) -> List[Tuple[str, JComment]]: - docstrings = {} - for file_path, list_of_comments in self.get_all_comments().items(): - javadoc_comments = [docstring for docstring in list_of_comments if docstring.is_javadoc] - if javadoc_comments: - docstrings[file_path] = javadoc_comments - return docstrings + """Raises, as :meth:`get_comment_in_file` does: the docstrings that *are* projected are a + strictly smaller set than "every comment", and returning them under this name would be a + silent partial rather than an empty one.""" + raise CodeanalyzerExecutionException(_COMMENTS_UNAVAILABLE.format(app=self.application_name)) + + def get_all_docstrings(self) -> Dict[str, List[JComment]]: + """The javadoc of each file's *declarations* — every declaration the projection gives a + ``docstring``: types, their callables, fields, enum constants and record components. That + is the only comment text in the graph. The in-memory backend reads the compilation unit's + own comment list instead, which additionally holds every file-level javadoc (a licence + header, say) and nothing per declaration; the two therefore report different sets for the + same file. + """ + out: Dict[str, List[JComment]] = {} + for name, t in self._types.items(): + path = self._idx[1][name] + javadoc = list(t.comments) + javadoc += [c for member in t.callables.values() for c in member.comments] + javadoc += [c for f in t.fields.values() for c in f.comments] + javadoc += [c for k in t.enum_constants for c in k.comments] + javadoc += [c for rc in t.record_components for c in rc.comments] + if javadoc: + out.setdefault(path, []).extend(javadoc) + return out + + def remove_all_comments(self, src_code: str) -> str: + raise NotImplementedError("This function is not implemented yet.") diff --git a/cldk/analysis/java/neo4j/reconstruct.py b/cldk/analysis/java/neo4j/reconstruct.py index 131c5209..2cf73ee4 100644 --- a/cldk/analysis/java/neo4j/reconstruct.py +++ b/cldk/analysis/java/neo4j/reconstruct.py @@ -14,274 +14,373 @@ # limitations under the License. ################################################################################ -"""Pure rehydration: Neo4j property maps → ``analysis.json``-shaped dicts for ``cldk.models.java``. - -:class:`~cldk.analysis.java.neo4j.JNeo4jBackend` bulk-fetches every node + relationship for an -application, groups children by parent, and feeds the grouped props here. Each function returns a -plain ``dict`` matching the corresponding pydantic model's field names, so the backend can assemble a -single ``analysis.json``-shaped payload and hand it to ``JApplication(**payload)`` — the exact same -constructor path the in-memory :class:`~cldk.analysis.java.codeanalyzer.JCodeanalyzer` uses -(``_init_japplication``). That guarantees the reconstructed objects are identical. - -The source graph is the one ``codeanalyzer-java`` (>= 2.4.0) emits with ``--emit neo4j`` — see its -``neo4j/GraphProjector.java`` / ``schema.neo4j.json`` for the property flattening these functions -invert. Java comments are first-class ``:JComment`` nodes (``J_HAS_COMMENT``), so unlike the Python -backend they round-trip losslessly. - -Parity caveats (inherent to what the projection stores, not bugs): a ``JType``'s -``is_class_or_interface_declaration`` and ``is_concrete_class`` flags are not projected (only the -``kind`` discriminator is), so they rehydrate to their defaults; the order of ``call_graph`` edges -is sorted rather than original-insertion order. +"""Rebuild the ``cldk.models.java`` (schema v2) models from codeanalyzer-java 3.0.1 Neo4j node and +edge property maps. + +Pure functions: they take the flat property dictionaries the analyzer's Neo4j projection wrote +(``schema.neo4j.json`` at the 3.0.1 tag is the authority for what each label carries) and return +the same pydantic objects the in-memory :class:`~cldk.analysis.java.codeanalyzer.JCodeanalyzer` +returns. :class:`~cldk.analysis.java.neo4j.JNeo4jBackend` fetches the rows and assembles the +containment tree; the per-node shape lives here. + +**Booleans.** The projection writes a boolean property only when it is ``True`` (verified: +``text_truncated`` exists on 16 of 5,007 ``:Artifact`` nodes, ``is_implicit`` on 99 of 1,216 +daytrader8 callables, ``is_wildcard`` only on wildcard imports). An absent boolean therefore *is* +``False`` in the contract, and reading one with a ``False`` default is not a default hiding drift. +Every non-boolean property the contract declares on a label is read with ``props[...]``. + +What the projection does **not** carry, and therefore comes back at the model's own empty default +(measured against the live graph, not assumed): + +* **``JModule.source``** -- the graph stores each *callable's* own text in ``JCallable.code`` and + nothing else, so a reconstructed :class:`JCompilationUnit` has ``source=""``. Its ``span`` is + unknown too (``:JModule`` carries no lines at all), so it rehydrates as the model's own ``-1``. + A callable is pointed at the text the graph did project through :func:`thread_code`; every other + node's ``code`` is ``""``. +* **every column and every byte offset** -- the projection writes ``start_line`` and ``end_line`` + and no position within a line, and no offset into a ``source`` it does not carry. Both are + reported as :data:`_UNKNOWN` (``-1``), the model's own "not known", on every node: a ``0`` would + read as column one and offset zero, which is a position, and a wrong one. **One exception, and it + is deliberate:** a :class:`JCallableParameter` comes back with the analyzer's own columns *and* + byte offsets, because the projection serialises the whole parameter list into + ``:JCallable.parameters_json`` and it round-trips exactly (see :func:`parameters`). Those byte + offsets index the module ``source`` the graph does not carry, so they locate the parameter in the + file on disk and nothing this backend can hand you; ``JCallableParameter.code`` is unreachable on + either backend (a parameter is never threaded to its compilation unit, so slicing raises rather + than returning a silent empty). +* **``JCallable.body_span``** -- the graph projects one line range per callable, the *declaration* + span. So ``JCallable.code`` here is the whole declaration (``public void f() {…}``), where the + local backend's is the body block (``{…}``); ``code_start_line`` is the declaration's first line, + which is the body block's first line too except where the opening brace sits on a later line. +* **comments** -- there are no ``:JComment`` nodes (0 in the reference graph). A type, callable, + field, enum constant and record component carries a single ``docstring`` property holding its + javadoc, rebuilt here as a one-element ``comments`` list; a non-javadoc comment on a declaration, + and every file-level comment, is not projected at all. +* **``JCallable.body``** -- only the ``call`` nodes are rebuilt (what ``call_sites`` is a view + over), which is roughly **30%** of what the graph holds (4,006 of daytrader8's 13,436 + ``:JBodyNode``); the ``entry``/``exit``/``statement``/``branch``/``loop``/``return`` nodes and the + parameter lattice are not. A call site's ``arguments`` (body-key references) and both columns are + not projected either. +* **``JBodyNode.callee``** (226 populated on the committed daytrader8 ``-a 4`` fixture, 0 here) -- the + ``can://`` id of the resolved callee. The projection puts that edge on ``J_RESOLVES_TO``, which + this module reads into ``callee_signature`` instead, and an id has no home on the public surface + (E6) anyway. **It is not an "unresolved" signal here:** ``node.callee is None`` classifies every + call on this backend as unresolved while ``callee_signature`` beside it is fully populated -- + test that instead. +* **``JCallSite.comment``** (47 on the same fixture, 0 here) -- the comment attached + to a call site. It follows from the comment gap above: the graph has no comment node to attach. +* ``cfg`` / ``cdg`` / ``ddg`` / ``summary`` (``None``: 3b reads them per callable on demand), + ``type_parameters``, ``JCompilationUnit.comments``, ``JApplication.param_in`` / ``param_out`` / + ``external_symbols``, and a decorator's / import's / enum constant's / record component's span. """ from __future__ import annotations import json -from typing import Any, Dict, List, Mapping +from typing import Any, Dict, List, Mapping, Optional + +from cldk.models.java.models import ( + JArtifact, + JBodyNode, + JCallable, + JCallableParameter, + JComment, + JCompilationUnit, + JConfigKey, + JDecorator, + JDependency, + JField, + JImport, + JLocalVariable, + JEnumConstant, + JRecordComponent, + JSpan, + JType, +) Props = Mapping[str, Any] -# -----[ helpers ]----- -def _arr(props: Props, key: str) -> List[str]: - return list(props.get(key, []) or []) - - - - -def _kind_flags(kind: str | None) -> Dict[str, bool]: - """Derive the type-discriminator booleans from the projected ``kind`` string.""" - return { - "is_interface": kind == "interface", - "is_enum_declaration": kind == "enum", - "is_annotation_declaration": kind == "annotation", - "is_record_declaration": kind == "record", - } - - -# -----[ leaf nodes ]----- -def comment(props: Props) -> dict: - return { - "content": props.get("content"), - "start_line": props.get("start_line", -1), - "end_line": props.get("end_line", -1), - "start_column": props.get("start_column", -1), - "end_column": props.get("end_column", -1), - "is_javadoc": props.get("is_javadoc", False), - } - - -def parameter(props: Props) -> dict: - return { - "name": props.get("name"), - "type": props.get("type", ""), - "annotations": _arr(props, "annotations"), - "modifiers": _arr(props, "modifiers"), - "start_line": props.get("start_line", -1), - "end_line": props.get("end_line", -1), - "start_column": props.get("start_column", -1), - "end_column": props.get("end_column", -1), - } - - -def field(props: Props, *, comment_node: dict | None = None) -> dict: - raw = props.get("variable_initializers_json") - return { - "comment": comment_node, - "type": props.get("type", ""), - "start_line": props.get("start_line", -1), - "end_line": props.get("end_line", -1), - "variables": _arr(props, "variables"), - "modifiers": _arr(props, "modifiers"), - "annotations": _arr(props, "annotations"), - "variable_initializers": json.loads(raw) if raw else {}, - } - - -def variable(props: Props, *, comment_node: dict | None = None) -> dict: - return { - "comment": comment_node, - "name": props.get("name", ""), - "type": props.get("type", ""), - "initializer": props.get("initializer", ""), - "start_line": props.get("start_line", -1), - "start_column": props.get("start_column", -1), - "end_line": props.get("end_line", -1), - "end_column": props.get("end_column", -1), - } - - -def enum_constant(props: Props) -> dict: - return {"name": props.get("name", ""), "arguments": _arr(props, "arguments")} - - -def record_component(props: Props, *, comment_node: dict | None = None) -> dict: - return { - "comment": comment_node, - "name": props.get("name", ""), - "type": props.get("type", ""), - "modifiers": _arr(props, "modifiers"), - "annotations": _arr(props, "annotations"), - "default_value": props.get("default_value"), - "is_var_args": props.get("is_var_args", False), - } - - -def crud_operation(props: Props) -> dict: - return {"line_number": props.get("line_number", -1), "operation_type": props.get("operation_type")} - - -def crud_query(props: Props) -> dict: - return { - "line_number": props.get("line_number", -1), - "query_arguments": props.get("query_arguments"), - "query_type": props.get("query_type"), - } - - -def callsite(props: Props, *, comment_node: dict | None = None, crud_op: dict | None = None, crud_q: dict | None = None) -> dict: - return { - "comment": comment_node, - "method_name": props.get("method_name", ""), - "receiver_expr": props.get("receiver_expr", ""), - "receiver_type": props.get("receiver_type", ""), - "argument_types": _arr(props, "argument_types"), - "argument_expr": _arr(props, "argument_expr"), - "return_type": props.get("return_type", ""), - "callee_signature": props.get("callee_signature", ""), - "is_static_call": props.get("is_static_call"), - "is_private": props.get("is_private"), - "is_public": props.get("is_public"), - "is_protected": props.get("is_protected"), - "is_unspecified": props.get("is_unspecified"), - "is_constructor_call": props.get("is_constructor_call", False), - "crud_operation": crud_op, - "crud_query": crud_q, - "start_line": props.get("start_line", -1), - "start_column": props.get("start_column", -1), - "end_line": props.get("end_line", -1), - "end_column": props.get("end_column", -1), - } - - -# -----[ declarations ]----- -def init_block( - props: Props, - *, - comments: List[dict] | None = None, - call_sites: List[dict] | None = None, - variable_declarations: List[dict] | None = None, -) -> dict: - return { - "file_path": props.get("file_path", ""), - "comments": comments or [], - "annotations": _arr(props, "annotations"), - "thrown_exceptions": _arr(props, "thrown_exceptions"), - "code": props.get("code", ""), - "start_line": props.get("start_line", -1), - "end_line": props.get("end_line", -1), - "is_static": props.get("is_static", False), - "referenced_types": _arr(props, "referenced_types"), - "accessed_fields": _arr(props, "accessed_fields"), - "call_sites": call_sites or [], - "variable_declarations": variable_declarations or [], - "cyclomatic_complexity": props.get("cyclomatic_complexity", 0), - } +class _ProjectedText: + """Stands in for the owning :class:`JCompilationUnit` on a callable, so its ``code`` view reads + the text the graph projected for it. + + ``:JModule`` carries no ``source``, so a reconstructed unit can slice nothing; ``:JCallable`` + carries its own ``code``. The models reach the unit only through the private ``_unit`` + back-reference, and only for :meth:`JCompilationUnit.slice` and ``package`` -- so pointing a + callable at one of these is what turns ``JCallable.code`` from an empty slice into that text + (see :func:`thread_code`). + """ + + __slots__ = ("_code", "package") + + def __init__(self, code: str, package: str) -> None: + self._code, self.package = code, package + + def slice(self, span: JSpan) -> str: + """The callable's whole projected text: the graph keeps one text per callable, not the + module source the span would index into.""" + return self._code +def thread_code(unit: JCompilationUnit, code: Mapping[str, str]) -> None: + """Point every callable in ``unit`` at its projected text, keyed by node id. + + Runs after the unit is validated, because :meth:`JCompilationUnit.model_post_init` threads + itself onto every node it owns and would otherwise win. + """ + + def walk(t: JType) -> None: + for c in t.callables.values(): + c._unit = _ProjectedText(code.get(c.id, ""), unit.package) + for local in c.types.values(): + walk(local) + for nested in t.types.values(): + walk(nested) + + for top in unit.types.values(): + walk(top) + + +# ---------------------------------------------------------------------------------------------- +# leaves +# ---------------------------------------------------------------------------------------------- +#: The model's own "not known". The graph stores ``start_line``/``end_line`` and nothing else, so +#: every column, every byte offset, and the whole span of a node it carries no lines for, are *not +#: projected* -- reported as this rather than as a ``0``, which would read as "column one, offset +#: zero" and index into a ``source`` that is ``""``. +_UNKNOWN = -1 + + +def span(props: Props) -> Optional[JSpan]: + """The line-only span the projection carries, or ``None`` when it carries no lines (an implicit + callable). Both columns and both byte offsets are :data:`_UNKNOWN`: the graph projects neither + (see the module docstring).""" + start, end = props.get("start_line"), props.get("end_line") + return None if start is None or end is None else JSpan(start=(start, _UNKNOWN), end=(end, _UNKNOWN), bytes=(_UNKNOWN, _UNKNOWN)) + + +def _unknown_span() -> JSpan: + """The span for a node the model requires one on and the projection carries no lines for.""" + return JSpan(start=(_UNKNOWN, _UNKNOWN), end=(_UNKNOWN, _UNKNOWN), bytes=(_UNKNOWN, _UNKNOWN)) + + +def docstring(props: Props) -> List[JComment]: + """The node's javadoc as the one-element ``comments`` list it stands in for (see the module + docstring); empty when the declaration carries none.""" + text = props.get("docstring") + return [JComment(content=text, is_javadoc=True)] if text is not None else [] + + +def decorator(node: Props, edge: Props) -> JDecorator: + """An annotation use from its ``:JAnnotation`` node (keyed by name) and the ``J_ANNOTATED_BY`` + edge's ``arguments`` (the source spellings).""" + return JDecorator(name=node["name"], args=list(edge.get("arguments") or [])) + + +def field(props: Props, decorators: List[JDecorator]) -> JField: + return JField( + id=props["id"], + name=props["name"], + type=props["type"], + modifiers=list(props.get("modifiers") or []), + decorators=decorators, + comments=docstring(props), + initializer=props.get("initializer"), + span=span(props), + ) + + +def variable(props: Props) -> JLocalVariable: + return JLocalVariable(name=props["name"], type=props["type"], initializer=props.get("initializer"), span=span(props)) + + +def enum_constant(props: Props) -> JEnumConstant: + return JEnumConstant(name=props["name"], arguments=list(props.get("arguments") or []), comments=docstring(props)) + + +def record_component(props: Props) -> JRecordComponent: + return JRecordComponent( + name=props["name"], + type=props["type"], + modifiers=list(props.get("modifiers") or []), + comments=docstring(props), + is_variadic=bool(props.get("is_variadic", False)), + ) + + +def parameters(props: Props) -> List[JCallableParameter]: + """``JCallable.parameters_json`` -- the analyzer's own serialisation of the parameter list, so + the parameters (names, types, spans with byte offsets, modifiers, annotations, variadic flag) + round-trip exactly. Absent on a callable that takes none.""" + raw = props.get("parameters_json") + return [JCallableParameter.model_validate(p) for p in json.loads(raw)] if raw else [] + + +def body_node(props: Props, callee_signature: Optional[str]) -> JBodyNode: + """A ``call`` body node. ``callee_signature`` is the ``signature`` of whatever the node's + ``J_RESOLVES_TO`` edge points at -- a project callable or an external -- and ``None`` when the + analyzer left the call unresolved. + + **Columns are the model's own ``-1``, not the body key's.** The key a body node's id ends with + (``@65:28``) spells a *different* position from the node's span: measured over daytrader8's + 4,006 call nodes, the key column equals the ``span.start`` column on only 629 of them, and on + the rest the difference runs from 1 to **110** columns, most often **4** (910 nodes). So the key + is used for what it is -- the ``body`` dict key -- and the column is reported as not projected + rather than as a number that would be wrong. (The graph carries no ``start_column`` on a + ``:JBodyNode`` at all; those figures are measured on the same analyzer's JSON, where the spans + the key would have to agree with do exist.) + """ + start, end = props.get("start_line"), props.get("end_line") + return JBodyNode( + kind=props["kind"], + span=None if start is None or end is None else JSpan(start=(start, _UNKNOWN), end=(end, _UNKNOWN), bytes=(_UNKNOWN, _UNKNOWN)), + method_name=props.get("method_name"), + receiver_expr=props.get("receiver_expr"), + receiver_type=props.get("receiver_type"), + return_type=props.get("return_type"), + accessibility=props.get("accessibility"), + argument_types=list(props.get("argument_types") or []), + argument_expr=list(props.get("argument_expr") or []), + callee_signature=callee_signature, + is_static_call=props.get("is_static_call"), + is_constructor_call=bool(props.get("is_constructor_call", False)), + ) + + +def imports(edge: Props) -> List[JImport]: + """One :class:`JImport` per spelling on a ``J_IMPORTS`` edge. The projection aggregates every + import of a module that resolves to the same target onto one edge carrying their full dotted + ``spellings``, so the simple name is the last dotted segment and the source order within a file + is not recoverable.""" + static, wildcard = bool(edge.get("is_static", False)), bool(edge.get("is_wildcard", False)) + return [JImport(name=s.rsplit(".", 1)[-1], path=s, is_static=static, is_wildcard=wildcard) for s in (edge.get("spellings") or [])] + + +# ---------------------------------------------------------------------------------------------- +# declarations +# ---------------------------------------------------------------------------------------------- def callable_( props: Props, *, - comments: List[dict] | None = None, - parameters: List[dict] | None = None, - call_sites: List[dict] | None = None, - variable_declarations: List[dict] | None = None, - crud_operations: List[dict] | None = None, - crud_queries: List[dict] | None = None, -) -> dict: - return { - "signature": props.get("signature", ""), - "is_implicit": props.get("is_implicit", False), - "is_constructor": props.get("is_constructor", False), - "comments": comments or [], - "annotations": _arr(props, "annotations"), - "modifiers": _arr(props, "modifiers"), - "thrown_exceptions": _arr(props, "thrown_exceptions"), - "declaration": props.get("declaration", ""), - "parameters": parameters or [], - "return_type": props.get("return_type"), - "code": props.get("code", ""), - "start_line": props.get("start_line", -1), - "end_line": props.get("end_line", -1), - "code_start_line": props.get("code_start_line", -1), - "referenced_types": _arr(props, "referenced_types"), - "accessed_fields": _arr(props, "accessed_fields"), - "call_sites": call_sites or [], - "is_entrypoint": props.get("is_entrypoint", False), - "variable_declarations": variable_declarations or [], - "crud_operations": crud_operations or [], - "crud_queries": crud_queries or [], - "cyclomatic_complexity": props.get("cyclomatic_complexity", 0), - } + decorators: List[JDecorator], + body: Dict[str, JBodyNode], + local_variables: List[JLocalVariable], + types: Dict[str, JType], +) -> JCallable: + metrics = props.get("cyclomatic_complexity") + # ``refs`` is a whole-object absence on the wire, not an empty one, and exactly for an implicit + # callable -- there is no body to analyse (measured: 99 of daytrader8's 1,216 callables carry no + # ``refs``, and all 99 are the implicit ones, while 225 non-implicit ones carry two empty + # lists). The graph omits both properties in either case, so ``is_implicit`` is what tells the + # two apart; deriving it from the properties' absence would report 225 as "not computed". + implicit = bool(props.get("is_implicit", False)) + return JCallable( + id=props["id"], + kind=props["kind"], + signature=props["signature"], + declaration=props.get("declaration"), + return_type=props.get("return_type"), + parameters=parameters(props), + modifiers=list(props.get("modifiers") or []), + error_channel=list(props.get("error_channel") or []), + decorators=decorators, + comments=docstring(props), + metrics=None if metrics is None else {"cyclomatic": metrics}, + refs=None if implicit else {"types": list(props.get("referenced_types") or []), "fields": list(props.get("accessed_fields") or [])}, + local_variables=local_variables, + body=body, + types=types, + is_implicit=implicit, + is_entrypoint=bool(props.get("is_entrypoint", False)), + span=span(props), + ) def type_( props: Props, *, - comments: List[dict] | None = None, - callable_declarations: Dict[str, dict] | None = None, - field_declarations: List[dict] | None = None, - enum_constants: List[dict] | None = None, - record_components: List[dict] | None = None, - initialization_blocks: List[dict] | None = None, -) -> dict: - out = { - "is_inner_class": props.get("is_inner_class", False), - "is_local_class": props.get("is_local_class", False), - "is_nested_type": props.get("is_nested_type", False), - "comments": comments or [], - "extends_list": _arr(props, "extends_list"), - "implements_list": _arr(props, "implements_list"), - "modifiers": _arr(props, "modifiers"), - "annotations": _arr(props, "annotations"), - "parent_type": props.get("parent_type", ""), - "nested_type_declarations": _arr(props, "nested_type_declarations"), - "callable_declarations": callable_declarations or {}, - "field_declarations": field_declarations or [], - "enum_constants": enum_constants or [], - "record_components": record_components or [], - "initialization_blocks": initialization_blocks or [], - "is_entrypoint_class": props.get("is_entrypoint_class", False), - } - out.update(_kind_flags(props.get("kind"))) - return out - - -def compilation_unit( - props: Props, - *, - comments: List[dict] | None = None, - import_declarations: List[dict] | None = None, - type_declarations: Dict[str, dict] | None = None, -) -> dict: - return { - "file_path": props.get("file_path", props.get("file_key", "")), - "package_name": props.get("package_name", ""), - "comments": comments or [], - "import_declarations": import_declarations or [], - "type_declarations": type_declarations or {}, - "is_modified": props.get("is_modified", False), - } - - -def call_edge(source: dict, target: dict, props: Props) -> dict: - """A ``JGraphEdges``-shaped raw dict; endpoints resolve via JApplication's lookup table.""" - weight = props.get("weight") - return { - "source": source, - "target": target, - "type": props.get("type", "CALL_DEP"), - "weight": str(weight) if weight is not None else "1", - "source_kind": props.get("source_kind"), - "destination_kind": props.get("destination_kind"), - } + decorators: List[JDecorator], + fields: Dict[str, JField], + callables: Dict[str, JCallable], + types: Dict[str, JType], + enum_constants: List[JEnumConstant], + record_components: List[JRecordComponent], +) -> JType: + return JType( + id=props["id"], + kind=props["kind"], + modifiers=list(props.get("modifiers") or []), + base_types=list(props.get("base_types") or []), + interfaces=list(props.get("interfaces") or []), + decorators=decorators, + comments=docstring(props), + enum_constants=enum_constants, + record_components=record_components, + fields=fields, + callables=callables, + types=types, + is_entrypoint_class=bool(props.get("is_entrypoint", False)), + # ``span`` is required on a type and the projection always carries its lines. + span=span(props) or _unknown_span(), + ) + + +def compilation_unit(props: Props, *, import_declarations: List[JImport], types: Dict[str, JType]) -> JCompilationUnit: + return JCompilationUnit( + id=props["id"], + package=props["package"], + source="", + content_hash=props.get("content_hash"), + imports=import_declarations, + types=types, + span=_unknown_span(), + ) + + +# ---------------------------------------------------------------------------------------------- +# the repository-artifact layer (unprefixed labels; the Java models, not the shared Py* ones -- +# the five ABC accessors convert, exactly as JCodeanalyzer does off the wire) +# ---------------------------------------------------------------------------------------------- +def config_key(props: Props) -> JConfigKey: + return JConfigKey( + id=props["id"], + key=props["key"], + namespace=props["namespace"], + value=props.get("value"), + references=list(props.get("references") or []), + span=span(props), + ) + + +def artifact(props: Props, *, config_keys: List[JConfigKey]) -> JArtifact: + return JArtifact( + id=props["id"], + path=props["path"], + format=props["format"], + roles=list(props.get("roles") or []), + size_bytes=props["size_bytes"], + sha256=props["sha256"], + source=props["source"], + text_truncated=bool(props.get("text_truncated", False)), + extraction=props["extraction"], + config_keys=config_keys, + ) + + +def dependency(edge: Props, package: Props, declared_in: str) -> JDependency: + """A declared dependency from the ``DECLARES_DEPENDENCY`` edge plus its endpoints: the + coordinate off the ``:Package`` node, the declaring manifest's id off the ``:Artifact``. + ``locked_version`` rides a separate ``LOCKS`` edge (a per-package fact, and no relationship of + that type exists in a Maven projection) and stays ``None``.""" + return JDependency( + group=package.get("group"), + name=package["name"], + ecosystem=package["ecosystem"], + spec=edge["spec"], + kind=edge["kind"], + extras=list(edge.get("extras") or []), + declared_in=declared_in, + direct=bool(edge.get("direct", False)), + prov=list(edge.get("prov") or []), + ) diff --git a/cldk/analysis/python/backend.py b/cldk/analysis/python/backend.py index 230b9cef..1d253a29 100644 --- a/cldk/analysis/python/backend.py +++ b/cldk/analysis/python/backend.py @@ -29,19 +29,49 @@ from __future__ import annotations -import base64 -import json -import os -import posixpath from abc import abstractmethod -from bisect import bisect_right -from typing import Callable, Dict, Iterable, List, NamedTuple, Sequence, Tuple +from typing import Dict, List, Sequence, Tuple import networkx as nx from cldk.analysis.commons.backend import AnalysisBackend -from cldk.analysis.commons.results import EdgePage, EntrypointCoverage, FlowPath, FlowPaths, LocateResult, PathHop, Slice, SliceNode -from cldk.utils.exceptions import SelectorNotInGraph + +# The language-neutral rulings live in ``commons`` (leg 2.5a, G4) and are re-exported here under +# the names this backend's callers and tests have always imported them by. +from cldk.analysis.commons.bounds import ( + DEFAULT_DEPTH, + DEFAULT_MAX_NODES, + DEFAULT_MAX_PATHS, + DEFAULT_PAGE_SIZE, + EdgeOrder, + check_depth, + check_distinct_endpoints, + check_max_nodes, + check_max_paths, + check_page_size, + check_selector, + cursor_params, + decode_cursor, + edge_page, + encode_cursor, + keyset_where, + reject_bare_string, +) +from cldk.analysis.commons.graphs import ( + as_slice_node, + bounded_subgraph, + cone_sinks, + edge_sort_key, + flow_path, + hop_sort_key, + sdg_rel_pattern, + sdg_rels, + shortest_walks, + slice_resolved, + via_table, +) +from cldk.analysis.commons.keys import body_key_column, call_graph_scope, resolve_module_key, scope_paths +from cldk.analysis.commons.results import EdgePage, EntrypointCoverage, FlowPaths, LocateResult, Slice, SliceNode from cldk.models.python import ( CdgEdge, CfgEdge, @@ -58,285 +88,24 @@ ) -def resolve_module_key(path: str, keys: Iterable[str]) -> str: - """The symbol-table / graph ``file_key`` naming ``path``, or ``path`` unchanged if none does. - - A caller of :meth:`PythonAnalysisBackend.locate` hands over whatever its scanner printed — - ``./src/app.py``, ``src/../src/app.py``, or an absolute path from the machine the scan ran on — - while both backends are keyed by the project-relative path the analyzer saw. Exact key first, - then the normalised form, then the longest known key the normalised path *ends on a segment - boundary* of (which is what an absolute path is). Returning ``path`` unchanged when nothing - matches is deliberate: the caller then gets ``file_not_in_graph`` naming the path it asked - about, not a silently substituted neighbour. - """ - keys = list(keys) - if path in keys: - return path - norm = posixpath.normpath(str(path).replace(os.sep, "/")) - if norm in keys: - return norm - suffix_matches = [k for k in keys if norm.endswith("/" + k)] - return max(suffix_matches, key=len) if suffix_matches else path - - -def body_key_column(key: str) -> int: - """The start column encoded in a body node's local key (``"21:12"`` -> ``12``), or ``-1``. - - Both backends need one tie-break for two body nodes that span the *same* line — ``if x: return x`` - emits an ``if`` and a ``return`` each spanning one line — and line numbers are the only positional - data the Neo4j projection carries, so the span cannot break it. The local *key* can: it is - ``:`` (sometimes suffixed, as in ``"22:8/actual_in:0"``), it exists on both sides - (locally the ``body`` dict key, over Neo4j the trailing segment of ``@``), and a - larger column is the more deeply nested statement. Comparing the keys as *strings* instead would - order ``"29:10"`` before ``"29:4"`` and pick the outer node, so the column is parsed as an int. - - ``-1`` for a key with no column (the synthetic ``@entry`` / ``@exit`` vertices) — they carry no - span, so they are filtered out before ranking and never reach this. - """ - _, _, col = key.split("/", 1)[0].partition(":") - return int(col) if col.isdigit() else -1 - - -def reject_bare_string(kind: str, values: object) -> None: - """Refuse a single string where a sequence of names is required. +# ``body_key_column`` moved to :mod:`cldk.analysis.commons.keys` (leg 2.5b): its key grammar is +# shared with TypeScript's body nodes, and a second copy is a second thing to keep in step. It is +# imported above and re-exported from here, which is where every Python caller already reads it. - ``paths='pkg/mod.py'`` is not a type error to Python — a string *is* a sequence, of ten - characters — so it used to reach :func:`check_selector` as ten requested paths and come back as - ``10 of 10 paths not in graph: 'p', 'k', 'g', '/', …``. The mistake is the likely one because - the sibling keyword ``module=`` genuinely is single-valued, so both spellings look plausible. - - Raises: - TypeError: ``values`` is a ``str``. - """ - if isinstance(values, str): - raise TypeError(f"{kind}= takes a sequence of names, not a string; pass [{values!r}] to select just that one") - - -def check_selector(kind: str, requested: Sequence[str], missing: Sequence[str]) -> None: - """The one place a scoping keyword's *selection* is judged, for both backends. - - Every scoped accessor — ``get_symbol_table(paths=)``, ``get_classes(module=)``, - ``get_call_graph(roots=)`` — narrows a whole-application enumeration to what the caller named. - Two ways of naming nothing must not both come back as an empty result: - - * **an empty sequence** (``paths=[]``, ``roots=[]``) selected nothing while missing nothing. It - is a caller bug — the argument to omit is the argument that means "everything" — and it - raises the same :class:`ValueError` ``depth=`` without ``roots=`` already does. - * **values that match nothing** are the ambiguous empty the parent spec's D7 calls a defect: - a mistyped path and a module that genuinely declares no classes were the same ``{}``. They - raise :class:`~cldk.utils.exceptions.SelectorNotInGraph`, which names them and stops. It - offers no near-miss candidates on purpose — leg 1.5's E8 puts typo-tolerant matching out of - scope "not in the resolver, not in the error path". - - A **partial** miss raises too. Returning the values that did match would make a result whose - size the caller cannot check against what it asked for, which is the same silence one step - quieter. - - Args: - kind: The keyword's name, as it appears in the caller's own call — ``"paths"``, - ``"module"`` or ``"roots"``. - requested: Everything the keyword named, in the caller's spelling. - missing: The subset of ``requested`` that matched nothing. Callers with no membership - information to bring (``call_graph_scope``, which has not seen the graph yet) pass an - empty sequence and get only the empty-selection check. - - Raises: - ValueError: ``requested`` is empty. - SelectorNotInGraph: ``missing`` is non-empty. - """ - if not requested: - raise ValueError(f"{kind}= selected nothing; omit it to enumerate the whole application") - if missing: - # ``roots=`` is an exact filter, unlike every name-taking accessor on this surface, so a - # correct short name and a typo miss the same way -- the message has to say which - # vocabulary it wanted (see the assessment on PythonAnalysisBackend.get_call_graph). - detail = ( - "roots= takes full signatures (as get_callables_overview() reports them) or @external ids, not bare names; " - "to address a callable by name use resolve_callable(name).callable, or backward_cone / callers_of / call_paths_between" - if kind == "roots" - else None - ) - raise SelectorNotInGraph(kind, list(missing), len(requested), detail=detail) - - -def scope_paths(paths: Sequence[str] | None, keys: Iterable[str], kind: str = "paths") -> List[str] | None: - """Resolve requested module paths to symbol-table keys, or ``None`` for "the whole application". - - Both backends route their ``paths=`` / ``module=`` keywords through here, so the lenient - resolution (:func:`resolve_module_key` — an absolute path or one with native separators finds - its module) and the strictness (:func:`check_selector` — a path naming no module raises) cannot - drift apart between them. - - Args: - paths: What the caller named, or ``None`` for the unscoped call. - keys: The symbol-table keys that exist — ``symbol_table.keys()`` locally, the - application's module ``file_key``s over Neo4j. - kind: The keyword's name for the error message; ``"module"`` for ``get_classes``, whose - single-valued keyword routes through here as a one-element sequence. - - **Resolution is many-to-one, and the result is de-duplicated.** Leniency is the whole point of - :func:`resolve_module_key` — ``"pkg/a.py"`` and ``"/abs/pkg/a.py"`` are two spellings a scanner - may plausibly hand over for the *same* module — so two requested paths legitimately collapse to - one key and the caller gets one entry back. Raising on the collapse would punish the very - caller the leniency exists for; de-duplicating explicitly is what keeps the returned list from - naming the same module twice and asking both backends to fetch it twice. - - Raises: - TypeError: ``paths`` is a bare string (see :func:`reject_bare_string`). - ValueError: ``paths`` is an empty sequence. - SelectorNotInGraph: a path names no module in this application. - """ - reject_bare_string(kind, paths) - if paths is None: - return None - known = list(keys) - resolved = [resolve_module_key(p, known) for p in paths] - check_selector(kind, list(paths), [p for p, r in zip(paths, resolved) if r not in known]) - return list(dict.fromkeys(resolved)) - - -def call_graph_scope(roots: Sequence[str] | None, depth: int | None) -> List[str] | None: - """Normalise :meth:`PythonAnalysisBackend.get_call_graph`'s scoping keywords. - - Returns the roots as a list, or ``None`` for "the whole application" — the unscoped call, - which must keep behaving exactly as it did before the keywords existed. - - Both backends route through this so the two cannot drift apart on what a keyword combination - means (the failure mode Fix 1 of leg 1.5 had to go back and repair on the child-fetch paths). - Whether each root *exists* is checked later, by whichever backend has the graph in hand, but - through the same :func:`check_selector` — see :func:`bounded_subgraph`. - - Raises: - TypeError: ``roots`` is a bare string (see :func:`reject_bare_string`). - ValueError: ``depth`` that is not a positive ``int``, ``depth`` without ``roots``, or an - empty ``roots``. A hop budget with no origin to count from has no meaning, and quietly - returning all 364,752 edges would be the worst of the available answers — the caller - asked for a bounded graph and would be handed an unbounded one with no signal. - ``depth`` is type-checked rather than merely range-checked because the two ways of - getting it wrong are silent otherwise: ``depth="2"`` raised ``TypeError`` from the - comparison, and ``depth=2.5`` was accepted and truncated to 2 by the Cypher/ego-graph - radius. ``bool`` is rejected for the same reason — ``depth=True`` is ``1`` by accident. - """ - check_depth(depth) - reject_bare_string("roots", roots) - if roots is None: - if depth is not None: - raise ValueError("depth= requires roots=; a hop budget needs an origin to count from") - return None - check_selector("roots", list(roots), ()) - return list(roots) - - -def check_depth(depth: int | None) -> int | None: - """``depth`` is a hop budget: ``None`` for unbounded, otherwise an ``int`` of at least 1. - - Type-checked and not merely range-checked, because the two ways of getting it wrong are silent - otherwise: ``depth="2"`` raised ``TypeError`` from somewhere further in, and ``depth=2.5`` was - accepted and truncated to 2 by the Cypher/ego-graph radius. ``bool`` is rejected for the same - reason — ``depth=True`` is ``1`` by accident. - - One function, so ``get_call_graph``, the slices and the reachability accessors cannot come to - disagree about what a hop budget is. - """ - if depth is not None and (not isinstance(depth, int) or isinstance(depth, bool) or depth < 1): - raise ValueError(f"depth must be an int >= 1, got {depth!r}") - return depth - - -def bounded_subgraph(graph: nx.DiGraph, roots: List[str], depth: int | None, declared: Iterable[str]) -> nx.DiGraph: - """The sub-call-graph reachable from ``roots``, within ``depth`` hops when given. - - **Induced**, not path-only: every edge between two reached nodes is kept, including one - pointing back towards a root. A path-only answer would let ``graph.predecessors(n)`` lie about - a node the caller can see, which is a worse defect than the extra edges are a cost. The Neo4j - backend's Cypher is written to produce the same induced shape rather than the cheaper - edges-along-the-path shape, for exactly this reason. - - **The domain a root is judged against — stated here because both backends must judge against - the same one — is the callable inventory, not this graph.** ``graph`` is built from call - *edges* alone, so a callable that neither calls nor is called by anything is not a node in it: - 444 of the live odoo application's 15,549 in-scope callables, 2.9%. Checking membership of - ``graph`` therefore raised for a callable that plainly exists, while the Neo4j backend — whose - Cypher matches a root by node *label*, not by edge participation — returned the one-node graph - it is. ``declared`` closes that gap: it carries every callable the application declares, and a - root is valid when it is **in the inventory or is a node of the graph**. The second disjunct is - not redundant — an ``@external`` ghost is a legitimate root, is a graph node, and is not a - declared callable — and the union is exactly what the Neo4j root match accepts (a - ``:PyCallable`` of this application, or a ``:PyExternal``). - - A root outside that domain raises (:func:`check_selector`) rather than contributing nothing: - "no such callable" and "a callable that calls nothing" are different answers, and before this - they were the same empty graph. - - The returned graph stays **edge-induced**. An isolated root is added back as a lone node — - which is the answer, and the one Neo4j gives — but nothing else the inventory knows about is - seeded into it. Seeding all declared callables would make the unbounded local graph disagree - with Neo4j's node-for-node, trading one parity defect for a larger one. - """ - inventory = set(declared) - check_selector("roots", roots, [r for r in roots if r not in graph and r not in inventory]) - nodes: set = set() - isolated: set = set() - for root in roots: - if root not in graph: - isolated.add(root) # declared, but in no call edge: its own one-node graph - elif depth is None: - nodes |= nx.descendants(graph, root) | {root} - else: - nodes |= set(nx.ego_graph(graph, root, radius=depth).nodes) - sub = graph.subgraph(nodes).copy() - sub.add_nodes_from(isolated) - return sub - - -# ---------------------------------------------------------------------------------------------- -# Paging the per-callable graphs (E5). -# -# THE CANONICAL ORDER, defined once here because it is the only thing that makes a page mean the -# same thing on both backends. Neo4j returns rows in no order unless told to, and the local -# backend returns the analyzer's emission order; without one stated sort, page two on Neo4j is a -# different set of edges from page two locally. Each backend uses these functions -- the local one -# sorts and slices with them directly, the Neo4j one writes the same components into its ORDER BY -# and rebuilds the cursor from them -- so a change here moves both at once. -# -# The order is over the edge's OWN fields, in the order a reader would name them: source, then -# target, then whatever else the edge carries. Nothing positional and nothing backend-specific -# (no relationship element id, no row number), because a key one backend cannot compute is not a -# shared order. -# -# TOTALITY. A keyset cursor resumes strictly *after* a key, so a repeated key would drop its twin. -# The full field tuple is unique on real data: measured across odoo-slim-19's 5,134,655 PY_DDG, -# 247,906 PY_CFG_NEXT and 139,065 PY_CDG edges, zero (src, dst, ...) tuples repeat -- and for CFG -# the endpoints alone are *not* enough (13,310 node pairs carry two edges of different ``kind``), -# which is why ``kind`` is in the key. On the graph side the emitter MERGEs these relationships on -# exactly these properties, so uniqueness is structural there rather than incidental. Two edges -# equal in every field would be equal as values -- the models carry nothing else -- so their -# relative order is unobservable, and the page boundary is the same either way. -# -# ``or ""`` / ``or []`` is not cosmetic: ``DdgEdge.var`` is ``Optional[str]``, and a ``None`` in a -# sort key raises in Python and silently drops the row in Cypher (``null > x`` is null). The -# Cypher spells the same normalisation with ``coalesce``. - -#: Edges per page when the caller does not say. 10,000 is where the measured distribution -#: splits: on odoo-slim-19, 15,520 of the 15,549 callables have fewer than 10,000 DDG edges, so -#: this default answers 99.8% of callables completely in one page and no caller of a normal -#: callable ever writes a loop -- while the 29 that are larger, up to 1,386,918 edges, are held to -#: a response a caller can actually hold. CFG and CDG max out at 402 and 314 edges on the same -#: application, so for them it is never reached. -DEFAULT_PAGE_SIZE = 10_000 +_CFG_KEY, _CDG_KEY, _DDG_KEY = edge_sort_key("cfg"), edge_sort_key("cdg"), edge_sort_key("ddg") def cfg_sort_key(edge: CfgEdge) -> Tuple: """The canonical order for :meth:`PythonAnalysisBackend.get_cfg`: source, target, kind.""" - return (edge.src, edge.dst, edge.kind or "") + return _CFG_KEY(edge) def cdg_sort_key(edge: CdgEdge) -> Tuple: """The canonical order for :meth:`PythonAnalysisBackend.get_cdg`: source, target. A control dependence carries nothing else to break a tie on, and needs nothing else: the pair is unique. """ - return (edge.src, edge.dst) + return _CDG_KEY(edge) def ddg_sort_key(edge: DdgEdge) -> Tuple: @@ -349,24 +118,7 @@ def ddg_sort_key(edge: DdgEdge) -> Tuple: Python and Cypher order lists the same way (element-wise, shorter first on a prefix), verified on the live graph rather than assumed: see ``test_neo4j_orders_a_page_exactly_as_python_would``. """ - return (edge.src, edge.dst, edge.var or "", list(edge.prov or [])) - - -class EdgeOrder(NamedTuple): - """One edge kind's canonical order, in both spellings that have to agree. - - The Python sort key and the Cypher expressions are the same components said twice, in two - languages, and the whole point of the order is that the two never disagree — so they are - written down once, together, and each backend takes the half it can run. ``len(exprs)`` is - also the order's arity, which is how a cursor from one accessor is refused by another - (:func:`decode_cursor`): the three arities are 3, 2 and 4. - - ``coalesce`` in the expressions is ``or ""`` / ``or []`` in the key: ``DdgEdge.var`` is - optional, and a ``None`` in a sort key raises in Python and silently drops the row in Cypher. - """ - - key: Callable[[object], Tuple] - exprs: Tuple[str, ...] + return _DDG_KEY(edge) #: The three orders. ``src``/``dst``/``kind``/``var``/``prov`` are the aliases the backends' @@ -375,335 +127,11 @@ class EdgeOrder(NamedTuple): CDG_ORDER = EdgeOrder(cdg_sort_key, ("src", "dst")) DDG_ORDER = EdgeOrder(ddg_sort_key, ("src", "dst", "coalesce(var,'')", "coalesce(prov,[])")) - -def encode_cursor(scope: str, key: Tuple) -> str: - """An opaque, round-trippable spelling of a sort key, stamped with the callable it came from. - - Opaque on purpose: the caller passes it back and never reads it, so the components of the - order stay an implementation detail rather than joining the caller's vocabulary. Base64 of - JSON, because the key holds strings and a list of strings, and both survive that unchanged. - - ``scope`` is the resolved callable signature, carried so that :func:`decode_cursor` can refuse - a cursor minted for a different callable. Without it, an agent looping over callables and - reusing the wrong ``next_cursor`` would get a plausible page of the *right* callable's edges - resumed from a position in the *wrong* one — silently, since body-node ids sort by callable id - and the filter would simply skip everything or nothing. - """ - return base64.urlsafe_b64encode(json.dumps([scope, list(key)]).encode("utf-8")).decode("ascii") - - -def decode_cursor(cursor: str, scope: str, arity: int) -> Tuple: - """Inverse of :func:`encode_cursor`, checked against the caller it is being used for. - - Three ways a cursor can be wrong, all of them raising rather than being read as "start from - the beginning" — which would silently hand back page one when page nine was asked for: - it does not decode; it was minted for another callable; or it has the wrong number of - components, which is what a cursor from a *different accessor* looks like (the three orders - have arities 3, 2 and 4, so no cursor is silently valid for the wrong graph). - """ - try: - got_scope, key = json.loads(base64.urlsafe_b64decode(cursor.encode("ascii")).decode("utf-8")) - except Exception as exc: # noqa: BLE001 -- any decode failure is the same caller error - raise ValueError(f"not a cursor from a previous page: {cursor!r}") from exc - if got_scope != scope: - raise ValueError(f"this cursor is from a page of {got_scope!r}, not {scope!r}") - if len(key) != arity: - raise ValueError(f"cursor has {len(key)} components, this accessor's order has {arity}: {cursor!r}") - return tuple(key) - - -def check_page_size(page_size: int) -> int: - """``page_size`` must ask for at least one edge. - - Zero is refused rather than treated as "no limit": a page of nothing whose ``next_cursor`` can - never advance is an infinite loop dressed as an empty answer. - """ - if page_size < 1: - raise ValueError(f"page_size must be at least 1, got {page_size}") - return page_size - - -def keyset_where(exprs: Sequence[str]) -> str: - """The Cypher for "strictly after the cursor", written out because Cypher has no tuple - comparison: ``(a, b) > ($c0, $c1)`` has to become - ``a > $c0 OR (a = $c0 AND (b > $c1))``. - - Keyset rather than ``SKIP``: measured on ``Website.configurator_apply`` (1,386,918 DDG edges, - 10,000 per page, query alone), ``SKIP`` costs 2.6s for the first page, 9.0s for the middle one - and 4.3s for the last -- it re-sorts a prefix that grows with the offset -- while this filter - is flat at 3.1s / 2.9s / 2.4s. The offset form is not wrong, it just gets worse the further in - the caller reads, which is the one direction pagination exists to make cheap. - """ - clause = "" - for i in reversed(range(len(exprs))): - expr, param = exprs[i], f"$c{i}" - clause = f"{expr} > {param}" + (f" OR ({expr} = {param} AND ({clause}))" if clause else "") - return clause - - -def cursor_params(cursor: str, scope: str, arity: int) -> Dict[str, object]: - """The ``$c0…$cN`` bindings :func:`keyset_where` reads, from an opaque cursor.""" - return {f"c{i}": v for i, v in enumerate(decode_cursor(cursor, scope, arity))} - - -def edge_page(model, scope: str, edges: List, order: EdgeOrder, page_size: int, cursor: str | None) -> EdgePage: - """One page of an edge set already held in memory. - - The local backend has every edge in hand, so it sorts by ``key`` and slices. The cursor is - resolved by binary search over the sorted keys -- ``bisect_right``, i.e. the first edge - strictly after it -- so it means exactly what :func:`keyset_where` makes it mean on the graph, - rather than an independently-invented position that happens to line up. - """ - check_page_size(page_size) - key = order.key - rows = sorted(edges, key=key) - start = bisect_right([key(e) for e in rows], decode_cursor(cursor, scope, len(order.exprs))) if cursor is not None else 0 - window = rows[start : start + page_size] - more = start + len(window) < len(rows) - return EdgePage[model](edges=window, total=len(rows), next_cursor=encode_cursor(scope, key(window[-1])) if more and window else None) - - -# ---------------------------------------------------------------------------------------------- -# Slicing and reachability (E2, E3, E5). -# -# THE FIVE RELATIONSHIP TYPES A SLICE FOLLOWS, verified against codeanalyzer's own -# ``neo4j/schema.py`` REL_TYPES and against ``CALL db.relationshipTypes()`` on odoo-slim-19 rather -# than copied from a plan -- the names in this leg's plan have been wrong before (PY_CFG_NEXT is -# not PY_CFG). All five exist, with these edge counts on that application: -# -# PY_DDG 5,134,655 data dependence, within a callable (var, prov) -# PY_CDG 139,065 control dependence, within a callable -# PY_PARAM_IN 229,035 actual_in -> formal_in : an argument entering a callee -# PY_PARAM_OUT 133,267 formal_out -> actual_out : a value coming back to the caller -# PY_SUMMARY 453,398 actual_in -> actual_out : a callee's pass-through, at the call site -# -# All five point WITH the flow -- verified on the live graph, where every PY_PARAM_IN runs -# actual_in -> formal_in and every PY_PARAM_OUT runs formal_out -> actual_out, with no exceptions -# in 362,302 edges. So a forward slice follows them and a backward slice follows them reversed; -# there is no per-type direction table to keep straight, which is why they can share one match. -# -# PY_CFG_NEXT is deliberately NOT here. Control *flow* says what runs next; a slice is about what -# a value or a decision depends on, and following successor edges would pull in every later -# statement whether or not it depends on anything -- the "returns the whole callable" bug that a -# non-emptiness assertion cannot catch. -SDG_RELS = ("PY_DDG", "PY_CDG", "PY_PARAM_IN", "PY_PARAM_OUT", "PY_SUMMARY") - -#: The Cypher spelling of :data:`SDG_RELS` for a relationship-type disjunction. -SDG_REL_PATTERN = "|".join(SDG_RELS) - -#: The caller's word for each relationship a path hop can be justified by (E6). The graph's own -#: ``PY_DDG``/``PY_PARAM_IN`` spelling never leaves the backend; both backends translate through -#: this one table so a hop cannot be labelled ``data`` over Neo4j and ``ddg`` locally. -#: -#: ``argument`` and ``return`` are the two interprocedural edges, and they are deliberately not -#: both called "parameter": ``PY_PARAM_IN`` binds a caller's argument to a callee's formal, and -#: ``PY_PARAM_OUT`` binds a callee's result back into the caller. A reader following a path needs -#: to know which way it just crossed a call boundary. -VIA = { - "PY_DDG": "data", - "PY_CDG": "control", - "PY_PARAM_IN": "argument", - "PY_PARAM_OUT": "return", - "PY_SUMMARY": "summary", - "PY_CALLS": "call", -} - -#: Paths per query when the caller does not say. A path list is a set of *witnesses* for a flow, -#: not the flow's extent, and ten worked examples is already more than a reader will follow; the -#: extent question is ``slice_forward``, which reports a ``total``. -DEFAULT_MAX_PATHS = 10 - - -def check_max_paths(max_paths: int) -> int: - """``max_paths`` must admit at least one path. Zero is refused for :func:`check_max_nodes`'s - reason: an empty list whose ``truncated`` says "there were more" answers nothing, and it is - indistinguishable at a glance from "there is no flow".""" - if max_paths < 1: - raise ValueError(f"max_paths must be at least 1, got {max_paths}") - return max_paths - - -def check_distinct_endpoints(src: SliceNode, dst: SliceNode) -> None: - """A path query must have two different endpoints. - - Neo4j's shortest-path search *refuses* a self-question outright ("the shortest path algorithm - does not work when the start and end nodes are the same"), which would otherwise surface as a - raw driver error from one backend and an empty list from the other. Both raise here instead, - and neither answers ``[]``: for a node that genuinely sits on a cycle, ``[]`` would be - indistinguishable from a proved absence of one, which is the ambiguous empty in another - costume. ``reaches(x, x)`` is the accessor that answers the existence question, and it does - terminate (measured: 0.03s, where the obvious ``EXISTS`` spelling never finished). - - Takes the *resolved* endpoints rather than their refs so the message speaks the caller's - vocabulary (E6/E7): a value is named ``'kwargs' within '….configurator_apply'``, a callable - by its signature, and the advice is a call that actually runs -- ``reaches`` takes callable - names, so for a value the cycle question is asked of its enclosing callable. - """ - if src.ref != dst.ref: - return - if src.kind == "callable": - raise ValueError(f"paths from {src.callable!r} to itself are not answered; ask reaches({src.callable!r}, {src.callable!r}) whether a cycle exists") - raise ValueError( - f"paths from {src.name!r} to itself (within {src.callable!r}) are not answered; a value reaches itself only through " - f"recursion, so ask reaches({src.callable!r}, {src.callable!r}) whether the callable is on a call cycle" - ) - - -def hop_sort_key(hops: Sequence[PathHop]) -> Tuple: - """The order two paths are compared in, in the caller's *own* vocabulary. - - E2 makes a path a sequence, which only means something if the *list* of paths is stable too: - ``max_paths`` truncates, and a truncation of a non-deterministic order is not reproducible. - So paths are ordered shortest first, then hop by hop on ``(via, var, to.ref)`` — every term of - which the caller can see in the result it gets back. - - Two hops that are indistinguishable in that vocabulary (parallel edges of the same kind, on - the same variable, between the same two nodes) are left to a backend-local tie-break: the - Neo4j backend appends the relationship's ``elementId``, the local backend keeps the order the - analyzer emitted them in. Either is stable for repeated calls against one graph; neither is - meaningful to a caller, which is why it is last and why nothing above depends on it. - """ - return (len(hops), tuple((h.via, h.var or "", h.to.ref) for h in hops)) - - -def flow_path(nodes: Sequence[SliceNode], edges: Sequence[Tuple[str, "str | None", "Sequence[str] | None"]]) -> FlowPath: - """Join a walk's ``n`` nodes and its ``n - 1`` edges into a :class:`FlowPath`. - - Both backends build paths through here, which is what makes the joining invariant - (``hops[i].to is hops[i + 1].frm``) a property of the construction rather than something each - backend has to be trusted to preserve. ``edges`` are the graph's own relationship types; they - are translated to the caller's word through :data:`VIA` exactly once, here. - - Raises: - KeyError: A relationship type with no word in :data:`VIA` — a new edge kind from a future - analyzer generation, which must be named before it can be reported rather than passed - through in the graph's spelling. - """ - return FlowPath(hops=[PathHop(frm=nodes[i], to=nodes[i + 1], via=VIA[rel], var=var, prov=list(prov or [])) for i, (rel, var, prov) in enumerate(edges)]) - - -def as_slice_node(node: object) -> SliceNode: - """The :class:`~cldk.analysis.commons.results.SliceNode` for anything carrying an address. - - :meth:`PythonAnalysisBackend.describe` takes "anything with a ``ref``" — slice nodes, the - endpoints of a :class:`~cldk.analysis.commons.results.PathHop`, a - :class:`~cldk.analysis.commons.results.LocateResult` — because the addressing layer hands a - caller three shapes and asking them to convert between shapes to hydrate one is the kind of - friction that gets worked around with string surgery. - - A ``SliceNode`` passes through untouched. A ``LocateResult`` is re-expressed as one, keeping - the vocabulary it already speaks: ``module.path`` is the file, ``callable.signature`` the - enclosing callable, ``node.kind`` the position's kind. - - Raises: - TypeError: ``node`` carries neither a ``ref`` nor a ``node_id``, so there is nothing to - look up. Guessing an address from a file and a line is what ``locate`` is for. - """ - if isinstance(node, SliceNode): - return node - ref = getattr(node, "node_id", None) - if ref is None: - raise TypeError(f"describe() needs something carrying a ref (a SliceNode, a path hop endpoint, a locate() result); got {type(node).__name__}") - module, callable_ref, body = node.module, node.callable, getattr(node, "node", None) - return SliceNode( - file=module.path, - line=node.span.start[0], - callable=callable_ref.signature if callable_ref else "", - kind=body.kind if body else "callable", - name=callable_ref.name if callable_ref else None, - source=node.source or None, - ref=ref, - ) - -#: Nodes per slice when the caller does not say. The same 10,000 as :data:`DEFAULT_PAGE_SIZE`, and -#: for a different reason: there, it is where 99.8% of callables fit in one page; here, nothing -#: fits, because the measured distribution has no middle (see -#: :class:`~cldk.analysis.commons.results.Slice`). 10,000 is the largest result that stays -#: readable, and every slice above it is one a caller should be re-asking with ``depth=``. -DEFAULT_MAX_NODES = 10_000 - -#: Hops from the seed when the caller does not say. **Finite, and that is the whole point.** -#: -#: The measured distribution has no middle (see :class:`~cldk.analysis.commons.results.Slice`), so -#: an unbounded default hands a connected seed 10,000 arbitrary nodes of a 195,819-node closure -- -#: an unprincipled 5%, honestly flagged ``truncated`` and useless either way. A finite default -#: answers a *narrower* question *completely* instead, and ``depth=None`` is how a caller asks for -#: the whole cone. -#: -#: 5 is the largest bound at which no measured slice needs ``max_nodes`` at all. Over 120 random -#: ``formal_in`` seeds with callers on odoo-slim-19, node counts by depth: -#: -#: ========= ===== ======= ======= ======= ======== -#: direction depth median p75 max > 10,000 -#: ========= ===== ======= ======= ======= ======== -#: backward 3 14 70 846 0 -#: backward 5 33 188 1,539 0 -#: backward 6 56 324 2,818 0 -#: backward 8 464 2,044 16,028 1 -#: backward None 195,786 195,787 198,306 79 -#: forward 3 12 34 440 0 -#: forward 5 24 63 1,053 0 -#: forward 6 35 166 14,260 1 -#: forward 8 48 402 37,326 2 -#: forward None 71 440,269 440,645 52 -#: ========= ===== ======= ======= ======= ======== -#: -#: 3 is informative but thin; 6 is where a forward slice first exceeds the cap and the default -#: would start truncating again. 5 is the last depth that never does, in either direction. -#: -#: **Which accessors take it, and which deliberately do not.** The three *slices* -#: (``slice_backward``, ``slice_forward``, ``backward_cone``) default to it: a bounded slice is a -#: *complete* answer to a narrower question, and ``total`` says so. The two *predicates* -#: (``reaches``, ``flows_to_call``, ``flows_to_argument``) and the two *path* queries -#: (``paths_between``, ``call_paths_between``) default to ``None`` -- unbounded -- because a hop -#: budget on a boolean or a path list is not a smaller answer but a **wrong** one: "no flow" and -#: "no flow within five hops" collapse into the same ``False`` / ``[]`` with nothing in the result -#: to tell them apart. Measured on odoo-slim-19: ``flows_to_call("kwargs", "Website.create", -#: within="Website.configurator_apply")`` is ``False`` at five hops and ``True`` unbounded, and -#: the matching ``paths_between`` is ``[]`` at five hops and ten paths at eight. ``depth=`` stays -#: on all five as an explicit narrowing a caller can name; it is only the *default* that differs. -DEFAULT_DEPTH = 5 - - -def check_max_nodes(max_nodes: int) -> int: - """``max_nodes`` must admit at least one node — the seed, if nothing else. - - Zero is refused rather than read as "no limit": a slice of nothing whose ``total`` says - 195,784 is a result no caller can act on, and "unbounded" is what ``max_nodes=None`` would - have to mean if it ever meant anything. - """ - if max_nodes < 1: - raise ValueError(f"max_nodes must be at least 1, got {max_nodes}") - return max_nodes - - -def cone_sinks(resolve: Callable[[str], SliceNode], sinks: Sequence[str]) -> List[SliceNode]: - """Resolve ``backward_cone``'s sinks, refusing the two ways of naming nothing. - - The same discipline :func:`check_selector` applies to ``roots=`` and ``paths=``: a bare string - is ten one-character sinks and is refused as a type error, and an empty sequence is refused - because "everything" is the argument omitted, not the argument emptied — and there is no - "everything" here to fall back to. Each surviving name goes through ``resolve``, so an - ambiguous sink raises listing candidates instead of one of them being picked. - - Duplicates are collapsed by resolved signature, not by the string the caller wrote: naming the - same callable twice, once bare and once qualified, is one sink. - """ - reject_bare_string("sinks", sinks) - if not sinks: - raise ValueError("sinks= names nothing to walk back from; pass at least one callable") - resolved = {node.callable: node for node in (resolve(s) for s in sinks)} - return list(resolved.values()) - - -def slice_resolved(roots: List[SliceNode]) -> str: - """The audit line on a :class:`~cldk.analysis.commons.results.Slice`: what the caller's names - matched, in the caller's vocabulary. - - Both backends build it here rather than each formatting its own, so a caller comparing two - results is comparing answers and not two spellings of one. - """ - return ", ".join(f"{r.callable} {r.kind} {r.name!r}" if r.kind != "callable" else r.callable for r in roots) +#: The five relationship types a slice follows and the caller's word for each, in this language's +#: ``PY_`` spelling -- see :func:`~cldk.analysis.commons.graphs.sdg_rels` for what they are and why. +SDG_RELS = sdg_rels("PY") +SDG_REL_PATTERN = sdg_rel_pattern("PY") +VIA = via_table("PY") class PythonAnalysisBackend(AnalysisBackend[PyApplication, PyModule, PyClass, PyCallable, PyClassAttribute, str]): @@ -983,15 +411,16 @@ def get_config_readers(self, key: str) -> List[PyCallableOverview]: # -----[ locate ]----- # The v2 query-facade spec's D3. Declared here rather than on the generic cross-language ABC - # because LocateResult carries codeanalyzer-python's BodyNode/Span — see - # cldk/analysis/commons/backend.py's module docstring for why that stays out of the shared - # contract until a second language implements it. + # because it has one implementation, not because of its types: TS-1 (leg 2.5b) made + # LocateResult language-neutral (BodyRef + the commons Span), so the type blocker is gone and + # the declaration hoists with TypeScript's implementation — see + # cldk/analysis/commons/backend.py's module docstring. @abstractmethod def locate(self, path: str, line: int) -> LocateResult: """Resolve a source position to its enclosing callable, with the source in hand. Four outcomes, kept distinguishable rather than collapsed into an ambiguous empty: inside a - callable (``callable`` set, and ``node`` set too when a body node is that precise); at module + callable (``callable`` set, and ``body`` set too when a body node is that precise); at module scope (a real position with no enclosing callable — a ``module_scope`` diagnostic); in the gap between two callables (also module scope, and never silently snapped to the nearest callable); or in a file the graph has no module for (``file_not_in_graph``). @@ -1117,7 +546,7 @@ def get_source(self, node_id: str) -> str: Generalises body access below callable granularity: ``node_id`` is either a callable's signature (the same key :meth:`get_method_bodies` uses) or the opaque body-node id - :attr:`LocateResult.node_id` hands back alongside :attr:`LocateResult.node`, so a caller + :attr:`LocateResult.node_id` hands back alongside :attr:`LocateResult.body`, so a caller can re-fetch the precise statement or call site :meth:`locate` found, not just its enclosing callable. The body-node form is the analyzer's own id (``"@"``) — round-tripped, never composed by the caller. diff --git a/cldk/analysis/python/codeanalyzer/codeanalyzer.py b/cldk/analysis/python/codeanalyzer/codeanalyzer.py index c27f62cd..88efdc4e 100644 --- a/cldk/analysis/python/codeanalyzer/codeanalyzer.py +++ b/cldk/analysis/python/codeanalyzer/codeanalyzer.py @@ -50,6 +50,7 @@ from __future__ import annotations import logging +from functools import partial from pathlib import Path from typing import Dict, Iterator, List, Sequence, Tuple, Union @@ -60,8 +61,9 @@ from codeanalyzer.schema import Analysis, model_dump_json from cldk.analysis import AnalysisLevel +from cldk.analysis.commons.levels import ANALYZER_LEVELS, LEVEL_NAMES, analyzer_level from cldk.analysis.commons.resolve import CallableCandidate, body_node_kind, resolve_callable_signature, resolve_value_name, resolve_within, value_candidate -from cldk.analysis.commons.results import CallableRef, Diagnostic, EdgePage, EntrypointCoverage, FlowPaths, LocateResult, ModuleRef, Slice, SliceNode, TypeRef +from cldk.analysis.commons.results import BodyRef, CallableRef, Diagnostic, EdgePage, EntrypointCoverage, FlowPaths, LocateResult, ModuleRef, Slice, SliceNode, TypeRef from cldk.utils.exceptions import CodeanalyzerUsageException from cldk.analysis.python.backend import ( CDG_ORDER, @@ -86,6 +88,7 @@ flow_path, resolve_module_key, scope_paths, + shortest_walks, slice_resolved, ) from cldk.models.python import ( @@ -119,39 +122,15 @@ #: table + Jedi call graph, 2 = + defuse-linker call graph, 3 = + intraprocedural dataflow #: (CFG/CDG/DDG), 4 = + interprocedural SDG (``formal_in``/``formal_out`` vertices, alias-aware #: DDG). The four SDK names line up with those four integers in order. -_ANALYZER_LEVELS = { - AnalysisLevel.symbol_table: 1, - AnalysisLevel.call_graph: 2, - AnalysisLevel.program_dependency_graph: 3, - AnalysisLevel.system_dependency_graph: 4, -} - -#: The inverse, by the member name a caller writes (``"call_graph"``, not ``"call graph"``) — so -#: an error about the level in use names it the way it was asked for. -_LEVEL_NAMES = {n: lvl.name for lvl, n in _ANALYZER_LEVELS.items()} - - -def analyzer_level(level: "AnalysisLevel | str") -> int: - """The analyzer's integer level for one of the SDK's :class:`~cldk.analysis.AnalysisLevel` - names. - - Accepts the enum, its value (``"call graph"``) and its member name (``"call_graph"``): the - facade's parameter is typed ``str``, and the underscore spelling is what a caller writing - ``analysis_level="system_dependency_graph"`` produces. An unrecognised name raises rather than - falling back to a default — a level that silently becomes 1 is the defect this function exists - to close. - - ``AnalysisOptions``'s other two dataflow knobs are left at their defaults on purpose: - ``graphs="cfg,dfg,pdg,sdg"`` already selects every section the SDK can surface (``sdg`` is - inert below level 4, where it only widens a ``want_pdg`` that ``pdg`` already sets), and - ``graph_field_depth=3`` is the analyzer's own access-path k-limit, which the SDK exposes no - parameter for. - """ - key = str(getattr(level, "value", level)).replace("_", " ") - try: - return _ANALYZER_LEVELS[AnalysisLevel(key)] - except ValueError: - raise ValueError(f"unknown analysis_level {level!r}; expected one of {[lvl.name for lvl in AnalysisLevel]}") from None +#: The analyzer's ``-a`` integer for each SDK level and its inverse — lifted to +#: :mod:`cldk.analysis.commons.levels` (TypeScript sends the same integers); re-bound here under +#: the names this module and its tests always used. ``AnalysisOptions``'s other two dataflow knobs +#: are left at their defaults on purpose: ``graphs="cfg,dfg,pdg,sdg"`` already selects every +#: section the SDK can surface (``sdg`` is inert below level 4, where it only widens a +#: ``want_pdg`` that ``pdg`` already sets), and ``graph_field_depth=3`` is the analyzer's own +#: access-path k-limit, which the SDK exposes no parameter for. +_ANALYZER_LEVELS = ANALYZER_LEVELS +_LEVEL_NAMES = LEVEL_NAMES def body_node_id(callable_id: str, body_key: str) -> str: @@ -1440,58 +1419,12 @@ def _call_neighbours(self, name: str, in_class: str | None, in_module: str | Non return out # -----[ paths, mixed queries, hydration ]----- - @staticmethod - def _shortest_walks(edges: Dict[str, Dict[str, list]], src: str, dst: str, depth: int | None, limit: int) -> List[list]: - """Up to ``limit`` shortest ``src``->``dst`` walks over ``edges``, in the documented order. - - The local twin of the graph's ``allShortestPaths``, and only shortest walks for its reason: - enumerating every walk does not terminate on a real dependence graph. - - Two passes. The first is the same breadth-first level walk :meth:`_reach` does, keeping the - hop count each node was *first* reached at; the second is a depth-first replay that only - ever steps to a node whose recorded distance is exactly one more than the walk so far, so - it visits shortest walks and nothing else. - - The replay's branch order is ``(via, var, to)`` -- exactly the per-hop key - :func:`~cldk.analysis.python.backend.hop_sort_key` documents -- and every walk found has - the same length, so a pre-order depth-first traversal emits them already sorted. That is - what makes ``limit`` a *prefix* of a total order rather than whichever ``limit`` walks the - recursion happened to find first. - """ - dist, frontier, hops = {src: 0}, [src], 0 - while frontier and dst not in dist and (depth is None or hops < depth): - hops += 1 - nxt = [] - for s in frontier: - for d in edges.get(s, ()): - if d not in dist: - dist[d] = hops - nxt.append(d) - frontier = nxt - if dst not in dist or dist[dst] == 0: - return [] - target, out = dist[dst], [] - - def walk(node: str, walked: list) -> None: - if len(walked) == target: - if node == dst: - out.append(list(walked)) - return - options = sorted( - (VIA[rel], var or "", d, (rel, var, prov)) - for d, labels in edges.get(node, {}).items() - if dist.get(d) == len(walked) + 1 - for rel, var, prov in labels - ) - for _, _, d, label in options: - walked.append((d, label)) - walk(d, walked) - walked.pop() - if len(out) >= limit: - return - - walk(src, []) - return out + #: Up to ``limit`` shortest walks over a ``{src: {dst: [label]}}`` adjacency, in + #: :func:`~cldk.analysis.python.backend.hop_sort_key` order. Lifted to + #: :func:`~cldk.analysis.commons.graphs.shortest_walks` (leg 2.5b) with the ``via`` table as its + #: one parameter -- it is a graph algorithm over strings and knows no language, and TypeScript's + #: local backend answers ``paths_between`` with the same two passes. + _shortest_walks = staticmethod(partial(shortest_walks, via=VIA)) def _value_paths(self, a: SliceNode, b: SliceNode, depth: int | None, max_paths: int) -> FlowPaths: """Build the :class:`FlowPaths` for value ``a`` -> value ``b``.""" @@ -1499,7 +1432,7 @@ def _value_paths(self, a: SliceNode, b: SliceNode, depth: int | None, max_paths: walks = self._shortest_walks(adjacency["forward"], a.ref, b.ref, depth, max_paths + 1) described = {ref: _local_slice_node(nodes[ref], ref) for walk in walks for ref, _ in walk if ref in nodes} described[a.ref] = a - paths = [flow_path([described[a.ref]] + [described[ref] for ref, _ in walk], [label for _, label in walk]) for walk in walks[:max_paths]] + paths = [flow_path([described[a.ref]] + [described[ref] for ref, _ in walk], [label for _, label in walk], via=VIA) for walk in walks[:max_paths]] return FlowPaths(paths=paths, complete=len(walks) <= max_paths) def paths_between(self, src: str, dst: str, *, src_within: str, dst_within: str, depth: int | None = None, max_paths: int = DEFAULT_MAX_PATHS) -> FlowPaths: @@ -1537,7 +1470,7 @@ def call_paths_between(self, src: str, dst: str, *, depth: int | None = None, ma externals = self.get_external_symbols() described = {sig: self._call_graph_node(sig, externals) for walk in walks for sig, _ in walk} described[a] = self._call_graph_node(a, externals) - paths = [flow_path([described[a]] + [described[sig] for sig, _ in walk], [label for _, label in walk]) for walk in walks[:max_paths]] + paths = [flow_path([described[a]] + [described[sig] for sig, _ in walk], [label for _, label in walk], via=VIA) for walk in walks[:max_paths]] return FlowPaths(paths=paths, complete=len(walks) <= max_paths) def _call_graph_node(self, signature: str, externals: Dict[str, PyExternalSymbol]) -> SliceNode: @@ -1734,7 +1667,7 @@ def _not_analysed(self, path: str, line: int) -> LocateResult: on_disk = Path(path).is_file() or bool(project_dir and (Path(project_dir) / path).is_file()) why = "the file exists but no analysed module covers it" if on_disk else "no such file in the analysed project" return LocateResult( - node=None, + body=None, callable=None, type=None, module=ModuleRef(path=str(path)), @@ -1758,7 +1691,7 @@ def _locate_one(self, path: str, line: int) -> LocateResult: found = _find_innermost(module, line) if found is None: return LocateResult( - node=None, + body=None, callable=None, type=None, module=module_ref, @@ -1776,8 +1709,11 @@ def _locate_one(self, path: str, line: int) -> LocateResult: # a bare ``"line:col"``, so this path was already right; it routes through the shared # helper so it stays right if that ever changes. node, node_id = (found_body[1], body_node_id(c.id, found_body[0])) if found_body else (None, None) + # ``BodyRef`` is the language-neutral handle (TS-1): id, kind, span, and -- unlike the graph + # backend, where callee resolution is a separate ``PY_RESOLVES_TO`` edge and not a node + # property -- the callee the analyzer already resolved on a call node. return LocateResult( - node=node, + body=BodyRef(id=node_id or "", kind=node.kind, span=node.span, callee=node.callee) if node else None, node_id=node_id, callable=CallableRef(signature=c.signature, name=c.name, class_signature=owner.signature if owner else None), type=TypeRef(signature=owner.signature, name=owner.name) if owner else None, diff --git a/cldk/analysis/python/neo4j/neo4j_backend.py b/cldk/analysis/python/neo4j/neo4j_backend.py index 30097ea5..688d28b4 100644 --- a/cldk/analysis/python/neo4j/neo4j_backend.py +++ b/cldk/analysis/python/neo4j/neo4j_backend.py @@ -81,7 +81,6 @@ from __future__ import annotations import logging -import re from collections import defaultdict from contextlib import contextmanager from functools import cached_property @@ -92,8 +91,10 @@ from codeanalyzer.schema.ids import application_id, module_id from codeanalyzer.schema.py_schema import PyEntrypointReport +from cldk.analysis.commons.backend import semver as _semver +from cldk.analysis.commons.keys import module_key_of from cldk.analysis.commons.resolve import CallableCandidate, body_node_kind, resolve_callable_signature, resolve_value_name, resolve_within, value_candidate -from cldk.analysis.commons.results import CallableRef, Diagnostic, EdgePage, EntrypointCoverage, FlowPath, FlowPaths, LocateResult, ModuleRef, PathHop, Slice, SliceNode, TypeRef +from cldk.analysis.commons.results import BodyRef, CallableRef, Diagnostic, EdgePage, EntrypointCoverage, FlowPath, FlowPaths, LocateResult, ModuleRef, PathHop, Slice, SliceNode, TypeRef from cldk.analysis.python.backend import ( CDG_ORDER, CFG_ORDER, @@ -151,11 +152,6 @@ logger = logging.getLogger(__name__) -def _semver(raw: Any) -> Tuple[int, int, int] | None: - """``"1.4.1"`` (or ``"1.4.1.post0"``) as ``(1, 4, 1)``; ``None`` for anything that does not - start with three dotted integers, so an unparsable version is *unknown*, never silently zero.""" - m = re.match(r"(\d+)\.(\d+)\.(\d+)", raw) if isinstance(raw, str) else None - return (int(m[1]), int(m[2]), int(m[3])) if m else None # One statement per parent->child collection, each fetching that whole collection for the *entire* # application in a single round trip and returning the parent's key as ``pk``. These are the bulk @@ -485,7 +481,7 @@ def _scope_prefix(self) -> str: @cached_property def _module_set(self) -> FrozenSet[str]: - """:attr:`_modules` as a set -- the membership side of :func:`~cldk.analysis.python.neo4j.reconstruct.module_key_of`. + """:attr:`_modules` as a set -- the membership side of :func:`~cldk.analysis.commons.keys.module_key_of`. The list stays the Cypher parameter (the driver does not pack a set); this is the view every projected row's key is verified against, built once.""" return frozenset(self._modules) @@ -502,13 +498,13 @@ def _module_key(self, node_id: str) -> str: miss is a genuine defect and is raised as such, without the id (E6). """ try: - return R.module_key_of(node_id, self._scope_prefix, self._module_set) + return module_key_of(node_id, self._scope_prefix, self._module_set) except KeyError: pass self._modules = self._load_module_keys() self.__dict__.pop("_module_set", None) # drop the cached frozenset; rebuilt on next read try: - return R.module_key_of(node_id, self._scope_prefix, self._module_set) + return module_key_of(node_id, self._scope_prefix, self._module_set) except KeyError: raise CodeanalyzerExecutionException( f"A node of application {self.application_name!r} belongs to none of the {len(self._module_set)} module keys the graph " @@ -1612,7 +1608,7 @@ def _paths(self, query: str, node_of, a: SliceNode, b: SliceNode, *, src: str, d are the keys the query matches them by.""" check_distinct_endpoints(a, b) rows = self._run(query.format(rels=SDG_REL_PATTERN, depth="" if depth is None else depth), src=src, dst=dst, cap=max_paths + 1, prefix=self._scope_prefix) - paths = [flow_path([node_of(n, self._module_key) for n in r["ns"]], [(e["via"], e["var"], e["prov"]) for e in r["rs"]]) for r in rows[:max_paths]] + paths = [flow_path([node_of(n, self._module_key) for n in r["ns"]], [(e["via"], e["var"], e["prov"]) for e in r["rs"]], via=VIA) for r in rows[:max_paths]] return FlowPaths(paths=paths, complete=len(rows) <= max_paths) # Argument validation precedes name resolution on every accessor below, as it does on the @@ -1964,7 +1960,7 @@ def _locate_result(self, path: str, line: int, rows: List[Dict[str, Any]]) -> Lo module_props = next((r["module_props"] for r in rows if r["module_props"] is not None), None) if module_props is None: return LocateResult( - node=None, + body=None, callable=None, type=None, module=ModuleRef(path=path), @@ -2005,7 +2001,7 @@ def _locate_result(self, path: str, line: int, rows: List[Dict[str, Any]]) -> Lo # built and may not have the project checked out), and concatenating the callables' # ``code`` would silently drop every module-level statement. return LocateResult( - node=None, + body=None, callable=None, type=None, module=module_ref, @@ -2026,8 +2022,11 @@ def _locate_result(self, path: str, line: int, rows: List[Dict[str, Any]]) -> Lo cprops, clsprops = best_row["callable_props"], best_row["class_props"] found_body = self._innermost_body_node(rows, cprops["signature"]) node, node_id = (found_body[1], found_body[0]) if found_body else (None, None) + # ``BodyRef.callee`` is projection-lossy here and always ``None``: callee resolution is the + # separate ``PY_RESOLVES_TO`` edge, not a property of the body node (see + # :func:`~cldk.analysis.python.neo4j.reconstruct.body_node`). The local backend fills it. return LocateResult( - node=node, + body=BodyRef(id=node_id or "", kind=node.kind, span=node.span, callee=node.callee) if node else None, node_id=node_id, callable=CallableRef(signature=cprops["signature"], name=cprops["name"], class_signature=clsprops["signature"] if clsprops else None), type=TypeRef(signature=clsprops["signature"], name=clsprops["name"]) if clsprops else None, diff --git a/cldk/analysis/python/neo4j/reconstruct.py b/cldk/analysis/python/neo4j/reconstruct.py index 20b895a9..165377b9 100644 --- a/cldk/analysis/python/neo4j/reconstruct.py +++ b/cldk/analysis/python/neo4j/reconstruct.py @@ -38,20 +38,24 @@ from __future__ import annotations import json -from typing import Any, Collection, Dict, List, Mapping +from typing import Any, Dict, List, Mapping +# ``module_key_of`` lives in ``commons.keys`` (leg 2.5a, G4); re-exported for the callers that have +# always addressed it through this module. +from cldk.analysis.commons.keys import module_key_of # noqa: F401 + +# The artifact-layer reconstructors (``artifact`` / ``config_key`` / ``dependency``) live in +# ``commons.artifacts`` (leg 2.5a): the layer is projected identically by every analyzer. +from cldk.analysis.commons.artifacts import artifact, config_key, dependency # noqa: F401 from cldk.models.python import ( BodyNode, - PyArtifact, PyCallable, PyCallableOverview, PyClass, PyClassAttribute, PyClassOverview, PyComment, - PyConfigKey, PyConfigRead, - PyDependency, PyExternalSymbol, PyImport, PyModule, @@ -68,26 +72,6 @@ Props = Mapping[str, Any] -# -----[ ids ]----- -def module_key_of(node_id: str, prefix: str, known: Collection[str]) -> str: - """The repo-relative module key embedded in a ``can://`` id (F4). - - Ids are ``/`` (or exactly ```` for a module), and a - file key can itself contain ``.py/`` as a directory name, so the key is never recovered by - splitting: every ``/``-boundary prefix of the id is tried longest first and the first that is - a member of ``known`` -- the application's verified module keys -- wins. A miss raises: a key - we cannot verify is a defect, not a guess. ``known`` should be a set; this runs once per row. - """ - if not node_id.startswith(prefix): - raise KeyError(node_id) - parts = node_id[len(prefix) :].split("/") - for n in range(len(parts), 0, -1): - candidate = "/".join(parts[:n]) - if candidate in known: - return candidate - raise KeyError(node_id) - - # -----[ helpers ]----- def comments(props: Props) -> List[PyComment]: """Rebuild the (lossy) comment list from the single ``docstring`` property.""" @@ -209,73 +193,6 @@ def body_node(props: Props) -> BodyNode: ) -def config_key(props: Props) -> PyConfigKey: - """Rebuild a :class:`PyConfigKey` from a ``:ConfigKey`` node's properties. - - Line-only ``span`` (see :func:`body_node`): the projection writes ``start_line``/``end_line`` - and nothing finer, so the columns and byte offsets rehydrate as ``0``. ``span`` stays ``None`` - when the node carries no lines at all (best-effort extraction never located the key in the - artifact's source). - """ - lines = (props.get("start_line"), props.get("end_line")) - return PyConfigKey( - id=props.get("id", ""), - key=props.get("key", ""), - namespace=props.get("namespace", ""), - value=props.get("value"), - span=Span(start=(lines[0], 0), end=(lines[1], 0), bytes=(0, 0)) if None not in lines else None, - references=list(props.get("references", []) or []), - ) - - -def artifact(props: Props, *, config_keys: List[PyConfigKey] | None = None) -> PyArtifact: - """Rebuild a :class:`PyArtifact` from an ``:Artifact`` node's properties plus its fetched - :class:`PyConfigKey` children (``[:DEFINES_CONFIG]``). - - ``kind`` is not a projected property — every ``PyArtifact`` the analyzer emits carries the - model's own default (``"artifact"``; see ``codeanalyzer/artifacts/discovery.py``), so it is - supplied here rather than queried for. - """ - return PyArtifact( - id=props.get("id", ""), - kind="artifact", - path=props.get("path", ""), - format=props.get("format", ""), - roles=list(props.get("roles", []) or []), - size_bytes=props.get("size_bytes", 0), - sha256=props.get("sha256", ""), - source=props.get("source", ""), - extraction=props.get("extraction", "none"), - config_keys=config_keys or [], - ) - - -def dependency(props: Props, *, name: str, ecosystem: str, declared_in: str) -> PyDependency: - """Rebuild a :class:`PyDependency` from a ``[:DECLARES_DEPENDENCY]`` edge's properties plus its - endpoints (``name``/``ecosystem`` off the ``:Package`` node, ``declared_in`` off the - ``:Artifact`` node). ``ecosystem`` is a real ``Package`` property (``neo4j/schema.py``'s - ``Package`` node type carries it); ``"pypi"`` is only ever what the analyzer happens to write - there today (its only ecosystem, per ``PyDependency.ecosystem``'s own docstring) — read off the - node rather than hardcoded, so this doesn't silently go stale the day a second ecosystem ships. - - ``locked_version``/``provides_imports`` are projection-lossy here: the graph carries them on - the separate ``[:LOCKS]``/``[:PY_PROVIDES]`` edges (per-package facts, not per-declaration), and - no caller of this reconstruction chases those yet, so they come back at the model's own empty - defaults — the same class of gap :func:`callsite` documents for ``argument_types``. - """ - return PyDependency( - name=name, - ecosystem=ecosystem, - spec=props.get("spec", ""), - kind=props.get("kind", "runtime"), - extras=list(props.get("extras", []) or []), - declared_in=declared_in, - direct=props.get("direct", True), - provides_imports=[], - prov=list(props.get("prov", []) or []), - ) - - def unresolved_config_read(props: Props, *, callee: str) -> PyConfigRead: """Rebuild a :class:`PyConfigRead` from a ``[:PY_READS_CONFIG_UNRESOLVED]`` edge's properties plus its ``:PyExternal`` ghost endpoint (``callee``). diff --git a/cldk/analysis/python/python_analysis.py b/cldk/analysis/python/python_analysis.py index 56b83966..85c08ab1 100644 --- a/cldk/analysis/python/python_analysis.py +++ b/cldk/analysis/python/python_analysis.py @@ -859,7 +859,7 @@ def locate(self, path: str, line: int) -> LocateResult: ``get_method``, falling back to ``get_callers``, falling back to scanning the symbol table by hand. Four outcomes stay distinguishable — see :class:`~cldk.analysis.commons.results.LocateResult`: inside a callable (``callable`` set, - plus ``node`` when a body node is that precise), at real module scope (``module_scope`` + plus ``body`` when a body node is that precise), at real module scope (``module_scope`` diagnostic), in the gap between two callables (also module scope, never snapped to the nearest callable), or in a file the graph has no module for (``file_not_in_graph``). @@ -943,7 +943,7 @@ def get_source(self, node_id: str) -> str: Generalises :meth:`get_method_bodies` below callable granularity: ``node_id`` is either a callable's signature, or the opaque body-node id :attr:`~cldk.analysis.commons.results.LocateResult.node_id` hands back alongside - :attr:`~cldk.analysis.commons.results.LocateResult.node`, so a statement or call site + :attr:`~cldk.analysis.commons.results.LocateResult.body`, so a statement or call site :meth:`locate` found can be re-fetched precisely, not just the callable enclosing it. Args: diff --git a/cldk/analysis/typescript/backend.py b/cldk/analysis/typescript/backend.py index 2ba00044..a69a58ed 100644 --- a/cldk/analysis/typescript/backend.py +++ b/cldk/analysis/typescript/backend.py @@ -24,74 +24,186 @@ * :class:`~cldk.analysis.typescript.neo4j.TSNeo4jBackend` — answers the *same* queries with Cypher over the graph ``codeanalyzer-typescript`` emits with ``--emit neo4j``. -This ABC formalizes the surface those two share so the façade↔backend relationship is enforced by -the type system (and at instantiation time) instead of matching only by convention. Both backends -subclass it; the façade is typed against it. Backend-specific lifecycle (e.g. the Neo4j driver's -``close()`` / context-manager support) is intentionally *not* part of the contract. - -The vocabulary mirrors :class:`~cldk.analysis.java.codeanalyzer.JCodeanalyzer` / -:class:`~cldk.analysis.python.codeanalyzer.PyCodeanalyzer`, but the node kinds are TypeScript-native -(interfaces, type aliases, enums, namespaces, decorators, ...). +The shape shared with every other language — application view, symbol table, call graph, the +class/method/field lookups and the repository-artifact layer — is inherited from the generic +:class:`~cldk.analysis.commons.backend.AnalysisBackend`; what is declared here is the +TypeScript-native remainder (interfaces, type aliases, enums, namespaces, decorators, the +1.x call-site accessors and the bulk projections). Both backends subclass it; the façade is typed +against it. Backend-specific lifecycle (e.g. the Neo4j driver's ``close()`` / context-manager +support) is intentionally *not* part of the contract. + +The call graph both backends return keeps TypeScript's own endpoints (decision TS-11): cants emits +a module as the caller of its top-level code and a class as the callee of ``new X()``, and both +are kept, tagged with a ``kind`` node attribute (:data:`CALL_GRAPH_NODE_KINDS`) so a +caller wanting Python's callable-only shape filters in one line rather than the SDK erasing every +top-level call. """ from __future__ import annotations -from abc import ABC, abstractmethod -from typing import Dict, List, Set, Tuple +from abc import abstractmethod +from functools import partial +from typing import ClassVar, Dict, List, Sequence, Set, Tuple import networkx as nx +from cldk.analysis.commons.backend import AnalysisBackend +from cldk.analysis.commons.bounds import ( + DEFAULT_DEPTH, + DEFAULT_MAX_NODES, + DEFAULT_MAX_PATHS, + DEFAULT_PAGE_SIZE, + EdgeOrder, +) +from cldk.analysis.commons.graphs import as_slice_node, edge_sort_key, sdg_rel_pattern, sdg_rels, via_table +from cldk.analysis.commons.keys import module_dotted +from cldk.analysis.commons.results import EdgePage, EntrypointCoverage, FlowPaths, LocateResult, Slice, SliceNode from cldk.models.typescript import ( TSApplication, TSCallable, TSCallableOverview, TSCallsite, + TSCdgEdge, + TSCfgEdge, TSClass, TSClassAttribute, + TSClassOverview, + TSDdgEdge, TSDecorator, TSEnum, TSEnumMember, TSExport, TSExternalSymbol, + TSField, TSImport, TSInterface, TSModule, TSSynthesizedCallable, + TSType, TSTypeAlias, TSVariableDeclaration, ) -class TSAnalysisBackend(ABC): +#: The ``kind`` vocabulary of a call-graph node: what the id index holds — a module, any of the +#: five type kinds (a class is the callee of ``new X()``; the others are indexed and would be kept +#: if the analyzer ever emitted an edge to one), a callable, or an external. +CALL_GRAPH_NODE_KINDS = frozenset({"module", "class", "interface", "enum", "type_alias", "namespace", "callable", "external"}) + +#: Every source extension a TypeScript module key can end in, across both id prefixes. Used to +#: derive a module's dotted name from its key, so ``in_module=`` can be written either way. +TS_EXTENSIONS = (".ts", ".tsx", ".js", ".jsx", ".mjs", ".cjs") + +#: How TypeScript spells a module key as a dotted name. ``package_index=None`` is the ruling, not +#: an omission: :func:`~cldk.analysis.commons.keys.module_dotted` strips a trailing ``/__init__`` +#: because that is how Python addresses a package, and nothing stops a TypeScript project having a +#: file called ``__init__.ts`` -- which is a module in its own right and must dot to +#: ``…__init__``. (superset-frontend has none, so this fires nowhere on the reference corpus; the +#: parameter is here so it cannot fire wrongly on a corpus that does.) TypeScript's own index +#: convention (``index.ts``) is deliberately *not* stripped either: ``src/foo/index.ts`` is +#: addressed as ``src/foo/index.ts`` everywhere else on this surface, so it dots to +#: ``src.foo.index``. +ts_module_dotted = partial(module_dotted, extensions=TS_EXTENSIONS, package_index=None) + + +# ---------------------------------------------------------------------------------------------- +# The dataflow surface's shared vocabulary (leg 2.5b, Task 2). Each of these is the language-neutral +# ruling from ``cldk.analysis.commons`` bound to TypeScript's relationship prefix and edge models, +# once, here -- so the two backends cannot come to disagree about what a page's order, a slice's +# edge set or a hop's word is. + +#: The canonical order of each per-callable graph, in the two spellings that have to agree: the +#: Python sort key (:func:`~cldk.analysis.commons.graphs.edge_sort_key`) and the Cypher +#: expressions. ``coalesce`` is ``or ""`` / ``or []``: an optional field's ``None`` raises in a +#: Python sort key and silently drops the row in Cypher. ``len(exprs)`` is also the order's arity, +#: which is how a cursor minted by one accessor is refused by another (3, 2 and 4). +CFG_ORDER = EdgeOrder(edge_sort_key("cfg"), ("src", "dst", "coalesce(kind,'')")) +CDG_ORDER = EdgeOrder(edge_sort_key("cdg"), ("src", "dst")) +DDG_ORDER = EdgeOrder(edge_sort_key("ddg"), ("src", "dst", "coalesce(var,'')", "coalesce(prov,[])")) + +#: The five relationship types a slice follows, spelled with TypeScript's ``TS_`` prefix, and the +#: Cypher disjunction of them. ``TS_CFG_NEXT`` is deliberately absent: control *flow* says what runs +#: next, while a slice is about what a value or a decision depends on. +SDG_RELS = sdg_rels("TS") +SDG_REL_PATTERN = sdg_rel_pattern("TS") + +#: The caller's word for each relationship a path hop can be justified by (E6). Both backends +#: translate through this one table, so a hop cannot be labelled ``data`` over Neo4j and ``ddg`` +#: locally. +VIA = via_table("TS") + + +def ts_body_node_kind(kind: str, of: "str | None") -> Tuple[str, "str | None"]: + """One body node's ``(kind, name)`` in the caller's vocabulary — TypeScript's own translation. + + :func:`~cldk.analysis.commons.resolve.body_node_kind` is the Python twin and is deliberately + **not** reused: cants' ``of`` grammar shares no token with codeanalyzer-python's ``var`` + grammar, so routing TypeScript through it would put three different internal spellings into + fields E6 reserves for the caller's vocabulary. Measured on superset-frontend (1.3.0): + + * a ``formal_in``'s ``of`` is the parameter's own source text on all 10,465 of them, with none + of Python's ``":mod::name"`` / ``":name"`` markers — so there is nothing to + translate and the ``kind`` is always ``parameter``; + * a ``formal_out``/``actual_out``'s ``of`` is the literal ``"$ret"`` where Python writes + ``""`` — a marker, not a name, so it becomes ``name=None``; + * an ``actual_in``'s ``of`` is ``"arg0"``, ``"arg1"``, … — a *position*, where Python names the + parameter the argument binds to. Reporting it would put an ordinal in a return field (E7), so + it too becomes ``name=None``; the argument's identity is recoverable from ``ref``, and + :meth:`TSAnalysisBackend.flows_to_argument` addresses arguments by the callee's parameter + name rather than by position for exactly this reason. + + Every other kind (``statement``, ``call``, ``entry``, ``exit``, ``config_access``) is already + English and passes through with no name, as it does in Python. + """ + if kind == "formal_in": + return "parameter", of + if kind == "actual_in": + return "argument", None + if kind in ("formal_out", "actual_out"): + return "return", None + return kind, None + + +class TSAnalysisBackend(AnalysisBackend[TSApplication, TSModule, TSType, TSCallable, TSField, str]): """Abstract base every TypeScript analysis backend implements. A backend owns *all* indexing and query logic for a TypeScript application; the :class:`TypeScriptAnalysis` façade is a one-line-delegation shim over it. Implementations must return the canonical ``cldk.models.typescript`` pydantic objects (or the documented NetworkX / dict / list shapes) so the two backends are behaviorally interchangeable. - """ - # -----[ application / whole-program ]----- - @abstractmethod - def get_application(self) -> TSApplication: - """The whole application view (symbol table + call graph + external symbols).""" + Inherited abstract (see :class:`~cldk.analysis.commons.backend.AnalysisBackend`): + ``get_application_view``, ``get_symbol_table``, ``get_call_graph``, ``get_all_classes``, + ``get_class``, ``get_all_methods_in_class``, ``get_method``, ``get_all_fields``, + ``get_method_parameters``, ``get_artifacts``, ``get_dependencies``, ``get_config_keys``, + ``get_config_uses``, ``get_unresolved_config_reads``. + """ - @abstractmethod - def get_symbol_table(self) -> Dict[str, TSModule]: - """The per-file symbol table, keyed by module file path.""" + P: ClassVar[str] = "TS" + N: ClassVar[str] = "TS" + # -----[ application / whole-program ]----- @abstractmethod def get_modules(self) -> List[TSModule]: """All modules (compilation units).""" @abstractmethod def get_external_symbols(self) -> Dict[str, TSExternalSymbol]: - """Phantom (external) call targets — imported/required library members.""" + """Phantom (external) call targets — imported/required library members and builtins — + keyed ``"."``, the key the call graph uses for them; the wire's ``can://`` + id is on the value.""" @abstractmethod def get_synthesized_callables(self) -> Dict[str, TSSynthesizedCallable]: - """Anonymous-callback endpoints the symbol table never names (Jelly-resolved). Keyed by the - synthesized signature that ``call_graph`` edges reference. Empty for the ``tsc`` resolver.""" + """The application's anonymous callables, each value carrying the ``can://`` tree id of + the callable it stands for. Empty below level 2. + + **The key is backend-dependent**, and each backend's own docstring says which it uses: a + backend reading ``analysis.json`` passes the analyzer's compatibility index through as + emitted, so the key is the *older* anonymous id and the value's ``id`` is the tree id that + replaced it (key != ``id``); a backend reading the Neo4j projection has the tree nodes and + not the index, so it keys by the node's own id (key == ``id``). Do not key a cross-backend + lookup on this map -- ask for the value's ``id``.""" @abstractmethod def get_typescript_file(self, qualified_name: str) -> str | None: @@ -102,10 +214,6 @@ def get_typescript_module(self, file_path: str) -> TSModule | None: """The module for a file path.""" # -----[ call graph ]----- - @abstractmethod - def get_call_graph(self) -> nx.DiGraph: - """NetworkX DiGraph of callable signatures (and phantom external symbols) + call edges.""" - @abstractmethod def get_call_graph_json(self) -> str: """The application serialized as JSON.""" @@ -129,7 +237,8 @@ def get_class_hierarchy(self) -> nx.DiGraph: # -----[ call sites ]----- @abstractmethod def get_call_sites(self, qualified_callable_name: str) -> List[TSCallsite]: - """The rich, syntactic call sites inside a callable.""" + """The syntactic call sites inside a callable — its ``body`` nodes of ``kind == "call"``, + with the resolved callee mapped to its signature.""" @abstractmethod def get_calling_lines(self, target_signature: str) -> List[int]: @@ -139,15 +248,7 @@ def get_calling_lines(self, target_signature: str) -> List[int]: def get_call_targets(self, source_signature: str) -> Set[str]: """The call targets invoked from a callable, derived from its call sites.""" - # -----[ classes / interfaces / enums / type-aliases ]----- - @abstractmethod - def get_all_classes(self) -> Dict[str, TSClass]: - """Every class, keyed by signature.""" - - @abstractmethod - def get_class(self, qualified_class_name: str) -> TSClass | None: - """A single class by signature.""" - + # -----[ interfaces / enums / type-aliases ]----- @abstractmethod def get_all_interfaces(self) -> Dict[str, TSInterface]: """Every interface, keyed by signature.""" @@ -166,7 +267,9 @@ def get_all_type_aliases(self) -> Dict[str, TSTypeAlias]: @abstractmethod def get_all_nested_classes(self, qualified_class_name: str) -> List[TSClass]: - """The classes declared inside a class.""" + """The classes declared inside a class -- on schema v2 always ``[]``, on every backend: a + class holds only ``callables`` and ``fields``, so no class nests a type. A class declared + inside a *callable* survives as ``TSCallable.inner_classes``. Kept for the 1.x surface.""" @abstractmethod def get_all_sub_classes(self, qualified_class_name: str) -> Dict[str, TSClass]: @@ -185,21 +288,6 @@ def get_implemented_interfaces(self, qualified_class_name: str) -> List[str]: def get_all_methods_in_application(self) -> Dict[str, Dict[str, TSCallable]]: """All methods grouped by their owning class/interface signature.""" - @abstractmethod - def get_all_methods_in_class(self, qualified_class_name: str) -> Dict[str, TSCallable]: - """The methods of a class/interface, keyed by short name.""" - - @abstractmethod - def get_method(self, qualified_class_name: str, qualified_method_name: str) -> TSCallable | None: - """A single method of a class/interface, or a module/namespace-level function. - ``qualified_class_name`` accepts either a class/interface signature (resolving to that - type's methods) or a module/namespace scope, in which case module-level functions are - resolved as a fallback; returns ``None`` if nothing resolves.""" - - @abstractmethod - def get_method_parameters(self, qualified_class_name: str, qualified_method_name: str) -> List[str]: - """The parameter names of a method.""" - @abstractmethod def get_all_constructors(self, qualified_class_name: str) -> Dict[str, TSCallable]: """The constructors of a class.""" @@ -208,10 +296,6 @@ def get_all_constructors(self, qualified_class_name: str) -> Dict[str, TSCallabl def get_all_functions(self) -> Dict[str, TSCallable]: """Top-level (module/namespace) functions, keyed by signature.""" - @abstractmethod - def get_all_fields(self, qualified_class_name: str) -> List[TSClassAttribute]: - """The attributes/fields of a class.""" - @abstractmethod def get_interface_properties(self, qualified_interface_name: str) -> List[TSClassAttribute]: """The properties of an interface.""" @@ -263,9 +347,9 @@ def get_callables_overview(self) -> List[TSCallableOverview]: @abstractmethod def get_method_bodies(self, signatures: List[str]) -> Dict[str, str]: """Source bodies for the given callable signatures, keyed by signature. Signatures with no - matching callable are omitted, as are callables whose ``code`` is ``None`` (e.g. implicit - constructors the analyzer synthesizes with no source text) — every returned value is a - real ``str``.""" + matching callable are omitted, as are callables with no source text (an implicit + constructor the analyzer synthesizes has an empty span, so its ``code`` is ``""``; 1.x + carried ``None``) — every returned value is a real, non-empty ``str``.""" @abstractmethod def get_decorated_callables(self, markers: List[str]) -> List[TSCallableOverview]: @@ -277,3 +361,595 @@ def get_callsites_for(self, signatures: List[str]) -> Dict[str, List[TSCallsite] """Call sites of the given callable signatures, keyed by owning signature. Each existing signature gets an entry (an empty list if it has no call sites); signatures with no matching callable are omitted.""" + + # -----[ entrypoints and the config readers (leg 2.5b, Task 3) ]----- + # Declared here rather than on the generic cross-language ABC for the same reason their Python + # twins are declared on ``PythonAnalysisBackend``: the return types are this language's own + # projections, and each analyzer spells the entrypoint mark differently. + @abstractmethod + def get_entrypoints(self) -> List[TSCallableOverview]: + """Overviews of every *callable* codeanalyzer-typescript marked as an entrypoint + (``TSCallable.is_entrypoint``) — a CLI command, route handler, or other externally-invoked + callable its entrypoint-detection pass already found. + + An empty list means the pass found no entrypoint *callables* — the ordinary "no + entrypoints in this project" case, never a stand-in for the mark not existing (1.3.0 + carries ``is_entrypoint`` as a real boolean on every callable, and a graph emitted below + 1.3.0 is refused at attach, so it is never ambiguous at the property level). + + Two things this accessor alone cannot tell you, each answered by a sibling rather than by + widening its frozen ``List[TSCallableOverview]`` return: + + * **Class-level entrypoints.** ``TSClass`` carries its own ``is_entrypoint``: a class the + rulesets matched with no individually-marked method. This walk is callables-only; use + :meth:`get_entrypoint_classes`. + * **Whether the pass itself had gaps.** Detection under-approximates by design, so silence + is its failure mode — an empty result here cannot distinguish "ran clean, found none" + from "had gaps". Use :meth:`get_entrypoint_coverage`.""" + + @abstractmethod + def get_entrypoint_classes(self) -> List[TSClassOverview]: + """Overviews of every *class* the analyzer marked as an entrypoint in its own right + (``TSClass.is_entrypoint``) — the class-level sibling of :meth:`get_entrypoints`, which + walks callables only. Same empty-vs-absent guarantee as :meth:`get_entrypoints`. + + **Classes only.** The 1.3.0 schema declares ``is_entrypoint`` on all five type kinds, but + the Neo4j projection stamps it onto ``:TSCallable`` and ``:TSClass`` nodes only (measured + on the reference graph: no other label carries the property at all), so widening this past + classes would make the two backends answer differently. That is a gap in the projection, + recorded rather than papered over — see :class:`~cldk.models.typescript.TSClassOverview`.""" + + @abstractmethod + def get_entrypoint_coverage(self) -> EntrypointCoverage: + """Coverage and failure record for the entrypoint-detection pass + (``TSApplication.entrypoint_report``), so a caller can tell "the pass ran clean and found + nothing" apart from "the pass had gaps" — a distinction :meth:`get_entrypoints`'s empty + list alone cannot make. + + See :class:`~cldk.analysis.commons.results.EntrypointCoverage` for the field-by-field + contract. Both TypeScript backends can normally supply it in full: the local backend + passes ``entrypoint_report`` through, and the Neo4j backend parses the + ``entrypoint_report_json`` string property 1.3.0 stamps on the ``:Application`` anchor + (alongside the derived ``entrypoint_frameworks``). A source that carries neither answers + with a ``diagnostics``-only result rather than fabricating empty-but-clean-looking + coverage fields — the same "say so honestly" precedent as ``LocateResult``'s + ``module_source_unavailable``.""" + + @abstractmethod + def get_config_readers(self, key: str) -> List[TSCallableOverview]: + """Overviews of every callable that reads configuration key ``key``, resolved from + :meth:`~cldk.analysis.commons.backend.AnalysisBackend.get_config_uses`'s edges. + + That generic accessor hands back ``PyConfigUseEdge.src`` as an opaque body-node id; + resolving it to "which callable" is a containment walk (``TS_HAS_BODY_NODE``, or the + callable's own ``body`` map in process), never a split on ``@`` — an anonymous callable's + own id contains one. Empty means no callable reads this key, which is not the same as "a + read exists but never resolved to a key": see + :meth:`~cldk.analysis.commons.backend.AnalysisBackend.get_unresolved_config_reads`.""" + + # ===================================================================================== + # The addressing surface (leg 2.5b, TS-2). Python's semantics are the contract: every + # signature below is `cldk/analysis/python/backend.py`'s, keyword-for-keyword. + # + # A caller names things the way it already thinks of them; the SDK resolves. Nothing here takes + # or returns a ``can://`` URI outside ``ref`` / ``node_id`` (E6), and nothing takes an ordinal + # (E7). The resolution *policy* is not implemented per backend -- both route through + # :mod:`cldk.analysis.commons.resolve`, so they cannot drift on what "ambiguous" means. What a + # backend implements is only how it produces the candidates. + # ===================================================================================== + @abstractmethod + def locate(self, path: str, line: int) -> LocateResult: + """Resolve a source position to its enclosing callable, with the source in hand. + + Four outcomes, kept distinguishable rather than collapsed into an ambiguous empty: inside a + callable (``callable`` set, and ``body`` set too when a body node is that precise); at + module scope (a real position with no enclosing callable -- a ``module_scope`` diagnostic); + in the gap between two callables (also module scope, and never silently snapped to the + nearest callable); or in a file the analysis has no module for (``file_not_in_graph``). + + Args: + path: The file path. Normalised against the backend's module keys + (:func:`~cldk.analysis.commons.keys.resolve_module_key`), so a ``./``-prefixed or + absolute path resolves rather than reading back as ``file_not_in_graph``. + line: The 1-based line number. + """ + + @abstractmethod + def locate_many(self, positions: Sequence[Tuple[str, int]]) -> List[LocateResult]: + """Resolve many positions in one round trip, in input order. + + The bulk form, not an optimisation over :meth:`locate`: a scanner hands over a whole alert + set at once, and round trips cost latency for a person and context for an agent. + """ + + @abstractmethod + def resolve_callable(self, name: str, *, in_class: str | None = None, in_module: str | None = None) -> SliceNode: + """Resolve a callable name to the callable it names. + + The **candidate domain is every callable in the analysed application** -- exactly the set + :meth:`get_callables_overview` reports: module- and namespace-level functions, class and + interface methods, and callables nested inside either. Both backends resolve against that + same domain; a shared *predicate* over different *sets* is not parity. + + ``name`` is matched whole or as a dotted suffix on segment boundaries (``"show"`` names any + ``….show``; ``"UserController.show"`` narrows), with an exact match winning outright. + ``in_class`` / ``in_module`` disambiguate rather than scope -- a callable is the unit of + address -- and are matched the same segment-wise way against the owning class's or + interface's signature and against the module. ``in_module`` takes a module-key suffix + (``"src/controllers.ts"``, ``"controllers.ts"``) **or** the dotted form + (``"src.controllers"``, ``"controllers"``) that TypeScript signatures are spelled in; the + two never cross, because a ``/`` spelling never matches a dotted candidate. + + **An anonymous callable is addressed by its signature, never by its name.** cants gives + every one of them the name ``"(anonymous)"`` (7,044 on superset-frontend) and a *unique* + signature ending in ````, so ``resolve_callable("")`` is the + address. The display name is not one: this resolver matches *signatures*, and no signature + carries ``"(anonymous)"``, so that spelling misses outright rather than becoming a + 7,044-way ambiguity -- an honest "no such callable", not a list nobody could choose from. + + **A declaration-merged name resolves to the callable facet or to nothing, never to the + wrong facet.** ``const X = () => …`` beside ``interface X`` shares one id, and the Neo4j + emitter collapses the two onto a single node carrying both labels and one ``kind`` (three + such nodes on superset-frontend, two of them callables). This accessor's domain is the + ``kind``, not the label: a node whose ``kind`` is a callable kind is a candidate here and a + node whose ``kind`` names a type facet is not, so a merged node can never come back + described as something it is not. + + Returns: + A :class:`~cldk.analysis.commons.results.SliceNode` with ``kind="callable"``, the + callable's dotted signature in ``callable``, and its opaque graph id in ``ref``. That + ``ref`` round-trips through :meth:`get_source` on either backend -- the one sanctioned + use of an opaque id. + + Raises: + AmbiguousName: More than one callable matched, listing every match and nothing else. + The resolver never picks: a guess presented as an answer is the confident wrong + answer this layer exists to prevent. + SelectorNotInGraph: Nothing matched, naming the selector as the caller spelled it -- + or, when the name matched and a keyword excluded every match, naming that keyword. + No near-miss suggestions: E8 puts typo-tolerant matching out of scope in the error + path as much as in the resolver. + """ + + @abstractmethod + def resolve_value(self, name: str, *, within: str) -> SliceNode: + """Resolve a value name inside a callable to the position that carries it. + + A value name is scoped by its callable, so ``within`` is required and is itself resolved by + :meth:`resolve_callable` -- ``within="UserController.show"`` is enough. + + The **candidate domain is the resolved callable's ``formal_in`` vertices**: every named + value that *enters* it, which is what a backward slice seeds from. In TypeScript those are + parameters and nothing else, so the answer's ``kind`` is always ``"parameter"`` and + ``defined_in`` is always ``None`` -- unlike Python, where 84% of entering values are + captured module globals and the analyzer marks them with a ``":mod::name"`` + grammar. cants emits no such grammar (measured on superset-frontend: of 10,465 ``formal_in`` + vertices none carries a marker prefix), so nothing is translated and the name a caller + writes is the name the analyzer wrote. + + The name is the parameter's source text, which for a destructured parameter is a pattern + (``"{ theme }"``, 948 of them on superset-frontend) rather than an identifier. That is + reported as it is: inventing an identifier for a pattern would be a fabricated address. + + The domain is deliberately *not* every body node carrying a value: ``of`` is non-null only + on the four parameter-passing kinds, and the same name also appears on the callable's + ``formal_out`` vertex and at each call site's actuals, so collapsing them would make every + parameter ambiguous with its own exit value. A local variable has no address here at all; + :meth:`locate` is what addresses those positions. + + Note: + The returned ``ref`` does **not** round-trip through :meth:`get_source` on either + backend -- a ``formal_in`` vertex is a dataflow position with no span, so there is no + text to return for one. Only a :meth:`resolve_callable` ``ref`` round-trips. + + Raises: + AmbiguousName: ``within`` named more than one callable, or more than one value matched. + SelectorNotInGraph: No such callable, or no such value in it. + """ + + @abstractmethod + def get_source(self, node_id: str) -> str: + """Source text for one node, named by ``node_id``. + + Generalises :meth:`get_method_bodies` below callable granularity: ``node_id`` is a + callable's signature, a callable's ``can://`` id, or the opaque body-node id + :attr:`~cldk.analysis.commons.results.LocateResult.node_id` hands back -- so a caller can + re-fetch the precise statement or call site :meth:`locate` found, not just the callable + enclosing it. Round-tripped, never composed by the caller: a TypeScript body-node id is + ``@`` and a callable id may itself contain an ``@`` + (``…/``), so it cannot be taken apart by splitting on one. + + Raises: + KeyError: Nothing carries that id or signature, or it carries no recoverable source + (a ``formal_in`` vertex, or the implicit constructor cants synthesizes with an + empty span -- both backends refuse rather than returning ``""`` as if it were a + body). + NotImplementedError: (Neo4j backend only) ``node_id`` names a body node. The graph + projects per-callable text (``:TSCallable.code``) but nothing below it -- + ``:TSBodyNode`` carries a line span and no text, and ``:TSModule`` carries no + source to slice one out of. Only the local backend, which holds the module text and + the analyzer's offsets, can answer for a statement or call site. + """ + + @property + @abstractmethod + def has_resolution_edges(self) -> bool: + """Whether this backend can resolve a call site's ``callee_signature`` at all right now. + + :meth:`get_callsites_for`'s per-site ``callee_signature`` is ``None`` both for "genuinely + unresolved" and for "this view was built below the analysis level where callee resolution + runs" -- :class:`~cldk.models.typescript.TSCallsite` has no field to carry the distinction. + This is the disambiguator: ``False`` means every ``None`` from :meth:`get_callsites_for` is + explained by that, not by individual call sites failing to resolve. + + Unlike Python's, the local TypeScript backend is **not** unconditionally ``True``: cants + resolves callees in its own level-2 pass and writes ``callee: null`` on every call node + below it (there is no Jedi-style resolver running regardless of level), so the honest + answer is whether the analysis actually reached that level. + """ + + # ===================================================================================== + # The dataflow surface (leg 2.5b, Task 2). Python's semantics are the contract: every signature + # below is `cldk/analysis/python/backend.py`'s, keyword-for-keyword, default-for-default. + # + # BOUNDS ARE ASYMMETRIC ON PURPOSE (E5). The three *slices* default ``depth`` to + # :data:`~cldk.analysis.commons.bounds.DEFAULT_DEPTH` and cap ``max_nodes``: a bounded traversal + # is a *complete* answer to a narrower question, and ``total`` says how much was left out. The + # two *predicates* and the two *path* queries default to ``depth=None`` -- unbounded -- because + # a hop budget on a boolean or a path list is not a smaller answer but a **wrong** one: "no + # flow" and "no flow within five hops" collapse into the same ``False`` / ``[]`` with nothing in + # the result to tell them apart. + # + # ONE COMPLETENESS PROTOCOL. Truncation is reported by ``complete`` on ``EdgePage`` / ``Slice`` + # / ``FlowPaths``, never by silently returning less. + # + # ANALYSIS LEVEL. cfg/cdg/ddg exist from analyzer level 3 and the interprocedural overlays from + # level 4. A backend attached to a shallower analysis MUST raise rather than return an empty + # page: an empty there is indistinguishable from a callable that genuinely has no dependence + # (D7). ``--emit neo4j`` is always full depth, so only the local backend can be below the line. + # + # THE CALL GRAPH'S VERTICES ARE TYPESCRIPT'S (TS-11). cants makes a *module* the caller of its + # own top-level code -- 1,464 of superset-frontend's 17,712 ``TS_CALLS`` edges -- so + # ``callers_of``, ``backward_cone`` and the call paths report a module vertex with + # ``kind="module"``, a value outside :attr:`~cldk.analysis.commons.results.SliceNode.KINDS`' + # Python-derived list. Dropping them would answer "nothing reaches this" where a module does. + # A module is only ever a *source* (verified: 0 incoming ``TS_CALLS`` on the reference graph), + # so it can never be the interior of a path; an external is only ever a *target*, so it cannot + # either. + # ===================================================================================== + @abstractmethod + def get_cfg(self, callable: str, *, in_class: str | None = None, page_size: int = DEFAULT_PAGE_SIZE, cursor: str | None = None) -> EdgePage[TSCfgEdge]: + """One page of the control flow edges within one callable. + + Args: + callable: The callable's name, resolved by :meth:`resolve_callable` — so an ambiguous + name raises listing candidates rather than being guessed at. + in_class: Disambiguate by owning class, as in :meth:`resolve_callable`. + page_size: Most edges to return. See + :data:`~cldk.analysis.commons.bounds.DEFAULT_PAGE_SIZE`. + cursor: ``next_cursor`` from a previous page; ``None`` starts at the beginning. + + Returns: + An :class:`~cldk.analysis.commons.results.EdgePage` of + :class:`~cldk.models.typescript.TSCfgEdge`, each carrying the analyzer's ``kind`` + (``fallthrough``, ``true``, ``false``, ``switch_case``, ``loop_back``, ``exception``, + ``return``, ``break``, ``continue``, ``yield``, ``await_resume``) — a conditional's two + successors stay two edges, discriminated by ``kind``, which is also why ``kind`` is part + of the order (:data:`CFG_ORDER`). Endpoints are the body nodes' own ``can://`` ids, the + spelling :meth:`get_source` accepts. + + Raises: + AmbiguousName: ``callable`` named more than one callable. + SelectorNotInGraph: Nothing matched. + ValueError: ``page_size`` below 1, or ``cursor`` not from a previous page of this + accessor and this callable. + CodeanalyzerUsageException: (local backend) built below + ``analysis_level="program_dependency_graph"``. + """ + + @abstractmethod + def get_cdg(self, callable: str, *, in_class: str | None = None, page_size: int = DEFAULT_PAGE_SIZE, cursor: str | None = None) -> EdgePage[TSCdgEdge]: + """One page of the control dependence edges within one callable. + + ``src`` is the branching node a ``dst`` is control dependent on — post-dominance over the + CFG :meth:`get_cfg` returns, computed by the analyzer, not re-derived here. Arguments, + bounds and failures are :meth:`get_cfg`'s; the order is :data:`CDG_ORDER`. + """ + + @abstractmethod + def get_ddg(self, callable: str, *, in_class: str | None = None, page_size: int = DEFAULT_PAGE_SIZE, cursor: str | None = None) -> EdgePage[TSDdgEdge]: + """One page of the data dependence edges within one callable. + + Each edge carries the variable it flows (``var``) and its evidence (``prov``). + + **TypeScript's DDG has exactly one provenance tier.** Every one of the 119,384 ``TS_DDG`` + edges on the reference application carries ``prov == ["reaching-defs"]`` — cants emits no + ``ssa`` and no ``points-to`` tier, so Python's three-way certainty ranking + (:func:`~cldk.analysis.commons.results.prov_rank`) collapses to a single value here. The + field and the ranking helper are kept, because the analyzer reserves further tiers and a + caller comparing two hops' certainty must keep working when it emits them; nothing here + invents one. + + Arguments, bounds and failures are :meth:`get_cfg`'s; the order is :data:`DDG_ORDER`, which + includes ``var`` and ``prov`` because the same statement pair legitimately appears more than + once when it carries several variables, and collapsing those would drop dependences. + """ + + # -----[ slicing and reachability ]----- + @abstractmethod + def slice_backward(self, src: str, *, within: str, depth: int | None = DEFAULT_DEPTH, max_nodes: int = DEFAULT_MAX_NODES) -> Slice: + """Everything the value ``src`` depends on: reverse reachability over the SDG. + + The edge set is :data:`SDG_RELS` — data and control dependence within a callable, the two + parameter-passing relationships across a call, and the callee summaries at a call site. All + five point *with* the flow, so a backward slice follows them reversed. + + ``within`` is **required**: a value name is scoped by its callable and + :meth:`resolve_value` cannot resolve one without it, so a ``None`` default would be a + signature that raises on its own default. + + Args: + src: The value's name, resolved by :meth:`resolve_value` — in TypeScript, a parameter. + within: The callable to look inside, resolved as in :meth:`resolve_callable`. + depth: Most hops from the seed. Defaults to + :data:`~cldk.analysis.commons.bounds.DEFAULT_DEPTH`; ``None`` for the whole cone. + max_nodes: Most nodes in the result. A cap that fires is reported by + :attr:`~cldk.analysis.commons.results.Slice.truncated` and quantified by + :attr:`~cldk.analysis.commons.results.Slice.total`; it is never silent. + + Returns: + A :class:`~cldk.analysis.commons.results.Slice` containing the seed, ordered by node + id, with ``source`` unhydrated on every node (:meth:`describe` fills it in). + + Raises: + AmbiguousName: ``within`` named more than one callable, or ``src`` more than one value. + SelectorNotInGraph: No such callable, or no such value in it. + ValueError: ``depth`` that is not a positive ``int``, or ``max_nodes`` below 1. + CodeanalyzerUsageException: (local backend) built below + ``analysis_level="program_dependency_graph"``. + """ + + @abstractmethod + def slice_forward(self, src: str, *, within: str, depth: int | None = DEFAULT_DEPTH, max_nodes: int = DEFAULT_MAX_NODES) -> Slice: + """Everything the value ``src`` can affect: forward reachability over the same edges. + + The usually-interesting direction for a value entering a callable: nothing flows *into* a + parameter except from its callers, so ``slice_backward`` from one is often the seed alone, + while this follows it through the body and out through every call it feeds. Arguments, + bounds and failures are :meth:`slice_backward`'s. + """ + + @abstractmethod + def reaches(self, src: str, dst: str, *, depth: int | None = None) -> bool: + """Is there a call path from ``src`` to ``dst``? + + A **call-graph** question, over ``TS_CALLS`` — "can control get from here to there at all", + the cheap check a caller makes before asking for the paths themselves. Both names go through + :meth:`resolve_callable`, so an ambiguous one raises listing candidates rather than being + guessed at, and both endpoints are therefore callables. + + Returns ``bool`` and nothing else: it is deliberately not a degenerate ``Slice``, because + "is there a path" and "what is on it" are different questions with different costs. + + **``depth`` defaults to ``None`` here, unlike the three slices.** A default that bounds a + *slice* trades size for a complete answer to a narrower question; a default that bounds a + *boolean* would turn "there is no path" and "there is no path within 5 hops" into the same + ``False``. + + Raises: + AmbiguousName: Either name matched more than one callable. + SelectorNotInGraph: Either name matched none. + ValueError: ``depth`` that is not a positive ``int``. + CodeanalyzerExecutionException: (Neo4j backend) the attached server predates the + quantified path pattern this compiles to (Neo4j 5.9). + """ + + @abstractmethod + def backward_cone(self, sinks: Sequence[str], *, depth: int | None = DEFAULT_DEPTH, max_nodes: int = DEFAULT_MAX_NODES) -> Slice: + """Every vertex that can reach any of ``sinks`` — "what could get here". + + A **call-graph** cone, so its nodes are call-graph vertices, not body nodes: callables + (``kind="callable"``) and the modules whose top-level code calls them (``kind="module"``, + TS-11). The sinks themselves are in the result, and in + :attr:`~cldk.analysis.commons.results.Slice.roots`. + + Args: + sinks: The callables to walk back from, each resolved by :meth:`resolve_callable`. + depth: Most call hops back. Defaults to + :data:`~cldk.analysis.commons.bounds.DEFAULT_DEPTH`; ``None`` for the whole cone. + max_nodes: Most nodes in the result. + + Raises: + AmbiguousName: A sink name matched more than one callable. + SelectorNotInGraph: A sink name matched none. + TypeError: ``sinks`` is a bare string. + ValueError: ``sinks`` is empty, ``depth`` is not a positive ``int``, or ``max_nodes`` + is below 1. + """ + + @abstractmethod + def callers_of(self, name: str, *, in_class: str | None = None, in_module: str | None = None) -> List[SliceNode]: + """Who calls this — one hop back over ``TS_CALLS``, addressed by name. + + The name-based sibling of :meth:`get_all_callers`, which takes a class signature plus a + method name and returns raw dicts. That one is a frozen leg-1 signature and is not touched; + this one takes a name the caller already has and returns + :class:`~cldk.analysis.commons.results.SliceNode` objects. + + **A module is a legitimate caller** (``kind="module"``): cants emits a module as the caller + of its own top-level code, and dropping those would report "nothing calls it" for every + function a module invokes at import time. An external ghost is never a caller — it was never + analysed, so it has no body to call from, and the reference graph has no ``TS_CALLS`` + originating at one. + + An empty list is unambiguous: a name that matches nothing raises, so ``[]`` means "nothing + calls it". + + Raises: + AmbiguousName: ``name`` matched more than one callable. + SelectorNotInGraph: Nothing matched. + """ + + @abstractmethod + def callees_of(self, name: str, *, in_class: str | None = None, in_module: str | None = None) -> List[SliceNode]: + """What this calls — one hop forward over ``TS_CALLS``, addressed by name. + + **Externals are included**, with ``kind="external"``: they are 6,537 of the reference + application's 17,712 call edges and they are what a caller tracing a sink is usually looking + for. An external was never analysed, so it has no position: ``file`` is ``""`` and ``line`` + is ``0``, and ``kind`` is what says why rather than leaving two sentinels to be discovered. + Its ``callable`` is the readable dotted name built from the node's own ``module`` and + ``name`` — never its ``can://`` id, which stays in ``ref`` where an opaque handle belongs. + + Raises: + AmbiguousName: ``name`` matched more than one callable. + SelectorNotInGraph: Nothing matched. + """ + + # -----[ paths and flow predicates ]----- + @abstractmethod + def paths_between(self, src: str, dst: str, *, src_within: str, dst_within: str, depth: int | None = None, max_paths: int = DEFAULT_MAX_PATHS) -> FlowPaths: + """How a value reaches another value — the *sequences*, where a slice is the set. + + Each :class:`~cldk.analysis.commons.results.FlowPath` is an ordered list of + :class:`~cldk.analysis.commons.results.PathHop` values, and each hop says what justified it: + the kind of edge (``data``/``control``/``argument``/``return``/``summary``), the variable + the dependence is on, and the provenance the analyzer established it with — which in + TypeScript is always ``["reaching-defs"]`` (see :meth:`get_ddg`). + + **Only shortest paths.** A search that enumerated every walk would not terminate on a real + dependence graph, and the tenth-longest way a value can reach another is not evidence anyone + wants. What comes back is the shortest hop-count, and every path of it up to ``max_paths``. + + **Two scopes, not one, and neither defaults to the other.** A value is addressed by a name + plus the callable it enters, so two values need two callables — and a single scope could + never find the cross-callable path this accessor exists for. + + Args: + src: The value the flow starts at, named as a caller would. + dst: The value it must reach. + src_within: The callable ``src`` enters. Required. + dst_within: The callable ``dst`` enters. Required, and not defaulted. + depth: Most hops a path may take; ``None`` (the default) for no bound. + max_paths: Most paths to return. The result's ``complete`` says whether more existed. + + Raises: + AmbiguousName: ``src``, ``dst`` or either callable name matched more than one thing. + SelectorNotInGraph: One of them matched nothing. + ValueError: ``depth`` is not a positive ``int``, ``max_paths`` is below 1, or ``src`` + and ``dst`` resolve to the same position (see + :func:`~cldk.analysis.commons.bounds.check_distinct_endpoints`). + """ + + @abstractmethod + def call_paths_between(self, src: str, dst: str, *, depth: int | None = None, max_paths: int = DEFAULT_MAX_PATHS) -> FlowPaths: + """How one callable reaches another — the same sequences, over the call graph. + + The evidence-carrying form of :meth:`reaches`: that answers *whether*, this answers *how*. + Every hop is ``via="call"`` with no ``var`` and no ``prov``, because a ``TS_CALLS`` edge + carries neither — a call is a syntactic fact, and saying so explicitly is better than + inventing a provenance for it. + + Takes no ``within``: a callable is addressed by name alone. ``depth`` defaults to ``None`` + as :meth:`reaches`'s does. + + Raises: + AmbiguousName: Either name matched more than one callable. + SelectorNotInGraph: Either matched nothing. + ValueError: ``depth`` is not a positive ``int``, ``max_paths`` is below 1, or ``src`` + and ``dst`` name the same callable. + """ + + @abstractmethod + def flows_to_call(self, src: str, callee: str, *, within: str, depth: int | None = None) -> bool: + """Does this value reach **any** argument of a call to ``callee``? + + The target is the set of ``callee``'s ``formal_in`` vertices — in TypeScript, its + parameters. Those are enterable only through ``TS_PARAM_IN`` from a caller's argument, so + reaching one means the value was passed into a real call, not merely that it sits in the + same program. + + A value that only *control*-dominates a call site without feeding any of its arguments is + deliberately **not** counted: "flows to" is a dataflow claim, and widening it to "was + executed before" would make the answer true almost everywhere. + + **One ``within``, scoping ``src`` only.** :meth:`paths_between` takes two callables because + it takes two *values*; here the second endpoint is ``callee``, a callable addressed by name + alone, so a second scope would have nothing to scope. ``depth`` defaults to ``None``: a bare + ``False`` on a boolean carries no signal that a bound fired. + + Raises: + AmbiguousName: ``src`` or ``callee`` matched more than one thing. + SelectorNotInGraph: Either matched nothing. + ValueError: ``depth`` is not a positive ``int``. + """ + + @abstractmethod + def flows_to_argument(self, src: str, callee: str, arg: str, *, within: str, depth: int | None = None) -> bool: + """Does this value reach the argument ``arg`` of a call to ``callee``? + + A **different question** from :meth:`flows_to_call`, and kept a separate implementation on + purpose: a tainted value routinely reaches a function without reaching the parameter that + matters. + + ``arg`` is resolved to the parameter **by name**, through the same :meth:`resolve_value` the + other accessors use, with ``within=callee`` — nothing here asks the caller to know which + slot a parameter occupies (E7). + + **The implication ``flows_to_argument`` ⟹ ``flows_to_call`` holds by construction**, not by + agreement between two queries: ``resolve_value(arg, within=callee)`` can only ever return + one of ``callee``'s ``formal_in`` vertices, and that set is exactly what + :meth:`flows_to_call` tests reachability of. + + Raises: + AmbiguousName: A name matched more than one thing. + SelectorNotInGraph: A name matched nothing — including ``arg`` naming no parameter of + ``callee``, which is a caller error and not a ``False``. + ValueError: ``depth`` is not a positive ``int``. + """ + + def describe(self, nodes: Sequence[object]) -> List[SliceNode]: + """Fill in :attr:`~cldk.analysis.commons.results.SliceNode.source` for these positions. + + A second call because addressing answers *where* and source answers *what*, and source is + the one field with no size ceiling (E4). Returns the **same** + :class:`~cldk.analysis.commons.results.SliceNode` type, so nothing downstream has to branch + on whether a node has been through here, and accepts anything carrying an address -- slice + nodes and :meth:`locate` results alike (see :func:`as_slice_node`). + + **One round trip regardless of node count.** Implemented here rather than in each backend + precisely so that cannot drift: the whole batch resolves through a single + :meth:`_sources_for` call. + + Afterwards ``source=None`` means exactly one thing: *this position exists and the backend + has no text for it*. It never means "the lookup failed", because a ref naming nothing raises + instead. Which positions have no text differs by backend, honestly: a ``kind="callable"`` + node hydrates on both; a ``formal_in`` vertex hydrates on neither (it has no span in the + analyzer's own model); a statement or call site hydrates only locally, because the graph + carries no text below callable granularity. + + Raises: + KeyError: A ``ref`` names nothing this backend can find -- a ref comes from this SDK, + so one that resolves to nothing means a stale or foreign address, which is worth + stopping on rather than discovering three layers later. The message names the + positions in the caller's vocabulary, never by ``ref`` (E6). + TypeError: An element carries no ``ref`` (see :func:`as_slice_node`). + """ + out = [as_slice_node(n) for n in nodes] + if not out: + return [] + sources = self._sources_for([n.ref for n in out]) + missing = [n for n in out if n.ref not in sources] + if missing: + named = [f"{n.callable} ({n.file}:{n.line})" if n.file else n.callable for n in missing[:5]] + raise KeyError(f"{len(missing)} of {len(out)} positions name nothing in this application: {', '.join(named)}") + return [n.model_copy(update={"source": sources[n.ref]}) for n in out] + + @abstractmethod + def _sources_for(self, refs: Sequence[str]) -> Dict[str, "str | None"]: + """``{ref: source or None}`` for every ref this backend can **find**, in one round trip. + + The seam :meth:`describe` is built on, and the reason its two kinds of "no source" stay + distinguishable: a ref that exists but has no recoverable text maps to ``None``; a ref that + names nothing is *absent from the mapping*, and :meth:`describe` raises on it. + """ diff --git a/cldk/analysis/typescript/codeanalyzer/codeanalyzer.py b/cldk/analysis/typescript/codeanalyzer/codeanalyzer.py index d173ec91..b248f02d 100644 --- a/cldk/analysis/typescript/codeanalyzer/codeanalyzer.py +++ b/cldk/analysis/typescript/codeanalyzer/codeanalyzer.py @@ -16,10 +16,10 @@ """TypeScript Codeanalyzer backend wrapper. -Subprocess wrapper around the ``codeanalyzer-typescript`` binary (built from ``codeanalyzer-ts`` -with ``bun build --compile``). Mirrors the Java ``JCodeanalyzer`` / Python ``PyCodeanalyzer`` -pattern: shell out to the analyzer, read ``analysis.json`` from stdout (or an output dir), -validate it into a ``TSApplication`` pydantic model, **and own all query/indexing logic**. The +Subprocess wrapper around the ``codeanalyzer-typescript`` binary (``cants``). Mirrors the Java +``JCodeanalyzer`` / Python ``PyCodeanalyzer`` pattern: shell out to the analyzer, read the +``analysis.json`` envelope (:class:`TSAnalysis`) from stdout or an output dir, keep its +``application`` as the queried :class:`TSApplication`, **and own all query/indexing logic**. The ``TypeScriptAnalysis`` facade is a thin delegating shell over this backend. """ @@ -30,22 +30,61 @@ import os import shlex import subprocess -from collections import deque +import warnings +from collections import defaultdict +from functools import cached_property, partial from pathlib import Path from subprocess import CompletedProcess -from typing import Dict, Iterator, List, Set, Tuple, Union +from typing import Dict, Iterator, List, Sequence, Set, Tuple, Union import networkx as nx -from cldk.analysis import AnalysisLevel -from cldk.analysis.typescript.backend import TSAnalysisBackend +from cldk.analysis.commons.bounds import ( + DEFAULT_DEPTH, + DEFAULT_MAX_NODES, + DEFAULT_MAX_PATHS, + DEFAULT_PAGE_SIZE, + check_depth, + check_distinct_endpoints, + check_max_nodes, + check_max_paths, + check_page_size, + edge_page, +) +from cldk.analysis.commons.graphs import cone_sinks, flow_path, shortest_walks, slice_resolved +from cldk.analysis.commons.keys import body_key_column, resolve_module_key +from cldk.analysis.commons.levels import ANALYZER_LEVELS, LEVEL_NAMES, analyzer_level +from cldk.analysis.commons.resolve import CallableCandidate, resolve_callable_signature, resolve_value_name, resolve_within +from cldk.analysis.commons.results import ( + BodyRef, + CallableRef, + Diagnostic, + EdgePage, + EntrypointCoverage, + FlowPaths, + LocateResult, + ModuleRef, + Slice, + SliceNode, + Span, + TypeRef, +) +from cldk.analysis.typescript.backend import CDG_ORDER, CFG_ORDER, DDG_ORDER, VIA, TSAnalysisBackend, ts_body_node_kind, ts_module_dotted +from cldk.models.python import PyArtifact, PyConfigKey, PyConfigRead, PyConfigUseEdge, PyDependency from cldk.models.typescript import ( + TSAnalysis, TSApplication, + TSBodyNode, TSCallable, + TSCdgEdge, + TSCfgEdge, + TSDdgEdge, TSCallableOverview, TSCallsite, TSClass, TSClassAttribute, + TSClassOverview, + TSConfigKey, TSDecorator, TSEnum, TSEnumMember, @@ -55,37 +94,50 @@ TSInterface, TSModule, TSNamespace, + TSSpan, TSSynthesizedCallable, TSTypeAlias, TSVariableDeclaration, ) -from cldk.utils.exceptions.exceptions import CodeanalyzerExecutionException +from cldk.analysis import AnalysisLevel +from cldk.utils.exceptions.exceptions import CodeanalyzerExecutionException, CodeanalyzerUsageException logger = logging.getLogger(__name__) +#: The codeanalyzer-typescript release that removed ``--tsc-only`` (the resolver is no longer a +#: choice; 1.x's ``tsc`` and ``defuse`` provenances are both emitted and tagged per edge). +_TSC_ONLY_REMOVED_IN = "1.0.0" + +#: The analyzer level at which cants resolves a call node's ``callee`` (its level-2 pass). Below +#: it every ``call`` body node carries ``callee: null`` -- what ``has_resolution_edges`` reports. +_CALLEE_RESOLUTION_LEVEL = 2 + class TSCodeanalyzer(TSAnalysisBackend): """Build and query the application view of a TypeScript project by invoking the codeanalyzer-typescript binary as a subprocess. This backend owns all indexing and query logic (symbol lookups, the NetworkX call graph, - class hierarchy, call sites, entrypoints, decorators, ...). The :class:`TypeScriptAnalysis` - facade simply delegates to it, mirroring how :class:`PythonAnalysis` delegates to - :class:`PyCodeanalyzer`. + class hierarchy, call sites, decorators, the artifact layer, ...). The + :class:`TypeScriptAnalysis` facade simply delegates to it, mirroring how + :class:`PythonAnalysis` delegates to :class:`PyCodeanalyzer`. Args: project_dir: Path to the root of the TypeScript project. - analysis_backend_path: Directory containing the ``codeanalyzer-typescript`` binary. If - None, falls back to ``$CODEANALYZER_TS_BIN`` then the ``codeanalyzer-typescript`` - PyPI package (``pip install codeanalyzer-typescript``). analysis_json_path: Directory to persist ``analysis.json``. If None, output is read from the subprocess stdout pipe. - analysis_level: ``AnalysisLevel.symbol_table`` (1) or ``AnalysisLevel.call_graph`` (2). - eager_analysis: If True, re-run the analyzer even if a cached ``analysis.json`` exists. + analysis_level: Any :class:`~cldk.analysis.AnalysisLevel` (or its name); sent to the + analyzer as ``-a 1..4`` — the backend requests what the caller asked for. + eager_analysis: If True, re-run the analyzer even if a cached ``analysis.json`` exists, and + tell the analyzer to rebuild its own cache (``--eager``). target_files: Restrict analysis to these files (incremental). - tsc_only: If True, restrict the analyzer to the tsc resolver call graph by passing - ``--tsc-only`` (codeanalyzer-typescript >= 0.4.2). Defaults to False (let the binary - choose its default). Replaces reliance on the obsolete ``--call-graph-provider both``. + tsc_only: Deprecated no-op. The flag was removed from codeanalyzer-typescript at 1.0.0; + passing ``True`` emits a :class:`DeprecationWarning` and changes nothing. + + Attributes: + analysis: The whole ``analysis.json`` envelope — ``max_level``, ``k_limit``, + ``analyzer.version`` — for callers that need to know what generation produced the view. + application: ``analysis.application``, the queried view. """ def __init__( @@ -102,10 +154,14 @@ def __init__( self.analysis_level = analysis_level self.eager_analysis = eager_analysis self.target_files = target_files - self.tsc_only = tsc_only - self.application: TSApplication = self._init_codeanalyzer( - analysis_level=1 if analysis_level == AnalysisLevel.symbol_table else 2 - ) + if tsc_only: + warnings.warn( + f"tsc_only is a no-op: codeanalyzer-typescript removed --tsc-only in {_TSC_ONLY_REMOVED_IN}; " "every call edge now carries its resolver in `prov` instead.", + DeprecationWarning, + stacklevel=4, # warn at the CLDK.typescript(...) call, through the facade + ) + self.analysis: TSAnalysis = self._init_codeanalyzer(analysis_level=analyzer_level(analysis_level)) + self.application: TSApplication = self.analysis.application self._call_graph: nx.DiGraph | None = None self._index() @@ -126,53 +182,46 @@ def _get_codeanalyzer_exec(self) -> List[str]: import codeanalyzer_typescript return [str(codeanalyzer_typescript.bin_path())] - except (ModuleNotFoundError, FileNotFoundError): - pass - - raise CodeanalyzerExecutionException( - "codeanalyzer-typescript binary not found. Install it with `pip install codeanalyzer-typescript`, " - "or set $CODEANALYZER_TS_BIN." - ) - - @staticmethod - def _init_tsapplication(data: str) -> TSApplication: - """Build a TSApplication from a stringified analysis.json.""" - return TSApplication(**json.loads(data)) - - def _init_codeanalyzer(self, analysis_level: int = 1) -> TSApplication: - """Run the analyzer and return the validated TSApplication.""" - codeanalyzer_exec = self._get_codeanalyzer_exec() - target_args: List[str] = [] - if self.target_files: - for tf in self.target_files: - target_args += ["-t", str(tf).strip()] - # Restrict the call graph to the tsc resolver path when requested, replacing the obsolete - # `--call-graph-provider both`. The `--tsc-only` flag lands in codeanalyzer-typescript - # 0.4.2; older binaries reject it, so only opt in when running >= 0.4.2. - if self.tsc_only: - target_args += ["--tsc-only"] - + except (ModuleNotFoundError, FileNotFoundError) as e: + raise CodeanalyzerExecutionException( + "codeanalyzer-typescript binary not found: $CODEANALYZER_TS_BIN is unset and the " + f"`codeanalyzer-typescript` wheel is not importable or carries no binary for this platform ({e}). " + "Install it with `pip install codeanalyzer-typescript`, or set $CODEANALYZER_TS_BIN." + ) from e + + def _argv(self, analysis_level: int, output_dir: Path | None) -> List[str]: + """The 1.2.0 command line: ``-i --app-name -a <1..4> [-o + --cache-dir ] --skip-tests [--eager] [-t ]...``. The application name is what + the analyzer stamps into every ``can://typescript//...`` id.""" + project = Path(self.project_dir) + args = self._get_codeanalyzer_exec() + ["-i", str(project), "--app-name", project.name, "-a", str(analysis_level)] + if output_dir is not None: + args += ["-o", str(output_dir), "--cache-dir", str(output_dir)] + args += ["--skip-tests"] + if self.eager_analysis: + args += ["--eager"] + for tf in self.target_files or []: + args += ["-t", str(tf).strip()] + return args + + def _init_codeanalyzer(self, analysis_level: int) -> TSAnalysis: + """Run the analyzer and return the validated envelope.""" if self.analysis_json_path is None: # Read compact JSON from the stdout pipe. - args = codeanalyzer_exec + ["-i", str(Path(self.project_dir)), "-a", str(analysis_level)] + target_args + args = self._argv(analysis_level, None) try: logger.info(f"Running codeanalyzer-typescript: {' '.join(args)}") - console_out: CompletedProcess[str] = subprocess.run( - args, capture_output=True, text=True, check=True - ) - return self._init_tsapplication(console_out.stdout) + console_out: CompletedProcess[str] = subprocess.run(args, capture_output=True, text=True, check=True) + return TSAnalysis.model_validate_json(console_out.stdout) except Exception as e: # noqa: BLE001 raise CodeanalyzerExecutionException(str(e)) from e # Persist to an output directory and read analysis.json back. - analysis_json_file = Path(self.analysis_json_path).joinpath("analysis.json") + output_dir = Path(self.analysis_json_path) + analysis_json_file = output_dir / "analysis.json" needs_run = self.eager_analysis or not analysis_json_file.exists() or bool(self.target_files) if needs_run: - args = ( - codeanalyzer_exec - + ["-i", str(Path(self.project_dir)), "-a", str(analysis_level), "-o", str(self.analysis_json_path)] - + target_args - ) + args = self._argv(analysis_level, output_dir) try: logger.info(f"Running codeanalyzer-typescript: {' '.join(args)}") subprocess.run(args, capture_output=True, text=True, check=True) @@ -180,12 +229,12 @@ def _init_codeanalyzer(self, analysis_level: int = 1) -> TSApplication: raise CodeanalyzerExecutionException("codeanalyzer-typescript did not generate analysis.json.") except Exception as e: # noqa: BLE001 raise CodeanalyzerExecutionException(str(e)) from e - with open(analysis_json_file, encoding="utf-8") as f: - return self._init_tsapplication(json.dumps(json.load(f))) + return TSAnalysis.model_validate_json(analysis_json_file.read_text(encoding="utf-8")) # -----[ indexing ]----- def _index(self) -> None: - """Flatten the (recursive) symbol table into signature-keyed lookups, built once.""" + """Flatten the (recursive) symbol table into signature-keyed lookups, built once, and the + id → (graph key, kind) index that joins ``can://`` edge endpoints to those keys (TS-10).""" self._classes: Dict[str, TSClass] = {} self._interfaces: Dict[str, TSInterface] = {} self._enums: Dict[str, TSEnum] = {} @@ -194,8 +243,12 @@ def _index(self) -> None: self._functions: Dict[str, TSCallable] = {} self._methods_by_class: Dict[str, Dict[str, TSCallable]] = {} self._file_of: Dict[str, str] = {} + #: ``can://`` id → (call-graph node key, kind). Modules key on the file key, classes and + #: callables on ``signature``, externals on ``"."``. + self._id_index: Dict[str, Tuple[str, str]] = {} for fp, mod in self.application.symbol_table.items(): + self._id_index[mod.id] = (fp, "module") for f in mod.functions.values(): self._add_callable(f, fp) self._functions[f.signature] = f @@ -205,16 +258,41 @@ def _index(self) -> None: self._add_interface(it, fp) for en in mod.enums.values(): self._enums[en.signature] = en - self._file_of[en.signature] = fp + self._add_type(en, fp) for ta in mod.type_aliases.values(): self._type_aliases[ta.signature] = ta - self._file_of[ta.signature] = fp + self._add_type(ta, fp) for ns in mod.namespaces.values(): self._add_namespace(ns, fp) + for key, ext in (self.application.external_symbols or {}).items(): + node = (f"{ext.module}.{ext.name}", "external") + self._id_index[key] = node + self._id_index[ext.id] = node + # The compatibility index: keyed by the older anonymous id, the value's id is the tree id + # that replaced it (already indexed above); a residual fallback node (key == id, no tree + # home) is keyed by its own name. Anything else is an analyzer defect, never a node keyed + # by a raw id. + for key, syn in (self.application.synthesized_callables or {}).items(): + if syn.id in self._id_index: + node = self._id_index[syn.id] + elif syn.id == key and syn.name: + node = (syn.name, "callable") + else: + raise CodeanalyzerExecutionException( + f"synthesized callable {key!r} resolves to {syn.id!r}, which is neither a callable of application " + f"{self.application.id!r} nor a named residual node: codeanalyzer-typescript " + f"{self.analysis.analyzer.version} emitted an unhomed endpoint" + ) + self._id_index.setdefault(key, node) + + def _add_type(self, t, fp: str) -> None: + self._file_of[t.signature] = fp + self._id_index[t.id] = (t.signature, t.kind) def _add_callable(self, c: TSCallable, fp: str) -> None: self._callables[c.signature] = c self._file_of[c.signature] = fp + self._id_index[c.id] = (c.signature, "callable") for ic in c.inner_callables.values(): self._add_callable(ic, fp) for cl in c.inner_classes.values(): @@ -222,18 +300,16 @@ def _add_callable(self, c: TSCallable, fp: str) -> None: def _add_class(self, cl: TSClass, fp: str) -> None: self._classes[cl.signature] = cl - self._file_of[cl.signature] = fp + self._add_type(cl, fp) methods: Dict[str, TSCallable] = {} for m in cl.methods.values(): self._add_callable(m, fp) methods[m.name] = m self._methods_by_class[cl.signature] = methods - for ic in cl.inner_classes.values(): - self._add_class(ic, fp) def _add_interface(self, it: TSInterface, fp: str) -> None: self._interfaces[it.signature] = it - self._file_of[it.signature] = fp + self._add_type(it, fp) methods: Dict[str, TSCallable] = {} for m in it.methods.values(): self._add_callable(m, fp) @@ -241,6 +317,7 @@ def _add_interface(self, it: TSInterface, fp: str) -> None: self._methods_by_class[it.signature] = methods def _add_namespace(self, ns: TSNamespace, fp: str) -> None: + self._add_type(ns, fp) for f in ns.functions.values(): self._add_callable(f, fp) self._functions[f.signature] = f @@ -250,13 +327,56 @@ def _add_namespace(self, ns: TSNamespace, fp: str) -> None: self._add_interface(it, fp) for en in ns.enums.values(): self._enums[en.signature] = en - self._file_of[en.signature] = fp + self._add_type(en, fp) for ta in ns.type_aliases.values(): self._type_aliases[ta.signature] = ta - self._file_of[ta.signature] = fp + self._add_type(ta, fp) for n in ns.namespaces.values(): self._add_namespace(n, fp) + def _node_of(self, node_id: str) -> Tuple[str, str]: + """The (graph key, kind) an endpoint id resolves to. Every endpoint the analyzer emits is + homed on the tree, the externals or the synthesized index; one that is not is the + analyzer's defect, surfaced rather than skipped or keyed by a raw id.""" + try: + return self._id_index[node_id] + except KeyError: + raise CodeanalyzerExecutionException( + f"call-graph endpoint {node_id!r} is not a module, type, callable, external or synthesized callable " + f"of application {self.application.id!r}: codeanalyzer-typescript {self.analysis.analyzer.version} " + "emitted an unhomed endpoint" + ) from None + + def _callee_signature(self, node: TSBodyNode) -> str | None: + """The graph key a call node's resolved ``callee`` id maps to; ``None`` when the analyzer + left it unresolved (``null``). A ``callee`` that is neither is the same unhomed-endpoint + defect :meth:`_node_of` raises for — a raw id never reaches a return field.""" + if node.callee is None: + return None + return self._node_of(node.callee)[0] + + def _callsite(self, key: str, node: TSBodyNode) -> TSCallsite: + """The 1.x per-call record, read off a ``kind == "call"`` body node.""" + span = node.span + return TSCallsite( + method_name=node.method_name or "", + receiver_expr=node.receiver_expr, + receiver_type=node.receiver_type, + argument_types=list(node.argument_types), + type_arguments=list(node.type_arguments), + return_type=node.return_type, + callee_signature=self._callee_signature(node), + is_constructor_call=node.is_constructor_call, + is_optional_chain=node.is_optional_chain, + start_line=span.start[0] if span else -1, + start_column=span.start[1] if span else -1, + end_line=span.end[0] if span else -1, + end_column=span.end[1] if span else -1, + ) + + def _call_nodes(self, c: TSCallable) -> Iterator[Tuple[str, TSBodyNode]]: + return ((k, n) for k, n in c.body.items() if n.kind == "call") + def _resolve_callable(self, class_or_module: str, method: str | None = None) -> TSCallable | None: """Resolve a callable from either a full signature (``method is None``) or a ``(class/module, member)`` pair. Mirrors :meth:`PyCodeanalyzer.get_method` resolution.""" @@ -285,7 +405,7 @@ def _resolve_signature(self, class_or_sig: str, member: str | None = None) -> st return callable_.signature if callable_ else f"{class_or_sig}.{member}" # -----[ application / whole-program ]----- - def get_application(self) -> TSApplication: + def get_application_view(self) -> TSApplication: return self.application def get_symbol_table(self) -> Dict[str, TSModule]: @@ -295,12 +415,10 @@ def get_modules(self) -> List[TSModule]: return list(self.application.symbol_table.values()) def get_external_symbols(self) -> Dict[str, TSExternalSymbol]: - return self.application.external_symbols + return {f"{ext.module}.{ext.name}": ext for ext in (self.application.external_symbols or {}).values()} def get_synthesized_callables(self) -> Dict[str, TSSynthesizedCallable]: - """Anonymous-callback endpoints Jelly resolves that the symbol table never names. Keyed by - the synthesized signature an edge's ``source``/``target`` references.""" - return self.application.synthesized_callables + return dict(self.application.synthesized_callables or {}) def get_typescript_file(self, qualified_name: str) -> str | None: return self._file_of.get(qualified_name) @@ -310,29 +428,19 @@ def get_typescript_module(self, file_path: str) -> TSModule | None: # -----[ call graph ]----- def get_call_graph(self) -> nx.DiGraph: - """Build (and cache) a NetworkX DiGraph whose nodes are callable signatures (plus phantom - external symbols and synthesized anonymous callbacks) and whose edges are the identity-only - call edges.""" + """Build (and cache) the call graph: nodes keyed as every other accessor keys them (module + file key, type/callable signature, ``"."`` for an external) with ``id`` and + ``kind`` attributes; edges carry ``type="CALL_DEP"``, ``weight`` and ``provenance`` as the + Python backend's do. Module callers and class callees are kept (TS-11).""" if self._call_graph is not None: return self._call_graph graph = nx.DiGraph() - for sig, callable_ in self._callables.items(): - graph.add_node(sig, callable=callable_, external=False) - # Phantom (external) nodes so that import-attributed edges don't dangle. - for sig, ext in self.application.external_symbols.items(): - graph.add_node(sig, external=True, module=ext.module, name=ext.name) - # Synthesized anonymous-callback nodes so Jelly's anonymous edges don't dangle. - for sig, syn in self.application.synthesized_callables.items(): - graph.add_node(sig, external=False, synthesized=True, name=syn.name, path=syn.path) for edge in self.application.call_graph: - graph.add_edge( - edge.source, - edge.target, - type=edge.type, - weight=edge.weight, - provenance=edge.provenance, - tags=edge.tags, - ) + src, src_kind = self._node_of(edge.src) + dst, dst_kind = self._node_of(edge.dst) + graph.add_node(src, id=edge.src, kind=src_kind) + graph.add_node(dst, id=edge.dst, kind=dst_kind) + graph.add_edge(src, dst, type="CALL_DEP", weight=edge.weight, provenance=tuple(edge.prov)) self._call_graph = graph return graph @@ -348,10 +456,7 @@ def get_all_callers(self, target_class_name: str, target_method_declaration: str target = self._resolve_signature(target_class_name, target_method_declaration) if target not in graph: return {"target_method": target, "caller_details": []} - callers = [ - {"caller_signature": src, "edge": graph.get_edge_data(src, target)} - for src in graph.predecessors(target) - ] + callers = [{"caller_signature": src, "edge": graph.get_edge_data(src, target)} for src in graph.predecessors(target)] return {"target_method": target, "caller_details": callers} def get_all_callees(self, source_class_name: str, source_method_declaration: str | None = None) -> Dict: @@ -361,34 +466,18 @@ def get_all_callees(self, source_class_name: str, source_method_declaration: str source = self._resolve_signature(source_class_name, source_method_declaration) if source not in graph: return {"source_method": source, "callee_details": []} - callees = [ - {"callee_signature": tgt, "edge": graph.get_edge_data(source, tgt)} - for tgt in graph.successors(source) - ] + callees = [{"callee_signature": tgt, "edge": graph.get_edge_data(source, tgt)} for tgt in graph.successors(source)] return {"source_method": source, "callee_details": callees} - def get_class_call_graph( - self, qualified_class_name: str, method_signature: str | None = None - ) -> List[Tuple[str, str]]: - """Call-graph edges reachable from a class (or one of its methods).""" - adjacency: Dict[str, List[str]] = {} - for e in self.application.call_graph: - adjacency.setdefault(e.source, []).append(e.target) + def get_class_call_graph(self, qualified_class_name: str, method_signature: str | None = None) -> List[Tuple[str, str]]: + """Call-graph edges reachable from a class (or one of its methods), in BFS order.""" + graph = self.get_call_graph() if method_signature is not None: seeds = [method_signature] else: seeds = [m.signature for m in self._methods_by_class.get(qualified_class_name, {}).values()] - edges: List[Tuple[str, str]] = [] - seen = set(seeds) - queue = deque(seeds) - while queue: - src = queue.popleft() - for dst in adjacency.get(src, []): - edges.append((src, dst)) - if dst not in seen: - seen.add(dst) - queue.append(dst) - return edges + seeds = [s for s in seeds if s in graph] + return list(nx.edge_bfs(graph, seeds)) if seeds else [] def get_class_hierarchy(self) -> nx.DiGraph: """Inheritance/implementation graph: an edge child → base for every base_class.""" @@ -405,28 +494,29 @@ def get_class_hierarchy(self) -> nx.DiGraph: # -----[ call sites ]----- def get_call_sites(self, qualified_callable_name: str) -> List[TSCallsite]: - """The rich, syntactic call sites *inside* a callable (receiver/argument types, resolved - ``callee_signature``, position). Distinct from the resolved call-graph edges.""" + """The syntactic call sites *inside* a callable (receiver/argument types, resolved + ``callee_signature``, position) — its ``body`` nodes of ``kind == "call"``. Distinct from + the resolved call-graph edges.""" callable_ = self._callables.get(qualified_callable_name) - return list(callable_.call_sites) if callable_ else [] + return [self._callsite(k, n) for k, n in self._call_nodes(callable_)] if callable_ else [] def get_calling_lines(self, target_signature: str) -> List[int]: """Sorted, de-duplicated source lines anywhere in the project where ``target_signature`` - is invoked (matched against each call site's resolved ``callee_signature``).""" + is invoked (matched against each call node's resolved callee).""" lines: Set[int] = set() for callable_ in self._callables.values(): - for cs in callable_.call_sites: - if cs.callee_signature == target_signature and cs.start_line >= 0: - lines.add(cs.start_line) + for _, n in self._call_nodes(callable_): + if n.span is not None and self._callee_signature(n) == target_signature: + lines.add(n.span.start[0]) return sorted(lines) def get_call_targets(self, source_signature: str) -> Set[str]: - """The set of call targets invoked from a callable, taken from its call sites. Resolved - ``callee_signature`` when available, otherwise the bare ``method_name``.""" + """The set of call targets invoked from a callable, taken from its call nodes. Resolved + callee signature when available, otherwise the bare ``method_name``.""" callable_ = self._callables.get(source_signature) if callable_ is None: return set() - return {cs.callee_signature or cs.method_name for cs in callable_.call_sites} + return {self._callee_signature(n) or n.method_name or "" for _, n in self._call_nodes(callable_)} # -----[ classes / interfaces / enums / type-aliases ]----- def get_all_classes(self) -> Dict[str, TSClass]: @@ -449,8 +539,9 @@ def get_all_type_aliases(self) -> Dict[str, TSTypeAlias]: return self._type_aliases def get_all_nested_classes(self, qualified_class_name: str) -> List[TSClass]: - cls = self._classes.get(qualified_class_name) - return list(cls.inner_classes.values()) if cls else [] + # The v2 class facet nests no types (only namespaces and callables do), so a class never + # has nested classes on this wire; kept for the 1.x surface. + return [] def get_all_sub_classes(self, qualified_class_name: str) -> Dict[str, TSClass]: return {sig: cls for sig, cls in self._classes.items() if qualified_class_name in cls.base_classes} @@ -499,11 +590,7 @@ def get_method_parameters(self, qualified_class_name: str, qualified_method_name return [p.name for p in method.parameters] if method else [] def get_all_constructors(self, qualified_class_name: str) -> Dict[str, TSCallable]: - return { - name: m - for name, m in self._methods_by_class.get(qualified_class_name, {}).items() - if m.kind == "constructor" - } + return {name: m for name, m in self._methods_by_class.get(qualified_class_name, {}).items() if m.kind == "constructor"} def get_all_functions(self) -> Dict[str, TSCallable]: return self._functions @@ -527,6 +614,56 @@ def get_all_variables(self) -> Dict[str, List[TSVariableDeclaration]]: """Module-level variable declarations per file.""" return {fp: list(m.variables) for fp, m in self.application.symbol_table.items()} + # -----[ repository artifacts — the shared Py* models, as the generic ABC promises ]----- + @staticmethod + def _py_config_key(ck: TSConfigKey) -> PyConfigKey: + """``TSConfigKey.value`` may be a JSON number or boolean (``"strict": true``); + ``PyConfigKey.value`` is a string, so a non-string value is rendered as its JSON text + (``true``, ``1``, ``1.5``), which is what the artifact itself says.""" + value = ck.value if isinstance(ck.value, str) or ck.value is None else json.dumps(ck.value) + return PyConfigKey(id=ck.id, key=ck.key, namespace=ck.namespace, value=value, span=ck.span.model_dump() if ck.span else None, references=list(ck.references)) + + def get_artifacts(self) -> Dict[str, PyArtifact]: + """Every non-code artifact (see :meth:`AnalysisBackend.get_artifacts`), keyed by + repo-relative path as the wire keys them; every ``TSArtifact`` field has a home on + :class:`PyArtifact`.""" + return { + path: PyArtifact(**a.model_dump(exclude={"config_keys"}), config_keys=[self._py_config_key(ck) for ck in a.config_keys]) + for path, a in self.application.artifacts.items() + } + + def get_dependencies(self, *, direct_only: bool = False, ecosystem: str | None = None, declared_in: str | None = None) -> List[PyDependency]: + """Every declared dependency, optionally filtered (see + :meth:`AnalysisBackend.get_dependencies`). The TypeScript wire carries no ``ecosystem`` + field — every dependency is an npm package (``pkg:npm/``), so that is what the + shared model's field says and what the ``ecosystem`` filter matches.""" + deps = [PyDependency(ecosystem="npm", **d.model_dump()) for d in self.application.dependencies] + if direct_only: + deps = [d for d in deps if d.direct] + if ecosystem is not None: + deps = [d for d in deps if d.ecosystem == ecosystem] + if declared_in is not None: + deps = [d for d in deps if d.declared_in == declared_in] + return deps + + def get_config_keys(self) -> Dict[str, PyConfigKey]: + """Every configuration key, flattened out of the artifact that defines it and keyed by id + (see :meth:`AnalysisBackend.get_config_keys`).""" + return {ck.id: self._py_config_key(ck) for a in self.application.artifacts.values() for ck in a.config_keys} + + def get_config_uses(self, key: str | None = None) -> List[PyConfigUseEdge]: + """Resolved code-to-config edges (see :meth:`AnalysisBackend.get_config_uses`).""" + edges = [PyConfigUseEdge(**u.model_dump()) for u in self.application.config_uses] + if key is None: + return edges + matching_ids = {ck.id for ck in self.get_config_keys().values() if ck.key == key} + return [e for e in edges if e.dst in matching_ids] + + def get_unresolved_config_reads(self) -> List[PyConfigRead]: + """Detector-matched config reads that resolved to no declared key (see + :meth:`AnalysisBackend.get_unresolved_config_reads`) — ``TSApplication.config_reads``.""" + return [PyConfigRead(**r.model_dump()) for r in self.application.config_reads] + # -----[ decorators ]----- def get_decorators(self, qualified_callable_name: str) -> List[TSDecorator]: callable_ = self._callables.get(qualified_callable_name) @@ -580,15 +717,15 @@ def _iter_callables(self) -> Iterator[Tuple[TSCallable, str | None, str | None]] def get_callables_overview(self) -> List[TSCallableOverview]: """Return a lightweight overview of every callable in the application (see :meth:`TSAnalysisBackend.get_callables_overview`).""" - return [TSCallableOverview.from_callable(c, owner_sig, owner_kind) for c, owner_sig, owner_kind in self._iter_callables()] + return [TSCallableOverview.from_callable(c, owner_sig, owner_kind, path=self._file_of[c.signature]) for c, owner_sig, owner_kind in self._iter_callables()] def get_method_bodies(self, signatures: List[str]) -> Dict[str, str]: - """Return ``{signature: code}`` for the requested signatures that exist and have a body - (omits callables whose ``code`` is ``None``, e.g. implicit constructors).""" + """Return ``{signature: code}`` for the requested signatures that exist and have source + text (omits an implicit constructor, whose empty span slices to ``""``).""" result: Dict[str, str] = {} for sig in signatures: c = self._callables.get(sig) - if c is not None and c.code is not None: + if c is not None and c.code: result[sig] = c.code return result @@ -596,7 +733,7 @@ def get_decorated_callables(self, markers: List[str]) -> List[TSCallableOverview """Return overviews of callables decorated with any of ``markers``.""" marker_set = set(markers) return [ - TSCallableOverview.from_callable(c, owner_sig, owner_kind) + TSCallableOverview.from_callable(c, owner_sig, owner_kind, path=self._file_of[c.signature]) for c, owner_sig, owner_kind in self._iter_callables() if marker_set.intersection(d.name for d in c.decorators) ] @@ -607,5 +744,653 @@ def get_callsites_for(self, signatures: List[str]) -> Dict[str, List[TSCallsite] for sig in signatures: c = self._callables.get(sig) if c is not None: - result[sig] = list(c.call_sites) + result[sig] = [self._callsite(k, n) for k, n in self._call_nodes(c)] return result + + # ----------------------------------------------------------- entrypoints / config readers + def get_entrypoints(self) -> List[TSCallableOverview]: + """Return overviews of every callable marked ``is_entrypoint`` (see + :meth:`TSAnalysisBackend.get_entrypoints`). ``is_entrypoint`` is ``Optional[bool]``, so + the test is truthiness, not ``is True``: below 1.3.0 it is ``None``, which is not a mark.""" + return [ + TSCallableOverview.from_callable(c, owner_sig, owner_kind, path=self._file_of[c.signature]) for c, owner_sig, owner_kind in self._iter_callables() if c.is_entrypoint + ] + + def get_entrypoint_classes(self) -> List[TSClassOverview]: + """Return overviews of every class marked ``is_entrypoint`` (see + :meth:`TSAnalysisBackend.get_entrypoint_classes`).""" + return [TSClassOverview.from_class(cl, path=self._file_of[sig]) for sig, cl in self._classes.items() if cl.is_entrypoint] + + def get_entrypoint_coverage(self) -> EntrypointCoverage: + """Return the entrypoint pass's coverage record (see + :meth:`TSAnalysisBackend.get_entrypoint_coverage`) -- a passthrough of + ``TSApplication.entrypoint_report``, which this backend has in full. + + The field is optional on the model (TS-1 kept it so, because the graph-backed application + view carries the report as a string property on the anchor rather than as a structured + field), so its absence is reported rather than fabricated.""" + report = self.application.entrypoint_report + if report is None: + return EntrypointCoverage( + diagnostics=[ + Diagnostic( + code="entrypoint_report_unavailable", + message="This analysis.json carries no TSApplication.entrypoint_report, so the entrypoint pass's coverage cannot be reported. " + "codeanalyzer-typescript 1.3.0 and newer always emit it.", + ) + ] + ) + return EntrypointCoverage( + frameworks_detected=list(report.frameworks_detected), + rulesets=list(report.rulesets), + unresolved=dict(report.unresolved), + errors=list(report.errors), + ) + + def get_config_readers(self, key: str) -> List[TSCallableOverview]: + """Return overviews of every callable reading configuration key ``key`` (see + :meth:`TSAnalysisBackend.get_config_readers`). + + ``PyConfigUseEdge.src`` is the reading call's own body-node id. It is matched against each + callable's ``body`` map rather than split on ``@`` back to an owner: an anonymous + callable's id contains an ``@`` of its own (``…/``), so the split the Python + twin can afford is not sound here.""" + reading = {e.src for e in self.get_config_uses(key)} + if not reading: + return [] + return [ + TSCallableOverview.from_callable(c, owner_sig, owner_kind, path=self._file_of[c.signature]) + for c, owner_sig, owner_kind in self._iter_callables() + if not reading.isdisjoint(node.id for node in (c.body or {}).values()) + ] + + # ===================================================================================== + # The addressing surface (leg 2.5b, TS-2) -- over the in-memory tree. + # ===================================================================================== + @cached_property + def _by_module(self) -> Dict[str, List[Tuple[TSCallable, str | None, str | None]]]: + """``module key -> the callables declared in it``, with their owner pair. + + Built lazily rather than in :meth:`_index`: every existing accessor answers without it, and + on a real application (superset-frontend: 11,085 callables) an index nobody asked for is + memory nobody asked for. :meth:`_iter_callables` is the domain, so ``locate`` and + ``resolve_callable`` see exactly the set ``get_callables_overview`` reports. + """ + out: Dict[str, List[Tuple[TSCallable, str | None, str | None]]] = defaultdict(list) + for c, owner_sig, owner_kind in self._iter_callables(): + out[self._file_of[c.signature]].append((c, owner_sig, owner_kind)) + return dict(out) + + def _owner_name(self, owner_sig: str | None) -> str | None: + owner = self._classes.get(owner_sig or "") or self._interfaces.get(owner_sig or "") + return owner.name if owner is not None else None + + @staticmethod + def _contains(span: TSSpan | None, line: int) -> bool: + return span is not None and span.start[0] <= line <= span.end[0] + + def _not_analysed(self, path: str, line: int) -> LocateResult: + """The ``file_not_in_graph`` outcome, with the one distinction this backend *can* draw. + + Unlike the Neo4j backend (which attaches to a graph and may not have the project checked + out), this one runs against the project directory, so it can tell "the file is there and + was not analysed" -- a ``--target-files`` narrowing, an excluded directory, a parse the + analyzer skipped -- from "there is no such file". The code stays ``file_not_in_graph`` + either way; the distinction rides in the message, which is the field an agent reads. + """ + on_disk = Path(path).is_file() or bool(self.project_dir and (Path(self.project_dir) / path).is_file()) + why = "the file exists but no analysed module covers it" if on_disk else "no such file in the analysed project" + return LocateResult( + body=None, + callable=None, + type=None, + module=ModuleRef(path=str(path)), + source="", + span=Span(start=(line, 0), end=(line, 0), bytes=(0, 0)), + diagnostics=[Diagnostic(code="file_not_in_graph", message=f"{path} is not covered by any analysed module ({why}).")], + ) + + def _body_ref(self, c: TSCallable, line: int) -> BodyRef | None: + """The innermost body node of ``c`` containing ``line``, as the language-neutral handle. + + ``None`` is a real outcome, not an error: a position on a declaration line or a blank line + inside a callable is contained by the callable and by no body node, and the caller still + gets the callable. Ties break the same way the Neo4j backend breaks them -- narrowest line + span, then the deeper column parsed out of the node's own key + (:func:`~cldk.analysis.commons.keys.body_key_column`), then the key -- so both backends + resolve a tie to the same node. + """ + matches = [(k, n) for k, n in (c.body or {}).items() if self._contains(n.span, line)] + if not matches: + return None + key, node = min(matches, key=lambda kn: (kn[1].span.end[0] - kn[1].span.start[0], -body_key_column(kn[0]), kn[0])) + if not node.id: + raise CodeanalyzerExecutionException( + f"body node {key!r} of {c.signature!r} carries no id: codeanalyzer-typescript {self.analysis.analyzer.version} " + "emitted an unaddressable body node (ids are required from 1.3.0, cants#165)" + ) + return BodyRef(id=node.id, kind=node.kind, span=node.span, callee=node.callee) + + def _locate_one(self, path: str, line: int) -> LocateResult: + # Whatever the caller's scanner printed ("./src/app.ts", an absolute path) is normalised to + # the symbol-table key first; an unnormalised path would otherwise read as file_not_in_graph. + key = resolve_module_key(str(path), self.application.symbol_table.keys()) + module = self.application.symbol_table.get(key) + if module is None: + return self._not_analysed(key, line) + module_ref = ModuleRef(path=key) + # Innermost callable = narrowest line span containing the position. A position between two + # callables, or at module scope, matches none and falls through to module_scope rather than + # snapping to a neighbour. Equal widths (an arrow inside a one-line function) tie, and the + # tie is broken on the longer signature, deeper first -- a nested callable's signature + # extends its owner's -- which is the rule the Neo4j backend applies to the same rows. + found = min( + ((c, o, k) for c, o, k in self._by_module.get(key, ()) if self._contains(c.span, line)), + key=lambda cok: (cok[0].span.end[0] - cok[0].span.start[0], -len(cok[0].signature), cok[0].signature), + default=None, + ) + if found is None: + return LocateResult( + body=None, + callable=None, + type=None, + module=module_ref, + source=module.source, + span=Span(start=(line, 0), end=(line, 0), bytes=(0, 0)), + diagnostics=[Diagnostic(code="module_scope", message=f"line {line} is at module scope in {key}.")], + ) + c, owner_sig, _ = found + owner_name = self._owner_name(owner_sig) + body = self._body_ref(c, line) + return LocateResult( + body=body, + node_id=body.id if body else None, + callable=CallableRef(signature=c.signature, name=c.name, class_signature=owner_sig), + type=TypeRef(signature=owner_sig, name=owner_name) if owner_sig and owner_name else None, + module=module_ref, + source=c.code or "", + span=c.span, + diagnostics=[], + ) + + def locate(self, path: str, line: int) -> LocateResult: + """Resolve a source position to its enclosing callable (see :meth:`TSAnalysisBackend.locate`).""" + return self._locate_one(path, line) + + def locate_many(self, positions: Sequence[Tuple[str, int]]) -> List[LocateResult]: + """Resolve many positions (see :meth:`TSAnalysisBackend.locate_many`). Purely in memory + here -- there is no round trip to batch -- but the results still come back in input order, + matching the Neo4j backend's contract.""" + return [self._locate_one(path, line) for path, line in positions] + + def resolve_callable(self, name: str, *, in_class: str | None = None, in_module: str | None = None) -> SliceNode: + """Resolve a callable name against the in-memory tree (see + :meth:`TSAnalysisBackend.resolve_callable`). + + The candidate domain is :meth:`_iter_callables` -- the same set + :meth:`get_callables_overview` reports and the same set the Neo4j backend resolves against. + Nothing is filtered before the shared policy runs: this backend has the whole list in memory + already, so handing the policy the unfiltered domain is the strongest form of "both backends + resolve over the same set". + """ + candidates = [CallableCandidate(c.signature, owner_sig, self._file_of[c.signature]) for c, owner_sig, _ in self._iter_callables()] + sig = resolve_callable_signature(name, candidates, in_class=in_class, in_module=in_module, dotted=ts_module_dotted) + c = self._callables[sig] + return SliceNode(file=self._file_of[sig], line=c.span.start[0], callable=sig, kind="callable", name=c.name, source=None, ref=c.id) + + def resolve_value(self, name: str, *, within: str) -> SliceNode: + """Resolve a value name inside a callable (see :meth:`TSAnalysisBackend.resolve_value`). + + ``TSBodyNode.of`` carries the value a ``formal_in`` vertex stands for -- the same fact the + graph projects as ``b.of`` -- and in TypeScript it is the parameter's own text, with none of + the ``":mod::name"`` grammar codeanalyzer-python marks captured globals with. So + there is nothing to translate and no kind to infer: it is a parameter. + """ + owner = resolve_within(self.resolve_callable, within) + c = self._callables[owner.callable] + # A list, not a dict keyed by name: two values that resolve to the same name are a genuine + # ambiguity the policy must see and raise on, and a dict would silently keep the last. + entries = [(n, n.of) for n in (c.body or {}).values() if n.kind == "formal_in" and n.of] + chosen = resolve_value_name(name, [v for _, v in entries], within=owner.callable) + node = next(n for n, v in entries if v == chosen) + return SliceNode(file=owner.file, line=owner.line, callable=owner.callable, kind="parameter", name=chosen, source=None, ref=node.id or "") + + def _sources_for(self, refs: Sequence[str]) -> Dict[str, "str | None"]: + """Source text for every ref this application holds (see + :meth:`TSAnalysisBackend._sources_for`). + + One walk of the callable tree for the whole batch, not one per ref. A callable answers to + both of its names (signature and ``can://`` id); a body node is sliced out of its module by + its span, so this backend fills in the statements and call sites the graph cannot. A vertex + with **no** span -- every ``formal_in``/``formal_out``/``actual_*`` -- maps to ``None``: it + is a dataflow position, not a region of the file, and there is nothing to read on either + backend. An external ghost is likewise found and textless, by definition. + """ + wanted = set(refs) + found: Dict[str, "str | None"] = {} + for key, ext in (self.application.external_symbols or {}).items(): + for ref in (key, ext.id): + if ref in wanted: + found[ref] = None + for c, _, _ in self._iter_callables(): + source = self.application.symbol_table[self._file_of[c.signature]].source + for ref in (c.signature, c.id): + if ref in wanted: + found[ref] = c.code or None + for node in (c.body or {}).values(): + if node.id in wanted: + found[node.id] = source[node.span.bytes[0] : node.span.bytes[1]] if node.span else None + return found + + def get_source(self, node_id: str) -> str: + """Source text for one node (see :meth:`TSAnalysisBackend.get_source`). + + Routed through :meth:`_sources_for` rather than re-walking the tree with its own splitting + rule: a TypeScript body-node id cannot be taken apart on ``@`` (an anonymous callable's own + id contains one), so the id is looked up whole, and the two ways of having no text stay + apart -- absent from the mapping is "nothing carries this id", ``None`` is "this exists and + has no recoverable source". + """ + found = self._sources_for([node_id]) + if node_id not in found: + raise KeyError(f"no callable, body node or external symbol of application {self.application.id!r} is addressed by {node_id!r}") + code = found[node_id] + if not code: + raise KeyError(f"no recoverable source for {node_id!r} (it carries no span, or the analyzer emitted no text for it)") + return code + + @property + def has_resolution_edges(self) -> bool: + """See :meth:`TSAnalysisBackend.has_resolution_edges`. ``True`` from analysis level 2, the + level at which cants resolves a call node's ``callee``; below it every call node carries + ``callee: null`` and every ``callee_signature`` from :meth:`get_callsites_for` is ``None`` + for that reason and not because the individual sites failed to resolve. Read off + ``analysis.max_level`` -- what the analyzer actually produced -- not off the level the + caller asked for.""" + return self.analysis.max_level >= _CALLEE_RESOLUTION_LEVEL + + # ===================================================================================== + # The dataflow surface (leg 2.5b, Task 2) -- over the in-memory tree. + # + # THE LOCAL BACKEND ANSWERS INTERPROCEDURALLY, as the Python twin does and for the same reason: + # a level-4 run carries the whole SDG in the model, so the cross-callable index this backend + # lacks it can BUILD out of the very lists ``--emit neo4j`` projects. + # + # TSCallable.ddg / .cdg / .summary endpoints are LOCAL body keys -> joined through the + # callable's own ``body`` map, whose nodes carry ``id`` + # TSApplication.param_in / param_out endpoints are ALREADY global ids + # + # ONE DIFFERENCE FROM PYTHON, AND IT IS THE ANALYZER'S, NOT A CHOICE HERE: Python composes a + # body node's global id from ``(callable id, body key)``. TypeScript reads it off the node -- + # 1.3.0 emits ``id`` on all 125,532 body nodes (cants#165) -- because a TypeScript id cannot be + # composed safely: an anonymous callable's own id already contains an ``@`` + # (``.../``), so the ``@`` grammar is not invertible by splitting + # and is not worth re-deriving when the analyzer states it. + # ===================================================================================== + #: The analyzer level at which ``cfg``/``cdg``/``ddg`` first exist. Read against + #: ``analysis.max_level`` -- what the analyzer actually produced -- not the level the caller + #: asked for, the same rule :attr:`has_resolution_edges` already applies. + _DATAFLOW_LEVEL = ANALYZER_LEVELS[AnalysisLevel.program_dependency_graph] + + def _require_dataflow(self) -> None: + """Refuse, naming both levels, when this analysis was built below the dataflow pass. + + The one guard, shared by the per-callable graphs, the slices and the value-flow accessors: + they come from the same analyzer pass and go dark together, and a second copy of this check + is a second thing to keep in step. It raises instead of returning empty because at a + shallower level an empty answer would mean "not analysed" while looking exactly like "no + dependence" (D7). The Neo4j backend has no such mode -- ``--emit neo4j`` is always full + depth -- so this is the only place the contract can be broken. + """ + level = self.analysis.max_level + if level < self._DATAFLOW_LEVEL: + raise CodeanalyzerUsageException( + f"control and data flow need analysis_level='program_dependency_graph' or deeper " + f"(analyzer level {self._DATAFLOW_LEVEL}); this analysis was produced at " + f"'{LEVEL_NAMES.get(level, level)}' (analyzer level {level}), where codeanalyzer-typescript emits no " + "cfg/cdg/ddg at all. Returning an empty result would be indistinguishable from a callable that has " + "no dependence, so this raises instead. Rebuild with " + "CLDK.typescript(..., analysis_level='system_dependency_graph')." + ) + + def _body_ids(self, c: TSCallable) -> Dict[str, str]: + """``local body key -> global body-node id`` for one callable. + + Read off the nodes, never composed (see the block comment above). A span-bearing node + without an id is an analyzer defect and is raised as one rather than being worked around -- + the same guard :meth:`_body_ref` applies to the addressing surface. + """ + out: Dict[str, str] = {} + for key, node in (c.body or {}).items(): + if not node.id: + raise CodeanalyzerExecutionException( + f"body node {key!r} of {c.signature!r} carries no id: codeanalyzer-typescript {self.analysis.analyzer.version} " + "emitted an unaddressable body node (ids are required from 1.3.0, cants#165)" + ) + out[key] = node.id + return out + + @staticmethod + def _endpoint(c: TSCallable, ids: Dict[str, str], key: str) -> str: + """One graph endpoint's global id, refusing a key the callable's ``body`` map does not hold. + + Silently dropping such an edge would make a page's ``total`` disagree with the graph's for + no reason a caller could see; the graph, whose relationships are between real + ``:TSBodyNode``s, cannot have the problem at all. + """ + try: + return ids[key] + except KeyError: + raise CodeanalyzerExecutionException( + f"an edge of {c.signature!r} names the body key {key!r}, which is not in that callable's body map: " + "codeanalyzer-typescript emitted a dangling intra-callable edge" + ) from None + + def _graphs_of(self, name: str, in_class: str | None, page_size: int) -> TSCallable: + """The callable ``name`` resolves to, once this backend is deep enough to have dataflow. + + ``page_size`` is validated *first*, before the level guard and before resolution, so a + malformed argument is a ``ValueError`` before anything else -- the order the Neo4j backend + also applies, so the two cannot answer the same bad call with different exceptions. + Resolution is :meth:`resolve_callable`'s, not a second path. + """ + check_page_size(page_size) + self._require_dataflow() + return self._callables[self.resolve_callable(name, in_class=in_class).callable] + + def get_cfg(self, callable: str, *, in_class: str | None = None, page_size: int = DEFAULT_PAGE_SIZE, cursor: str | None = None) -> EdgePage[TSCfgEdge]: + """One page of control flow within one callable (see :meth:`TSAnalysisBackend.get_cfg`).""" + c = self._graphs_of(callable, in_class, page_size) + ids = self._body_ids(c) + edges = [TSCfgEdge(src=self._endpoint(c, ids, e.src), dst=self._endpoint(c, ids, e.dst), kind=e.kind) for e in c.cfg or []] + return edge_page(TSCfgEdge, c.signature, edges, CFG_ORDER, page_size, cursor) + + def get_cdg(self, callable: str, *, in_class: str | None = None, page_size: int = DEFAULT_PAGE_SIZE, cursor: str | None = None) -> EdgePage[TSCdgEdge]: + """One page of control dependence within one callable (see :meth:`TSAnalysisBackend.get_cdg`).""" + c = self._graphs_of(callable, in_class, page_size) + ids = self._body_ids(c) + edges = [TSCdgEdge(src=self._endpoint(c, ids, e.src), dst=self._endpoint(c, ids, e.dst)) for e in c.cdg or []] + return edge_page(TSCdgEdge, c.signature, edges, CDG_ORDER, page_size, cursor) + + def get_ddg(self, callable: str, *, in_class: str | None = None, page_size: int = DEFAULT_PAGE_SIZE, cursor: str | None = None) -> EdgePage[TSDdgEdge]: + """One page of data dependence within one callable (see :meth:`TSAnalysisBackend.get_ddg`).""" + c = self._graphs_of(callable, in_class, page_size) + ids = self._body_ids(c) + edges = [TSDdgEdge(src=self._endpoint(c, ids, e.src), dst=self._endpoint(c, ids, e.dst), var=e.var, prov=list(e.prov or [])) for e in c.ddg or []] + return edge_page(TSDdgEdge, c.signature, edges, DDG_ORDER, page_size, cursor) + + # -----[ slicing and reachability ]----- + @cached_property + def _sdg(self) -> Tuple[Dict[str, Dict[str, Dict[str, list]]], Dict[str, Tuple[TSCallable, TSBodyNode, str]]]: + """``(adjacency, node index)`` over the whole application's SDG, built once and cached. + + ``adjacency`` is ``{"forward": {src: {dst: [label]}}, "backward": {dst: {src: [label]}}}`` -- + both directions, because a backward slice is not derivable from a forward index without + inverting it, and inverting it per call is the same work done repeatedly. A ``label`` is + ``(relationship type, var, prov)``: what a path hop has to report, and what the graph carries + on the corresponding relationship. It is a **list** per pair because parallel edges are + ordinary and collapsing them would merge several pieces of evidence into one. + + ``node index`` maps a global body-node id to ``(owning callable, body node, module key)``, + which is everything :meth:`_slice_node` needs to describe a reached node without a second + walk. It is built from :meth:`_iter_callables`, so the described set is exactly the graph's: + ``_SLICE`` there joins each reached node back to a ``:TSCallable`` and drops what it cannot. + """ + forward: Dict[str, Dict[str, list]] = {} + backward: Dict[str, Dict[str, list]] = {} + nodes: Dict[str, Tuple[TSCallable, TSBodyNode, str]] = {} + + def link(src: str, dst: str, label: tuple) -> None: + forward.setdefault(src, {}).setdefault(dst, []).append(label) + backward.setdefault(dst, {}).setdefault(src, []).append(label) + + for c, _, _ in self._iter_callables(): + path = self._file_of[c.signature] + ids = self._body_ids(c) + for key, node in (c.body or {}).items(): + nodes[ids[key]] = (c, node, path) + # The relationship name each list is projected as, so a hop reports the same ``via`` + # here as it does over the graph -- ``VIA`` is the single translation table. + for rel, edges in (("TS_DDG", c.ddg), ("TS_CDG", c.cdg), ("TS_SUMMARY", c.summary)): + for e in edges or []: + link( + self._endpoint(c, ids, e.src), + self._endpoint(c, ids, e.dst), + (rel, getattr(e, "var", None), tuple(getattr(e, "prov", None) or ())), + ) + # Endpoints here are already global, so they are used as-is -- joining them again would + # mint ids that name nothing. + for rel, edges in (("TS_PARAM_IN", self.application.param_in), ("TS_PARAM_OUT", self.application.param_out)): + for e in edges or []: + link(e.src, e.dst, (rel, e.var, ())) + return {"forward": forward, "backward": backward}, nodes + + def _reach(self, ref: str, direction: str, depth: int | None) -> set: + """The set of node ids reachable from ``ref`` in at most ``depth`` hops. + + Level-by-level rather than a plain stack, because ``depth`` is a hop budget and a + depth-first walk cannot count hops without revisiting. Shared by the slices and by the two + flow predicates so "reachable" means one thing on this backend. + """ + edges = self._sdg[0][direction] + seen, frontier, hops = {ref}, [ref], 0 + while frontier and (depth is None or hops < depth): + nxt = [d for src in frontier for d in edges.get(src, ()) if d not in seen] + seen.update(nxt) + frontier = nxt + hops += 1 + return seen + + def _slice_node(self, ref: str) -> SliceNode: + """One reached body node in the caller's vocabulary. + + A parameter-passing vertex has no span of its own -- it is a dataflow position, not a region + of the file (55,778 of the reference application's 125,532 body nodes carry no lines at + all) -- so the *callable's* first line stands in, which is where a reader would go looking + for it and what the Neo4j projection's ``coalesce`` produces from the same two properties. + """ + c, node, path = self._sdg[1][ref] + kind, name = ts_body_node_kind(node.kind, node.of) + return SliceNode(file=path, line=node.span.start[0] if node.span else c.span.start[0], callable=c.signature, kind=kind, name=name, source=None, ref=ref) + + def _slice_from(self, root: SliceNode, direction: str, depth: int | None, max_nodes: int) -> Slice: + """:meth:`_reach`'s closure from ``root``, described and capped like the graph's. + + The whole closure is computed and *then* cut: ``total`` has to be the size of the whole + slice for the cap to be reportable (E5), and there is nothing cheaper to compute it from -- + the same reason the Cypher counts before it pages. + """ + nodes = self._sdg[1] + found = [self._slice_node(ref) for ref in sorted(self._reach(root.ref, direction, depth)) if ref in nodes] + return Slice(nodes=found[:max_nodes], roots=[root], resolved=slice_resolved([root]), total=len(found)) + + def slice_backward(self, src: str, *, within: str, depth: int | None = DEFAULT_DEPTH, max_nodes: int = DEFAULT_MAX_NODES) -> Slice: + """What affects this value (see :meth:`TSAnalysisBackend.slice_backward`).""" + check_depth(depth) + check_max_nodes(max_nodes) + self._require_dataflow() + return self._slice_from(self.resolve_value(src, within=within), "backward", depth, max_nodes) + + def slice_forward(self, src: str, *, within: str, depth: int | None = DEFAULT_DEPTH, max_nodes: int = DEFAULT_MAX_NODES) -> Slice: + """What this value affects (see :meth:`TSAnalysisBackend.slice_forward`).""" + check_depth(depth) + check_max_nodes(max_nodes) + self._require_dataflow() + return self._slice_from(self.resolve_value(src, within=within), "forward", depth, max_nodes) + + # -----[ the call graph, in the caller's vocabulary ]----- + @cached_property + def _externals_by_key(self) -> Dict[str, TSExternalSymbol]: + """``"." -> the external`` — the key the call graph uses for a ghost, which is + not the key :meth:`get_external_symbols` is keyed by (that is the analyzer's own wire key).""" + return {f"{e.module}.{e.name}": e for e in (self.application.external_symbols or {}).values()} + + def _vertex(self, key: str, kind: str, node_id: str) -> SliceNode: + """One call-graph vertex as a :class:`SliceNode`, whatever kind of thing it is. + + Three shapes, because TypeScript's call graph has three (TS-11): a declared callable; a + **module**, which cants makes the caller of its own top-level code and which is addressed by + its file key everywhere on this surface; and an **external** ghost, which was never analysed + and so has no position -- ``file=""`` and ``line=0``, with ``kind`` saying why. Its + ``callable`` is the readable ``"."``, never its ``can://`` id (E6). + """ + if kind == "module": + module = self.application.symbol_table.get(key) + return SliceNode(file=key, line=module.span.start[0] if module else 0, callable=key, kind="module", name=key, source=None, ref=node_id) + if kind == "external": + ext = self._externals_by_key.get(key) + return SliceNode(file="", line=0, callable=key, kind="external", name=ext.name if ext else key, source=None, ref=node_id) + c = self._callables.get(key) + if c is not None: + return SliceNode(file=self._file_of[key], line=c.span.start[0], callable=key, kind="callable", name=c.name, source=None, ref=node_id) + # A type vertex -- a class as the callee of ``new X()``. None occur on the reference + # application (its 17,712 call edges name only callables, modules and externals), but the + # id index keeps the five type kinds, so the shape is described rather than dropped. + return SliceNode(file=self._file_of.get(key, ""), line=0, callable=key, kind=kind, name=key.rsplit(".", 1)[-1], source=None, ref=node_id) + + def _call_graph_node(self, key: str) -> SliceNode: + """A vertex of :meth:`get_call_graph`, described from the ``id``/``kind`` it already carries.""" + attrs = self.get_call_graph().nodes[key] + return self._vertex(key, attrs["kind"], attrs["id"]) + + @property + def _callable_call_graph(self) -> nx.DiGraph: + """The call graph restricted to callable vertices — the domain ``reaches`` and + ``call_paths_between`` walk. + + A view, not a copy. It is what the Neo4j backend's hop-by-hop ``:TSCallable`` predicate + selects, written here so the two cannot disagree: a module has no *incoming* call edge and + an external no *outgoing* one on the reference graph, so restricting changes no answer + today, and it is what keeps a future emitter from silently widening one. + """ + graph = self.get_call_graph() + return graph.subgraph([n for n, a in graph.nodes(data=True) if a.get("kind") == "callable"]) + + def reaches(self, src: str, dst: str, *, depth: int | None = None) -> bool: + """Is there a call path (see :meth:`TSAnalysisBackend.reaches`)?""" + check_depth(depth) + a = self.resolve_callable(src).callable + b = self.resolve_callable(dst).callable + graph = self._callable_call_graph + if a not in graph or b not in graph: + return False + # ``nx.descendants`` is unbounded and ``ego_graph`` the bounded form; both exclude the + # zero-hop case, which is what makes ``reaches(x, x)`` false unless a real cycle exists. + reachable = nx.descendants(graph, a) if depth is None else set(nx.ego_graph(graph, a, radius=depth).nodes) - {a} + return b in reachable + + def backward_cone(self, sinks: Sequence[str], *, depth: int | None = DEFAULT_DEPTH, max_nodes: int = DEFAULT_MAX_NODES) -> Slice: + """Everything that can reach these sinks (see :meth:`TSAnalysisBackend.backward_cone`). + + Walked over the **whole** call graph and not the callable-only view: a module is the caller + of its own top-level code, so it is part of the answer to "what could get here" even though + it can never be the interior of a path. + """ + check_depth(depth) + check_max_nodes(max_nodes) + roots = cone_sinks(self.resolve_callable, sinks) + graph = self.get_call_graph() + described: Dict[str, SliceNode] = {r.callable: r for r in roots} + for root in roots: + if root.callable not in graph: + continue + # ``ego_graph`` follows *successors*, so a backward question needs the reversed view. + back = graph.reverse(copy=False) + reached = nx.ancestors(graph, root.callable) if depth is None else set(nx.ego_graph(back, root.callable, radius=depth).nodes) + for key in reached: + described.setdefault(key, self._call_graph_node(key)) + # Ordered by ``ref``, the order ``Slice.nodes`` documents and the graph's ``ORDER BY m.id`` + # produces -- not by signature, which would make a capped cone return different vertices per + # backend. + found = sorted(described.values(), key=lambda n: n.ref) + return Slice(nodes=found[:max_nodes], roots=roots, resolved=slice_resolved(roots), total=len(found)) + + def callers_of(self, name: str, *, in_class: str | None = None, in_module: str | None = None) -> List[SliceNode]: + """Who calls this, module callers included (see :meth:`TSAnalysisBackend.callers_of`).""" + return self._call_neighbours(name, in_class, in_module, callers=True) + + def callees_of(self, name: str, *, in_class: str | None = None, in_module: str | None = None) -> List[SliceNode]: + """What this calls, externals included (see :meth:`TSAnalysisBackend.callees_of`).""" + return self._call_neighbours(name, in_class, in_module, callers=False) + + def _call_neighbours(self, name: str, in_class: str | None, in_module: str | None, *, callers: bool) -> List[SliceNode]: + """One hop of the call graph, in the caller's vocabulary, ordered by ``ref``. + + The order is stated rather than left to the graph: NetworkX hands back insertion order and + Cypher hands back none, so a caller comparing the two backends would be comparing two + arbitrary orders. ``ref`` is the one total order both can compute. + """ + sig = self.resolve_callable(name, in_class=in_class, in_module=in_module).callable + graph = self.get_call_graph() + if sig not in graph: + return [] + others = graph.predecessors(sig) if callers else graph.successors(sig) + return sorted((self._call_graph_node(other) for other in others), key=lambda n: n.ref) + + # -----[ paths and flow predicates ]----- + #: Up to ``limit`` shortest walks over a ``{src: {dst: [label]}}`` adjacency, in + #: :func:`~cldk.analysis.commons.graphs.hop_sort_key` order -- the shared implementation, bound + #: to TypeScript's ``via`` table. + _shortest_walks = staticmethod(partial(shortest_walks, via=VIA)) + + def paths_between(self, src: str, dst: str, *, src_within: str, dst_within: str, depth: int | None = None, max_paths: int = DEFAULT_MAX_PATHS) -> FlowPaths: + """How a value reaches another value (see :meth:`TSAnalysisBackend.paths_between`).""" + check_depth(depth) + check_max_paths(max_paths) + self._require_dataflow() + a = self.resolve_value(src, within=src_within) + b = self.resolve_value(dst, within=dst_within) + check_distinct_endpoints(a, b) + adjacency, nodes = self._sdg + walks = self._shortest_walks(adjacency["forward"], a.ref, b.ref, depth, max_paths + 1) + described = {ref: self._slice_node(ref) for walk in walks for ref, _ in walk if ref in nodes} + described[a.ref] = a + paths = [flow_path([described[a.ref]] + [described[ref] for ref, _ in walk], [label for _, label in walk], via=VIA) for walk in walks[:max_paths]] + return FlowPaths(paths=paths, complete=len(walks) <= max_paths) + + def call_paths_between(self, src: str, dst: str, *, depth: int | None = None, max_paths: int = DEFAULT_MAX_PATHS) -> FlowPaths: + """How one callable reaches another (see :meth:`TSAnalysisBackend.call_paths_between`). + + Over the callable-only view :meth:`reaches` walks, so the paths cannot disagree with the + boolean that summarises them. + """ + check_depth(depth) + check_max_paths(max_paths) + a_node, b_node = self.resolve_callable(src), self.resolve_callable(dst) + check_distinct_endpoints(a_node, b_node) + a, b = a_node.callable, b_node.callable + graph = self._callable_call_graph + if a not in graph or b not in graph: + return FlowPaths(paths=[], complete=True) + # The call graph re-projected as this module's ``{src: {dst: [label]}}`` adjacency, so one + # walker serves both kinds of path. + edges: Dict[str, Dict[str, list]] = {n: {m: [("TS_CALLS", None, ())] for m in graph.successors(n)} for n in graph} + walks = self._shortest_walks(edges, a, b, depth, max_paths + 1) + described = {key: self._call_graph_node(key) for walk in walks for key, _ in walk} + described[a] = a_node + paths = [flow_path([described[a]] + [described[key] for key, _ in walk], [label for _, label in walk], via=VIA) for walk in walks[:max_paths]] + return FlowPaths(paths=paths, complete=len(walks) <= max_paths) + + def _callee_values(self, signature: str) -> List[str]: + """The ids of every value that *enters* ``signature`` -- in TypeScript, its parameters. The + local twin of the graph's ``formal_in`` body nodes.""" + c = self._callables.get(signature) + return [n.id for n in (c.body or {}).values() if n.kind == "formal_in" and n.id] if c else [] + + def flows_to_call(self, src: str, callee: str, *, within: str, depth: int | None = None) -> bool: + """Does this value reach any argument of a call to ``callee`` + (see :meth:`TSAnalysisBackend.flows_to_call`)?""" + check_depth(depth) + self._require_dataflow() + root = self.resolve_value(src, within=within) + targets = self._callee_values(self.resolve_callable(callee).callable) + return bool(targets) and not self._reach(root.ref, "forward", depth).isdisjoint(set(targets) - {root.ref}) + + def flows_to_argument(self, src: str, callee: str, arg: str, *, within: str, depth: int | None = None) -> bool: + """Does this value reach ``callee``'s ``arg`` + (see :meth:`TSAnalysisBackend.flows_to_argument`)?""" + check_depth(depth) + self._require_dataflow() + root = self.resolve_value(src, within=within) + target = self.resolve_value(arg, within=callee).ref + return target != root.ref and target in self._reach(root.ref, "forward", depth) diff --git a/cldk/analysis/typescript/neo4j/neo4j_backend.py b/cldk/analysis/typescript/neo4j/neo4j_backend.py index 02891c7d..1d3686d3 100644 --- a/cldk/analysis/typescript/neo4j/neo4j_backend.py +++ b/cldk/analysis/typescript/neo4j/neo4j_backend.py @@ -14,61 +14,144 @@ # limitations under the License. ################################################################################ -"""Neo4j-backed TypeScript analysis backend (read-only Cypher client). - -A drop-in alternative to :class:`TSCodeanalyzer`: it exposes the **same query -method surface** (``get_all_classes``, ``get_call_graph``, ``get_all_callers``, -...) so the :class:`TypeScriptAnalysis` facade can delegate to either one, but -every method answers by running **Cypher over a live Neo4j graph** instead of -walking the in-memory pydantic/NetworkX structures. - -This class is purely a **query client**: it never builds the graph and has no -dependency on the ``codeanalyzer-typescript`` binary or the project sources. It -assumes the database is already populated and just polls it — the shape a cloud -deployment wants, where a third-party job (e.g. inside Kubernetes) loads the -graph out of band and the SDK only reads it. - -The graph is the one ``codeanalyzer-typescript`` emits with ``--emit neo4j`` -(schema: ``codeanalyzer-ts/schema.neo4j.json``). Populating it always happens out -of band — never from this backend. - -Identity model (must match the in-memory backend): - -* a callable/class/interface/enum/type-alias is a ``:Symbol`` keyed by ``signature``; -* call-graph edges are ``(:Symbol)-[:CALLS]->(:Symbol|:External)``; -* every project-owned node carries a ``_module`` provenance property, so a single - database may hold several applications — all queries here are scoped to this - backend's application by the set of its module ``file_key``s. - -Parity caveats (inherent to what the projection stores, not bugs): - -* ``CALLS`` edge ``tags`` only round-trip the three keys the projection keeps - (``ts.dispatch`` / ``ts.external`` / ``ts.module``); -* ``get_imports`` / ``get_all_exports`` are reconstructed from the *aggregated* - ``IMPORTS`` / ``RE_EXPORTS`` edges (individual bindings, aliases and positions - are not stored); -* comments collapse to a single docstring, type-parameters keep only their names. +"""Neo4j-backed TypeScript analysis backend (read-only Cypher client) on the codeanalyzer-typescript +1.2.0 graph vocabulary. + +A drop-in alternative to :class:`TSCodeanalyzer`: the same query surface, every method answered by +Cypher over a live graph that ``codeanalyzer-typescript --emit neo4j`` populated out of band. This +class never writes and needs neither the analyzer binary nor the sources. + +**The graph it reads** (``schema.neo4j.json`` at the 1.2.0 tag; ``main`` renames nothing): +``:Application {id: can://typescript/}`` anchors the application and stamps +``analyzer_version``; every project node carries a ``can://`` ``id`` under the merge label +``CanNode`` -- ``TSModule`` (``name`` holds the file key), ``TSClass``/``TSInterface``/``TSEnum``/ +``TSTypeAlias``/``TSNamespace``, ``TSCallable`` (all seven kinds; anonymous ones also +``TSAnonymousCallable``), ``TSField``, ``TSBodyNode``, ``TSExternal``; containment is +``TS_HAS_MODULE`` / ``TS_DECLARES`` / ``TS_HAS_METHOD`` / ``TS_HAS_FIELD``; calls are ``TS_CALLS +{weight, prov}`` and a call site is a ``TSBodyNode {kind:'call'}`` under ``TS_HAS_BODY_NODE`` +resolving over ``TS_RESOLVES_TO``. + +**Scope (TS-3).** A signature is not application-stamped, so every statement that could match +another application's node carries the two-prefix predicate :func:`_scoped` spells -- +``can://typescript//`` and ``can://javascript//`` -- or is keyed by an id that embeds the +application, or walks out from the ``:Application`` anchor. There is no ``_module`` property to +fall back on (retired on ``main``, #166). + +**Seek labels (measured on the superset graph).** What decides the anchor is how narrow the +predicate is, not what shape it has. Statements scoped by the two *application* prefixes -- nearly +every node -- anchor on the specific label alone (``:TSCallable``): 11,085 callables scan in ~8 ms, +and ``:CanNode`` turns the two-prefix predicate into a slower range-seek union (``resolve_callable`` +re-measured this at 24-28 ms bare against 44-51 ms on ``:CanNode``). Id-equality point lookups +anchor on ``:CanNode: