Skip to content

fix(neo4j)!: scope every destructive statement on the can:// id prefix; retire _module #166

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rahlk merged 1 commit into from
Sep 6, 2026
Merged

fix(neo4j)!: scope every destructive statement on the can:// id prefix; retire _module #166

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .claude/SCHEMA_DECISIONS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,3 +163,13 @@ an unresolved read is `reason: "non-literal"` (key never closes on one literal)
| I2 | **`TSCallableParameter.id`** = `<callable-id>@formal_in:<i>`, present at EVERY level | a forward reference below L4 to the vertex that carries the parameter; at L4 `body["@formal_in:i"].id === parameters[i].id` and `.of === parameters[i].name` | python emits it at every level for the same reason; consumers key parameter flow on it before L4 exists |
| I3 | **One definition**: `globalOrdinal` in `src/schema/ids.ts`; `project.ts` and `attach.ts` delegate | the rule used to live in two private `fq` copies | two copies of a join key drift; the agreement test pins JSON `id` == graph merge key |
| I4 | **`SCHEMA_VERSION` unmoved** (2.0.0) | additive fields; no label/relationship/property change in the graph | #144: one version until every analyzer re-baselines together |

## Prefix-scoped destructive statements (2026-09-06, #140 — org spec `2026-09-02-prune-scope-on-can-id-prefix.md`)

| # | Concept | Decision | Rationale |
|---|---|---|---|
| P1 | **Every destructive statement scopes on the `can://` id prefix**: the node by equality, descendants by `id + '/'` | `--eager` purge, per-module purge, orphan prune (bolt) and the snapshot wipe (cypher) | `_module` was application-blind: two apps sharing a file key deleted each other's nodes. A bare `STARTS WITH id` is wrong too — it also matches `…/foo.tsx` under `…/foo.ts`, and `appXtra` under `app` |
| P2 | **`_module` retired from the graph**; `NodeRow.module` keeps the grouping in memory | the incremental diff is keyed by module id inside the app prefixes | the property carried no scope; the id carries language, app and file |
| P3 | **Markers `TSCanNode` / `JSCanNode`** on every `can://<lang>/` id, with a range index on `id` each | index anchors only; anchor label chosen from the id's own namespace | property indexes are label-scoped; `STARTS WITH` seeks only on a range index. Two markers because this analyzer emits two namespaces. `CanNode` stays until #95 |
| P4 | **Empty application refused** (`applicationPrefixes` throws) | on any push: the diff itself is app-scoped | `STARTS WITH ''` matches the whole store |
| P5 | **`SCHEMA_VERSION` stays 2.0.0** despite a removed property | supersedes #140's "MAJOR bump" goal | #144 / python #186: one version until every analyzer re-baselines together |
19 changes: 6 additions & 13 deletions schema.neo4j.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
{
"schema_version": "2.0.0",
"generator": "codeanalyzer-typescript",
"marker_labels": [],
"marker_labels": [
"TSCanNode",
"JSCanNode"
],
"node_labels": [
{
"label": "TSApplication",
Expand Down Expand Up @@ -73,7 +76,6 @@
"properties": {
"id": "string",
"kind": "string",
"_module": "string",
"name": "string",
"is_tsx": "boolean",
"is_declaration_file": "boolean",
Expand All @@ -89,7 +91,6 @@
"properties": {
"id": "string",
"kind": "string",
"_module": "string",
"signature": "string",
"name": "string",
"base_classes": "string[]",
Expand All @@ -111,7 +112,6 @@
"properties": {
"id": "string",
"kind": "string",
"_module": "string",
"signature": "string",
"name": "string",
"base_classes": "string[]",
Expand All @@ -129,7 +129,6 @@
"properties": {
"id": "string",
"kind": "string",
"_module": "string",
"signature": "string",
"name": "string",
"is_const": "boolean",
Expand All @@ -147,7 +146,6 @@
"properties": {
"id": "string",
"kind": "string",
"_module": "string",
"signature": "string",
"name": "string",
"aliased_type": "string",
Expand All @@ -165,7 +163,6 @@
"properties": {
"id": "string",
"kind": "string",
"_module": "string",
"signature": "string",
"name": "string",
"is_exported": "boolean",
Expand All @@ -182,7 +179,6 @@
"properties": {
"id": "string",
"kind": "string",
"_module": "string",
"signature": "string",
"name": "string",
"return_type": "string",
Expand Down Expand Up @@ -210,7 +206,6 @@
"properties": {
"id": "string",
"kind": "string",
"_module": "string",
"name": "string",
"type": "string",
"start_line": "integer",
Expand All @@ -224,7 +219,6 @@
"properties": {
"id": "string",
"kind": "string",
"_module": "string",
"of": "string",
"parent": "string",
"callee": "string",
Expand All @@ -239,7 +233,6 @@
"properties": {
"id": "string",
"kind": "string",
"_module": "string",
"name": "string",
"module": "string"
}
Expand All @@ -251,7 +244,6 @@
"properties": {
"id": "string",
"kind": "string",
"_module": "string",
"signature": "string",
"name": "string",
"return_type": "string",
Expand Down Expand Up @@ -573,6 +565,7 @@
"CREATE INDEX callable_name IF NOT EXISTS FOR (c:TSCallable) ON (c.name)",
"CREATE FULLTEXT INDEX ts_code_fts IF NOT EXISTS FOR (c:TSCallable) ON EACH [c.code]",
"CREATE INDEX cannode_kind IF NOT EXISTS FOR (n:CanNode) ON (n.kind)",
"CREATE INDEX cannode_module IF NOT EXISTS FOR (n:CanNode) ON (n._module)"
"CREATE INDEX tscannode_id IF NOT EXISTS FOR (n:TSCanNode) ON (n.id)",
"CREATE INDEX jscannode_id IF NOT EXISTS FOR (n:JSCanNode) ON (n.id)"
]
}
104 changes: 74 additions & 30 deletions src/build/neo4j/bolt.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@

import type { Logger } from "../../utils";
import type { EdgeRow, GraphRows, NodeRow, Prop } from "./rows";
import { chunk } from "./rows";
import { JS_MARKER, TS_MARKER, applicationPrefixes, chunk, descendantPrefix, markerFor } from "./rows";
import { CONSTRAINTS, INDEXES, SCHEMA_VERSION } from "./schema";

export interface BoltConfig {
Expand All @@ -30,7 +30,6 @@ export interface BoltConfig {
database: string | null;
}

const DESCENDANTS = "[:TS_DECLARES|TS_HAS_METHOD|TS_HAS_FIELD|TS_HAS_BODY_NODE*1..]";
const BATCH = 1000;

/** #68: a DB written by a different schema version must be fully re-upserted, not hash-diffed —
Expand All @@ -52,9 +51,36 @@ export function shouldForceFullUpsert(dbVersion: string | null, producerVersion:
* Batched, because deleting a whole application in one transaction exhausts
* `dbms.memory.transaction.total.max` on a modestly-sized server (#116, measured at 2.7 GiB).
*/
export const EAGER_PURGE =
"MATCH (n:CanNode) WHERE n.id STARTS WITH $prefix " +
"CALL { WITH n DETACH DELETE n } IN TRANSACTIONS OF 5000 ROWS";
/**
* `--eager` purge (#140): everything under this application's prefix, per language namespace,
* anchored on that namespace's marker so the prefix predicate seeks an index. `$prefix` is the
* `/`-terminated descendant prefix from `applicationPrefixes`, never a bare app id — a bare id also
* matches `can://typescript/appXtra/...`.
*/
export const EAGER_PURGE = eagerPurge(TS_MARKER);
export const EAGER_PURGE_JS = eagerPurge(JS_MARKER);
function eagerPurge(marker: string): string {
return `MATCH (n:${marker}) WHERE n.id STARTS WITH $prefix CALL { WITH n DETACH DELETE n } IN TRANSACTIONS OF 5000 ROWS`;
}
/** The per-module purge (#140): the module by equality, its subtree by `/`-prefix; `$keys` survive. */
function purgeModuleEdges(marker: string): string {
return `MATCH (x:${marker}) WHERE x.id = $mid OR x.id STARTS WITH $pre MATCH (x)-[r]->() DELETE r`;
}
function purgeModuleStale(marker: string): string {
return `MATCH (x:${marker}) WHERE (x.id = $mid OR x.id STARTS WITH $pre) AND NOT x.id IN $keys DETACH DELETE x`;
}
/** The orphan prune (#140): modules inside this app's prefix that the run no longer emits, with their subtrees. */
function pruneVanished(marker: string): string {
return (
`MATCH (m:TSModule:${marker}) WHERE m.id STARTS WITH $prefix AND NOT m.id IN $present ` +
`CALL { WITH m MATCH (x:${marker}) WHERE x.id = m.id OR x.id STARTS WITH m.id + '/' DETACH DELETE x } ` +
`IN TRANSACTIONS OF 1000 ROWS RETURN count(DISTINCT m) AS pruned`
);
}
/** The TSModule row of one module's group carries the module's `can://` id and its content_hash. */
function moduleRow(nodes: NodeRow[]): NodeRow | undefined {
return nodes.find((n) => n.labels.includes("TSModule"));
}

export async function boltWriter(
rows: GraphRows,
Expand All @@ -80,7 +106,7 @@ export async function boltWriter(
const shared: NodeRow[] = [];
const moduleOf = new Map<string, string>(); // node value → owning module
for (const n of rows.nodes) {
const m = n.props._module;
const m = n.module;
if (typeof m === "string") {
bucket(byModule, m).push(n);
moduleOf.set(n.value, m);
Expand All @@ -93,6 +119,10 @@ export async function boltWriter(
// app's :Application (by id) — an unscoped `MATCH (a:Application)` could read a foreign analyzer's
// node in a shared database and misjudge the version. Absent id → null → forces (safe default).
const appId = rows.nodes.find((n) => n.labels[0] === "Application")?.value ?? null;
// Every scoped statement below — the diff, the purges, the prune — needs the application's
// prefixes, and applicationPrefixes refuses an empty one (#140). project() always emits the
// Application row, so this only trips on a hand-built GraphRows.
const prefixes = applicationPrefixes(appId);
let dbSchemaVersion: string | null = null;
if (appId !== null) {
await withSession(session, async (s) => {
Expand All @@ -114,21 +144,35 @@ export async function boltWriter(

// --eager: drop this application's own nodes and rebuild. Without it the push only ever adds
// and updates -- managing the database's lifetime is the operator's call, not the analyzer's.
if (eager && appId !== null) {
await withSession(session, (s) => s.run(EAGER_PURGE, { prefix: appId }));
log.info(`neo4j(bolt): --eager, purged the existing graph for ${appId}`);
if (eager) {
await withSession(session, async (s) => {
await s.run(EAGER_PURGE, { prefix: prefixes.ts });
await s.run(EAGER_PURGE_JS, { prefix: prefixes.js });
});
log.info(`neo4j(bolt): --eager, purged the existing graph under ${prefixes.ts} and ${prefixes.js}`);
}

// 3. diff content_hash.
const dbHash = new Map<string, string | null>();
await withSession(session, async (s) => {
const res = await s.run("MATCH (m:TSModule) RETURN m._module AS k, m.content_hash AS h");
// Keyed by module ID inside this application's prefixes (#140): a file key alone collides
// across applications; the id carries language, application and file.
const res = await s.run(
"MATCH (m:TSModule) WHERE m.id STARTS WITH $ts OR m.id STARTS WITH $js RETURN m.id AS k, m.content_hash AS h",
{ ts: prefixes.ts, js: prefixes.js },
);
for (const rec of res.records) dbHash.set(rec.get("k"), rec.get("h"));
});
const moduleIdOf = new Map<string, string>(); // file key → the module's can:// id
const changed = new Set<string>();
for (const [m, nodes] of byModule) {
const mid = moduleRow(nodes)?.value;
if (!mid || !(mid.startsWith(prefixes.ts) || mid.startsWith(prefixes.js))) {
throw new Error(`neo4j: module ${m} has no can:// id under ${prefixes.ts} / ${prefixes.js}; refusing to scope a purge on it`);
}
moduleIdOf.set(m, mid);
const rowHash = hashOf(nodes, m);
if (forceAll || !dbHash.has(m) || rowHash === undefined || rowHash !== dbHash.get(m)) changed.add(m);
if (forceAll || !dbHash.has(mid) || rowHash === undefined || rowHash !== dbHash.get(mid)) changed.add(m);
}
log.info(
`neo4j(bolt): ${byModule.size} modules (${changed.size} changed), ${shared.length} shared nodes, ` +
Expand All @@ -147,13 +191,15 @@ export async function boltWriter(
// operator's call (#116). Anchored on :CanNode either way, so a sibling analyzer's nodes
// sharing this `_module` key are never in scope.
if (eager) {
// The module by equality, its subtree by `/`-prefix, anchored on the module's own
// language marker (#140). Application-scoped by construction: the id carries the app.
const mid = moduleIdOf.get(m)!;
const marker = markerFor(mid)!;
const params = { mid, pre: descendantPrefix(mid), keys };
await withSession(session, async (s) => {
await s.executeWrite(async (tx: any) => {
await tx.run(`MATCH (x:CanNode {_module: $m})-[r]->() DELETE r`, { m });
await tx.run(
`MATCH (x:CanNode {_module: $m}) WHERE x.id IS NULL OR NOT x.id IN $keys DETACH DELETE x`,
{ m, keys },
);
await tx.run(purgeModuleEdges(marker), params);
await tx.run(purgeModuleStale(marker), params);
});
});
}
Expand All @@ -169,20 +215,18 @@ export async function boltWriter(

// 7. orphan prune — only safe on a full run (a targeted run can't tell deleted from untargeted).
// appId === null would make `STARTS WITH ""` match every node in the store.
if (fullRun && eager && appId !== null) {
const present = [...byModule.keys()];
if (fullRun && eager) {
// Scoped on this app's prefixes and the module's own marker (#140); a second TypeScript app
// in the same database, whose modules are all "not in $present", is outside the prefix.
const present = [...moduleIdOf.values()];
let pruned = 0;
await withSession(session, async (s) => {
// Anchored on :CanNode AND this app's id prefix, same as EAGER_PURGE. `MATCH (m:TSModule)`
// alone would reach a SECOND TypeScript application in the same database -- every one of
// its modules is "not in this app's $present" -- and any 1.x twin-labelled node too (#116).
const res = await s.run(
`MATCH (m:TSModule:CanNode) WHERE m.id STARTS WITH $prefix AND NOT m._module IN $present ` +
`OPTIONAL MATCH (m)-${DESCENDANTS}->(x) DETACH DELETE x, m RETURN count(DISTINCT m) AS pruned`,
{ present, prefix: appId },
);
const pruned = res.records[0]?.get("pruned") ?? 0;
log.info(`neo4j(bolt): pruned ${pruned} vanished module(s)`);
for (const [marker, prefix] of [[TS_MARKER, prefixes.ts], [JS_MARKER, prefixes.js]] as const) {
const res = await s.run(pruneVanished(marker), { present, prefix });
pruned += Number(res.records[0]?.get("pruned") ?? 0);
}
});
log.info(`neo4j(bolt): pruned ${pruned} vanished module(s)`);
} else {
log.info("neo4j(bolt): orphan pruning skipped (use --eager to remove vanished modules)");
}
Expand Down Expand Up @@ -257,8 +301,8 @@ function bucket<K, V>(map: Map<K, V[]>, key: K): V[] {
}

function hashOf(nodes: NodeRow[], _fileKey: string): string | undefined {
// Every node in `nodes` shares the same _module; the Module row (labels include "TSModule") carries the hash.
const mod = nodes.find((n) => n.labels.includes("TSModule"));
// Every node in `nodes` shares one owning module; its TSModule row carries the hash.
const mod = moduleRow(nodes);
const h = mod?.props.content_hash;
return typeof h === "string" ? h : undefined;
}
Expand Down
24 changes: 17 additions & 7 deletions src/build/neo4j/cypher.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@

import * as fs from "node:fs";
import type { EdgeRow, GraphRows, NodeRow, Props } from "./rows";
import { JS_MARKER, TS_CAN_PREFIX, TS_MARKER, applicationPrefixes } from "./rows";
import { cypherMap, cypherValue } from "./rows";
import { CONSTRAINTS, INDEXES } from "./schema";

Expand Down Expand Up @@ -38,7 +39,7 @@ function* cypherBlocks(rows: GraphRows, appId: string): Generator<string> {

yield "";
yield "// ── wipe this project's prior subgraph (external targets are shared) ──";
yield wipe(appId);
yield wipe(rows, appId);

yield "";
yield "// ── nodes ──";
Expand All @@ -50,13 +51,22 @@ function* cypherBlocks(rows: GraphRows, appId: string): Generator<string> {
yield "";
}

function wipe(appId: string): string {
const id = cypherValue(appId);
function wipe(rows: GraphRows, appIdArg: string): string {
// Scoped on the `can://` id prefix per namespace (#140), not on a relationship walk from the
// Application node: the prefix reaches every node the app owns — including ones a walk would
// miss — and nothing another app owns, even one whose file keys collide. The id comes from the
// rows' own Application node (the argument is a fallback for callers that pass the bare name);
// rows with no application id get NO destructive statement — refused visibly, never `STARTS
// WITH ''`.
const appId = rows.nodes.find((n) => n.labels[0] === "Application")?.value ?? appIdArg;
if (!appId.startsWith(TS_CAN_PREFIX)) {
return "// no can:// application id in these rows — no wipe emitted (#140 refuses an unscoped delete)";
}
const { ts, js } = applicationPrefixes(appId);
return [
`MATCH (a:Application {id: ${id}})`,
"OPTIONAL MATCH (a)-[:TS_HAS_MODULE]->(m:TSModule)",
"OPTIONAL MATCH (m)-[:TS_DECLARES|TS_HAS_METHOD|TS_HAS_FIELD|TS_HAS_BODY_NODE*1..]->(x)",
"DETACH DELETE x, m, a;",
`MATCH (x:${TS_MARKER}) WHERE x.id STARTS WITH ${cypherValue(ts)} DETACH DELETE x;`,
`MATCH (x:${JS_MARKER}) WHERE x.id STARTS WITH ${cypherValue(js)} DETACH DELETE x;`,
`MATCH (a:Application {id: ${cypherValue(appId)}}) DETACH DELETE a;`,
].join("\n");
}

Expand Down
3 changes: 2 additions & 1 deletion src/build/neo4j/project.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@
* No I/O: the writers (cypher snapshot / bolt incremental) consume the returned `GraphRows`.
*
* The graph is a second projection of the SAME v2 envelope the JSON path emits (finalizeAnalysis),
* so JSON and graph never diverge. Every project-owned node carries `_module` (its owning file key,
* so JSON and graph never diverge. Every project-owned node passes `_module` (its owning file key)
* to the RowBuilder, which lifts it OFF the graph into NodeRow.module for the incremental diff (#140);
* for the incremental writer's per-module isolation); shared nodes (External) carry none.
*/

Expand Down
Loading
Loading