fix(schema)!: span.bytes are UTF-8 byte offsets, every node and level (#179)

.claude/SCHEMA_DECISIONS.md

@@ -196,3 +196,13 @@ Spec: `docs/design/specs/neo4j-bindings-parameters-config.md`. Additive on contr
 | D4 | **`parameters_json`** on `:TSCallable` | `JSON.stringify(c.parameters)` verbatim, `null` when empty | python's property and encoding; SDK already decodes it; 1.4 % of graph on cants self; a property on the existing node does not worsen #177 |
 | D5 | **`TS_READS_CONFIG_UNRESOLVED`** | `:TSApplication → :TSExternal \| :TSCallable`, `key`/`reason`/`prov`, `_k = key\|reason`, no `site`; env-root reads ghost under `@external/<root>`, call-rule reads target the resolved callee id | python shape verbatim incl. its documented per-site collapse; retires the #101 "config_reads stay JSON-only" note (python overturned it in #162) |
 | D6 | **Version / tracking** | contract `2.0.0`, analyzer 1.4.0, one PR closing #182 | every addition optional-with-absent; the SDK pins `analyzer_version` |
+
+## `span.bytes` are UTF-8 byte offsets (2026-09-07, #179 — python `byte_offsets` parity)
+
+Spec: `docs/design/specs/span-bytes-are-bytes.md`. Analyzer 1.5.0; Neo4j contract unchanged.
+
+| # | Concept | Decision | Rationale |
+|---|---|---|---|
+| D1 | **`span.bytes` meaning** | UTF-8 byte offsets into `module.source` (and an artifact's `source` for `ConfigKey` spans), every node and level; `Buffer.from(source).subarray(lo, hi)` reproduces the text | the keystone's `module.source[span.bytes]` and python's `byte_offsets`/`_span_code` mean bytes; TS emitted UTF-16 char offsets, so the graph's Buffer-sliced `code` ran short by the multibyte surplus inside the span and one slicing rule could not hold across languages |
+| D2 | **Conversion point** | `src/schema/offsets.ts` (`offsetMapOf`: ASCII identity, else one cumulative table per text, cached per owner). Producers convert on the way out (builders, `dataflow/attach`, `artifacts/yamlKeys`); consumers needing compiler positions convert on the way in (`configUse.nodeAtSpan`, defuse-linker factory lookup, `entrypoints/matching` default export). The dataflow IR keeps char offsets — internal, never on the wire | ts-morph positions stay native inside the analyzer; one definition of the mapping |
+| D3 | **Version** | 1.5.0 minor; a documented field's values move to the documented contract; ASCII files are byte-identical before and after | not a new field or shape |

CLAUDE.md

@@ -22,7 +22,8 @@ table, call graph, CFG, PDG, SDG — is *projection* of that one structure. Anal
 - **L1** (`-a 1`): tree to callable depth — `application → symbol_table{module} →
   types{}/functions{}/fields{} → callables{}` — plus `call` nodes in each callable's
   `body{}` (`callee` unresolved). `source` stored once per module; every node's
-  text slices off it via `span.bytes`.
+  text slices off it via `span.bytes` (UTF-8 BYTE offsets, #179 — `Buffer` slice, never `String.slice`;
+  producers/consumers convert through `src/schema/offsets.ts`).
 - **L2** (`-a 2`): `call_graph` edge list (callable→callable) at application scope,
   and `callee` slot on each call node refined `null → id` (only sanctioned mutation).
 - **L3** (`-a 3`): rest of `body{}` (statements + `@entry`/`@exit`) and intra-callable

docs/design/specs/span-bytes-are-bytes.md

@@ -0,0 +1,36 @@
+# `span.bytes` are UTF-8 byte offsets
+
+Tracking: codeanalyzer-typescript #179. Ships in 1.5.0. Neo4j contract unchanged at 2.0.0; the
+`analysis.json` field keeps its name and shape — its VALUES change for every node that follows a
+non-ASCII character in its file.
+
+## Problem
+
+`span.bytes` was documented (`src/schema/schema.ts`) and produced as UTF-16 char offsets — ts-morph's
+`getStart()`/`getEnd()` — while the canonical keystone (`get_method_body(sig)` →
+`module.source[callable.span.bytes]`) and codeanalyzer-python (`byte_offsets`, `_span_code` doing
+`source.encode("utf-8")[lo:hi]`) mean bytes. The Neo4j projection's `spanCode` already sliced a
+UTF-8 `Buffer` by them, so `:TSCallable.code` ran short by exactly the multibyte surplus inside the
+span: python-sdk's sample `src/index.main` contains one em dash (3 bytes) → 2 bytes short → the
+closing `\n}` missing. A consumer with one slicing rule across languages was wrong on every
+non-ASCII TypeScript file.
+
+## Decision
+
+| # | Concept | Decision | Rationale |
+|---|---|---|---|
+| D1 | **`span.bytes` meaning** | UTF-8 byte offsets into the module's `source`, every node, every level: declarations, body nodes (`call`/`config_access` at L1, statements and `@entry`/`@exit` at L3), the module's own span (`[0, byteLength]`), and artifact `ConfigKey` spans. `Buffer.from(source).subarray(lo, hi)` reproduces the text; `source.slice(lo, hi)` no longer does on a non-ASCII file. | Python parity and the keystone's meaning; one slicing rule per SDK, not one per language. |
+| D2 | **Where conversion happens** | One utility, `src/schema/offsets.ts`: `offsetMapOf(source)` (ASCII fast path = identity; otherwise a cumulative `Uint32Array` built once per text and cached per owning object). Producers convert on the way OUT (builders' `richSpan`, call-site and config-access spans, the module span, `dataflow/attach` for the L3 IR's char offsets, `artifacts/yamlKeys`). Consumers that need a compiler position convert on the way IN (`dataflow/configUse.nodeAtSpan`, `semantic_analysis/defuseLinker` factory lookup, `entrypoints/matching` default-export resolution). The dataflow IR (`schema/graphs.ts` `start_offset`/`end_offset`) stays in char units — internal, never on the wire. | A single definition of the mapping; ts-morph keeps its native positions internally so no compiler call changes. |
+| D3 | **Version** | Analyzer 1.5.0 (minor): a documented field's values move toward the documented contract; Neo4j contract stays 2.0.0. Cache invalidates by analyzer version. | Not a new field or shape; consumers on ASCII files see no change at all. |
+| D4 | **Consumers** | python-sdk's TypeScript leg (2.5, #343) slices bytes as its python side already does; the four `xfail(strict=True)` marks in its parity harness flip on the release. | The SDK's shared `_slice` rule becomes language-neutral. |
+
+## Definition of done
+
+- A fixture with multibyte chars before and inside declarations: for every callable, call-site body
+  node, L3 statement, `@entry`/`@exit`, `config_access`, and the module span, `Buffer` slicing by
+  `span.bytes` reproduces the node's text byte-for-byte; `:TSCallable.code` equals that slice.
+- Existing consumers still resolve: config-use call rules, the defuse linker's factory tier, the
+  Next.js default-export file rule — all exercised by their existing tests on ASCII fixtures plus
+  one non-ASCII case each.
+- `bun test` green, `-j` determinism unchanged, docs (`schema.ts` comments, `vocabulary.md`,
+  `CLAUDE.md`) say bytes.

docs/skills/analyzing-cants-graphs/references/vocabulary.md

@@ -45,7 +45,7 @@ matching rule): `dependency-manifest`, `tool-config`, `container-image`, `servic
 Code nodes are different: neither `TSModule` nor `TSCallable` carries source text, a file path, or
 column positions — only `_module`/`path` (the file key) and `start_line`/`end_line`. To read exact
 code text, re-open the file at those lines, or read `analysis.json`, where every module's `source`
-is stored once and every node's exact text is `source.slice(...span.bytes)`.
+is stored once and every node's exact text is the UTF-8 byte slice `Buffer.from(source).subarray(...span.bytes)` (bytes, not chars — the same rule as codeanalyzer-python's `source.encode()[lo:hi]`).
 
 ### External ghosts (`TSExternal`) — two grains, one label
 

src/artifacts/yamlKeys.ts

@@ -39,18 +39,20 @@ import {
 import { keyNode } from "./configKeys";
 import type { TSConfigKey, TSSpan } from "../schema";
 
+import { offsetMapOf } from "../schema/offsets";
 export function parseYamlKeys(text: string): TSConfigKey[] {
   const lc = new LineCounter();
   const docs = parseAllDocuments(text, { lineCounter: lc, keepSourceTokens: false });
   const bad = docs.find((d) => d.errors.length);
   if (bad) throw bad.errors[0];
   const out: TSConfigKey[] = [];
+  const offsets = offsetMapOf(text); // yaml ranges are char offsets; `span.bytes` are bytes (#179)
   const spanOf = (n: YamlNode): TSSpan | undefined => {
     const r = n.range;
     if (!r) return undefined;
     const s = lc.linePos(r[0]);
     const e = lc.linePos(r[1]);
-    return { start: [s.line, s.col], end: [e.line, e.col], bytes: [r[0], r[1]] };
+    return { start: [s.line, s.col], end: [e.line, e.col], bytes: [offsets.toByte(r[0]), offsets.toByte(r[1])] };
   };
   const multi = docs.length > 1;
   for (const [i, doc] of docs.entries()) {

src/build/neo4j/project.ts

@@ -14,6 +14,7 @@
 import { specifierRoot } from "../../artifacts/binding";
 import type { TSAnalysis, TSApplication, TSBodyNode, TSCallable, TSDecorator, TSEntrypoint, TSEntrypointReport, TSField, TSModule, TSType } from "../../schema";
 import { globalOrdinal, purlNpm } from "../../schema/ids";
+import { sliceBytes } from "../../schema/offsets";
 import { SCHEMA_VERSION } from "./schema";
 import { type GraphRows, type NodeRef, type Props, RowBuilder, prune } from "./rows";
 
@@ -350,7 +351,7 @@ function spanCode(source: string, sp: { bytes?: [number, number] } | undefined):
   if (!source || !bytes) return null;
   const [lo, hi] = bytes;
   if (hi <= lo) return null;
-  return Buffer.from(source, "utf8").subarray(lo, hi).toString("utf8");
+  return sliceBytes(source, bytes); // #179: `span.bytes` are real byte offsets now, so this is exact
 }
 
 function moduleProps(mod: TSModule, fileKey: string): Props {

src/dataflow/attach.ts

@@ -18,20 +18,23 @@
  */
 
 import type { CfgEdge, FunctionGraphs, GraphNode, PdgEdge, ProgramGraphs } from "../schema/graphs";
-import type { TSApplication, TSCallable, TSParamEdge } from "../schema";
+import { type TSApplication, type TSCallable, type TSModule, type TSParamEdge, forEachCallable } from "../schema";
 import { globalOrdinal, stampBodyIds } from "../schema/ids";
+import type { OffsetMap } from "../schema/offsets";
 
+import { offsetMapFor } from "../schema/offsets";
 interface LocalIds {
   canId: string;
   callable: TSCallable;
   stmtLocal: Map<number, string>; // entry/exit/statement node id → body local id
   paramN: Map<number, number>; // param node id → declaration index N
   paramName: Map<number, string>; // param node id → the `of` name
   exitId: number;
+  offsets: OffsetMap; // the owning module's char→byte map: IR offsets are chars, `span.bytes` are bytes (#179)
 }
 
 /** Build the per-callable node_id→local-id maps (single source-of-truth for every edge rewrite). */
-function buildLocalIds(canId: string, callable: TSCallable, nodes: GraphNode[]): LocalIds {
+function buildLocalIds(canId: string, callable: TSCallable, nodes: GraphNode[], offsets: OffsetMap): LocalIds {
   const stmtLocal = new Map<number, string>();
   const paramN = new Map<number, number>();
   const paramName = new Map<number, string>();
@@ -55,7 +58,7 @@ function buildLocalIds(canId: string, callable: TSCallable, nodes: GraphNode[]):
       stmtLocal.set(n.id, key);
     }
   }
-  return { canId, callable, stmtLocal, paramN, paramName, exitId };
+  return { canId, callable, stmtLocal, paramN, paramName, exitId, offsets };
 }
 
 /** L3/CDG/DDG node resolution: a param folds onto `@entry` (params are defined at entry at L3). */
@@ -73,8 +76,9 @@ function fq(callableId: string, bodyKey: string): string {
   return globalOrdinal(callableId, bodyKey); // single definition lives in schema/ids.ts (#164)
 }
 
-function spanOf(n: GraphNode): { start: [number, number]; end: [number, number]; bytes: [number, number] } {
-  return { start: [n.start_line, n.start_column], end: [n.end_line, n.end_column], bytes: [n.start_offset, n.end_offset] };
+/** The IR's char offsets become the wire's UTF-8 byte offsets here (#179). */
+function spanOf(n: GraphNode, offsets: OffsetMap): { start: [number, number]; end: [number, number]; bytes: [number, number] } {
+  return { start: [n.start_line, n.start_column], end: [n.end_line, n.end_column], bytes: [offsets.toByte(n.start_offset), offsets.toByte(n.end_offset)] };
 }
 
 // ----------------------------------------------------------------------------------------------
@@ -88,7 +92,7 @@ function emitL3(li: LocalIds, nodes: GraphNode[], cfgEdges: CfgEdge[] | undefine
     if (n.kind === "param") continue;
     const key = li.stmtLocal.get(n.id) as string;
     if (key in c.body) continue; // keep the richer L1 `call` node on a position collision
-    c.body[key] = n.kind === "statement" ? { kind: "statement", span: spanOf(n) } : { kind: n.kind, span: spanOf(n) };
+    c.body[key] = n.kind === "statement" ? { kind: "statement", span: spanOf(n, li.offsets) } : { kind: n.kind, span: spanOf(n, li.offsets) };
   }
 
   if (cfgEdges) {
@@ -290,12 +294,17 @@ export function applyDataflow(
 ): void {
   if (level < 3) return;
 
+  // Each callable's owning module, for the char→byte map its body-node spans need (#179).
+  const moduleOf = new Map<TSCallable, TSModule>();
+  for (const mod of Object.values(root.symbol_table)) forEachCallable(mod, (c) => moduleOf.set(c, mod));
+
   const info = new Map<string, LocalIds>();
   for (const [sig, fg] of Object.entries(pg.functions)) {
     const callable = callableBySig.get(sig);
     const canId = idBySig.get(sig);
-    if (!callable || !canId || !fg.cfg) continue;
-    info.set(sig, buildLocalIds(canId, callable, fg.cfg.nodes));
+    const mod = callable && moduleOf.get(callable);
+    if (!callable || !canId || !fg.cfg || !mod) continue;
+    info.set(sig, buildLocalIds(canId, callable, fg.cfg.nodes, offsetMapFor(mod, mod.source)));
   }
 
   for (const [sig, fg] of Object.entries(pg.functions)) {

src/dataflow/configUse.ts

@@ -26,6 +26,7 @@ import { symbolAt } from "../schema/checker";
 import { type ConfigUseSets, keyIndex } from "../semantic_analysis/configUse";
 import { ACCESS_RULES } from "../semantic_analysis/configUseRules";
 
+import { offsetMapFor } from "../schema/offsets";
 /**
  * `project` gives the AST the tiers read. A no-op below `-a 3` (the literal tier alone stands).
  * Deterministic: `uses` stays sorted.
@@ -101,8 +102,14 @@ function findCallable(app: AnalysisInternal, pred: (c: TSCallable) => boolean):
 }
 
 /** The AST node whose exact byte span is [start, end) in `absPath`, or undefined. */
-function nodeAtSpan(project: Project, absPath: string, start: number, end: number): Node | undefined {
-  let n = project.getSourceFile(absPath)?.getDescendantAtPos(start);
+function nodeAtSpan(project: Project, absPath: string, startByte: number, endByte: number): Node | undefined {
+  const sf = project.getSourceFile(absPath);
+  if (!sf) return undefined;
+  // `span.bytes` are UTF-8 byte offsets on the wire (#179); ts-morph positions are char units.
+  const m = offsetMapFor(sf, sf.getFullText());
+  const start = m.toChar(startByte);
+  const end = m.toChar(endByte);
+  let n = sf.getDescendantAtPos(start);
   while (n && (n.getStart() !== start || n.getEnd() !== end)) n = n.getParent();
   return n;
 }

src/entrypoints/matching.ts

@@ -10,6 +10,7 @@ import { forEachCallable, type AnalysisInternal, type TSCallable, type TSCallsit
 import { callBodyKeys } from "../schema/l1Body";
 import type { ArgSpec, BaseRule, CallRule, DecoratorRule, FileRule, ManifestRule } from "./rules";
 
+import { offsetMapFor, sliceBytes } from "../schema/offsets";
 export class PatternError extends Error {}
 
 const cache = new Map<string, RegExp>();
@@ -313,8 +314,9 @@ function resolveDefaultExport(mod: TSModule): TSCallable | undefined {
   }
   DEFAULT_EXPORT_TOKEN.lastIndex = 0;
   let m: RegExpExecArray | null;
+  const offsets = offsetMapFor(mod, mod.source); // regex indices are chars; `span.bytes` are bytes (#179)
   while ((m = DEFAULT_EXPORT_TOKEN.exec(mod.source))) {
-    const end = m.index + m[0].length;
+    const end = offsets.toByte(m.index + m[0].length);
     const target = Object.values(mod.functions).find((c) => c.name === "(anonymous)" && c.span.bytes[0] === end);
     if (target) return target;
   }
@@ -343,7 +345,7 @@ export function entrypointsFromFiles(
     if (!globToRegExp(rule.match).test(fileKey)) continue;
     for (const exp of rule.exports) {
       let target = Object.values(mod.functions).find((c) => c.is_exported && (exp === "default"
-        ? mod.source.slice(c.span.bytes[0], c.span.bytes[1]).trimStart().startsWith("export default")
+        ? sliceBytes(mod.source, c.span.bytes).trimStart().startsWith("export default")
         : c.name === exp));
       if (!target && exp === "default") target = resolveDefaultExport(mod);
       if (!target) {

src/schema/graphs.ts

@@ -34,7 +34,8 @@ export interface GraphNode {
   start_column: number;
   end_line: number;
   end_column: number;
-  /** UTF-16 char offsets into the owning module's `source` (same convention as L1 `span.bytes`). */
+  /** UTF-16 char offsets (ts-morph positions) — INTERNAL; `dataflow/attach` converts them to the
+   * UTF-8 byte offsets `span.bytes` carries on the wire (#179). */
   start_offset: number;
   end_offset: number;
 }

src/schema/offsets.ts

@@ -0,0 +1,66 @@
+/**
+ * Char ↔ UTF-8 byte offset conversion for one source text (#179).
+ *
+ * `span.bytes` is a UTF-8 BYTE range into the owning module's `source` — the canonical keystone's
+ * meaning and codeanalyzer-python's (`byte_offsets`), so one slicing rule holds across languages:
+ * `Buffer.from(source, "utf8").subarray(lo, hi)`. ts-morph and the yaml parser position nodes in
+ * UTF-16 code units, so every producer converts on the way out and every consumer that needs a
+ * compiler position converts on the way back in. Both directions go through here.
+ *
+ * ASCII fast path: when the text's byte length equals its char length, every offset is its own
+ * byte offset and no table is built. Otherwise a cumulative table (one entry per char) is built
+ * once per text and cached on the object that owns it (a SourceFile, a TSModule, an artifact) —
+ * strings cannot key a WeakMap, so the caller hands in the owner.
+ */
+
+export interface OffsetMap {
+  toByte(charPos: number): number;
+  toChar(bytePos: number): number;
+}
+
+const IDENTITY: OffsetMap = { toByte: (c) => c, toChar: (b) => b };
+
+export function offsetMapOf(source: string): OffsetMap {
+  if (Buffer.byteLength(source, "utf8") === source.length) return IDENTITY;
+  // byteAt[i] = byte offset of char i; byteAt[n] = total byte length. Surrogate pairs: both code
+  // units map to the pair's START and the pair advances 4 bytes — ts-morph never points between
+  // the two, and a byte inside the pair maps back to its high surrogate.
+  const n = source.length;
+  const byteAt = new Uint32Array(n + 1);
+  let b = 0;
+  for (let i = 0; i < n; i++) {
+    byteAt[i] = b;
+    const cu = source.charCodeAt(i);
+    if (cu < 0x80) b += 1;
+    else if (cu < 0x800) b += 2;
+    else if (cu >= 0xd800 && cu <= 0xdbff && i + 1 < n) { byteAt[i + 1] = b; b += 4; i++; }
+    else b += 3;
+  }
+  byteAt[n] = b;
+  return {
+    toByte: (c) => byteAt[Math.min(Math.max(c, 0), n)] as number,
+    toChar: (bytePos) => {
+      // binary search for the first char whose byte offset >= bytePos
+      let lo = 0, hi = n;
+      while (lo < hi) {
+        const mid = (lo + hi) >>> 1;
+        if ((byteAt[mid] as number) < bytePos) lo = mid + 1;
+        else hi = mid;
+      }
+      return lo;
+    },
+  };
+}
+
+/** Per-owner cache: the same text is mapped once per run. */
+const cache = new WeakMap<object, OffsetMap>();
+export function offsetMapFor(owner: object, source: string): OffsetMap {
+  let m = cache.get(owner);
+  if (!m) cache.set(owner, (m = offsetMapOf(source)));
+  return m;
+}
+
+/** The node's text — the one slicing rule every consumer of `span.bytes` uses. */
+export function sliceBytes(source: string, bytes: [number, number]): string {
+  return Buffer.from(source, "utf8").subarray(bytes[0], bytes[1]).toString("utf8");
+}