Skip to content

fix(schema)!: span.bytes are UTF-8 byte offsets, every node and level (#179) - #189

Merged
rahlk merged 1 commit into
mainfrom
fix/issue-179-byte-offsets
Sep 7, 2026
Merged

fix(schema)!: span.bytes are UTF-8 byte offsets, every node and level (#179)#189
rahlk merged 1 commit into
mainfrom
fix/issue-179-byte-offsets

Conversation

@rahlk

@rahlk rahlk commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Closes #179. Spec: docs/design/specs/span-bytes-are-bytes.md (D1–D3, mirrored in .claude/SCHEMA_DECISIONS.md). Analyzer 1.5.0; Neo4j contract stays 2.0.0.

What was wrong

span.bytes was produced (and documented in schema.ts) as UTF-16 char offsets — ts-morph's getStart()/getEnd() — while the canonical keystone and codeanalyzer-python mean UTF-8 bytes (byte_offsets, _span_code = source.encode()[lo:hi]). The projection's spanCode already sliced a Buffer by them, so :TSCallable.code ran short by exactly the multibyte surplus inside the span: python-sdk's sample src/index.main holds one em dash (3 bytes) → 2 bytes short → the closing \n} gone.

What changes

  • span.bytes = UTF-8 byte offsets on every node at every level: declarations, call/config_access (L1), statements and @entry/@exit (L3), the module span ([0, byteLength]), yaml ConfigKey spans. Buffer.from(source).subarray(lo, hi) reproduces the text; source.slice no longer does on a non-ASCII file. Values change only for nodes after a non-ASCII character; ASCII files are byte-identical before and after (-j1/-j4 determinism re-checked).
  • One utility, src/schema/offsets.ts: ASCII fast path is the identity; otherwise one cumulative table per text, cached per owning object (SourceFile / TSModule). Producers convert on the way out; the three consumers that need compiler positions (configUse.nodeAtSpan, the defuse linker's factory lookup, entrypoints/matching default-export resolution) convert on the way in. The dataflow IR keeps char offsets internally.
  • Docs: schema.ts comments, the consumer skill's vocabulary, CLAUDE.md.

Tests

test/byte-offsets.test.ts: a fixture with an em dash, CJK and an emoji before and inside declarations — every callable, field, call site, config_access, L3 statement, @entry, the module span, the yaml key span, and the graph's code slice byte-exact; the config-use literal tier still resolves past multibyte text; offsetMapOf round-trips incl. surrogate pairs. schema-v2.test.ts's slice assertions now use the byte rule. bun test: 339 pass.

Downstream

python-sdk #343 (leg 2.5b): slice span.bytes as bytes on the TypeScript leg too — its python side already does — and the four xfail(strict=True) marks on test_typescript_bulk_parity_live.py flip when the pin moves to the release carrying this.

…#179)

ts-morph positions are UTF-16 code units; the wire said bytes and the graph sliced a Buffer by
them, so :TSCallable.code ran short by the multibyte surplus inside the span. One conversion
utility (schema/offsets.ts): producers convert on the way out (builders, dataflow/attach,
artifacts/yamlKeys); consumers needing compiler positions convert on the way in (configUse,
defuse-linker factory lookup, entrypoint default-export resolution). Values change only for
nodes after a non-ASCII char; ASCII files are byte-identical.
@rahlk
rahlk merged commit b216b75 into main Sep 7, 2026
1 check passed
@rahlk
rahlk deleted the fix/issue-179-byte-offsets branch September 7, 2026 22:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Neo4j projection writes :TSCallable.code one line short of the callable's span

1 participant