Skip to content

span.bytes are declared as UTF-16 char offsets but the Neo4j projection slices code as UTF-8 bytes #174

Description

@rahlk

Observation

src/schema/schema.ts:29-31 declares TSSpan.bytes as UTF-16 character offsets into TSModule.source, and the level-3 builder produces them that way (src/schema/graphs.ts:38). The Neo4j projection derives each declaration's code property by slicing module.source with those offsets as if they were UTF-8 byte offsets (src/build/neo4j/project.ts:274-280).

The two agree on ASCII and disagree on any file containing a non-ASCII character before the declaration — the slice is shifted by the cumulative multi-byte surplus, so code on the graph starts or ends mid-token.

Why it matters

  • A consumer that slices source by span.bytes (as declared) and one that reads code off the graph get different text for the same callable — exactly the kind of cross-backend disagreement the SDK's parity tests exist to catch, and it would show up only on a corpus with non-ASCII source (comments and string literals are enough).
  • schema.neo4j.json documents code without saying which offset semantics produced it.

Suggestion

Pick one and make both sides use it: either keep bytes as character offsets and slice source by characters in the projection, or rename/redefine the field as byte offsets (a schema-major change) and fix the builder. The first is the smaller change. Add one fixture file with a non-ASCII comment above a declaration and assert the graph's code equals the character slice.

Definition of done

  • code on :TSCallable equals module.source.slice(span.bytes[0], span.bytes[1]) (as characters) for a declaration preceded by non-ASCII text.
  • The schema doc states the offset semantics once.

Found while inventorying the 1.2.0 wire schema for python-sdk leg 2.5.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions