Skip to content

KGX export: literal newlines in node descriptions make the TSV mean two things #723

Open

Description

@realmarcin

Found by claw's shared KGX checker (culturebotai-claw#273), run against this
repository's own output/kgx/.

What

output/kgx/nodes.tsv is 1,007 rows to anything that splits on newlines and
998 to Python's csv module. Nine node description fields contain a literal
newline — some wrapped in quotes, some not — so the two most obvious readers
disagree about the file.

line 63: CommunityMech:000031  biolink:OrganismalEntity  Geobacter-Clostridium…
line 64: MECHANISM CONTESTED - do not read this record as establishing…   ← same record

line 359: ENVO:00000003  biolink:EnvironmentalFeature  mine tailing  "Abandoned…
line 360: "  infores:communitymech                                      ← same record

Why it matters beyond tidiness

The split shifts columns. provided_by lands in the category position, which
is why a Biolink audit of this file reports infores:communitymech as a node
category
on nine nodes (culturebotai-claw#275). There is nothing wrong with the
vocabulary — every biolink: CURIE this repository emits is genuinely in the
Biolink model. The nine are an artefact of the serialisation.

Anything downstream that reads these as TSV rows — cut, awk, pandas with
quoting=QUOTE_NONE, a KGX merge — sees nine malformed records and, depending on
the reader, either drops them or mis-attributes their fields.

Fix

Escape or strip newlines in text fields when writing, or quote consistently.
src/communitymech/export/kgx_export.py writes these; the simplest correct
change is to replace \r and \n with a space (or \\n) in description,
name and supporting_text before writing.

Worth adding a check to validate_kgx.py too, or switching it to the shared
checker:

uv run kg-microbe-kgx check --mech communitymech

which reports exactly this, and which found it here.

Not this repository's problem, but adjacent

The vendored app/kgm/merged-kg_edges.tsv (kg-microbe's merged graph, 1 GB) has
a worse version: 190,506 bare carriage returns and zero CRLF, one of them
inside the header after a duplicated agent_type column. awk reads 11 columns
there; Python reads 7 and silently misaligns everything after it. Raised in
culturebotai-claw#275 — flagging it here only because the copy lives in this
repository.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions