You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
KGX export: literal newlines in node descriptions make the TSV mean two things #723
Found by claw's shared KGX checker (culturebotai-claw#273), run against this
repository's own output/kgx/.
What
output/kgx/nodes.tsv is 1,007 rows to anything that splits on newlines and 998 to Python's csv module. Nine node description fields contain a literal
newline — some wrapped in quotes, some not — so the two most obvious readers
disagree about the file.
line 63: CommunityMech:000031 biolink:OrganismalEntity Geobacter-Clostridium…
line 64: MECHANISM CONTESTED - do not read this record as establishing… ← same record
line 359: ENVO:00000003 biolink:EnvironmentalFeature mine tailing "Abandoned…
line 360: " infores:communitymech ← same record
Why it matters beyond tidiness
The split shifts columns. provided_by lands in the category position, which
is why a Biolink audit of this file reports infores:communitymech as a node
category on nine nodes (culturebotai-claw#275). There is nothing wrong with the
vocabulary — every biolink: CURIE this repository emits is genuinely in the
Biolink model. The nine are an artefact of the serialisation.
Anything downstream that reads these as TSV rows — cut, awk, pandas with quoting=QUOTE_NONE, a KGX merge — sees nine malformed records and, depending on
the reader, either drops them or mis-attributes their fields.
Fix
Escape or strip newlines in text fields when writing, or quote consistently. src/communitymech/export/kgx_export.py writes these; the simplest correct
change is to replace \r and \n with a space (or \\n) in description, name and supporting_text before writing.
Worth adding a check to validate_kgx.py too, or switching it to the shared
checker:
uv run kg-microbe-kgx check --mech communitymech
which reports exactly this, and which found it here.
Not this repository's problem, but adjacent
The vendored app/kgm/merged-kg_edges.tsv (kg-microbe's merged graph, 1 GB) has
a worse version: 190,506 bare carriage returns and zero CRLF, one of them
inside the header after a duplicated agent_type column. awk reads 11 columns
there; Python reads 7 and silently misaligns everything after it. Raised in
culturebotai-claw#275 — flagging it here only because the copy lives in this
repository.
Activity