feat(extract): DOCX text extraction with structural locators (RFC 0005)

[teakesdev]

Extends the RFC 0004 extraction slice from PDF-only to DOCX. Stdlib only (zipfile + xml.etree); no python-docx, no lxml.

Do not merge — review gate first: @adversarial-review-deepseek-agent on (1) locator collision/uniqueness, (2) every ZIP/XML failure path actually failing, (3) zero fabricated pagination anywhere.

The hard scope point

DOCX carries no rendered page numbers. Pagination is produced by the renderer at layout time from font metrics, page setup, field results and hyphenation — it is not in the file, and <w:br w:type="page"/> is a hint the content still reflows around. Any page number attached to DOCX content would be one this tool invented.

So none is. Enforced in four places rather than by convention:

extract_docx returns pages == [] in every case, success and failure	matterkit/extract.py
document_blocks has no page_number column, and CHECKs locator_scheme to docx-structural	matterkit/store.py
record_blocks raises on any other scheme — the block store cannot launder a page locator	matterkit/extract.py
matter pages writes no document_pages row for a .docx	matter.py

Storage is a sibling table, not a locator_scheme column on document_pages: with a shared table "never put a paragraph index in page_number" is a rule someone has to keep remembering and one careless INSERT breaks silently; with a separate table there is no column to misuse. Purely additive migration — CREATE TABLE IF NOT EXISTS on next connect(), no backfill, document_pages untouched.

The locator contract

Full text in rfcs/0005-docx-structural-extraction.md §2 and in the extract.py module docstring. A citation is (document_sha256, part, locator):

word/document.xml#b3                    4th block child of the body
word/document.xml#b3/r1/c0/b2           3rd paragraph of row 1, cell 0 of that table
word/document.xml#b0/r0/c0/b1/r2/c1/b0  nested one table deeper

Prefixes cycle b (block child) / r (row) / c (cell), so every path has length 3n+1 and the addressed unit is always a paragraph.

• Uniqueness: a path is a coordinate in a tree — two distinct blocks differ at the first divergent index, so collision within a part is impossible; across parts part separates them, across documents document_sha256 does. PK (document_sha256, part, locator) enforces it and makes re-extraction idempotent.
• Identical text is not identical identity: two paragraphs reading "Same text." share a text_sha256 and have two different locators. The locator names a position; the hash describes content.
• Character offsets, not run indexes — doc.text[b.char_start:b.char_end] == b.text. A w:r boundary is a formatting artefact Word re-splits on any edit, so "run 3 of paragraph 7" names a different substring tomorrow. run_count is kept as a diagnostic and is never part of a citation. (§2d explains the deviation from the brief's sketch.)
• The joiner is \n but is not a delimiter — a w:br renders as \n too, so doc.text must not be re-split to recover blocks; the stored offsets are authoritative.

Honest failure — never a silent empty

Every path below returns extraction-failed with a distinct reason and zero blocks (§3 has the full table with the reason strings):

not-a-zip (magic checked before zipfile, so a renamed .txt/.pdf and an appended-zip polyglot are refused, not re-dispatched) · truncated archive · CRC mismatch · encrypted member · zip bomb (declared size and bounded read both capped) · missing / duplicate / blank word/document.xml · DOCTYPE · non-UTF-8 part · malformed XML with no partial salvage · strict-OOXML namespace · wrong root · missing w:body.

Two that came out of adversarial reading rather than the brief, both verified to work against the un-guarded code first:

• DOCTYPE. xml.etree.ElementTree expands internal general entities (billion laughs) and the CPython C parser exposes no expat handle to disable it. A conforming OOXML part has no DTD, so it is refused — scanned per the XML prolog grammar so document text containing <!DOCTYPE can't trip it, and with a UTF-8 BOM stripped first.
• Non-UTF-8 parts, because that DOCTYPE guard is a byte scan and a UTF-16 part would slip past it.
• Duplicate word/document.xml entries — zipfile resolves to the last, another reader may take the first; which bytes the citation pins to would be reader-dependent, so there is no honest answer.

Three states the store must distinguish, and does: never extracted (no row) · extracted, could not be read (one row, failed, text IS NULL, reason kept) · extracted, genuinely empty (one row, extracted, container_type='part', text='').

What counts as text

w:delText (tracked deletions) and w:instrText (field codes) are never extracted — quoting struck-out or machine-internal text as content misstates the evidence. w:pPr is skipped whole, because w:pPr/w:tabs/w:tab is a tab-stop definition a naive walk would render as a tab that isn't in the document. Element tails are ignored so pretty-printing whitespace never enters the text.

PartInventory records seen / extracted / skipped (headers, footers, footnotes, endnotes, comments) / unhandled (e.g. w:sdt), so "skipped by design" is distinguishable from "missed" — matter pages prints both.

Tests

57 new in tests/test_extract_docx.py, 150 total green under python -m unittest discover -s tests (the CI runner) on 3.11 and 3.14. Every fixture is synthesised in-process from XML strings + zipfile — no binary fixtures, no client documents. The stdlib floor is asserted by a subprocess import with site-packages stripped from sys.path, not by a comment. Source immutability is a re-hash after extraction.

Brief's list (a)–(g) all covered; §8 maps each to its test class.

Named gaps, documented not hidden (§10)

1. The main part is addressed by its conventional name rather than by resolving the officeDocument package relationship — a producer naming it otherwise fails loudly instead of being read.
2. w:sdt content controls aren't descended into (they appear in unhandled); doing so changes the locator grammar and wants its own review.
3. Headers/footers/footnotes need a second part per document — the schema already allows it, the extractor doesn't produce it.
4. Pre-existing: .txt/.md sit in document_pages as "page 1", which is a whole-file locator wearing a page number. Same category of claim; left alone here rather than widened into this RFC.

matter-mcp.py untouched.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Kz5aom58RiiCEzewHdvUec