docs: take over #678's embedding-API spec, reconciled with the tree (supersedes #678)

[dpsiderius]

What this is

#678's spec, landed off current main. This supersedes #678 — close that
one when this merges; do not merge both.

Jacob's three commits are cherry-picked verbatim (same content, same
messages) and keep their authorship: GitHub shows
Jacob Verhoeks as author, me as committer, and he lands in the
contributor list. Everything else is one reconciliation commit of mine.

⚠️ Merge or Rebase — never Squash. Squashing rewrites all four commits
into one authored by whoever presses the button, which erases Jacob from the
history entirely. All three merge strategies are enabled on this repo, so
this is possible to do by accident.

Why a takeover rather than pushing to #678

#678 was opened from a fork point well behind main and its spec makes
several claims that were true when written and are not now. Rather than ask
for a rebase and a round of edits on someone else's branch, this takes the
content over and states the corrections in a commit with my name on it — so a
reviewer can see exactly what I changed and disagree with any of it
separately from Jacob's original work.

Why this should merge before the other four

Spec 013 and spec 010/Requirement 8 exist only on this branch. #689
(#685), #690 (#688), #691 (#683) and the rows-changed PR all carry Refs:
lines naming 010/Req-8, 013/Req-1, 013/Req-4 and 013/Req-7. Until
this lands, those references point at requirements the tree does not contain,
and #689's seven new corpus tests cannot get the per-scenario **Tests:**
links this repo's traceability convention asks for.

Nothing else about the ordering is load-bearing. ADR numbers only need to be
unique, so the 0039/0040 gap this leaves in index.md closes itself when
#690 and #691 land.

The ADR is renumbered 0034 → 0041

0034 was already taken by 0034-index-range-seeks.md when #678 was opened,
and 0038/0039/0040 have since gone to the Cargo registry decision (#684),
Value's Arc payloads (#690) and the streaming primitive (#691). Three
citations inside spec 013 and the index.md row move with it.

Five claims that stopped being true

Spec said	Reality
item 7: "nothing in src/vdbe/ offers a step or iterator API"	vdbe::Execution is that API (#691), with run() reimplemented as a wrapper over it
item 3: !Send is the pager's fault	Value::Text/Blob held Rc, so a result row — the one thing that must leave a worker thread — was unsendable too (#690)
the composite-PK gap is one item	It is two. Maintenance is fixed (#689); creating an autoindex on CREATE TABLE is still open (#687)
item 6: "synchronous has no handler"	It has a full one — query form, all three levels, decided per-level fsync policy (#645, ADR-0036). This one was already false when #678 was written, and it means Requirement 5 is further along than it claims
tree at 0.18.5	0.18.10

The synchronous one is the one to take as a warning: four of these five were
falsified by my own branches, but that one had been stale since before #678
was opened, and I only caught it while checking whether SQE can use this yet.
The list was written against 0.18.5 and the tree is 0.18.10 — treat every
"is missing" line in this spec as a claim to re-verify, not a fact.

The Send one is worth a reviewer's attention: both this spec and
ADR-0041 attribute the problem to the pager alone, and neither mentions
Value. The pager half is still real and is still what Requirement 4's
worker thread exists for — but the Value half was the part that made a
Send + Sync handle impossible, and nothing in the original spec saw it.

Requirement 7's acceptance scenario could not pass

It asked that "peak allocation is proportional to the ten rows". No
correct implementation satisfies that wording, because peak heap for a
streaming read is a floor set by the page cache, not a slope in rows
pulled. Measured on 1,000,000 rows (spike 014, #682): 137.7 MB materialized
against 8.68 MB streamed, and the 8.68 MB does not move with result size.

Restated as independence from result size, which is both the property a
consumer needs and — unlike proportionality — testable. The whole floor turns
out to be one constant, DEFAULT_PAGE_CACHE_CAPACITY (src/pager.rs:63):
2000 pages → 8.68 MB, 256 → 1.10 MB, 64 → 291 KB, at ~4.5% streaming
throughput for the smallest. So a caller who needs the floor lower has a
knob, and the scenario now describes something that can be asserted.

Both specs cited the wrong oracle

Measurements were attributed to "stock sqlite3 3.51.0". The pinned oracle
is 3.53.4 (tests/corpus/oracle.rs:22, Cargo.toml [package.metadata.oracle]). On macOS a bare sqlite3 is Apple's 3.51.0
codec build, which tools/gen_fixtures.sh refuses by design.

I re-derived the autoindex numbering rule against the pinned 3.53.4 across
eleven DDL shapes, comparing pragma_index_info key lists rather than index
counts. Every case agrees, including the counter-intuitive ones —
declaration order beats primary-key-first, and a rowid alias consumes no
number. So the citation was wrong, not the rule. Fixing it anyway, because
"measured against the pinned oracle" is this repo's entire assurance basis
and it has to mean what it says.

What is deliberately left alone

Spec 010/Req 8's three scenario **Tests:** links already name the exact test
functions #689 created — Jacob predicted the file and function names
correctly. They stay (planned) here: the flip belongs to the PR that
discharges the requirement, and it cannot happen on this branch because those
tests do not exist on it. #689 should get a follow-up commit dropping the
three markers once this lands.

Test plan

• make check-assurance — 86/86 and 276/276, unchanged, no dead
  links. A spec-only PR moving nothing is the correct outcome here:
  every requirement added is (planned), which tools/assurance.py
  excludes from all scoring by design. The dashboard moves when the
  implementing tickets land, not when the promise does
• make lint (both clippy passes) and make check-mod-files clean
• No source files touched — .openspec/ only
• Authorship verified on all three cherry-picks:
  author=Jacob Verhoeks <jjverhoeks@schubergphilis.com>

Three design questions for Jacob, not blocking

These are genuine choices the spec makes that I did not want to silently
overrule, and they belong in conversation rather than in my commit:

1. Requirement 4's worker thread. Now that rows are Send (feat: make Value Send by switching Text/Blob payloads to Arc (#688) #690), is one
   thread per connection still the shape you want, or does the pager's Rc
   graph want revisiting on its own terms?
2. Requirement 7's chunking. spike: 014 embedding-API kernel — Send+Sync handle over a streaming VDBE #682 measured one-row-per-message transport
   at ~4.5 µs/row and chunking at ~1024 erasing it. Adaptive chunking
   (1, 2, 4 … 1024) came within 5% of the best fixed size on drain and 10% on
   first-row latency, with no knob. Should the facade do that, or expose the
   chunk size?
3. The page-cache floor. Should DEFAULT_PAGE_CACHE_CAPACITY become
   per-connection configuration, given it is the entire streaming memory
   floor?

Spend: small — the cherry-picks are mechanical; the cost was reading the
spec against the tree closely enough to find the four stale claims and the
unsatisfiable scenario.

Refs: 010/Req-8, 013/Req-1, 013/Req-4, 013/Req-7, #682, #685, #687

Supersedes #678

🤖 Generated with Claude Code