feat: streaming Execution primitive, with run() as its wrapper (#683)

[dpsiderius]

Summary

Every execution entry point (execute_with_db,
execute_with_db_and_params, execute_transaction_step) materialized a whole
Vec<Vec<Value>> before returning, so reading row 0 of a large result cost
building row N, and a caller could not stop early. It was also not fixable
from outside the crate: fn dispatch (src/vdbe/exec.rs:704) and fn run are
private, and ResultRow hands its row to Vm::emit_row, which pushes into a
Vec owned by the Vm.

Spike 014 (#682) measured the cost on a 1,000,000-row result:

	batch	streaming
peak heap	137.7 MB	8.68 MB
time to first row	5.36 ms	44.7 µs

This adds pub struct Execution<'p> (new / next_row / autocommit) and
reimplements run() as a wrapper that collects next_row into the same
Vec it already returned.

The wrapper is the load-bearing design decision, not a convenience. Batch and
streaming become literally the same loop, so they cannot drift, and the
existing suite becomes the equivalence proof: 1562 passed / 0 failed before,
1568 / 0 after, the delta being exactly this ticket's six new tests. The
alternative — a second, parallel execution path — is what ADR-0040 rejects.

The pending FIFO is not an accident

pending is a VecDeque drained from the front, not a pop off the back,
because pragma::integrity_check (src/vdbe/pragma.rs:110) emits one row per
problem from a single dispatch. Draining from the back would silently
reverse that output.

No test in the suite reached that path, because no existing test produces
more than one problem row — so this PR builds one:
vdbe_streaming_execution_test.rs empties three indexes (writing a
valid-but-empty index leaf, page[0] = 0x0A) to force a genuinely multi-row
dispatch. Mutation-checked: replacing the FIFO with a pop fails that test, and
only that test. Without it the guard would have been free to regress.

Deliberately not included

• Any transaction-aware streaming constructor. Vm::autocommit is
  private, and read-only streaming is what spec 013/Req 7 targets.
• Chunking. spike: 014 embedding-API kernel — Send+Sync handle over a streaming VDBE #682 measured a one-row-per-message transport at ~4.5 µs per
  row and found chunking at ~1024 erases it, but Execution has no channel —
  that is a facade concern. Recorded in ADR-0040 so the facade ticket does not
  rediscover it.
• Making Value Send — that is feat: make Value Send by switching Text/Blob payloads from Rc to Arc #688, in review alongside this.
• Rows-affected (spec 013/Req 1) — separate ticket.

Notes for a reviewer

• Two things about ADR numbering. The ADR file here is
  0040-streaming-execution-with-batch-as-wrapper.md, but the commit message
  says ADR-0038 — it was written before feat: private crate registry via JFrog Artifactory (#12) #684 landed and took 0038 for the
  Cargo registry decision. The code citation (exec.rs) says 0040 and is
  correct; the commit message is stale and I have left it rather than
  force-push over a pushed branch. Merge feat: make Value Send by switching Text/Blob payloads from Rc to Arc #688 (ADR-0039) before this one
  or the ADR sequence gains a gap.
• make assurance does not move on this PR — still 86/86 and 276/276.
  Spec 013 is the subject of Feat/embedding api spec #678, which is still Status: Proposed and
  unmerged, so Req 7 is not yet in the dashboard's denominator. This PR
  implements the engine half of Req 7 ahead of the spec landing; the
  dashboard should move when Feat/embedding api spec #678 does.
• Spec 013/Req 7's acceptance scenario as written in Feat/embedding api spec #678 cannot pass, and
  that is worth settling in that PR rather than here. It asks that "peak
  allocation is proportional to the ten rows"; the measured streaming floor is
  8.68 MB, dominated by the page cache and flat in result size. spike: 014 embedding-API kernel — Send+Sync handle over a streaming VDBE #682 found
  the whole floor is one constant — DEFAULT_PAGE_CACHE_CAPACITY
  (pager.rs:63): 2000 pages → 8.68 MB, 256 → 1.10 MB, 64 → 291 KB (445x
  better than batch), at a cost of ~4.5% on streaming throughput and nothing
  on batch. The provable claim is "peak allocation is independent of result
  size", which is the stronger property anyway. Filed as a follow-up on
  Feat/embedding api spec #678, not fixed here.
• No CHANGELOG.md entry or version bump: this repo folds those into a
  separate chore/*-fold-into-0.18.x PR (precedent chore: fold #663 into 0.18.9, revert premature 0.18.10 bump #671). feat: streaming Execution primitive — read a result row without materializing the rest #683's own
  acceptance criteria ask for a CHANGELOG entry and are wrong about the
  convention — I wrote them; the ticket needs correcting, not this PR.

A second commit, from the fuzz gate

fix(fuzz): bound vdbe_exec's row drain so an emit-loop can't OOM the gate
(+37/−4, plus one seed) is a direct consequence of this change rather than
unrelated housekeeping. Making a row drain reachable means a fuzz-generated
program can drive an unbounded ResultRow emit loop, which OOMs the fuzz
target rather than reporting a finding. The drain is now bounded and a
result_row_emit_loop seed pins the case. Touches only tests/fuzz/.

Test plan

• tests/unit/vdbe_streaming_execution_test.rs — 6 new tests: row/order
  equivalence with run for a scan, a LIMIT, an aggregate and an empty
  result; multi-row-single-dispatch order; errors and halts are terminal
  (a caller that keeps polling gets None, not a re-entered program)
• cargo test --locked — 1568 passed / 0 failed (origin/main
  measured 1562 the same way: +6, exactly this ticket's tests)
• cargo test --locked --test corpus — 380, unchanged from baseline
• FIFO guard mutation-checked (pop-off-back → exactly one test fails)
• make lint, cargo fmt --check, make check-mod-files clean
• make assurance — 86/86, 276/276, no dead links
• make lint — both clippy passes, including the second one naming
  the test = false corpus/parity/sqllogictest targets that --tests
  does not build. cargo clippy --all-targets does not cover them, and
  is the check I originally used
• All of the above re-verified at 77d76e9 (branch tip, including the
  fuzz commit) from a clean detached worktree, not the working tree

Spend: matched the small estimate. The design was already paid for by
spike 014; what was not in the estimate was the multi-row-dispatch fixture,
since the suite had no way to produce one.

Refs: 013/Req-7, #682, #678, #688

Closes #683

🤖 Generated with Claude Code