Remove restaurant bench defects, and correct one this branch introduced

[Nash0x7E2]

Six restaurant gates could not be passed by a correct agent, or could not say why they failed. This removes them. It also reverts a contract change from earlier in this branch that an A/B showed was a regression, and records the noise floor that made that regression hard to see.

Scenarios and gates no correct agent could pass

interrupt demanded allergen: gluten while the contract said dietary preferences are not allergens. The correction is now a shellfish allergy.

entity_dense expected a pickup_window the schema left optional, compared as an exact string. create_order requires it, and 20, 20 min and in 20 minutes now compare equal.

golden and coherence require the exact callback number on create_reservation and the caller said it once. When the agent's own transcriber dropped digits — 512-555-0142 heard as "five one nine two five zero one four two" — no later turn carried the number again, while the contract told the agent to read a phone number back before relying on it. The caller now repeats the digits in groups when asked to confirm, which is what makes the read-back worth doing. coherence-t1 went from an empty phone to 512-555-0142 on create_reservation.

The judge read a required policy as a prohibition. A scenario with no judge.must_refuse still got an empty Must refuse: heading directly above the policy list, and the judge attached the heading to the list. noise_kitchen asks the agent to complete the booking despite the noise, so the verdict came back Must not complete booking despite kitchen noise for doing what the scenario wanted. noise_kitchen went 0/3 to 2/3 and pack-wide policy failures went 7 to 3.

end_state compared with fmt.Sprint while expected_tools went through MatchStructuredValue, so an agent that wrote Peanut satisfied one gate and failed the other. Both use the same comparator.

The filler gate, and the contract change it provoked

The filler gate required one of a set of stall phrases inside the delayed tool's window, which scores the words rather than the property. An agent that read the booking back across the whole 3 s lookup — never leaving a pause, saying something more useful than "one moment" — failed. An agent that said "one moment" and then went quiet passed.

Earlier on this branch I answered that by rewriting the contract to lead every tool-calling turn with the phrase. It passed the gate, tool_filler 0/3 to 2/3, and it was the wrong fix: it moved the substantive reply out of the turn that already held the floor and into the turn after the tool, where noise cut it. Trials booked correctly and told the caller nothing. Counting trials that wrote a correct booking while the caller heard under 25 words, across four k=3 runs: 4 with the phrase leading, and 1, 2, 1 without.

So the gate changed instead. filler_silence_ms is the longest stretch of the window the caller heard nothing in, and more than 800 ms of it fails — the agent's own workingGap, where the implementation already promises to speak up. filler_heard stays recorded, describing how the wait was covered rather than requiring a phrase. tool_filler-t3 now passes on a read-back with no stall phrase in it, which is the case that used to fail.

Diagnostics

barge_in_stop_ms was one unmeasurable -1. barge_in_reason now says whether the call recorded no barge-in, the agent only started speaking after it, or it was already quiet, and the stop edge reads across pauses up to 300 ms rather than 700 ms so a prompt stop is not merged into the reply after it. On interrupt this turned a blank into a number: the agent takes 1240–1640 ms to stop against a 800 ms budget, and agent_jitter_max_ms stays under 100 ms, so buffered audio is not where it goes.

metrics.json also gains heard_utterances and heard_ignored, and heard.json keeps rulings that carry no words.

Agent

A tool call cancelled by a correction used to hand back only "it was not run", so a caller who changed one detail mid-booking was asked for the rest again. The result now carries the arguments the call was about to send.

What the numbers are worth

k=3 is below the noise floor for a pack-level number, which is why the regression above took an A/B to see. Two runs here share a scenario_hash and a contract_hash and differ only in the judge-prompt edit, which cannot reach the tool log; they disagree by 4 on expected_tools and by 1 each on end_state and entity_tools. A one- or two-trial move is not a result, and benchmark/RESEARCH.md now says so.

The A/B is the one controlled comparison: same scenarios, same pipeline, contract the only variable, 5/24 with the phrase leading against 8/24 without — suggestive on its own, and backed by the per-trial silent-booking count that held across all four runs.

None of these runs are a baseline for the project. The as-shipped triple in RESEARCH.md is gemini-3.5-transcribe-live / gemini-3.8-flash / inworld-tts-2-flash; there is no Inworld key on this machine and Cartesia answers 402, so everything here ran deepgram/flux-general-en and elevenlabs/eleven_flash_v2_5. Treat the numbers as diagnostics.

Remaining failures, by cause

The agent books correctly and never says so, which is most of entity_speech. Barge-in stop latency is 1.2–1.6 s against 800 ms. Flux loses digits in a spoken phone number. The rest are judge verdicts that read as real agent errors: overbooking a full slot, and keeping peanut, shellfish after the caller said "instead".

MethodologyVersion is voicebench-live-v5; v4 baselines are a different series.

  

[Nash0x7E2]

Restaurant Voicebench validation (voicebench-live-v4)

Reran the same Go acceleration recipe as 20260909T002419Z / 20260909T014745Z (Flux default, Inworld TTS, local-dev, --spawn /tmp/accel-router). These numbers are a new series. Do not read v4 pass rates as if they were v3.

Live OpenAI judge returned 429 credit_balance_exhausted on every trial (same as the Gemma baseline). Deterministic gates were computed live. Policy/say-do/coherence were re-scored in place with OpenRouter openai/gpt-4.1-mini (max_tokens=2048). No product code was changed for this validation.

Ignored benchmark/out/20260909T040401Z (24/24 invalid; injected ElevenLabs voice 404’d). Canvas agent-failure-before-after.canvas.tsx is not in this workspace.

Runs

Pack	Run dir	Model	k
Flash	benchmark/out/20260909T040542Z	gemini/gemini-3.8-flash	3
Gemma	benchmark/out/20260909T044516Z	gemma/gemma-4-26B-A4B-it	1

Flash k=3 vs baseline 7/24 (20260909T002419Z, v3)

9/24 trials passed (0 invalid after re-score). 4/8 scenarios pass@k.

Scenario	This run (v4)	pass@k	vs 7/24
restaurant.adversarial	2/3	yes	still a pass@k cell
restaurant.coherence	0/3	no	still miss
restaurant.entity_dense	2/3	yes	allergen on the ticket
restaurant.golden	2/3	yes
restaurant.interrupt	0/3	no	barge_in still unmeasured
restaurant.noise_kitchen	0/3	no
restaurant.selectivity	3/3	yes	v3 had 3 selectivity + 3 hold fails
restaurant.tool_filler	0/3	no	v3 had 3 filler fails

Gate failure counts (trial-level, not comparable 1:1 to v3):

Gate	v3 baseline	this v4 run
expected_tools	10	11
end_state	7	6
entity_tools	7	6
policy	6	5
entity_speech	4	6
say_do	3	0
barge_in	3	3
selectivity	3	0
hold	3	0
filler	3	3

Gemma k=1 vs baseline 3/8 (20260909T014745Z, v3)

2/8 passed. Baseline passing cells: adversarial, coherence, tool_filler. This run: adversarial, golden.

Scenario	This run	vs 3/8
adversarial	pass	same
coherence	fail (create_reservation never called)	lost
entity_dense	fail (pickup_window missing; allergen=peanut on the order)	still miss
golden	pass	gained
interrupt	fail (no create_reservation; barge-in stop 220 ms)	barge gate now passes; booking still miss
noise_kitchen	fail (entity_speech peanut)	booked, but did not say peanut
selectivity	fail (hold)	still miss
tool_filler	fail (entity_speech peanut)	filler_heard=true; lost the cell on speech, not filler

Methodology vs agent (barge_in / selectivity / hold)

Re-scored agent.wav with merge gap 0 vs UtteranceMergeGapMS=700.

Flash restaurant.selectivity 3/3: methodology. Unmerged spans fail selectivity+hold because the background talker sits in a sentence gap (started_turn=true, continued=false). After span merge all three pass. Cough already continued without merge; the talker gap is what v4 flips. This is the voicebench-live-v4 series change, not continuous agent audio. Clause chunking did not close those gaps (raw span counts 10/9/16 → merged 3/5/5).

Flash restaurant.interrupt barge_in 0/3: agent. barge_in_stop_ms=-1 with and without merge. The agent was not in a measurable overlapping span when the barge-in played, so span merge cannot help. create_reservation missing on t1/t2; t3 booked peanut (gluten in notes) and a mangled phone.

Gemma restaurant.selectivity hold fail: agent. Cough and talker both continued=false with and without merge. The agent stopped. Selectivity (no new turn) already passed without merge.

Gemma restaurant.interrupt barge_in: agent audio, and it already passed without merge (raw 100 ms, v4 220 ms, both ≤ 800). Trial still fails because booking never completed.

Hold on Flash flipped only because of span merge (same three selectivity trials). Barge_in did not flip on Flash. Gemma hold did not flip.

Requested metrics

filler_heard

• Flash tool_filler: false on all 3. Agent did say “One moment checking”, but the words land at ~18–22 s, after the 3 s delayed check_availability window. filler_non_blocking=true (speech onset ~150–170 ms into the tool). Filler-during-tools is in the transcript, not in the timed gate.
• Gemma tool_filler: true. Gate is filler-clean; trial fails entity_speech (agent never said peanut: “How can I help? One moment. Let me check that.”).

coherence phone 512-555-0142

• Flash: all 3 book, all 3 with the wrong number (51125550142 / 511-255-50142). Agent repeats that number. Canonical 512-555-0142 is not in agent speech. Pack keyterms did not recover it.
• Gemma: create_reservation never called. Canonical number not in agent speech.

interrupt create_reservation

• Flash: t1/t2 never called; t3 called with allergen=peanut (not gluten), party_size=6.
• Gemma: never called. Agent stopped in 220 ms and asked for name + allergen again.

noise_kitchen agent_turns

• Flash: 1 on all 3 (“How can I help?”, no tools). Flux EoT 0.8 did not get a booking under kitchen noise.
• Gemma: 2. Did book (allergen=peanut on the tool) but failed entity_speech and stored phone 512555042.

entity_dense allergen

• Flash: create_order.allergen=peanut on all 3. t1/t3 pass; t2 fails pickup_window missing.
• Gemma: allergen=peanut on the order; fail is pickup_window missing. Speech did mention pickup in twenty minutes.

What actually moved

Change	Effect on this pair of runs
Span merge (voicebench-live-v4)	Flash selectivity+hold 0/3 → 3/3. Only methodology win that changed pass@k.
Filler during tools	Phrase is spoken. Flash still fails the timed filler gate. Gemma passes filler, fails peanut-in-speech.
Pack keyterms	Did not recover 512-555-0142 on Flash coherence (STT/TTS of the caller number is 51125550142).
Book when last detail arrives	Golden/entity_dense often book. Interrupt still does not complete create_reservation after the barge-in.
Clause chunking	Did not make Flash selectivity continuous enough to pass without merge.
Flux EoT 0.8	Flash kitchen still 1 agent turn. Gemma kitchen books (2 turns) but misses peanut in speech.

Net Flash trial pass 7/24 → 9/24 is the three new selectivity passes minus other cells that did not hold. Do not treat that as an apples-to-apples v3 lift.