feat(voice): experimental voice mode with an orb surface (desktop, OpenAI Realtime)

[Leo310]

What

Experimental, desktop-only voice mode: a full-duplex conversation with OpenAI's realtime speech model, which does the talking itself and hands anything note-related to the regular agent through a single function call (ask_second_brain, the chat-supervisor pattern). A delegated request runs as an ordinary chat turn on the open thread — persisted, streamed, reviewable on screen — and the spoken answer is read back. The user can keep talking while the agent works; the result is handed to the speech model once it is idle and the user is silent.

While voice is bound to a thread, the chat view swaps to an audio-reactive orb (breathes while listening, pulses with the model's voice, spins a ring while the agent works) with the last transcript lines, a stop button and a "show chat" toggle that reveals the message list under a compact orb strip. Settings: General → Voice (experimental). Composer gets a mic button; there is a "Toggle voice mode" command.

Design notes, all in src/voice/:

• realtimeProtocol.ts holds every event name and the PCM16 codecs, so API drift is a one-file fix.
• turnCoordinator.ts is a pure reducer that holds finished results until the model is idle and the user silent (server VAD auto-creates a response after speech_stopped, so a naive response.create collides).
• supervisorBridge.ts serialises delegations and maps the turn's outcome to JSON the model reads back; ChatSession.sendMessageAndAwait is the new completion hook it relies on.
• Capture runs in an AudioWorklet shipped as raw text through a blob URL (the ?worker&inline route yields a constructor, not a URL); playback tracks how much of an item was heard so barge-in can truncate the server's transcript.
• Auth is the openai-insecure-api-key WebSocket subprotocol with the configured OpenAI provider's key (browser sockets cannot set headers; the key already lives on this machine). No new dependencies.

Known limitations of the prototype: headphones assumed (Chromium cannot echo-cancel WebAudio output, so on speakers the model can hear itself); starting voice mode unmounts the composer, so an unsent typed draft is lost; only delegated turns land in the .chat thread (chit-chat since the last delegation is prefixed to the next request as context); no reconnect.

No linked issue — scoped in-session with the maintainer.

How I tested it

bun run check, format, lint, test (1893 tests, including new suites for the protocol codecs, the delivery coordinator, the supervisor bridge, the WebSocket client with a fake socket, the awaitable chat turn, and the v13→v14 migration) and madge --circular with type imports skipped all pass. A one-shot dev build inlines the worklet (registerProcessor present in main.js).

Not yet tried live. A real run needs an OpenAI API key in the slot vault (S2B WT2) and a microphone; the live checklist is: enable in settings → open a chat → mic → say "hello" (voice-only reply) → ask about a note (filler phrase, orb ring, a delegated user message appears in the chat, answer spoken) → talk over the model → ask "are you done?" during a long delegation → stop. Event names were written against the current GA docs but only a live session proves them.

AI assistance: Claude Code (Fable 5.1) designed and wrote the whole change from Leo's brief after a research discussion on speech-to-speech models; Leo chose the backend, platform, UI shape and transcript policy; live verification in a vault is still pending.

Checklist

• bun run check, bun run format, bun run lint, and bun run test pass locally
• I tried the change in a real Obsidian vault (or explained above why that isn't applicable) — pending, see above
• I read CONTRIBUTING.md, including the section on AI assistance
• If this adds a provider, a bundled skill, a built-in tool, or changes manifest.json: I noted that the docs site needs updating — not applicable yet (experimental, undocumented on the site until it ships)

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; no new actionable failures remain, and all previous findings were resolved.

Summary

This PR adds an experimental desktop voice-conversation mode backed by OpenAI Realtime and integrates delegated requests with ordinary persisted chat turns. Changes since the previous review also redesign list_directory to provide compact, context-budgeted folder overviews and update the associated UI and bundled skill guidance.

• Adds microphone capture, streamed PCM playback, barge-in handling, transcript coordination, and an audio-reactive voice surface.
• Adds awaitable chat turns, serialized voice-to-agent delegation, progress narration, and thread/view lifecycle handling.
• Adds persisted voice settings and migrations, desktop settings controls, and a composer/command entry point.
• Reworks directory listings around recursive counts, explicit omission markers, model-aware output budgets, and matching tool-output rendering.
• Advances the Dataview and explore-vault bundled skills while retaining prior shipped bodies for safe upgrades.

Diagram

sequenceDiagram
  participant U as User
  participant V as Voice session
  participant O as OpenAI Realtime
  participant C as Chat session
  participant A as Regular agent

  U->>V: Start voice conversation
  V->>O: Open realtime socket and stream microphone audio
  O-->>V: Speech audio and transcript
  O->>V: ask_second_brain request
  V->>C: sendMessageAndAwait(request)
  C->>A: Run ordinary persisted chat turn
  A-->>C: Stream progress and final answer
  C-->>V: Settled turn result
  V->>O: Function result when model is idle and user is silent
  O-->>V: Spoken answer audio
  V-->>U: Playback and orb/transcript updates

Loading

_{Reviews (18) · Last reviewed commit: "Merge remote-tracking branch 'origin/mai..."}