docs: evals cover scripted and simulated scenarios by aconchillo · Pull Request #1259 · pipecat-ai/docs

aconchillo · 2026-09-09T06:57:08Z

Summary

Simulations landed in pipecat (#5646): a scenario with a persona: and a goal: hands the user's side to an LLM, and a judge decides success: and scores per-reply metrics:. The Evals section now explains the two kinds of scenario, scripted and simulated.

Overview leads with a scripted and a simulated example, adds a "Two kinds of scenario" section with the kind × modality grid, and repositions third-party platforms around deployed paths, caller populations at scale, and production monitoring.
Writing Scenarios becomes Scripted Scenarios (/pipecat/evals/scripted-scenarios, redirect from the old path) beside a new Simulated Scenarios page: persona and goal, the simulator: block, modalities, the success: criterion and what the judge sees, judged metrics with the per-reply yes/no rule, a table of the built-in measures, function_calls and calls: [], caps and runs:, how a run ends, sample output, and authoring advice.
Quickstart ends with a simulated caller against the same agent.
Lifecycle distinguishes one local persona from platform caller populations and adds a simulation to the booking example and the adoption path.
Suites covers mixed manifests, -k, runs: as a requirement versus --repeat as a measurement, the simulation record in results.jsonl, and per-run subprocesses.
Using the Library moves to the 1.9.0 names (EvalScriptSession, EvalSimulationSession, load_scenario_file, persona_llm=, user_tts=/bot_stt=) with a migration table for the deprecated ones.
The Eval Loop and the CLI reference pick up simulations, -k, the new --repeat semantics, and .env loading.

The pipecat AGENTS.md template was aligned to the new page titles in pipecat-ai/pipecat branch aleix/flows-hello-world-greeting.

Checks

node scripts/check-imports.mjs resolves every import against pipecat main (daf5902d9).
npx mint broken-links --check-anchors --check-redirects: no broken links.
node scripts/docs-meta-lint.mjs: no errors; llms.txt and llms-full.txt regenerated.
Prettier clean.

github-actions · 2026-09-09T06:57:19Z

🔍 Mintlify preview for this branch: https://daily-aleix-evals-simulated-scenarios.mintlify.site

pipecat main deprecated pipecat.evals.harness in 1.9.0 and re-exports EvalSession through a star import, which the import checker cannot resolve. Point the library page at the module that defines it, and drop the unused EvalTurnProgress import. A fuller rewrite for the renamed script/simulation API lands in #1259. Claude-Session: https://claude.ai/code/session_013Y5KvVSbnLLMWmMk41duMr

Simulations landed in pipecat (#5646): a scenario with a persona: and a goal: hands the user's side to an LLM, and a judge decides success and scores per-reply metrics. The Evals section now explains the two kinds. - Overview leads with both kinds, adds the kind x modality grid, and repositions third-party platforms around deployed paths and scale. - Writing Scenarios becomes Scripted Scenarios (redirect kept), beside a new Simulated Scenarios page: persona, goal, simulator, success, judged and measured metrics with a table of the built-in measures, function_calls, caps, runs, endings, output, and authoring advice. - Quickstart ends with a simulated caller; Lifecycle no longer calls simulations out of scope; Suites covers mixed manifests, -k, runs vs --repeat, and the results.jsonl record; the Library page moves to the 1.9.0 names with a migration table; the Eval Loop and CLI reference pick up simulations, -k, and .env loading.

…tParams

The user: and judge: blocks, factory:, !include, back-to-back runs, and the disconnect path apply to scripted and simulated scenarios alike, so they move from the scripted page to Scenario Configuration. The scripted page keeps the two fields only it has, context: and stop_on_failure:, and both kinds link to the new page.

Follows pipecat #5676: EvalSession is the base of both session kinds and builds either from a loaded scenario, and how a run behaves is one EvalSessionParams. The kind-specific from_scenario() calls stay as the typed alternative, the run knobs move to a params table, and the migration table drops the EvalSession rename.

Follows pipecat #5676's follow-ups: min_quality is min_score, the persona runs on the local Ollama judge model unless a simulator: block says otherwise, service: openai and service: cartesia give way to a factory: that takes the block alone, a run ends as silence after max_silence_s, and results tell judge trouble from bot trouble with a verdict of none, a metric kind, and a judge_no_verdict error on the goal.

markbackman · 2026-09-10T11:07:00Z

 - A judge LLM. Either:
  - **Ollama** (local, the default): install [Ollama](https://ollama.com) and run `ollama pull gemma4:12b`, or
-  - **OpenAI**: set `OPENAI_API_KEY` and point the scenario's `judge:` block at it (shown below).
+  - **OpenAI**: set `OPENAI_API_KEY` and point the scenario's `judge:` block at it through a small factory (shown below).


For a quickstart, does it make sense to set up a factory or just use the default service to simulate turns?

That said, it is nice to teach the factory option to provide your own. No strong preference.

I have removed OpenAI. It's deprecated now, you can still use the factory function.

markbackman · 2026-09-10T12:14:41Z

+| `turns`          | The persona's turns in the conversation. A reply that only calls `end_call` is not a turn.                                                                                                                        | count   |
+| `duration`       | The conversation's length, from its first line to the hang-up.                                                                                                                                                    | seconds |
+| `words`          | The longest agent reply. Bounds every reply.                                                                                                                                                                      | words   |
+| `latency`        | The slowest agent reply. In text mode, from the persona's send to the reply's first LLM token; in audio mode, from the agent noticing the persona stop speaking to its first spoken sentence. Bounds every reply. | seconds |


I think in text mode, latency isn't relevant, right?

It still measures time to first token. I've updated the description.

markbackman · 2026-09-10T12:28:32Z

-`EvalManifest.load()` accepts keyword overrides for every manifest value (`concurrency`, `base_port`, `spawn`, `scenarios_dir`, and so on), mirroring the CLI flags.
+`EvalManifest.load()` accepts keyword overrides for every manifest value (`concurrency`, `base_port`, `spawn`, `scenarios_dir`, `repeat`, and so on), mirroring the CLI flags.
+
+## Migrating from 1.8


markbackman · 2026-09-10T12:29:50Z

 - **Actionable failures.** Failures name the turn, the expectation, and the reason, including what the judge said. The `.eval.log` decision trace shows every event the harness observed, so "why did this fail" is answerable from files.
 - **Suites are self-contained.** `pipecat eval suite` spawns the agents itself, so an autonomous loop doesn't need to manage processes: edit, run one command, read the result.
 - **Text mode is fast and cheap.** Iterating on prompts and logic skips STT and TTS entirely, so an assistant can afford to run the evals after every change.
+- **Simulations find, scripts pin.** A failing simulation shows where the agent lost the caller, with the conversation printed under `-v`. The assistant reproduces that exchange as a scripted scenario, fixes against it, and keeps both.


Claude keeps using this phrase of "scripts pin", but it seems a bit weird. Maybe we can find a better way to say it. It's used here and in a few other places.

Yeah, that is actually wrong. Simplified.

markbackman

LGTM! Nice improvement. Just a few minor things.

… judge is a default

mintlify Bot deployed to staging September 9, 2026 17:13 View deployment

mintlify Bot deployed to staging September 9, 2026 17:16 View deployment

mintlify Bot deployed to staging September 9, 2026 17:30 View deployment

mintlify Bot deployed to staging September 9, 2026 17:36 View deployment

aconchillo force-pushed the aleix/evals-simulated-scenarios branch from 31cf3c8 to 2e2acb4 Compare September 9, 2026 17:37

mintlify Bot deployed to staging September 9, 2026 20:57 View deployment

mintlify Bot deployed to staging September 10, 2026 00:53 View deployment

aconchillo added 10 commits September 9, 2026 18:14

docs: simpler description for the evals overview

8335b35

docs: the evals overview says a script's eval: is judged too

fa0b457

docs: the evals quickstart names the eval transport with EvalTranspor…

c173724

…tParams

chore: regenerate llms-full.txt after rebase

ace7812

docs: a metric's failure_kind, renamed from kind

903fa47

chore: regenerate llms-full.txt after rebase

e041a3a

aconchillo force-pushed the aleix/evals-simulated-scenarios branch from 006f849 to d1f9a89 Compare September 10, 2026 01:14

mintlify Bot deployed to staging September 10, 2026 01:15 View deployment

aconchillo requested a review from markbackman September 10, 2026 01:22

docs: a lull is measured from either side's activity

16f9143

mintlify Bot deployed to staging September 10, 2026 01:23 View deployment

aconchillo force-pushed the aleix/evals-simulated-scenarios branch from ff9121b to 16f9143 Compare September 10, 2026 02:54

mintlify Bot deployed to staging September 10, 2026 02:56 View deployment

markbackman reviewed Sep 10, 2026

View reviewed changes

Update simulated anatomy for no-reasoning default

1e56314

mintlify Bot deployed to staging September 10, 2026 12:11 View deployment

More updates for simulated guide

06fcacf

markbackman reviewed Sep 10, 2026

View reviewed changes

markbackman approved these changes Sep 10, 2026

View reviewed changes

aconchillo added 2 commits September 10, 2026 08:34

docs: the evals quickstart runs on Ollama alone

8d96000

chore: regenerate llms.txt for the simulated guide updates

9a7b990

mintlify Bot deployed to staging September 10, 2026 15:36 View deployment

docs: latency is the audio-mode example, and the quickstart notes the…

5ce2b81

… judge is a default

mintlify Bot deployed to staging September 10, 2026 15:42 View deployment

docs: scripts are for exact control, simulations for a goal

1c13ff5

aconchillo mentioned this pull request Sep 10, 2026

cli: the agent template says scripts are for exact control and simulations for a goal pipecat-ai/pipecat#5686

Merged

mintlify Bot deployed to staging September 10, 2026 15:49 View deployment

docs: plainer wording across the evals pages

00d2bda

mintlify Bot deployed to staging September 10, 2026 15:53 View deployment

docs: no em dashes in the evals pages

66798ea

aconchillo merged commit 177aa3a into main Sep 10, 2026
4 checks passed

mintlify Bot deployed to staging September 10, 2026 16:20 View deployment

Navigation Menu

Repository navigation

docs: evals cover scripted and simulated scenarios - #1259

docs: evals cover scripted and simulated scenarios#1259
aconchillo merged 19 commits into
mainfrom
aleix/evals-simulated-scenarios

aconchillo commented Sep 9, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

markbackman Sep 10, 2026 •

edited

Loading

Uh oh!

aconchillo Sep 10, 2026

Uh oh!

markbackman Sep 10, 2026

Uh oh!

aconchillo Sep 10, 2026

Uh oh!

markbackman Sep 10, 2026

Uh oh!

markbackman Sep 10, 2026

Uh oh!

aconchillo Sep 10, 2026

Uh oh!

markbackman left a comment

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

aconchillo commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checks

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

markbackman Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aconchillo Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

markbackman Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

aconchillo Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

markbackman Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

markbackman Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

aconchillo Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

markbackman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aconchillo commented Sep 9, 2026 •

edited

Loading

markbackman Sep 10, 2026 •

edited

Loading