[AI-795] Add vision-agents agent simulate with table and report output - #656
darkoatanasovski wants to merge 4 commits into
Conversation
Let developers run text-mode simulations from the terminal and in CI, next to `run` and `serve` in `Runner.cli()`. `vision-agents agent simulate <dir-or-file>` loads TOML scenario files (a brief for the caller plus the criteria the agent must meet), plays each one against a fresh agent from the project's `pyproject.toml` entrypoint, and has a judge model rule on every criterion. It prints a table (scenario, variations, pass@k, turns, P50 turn latency, failed criteria), writes `report.json` and `report.md` with the full transcripts and verdicts, and exits 0/1/2 for passed/failed/errored so it can gate CI. `--repeat`, `--variations`, `--judge`, `--report` and `--filter` shape the run; `--judge` takes `provider/model` or a `module:attribute` callable that returns an LLM. The engine lives in `vision_agents.testing` (`Simulator`, `Scenario`, `load_scenario`, `find_scenarios`) and its report shape follows the hosted `SimulationRun` schema. `LLMJudge`'s verdict parsing is extracted into a shared `parse_verdict` so both judges agree on the format. `vision-agents agent` now stops parsing its own options after the first positional, so `agent simulate --help` (and `agent run --help`) reach the Runner CLI instead of printing the dispatcher's help.
- A caller model that returns an empty message, or ends before saying anything, now errors the case (exit 2) instead of judging an empty conversation as a failed scenario. - The `--judge` LLM is probed inside the event loop and closed, so a bad spec still fails fast without leaking a provider client. - Report writing failures and the empty-table edge case raise cleanly. - Scenario discovery only looks at `*.toml` files directly in the given directory, so `simulate .` no longer picks up `pyproject.toml`. - Drop a `getattr` in favour of attribute access. - Tests: silent-caller and bare `[END]` cases, errored-over-failed precedence, fresh-interpreter import order for the core/testing cycle, and `agent run --help` reaching the Runner CLI. - Scaffold README no longer advertises `simulate` against the realtime template agent without explaining the text-LLM requirement.
Plugin LLMs report provider failures through `LLMErrorEvent` and return an empty reply instead of raising, so the simulator now subscribes to the agent's events during a conversation and turns such an error into an errored case (exit 2) rather than letting the judge fail the scenario. Also export `parse_verdict` from `vision_agents.testing`, dedupe repeated errors in the table's failed-criteria cell, note that `--entrypoint` must precede the subcommand, and document what `Runner.simulate()` raises.
📝 WalkthroughWalkthroughThe change adds TOML scenario loading and a text-mode simulation harness. The simulator runs caller and agent conversations, applies repeats and variations, evaluates criteria with a judge LLM, and produces structured reports. The Runner exposes Priority: ➖ Normal Merge Risk: 🔵 Low · up to The feature remains usable, but some CI error classification, model-response parsing, and programmatic configuration behavior should be corrected or explicitly accepted before merging. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Advanced
Run ID: e03a28f3-4957-478e-9c87-b80617bed026
📒 Files selected for processing (12)
README.mdagents-core/vision_agents/cli/agent/command.pyagents-core/vision_agents/cli/init/templates/README.md.j2agents-core/vision_agents/core/runner/runner.pyagents-core/vision_agents/core/runner/simulate.pyagents-core/vision_agents/testing/__init__.pyagents-core/vision_agents/testing/_judge.pyagents-core/vision_agents/testing/_scenario.pyagents-core/vision_agents/testing/_simulation.pyagents-core/vision_agents/testing/_utils.pytests/test_cli/test_cli.pytests/test_testing/test_simulation.py
Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.
| Returns: | ||
| The process exit code: 0 all passed, 1 a scenario failed, 2 an error. | ||
| """ | ||
| configure_sdk_logger(level=getattr(logging, log_level.upper(), logging.WARNING)) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🔎 Supported by static analysis
🏁 Script executed:
sed -n '45,95p' agents-core/vision_agents/core/runner/simulate.py
sed -n '220,275p' agents-core/vision_agents/core/runner/runner.py
sed -n '445,535p' agents-core/vision_agents/core/runner/runner.py
rg -n 'run_simulation\(|\.simulate\(' agents-core tests | head -80Repository: GetStream/Vision-Agents
Length of output: 7134
🏁 Script executed:
#!/bin/bash
rg -n -C 4 'def configure_sdk_logger|configure_sdk_logger\(|def run_simulation|def simulate\(|run_simulation\(' agents-core/vision_agents agents-core/tests tests 2>/dev/null | head -240
printf '\n--- candidate tests/usages ---\n'
rg -n -C 3 'log_level|--log-level|run_simulation|\.simulate\(' agents-core/tests tests agents-core/vision_agents 2>/dev/null | head -260
printf '\n--- package exports ---\n'
rg -n -C 3 'run_simulation|Runner|simulate' agents-core/vision_agents/core/__init__.py agents-core/vision_agents/core/runner/__init__.py agents-core/vision_agents/__init__.py 2>/dev/nullRepository: GetStream/Vision-Agents
Length of output: 21862
Reject unsupported log levels.
Runner.simulate calls run_simulation directly, so programmatic callers bypass Click's Choice validation. An unsupported value silently selects WARNING, which hides the caller's configuration error. Raise ValueError for invalid values.
Proposed fix
- configure_sdk_logger(level=getattr(logging, log_level.upper(), logging.WARNING))
+ levels = {
+ "DEBUG": logging.DEBUG,
+ "INFO": logging.INFO,
+ "WARNING": logging.WARNING,
+ "ERROR": logging.ERROR,
+ "CRITICAL": logging.CRITICAL,
+ }
+ try:
+ numeric_level = levels[log_level.upper()]
+ except KeyError as err:
+ raise ValueError(f"unsupported log level: {log_level}") from err
+ configure_sdk_logger(level=numeric_level)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| configure_sdk_logger(level=getattr(logging, log_level.upper(), logging.WARNING)) | |
| levels = { | |
| "DEBUG": logging.DEBUG, | |
| "INFO": logging.INFO, | |
| "WARNING": logging.WARNING, | |
| "ERROR": logging.ERROR, | |
| "CRITICAL": logging.CRITICAL, | |
| } | |
| try: | |
| numeric_level = levels[log_level.upper()] | |
| except KeyError as err: | |
| raise ValueError(f"unsupported log level: {log_level}") from err | |
| configure_sdk_logger(level=numeric_level) |
| raise SimulateError( | ||
| f"--judge {spec!r}: expected an LLM instance, got {type(llm).__name__}" | ||
| ) | ||
| await llm.close() |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🔎 Supported by static analysis
🏁 Script executed:
sed -n '120,138p' agents-core/vision_agents/core/runner/simulate.py
sed -n '485,535p' agents-core/vision_agents/core/runner/runner.py
rg -n 'exit code 2|exit_code = 2|provider error|probe' README.md agents-core tests/test_cli/test_cli.pyRepository: GetStream/Vision-Agents
Length of output: 2882
Convert probe cleanup failures to exit code 2.
If llm.close() raises, the exception escapes probe_llm_factory as an unclassified nonzero failure. This prevents CI from distinguishing the provider error from a scenario failure, even though the command still terminates.
Wrap the close failure in SimulateError.
Proposed fix
- await llm.close()
+ try:
+ await llm.close()
+ except Exception as err:
+ raise SimulateError(
+ f"--judge {spec!r}: failed to close the probe LLM: {err}"
+ ) from err📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| await llm.close() | |
| try: | |
| await llm.close() | |
| except Exception as err: | |
| raise SimulateError( | |
| f"--judge {spec!r}: failed to close the probe LLM: {err}" | |
| ) from err |
| if cleaned.startswith("```"): | ||
| cleaned = cleaned.split("\n", 1)[-1].rsplit("```", 1)[0].strip() |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Single-line code fences are not stripped.
Line 32 removes the opening fence by splitting on the first newline. A one-line fenced response has no newline, so split("\n", 1)[-1] returns the whole string, including the leading ```. Only the trailing fence is removed.
Input ```{"verdict": "pass"}``` returns ```{"verdict": "pass"}. parse_verdict then fails json.loads and raises ValueError, and _parse_string_list raises SimulationError. The case is reported as errored instead of judged.
Strip the opening fence directly:
Proposed fix
def strip_code_fences(text: str) -> str:
"""Return ``text`` without a surrounding markdown code fence, if any."""
cleaned = text.strip()
if cleaned.startswith("```"):
- cleaned = cleaned.split("\n", 1)[-1].rsplit("```", 1)[0].strip()
+ cleaned = cleaned[3:]
+ head, newline, rest = cleaned.partition("\n")
+ # Drop an info string such as ``json`` on the opening fence line.
+ cleaned = rest if newline and "`" not in head else cleaned
+ cleaned = cleaned.rsplit("```", 1)[0].strip()
return cleaned📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if cleaned.startswith("```"): | |
| cleaned = cleaned.split("\n", 1)[-1].rsplit("```", 1)[0].strip() | |
| if cleaned.startswith("```"): | |
| cleaned = cleaned[3:] | |
| head, newline, rest = cleaned.partition("\n") | |
| # Drop an info string such as ``json`` on the opening fence line. | |
| cleaned = rest if newline and "`" not in head else cleaned | |
| cleaned = cleaned.rsplit("```", 1)[0].strip() |
| model follows the scenario brief, your agent replies turn by turn, and a judge model rules on each criterion | ||
| once the conversation ends. It prints a table with the scenario, variations, pass@k, turns, P50 turn latency | ||
| and failed criteria, writes `report.json` and `report.md` with the full transcripts and verdicts, and exits | ||
| `0` when every scenario passed, `1` when any failed and `2` on a judge or provider error, so it can gate CI. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Document all exit code 2 conditions.
Scenario, configuration, and execution errors also return exit code 2. The current text limits this code to judge or provider errors. This can cause incorrect CI error handling.
1 participant
Why
Developers can already
runandservean agent from the CLI, but there was no way to put an agent through a set of conversations and get a pass/fail answer, locally or in CI. The hosted dashboard trigger for this exists in Go on theacceleratebranch; this is its local and CI counterpart.Linear: AI-795
Changes
vision-agents agent simulate <dir-or-file>(alsouv run agent.py simulate ...), added next torunandserveinRunner.cli(). It loads every*.tomlscenario file, resolves the agent frompyproject.toml, and runs each scenario in text mode: a caller model follows the scenario brief, the agent replies turn by turn throughTestSession, and a judge model rules on every criterion once the conversation ends. Every conversation gets a fresh agent so no history leaks between cases.--repeat N,--variations N(LLM-generated rephrasings of the brief; the brief as written is always variation 0),--judge MODEL(provider/modelsuch asgemini/gemini-2.5-flash, ormodule:attributenaming a callable that returns an LLM, which is also how the tests inject a scripted judge),--report DIR,--filter NAME, plus--log-levelfor symmetry withrun.report.json/report.mdin the report directory with full transcripts (including tool calls) and per-criterion verdicts. The JSON shape follows the accelerationSimulationRun/SimulationCase/SimulationLineschema (state,cases,passed,failed,conversations[]withvariation,scenario,transcript[{caller, text, at}],turns,passed,verdict,ended,error, timestamps), withattempt,criteria[]andlatency_msadded for the local use case.--judgespec, missing scenarios and malformed scenario files also exit 2).name(defaults to the file stem),scenario(the caller brief),criteria(list), optionalmax_turns(default 12) andvariations(default 1). Documented in the root README together with the rest of the CLI, and referenced from the scaffold README.vision_agents.testing:Scenario,load_scenario,find_scenarios,Simulator,SimulationReport/SimulationRun/SimulationCase/TranscriptLine/CriterionVerdict/ToolCall.LLMJudge's verdict parsing is extracted into a sharedparse_verdictso both judges accept the same JSON format; parse failures now surface as aValueErrorthatLLMJudge.evaluatestill turns into a failed verdict.vision-agents agentnow setsallow_interspersed_args = False, so options after the first positional (e.g.agent simulate --help,agent run --help) reach the Runner CLI instead of being parsed by the dispatcher.--entrypointkeeps working when given before the subcommand.tests/test_clicovers--help, exit codes 0/1/2, every option being reflected in the run, bad--judgespecs, missing and malformed scenarios, and the contents ofreport.json/report.md, all against a stub project with a scripted LLM (no real provider).tests/test_testing/test_simulation.pycovers scenario loading, the conversation loop (end token,max_turnscutoff, tool-call capture), variations and repeat semantics, error propagation and the report shape.Realtime (audio) LLMs cannot be simulated in text mode; such agents get a clear per-case error and exit code 2. This is called out in the README.