[AI-793] Polish vision_agents.testing: conversation judge, assertions, Agent support - #652
darkoatanasovski wants to merge 3 commits into
Conversation
…n_agents.testing LLMJudge now evaluates a whole transcript (list of RunEvent) against named criteria and returns a per-criterion verdict with pass/fail, score and reason. Four built-in criteria ship: say-do consistency, stays in scope, concise, and responds in the user's language. The single-message evaluate(event, intent) API is kept and routed through the same path. TestResponse gains assert_function_not_called and assert_function_call_order, both listing the actual calls on failure. TestSession can wrap an Agent so the agent's instructions and MCP tools are exercised as in production. Tool calls are now captured from the LLM's ToolStartEvent/ToolEndEvent instead of monkeypatching the function registry, which removes the two TODOs: overlapping simple_response calls no longer touch registry restoration (turns are separated via a ContextVar), and parallel calls of the same tool pair by tool_call_id. Each turn's events start with the user message and the session exposes the accumulated transcript. Pytest fixtures (test_session, judge, agent_llm, agent_instructions, judge_llm) live in vision_agents.testing.fixtures; the simple agent example uses them together with the conversation judge.
…lose() unsubscribes tool events
📝 WalkthroughWalkthroughThe testing package now evaluates complete event transcripts against multiple criteria and returns per-criterion scores. Priority: ⬇️ Low Merge Risk: 🟡 Moderate · up to Existing consumers using evaluate may receive failing verdicts from unchanged judge responses; preserve compatibility before merging. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Advanced
Run ID: 043c6d2e-05e2-4df3-937c-1e2bfd34ac88
📒 Files selected for processing (14)
agents-core/vision_agents/testing/__init__.pyagents-core/vision_agents/testing/_judge.pyagents-core/vision_agents/testing/_run_result.pyagents-core/vision_agents/testing/_session.pyagents-core/vision_agents/testing/fixtures.pyconftest.pyexamples/01_simple_agent_example/README.mdexamples/01_simple_agent_example/conftest.pyexamples/01_simple_agent_example/test_simple_agent.pytests/test_testing/fake_llms.pytests/test_testing/test_eval.pytests/test_testing/test_fixtures.pytests/test_testing/test_judge.pytests/test_testing/test_session.py
Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.
…m the multi-turn example test
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to GitHub limitations.
🟠 Major · Preserve the legacy response path in evaluate(). · _judge.py:162-218
agents-core/vision_agents/testing/_judge.py:162-218
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winPreserve the legacy response path in
evaluate().LLMJudge.evaluate()remains part of the publicJudgeprotocol, but it now delegates toevaluate_conversation(), whose_parse_verdict()requires a top-levelresultslist. The previous implementation documented and parsed{"verdict": "pass" or "fail", "reason": "..."}. A caller or custom judge LLM that still returns that contract now receivesJudgeVerdict(success=False, reason="Missing 'results' list ...")for every evaluation. No changelog entry documents this breaking change. Keep the legacy prompt/parser inevaluate(), or add an explicit compatibility branch.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Advanced
Run ID: dc42db6a-6a9a-46e2-9aea-63df8bbd0232
📒 Files selected for processing (2)
agents-core/vision_agents/testing/_session.pyexamples/01_simple_agent_example/test_simple_agent.py
🚧 Files skipped from review as they are similar to previous changes (1)
- agents-core/vision_agents/testing/_session.py
Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.
1 participant
Why
The eval package in
agents-core/vision_agents/testingcould only judge oneChatMessageEventagainst one intent, andTestSessiononly wrapped a bareLLM. Real agent behaviour is a sequence of turns, tool calls and tool results, and production agents carry instructions and MCP tools that tests should exercise too._session.pyalso had two known TODOs: tool observation monkeypatched the function registry (unsafe under overlappingsimple_responsecalls) and parallel calls of the same tool could not be paired with their outputs.Linear: https://linear.app/stream/issue/AI-793/polish-vision-agentstesting-conversation-judge-assertions-agent
Changes
LLMJudge.evaluate_conversation(events, criteria, instructions=...)judges a whole transcript (list ofRunEvent) and returns aJudgeVerdictwith pass/fail, mean score and aCriterionVerdict(pass/fail, score, reason) per criterion. Criteria can beCriterionobjects or plain strings. The oldevaluate(event, intent)is kept and routed through the same path.SAY_DO_CONSISTENCY,STAYS_IN_SCOPE,CONCISE,RESPONDS_IN_USER_LANGUAGE. Each has a real-LLM test with a passing and a failing transcript (@pytest.mark.integration, Gemini), plus LLM-free unit tests for prompt building and verdict parsing.TestResponse.assert_function_not_called(name, arguments=...)andassert_function_call_order([...]). Failure messages list the actual calls and, for order checks, the actual order.TestSession(agent=agent)wraps anAgent: the agent's instructions are used and its MCP servers are connected onstart()so their tools are registered on the LLM and captured like any other tool call.ToolStartEvent/ToolEndEventinstead of wrapping registry functions. Turns are separated with aContextVar, so overlappingsimple_responsecalls no longer interfere with each other or with the registry, and parallel calls of the same tool are paired viatool_call_id.ChatMessageEvent, andTestSession.transcriptexposes the accumulated conversation for the judge.test_session,judge,agent_llm,agent_instructions,judge_llminvision_agents.testing.fixtures(enable withpytest_plugins = ["vision_agents.testing.fixtures"]). The simple agent example uses them and judges conversations with the built-in criteria.Note: the Gemini-backed integration tests (built-in criteria and
examples/01_simple_agent_example) needGOOGLE_API_KEYand were not run locally; the unit suite, ruff and mypy pass viadev.py check. The nineplugins/anthropicunit-test errors seen locally are pre-existing and environmental (the Anthropic SDK reads a credentials profile path when noANTHROPIC_API_KEYis set, tripping blockbuster); they pass with the key present.