fix(agent): empty assistant turn never completes the node - #74
Merged
Conversation
An empty string is truthy in Lua, so tool_calling=auto treated a final turn with no text and no tool calls as a successful completion and routed "" downstream as the node result. NONE rejected "" but not whitespace. Both modes now share one predicate: text needs a non-whitespace character, a structured table needs at least one field, nil/false never complete. An empty turn records an EMPTY_RESULT observation so the model is asked for a real answer. The observation check_completion records was queued after the turn's durable yield, so the model only saw it one turn late; finalize now yields right after recording feedback. Control artifacts are persisted with the creating node's id so they can be attributed and recovered. Tests: check_completion matrix across NONE/AUTO/ANY (empty, whitespace, nil, false, empty table, structured, pending tool calls, min_iterations); a loop-level run where the model creates an artifact, returns an empty turn, then answers (artifact retained, exactly one empty-result observation, output is the real answer); an all-empty run failing at the iteration limit. Claude-Session: https://claude.ai/code/session_01TLD4uyqDLGD9JfgZESaaXH
wippy 0.3.35a lint loads entries from lock paths only, so the workspace-replaced module has to be listed for the harness to link; install leaves the entry as is, so verify-lock stays clean. Claude-Session: https://claude.ai/code/session_01TLD4uyqDLGD9JfgZESaaXH
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
tool_calling = "auto"completed the agent node on an empty assistant turn:agent_result.result ~= nilis true for""(the empty string is truthy in Lua), so a model that had just created its answer throughCreateArtifactand then returned{"result":"","tool_calls":[]}ended the node "successfully" with a zero-byte result.nonerejected""but accepted whitespace. Downstream consumers (kickside/researchdeliver) received an empty report and recorded a false success. Report: "Kickside Research false-success with empty report and orphaned artifact".Two adjacent defects surfaced while proving the fix at loop level:
NO_TOOLS_CALLEDfeedback observation was queued after the turn's durable yield, so the next prompt (built from persisted history) did not contain it; the model only saw it one turn late.node_id, so they could not be attributed to the creating agent node.Change
has_usable_result: text needs a non-whitespace character, a structured table needs at least one field,nil/falsenever complete. Shared bynoneandauto.autorecordsFEEDBACK.EMPTY_RESULT(key<iteration>_empty_result); a nil result keepsNO_TOOLS_CALLED.check_completionreports whether it recorded feedback;finalize_iterationyields immediately in that case so the feedback is applied before the next prompt (yieldwaits for the orchestrator to apply the commit; a baresubmitonly queues it).process_artifactssetsnode_idon the artifact row.Tests
check_completion_test: 19 cases acrossnone/auto/any(empty, whitespace, nil, false, empty table, structured, numeric, pending tool calls,min_iterations, exit-tool hint).agent_empty_result_test(loop level, deterministic model stub): the model creates an artifact through a control tool, returns an empty turn, then answers once the feedback is in its context. Asserts the artifact is retained withnode_id, exactly one empty-result observation exists, and the workflow output is the real answer. A second case runs all-empty turns to the iteration limit and assertsCOMPLETED_FAILUREwith no output.control_handler_test: artifact rows carrynode_id.wippy test --profile sqlitesuite: 994 passed, 0 failed (two consecutive runs).https://claude.ai/code/session_01TLD4uyqDLGD9JfgZESaaXH