Skip to content

docs(contract): document what a job must write to be rollout-browsable - #12

Merged
maharajamihir merged 1 commit into
mainfrom
mihir/document-rollout-format
Aug 18, 2026
Merged

maharajamihir merged 1 commit into
mainfrom
mihir/document-rollout-format

Conversation

@maharajamihir

Copy link
Copy Markdown
Contributor

Nothing documented what a job has to write for its output to be browsable in the UI's rollout viewer, so every team reverse-engineers it from server.rs. Adds one ### Browsable rollouts subsection to docs/RECIPE_CONTRACT.md, at the end of "What labctl demands from your recipe". No restructuring of the document.

Facts covered, all verified in the code:

  • type = "eval_result" + a marker is the entry ticket. runner.rs::register_outputs parses the marker file as JSON and stores it verbatim as the artifact's metadata.result; that blob is everything the viewer sees. Without a marker there is no result key at all.
  • Browsable predicate (server.rs, eval-series builder): result contains traj_path, or gif_path, or a non-empty runs array.
  • Two layouts (resolve_rollout_paths): multi (result.runs[].subdir → <artifact_path>/<subdir>/trajectory.jsonl + .../steps/) and single (absolute result.traj_path, steps/ beside it).
  • The two hardcoded filenames, called out explicitly because they are where this goes wrong:
    • the multi shape looks for exactly trajectory.jsonl — nothing else;
    • frames must be step_%03d.png. get_artifact_frame builds steps_dir.join(format!("step_{n:03}.png")) from the requested index; it never lists or sorts the directory, so any other name is unreachable and a numbering gap is a hole the viewer can't skip.
  • The frame_count trap: get_artifact_rollout counts every *.png under steps/, so the count and the fetchable index space agree only when there is exactly one PNG per step. The production OSWorld eval also writes step_NNN_after.png, so it reports ~2x its real step count and the back half of the scrubber 404s. Documented as the trap, not as the pattern.
  • Trajectory fields the viewer reads per line: step_num, action, response (literal "<reset>" renders as a dash), reward. Lines must be in frame order — RolloutViewer.svelte indexes steps[frame] positionally for the frame caption and uses step_num for the click-to-jump target.
  • Tab label is runs[].instruction, falling back to slug.

Worked example is the real artifact eval_logs/alfred.nguyen/osworld_cua_micro_ckpt_35k_run_01a01316fcde7a609d00ff87ac353ac9 (marker completed.json, 11 runs[] entries, task_000_.../trajectory.jsonl with 65 lines and steps/ holding 129 PNGs).

Two smaller things found while verifying, both written into the doc as one-clause caveats rather than as separate issues:

  • gif_path alone makes a point look browsable in the eval series, but resolve_rollout_paths handles only runs and traj_path, so opening it 404s. The doc says to pair gif_path with traj_path.
  • The UI sends ?task=<runs[].index> while the server resolves runs by array position, so an index that doesn't match its position selects the wrong episode. The doc says to keep them equal.

Docs-only change; cargo test --all-features unaffected (36 unit + 9 smoke pass).

🤖 Generated with Claude Code

Nothing described the on-disk shape the UI's rollout viewer expects, so
every team reverse-engineers it from `server.rs`. Adds a "Browsable
rollouts" subsection to the recipe contract covering: the
`type = "eval_result"` + `marker` requirement (the marker file's parsed
JSON *is* `metadata.result`), the browsable predicate (`traj_path` /
`gif_path` / non-empty `runs[]`), the multi and single path layouts, the
four trajectory fields the viewer reads, and the tab label.

Calls out the two hardcoded filenames explicitly, since they are where
this goes wrong in practice: the multi shape looks for exactly
`trajectory.jsonl`, and frames are addressed as `step_%03d.png` built
from the frame index rather than by listing the directory. Also notes
that `frame_count` counts every `*.png` under `steps/`, so extra PNGs
(the OSWorld eval's `step_NNN_after.png`) inflate it past the fetchable
index space.

Worked example is the OSWorld CUA eval artifact in the shared registry.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@maharajamihir
maharajamihir merged commit bfcc5cf into main Aug 18, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant