Skip to content

Docs: Manitos exports tool spans now #55

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
josedaniel-dev merged 1 commit into from
Oct 16, 2026
Merged

Docs: Manitos exports tool spans now #55

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -289,8 +289,9 @@ must or must not be used, health conditions (no error, no truncation, no degrada
maximum duration), phrases the reply must or must not contain (typed by the person), and an
optional human verdict (correct / incorrect). Cases keep their source trace id but outlive it when
retention purges the trace. `GET /v1/datasets/{id}/export` returns the `observer.dataset.v1` JSON
that later steps (run comparison, a CI regression command) read. Note: Manitos does not export
tool-call spans yet, so `observed.tools` is empty until it does.
that later steps (run comparison, a CI regression command) read. Note: Manitos exports one metadata-only
tool span per executed call (name, outcome, duration; never arguments or results) since its PR #36,
so `observed.tools` is filled for turns recorded after that version; earlier traces have none.

### Runs and version comparison

Expand Down
12 changes: 8 additions & 4 deletions docs/roadmap.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,10 +59,12 @@ by a person before saving.
copied from a trace; phrases are typed by a person. Runs are checked by re-scoring traces Manitos
already sends (Observer does not execute the agent).
- Done when: one action on the trace page creates an item and it can be listed and exported.
- **Dependency found:** Manitos traces have no tool-call spans (`span_type = tool`) and do not
export reply text, so tool and text expectations cannot be checked against real traces yet. Two
changes are needed on the Manitos side before step 3 can verify them: emit one tool span per
call (name, status), and decide whether a redacted reply may be exported for text checks.
- **Dependency found, tool half resolved:** Manitos traces had no tool-call spans
(`span_type = tool`). Manitos PR #36 (merged 2026-10-16) now exports one per executed call (name,
outcome, duration; no arguments or results), so tool expectations are verifiable on traces
recorded since its `manitos-server` restart. Still open: Manitos exports no reply text, so text
phrases stay `unknown`; whether a redacted reply may be exported is a privacy decision for the
user (it would break the metadata-only rule).

## Step 3: compare runs by version

Expand Down Expand Up @@ -134,3 +136,5 @@ investigate without screenshots. Read-only first.
- 2026-10-16: step 4 done. `observer-regression` CLI with exit codes 0/1/2, report
`observer.regression.v1`. Next: step 5 (OTLP). Tool and text expectations still need Manitos to
export tool spans and reply text before the gate can verify them on real traces.
- 2026-10-16: Manitos PR #36 merged: tool spans exported. Tool expectations are now verifiable on
new traces; text phrases remain `unknown`. Observer services migrated and restarted.
Loading