Skip to content

Add shared contracts for Atomic Skill benchmark suites - #675

Draft
yuecideng wants to merge 1 commit into
mainfrom
codex/atomic-benchmark-contracts
Draft

yuecideng wants to merge 1 commit into
mainfrom
codex/atomic-benchmark-contracts

Conversation

@yuecideng

Copy link
Copy Markdown
Contributor

Description

Atomic Skill benchmark case construction and task scoring currently share a large scenario module, while raw cases have no explicit evaluation-domain identity. This draft establishes the first shared interfaces for #669 so embodiment/provider integration and physical evaluator work can proceed against one runner and artifact contract.

  • Extract AtomicSkillCaseProvider into atomic_skill.py and expose the measured ExecutionObservation contract. Preserve the existing provider import and legacy task_result() behavior.
  • Add an opt-in PhysicalStageEvaluator interface that receives case inputs, measured observations, and motion outcomes, without a compiled trajectory or projected context. Validate ordered stage states and complete per-environment coverage; retain stage evidence and failure attribution in raw trial outcomes. Physical success cannot override failed motion or execution.
  • Add optional, validated track-level domain provenance (id, version, kind). Snapshot it onto cases and all associated lifecycle/trial records. Keep undeclared legacy domains unclassified and planner hashes separate from case content.
  • Write each track/batch manifest population before candidate evaluation. Advance the case manifest schema to version 3 for domain provenance.
  • Document the interfaces and follow-up integration boundaries in the benchmark README.

Refs #669. This is the foundation PR, not completion of the issue.

Follow-up boundaries

  • Embodiment/provider integration: official component loading, endpoint/runtime-service binding, planner compatibility, and migration of skill cases.
  • Physical evaluation: measured segment/contact collection, corrected per-skill criteria from Benchmark seven more atomic skills #642, counterexample tests, and opt-in migration of providers. This PR preserves existing scoring and does not claim to fix grasp/release/retention correctness.
  • Generalization/statistics: versioned domain generators, frozen inclusion/eligibility rules, stage/domain aggregation, and report extensions. The new domain declaration only labels an existing population; it does not generate held-out cases or prove train/evaluation separation. Existing report tables are unchanged.

Dependencies: no new packages; independent of merging #642.

Type of change

  • Enhancement (non-breaking change which improves an existing functionality)

Validation

  • /home/dex/miniconda3/envs/open/bin/python -m pytest -q -rs tests/benchmark/motion_generation: 137 passed, 1 skipped. The existing GPU test requires --run-gpu; no live simulator/GPU replay was performed.
  • Standalone contract tests run without the simulator engine: 21 passed. Coverage includes config round trips, invalid domain declarations, JSON provenance/stage serialization, incomplete stage chains, and batch coverage/order.
  • Runner/evaluator integration tests cover manifest persistence before both candidate planners, per-environment evidence, legacy bypass on opt-in, and motion/execution success gates.
  • black . using 26.3.1, black --check ., and git diff --check: passed.
  • python docs/scripts/check_api_docs.py: 2229/2229 exports documented.
  • Independent read-only review: no actionable findings.

The default Python environment lacks simulator dependencies; the existing open environment was used for the complete focused regression suite.

Context review: context.py affected --base origin/main --explain reports no affected topics. Changes are confined to benchmark scripts/tests; production simulation and Atomic Skill behavior are unchanged. The benchmark README documents the new contracts, so no agent-context revision was needed.

Screenshots

Not applicable; this change adds benchmark interfaces and raw artifact fields.

Checklist

  • I have run the black . command to format the code base.
  • I reviewed affected documentation and agent context, updated it where needed, or explained why no update was needed.
  • Public API changes are reflected in the API docs (python docs/scripts/check_api_docs.py), if applicable. No packaged public API changes.
  • I have added tests that prove the new behavior.
  • Dependencies have been updated, if applicable. No dependency changes.

@yuecideng yuecideng added enhancement New feature or request motion gen Things related to motion generation for robot atomic action atomic action related functionality labels Sep 23, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

atomic action atomic action related functionality enhancement New feature or request motion gen Things related to motion generation for robot

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant