Benchmark seven more atomic skills

[Yuan-Xinyi]

Description

#641 is merged, so this now targets main directly and carries only its own
commits.

Extends the suite from five skills to twelve, measured the way #641's standard
defines: an ordered list of stages per skill, each a scene measurement, chained
into one success_rate.

stage_success_rate[i] = passed(i) / reached(i)      reached(i) = passed(i-1)
success_rate          = passed(last) / total

Results

--profile coverage --device cpu, --repeat 1, one environment. Full report in
scripts/benchmark/atomic_action/BENCHMARK_REPORT.md, per-case tables in
outputs/benchmarks/.

stage_success_rate[i] = passed(i) / reached(i)      reached(i) = passed(i-1)
success_rate          = passed(last) / measured     measured = total - unsupported
coverage_rate         = measured / total

Skill	Cases	Coverage	Stage chain	success_rate
move_joints	3	100%	reached 100%	100%
move_end_effector	4	100%	reached 100%	100%
slide	3	100%	grasped 100% → slid 100% → released 100%	100%
press	3	100%	contacted 100% → pressed 100%	100%
pour	3	100%	poured 100%	100%
twist	3	100%	grasped 100% → twisted 100% → released 100%	100%
axis_align	2	100%	grasped 100% → aligned 100% → held 100%	100%
open_door	3	100%	grasped 66.67% → opened 100% → released 100%	66.67%
pick_up	48	64.58%	grasped 100% → lifted 74.19% → held 100%	74.19%
place	48	60.42%	released 100% → placed 48.28% → stable 100%	48.28%
move_held_object	48	56.25%	transported 48.15%	48.15%
hand_over	2	100%	grasped → transferred → handed_over 100% → placed 0%	0%

About 40 % of the grasp cases are ones this embodiment cannot serve

unsupported_capability now covers them: the case leaves every denominator and
lowers coverage_rate, which is what BENCHMARK_DESIGN.md section 5
prescribes and what the standard now states for success_rate.

Skill	Unsupported	In q1, near the base	With a side approach
pick_up	17 of 48	9 of 12	13 of 24
place	19 of 48	11 of 12	14 of 24
move_held_object	21 of 48	11 of 12	16 of 24

Both groups were measured, not assumed. For an object 0.18 m from the base
axis, all three distinct IK branches of the grasp pose exist and driving the arm
to each one stalls, the shoulder stopping 0.44 rad short while every other joint
tracks to 0.01 rad. For a side approach, probing one case at increasing heights,
0 of the sampled grasps are attainable at the object's own height on the ground
and 3 of 3 become attainable once lifted 0.20 m, while the top approach is
attainable at every height — the gripper is hitting the ground.

Both produce a plan that executes into thin air, and the skill's own screen
cannot see either: it accepts a candidate when IK returns a solution, which says
a configuration exists, not that the arm can get to it.
has_attainable_grasp_candidate() drives each sampled grasp in physics and
marks the case unsupported only when none of them arrives. The case set is
untouched — positions, approaches and object presets are the ones #318
shipped.

Excluding those cases raised pick_up from 52.08 % to 74.19 %, place from
37.50 % to 48.28 % and move_held_object from 27.08 % to 48.15 %, and took
grasped and released to 100 %: the cases where the skill reported a held
object the scene did not show were all unattainable ones.

Where the remaining failures are

pick_up, all eight losses at lifted: the hand closes, the arm follows
its plan, the object does not rise 4 cm. Lift over the measured cases has a
median of 0.1563 m, so these are not near misses.

place, fifteen at placed: median XY error 0.0101 m against the 1 cm
tolerance, with a tail to 0.6373 m.

move_held_object: median 0.0064 m, inside tolerance, with a tail to
0.6478 m — 14 of 26 scored cases land inside 1 cm and the rest are not close.
One further case failed its PickUp precondition and is reported as an
exception.

hand_over, both cases at placed only: 0.1048 m and 0.0680 m against the
derived 0.03 m budget, neither dropped.

open_door, open_90 at grasped: the hinge never moves.

One recorded failure per skill, captured with the suite's own
--record_video --record_failed_video --video_case_limit 1:

These are single-sample estimates

Everything is --repeat 1 and grasp sampling is stochastic. open_door shows
what that means: the same three cases, the same code, run twice, scored 100 %
and then 66.67 %, because open_90 grasped the handle on one run and not the
other. Every 100 % above carries that caveat.

The replay rate is not what produced these numbers

The physical-validation replay moved from 4 to the standard's calibrated 16
physics steps per waypoint. pick_up coverage at both rates:

Steps per waypoint	grasped	lifted	held	success_rate
4 (previous)	87.50%	59.52%	100.00%	52.08%
16 (standard)	87.50%	59.52%	100.00%	52.08%

Identical in aggregate; six of 48 cases swap verdicts, three each way.

What changed since the previous revision of this branch

Rebased onto main with #641 in it. The branch sat on the pre-rebase stack. Four
of its commits had reached main through #620 — their five benchmark files are
now byte-identical to main, and replaying them would have removed the
affordance_sampling support that landed after the branch point. Its own copy
of BENCHMARK_STANDARD.md is dropped as superseded, and so is its press/slide
tutorial strategy fix, which is already on main: both tutorials request
strategy="ik_interp" there today.

A case the embodiment cannot serve is no longer scored as a skill failure.
has_attainable_grasp_candidate() drives each sampled grasp in physics,
pre-grasp then grasp, and stops at the first one that arrives. When none does,
the case is recorded with unsupported_capability, leaves every denominator and
lowers coverage_rate instead. Supported cases pay one probe; only an
unsupported one pays for the whole candidate list. build_stage_leaderboard
reports coverage_rate and unsupported_cases alongside the stage rates, and
the standard's section 1 now states the denominator.

The measurement layer now follows the standard, which had replaced all three
of the things this branch shipped:

• StageLadder and SKILL_STAGES replace the four-level ladder
  (planning_success → motion_valid → execution_success → task_success).
  Stages are declared per skill for all fifteen; a stage is reached only when
  every earlier one passed, and recording past the first failure is ignored.
• Plan validity and replay tracking are diagnostics. The tracking gate and
  REPLAY_TRACKING_TOLERANCE_RAD are gone, and the taxonomy is the nine
  task-level reasons rather than seventeen. open_door's 4x replay rate stays,
  now justified as measurement fidelity rather than as a way past the gate.
• PRIMITIVE_* / TASK_* tolerances replace PHYSICAL_PLACE_XY_TOLERANCE_M
  (0.10) and PHYSICAL_MOVE_HELD_OBJECT_XYZ_TOLERANCE_M (0.12).
• MoveJoints and MoveEndEffector are measured on the robot after a physical
  replay and a terminal settle. From traj[:, -1] they reported 0.000000 rad
  and 0.00002 m, which describes the solver; they now measure 0.002996 rad and
  0.0006–0.0015 m.
• Twist scores the executed end-effector rotation about the knob axis, with
  the axis probed off the asset. It was failing 0/3 on the knob joint, which
  over-rotates to 2.03–3.14 rad because the gripper wedges against it; that
  value is now a diagnostic and the skill passes 3/3 at ≤ 0.0085 rad.
• HandOver's delivery budget is the derived 3 x TASK_POSITION_TOLERANCE_M
  instead of a borrowed 0.12 m.
• Release and settle stages are measured rather than assumed: hand_is_released
  compares the hand against the skill's own open and closed commands, and
  dropped_below_support applies the cross-cutting drop rule to the lowest
  point of a replay.
• replay_and_track_channels samples several scalars in one replay, so a skill
  scored on one signal can report another; a missing actuation segment now
  fails the case as invalid_case instead of silently scoring the trajectory's
  last waypoint, which is the rebound the helper exists to avoid.

Two defects on the aggregate --in_process path

Found while validating this branch, both fixed here.

_make_child_args() built the child namespace from a hand-kept list of the
five original benchmarks' arguments, so every action added here raised
AttributeError on its own case selection — door_cases, press_cases,
slide_cases, twist_cases, align_cases, pour_cases, hand_over_cases —
the smoke profile included. The namespace now starts from the selected module's
own add_benchmark_args() defaults, with the aggregate run's shared settings
applied over them.

With that fixed, --action all --in_process reached hand_over and failed with
Robot DualUr5HandOver not found: the simulator is a singleton keyed by
instance id, and leaving the previous benchmark's scene standing makes the next
one a second instance whose planner resolves against the first benchmark's
robots. HandOver's own teardown helper moved to common.release_simulation()
and the dispatcher calls it after each in-process benchmark.

--action all --profile smoke --device cpu --in_process now runs all twelve and
writes twelve reports. Neither defect touched the measurements above: every
coverage run was a direct module invocation, and both failures produced no
report rather than a wrong number.

Type of change

• New feature (non-breaking change which adds functionality)

Screenshots

N/A — reports are Markdown tables written to outputs/benchmarks/.

Checklist

• I have run the black . command to format the code base.
• I have made corresponding changes to the documentation
• Public API changes are reflected in the API docs (none; benchmark scripts)
• I have added tests that prove my feature works
• Dependencies have been updated, if applicable (none)

Validation

tests/benchmark/atomic_action, 41 tests.
test_measurement_standard.py carries 13 of them. Three of
them read BENCHMARK_STANDARD.md and check the shipped stage table, failure
taxonomy and tolerance constants against the document, so code and standard
cannot drift apart. The rest cover the stage bookkeeping the benchmarks depend
on: a stage after a failure is never reached, a closing stage fails the case,
out-of-order recording and non-taxonomy reasons raise, an unreached stage is
N/A rather than zero, diagnostics change no outcome, stage rates divide by the
cases that reached the stage, and the two shared scene rules.

test_aggregate_dispatch.py covers the --in_process path: parametrized over
every registered action, it compares the child namespace against the args.*
attributes each benchmark module actually reads, checks each action keeps its
own case-selection defaults, and asserts the simulation is released between
benchmarks. Against the namespace those commits replace, it fails for all seven
added actions.

python -m pytest tests/benchmark/atomic_action -q     # 41 passed
black --check scripts/benchmark/atomic_action tests/benchmark/atomic_action

All twelve benchmarks were run at --profile coverage; the commands are in the
report's reproducing section.

Open

1. move_held_object has a half-metre tail. Median position error is inside
   the 1 cm tolerance, but 12 of 26 scored cases are not close, the worst at
   0.6478 m. First thing this suite should be pointed at.
2. pick_up loses every one of its cases at lifted — the hand closes on a
   reachable grasp and the object does not come up. place's fifteen placed
   failures are the same kind of question one stage later.
3. press's criterion does not discriminate commanded depth. Commanding 10,
   20 or 30 mm all drive the button to 6.00 mm of its 6 mm stroke.
4. Single-sample estimates. Everything is --repeat 1; open_door scored
   100 % and then 66.67 % on two runs of the same code.
5. The unsupported probe is a benchmark-side measurement, not a skill
   change. It says the arm cannot reach a grasp; it does not tell PickUp to
   choose differently. Whether the skill should screen candidates by
   executability is a separate question, and a bigger one: nothing in these
   runs shows it picking a bad candidate when a good one existed.
6. Three skills still have no benchmark module: push_object,
   coordinated_pickment, coordinated_placement.

🤖 Generated with Claude Code

[greptile-apps]

Confidence Score: 0/5

Do not merge yet: Press and Twist setup is broken by the asset migration, and outstanding measurement defects still undermine the benchmark results.

Findings

1.  Asset migration breaks benchmark setup ▶
2.  In-process actions lack required arguments ▶
3.  Off-axis tumbles count as pours ▶
4.  Reversed pours pass the gate ▶
5.  Tabletop drops escape detection ▶
6.  Transfer passes during pickup ▶
7.  Stability excludes withdrawal motion ▶
8.  Planned effects count as observations ▶
9.  Incomplete probes exclude supported cases ▶
10.  Probe attempts share displaced objects ▶
11.  Tutorial test coverage exceeds scope ▶
12.  Videos replay a different execution ▶
13.  Failed recordings consume video quota ▶
14.  Recording changes the height diagnostic ▶
15.  Exported helpers lack parameter annotations ▶
16.  Report describes outdated replay settings ▶
17.  Release uses mismatched closed command ▶
18.  Report counts contradict the leaderboard ▶

Fix with agent prompt

### Issue 1
scripts/tutorials/atomic_action/press.py:66-67
The atomic Press and Twist benchmarks import these tutorial scene factories, but still look up `start_button_press` and `power_knob_rotation`. The migrated Microwave configuration uses `button_joint` and `knob_joint` instead. Both benchmarks therefore fail at `joint_names.index(...)` during setup, before writing a report; aggregate runs are affected too. Update both benchmark consumers alongside the factories, preferably resolving the joint from the target link.

### Issue 2
scripts/benchmark/atomic_action/run_benchmark.py:48-54
The new actions cannot run through `--in_process`: `_make_child_args()` omits their case-selection fields, but each new `run_all_benchmarks()` accesses its field directly before applying the smoke profile. For example, `--action open_door --in_process --profile smoke` raises `AttributeError` on `args.door_cases` instead of producing a report. Populate the new action-specific defaults when constructing the child namespace, preferably using each module's argument parser.

### Issue 3
scripts/benchmark/atomic_action/pour_benchmark.py:147-151
This helper returns the total relative rotation angle; the requested axis only determines its sign. A 45-degree tumble perpendicular to the pour axis therefore returns positive 45 degrees because the zero dot product is forced to a positive sign. The 45-degree benchmark can consequently credit an off-axis slip as a successful pour. Measure rotation about the requested axis and add a perpendicular-axis counterexample test.

### Issue 4
scripts/benchmark/atomic_action/pour_benchmark.py:292-294
Taking the absolute value of both rotations removes the direction check from the signed task metric. If a slipping object rotates −45 degrees around the requested axis for the +45-degree case, this reports zero error and `task_success=True`. The measurement and primitive use the same axis convention, so compare the signed achieved rotation with the signed command rather than their magnitudes.

```suggestion
        rotation_error = abs(
            trace.peak_signed_displacement - case.rotate_angle_rad
        )
```

### Issue 5
scripts/benchmark/atomic_action/hand_over_benchmark.py:238-247
The drop threshold is below the support table, so it cannot detect an object dropped onto that table during transfer. The target lies inside the table footprint, only 0.10 m above its surface. An object resting near the target at z≈0.53 m passes both the 0.12 m delivery tolerance and the z≥0.35 m check despite losing its grasp prematurely. Track grasp retention or an unexpected return to the support surface during transfer, while allowing the primitive's intentional final release.

### Issue 6
scripts/benchmark/atomic_action/hand_over_benchmark.py:313-316
`hand_balance` is distance-to-left minus distance-to-right, so a negative value means the object is closer to the left hand-the opposite of the comment's convention for the configured left-to-right transfer. Its minimum also includes the initial pickup, not just the transfer boundary. A successful source grasp followed by a missed receiving grasp can therefore report `transferred=True`, incorrectly blaming a later stage instead.

Measure both hands holding at the transfer boundary, using the primitive's actual selected arm assignment rather than a whole-replay distance minimum.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

### Issue 7
scripts/benchmark/atomic_action/place_benchmark.py:406-409
`segment_end_pose` is overwritten at every waypoint, so the stability reference is taken after the complete trajectory. Place has a separate `retract` segment after `release`: an object can topple during withdrawal and stop before the final waypoint, producing zero measured settle drift. If its XY position remains within tolerance, all stages pass despite the topple.

Capture the pose at the release boundary and compare it with the settled pose so the stability check includes movement during withdrawal.

### Issue 8
scripts/benchmark/atomic_action/pickup_benchmark.py:291-293
`held_created` comes from `result.projected_context`, which creates a hypothetical held relation during compilation without checking physical attachment. A missed physical grasp therefore counts as `grasped=True`. Place's new `released` stage similarly uses the planner's projected removal of that relation, so an object still physically held can count as released.

These values corrupt the physical stage rates and failure localization promised by the measurement standard. Keep projected state as a diagnostic, and determine grasp and release from replay observations at their respective boundaries.

### Issue 9
scripts/benchmark/atomic_action/common.py:1415-1420
The probe stops after eight IK-feasible candidates, but the generator can return 50 and PickUp considers the full set, including mirrored TCP-roll variants that this probe never checks. When the tested poses stall but a later candidate or mirrored variant is attainable, this returns false before the skill runs. PickUp, Place and MoveHeldObject then mark a supported case as unsupported and remove it from their success denominators.

Check the skill's supported candidate variants exhaustively before declaring unsupported, or distinguish an exhausted probe budget from demonstrated lack of capability.

### Issue 10
scripts/benchmark/atomic_action/common.py:1434-1437
Each attempt restores the robot but leaves the dynamic object wherever the previous attempt pushed it. All candidates were sampled against the original object pose, so later attempts can run against a displaced target. An arm configuration can then appear attainable because an earlier attempt pushed the obstruction away, or become blocked by the displaced object. The resulting unsupported classification depends on probe order rather than the original benchmark scene.

Restore the object's pose and clear its dynamics before every candidate attempt, not only when the helper returns.

### Issue 11
tests/sim/atomic_actions/test_tutorial_utils.py:713-718
This parametrization adds a tutorial-helper test case for the default Robotiq close command. The repository explicitly prohibits creating or expanding tests whose primary subject is `scripts/tutorials/**` unless tutorial coverage was requested, and no such request accompanies this change. Keep tutorial-only validation within the existing static checks, or move reusable behavior into production code and test that contract. This repository requirement must be satisfied before merging.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

### Issue 12
scripts/benchmark/atomic_action/open_door_benchmark.py:298-303
The recorder uses four physics steps per waypoint, whereas OpenDoor is scored at 64 and the other new benchmarks at 16. For these contact tasks, that changes tracking and achieved motion-not just playback speed-so a video attached to a successful case can depict a different outcome. Record during the measured replay or give the recording path the same replay cadence and terminal hold settings.

### Issue 13
scripts/benchmark/atomic_action/slide_benchmark.py:335-336
A failed or unavailable recording is appended to `video_paths` as an empty string. With `--record_video --record_failed_video` and the default case limit, a planning failure with no trajectory can consume the quota, suppress a later recordable case, and inflate the reported video count. Append only nonempty recording paths, as the other new benchmarks do. Twist has the same issue.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

(5 more issues omitted due to length limits.)

Summary

Expands the atomic-action benchmark suite to twelve skills with ordered stage measurements, coverage reporting, physical replay diagnostics, and aggregate dispatch support.

• The latest revision changes HandOver to UR10/Robotiq and separates horizontal delivery accuracy from downward settling.
• It also migrates Microwave assets, adds the FEP kinematic solver and benchmark, and consolidates render-state publication.
• The Microwave migration needs corresponding Press/Twist benchmark joint-name updates.
• The added tutorial-helper test case conflicts with the repository’s explicit testing scope.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Selected benchmark case] --> B{Capability supported?}
    B -- No --> C[Unsupported: lowers coverage]
    B -- Yes --> D[Compile and physically replay]
    D --> E[Measure ordered skill stages]
    E --> F{Current stage passes?}
    F -- No --> G[Record first failure]
    F -- Yes --> H{More stages?}
    H -- Yes --> E
    H -- No --> I[Successful case]
    C --> J[Coverage and stage leaderboard]
    G --> J
    I --> J

Loading

_{Reviews (6) · Last reviewed commit: "Fix false placement failures in the Hand..."}

[greptile-apps]

In-process actions lack required arguments

The new actions cannot run through --in_process: _make_child_args() omits their case-selection fields, but each new run_all_benchmarks() accesses its field directly before applying the smoke profile. For example, --action open_door --in_process --profile smoke raises AttributeError on args.door_cases instead of producing a report. Populate the new action-specific defaults when constructing the child namespace, preferably using each module's argument parser.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/run_benchmark.py
Line: 46-52

Comment:
**In-process actions lack required arguments**

The new actions cannot run through `--in_process`: `_make_child_args()` omits their case-selection fields, but each new `run_all_benchmarks()` accesses its field directly before applying the smoke profile. For example, `--action open_door --in_process --profile smoke` raises `AttributeError` on `args.door_cases` instead of producing a report. Populate the new action-specific defaults when constructing the child namespace, preferably using each module's argument parser.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Off-axis tumbles count as pours

This helper returns the total relative rotation angle; the requested axis only determines its sign. A 45-degree tumble perpendicular to the pour axis therefore returns positive 45 degrees because the zero dot product is forced to a positive sign. The 45-degree benchmark can consequently credit an off-axis slip as a successful pour. Measure rotation about the requested axis and add a perpendicular-axis counterexample test.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/pour_benchmark.py
Line: 146-150

Comment:
**Off-axis tumbles count as pours**

This helper returns the total relative rotation angle; the requested axis only determines its sign. A 45-degree tumble perpendicular to the pour axis therefore returns positive 45 degrees because the zero dot product is forced to a positive sign. The 45-degree benchmark can consequently credit an off-axis slip as a successful pour. Measure rotation about the requested axis and add a perpendicular-axis counterexample test.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Reversed pours pass the gate

Taking the absolute value of both rotations removes the direction check from the signed task metric. If a slipping object rotates −45 degrees around the requested axis for the +45-degree case, this reports zero error and task_success=True. The measurement and primitive use the same axis convention, so compare the signed achieved rotation with the signed command rather than their magnitudes.

Suggested change

	rotation_error = abs(
	abs(trace.peak_signed_displacement) - abs(case.rotate_angle_rad)
	)
	rotation_error = abs(
	trace.peak_signed_displacement - case.rotate_angle_rad
	)

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/pour_benchmark.py
Line: 294-296

Comment:
**Reversed pours pass the gate**

Taking the absolute value of both rotations removes the direction check from the signed task metric. If a slipping object rotates −45 degrees around the requested axis for the +45-degree case, this reports zero error and `task_success=True`. The measurement and primitive use the same axis convention, so compare the signed achieved rotation with the signed command rather than their magnitudes.

```suggestion
        rotation_error = abs(
            trace.peak_signed_displacement - case.rotate_angle_rad
        )
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Tabletop drops escape detection

The drop threshold is below the support table, so it cannot detect an object dropped onto that table during transfer. The target lies inside the table footprint, only 0.10 m above its surface. An object resting near the target at z≈0.53 m passes both the 0.12 m delivery tolerance and the z≥0.35 m check despite losing its grasp prematurely. Track grasp retention or an unexpected return to the support surface during transfer, while allowing the primitive's intentional final release.

Knowledge Base Used: Simulation lab

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/hand_over_benchmark.py
Line: 238-247

Comment:
**Tabletop drops escape detection**

The drop threshold is below the support table, so it cannot detect an object dropped onto that table during transfer. The target lies inside the table footprint, only 0.10 m above its surface. An object resting near the target at z≈0.53 m passes both the 0.12 m delivery tolerance and the z≥0.35 m check despite losing its grasp prematurely. Track grasp retention or an unexpected return to the support surface during transfer, while allowing the primitive's intentional final release.

**Knowledge Base Used:** [Simulation lab](https://app.greptile.com/dexforce/-/custom-context/knowledge-base/dexforce/embodichain/-/docs/simulation-lab.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Videos replay a different execution

The recorder uses four physics steps per waypoint, whereas OpenDoor is scored at 64 and the other new benchmarks at 16. For these contact tasks, that changes tracking and achieved motion—not just playback speed—so a video attached to a successful case can depict a different outcome. Record during the measured replay or give the recording path the same replay cadence and terminal hold settings.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/open_door_benchmark.py
Line: 286-291

Comment:
**Videos replay a different execution**

The recorder uses four physics steps per waypoint, whereas OpenDoor is scored at 64 and the other new benchmarks at 16. For these contact tasks, that changes tracking and achieved motion—not just playback speed—so a video attached to a successful case can depict a different outcome. Record during the measured replay or give the recording path the same replay cadence and terminal hold settings.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Failed recordings consume video quota

A failed or unavailable recording is appended to video_paths as an empty string. With --record_video --record_failed_video and the default case limit, a planning failure with no trajectory can consume the quota, suppress a later recordable case, and inflate the reported video count. Append only nonempty recording paths, as the other new benchmarks do. Twist has the same issue.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/slide_benchmark.py
Line: 318-319

Comment:
**Failed recordings consume video quota**

A failed or unavailable recording is appended to `video_paths` as an empty string. With `--record_video --record_failed_video` and the default case limit, a planning failure with no trajectory can consume the quota, suppress a later recordable case, and inflate the reported video count. Append only nonempty recording paths, as the other new benchmarks do. Twist has the same issue.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Recording changes the height diagnostic

final_object_z_m is read after the optional reset and recording replay, while the rotation trace and starting height describe the scored run. Enabling recording can therefore replace this diagnostic with the height from a different execution—or the reset pose if recording cannot start. Capture the final height immediately after physical validation, before resetting the scene for video.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/pour_benchmark.py
Line: 329-330

Comment:
**Recording changes the height diagnostic**

`final_object_z_m` is read after the optional reset and recording replay, while the rotation trace and starting height describe the scored run. Enabling recording can therefore replace this diagnostic with the height from a different execution—or the reset pose if recording cannot start. Capture the final height immediately after physical validation, before resetting the scene for video.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Exported helpers lack parameter annotations

The exported object_axis_angle_rad() leaves all three parameters unannotated, violating the repository directive to fully annotate public APIs. Add RigidObject and tensor parameter annotations, using type-only imports where appropriate. The exported signed_rotation_about_axis_rad() in Pour has the same omission. This repository requirement must be satisfied before merging.

Context Used: CLAUDE.md (source)

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/axis_align_benchmark.py
Line: 115

Comment:
**Exported helpers lack parameter annotations**

The exported `object_axis_angle_rad()` leaves all three parameters unannotated, violating the repository directive to fully annotate public APIs. Add `RigidObject` and tensor parameter annotations, using type-only imports where appropriate. The exported `signed_rotation_about_axis_rad()` in Pour has the same omission. This repository requirement must be satisfied before merging.

**Context Used:** CLAUDE.md ([source](https://github.com/dexforce/embodichain/blob/main/CLAUDE.md))

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Report describes outdated replay settings

The report presents 16 steps per waypoint as OpenDoor's chosen replay rate, but the committed benchmark uses 64 because 16 fails its tracking gate. Together with the opening claim that the changes are uncommitted, this leaves readers unable to tie the reported results to the reviewed implementation. Identify the measured revision and settings, and separate historical experiments from the current benchmark configuration.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/BENCHMARK_REPORT.md
Line: 73-86

Comment:
**Report describes outdated replay settings**

The report presents 16 steps per waypoint as OpenDoor's chosen replay rate, but the committed benchmark uses 64 because 16 fails its tracking gate. Together with the opening claim that the changes are uncommitted, this leaves readers unable to tie the reported results to the reviewed implementation. Identify the measured revision and settings, and separate historical experiments from the current benchmark configuration.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Something wrong

[greptile-apps]

Transfer passes during pickup

hand_balance is distance-to-left minus distance-to-right, so a negative value means the object is closer to the left hand—the opposite of the comment's convention for the configured left-to-right transfer. Its minimum also includes the initial pickup, not just the transfer boundary. A successful source grasp followed by a missed receiving grasp can therefore report transferred=True, incorrectly blaming a later stage instead.

Measure both hands holding at the transfer boundary, using the primitive's actual selected arm assignment rather than a whole-replay distance minimum.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/hand_over_benchmark.py
Line: 292-295

Comment:
**Transfer passes during pickup**

`hand_balance` is distance-to-left minus distance-to-right, so a negative value means the object is closer to the left hand—the opposite of the comment's convention for the configured left-to-right transfer. Its minimum also includes the initial pickup, not just the transfer boundary. A successful source grasp followed by a missed receiving grasp can therefore report `transferred=True`, incorrectly blaming a later stage instead.

Measure both hands holding at the transfer boundary, using the primitive's actual selected arm assignment rather than a whole-replay distance minimum.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

Stability excludes withdrawal motion

segment_end_pose is overwritten at every waypoint, so the stability reference is taken after the complete trajectory. Place has a separate retract segment after release: an object can topple during withdrawal and stop before the final waypoint, producing zero measured settle drift. If its XY position remains within tolerance, all stages pass despite the topple.

Capture the pose at the release boundary and compare it with the settled pose so the stability check includes movement during withdrawal.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/place_benchmark.py
Line: 384-387

Comment:
**Stability excludes withdrawal motion**

`segment_end_pose` is overwritten at every waypoint, so the stability reference is taken after the complete trajectory. Place has a separate `retract` segment after `release`: an object can topple during withdrawal and stop before the final waypoint, producing zero measured settle drift. If its XY position remains within tolerance, all stages pass despite the topple.

Capture the pose at the release boundary and compare it with the settled pose so the stability check includes movement during withdrawal.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Planned effects count as observations

held_created comes from result.projected_context, which creates a hypothetical held relation during compilation without checking physical attachment. A missed physical grasp therefore counts as grasped=True. Place's new released stage similarly uses the planner's projected removal of that relation, so an object still physically held can count as released.

These values corrupt the physical stage rates and failure localization promised by the measurement standard. Keep projected state as a diagnostic, and determine grasp and release from replay observations at their respective boundaries.

Knowledge Base Used: Simulation lab

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/pickup_benchmark.py
Line: 268-270

Comment:
**Planned effects count as observations**

`held_created` comes from `result.projected_context`, which creates a hypothetical held relation during compilation without checking physical attachment. A missed physical grasp therefore counts as `grasped=True`. Place's new `released` stage similarly uses the planner's projected removal of that relation, so an object still physically held can count as released.

These values corrupt the physical stage rates and failure localization promised by the measurement standard. Keep projected state as a diagnostic, and determine grasp and release from replay observations at their respective boundaries.

**Knowledge Base Used:** [Simulation lab](https://app.greptile.com/dexforce/-/custom-context/knowledge-base/dexforce/embodichain/-/docs/simulation-lab.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Release uses mismatched closed command

The engine uses close_qpos=0.04, but this call uses the helper's default 0.036. With an open command of zero, an achieved jaw position of 0.019 is closer to the actual open command than to 0.04, yet this check reports release failure because it compares against 0.036. This can misclassify borderline releases. Use the same explicit close value here, or pass the original command tensors into the evaluator.

Suggested change

	hand_open, hand_close = get_hand_open_close_qpos(robot)
	hand_open, hand_close = get_hand_open_close_qpos(robot, close_qpos=0.04)

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/open_door_benchmark.py
Line: 288

Comment:
**Release uses mismatched closed command**

The engine uses `close_qpos=0.04`, but this call uses the helper's default `0.036`. With an open command of zero, an achieved jaw position of `0.019` is closer to the actual open command than to `0.04`, yet this check reports release failure because it compares against `0.036`. This can misclassify borderline releases. Use the same explicit close value here, or pass the original command tensors into the evaluator.

```suggestion
        hand_open, hand_close = get_hand_open_close_qpos(robot, close_qpos=0.04)
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Incomplete probes exclude supported cases

The probe stops after eight IK-feasible candidates, but the generator can return 50 and PickUp considers the full set, including mirrored TCP-roll variants that this probe never checks. When the tested poses stall but a later candidate or mirrored variant is attainable, this returns false before the skill runs. PickUp, Place and MoveHeldObject then mark a supported case as unsupported and remove it from their success denominators.

Check the skill's supported candidate variants exhaustively before declaring unsupported, or distinguish an exhausted probe budget from demonstrated lack of capability.

Knowledge Base Used: GraspKit workflows

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/common.py
Line: 1415-1420

Comment:
**Incomplete probes exclude supported cases**

The probe stops after eight IK-feasible candidates, but the generator can return 50 and PickUp considers the full set, including mirrored TCP-roll variants that this probe never checks. When the tested poses stall but a later candidate or mirrored variant is attainable, this returns false before the skill runs. PickUp, Place and MoveHeldObject then mark a supported case as unsupported and remove it from their success denominators.

Check the skill's supported candidate variants exhaustively before declaring unsupported, or distinguish an exhausted probe budget from demonstrated lack of capability.

**Knowledge Base Used:** [GraspKit workflows](https://app.greptile.com/dexforce/-/custom-context/knowledge-base/dexforce/embodichain/-/docs/graspkit.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Probe attempts share displaced objects

Each attempt restores the robot but leaves the dynamic object wherever the previous attempt pushed it. All candidates were sampled against the original object pose, so later attempts can run against a displaced target. An arm configuration can then appear attainable because an earlier attempt pushed the obstruction away, or become blocked by the displaced object. The resulting unsupported classification depends on probe order rather than the original benchmark scene.

Restore the object's pose and clear its dynamics before every candidate attempt, not only when the helper returns.

Knowledge Base Used: Simulation lab

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/common.py
Line: 1434-1437

Comment:
**Probe attempts share displaced objects**

Each attempt restores the robot but leaves the dynamic object wherever the previous attempt pushed it. All candidates were sampled against the original object pose, so later attempts can run against a displaced target. An arm configuration can then appear attainable because an earlier attempt pushed the obstruction away, or become blocked by the displaced object. The resulting unsupported classification depends on probe order rather than the original benchmark scene.

Restore the object's pose and clear its dynamics before every candidate attempt, not only when the helper returns.

**Knowledge Base Used:** [Simulation lab](https://app.greptile.com/dexforce/-/custom-context/knowledge-base/dexforce/embodichain/-/docs/simulation-lab.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Report counts contradict the leaderboard

The leaderboard now excludes unsupported cases from evaluated_cases, but summarize_video_recording() still counts every result as evaluated and every unsuccessful result as a failure. One successful case and one unsupported case therefore produce evaluated_cases=1 with 100% success in the leaderboard, but evaluated=2, success=1, failure=1 in the same report's notes. This makes the report harder to interpret. Report unsupported cases separately in the shared summary and use the same measured-case population as the leaderboard.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/common.py
Line: 2000-2001

Comment:
**Report counts contradict the leaderboard**

The leaderboard now excludes unsupported cases from `evaluated_cases`, but `summarize_video_recording()` still counts every result as evaluated and every unsuccessful result as a failure. One successful case and one unsupported case therefore produce `evaluated_cases=1` with 100% success in the leaderboard, but `evaluated=2, success=1, failure=1` in the same report's notes. This makes the report harder to interpret. Report unsupported cases separately in the shared summary and use the same measured-case population as the leaderboard.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[yuecideng]

Additional inline findings that are not already covered by the existing review threads.

[yuecideng]

[P1] Measure retention from the physical replay

still_held is derived from result.projected_context, which is a hypothetical state produced during side-effect-free compilation before this replay runs. MoveHeldObject preserves that symbolic held relation whenever planning succeeds, so this guard remains true even if the gripper physically loses the object. If the dropped object happens to settle within the pose tolerances below, the transported stage can pass. Determine retention/drop from replay observations at the action boundary and keep projected state diagnostic-only.

[yuecideng]

[P2] Verify that the object is still held

This held stage checks only that the object never fell more than the drop margin and that its settled axis remains aligned. An object released onto the same support surface can satisfy both conditions while no longer being grasped, so AxisAlign can report full success after losing the object. The standard defines this stage as still held; add physical grasp-retention evidence (for example contact or stable hand/object relative pose) at the held boundary.

[yuecideng]

[P2] Apply the cross-cutting drop rule to Pour

The replay samples only the object's rotation, so a case that reaches the commanded angle and then drops the object during the return or terminal hold still records poured=True. BENCHMARK_STANDARD.md requires any skill holding an object to fail the current stage with object_dropped when the replay crosses the drop margin. Track object height in the same replay and include that result in this verdict.

[yuecideng]

[P2] Score placed at the actuation boundary

place_xy_error_m is computed from final_obj_position, which replay_trajectory_for_physical_validation() returns only after its terminal hold, even though segment_end_pose was captured during replay. An object can therefore miss the target when Place stops acting, roll into tolerance during settle, and pass placed; the reverse is attributed to the wrong stage. Compute this stage from the object pose at the placing/release boundary and reserve the settled pose for stable.

[greptile-apps]

Asset migration breaks benchmark setup

The atomic Press and Twist benchmarks import these tutorial scene factories, but still look up start_button_press and power_knob_rotation. The migrated Microwave configuration uses button_joint and knob_joint instead. Both benchmarks therefore fail at joint_names.index(...) during setup, before writing a report; aggregate runs are affected too. Update both benchmark consumers alongside the factories, preferably resolving the joint from the target link.

Knowledge Base Used: Asset and dataset management

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/tutorials/atomic_action/press.py
Line: 66-67

Comment:
**Asset migration breaks benchmark setup**

The atomic Press and Twist benchmarks import these tutorial scene factories, but still look up `start_button_press` and `power_knob_rotation`. The migrated Microwave configuration uses `button_joint` and `knob_joint` instead. Both benchmarks therefore fail at `joint_names.index(...)` during setup, before writing a report; aggregate runs are affected too. Update both benchmark consumers alongside the factories, preferably resolving the joint from the target link.

**Knowledge Base Used:** [Asset and dataset management](https://app.greptile.com/dexforce/-/custom-context/knowledge-base/dexforce/embodichain/-/docs/asset-data-management.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Tutorial test coverage exceeds scope

This parametrization adds a tutorial-helper test case for the default Robotiq close command. The repository explicitly prohibits creating or expanding tests whose primary subject is scripts/tutorials/** unless tutorial coverage was requested, and no such request accompanies this change. Keep tutorial-only validation within the existing static checks, or move reusable behavior into production code and test that contract. This repository requirement must be satisfied before merging.

Context Used: AGENTS.md (source)

Prompt To Fix With AI

This is a comment left during a code review.
Path: tests/sim/atomic_actions/test_tutorial_utils.py
Line: 713-718

Comment:
**Tutorial test coverage exceeds scope**

This parametrization adds a tutorial-helper test case for the default Robotiq close command. The repository explicitly prohibits creating or expanding tests whose primary subject is `scripts/tutorials/**` unless tutorial coverage was requested, and no such request accompanies this change. Keep tutorial-only validation within the existing static checks, or move reusable behavior into production code and test that contract. This repository requirement must be satisfied before merging.

**Context Used:** AGENTS.md ([source](https://github.com/dexforce/embodichain/blob/main/AGENTS.md))

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!