[python] Preserve LeRobot recording metadata in direct writer - #9708
[python] Preserve LeRobot recording metadata in direct writer#9708YannByron wants to merge 4 commits into
Conversation
Write episodes, tasks, info, and native LeRobot statistics alongside directly captured frames. Restore writer state from companion metadata without scanning the frame table. Co-Authored-By: Codex <noreply@anthropic.com> AI-Model: gpt-5 Co-Authored-By: Codex <noreply@ai-tool.com> Co-Authored-By: Codex <noreply@openai.com> AI-Contributed/Feature: 518/518 AI-Contributed/UT: 330/330
JingsongLi
left a comment
There was a problem hiding this comment.
Two regressions found in the direct writer changes: optional-dependency handling in the test matrix and aggregation of numeric features whose names contain image.
Keep optional-dependency tests isolated and aggregate statistics by declared feature dtype so numeric fields containing image in their names remain valid. Co-Authored-By: Codex <noreply@anthropic.com> AI-Model: gpt-5 Co-Authored-By: Codex <noreply@ai-tool.com> Co-Authored-By: Codex <noreply@openai.com> AI-Contributed/Feature: 19/19 AI-Contributed/UT: 45/45
Persist an ordered subtask vocabulary in a managed companion table, validate frame-level subtask indices, and restore the immutable mapping when appending to an existing table group. Co-Authored-By: Codex <noreply@anthropic.com> AI-Model: gpt-5 Co-Authored-By: Codex <noreply@ai-tool.com> Co-Authored-By: Codex <noreply@openai.com> AI-Contributed/Feature: 170/170 AI-Contributed/UT: 129/129
JingsongLi
left a comment
There was a problem hiding this comment.
The previous two findings are fixed. One integration issue remains with the PaimonLeRobotDataset reader now merged into master in #9498.
| "tasks_table": pa.Table.from_pylist( | ||
| [], schema=_EMPTY_TASKS_SCHEMA), |
There was a problem hiding this comment.
[P1] Preserve the text index metadata for tasks and subtasks
These companion tables store labels as ordinary columns without pandas index metadata, but the PaimonLeRobotDataset reader merged in #9498 restores a DataFrame and uses its index as the task/subtask text (_component_dataframe and _index_names). Consequently, a recording with task="pick" is decoded as task "0", and its first frame fails _validate_control_row with Paimon task at LeRobot index 0 is not assigned to Episode 0, because the episode correctly lists "pick". Subtask labels are likewise decoded as numeric strings. I reproduced this using the PR writer's persisted output and the current master reader; restoring the text indexes makes the same frame pass validation. Please preserve the native text-index pandas metadata for both companion schemas using the existing _prepare_metadata_tables mechanism, or make the reader explicitly support these plain text columns, and add a writer-to-reader regression test.
Store task and subtask labels as Pandas indexes so the Paimon LeRobot reader restores their text mappings correctly.\n\nCo-Authored-By: Codex <noreply@anthropic.com>\nAI-Model: gpt-5\nAI-Contributed/Feature: 27/27\nAI-Contributed/UT: 13/13 Co-Authored-By: Codex <noreply@ai-tool.com> Co-Authored-By: Codex <noreply@openai.com> AI-Contributed/Feature: 27/27 AI-Contributed/UT: 13/13
Summary
Follow up on #9665 by preserving native LeRobot recording metadata when frames are captured directly into Paimon. The writer now publishes a managed table group containing frames, episodes, tasks, optional subtasks, dataset information, and statistics without creating an intermediate LeRobot dataset directory.
Changes
__episodes,__tasks,__info, and__statscompanion tables for direct capture.task_index.__subtasks, validate frame-levelsubtask_indexvalues, and restore the vocabulary on resume.Testing
observation.image_embeddingfeature across multiple episodes and resume.git diff --check.Notes
subtask_indexrequires an ordered, uniquesubtasksvocabulary. Existing groups restore it from__subtasksand reject conflicting vocabularies.