Repository navigation
[python] Use UUIDs for LeRobot dataset association keys - #11113
Conversation
JingsongLi
left a comment
There was a problem hiding this comment.
Found two regressions in the shared Dataset schema and the unified single-source import path.
| ])) | ||
| _DATASETS_SCHEMA = pa.schema([ | ||
| pa.field("index", pa.int64(), nullable=False), | ||
| pa.field("index", pa.string(), nullable=False), |
There was a problem hiding this comment.
[P1] Keep the capture writer compatible with this schema change
PaimonLeRobotWriter also imports _DATASETS_SCHEMA, but flush() still builds a Dataset row with integer index=0 and passes it to pa.Table.from_pylist(..., schema=_DATASETS_SCHEMA) (writer.py:693–701). Changing this field to pa.string() therefore makes every non-empty writer flush/finalize fail with ArrowTypeError: Expected bytes, got a 'int' object. I reproduced this through the real writer and catalog, stubbing only the unavailable statistics dependency; the Episode snapshot was already committed, while Frames and Datasets had no committed snapshot. Restoring the integer schema made the same flow succeed.
Please either preserve a separate schema for the capture writer or update its identifier generation, flush, and resume validation together; _state_from_companion_tables() also still requires index == 0.
| from pypaimon.multimodal.lerobot.aggregate import _import_sources | ||
| return _import_sources( | ||
| connection, table_name, source, batch_size, options, | ||
| _validated_source_options(source_options), tag_name, parallelism, | ||
| dataset_name, extra_dataset_fields, schema_evolution, append) |
There was a problem hiding this comment.
[P2] Preserve feature quantiles when importing a single source
Routing a single source string through _import_sources also switches its reader metadata to _merged_source_stats(). That function calls _merge_stats() even for one source, dropping q01/q99 (and the other quantiles) from non-control features such as action and observation.state. I compared the same single-source import against the base commit: the base reader preserves these keys, whereas this version exposes only min/max/mean/std/count.
This breaks existing single-source training with LeRobot's QUANTILES normalization, which requires q01/q99 in dataset.meta.stats and raises when they are missing (supported LeRobot 0.4.4 implementation). The original quantiles remain in _datasets.stats, but the reader does not restore them. Please preserve the original non-control feature quantiles for a singleton source, where no statistical merge is needed.
LeRobot sources use UUIDv7 association keys across the three-table group:
__datasets.ididentifies a source, and Frames/Episodes reference it throughdataset_id. The frame-levelindexremains the original source frame index; the reader exposes dense sample positions and acceptsdataset_index=Nto select the Nth source.PaimonLeRobotWritergenerates and persists one UUIDv7 per source, reusing it across flushes and reopen. Single-source import follows the same layout. The importer rejects Episode controls changed after validation while accepting both scalar and list-valued Episode tasks; single-source feature statistics retain their original quantiles. Tests cover the UUID schema, writer recovery, concurrent import, tag recovery, metadata changes, and quantile-preserving reader behavior.Validation: 405 passed, 1 skipped across aggregate, multimodal, writer, text-field, and Ray tests with LeRobot installed. Flake8 critical checks and
git diff --checkpassed. The previous integer-key layout is not migrated here; concurrent append coordination remains separate in #11097.