Skip to content

[python] Use UUIDs for LeRobot dataset association keys - #11113

Merged
JingsongLi merged 5 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/lerobot-dataset-uuid
Nov 12, 2026
Merged

JingsongLi merged 5 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/lerobot-dataset-uuid

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

LeRobot sources use UUIDv7 association keys across the three-table group: __datasets.id identifies a source, and Frames/Episodes reference it through dataset_id. The frame-level index remains the original source frame index; the reader exposes dense sample positions and accepts dataset_index=N to select the Nth source.

PaimonLeRobotWriter generates and persists one UUIDv7 per source, reusing it across flushes and reopen. Single-source import follows the same layout. The importer rejects Episode controls changed after validation while accepting both scalar and list-valued Episode tasks; single-source feature statistics retain their original quantiles. Tests cover the UUID schema, writer recovery, concurrent import, tag recovery, metadata changes, and quantile-preserving reader behavior.

Validation: 405 passed, 1 skipped across aggregate, multimodal, writer, text-field, and Ray tests with LeRobot installed. Flake8 critical checks and git diff --check passed. The previous integer-key layout is not migrated here; concurrent append coordination remains separate in #11097.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found two regressions in the shared Dataset schema and the unified single-source import path.

]))
_DATASETS_SCHEMA = pa.schema([
pa.field("index", pa.int64(), nullable=False),
pa.field("index", pa.string(), nullable=False),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Keep the capture writer compatible with this schema change

PaimonLeRobotWriter also imports _DATASETS_SCHEMA, but flush() still builds a Dataset row with integer index=0 and passes it to pa.Table.from_pylist(..., schema=_DATASETS_SCHEMA) (writer.py:693–701). Changing this field to pa.string() therefore makes every non-empty writer flush/finalize fail with ArrowTypeError: Expected bytes, got a 'int' object. I reproduced this through the real writer and catalog, stubbing only the unavailable statistics dependency; the Episode snapshot was already committed, while Frames and Datasets had no committed snapshot. Restoring the integer schema made the same flow succeed.

Please either preserve a separate schema for the capture writer or update its identifier generation, flush, and resume validation together; _state_from_companion_tables() also still requires index == 0.

Comment on lines +100 to +104
from pypaimon.multimodal.lerobot.aggregate import _import_sources
return _import_sources(
connection, table_name, source, batch_size, options,
_validated_source_options(source_options), tag_name, parallelism,
dataset_name, extra_dataset_fields, schema_evolution, append)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve feature quantiles when importing a single source

Routing a single source string through _import_sources also switches its reader metadata to _merged_source_stats(). That function calls _merge_stats() even for one source, dropping q01/q99 (and the other quantiles) from non-control features such as action and observation.state. I compared the same single-source import against the base commit: the base reader preserves these keys, whereas this version exposes only min/max/mean/std/count.

This breaks existing single-source training with LeRobot's QUANTILES normalization, which requires q01/q99 in dataset.meta.stats and raises when they are missing (supported LeRobot 0.4.4 implementation). The original quantiles remain in _datasets.stats, but the reader does not restore them. Please preserve the original non-control feature quantiles for a singleton source, where no statistical merge is needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants