Skip to content

[MaxText] Add single-host and multislice verification for constant_padded elastic resizes. - #6372

Open
copybara-service[bot] wants to merge 1 commit into
mainfrom
test_1019389712
Open

copybara-service[bot] wants to merge 1 commit into
mainfrom
test_1019389712

Conversation

@copybara-service

Copy link
Copy Markdown
Contributor

[MaxText] Add single-host and multislice verification for constant_padded elastic resizes.

The convergence guarantee of the constant-global-batch elastic modes (elastic_global_batch_size_mode constant_padded and constant_grad_accum) is that a resized run trains on exactly the same real tokens per step as the full-allocation run, with the same optimizer mathematics. Until now the only verification was manual. This change adds:

  • elastic_simulate_job_devices (testing only): sizes the logical global batch for a pretended job and plans and pads the physical batch for the devices present, without Pathways. The synthetic data iterator now generates only the logical rows and appends zero rows, so every padded run trains on the same rows as the unpadded batch of the pretended job.
  • tests/integration/elastic_constant_padded_test.py (TPU, single host): trains the same logical batch under the legacy factoring (wider per-device batch), the fixed per-device batch factoring (more accumulation steps), a bounded search and constant_grad_accum, and asserts identical learning/total_weights and matching loss and gradient norm.
  • tests/utils/elastic_resize_checks.py: the assertion tool for the multislice end-to-end test. Given the metrics files of one or more full-allocation runs and of a resized run it requires exact learning/total_weights at every common step and keeps learning/loss and learning/grad_norm within a multiple of the seed-to-seed noise floor measured between the reference runs (or an absolute tolerance with a single reference). Rewound steps are handled by keeping the later entry. Unit-tested in tests/unit/elastic_resize_checks_test.py.

Tested: tests/unit/elastic_resize_checks_test.py, tests/unit/configs_value_test.py (test_elastic_simulate_job_devices_plans_without_pathways), tests/unit/synthetic_data_test.py, and tests/integration/elastic_constant_padded_test.py on a TPU VM.

…added` elastic resizes.

The convergence guarantee of the constant-global-batch elastic modes (`elastic_global_batch_size_mode` `constant_padded` and `constant_grad_accum`) is that a resized run trains on exactly the same real tokens per step as the full-allocation run, with the same optimizer mathematics. Until now the only verification was manual. This change adds:

*   `elastic_simulate_job_devices` (testing only): sizes the logical global batch for a pretended job and plans and pads the physical batch for the devices present, without Pathways. The synthetic data iterator now generates only the logical rows and appends zero rows, so every padded run trains on the same rows as the unpadded batch of the pretended job.
*   `tests/integration/elastic_constant_padded_test.py` (TPU, single host): trains the same logical batch under the legacy factoring (wider per-device batch), the fixed per-device batch factoring (more accumulation steps), a bounded search and `constant_grad_accum`, and asserts identical `learning/total_weights` and matching loss and gradient norm.
*   `tests/utils/elastic_resize_checks.py`: the assertion tool for the multislice end-to-end test. Given the metrics files of one or more full-allocation runs and of a resized run it requires exact `learning/total_weights` at every common step and keeps `learning/loss` and `learning/grad_norm` within a multiple of the seed-to-seed noise floor measured between the reference runs (or an absolute tolerance with a single reference). Rewound steps are handled by keeping the later entry. Unit-tested in `tests/unit/elastic_resize_checks_test.py`.

Tested: `tests/unit/elastic_resize_checks_test.py`, `tests/unit/configs_value_test.py` (`test_elastic_simulate_job_devices_plans_without_pathways`), `tests/unit/synthetic_data_test.py`, and `tests/integration/elastic_constant_padded_test.py` on a TPU VM.
PiperOrigin-RevId: 1019389712

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant