Repository navigation
[MaxText] Add single-host and multislice verification for constant_padded elastic resizes. - #6372
Open
copybara-service[bot] wants to merge 1 commit into
Open
copybara-service[bot] wants to merge 1 commit into
copybara-service[bot] wants to merge 1 commit into
Conversation
…added` elastic resizes. The convergence guarantee of the constant-global-batch elastic modes (`elastic_global_batch_size_mode` `constant_padded` and `constant_grad_accum`) is that a resized run trains on exactly the same real tokens per step as the full-allocation run, with the same optimizer mathematics. Until now the only verification was manual. This change adds: * `elastic_simulate_job_devices` (testing only): sizes the logical global batch for a pretended job and plans and pads the physical batch for the devices present, without Pathways. The synthetic data iterator now generates only the logical rows and appends zero rows, so every padded run trains on the same rows as the unpadded batch of the pretended job. * `tests/integration/elastic_constant_padded_test.py` (TPU, single host): trains the same logical batch under the legacy factoring (wider per-device batch), the fixed per-device batch factoring (more accumulation steps), a bounded search and `constant_grad_accum`, and asserts identical `learning/total_weights` and matching loss and gradient norm. * `tests/utils/elastic_resize_checks.py`: the assertion tool for the multislice end-to-end test. Given the metrics files of one or more full-allocation runs and of a resized run it requires exact `learning/total_weights` at every common step and keeps `learning/loss` and `learning/grad_norm` within a multiple of the seed-to-seed noise floor measured between the reference runs (or an absolute tolerance with a single reference). Rewound steps are handled by keeping the later entry. Unit-tested in `tests/unit/elastic_resize_checks_test.py`. Tested: `tests/unit/elastic_resize_checks_test.py`, `tests/unit/configs_value_test.py` (`test_elastic_simulate_job_devices_plans_without_pathways`), `tests/unit/synthetic_data_test.py`, and `tests/integration/elastic_constant_padded_test.py` on a TPU VM. PiperOrigin-RevId: 1019389712
copybara-service
Bot
requested review from
A9isha,
NuojCheng,
RissyRan,
SurbhiJainUSC,
abhinavclemson,
aireenmei,
bvandermoon,
darisoy,
dipannita08,
gagika,
gobbleturk,
hengtaoguo,
igorts-git,
jacoguzo,
jiangjy1982,
khatwanimohit,
shralex,
shuningjin and
xibinliu
as code owners
November 16, 2026 20:26
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[MaxText] Add single-host and multislice verification for
constant_paddedelastic resizes.The convergence guarantee of the constant-global-batch elastic modes (
elastic_global_batch_size_modeconstant_paddedandconstant_grad_accum) is that a resized run trains on exactly the same real tokens per step as the full-allocation run, with the same optimizer mathematics. Until now the only verification was manual. This change adds:elastic_simulate_job_devices(testing only): sizes the logical global batch for a pretended job and plans and pads the physical batch for the devices present, without Pathways. The synthetic data iterator now generates only the logical rows and appends zero rows, so every padded run trains on the same rows as the unpadded batch of the pretended job.tests/integration/elastic_constant_padded_test.py(TPU, single host): trains the same logical batch under the legacy factoring (wider per-device batch), the fixed per-device batch factoring (more accumulation steps), a bounded search andconstant_grad_accum, and asserts identicallearning/total_weightsand matching loss and gradient norm.tests/utils/elastic_resize_checks.py: the assertion tool for the multislice end-to-end test. Given the metrics files of one or more full-allocation runs and of a resized run it requires exactlearning/total_weightsat every common step and keepslearning/lossandlearning/grad_normwithin a multiple of the seed-to-seed noise floor measured between the reference runs (or an absolute tolerance with a single reference). Rewound steps are handled by keeping the later entry. Unit-tested intests/unit/elastic_resize_checks_test.py.Tested:
tests/unit/elastic_resize_checks_test.py,tests/unit/configs_value_test.py(test_elastic_simulate_job_devices_plans_without_pathways),tests/unit/synthetic_data_test.py, andtests/integration/elastic_constant_padded_test.pyon a TPU VM.