Repository navigation
Extend log_required_ragged_buffer_factor to the ragged all-to-all path (no new flag) - #6299
Shuwen-Fang wants to merge 1 commit into
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request adds a diagnostic tool for the ragged all-to-all Mixture-of-Experts (MoE) path. By introducing a new configuration flag, users can now identify the optimal buffer size required to avoid token drops, facilitating better benchmarking of routing skew and load-balancing strategies without altering the default execution behavior. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
🤖 Hi @Shuwen-Fang, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
There was a problem hiding this comment.
📋 Review Summary
Verdict: Bypasses existing MoE probe architecture, introduces uncoordinated host print storm in shard_map, and lacks automated tests.
Top issues:
- [Alignment] MaxText already implements
log_required_ragged_buffer_factor(inconfigs/types.py,base.yml, andlayers/moe.py) with complete plumbing for intermediate sowing and structured per-step logging intrain.py. Introducinglog_ra2a_required_rbfcreates a redundant second configuration flag and duplicates this diagnostic concept. → Deletelog_ra2a_required_rbf, populateRouteOutput.required_rbfinsidera2a_and_routeunderlog_required_ragged_buffer_factor, and relax the config validator intypes.py(line 4665) to permit ragged all-to-all. - [Perf] Calling
jax.debug.printwithordered=Falseinsideshard_mapforces every device across the mesh to execute host callbacks on every layer and step, generating a severe I/O storm on large meshes. Furthermore, omitting step number and layer metadata causes outputs from all devices and layers to interleave as scrambled, anonymous floats on stdout. → Removejax.debug.printand route the computed required factor throughRouteOutput.required_rbfto leverageself.sow(nnx.Intermediate, "moe_required_rbf", ...)and_maybe_log_required_rbfintrain.py. - [Bug]
all_shards_group_sizesis all-gathered acrossbatch_axisonly; on topologies with orthogonal mesh axes (such as data or FSDP parallelism),_reqis not reduced globally across replicas. Different devices will compute and report conflicting required buffer factors for the same layer. → Applyjax.lax.pmax(required, tuple(self.mesh.axis_names))whenself.meshis present, consistent withRoutedMoE.required_ragged_buffer_factor. - [Tests] The PR adds no unit or integration tests for the required buffer factor computation or config plumbing. → Add a unit test in
tests/unit/moe_test.pythat passes an imbalanced expert assignment and verifies the computed required RBF matches the expected traffic peak.
Inferred spec & validation
- Sources: PR description; code before/after (
src/maxtext/configs/base.yml,src/maxtext/configs/types.py,src/maxtext/layers/moe.py). - Problem: The ragged all-to-all MoE path lacked probe logging for the minimum
ragged_buffer_factorrequired to prevent token drops under routing skew.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): Calculate per-layer minimum ragged_buffer_factor to avoid token drops on ragged all-to-all path (num_ep > 1, is_batch_sharded_by_expert=True) |
pmax across non-EP axes) |
❌ none |
| R2 (stated): Report or log the per-layer required RBF when enabled | 🩹 symptom only (emits via uncoordinated jax.debug.print in shard_map) |
❌ none |
R3 (stated): When disabled (false, default), model execution and compiled HLO remain unchanged |
✅ met | |
C1 (inferred): Architectural alignment with existing MaxText probe infrastructure (log_required_ragged_buffer_factor) |
❌ violated (duplicates flag and bypasses logging pipeline) | ❌ none |
C2 (inferred): Multi-device execution safety in shard_map hot path without host callback storms or interleaved stdout |
❌ violated (jax.debug.print executed per core on every layer/step) |
❌ none |
| all_shards_group_sizes = jax.lax.all_gather(reshaped_group_sizes, axis_name=batch_axis) | ||
| if self.config.log_ra2a_required_rbf: | ||
| _req = jnp.max(jnp.sum(all_shards_group_sizes, axis=0)) / jnp.shape(x)[0] |
There was a problem hiding this comment.
🟠 This implementation diverges from the existing log_required_ragged_buffer_factor infrastructure (RouteOutput.required_rbf, self.sow('moe_required_rbf', ...), and _maybe_log_required_rbf in train.py). Additionally, executing jax.debug.print with ordered=False inside shard_map triggers uncoordinated host callbacks across all mesh devices on every layer and step, scrambling stdout and degrading performance. Please remove jax.debug.print, reduce across self.mesh.axis_names with jax.lax.pmax, and assign to RouteOutput.required_rbf under self.config.log_required_ragged_buffer_factor.
There was a problem hiding this comment.
Fixed in 78a8c58: removed jax.debug.print and log_ra2a_required_rbf, and populated RouteOutput.required_rbf in ra2a_and_route via self.required_ragged_buffer_factor(...) (with mesh-wide jax.lax.pmax) under self.config.log_required_ragged_buffer_factor, plus added multi-device unit tests in tests/unit/moe_test.py.
| log_ra2a_required_rbf: bool = Field( | ||
| False, description="Debug: print the per-layer ragged_buffer_factor needed to avoid drops (ragged a2a path)." | ||
| ) |
There was a problem hiding this comment.
🟠 Redundant configuration flag. MoEGeneral already defines log_required_ragged_buffer_factor (line 1225) for this diagnostic purpose. Instead of adding log_ra2a_required_rbf, remove this field and update the validator in types.py (line 4665) to permit log_required_ragged_buffer_factor on ragged all-to-all configurations.
There was a problem hiding this comment.
Fixed in 78a8c58: deleted log_ra2a_required_rbf from base.yml and types.py and relaxed the log_required_ragged_buffer_factor validator in types.py to accept ragged all-to-all (use_ring_of_experts=False, sparse_matmul=True).
|
@gemini-cli /review |
|
🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
|
@gemini-cli /review |
|
🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
There was a problem hiding this comment.
📋 Review Summary
Verdict: No significant issues found.
Inferred spec & validation
- Sources: PR description; code before/after.
- Problem:
log_required_ragged_buffer_factorprobe is not supported for the ragged all-to-all MoE routing path (use_ring_of_experts=False).
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): Allow log_required_ragged_buffer_factor=True for ragged all-to-all with sparse_matmul=True. |
✅ | test_validator_rules_for_log_required_ragged_buffer_factor ✅ |
| R2 (stated): Compute exact pre-truncation peak demand and sow as "moe_required_rbf". | ✅ | test_ra2a_sows_mesh_max_required_rbf ✅ |
R3 (stated): Support both num_ep == 1 and num_ep > 1 cases (including FSDP-only sharding). |
✅ | test_ra2a_ep1_and_unsharded_batch_required_rbf ✅ |
C1 (stated): Zero overhead when disabled (log_required_ragged_buffer_factor=False). |
✅ preserved | ❌ none (verified by manual E2E parity run) |
C2 (inferred): Maintain existing config restrictions for ring of experts (e.g. use_ragged_sort). |
✅ preserved | test_validator_rules_for_log_required_ragged_buffer_factor ✅ |
|
@gemini-cli /review |
|
🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
85f7c74 to
6dee19c
Compare
Summary
Extend
log_required_ragged_buffer_factorto the ragged all-to-all path (no new flag).log_required_ragged_buffer_factorflag & structured logger: PopulatesRouteOutput.required_rbfinsideRoutedMoE.ra2a_and_route(use_ring_of_experts=False) viaself.required_ragged_buffer_factor(...)(mesh-widejax.lax.pmaxovertuple(self.mesh.axis_names)), emitting the standardREQUIRED_RBF step=... max=... per_layer=[...]log line once per step intrain.pywith zero device-to-host sync inside the layer.ragged_all_to_allbuffers: Inra2a_and_route, onlynum_ep > 1withis_batch_sharded_by_expertallocates aragged_buffer_factor-sized receive buffer (get_ragged_buffer_size). Whennum_ep == 1oris_batch_sharded_by_ep(inputs)isFalse(batch size 1 decode), noragged_all_to_allreceive buffer exists (ragged_buffer_factorcannot truncate tokens), sorequired_rbfstaysNoneand"moe_required_rbf"is not sown.src/maxtext/configs/types.py): Relaxeslog_required_ragged_buffer_factor=Trueto allowuse_ring_of_experts=Falsewheneversparse_matmul=True(while still requiringuse_ragged_sort=Truewhenuse_ring_of_experts=True).Validation
1. TPU v6e-8 Probe Parity & 128-Row-Aligned Drop Boundary (
85f7c74d2)(a) Exact parity with prior
log_ra2a_required_rbfdebug probe (17e928f11)Verified over 4 steps × 4 layers (
EP=8,per_device_batch_size=2,max_target_length=1024,top_k=10→balanced_size = 20,480rows/shard,use_ring_of_experts=false,seed=0) using#6298synthetic routing:L=16/shard),lognormal(sigma=0.9, row_tau=1.3, row_len=768): all 16(step, layer)values match identically (step=0: [1.3676, 1.5240, 1.2921, 1.2648],step=1: [1.2732, 1.3465, 1.2262, 1.1531],step=2: [1.1669, 1.6920, 1.2893, 1.1545],step=3: [1.1877, 1.4992, 1.3093, 1.2880]).L=64/shard), empirical Qwen3.5 profile (qwen397b_routing_profile.npz,row_tau=1.3, row_len=768): all 16(step, layer)values match identically (step=0: [1.3205, 1.0804, 1.2630, 1.3212],step=1: [1.2775, 1.1416, 1.2175, 1.3247],step=2: [1.3638, 1.0943, 1.3478, 1.2827],step=3: [1.2427, 1.0994, 1.1789, 1.3839]; atEP=32on SFS this same profile gives2.4823).(b) Exact 128-row-aligned drop-prediction boundary on the
ra2apathOn TPU v6e-8 (
EP=8, 2-layerqwen3.5-397b-a17b,base_emb_dim=2048,num_experts=128,use_ring_of_experts=false,use_ragged_sort=true,balanced_size = 20,480rows/shard), the probe reports:step=0:[1.5240, 1.2921],step=1:[1.3465, 1.2262],step=2:[1.6920, 1.2893](1.6920 × 20,480 = 34,652rows on layer 0),step=3:[1.4992, 1.3093].>= 34,652:34,688rows (ragged_buffer_factor = 1.69376221).< 34,652:34,560rows (ragged_buffer_factor = 1.68751221, 128 rows lower).| Step | Force-dropless (
rbf=-1.0) loss |rbf = 1.69376221(34,688rows,>=probe)|Δloss|/|Δgrad_norm|/fallback_layers|rbf = 1.68751221(34,560rows,<probe)|Δloss|/|Δgrad_norm|/fallback_layers||---|---|---|---|
| 0 |
10.89864540|0.000e+00/0.000e+00/0.0|0.000e+00/0.000e+00/0.0|| 1 |
10.77676964|0.000e+00/0.000e+00/0.0|0.000e+00/0.000e+00/0.0|| 2 |
10.68296623|0.000e+00/0.000e+00/0.0|7.629e-06/2.980e-07/1.0(layer 0 overflows) || 3 |
10.64385128|0.000e+00/0.000e+00/0.0|2.956e-05/2.325e-06/0.0|(c) Flag-off bitwise parity vs
main(561de23e8)8-step TPU v6e-8 training (
log_required_ragged_buffer_factor=false):0.000e+00diff onloss,raw_grad_norm, andparam_normacross all 8 steps.2. Automated CPU Unit Tests (
tests/unit/moe_test.py::RequiredRaggedBufferFactorTest— 5 passed)test_matches_buffer_size_boundarytest_return_ra2a_zeros_dropped_token_rowstest_ra2a_sows_mesh_max_required_rbftest_ra2a_ep1_and_unsharded_batch_do_not_sow_required_rbftest_validator_rules_for_log_required_ragged_buffer_factor(Note on CI
codecov/patch: fork PRs upload coverage tokenlessly across 52 matrix sessions; on prior run78a8c583c, Codecov ingested 38/52 sessions even though all 4cpu-unitshards passed, which can cause spurious partial-uploadcodecov/patchreports on large files likemoe.py.)Checklist
Before submitting this PR, please make sure (put X in square brackets):