Repository navigation
[MoE replication 4/4] Wire online hot-expert replication into the ragged all-to-all MoE path - #6302
Shuwen-Fang wants to merge 24 commits into
Conversation
Default off. When on, each MoE layer prints the ragged_buffer_factor it needs to avoid dropping tokens (max tokens received by an EP shard / tokens per shard).
Pure-jnp planners (greedy, vectorized rounds, and lagged placement) that choose which hot experts get replica slots on which EP devices and reroute whole source shards to them. Not wired into the MoE layer yet.
Custom-VJP ops that move replica expert weights to their receiving EP devices with one ragged all-to-all per weight array (gather, direct and split variants) and add replica gradients back into the home experts. Not wired into the MoE layer yet.
Adds the expert_replica_* flags (expert_replicas_per_device=0, i.e. off, by default) and uses moe_replica_planner and moe_replica_weights in RoutedMoE: tokens are remapped to physical (home or replica) slots before the dispatch sort, replica weights are moved per layer, and replica slots are computed by the expert GMMs.
1242c48 to
eed7de2
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…nd CPU wiring tests
…t-0 route clobber
…ngle rounds+direct path
|
@gemini-cli /review |
|
🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
|
🤖 I'm sorry @Shuwen-Fang, but I was unable to process your request. Please see the logs for more details. |
|
@gemini-cli /review |
|
🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
|
🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
There was a problem hiding this comment.
📋 Review Summary
Verdict: Spec met across all acceptance criteria with robust test coverage; no significant issues found.
Inferred spec & validation
- Sources: PR description (#6302), linked PRs (#6299, #6300, #6301), code before/after across
src/maxtext/layers/moe.py,src/maxtext/layers/moe_replica_planner.py,src/maxtext/layers/moe_replica_weights.py, andsrc/maxtext/configs/. - Problem: Token routing hotspots in MoE overload individual expert-parallel devices on the ragged all-to-all path, forcing high ragged buffer factors and stranding idle compute on cold devices.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): Online hot-expert replication is wired into the use_ring_of_experts=False ragged all-to-all path (ra2a_and_route) when expert_replicas_per_device > 0, lowering required buffer factors under skewed traffic. |
✅ met | moe_expert_replication_test.py::RoutedMoEReplicationWiringTest::test_end_to_end_sparse_matmul_parity_and_required_rbf ✅ |
R2 (stated): Default configuration (expert_replicas_per_device=0) remains a zero-overhead no-op with bitwise numerical parity. |
✅ met | moe_expert_replication_test.py::ExpertReplicationConfigTest::test_defaults_off ✅, moe_replica_weights_test.py::ExtendWithReplicasTest::test_noop_when_zero_replicas_or_single_device ✅ |
R3 (stated): ReplicaPlan threads through RouteMetadata as a registered Flax PyTree, safely crossing JIT compilation, jax.checkpoint remat, and jax.lax.cond. |
✅ met | moe_expert_replication_test.py::RoutedMoEReplicationWiringTest::test_route_metadata_with_replica_plan_is_valid_pytree ✅, test_remat_and_multi_branch_trace_parity_with_r0 ✅ |
| R4 (stated): Gradients of replica weights on receiver devices return to home devices via custom VJP and accumulate into home expert gradients in master weight dtype. | ✅ met | moe_replica_weights_test.py::ExtendWithReplicasTest::test_forward_and_backward_sum_multiple_replicas_to_home ✅, test_slot0_with_unused_send_slots_padded_to_zero ✅, test_mixed_precision_fp32_master_weights_to_bf16_compute ✅ |
| R5 (stated): Config validation rejects incompatible MoE features (Ring of Experts, FP8/quantized MoE, dropless fallback, biases, embedding chunking, EP=1) fail-fast. | ✅ met | moe_expert_replication_test.py::ExpertReplicationConfigTest::test_config_validator_rejects_unsupported_combinations ✅, test_guardrails_raise_on_unsupported_configs ✅ |
| C1 (stated): Preserves backward compatibility of existing checkpoints and YAML configs when replication is disabled. | ✅ preserved | moe_expert_replication_test.py::ExpertReplicationConfigTest::test_defaults_off ✅ |
| C2 (stated/inferred): Zero host-device synchronization, tracer leaks, or dynamic control flow inside JIT/shard_map hot paths. | ✅ preserved | moe_expert_replication_test.py::RoutedMoEReplicationWiringTest::test_end_to_end_sparse_matmul_parity_and_required_rbf ✅ |
| C3 (inferred): Gradient accumulation on home devices uses XLA-fused elementwise loops without materializing intermediate dynamic scatter-add copies in HBM. | ✅ preserved | moe_replica_weights_test.py::ExtendWithReplicasTest::test_forward_and_backward_sum_multiple_replicas_to_home ✅ |
|
🤖 Hi @Shuwen-Fang, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request integrates online hot-expert replication into the ragged all-to-all MoE path in MaxText. By allowing hot experts to be replicated across devices, the system can better balance token loads during training. The implementation includes a planner to compute replication plans, weight management logic to handle replicated expert weights, and strict guardrails to ensure compatibility with existing MoE configurations. The changes are designed to be opt-in via configuration flags and have been validated for performance and correctness on TPU hardware. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
There was a problem hiding this comment.
📋 Review Summary
Verdict: No significant issues found.
Inferred spec & validation
- Sources: PR description, code before/after.
- Problem: Need to wire online hot-expert replication into the ragged all-to-all MoE path to improve load balancing for models with skewed routing.
| Spec item | Implementation | Validating test |
|---|---|---|
| R1 (stated): Rejects unsupported config combinations at init | ✅ met | moe_expert_replication_test.py::test_config_validator_rejects_unsupported_combinations ✅ |
R2 (stated): Validates runtime preconditions (num_ep > 1, is_batch_sharded_by_expert=True) in ra2a_and_route |
✅ met | moe_expert_replication_test.py::test_guardrails_raise_on_unsupported_configs ✅ |
R3 (stated): PyTree-safely threads replica_plan through RouteMetadata into _moe_body for weight extension |
✅ met | moe_expert_replication_test.py::test_route_metadata_with_replica_plan_is_valid_pytree ✅ |
| C1 (stated): Off by default with zero graph/HLO change | ✅ preserved | moe_expert_replication_test.py::test_defaults_off ✅ |
| C2 (stated): At most one replica per logical expert per step | ✅ preserved | moe_replica_planner_test.py::test_plan_satisfies_weight_move_invariants ✅ |
Description
Wires online hot-expert replication (
moe_replica_planner+moe_replica_weights, stacked on#6299,#6300,#6301) into theuse_ring_of_experts=Falseragged all-to-all (ra2a_and_route) MoE path behind 3 default-off configuration flags:expert_replicas_per_device: 0(>= 0;0= off, zero graph/HLO change when off)expert_replica_max_send: 4(1 <= max_send <= 8; bounds per-home fan-out and the backward return buffer)expert_replica_board_size: 8(>= 1; number of JAX EP devices — i.e., TensorCores along theexpertmesh axis, which is2per chip on 2-TC/chip TPUs whenici_tensor_parallelism=1and1per chip on 1-TC/chip TPUs — per physical board/tray so the planner prefers same-board sources:4or8on TPU v6e-8,8on TPU v7x (4chips ×2TCs), and8per 4-chip board or16per 8-chip paired board on SFS4x4x1)Architecture & Guardrails
RouteMetadata.replica_planPyTree threading:ReplicaPlanis a@flax.struct.dataclassPyTree andra2a_and_routereturns it onRouteMetadata.replica_plan(Optional[Any] = None), consumed directly by_moe_bodywith zero mutable closure state and full PyTree safety acrossjax.jit,jax.checkpointremat, and multi-branchjax.lax.cond.MaxTextConfig.validate_expert_replication) + runtime checks: Rejects unsupported combinations before compilation (num_experts <= 1,sparse_matmul=False,te_moe_block=True,use_ring_of_experts=True,num_moe_emb_chunks > 0,mlp_bias=True,moe_accumulate_chunk_wgrad=True,moe_dropless_fallback != None,fp8_moe=True,fp8_moe_fake_quant=True,bool(quantization),bool(te_gmm_quantization), andEP == 1), and raisesValueErrorinra2a_and_routeifnum_ep <= 1,is_batch_sharded_by_expert=False(batch size 1 decode), ornum_experts % num_ep != 0so replication can never silently no-op asR=0.4x4x1(EP=32) BF16 MoE GEMM recipe (model_name=qwen3.5-397b-a17b,dtype=bfloat16,weight_dtype=float32,fp8_moe=false,quantization="",ici_expert_parallelism=32,ici_fsdp_parallelism=1,per_device_batch_size=2,max_target_length=4096,use_ring_of_experts=false,sparse_matmul=true,megablox=true,expert_replicas_per_device=4,expert_replica_max_send=4,expert_replica_board_size=8/16,ragged_buffer_factor=1.1/1.2), which usesmlp_bias=false,num_moe_emb_chunks=0,moe_accumulate_chunk_wgrad=false, andmoe_dropless_fallback=None.Known Limitations (v1)
QArraywrappers (fp8_moe,fp8_moe_fake_quant,quantization,te_gmm_quantization), so it cannot currently be combined with FP8 MoE recipes (such as those used in MLPerf / NPI FP8 configurations). All reported Qwen3.5-397B SFS4x4x1results used BF16 MoE GEMMs, which is a supported configuration.moe_dropless_fallbackin v1 (operational guidance & follow-up):moe_dropless_fallback(along withuse_ring_of_experts,mlp_bias,num_moe_emb_chunks,moe_accumulate_chunk_wgrad, andte_moe_block) is rejected at config initialization. Operational guidance: with replication, pickragged_buffer_factorfrom a calibration run withlog_required_ragged_buffer_factor=trueplus headroom; tokens beyond the buffer are dropped, same as baseline with a fixedragged_buffer_factor. Follow-up: the fallback (moe_dropless_fallback='layer') could use post-replication physical counts — which thelog_required_ragged_buffer_factorprobe already computes — to detect per-shard overflow (~35–50 LoC inRoutedMoE._check_ragged_overflow/sparse_matmulplus unit tests).home + 1 replica), so a single expert exceeding roughly2xa device's mean load cannot be balanced across more than two devices.Validation
1. TPU Performance & Correctness Evidence
fa4b588e6(http://xid/1000000001525390, SFS 4x4x1, n=3, 0 drops, aligned rbf): σ=0.60958.96±0.26→916.62±0.19 ms(−42.3 ms, −4.4%); Qwen empirical replay (rbf2.50→1.10)1064.42±1.39→928.71±0.46 ms(−135.7 ms, −12.8%); 30-step max|Δloss| ≤ 5.8e-5.sfs=4x4x1sweeps (EP=32, 4-layer Qwen3.5-397B BF16 MoE GEMMs,pdbs=2x4096) — measured on pre-split prototype tree (closed PR#6295@0c1f24a64, internal Fig commitscea9cbbc82bd/f6d9750cb364, prior to splittingmoe_expert_replication.pyinto#6298–#6302):rbf=2.7vs1.2): reduces median step time from1097.8 ± 0.8 ms(req=2.482) to959.5 ± 0.0 ms(-12.6%,req=1.061, zero drops); at tightenedrbf(2.5vs1.1),1066.5 -> 937.2 ms(-12.1%).moe_use_direct_token_gather=true + use_tokamax_gmm=true + use_gmm_v2=true,rbf=2.5vs1.1, 3 interleaved repeats onf6d9750cb364): reduces median step time from1054.8 ± 0.3 msto925.4 ± 0.2 ms(-129.4 ms/-12.3%additive;-13.2%vs default-opsR=01066.1 ± 0.0 ms), 30-step BF16 relative loss diff4.43e-6.57bdd8933vs822ea140d/fa4b588e6,EP=32, E=512, K=8, R=4, max_send=4):48/48bit-identicalroute_tables across the Qwen3.5-397B empirical profile and synthetic lognormalsigma=0.40..1.60at bothboard_size=8andboard_size=16.fa4b588e6):main(561de23e8, 8 steps, defaultexpert_replicas_per_device=0,EP=8): bit-identical on all 8 steps (0.000e+00max rel diff onloss,raw_grad_norm, andparam_norm;loss10.900139808654785→10.591382026672363).expert_replicas_per_device=4, 12 steps,JAX_LOG_COMPILES=1):0XLA compilation events after step 0 across steps1..11.EP=8,R=2vsR=0, 4 steps):step=0lossrel_diff=0.000e+00, maxrel_diff=1.075e-05across steps0..3.#6295@0c1f24a64): 12-stepR=4vsR=0(EP=8, skewed routing) max rel diffloss=3.013e-06,raw_grad_norm=2.676e-06,param_norm=1.685e-07,REQUIRED_RBF1.425-1.619 -> 1.017-1.036.2. Automated CPU Unit Tests (
tests/unit/moe_expert_replication_test.py— 11 passed, 28 subtests passed)ExpertReplicationConfigTest::test_config_validator_rejects_unsupported_combinationsExpertReplicationConfigTest::test_defaults_offExpertReplicationConfigTest::test_expert_replica_max_send_capped_at_8ExpertReplicationConfigTest::test_flags_parseExpertReplicationConfigTest::test_invalid_bounds_rejectedRoutedMoEReplicationWiringTest::test_end_to_end_sparse_matmul_parity_and_required_rbfRoutedMoEReplicationWiringTest::test_guardrails_raise_on_unsupported_configsRoutedMoEReplicationWiringTest::test_permute_with_physical_route_fnRoutedMoEReplicationWiringTest::test_remat_and_multi_branch_trace_parity_with_r0RoutedMoEReplicationWiringTest::test_route_metadata_with_replica_plan_is_valid_pytreeRoutedMoEReplicationWiringTest::test_routed_bias_updates_match_r0Checklist
Before submitting this PR, please make sure (put X in square brackets):