Repository navigation
[MoE replication 4b/5] Wire online hot-expert replication into RoutedMoE - #6375
Shuwen-Fang wants to merge 4 commits into
Conversation
… flag-off test cleanup
|
🤖 Hi @Shuwen-Fang, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request implements online hot-expert replication within the RoutedMoE module to mitigate load imbalance in Mixture-of-Experts models. By dynamically creating replicas of high-demand experts across devices, the system can distribute token processing more effectively. The changes include a new planner for determining replica placement, custom VJP logic for handling weight replication and gradient accumulation, and updated routing mechanisms to support physical slot remapping. The implementation is gated behind new configuration flags and includes strict validation to prevent unsupported feature combinations. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
📋 Review Summary
Verdict: Spec partially met; core fix untested for padded/masked tokens.
Top issues:
- [Bug] Masked/padding tokens cause out-of-bounds indexing in
physical_route_fn, silently mapping them to the last physical expert's slot and inflating its load. → Clampflat_logicalbefore indexingroute_table, and map the masked tokens tonum_sort_binsso they are correctly dropped. (See inline comment inmoe.py). - [Tests] Masked/padding token paths are unvalidated. The CPU tests either pass
forced_routed_experts(which bypasses therouting_masklogic) or use a fully populated sequence with no capacity limits, leaving the out-of-bounds bug undetected. → Add a test case that passes padded sequences (or hits capacity limits forcing drops) withoutforced_routed_expertsto ensure replication handles masked tokens correctly.
Inferred spec & validation
- Sources: PR description, code before/after, b/585678451 (inferred).
- Problem: Hot experts cause token dropping or delays. Replicating hot experts online (during step execution) can improve performance and balance loads.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): Wire moe_replica_planner and moe_replica_weights into RoutedMoE behind expert_replicas_per_device > 0 on ragged all-to-all path. |
test_end_to_end_sparse_matmul_parity_and_required_rbf routing_mask logic for padded/dropped tokens) |
|
R2 (stated): Remap logical experts to physical slots in permute. |
❌ not met (out-of-bounds bug for masked tokens) | test_permute_with_physical_route_fn |
C1 (stated): No-op and bit-identical when expert_replicas_per_device == 0. |
✅ preserved | test_flag_off_skips_replication_modules ✅ |
C2 (inferred): Raise NotImplementedError on unsupported configs (e.g. moe_use_fused_gmm3d, use_ring_of_experts=True, moe_bias, etc). |
✅ preserved | test_runtime_guardrails_raise ✅ |
| self.config.expert_replica_max_send, | ||
| self.config.expert_replica_board_size, | ||
| ) | ||
| phys = replica_plan.route_table[expert_shard_id][flat_logical] |
There was a problem hiding this comment.
🔴 Masked and padded tokens (which have expert_index == self.config.num_experts) cause an out-of-bounds array access here. JAX silently wraps out-of-bounds indices to the last element, so masked tokens will be incorrectly mapped to the physical slot of the last expert, artificially inflating its load and causing incorrect gradients. We need to clamp the logical index for the lookup and then manually map masked tokens out of bounds (to num_sort_bins) so jnp.bincount properly ignores them.
| phys = replica_plan.route_table[expert_shard_id][flat_logical] | |
| # Masked tokens have value self.config.num_experts. Map them out of bounds so they are ignored. | |
| valid_logical = jnp.minimum(flat_logical, self.config.num_experts - 1) | |
| phys = replica_plan.route_table[expert_shard_id][valid_logical] | |
| num_sort_bins = num_ep * replica_plan.num_slots | |
| phys = jnp.where(flat_logical == self.config.num_experts, num_sort_bins, phys) |
There was a problem hiding this comment.
Not reachable on this path. Replication is rejected with use_ring_of_experts=True, the only branch that maps invalid slots to num_experts. On the ragged all-to-all branch, get_topk returns [0, num_experts) or -1 for invalid/padded slots (forced routing, and token_mask in #6373), and permute() remaps those -1s round-robin into [0, num_experts) before physical_route_fn runs.
Description
Wires the online hot-expert planner (#6300) and SparseCore weight-replication custom VJP (#6301) into
RoutedMoEbehindexpert_replicas_per_device(default0, a complete no-op).When
expert_replicas_per_device > 0on the ragged all-to-all path,ra2a_and_routegathers[EP, E]token counts, computes aReplicaPlan, remaps logical experts intoEP * (L + R)physical slots insidepermute(), threadsreplica_planviaRouteMetadata, and extends(w0, w1, wo)to[L + R, ...]before grouped GEMMs in_moe_body. On SFS (EP=32, 4 layers, Qwen3.5-35B C4 routing replay resampled to 512 experts), R=4 cuts step time 12.8% (1064.4 → 928.7 ms).BUGS: b/585678451
Stack
log_required_ragged_buffer_factoron ragged all-to-allmoe_replica_planner.py)moe_replica_weights.py)expert_replicas_per_device=0)RoutedMoEwiring & CPU end-to-end parity testsTests
Tested on CPU in
tests/unit/moe_expert_replication_test.py(flag-off skips planner and weight replication,permuteslot routing and packed-sort guard, forward/backward parity +required_rbfdrop vs.R=0,routed_biasparity,jax.checkpoint/lax.condremat, and runtime guardrails; 14 passed) and on TPU v6e-8 / SFS.Flag-off (
expert_replicas_per_device=0) verified againstmaineb257bef: bit-identical forward/backward on a CPU EP=4 harness, and identical compiled HLO plus identical 20-step loss/grad-norm on TPU v6e-8.Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.