Repository navigation
[MoE replication 3/4] Add replica weight-move ops for online hot-expert replication - #6301
Shuwen-Fang wants to merge 1 commit into
Conversation
8a1c999 to
9360172
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
@gemini-cli /review |
|
🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
There was a problem hiding this comment.
📋 Review Summary
Verdict: Spec met; solid test coverage, but unrolled Python loop causes severe HLO bloat in the backward pass.
Top issues:
- [Perf] The Python loop over
max_sendin the backward pass creates massive XLA graph unrolling. → Replace it with JAX's nativescatter_addvia.at[send_slots].add(back).
Inferred spec & validation
- Sources: PR description, code before/after.
- Problem: The MoE ragged all-to-all path needs to efficiently extend local weights with replicas and accumulate replica gradients back to their home devices without intermediate host/device scatter buffers.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): Forward pass populates replica slots [L, L+R) with home weights sent via ragged_all_to_all |
✅ | test_forward_and_backward_sum_multiple_replicas_to_home ✅ |
| R2 (stated): Backward pass returns replica gradients and accumulates them into home gradients | ✅ | test_forward_and_backward_sum_multiple_replicas_to_home ✅ |
R3 (stated): Support mixed precision (e.g. fp32 master, bf16 compute) |
✅ | test_mixed_precision_fp32_master_weights_to_bf16_compute ✅ |
| C1 (inferred): Unused/padded send slots must not corrupt local slot 0's gradient | ✅ preserved | test_slot0_with_unused_send_slots_padded_to_zero ✅ |
| C2 (inferred): Empty plan leaves replicas zero and home grad unmodified | ✅ preserved | test_empty_plan_leaves_replicas_zero_and_home_grad_exact ✅ |
| rows = jnp.arange(local, dtype=jnp.int32).reshape((local,) + (1,) * (len(shape) - 1)) | ||
| g_home = g[:local].astype(wdtype) | ||
| for c in range(max_send): | ||
| g_home = g_home + jnp.where(rows == send_slots[c], back[c][None], jnp.zeros((), wdtype)) |
There was a problem hiding this comment.
🟠 This Python loop over max_send unrolls into the XLA graph, creating a massive number of where and add operations if max_send is large. JAX's .at[indices].add() natively compiles to an efficient scatter-add and inherently handles accumulation if send_slots contains duplicates, eliminating the need for the unrolled loop and the rows mask tensor.
| rows = jnp.arange(local, dtype=jnp.int32).reshape((local,) + (1,) * (len(shape) - 1)) | |
| g_home = g[:local].astype(wdtype) | |
| for c in range(max_send): | |
| g_home = g_home + jnp.where(rows == send_slots[c], back[c][None], jnp.zeros((), wdtype)) | |
| g_home = g[:local].astype(wdtype) | |
| g_home = g_home.at[send_slots].add(back) |
There was a problem hiding this comment.
Intentional for TPU XLA fusion (expert_replica_max_send defaults to 4): with g[:local].astype(wdtype).at[send_slots].add(back), XLA cannot fuse scatter-add into the g[:local] slice of the [L + R, D, M] tgmm output and materializes an extra 0.62 ms/layer HBM slice copy in hlo_stats (measured on TPU v6e-8 at L=16, D=4096, M=1024). Unrolling the max_send (4) elementwise where+add terms fuses the g[:local] slice, astype(wdtype), and home-grad accumulation into a single pass (0.0 ms standalone slice/scatter time).
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request implements the core logic for online hot-expert replication within a Mixture-of-Experts (MoE) architecture. It provides a mechanism to distribute expert weights across devices using a ragged all-to-all communication pattern and ensures that gradients are correctly accumulated back to the home devices during the backward pass. The implementation is optimized for performance by leveraging XLA-fused elementwise operations and custom VJP definitions to minimize memory overhead on TPUs. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
🤖 Hi @Shuwen-Fang, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
There was a problem hiding this comment.
📋 Review Summary
Verdict: No significant issues found.
Inferred spec & validation
- Sources: PR description, code before/after.
- Problem: Needs a way to efficiently pad and route expert weights to replicas for online hot-expert replication and sum gradients back to their home devices without dynamic scatter-add HBM copies.
| Spec item | Implementation | Validating test |
|---|---|---|
Pad local weights to [L+R, ...] via ragged_all_to_all (stated) |
✅ | test_forward_and_backward_sum_multiple_replicas_to_home ✅ |
| Custom VJP routes replica gradients back to home (stated) | ✅ | test_forward_and_backward_sum_multiple_replicas_to_home ✅ |
| Accumulate home gradients via unrolled XLA-fused loop (stated) | ✅ | test_forward_and_backward_sum_multiple_replicas_to_home ✅ |
| Handle edge cases (0 replicas, 1 EP) safely (inferred) | ✅ | test_noop_when_zero_replicas_or_single_device, test_empty_plan_leaves_replicas_zero_and_home_grad_exact ✅ |
| Mixed precision compute vs master weights (stated) | ✅ | test_mixed_precision_fp32_master_weights_to_bf16_compute ✅ |
adb1068 to
b0f57e7
Compare
Description
Adds the replica weight-movement module (
src/maxtext/layers/moe_replica_weights.py) for online hot-expert replication on the ragged all-to-all MoE path (part 3 of the stack; no callers wired yet):extend_with_replicas: pads local expert weights[L, ...]to[L + R, ...]in computedtypeand populates replica slots[L, L + R)in place via a singlejax.lax.ragged_all_to_alldirectly from the home weight buffer (with a zero-collective fast path whennum_replicas == 0ornum_ep == 1)._extend_direct_fwd/_extend_direct_bwd): returns replica gradients[L, L + R)to their home devices viaragged_all_to_alland accumulates them into home expert gradients in master weightdtypevia an unrolledO(max_send)XLA-fused elementwise loop (1 <= max_send <= MAX_REPLICA_SEND = 8, avoiding the0.62 ms/layerHBM copy materialized by dynamicscatter-addon TPU).moe_replica_planner.ReplicaPlaninvariants verified inmoe_replica_planner_test.test_plan_satisfies_weight_move_invariants.Tests
JAX_PLATFORMS=cpu pytest -v tests/unit/moe_replica_weights_test.py(7 passed):MoveParamsTest::test_send_and_recv_sizes_match_across_peersExtendWithReplicasTest::test_empty_plan_leaves_replicas_zero_and_home_grad_exactExtendWithReplicasTest::test_forward_and_backward_sum_multiple_replicas_to_homeExtendWithReplicasTest::test_invalid_plan_parameters_raise_value_errorExtendWithReplicasTest::test_mixed_precision_fp32_master_weights_to_bf16_computeExtendWithReplicasTest::test_noop_when_zero_replicas_or_single_deviceExtendWithReplicasTest::test_slot0_with_unused_send_slots_padded_to_zeroChecklist
Before submitting this PR, please make sure (put X in square brackets):