Skip to content

[MoE replication 3/4] Add replica weight-move ops for online hot-expert replication - #6301

Open
Shuwen-Fang wants to merge 1 commit into
AI-Hypercomputer:mainfrom
Shuwen-Fang:moe-replica-weights
Open

Shuwen-Fang wants to merge 1 commit into
AI-Hypercomputer:mainfrom
Shuwen-Fang:moe-replica-weights

Conversation

@Shuwen-Fang

@Shuwen-Fang Shuwen-Fang commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Adds the replica weight-movement module (src/maxtext/layers/moe_replica_weights.py) for online hot-expert replication on the ragged all-to-all MoE path (part 3 of the stack; no callers wired yet):

  • extend_with_replicas: pads local expert weights [L, ...] to [L + R, ...] in compute dtype and populates replica slots [L, L + R) in place via a single jax.lax.ragged_all_to_all directly from the home weight buffer (with a zero-collective fast path when num_replicas == 0 or num_ep == 1).
  • Custom VJP (_extend_direct_fwd / _extend_direct_bwd): returns replica gradients [L, L + R) to their home devices via ragged_all_to_all and accumulates them into home expert gradients in master weight dtype via an unrolled O(max_send) XLA-fused elementwise loop (1 <= max_send <= MAX_REPLICA_SEND = 8, avoiding the 0.62 ms/layer HBM copy materialized by dynamic scatter-add on TPU).
  • Documents and cross-references the moe_replica_planner.ReplicaPlan invariants verified in moe_replica_planner_test.test_plan_satisfies_weight_move_invariants.

Tests

  • JAX_PLATFORMS=cpu pytest -v tests/unit/moe_replica_weights_test.py (7 passed):
    • MoveParamsTest::test_send_and_recv_sizes_match_across_peers
    • ExtendWithReplicasTest::test_empty_plan_leaves_replicas_zero_and_home_grad_exact
    • ExtendWithReplicasTest::test_forward_and_backward_sum_multiple_replicas_to_home
    • ExtendWithReplicasTest::test_invalid_plan_parameters_raise_value_error
    • ExtendWithReplicasTest::test_mixed_precision_fp32_master_weights_to_bf16_compute
    • ExtendWithReplicasTest::test_noop_when_zero_replicas_or_single_device
    • ExtendWithReplicasTest::test_slot0_with_unused_send_slots_padded_to_zero

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests locally and provided test artifacts above, if applicable.
  • I have made or will make corresponding changes to the doc if needed.

@github-actions github-actions Bot added the skip-ci-tests Add this label to the PR to skip running CI tests label Nov 12, 2026
Shuwen-Fang added a commit to Shuwen-Fang/maxtext that referenced this pull request Nov 13, 2026
@Shuwen-Fang Shuwen-Fang changed the title [MoE replication 4/5] Add replica weight-move ops for online hot-expert replication [MoE replication 3/4] Add replica weight-move ops for online hot-expert replication Nov 13, 2026
@codecov

codecov Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 20.63492% with 50 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/maxtext/layers/moe_replica_weights.py 20.63% 50 Missing ⚠️

📢 Thoughts on this report? Let us know!

@Shuwen-Fang

Copy link
Copy Markdown
Collaborator Author

@gemini-cli /review

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: Spec met; solid test coverage, but unrolled Python loop causes severe HLO bloat in the backward pass.

Top issues:

  1. [Perf] The Python loop over max_send in the backward pass creates massive XLA graph unrolling. → Replace it with JAX's native scatter_add via .at[send_slots].add(back).
Inferred spec & validation
  • Sources: PR description, code before/after.
  • Problem: The MoE ragged all-to-all path needs to efficiently extend local weights with replicas and accumulate replica gradients back to their home devices without intermediate host/device scatter buffers.
Spec item Implementation Validating test
R1 (stated): Forward pass populates replica slots [L, L+R) with home weights sent via ragged_all_to_all ✅ test_forward_and_backward_sum_multiple_replicas_to_home ✅
R2 (stated): Backward pass returns replica gradients and accumulates them into home gradients ✅ test_forward_and_backward_sum_multiple_replicas_to_home ✅
R3 (stated): Support mixed precision (e.g. fp32 master, bf16 compute) ✅ test_mixed_precision_fp32_master_weights_to_bf16_compute ✅
C1 (inferred): Unused/padded send slots must not corrupt local slot 0's gradient ✅ preserved test_slot0_with_unused_send_slots_padded_to_zero ✅
C2 (inferred): Empty plan leaves replicas zero and home grad unmodified ✅ preserved test_empty_plan_leaves_replicas_zero_and_home_grad_exact ✅

Comment on lines +85 to +88
rows = jnp.arange(local, dtype=jnp.int32).reshape((local,) + (1,) * (len(shape) - 1))
g_home = g[:local].astype(wdtype)
for c in range(max_send):
g_home = g_home + jnp.where(rows == send_slots[c], back[c][None], jnp.zeros((), wdtype))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 This Python loop over max_send unrolls into the XLA graph, creating a massive number of where and add operations if max_send is large. JAX's .at[indices].add() natively compiles to an efficient scatter-add and inherently handles accumulation if send_slots contains duplicates, eliminating the need for the unrolled loop and the rows mask tensor.

Suggested change
rows = jnp.arange(local, dtype=jnp.int32).reshape((local,) + (1,) * (len(shape) - 1))
g_home = g[:local].astype(wdtype)
for c in range(max_send):
g_home = g_home + jnp.where(rows == send_slots[c], back[c][None], jnp.zeros((), wdtype))
g_home = g[:local].astype(wdtype)
g_home = g_home.at[send_slots].add(back)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intentional for TPU XLA fusion (expert_replica_max_send defaults to 4): with g[:local].astype(wdtype).at[send_slots].add(back), XLA cannot fuse scatter-add into the g[:local] slice of the [L + R, D, M] tgmm output and materializes an extra 0.62 ms/layer HBM slice copy in hlo_stats (measured on TPU v6e-8 at L=16, D=4096, M=1024). Unrolling the max_send (4) elementwise where+add terms fuses the g[:local] slice, astype(wdtype), and home-grad accumulation into a single pass (0.0 ms standalone slice/scatter time).

@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request implements the core logic for online hot-expert replication within a Mixture-of-Experts (MoE) architecture. It provides a mechanism to distribute expert weights across devices using a ragged all-to-all communication pattern and ensures that gradients are correctly accumulated back to the home devices during the backward pass. The implementation is optimized for performance by leveraging XLA-fused elementwise operations and custom VJP definitions to minimize memory overhead on TPUs.

Highlights

  • Module Addition: Introduced src/maxtext/layers/moe_replica_weights.py to handle replica weight movement for online hot-expert replication in MoE models.
  • Efficient Weight Replication: Implemented extend_with_replicas which uses jax.lax.ragged_all_to_all to efficiently pad and populate replica expert weights.
  • Custom VJP Implementation: Added a custom vector-Jacobian product (VJP) to return replica gradients to home devices and accumulate them efficiently, avoiding expensive HBM copies.
  • Unit Testing: Added comprehensive unit tests in tests/unit/moe_replica_weights_test.py covering forward/backward passes, mixed precision, and error handling.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @Shuwen-Fang, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: No significant issues found.

Inferred spec & validation
  • Sources: PR description, code before/after.
  • Problem: Needs a way to efficiently pad and route expert weights to replicas for online hot-expert replication and sum gradients back to their home devices without dynamic scatter-add HBM copies.
Spec item Implementation Validating test
Pad local weights to [L+R, ...] via ragged_all_to_all (stated) ✅ test_forward_and_backward_sum_multiple_replicas_to_home ✅
Custom VJP routes replica gradients back to home (stated) ✅ test_forward_and_backward_sum_multiple_replicas_to_home ✅
Accumulate home gradients via unrolled XLA-fused loop (stated) ✅ test_forward_and_backward_sum_multiple_replicas_to_home ✅
Handle edge cases (0 replicas, 1 EP) safely (inferred) ✅ test_noop_when_zero_replicas_or_single_device, test_empty_plan_leaves_replicas_zero_and_home_grad_exact ✅
Mixed precision compute vs master weights (stated) ✅ test_mixed_precision_fp32_master_weights_to_bf16_compute ✅

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-ci-tests Add this label to the PR to skip running CI tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant