Skip to content

[MoE replication 4b/5] Wire online hot-expert replication into RoutedMoE - #6375

Open
Shuwen-Fang wants to merge 4 commits into
AI-Hypercomputer:mainfrom
Shuwen-Fang:moe-expert-replication-wiring-v2
Open

Shuwen-Fang wants to merge 4 commits into
AI-Hypercomputer:mainfrom
Shuwen-Fang:moe-expert-replication-wiring-v2

Conversation

@Shuwen-Fang

@Shuwen-Fang Shuwen-Fang commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Review only this PR's changes: Compare c12707c39...moe-expert-replication-wiring-v2 · commits 5e886906 + 07d9c17c (plus 03814ead, cherry-picking #6374's b146f04f fix until #6374 merges)

Description

Wires the online hot-expert planner (#6300) and SparseCore weight-replication custom VJP (#6301) into RoutedMoE behind expert_replicas_per_device (default 0, a complete no-op).

When expert_replicas_per_device > 0 on the ragged all-to-all path, ra2a_and_route gathers [EP, E] token counts, computes a ReplicaPlan, remaps logical experts into EP * (L + R) physical slots inside permute(), threads replica_plan via RouteMetadata, and extends (w0, w1, wo) to [L + R, ...] before grouped GEMMs in _moe_body. On SFS (EP=32, 4 layers, Qwen3.5-35B C4 routing replay resampled to 512 experts), R=4 cuts step time 12.8% (1064.4 → 928.7 ms).

BUGS: b/585678451

Stack

# PR Scope
1/5 #6299 log_required_ragged_buffer_factor on ragged all-to-all
2/5 #6300 Online hot-expert replica planner (moe_replica_planner.py)
3/5 #6301 Custom-VJP SparseCore weight replication (moe_replica_weights.py)
4a/5 #6374 Config flags & pre-compile validation (expert_replicas_per_device=0)
4b/5 #6375 (this PR) RoutedMoE wiring & CPU end-to-end parity tests
Bench #6298 Standalone synthetic routing replay benchmark

Tests

Tested on CPU in tests/unit/moe_expert_replication_test.py (flag-off skips planner and weight replication, permute slot routing and packed-sort guard, forward/backward parity + required_rbf drop vs. R=0, routed_bias parity, jax.checkpoint/lax.cond remat, and runtime guardrails; 14 passed) and on TPU v6e-8 / SFS.

Flag-off (expert_replicas_per_device=0) verified against main eb257bef: bit-identical forward/backward on a CPU EP=4 harness, and identical compiled HLO plus identical 20-step loss/grad-norm on TPU v6e-8.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @Shuwen-Fang, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@github-actions github-actions Bot added the skip-ci-tests Add this label to the PR to skip running CI tests label Nov 16, 2026
@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request implements online hot-expert replication within the RoutedMoE module to mitigate load imbalance in Mixture-of-Experts models. By dynamically creating replicas of high-demand experts across devices, the system can distribute token processing more effectively. The changes include a new planner for determining replica placement, custom VJP logic for handling weight replication and gradient accumulation, and updated routing mechanisms to support physical slot remapping. The implementation is gated behind new configuration flags and includes strict validation to prevent unsupported feature combinations.

Highlights

  • Online Hot-Expert Replication: Integrated an online hot-expert planner and SparseCore weight-replication into RoutedMoE, allowing for expert replication on the ragged all-to-all path to improve load balancing.
  • Configuration and Validation: Added new configuration flags (expert_replicas_per_device, expert_replica_max_send, expert_replica_board_size) and implemented comprehensive validation logic to ensure compatibility with existing MoE features.
  • Performance Impact: Demonstrated a 12.8% reduction in step time on SFS (Qwen3.5-35B) by utilizing 4 expert replicas per device.
  • Testing: Added unit tests for configuration validation, planner logic, and end-to-end parity on CPU, ensuring robustness and correctness of the new replication path.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@codecov

codecov Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 25.98039% with 151 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/maxtext/layers/moe_replica_planner.py 15.62% 81 Missing ⚠️
src/maxtext/layers/moe_replica_weights.py 20.63% 50 Missing ⚠️
src/maxtext/layers/moe.py 55.55% 16 Missing and 4 partials ⚠️

📢 Thoughts on this report? Let us know!

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: Spec partially met; core fix untested for padded/masked tokens.

Top issues:

  1. [Bug] Masked/padding tokens cause out-of-bounds indexing in physical_route_fn, silently mapping them to the last physical expert's slot and inflating its load. → Clamp flat_logical before indexing route_table, and map the masked tokens to num_sort_bins so they are correctly dropped. (See inline comment in moe.py).
  2. [Tests] Masked/padding token paths are unvalidated. The CPU tests either pass forced_routed_experts (which bypasses the routing_mask logic) or use a fully populated sequence with no capacity limits, leaving the out-of-bounds bug undetected. → Add a test case that passes padded sequences (or hits capacity limits forcing drops) without forced_routed_experts to ensure replication handles masked tokens correctly.
Inferred spec & validation
  • Sources: PR description, code before/after, b/585678451 (inferred).
  • Problem: Hot experts cause token dropping or delays. Replicating hot experts online (during step execution) can improve performance and balance loads.
Spec item Implementation Validating test
R1 (stated): Wire moe_replica_planner and moe_replica_weights into RoutedMoE behind expert_replicas_per_device > 0 on ragged all-to-all path. ⚠️ partial test_end_to_end_sparse_matmul_parity_and_required_rbf ⚠️ superficial (bypasses routing_mask logic for padded/dropped tokens)
R2 (stated): Remap logical experts to physical slots in permute. ❌ not met (out-of-bounds bug for masked tokens) test_permute_with_physical_route_fn ⚠️ superficial (bypasses masked tokens)
C1 (stated): No-op and bit-identical when expert_replicas_per_device == 0. ✅ preserved test_flag_off_skips_replication_modules ✅
C2 (inferred): Raise NotImplementedError on unsupported configs (e.g. moe_use_fused_gmm3d, use_ring_of_experts=True, moe_bias, etc). ✅ preserved test_runtime_guardrails_raise ✅

Comment thread src/maxtext/layers/moe.py
self.config.expert_replica_max_send,
self.config.expert_replica_board_size,
)
phys = replica_plan.route_table[expert_shard_id][flat_logical]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Masked and padded tokens (which have expert_index == self.config.num_experts) cause an out-of-bounds array access here. JAX silently wraps out-of-bounds indices to the last element, so masked tokens will be incorrectly mapped to the physical slot of the last expert, artificially inflating its load and causing incorrect gradients. We need to clamp the logical index for the lookup and then manually map masked tokens out of bounds (to num_sort_bins) so jnp.bincount properly ignores them.

Suggested change
phys = replica_plan.route_table[expert_shard_id][flat_logical]
# Masked tokens have value self.config.num_experts. Map them out of bounds so they are ignored.
valid_logical = jnp.minimum(flat_logical, self.config.num_experts - 1)
phys = replica_plan.route_table[expert_shard_id][valid_logical]
num_sort_bins = num_ep * replica_plan.num_slots
phys = jnp.where(flat_logical == self.config.num_experts, num_sort_bins, phys)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not reachable on this path. Replication is rejected with use_ring_of_experts=True, the only branch that maps invalid slots to num_experts. On the ragged all-to-all branch, get_topk returns [0, num_experts) or -1 for invalid/padded slots (forced routing, and token_mask in #6373), and permute() remaps those -1s round-robin into [0, num_experts) before physical_route_fn runs.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-ci-tests Add this label to the PR to skip running CI tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant