Repository navigation
Add online hot-expert replication for ragged all-to-all expert parallelism - #6295
Closed
Shuwen-Fang wants to merge 1 commit into
Closed
Shuwen-Fang wants to merge 1 commit into
Shuwen-Fang wants to merge 1 commit into
Conversation
…elism Adds flag-gated (default off) hot-expert replication to RoutedMoE on the ragged all-to-all EP path: per-step plan from all-gathered expert counts, replica weights moved by ragged_all_to_all into R extra local slots, tokens rerouted to replicas, replica grads returned to home experts. Also adds a synthetic MoE routing generator for reproducible load-imbalance benchmarks, plus unit tests.
This was referenced Nov 12, 2026
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds optional, flag-gated hot-expert replication to
RoutedMoEon the ragged all-to-all EP path. It also adds a synthetic MoE routing generator for reproducible load-imbalance benchmarks. Everything is off by default; with the flags off the code path is unchanged.Motivation
With skewed routing, the ragged all-to-all receive buffer has to be sized for the hottest device (
ragged_buffer_factor, rbf) to stay dropless. The sort, gather, combine and GMM costs scale with that buffer, and collectives wait on the slowest device. Placing replicas of the hottest experts on the coldest devices lowers the dropless rbf and evens out the per-device work.How it works (
expert_replicas_per_device=R)expert_replica_planner:rounds: vectorized; recommended.greedy: the original serial algorithm.ragged_all_to_allintoRextra local slots, so each device holdsL + Rexperts.expert_replica_move:direct: in place, no gather or concat.gather: copies the replica rows first.expert_replica_impl(auto|concat|split)expert_replica_plan(online|lagged)expert_replica_max_send,expert_replica_board_sizeexpert_replica_overhead_control: measurement only. It moves replicas but routes no tokens to them.log_ra2a_required_rbf: prints the minimum dropless rbf.Synthetic routing (
synthetic_routing=lognormal|balanced)forced_routed_expertsbatch key.synthetic_routing_profile).Results
Setup: Qwen3.5-397B-A17B, 4 MoE layers, per_device_batch_size 2 × 4096, EP 32 on 16 TPU 8i chips, bf16 dispatch. Routing matches a real Qwen3.5 routing profile. Each arm uses its own minimum dropless rbf. Medians over 3 interleaved repeats; pooled std 0.3 ms.
rounds+directTesting
tests/unit/moe_expert_replication_test.py: planner validity for greedy and rounds, send limits, load reduction, identity table, config parsing.tests/unit/synthetic_routing_test.py: determinism, shape and range, distinct top-k, exact balance.