Skip to content

Add online hot-expert replication for ragged all-to-all expert parallelism - #6295

Closed
Shuwen-Fang wants to merge 1 commit into
AI-Hypercomputer:mainfrom
Shuwen-Fang:moe-expert-replication
Closed

Shuwen-Fang wants to merge 1 commit into
AI-Hypercomputer:mainfrom
Shuwen-Fang:moe-expert-replication

Conversation

@Shuwen-Fang

Copy link
Copy Markdown
Collaborator

Adds optional, flag-gated hot-expert replication to RoutedMoE on the ragged all-to-all EP path. It also adds a synthetic MoE routing generator for reproducible load-imbalance benchmarks. Everything is off by default; with the flags off the code path is unchanged.

Motivation

With skewed routing, the ragged all-to-all receive buffer has to be sized for the hottest device (ragged_buffer_factor, rbf) to stay dropless. The sort, gather, combine and GMM costs scale with that buffer, and collectives wait on the slowest device. Placing replicas of the hottest experts on the coldest devices lowers the dropless rbf and evens out the per-device work.

How it works (expert_replicas_per_device=R)

  • Plan. Every step, each layer builds a plan from the all-gathered per-shard expert counts. Every device computes the same plan. expert_replica_planner:
    • rounds: vectorized; recommended.
    • greedy: the original serial algorithm.
  • Weights. The chosen replica weights move with ragged_all_to_all into R extra local slots, so each device holds L + R experts. expert_replica_move:
    • direct: in place, no gather or concat.
    • gather: copies the replica rows first.
  • Tokens. Tokens are rerouted to replica slots through a physical route table. Sorting and group sizes run over the physical slots.
  • Backward. Replica gradients are returned to the home experts with the reverse ragged all-to-all.
  • Other flags:
    • expert_replica_impl (auto|concat|split)
    • expert_replica_plan (online|lagged)
    • expert_replica_max_send, expert_replica_board_size
    • expert_replica_overhead_control: measurement only. It moves replicas but routes no tokens to them.
    • log_ra2a_required_rbf: prints the minimum dropless rbf.

Synthetic routing (synthetic_routing=lognormal|balanced)

  • Forces top-k routing from a lognormal popularity model with document-level clustering. It adds the forced_routed_experts batch key.
  • Optionally loads a fitted profile (synthetic_routing_profile).
  • Intended for perf experiments that need real-steady-state-like imbalance without a trained checkpoint.

Results

Setup: Qwen3.5-397B-A17B, 4 MoE layers, per_device_batch_size 2 × 4096, EP 32 on 16 TPU 8i chips, bf16 dispatch. Routing matches a real Qwen3.5 routing profile. Each arm uses its own minimum dropless rbf. Medians over 3 interleaved repeats; pooled std 0.3 ms.

Arm rbf Step time
Baseline 2.5 1066.5 ms
R = 4, rounds + direct 1.1 937.2 ms (−12.1%)
Perfectly balanced routing (ideal) 1.0 912.0 ms (separate sweep)
  • Replication overhead, measured with the overhead-control arm: about 23 ms/step (2%).
  • XProf shows savings in token sort/gather (−74 ms) and collective skew wait (−62 ms).
  • Peak HBM is lower than the baseline because of the smaller receive buffer.

Testing

  • New unit tests, which run on CPU:
    • tests/unit/moe_expert_replication_test.py: planner validity for greedy and rounds, send limits, load reduction, identity table, config parsing.
    • tests/unit/synthetic_routing_test.py: determinism, shape and range, distinct top-k, exact balance.
  • TPU parity on v6e-8:
    • Flags off: bit-identical to main.
    • R = 2 and R = 4: loss within 3e-5 of R = 0 over 50 steps.
    • Overhead control: bit-identical to R = 0.
    • Checkpoints round-trip between feature and main.
    • No recompiles over 50 steps.

…elism

Adds flag-gated (default off) hot-expert replication to RoutedMoE on the ragged
all-to-all EP path: per-step plan from all-gathered expert counts, replica weights
moved by ragged_all_to_all into R extra local slots, tokens rerouted to replicas,
replica grads returned to home experts. Also adds a synthetic MoE routing generator
for reproducible load-imbalance benchmarks, plus unit tests.
@Shuwen-Fang

Copy link
Copy Markdown
Collaborator Author

Split into smaller PRs for review: #6298 (synthetic routing), #6299 (required-rbf debug log), #6300 (replica planner), #6301 (replica weight moves), #6302 (RoutedMoE wiring; depends on #6299-#6301). Same code, verified bitwise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-ci-tests Add this label to the PR to skip running CI tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant