Skip to content

[MoE replication 4/4] Wire online hot-expert replication into the ragged all-to-all MoE path - #6302

Closed
Shuwen-Fang wants to merge 24 commits into
AI-Hypercomputer:mainfrom
Shuwen-Fang:moe-expert-replication-wiring
Closed

Shuwen-Fang wants to merge 24 commits into
AI-Hypercomputer:mainfrom
Shuwen-Fang:moe-expert-replication-wiring

Conversation

@Shuwen-Fang

@Shuwen-Fang Shuwen-Fang commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Wires online hot-expert replication (moe_replica_planner + moe_replica_weights, stacked on #6299, #6300, #6301) into the use_ring_of_experts=False ragged all-to-all (ra2a_and_route) MoE path behind 3 default-off configuration flags:

  • expert_replicas_per_device: 0 (>= 0; 0 = off, zero graph/HLO change when off)
  • expert_replica_max_send: 4 (1 <= max_send <= 8; bounds per-home fan-out and the backward return buffer)
  • expert_replica_board_size: 8 (>= 1; number of JAX EP devices — i.e., TensorCores along the expert mesh axis, which is 2 per chip on 2-TC/chip TPUs when ici_tensor_parallelism=1 and 1 per chip on 1-TC/chip TPUs — per physical board/tray so the planner prefers same-board sources: 4 or 8 on TPU v6e-8, 8 on TPU v7x (4 chips × 2 TCs), and 8 per 4-chip board or 16 per 8-chip paired board on SFS 4x4x1)

Architecture & Guardrails

  • Explicit RouteMetadata.replica_plan PyTree threading: ReplicaPlan is a @flax.struct.dataclass PyTree and ra2a_and_route returns it on RouteMetadata.replica_plan (Optional[Any] = None), consumed directly by _moe_body with zero mutable closure state and full PyTree safety across jax.jit, jax.checkpoint remat, and multi-branch jax.lax.cond.
  • Fail-fast config-time validation (MaxTextConfig.validate_expert_replication) + runtime checks: Rejects unsupported combinations before compilation (num_experts <= 1, sparse_matmul=False, te_moe_block=True, use_ring_of_experts=True, num_moe_emb_chunks > 0, mlp_bias=True, moe_accumulate_chunk_wgrad=True, moe_dropless_fallback != None, fp8_moe=True, fp8_moe_fake_quant=True, bool(quantization), bool(te_gmm_quantization), and EP == 1), and raises ValueError in ra2a_and_route if num_ep <= 1, is_batch_sharded_by_expert=False (batch size 1 decode), or num_experts % num_ep != 0 so replication can never silently no-op as R=0.
  • Target Qwen3.5-397B SFS recipe compatibility: Confirmed compatible with the benchmarked Qwen3.5-397B SFS 4x4x1 (EP=32) BF16 MoE GEMM recipe (model_name=qwen3.5-397b-a17b, dtype=bfloat16, weight_dtype=float32, fp8_moe=false, quantization="", ici_expert_parallelism=32, ici_fsdp_parallelism=1, per_device_batch_size=2, max_target_length=4096, use_ring_of_experts=false, sparse_matmul=true, megablox=true, expert_replicas_per_device=4, expert_replica_max_send=4, expert_replica_board_size=8/16, ragged_buffer_factor=1.1/1.2), which uses mlp_bias=false, num_moe_emb_chunks=0, moe_accumulate_chunk_wgrad=false, and moe_dropless_fallback=None.

Known Limitations (v1)

  • No FP8 / quantized MoE GEMM support yet: Online expert replication does not yet replicate per-expert quantization scales or QArray wrappers (fp8_moe, fp8_moe_fake_quant, quantization, te_gmm_quantization), so it cannot currently be combined with FP8 MoE recipes (such as those used in MLPerf / NPI FP8 configurations). All reported Qwen3.5-397B SFS 4x4x1 results used BF16 MoE GEMMs, which is a supported configuration.
  • No moe_dropless_fallback in v1 (operational guidance & follow-up): moe_dropless_fallback (along with use_ring_of_experts, mlp_bias, num_moe_emb_chunks, moe_accumulate_chunk_wgrad, and te_moe_block) is rejected at config initialization. Operational guidance: with replication, pick ragged_buffer_factor from a calibration run with log_required_ragged_buffer_factor=true plus headroom; tokens beyond the buffer are dropped, same as baseline with a fixed ragged_buffer_factor. Follow-up: the fallback (moe_dropless_fallback='layer') could use post-replication physical counts — which the log_required_ragged_buffer_factor probe already computes — to detect per-shard overflow (~35–50 LoC in RoutedMoE._check_ragged_overflow / sparse_matmul plus unit tests).
  • At most one replica per logical expert per step: Each logical expert receives at most one replica slot (home + 1 replica), so a single expert exceeding roughly 2x a device's mean load cannot be balanced across more than two devices.

Validation

1. TPU Performance & Correctness Evidence

  • Confirmed on the PR stack @ fa4b588e6 (http://xid/1000000001525390, SFS 4x4x1, n=3, 0 drops, aligned rbf): σ=0.60 958.96±0.26 → 916.62±0.19 ms (−42.3 ms, −4.4%); Qwen empirical replay (rbf 2.50 → 1.10) 1064.42±1.39 → 928.71±0.46 ms (−135.7 ms, −12.8%); 30-step max |Δloss| ≤ 5.8e-5.
  • Earlier SFS sfs=4x4x1 sweeps (EP=32, 4-layer Qwen3.5-397B BF16 MoE GEMMs, pdbs=2x4096) — measured on pre-split prototype tree (closed PR #6295 @ 0c1f24a64, internal Fig commits cea9cbbc82bd / f6d9750cb364, prior to splitting moe_expert_replication.py into #6298–#6302):
    • Default MoE ops (rbf=2.7 vs 1.2): reduces median step time from 1097.8 ± 0.8 ms (req=2.482) to 959.5 ± 0.0 ms (-12.6%, req=1.061, zero drops); at tightened rbf (2.5 vs 1.1), 1066.5 -> 937.2 ms (-12.1%).
    • On top of optimized MoE ops (moe_use_direct_token_gather=true + use_tokamax_gmm=true + use_gmm_v2=true, rbf=2.5 vs 1.1, 3 interleaved repeats on f6d9750cb364): reduces median step time from 1054.8 ± 0.3 ms to 925.4 ± 0.2 ms (-129.4 ms / -12.3% additive; -13.2% vs default-ops R=0 1066.1 ± 0.0 ms), 30-step BF16 relative loss diff 4.43e-6.
    • CPU plan-equivalence check (57bdd8933 vs 822ea140d / fa4b588e6, EP=32, E=512, K=8, R=4, max_send=4): 48/48 bit-identical route_tables across the Qwen3.5-397B empirical profile and synthetic lognormal sigma=0.40..1.60 at both board_size=8 and board_size=16.
  • TPU v6e-8 Correctness Gates:
    • PR-stack head (fa4b588e6):
      • Flag-off bitwise parity vs main (561de23e8, 8 steps, default expert_replicas_per_device=0, EP=8): bit-identical on all 8 steps (0.000e+00 max rel diff on loss, raw_grad_norm, and param_norm; loss 10.900139808654785 → 10.591382026672363).
      • Zero recompiles (expert_replicas_per_device=4, 12 steps, JAX_LOG_COMPILES=1): 0 XLA compilation events after step 0 across steps 1..11.
      • Active replication parity (EP=8, R=2 vs R=0, 4 steps): step=0 loss rel_diff=0.000e+00, max rel_diff=1.075e-05 across steps 0..3.
    • Pre-split prototype (#6295 @ 0c1f24a64): 12-step R=4 vs R=0 (EP=8, skewed routing) max rel diff loss=3.013e-06, raw_grad_norm=2.676e-06, param_norm=1.685e-07, REQUIRED_RBF 1.425-1.619 -> 1.017-1.036.

2. Automated CPU Unit Tests (tests/unit/moe_expert_replication_test.py — 11 passed, 28 subtests passed)

  • ExpertReplicationConfigTest::test_config_validator_rejects_unsupported_combinations
  • ExpertReplicationConfigTest::test_defaults_off
  • ExpertReplicationConfigTest::test_expert_replica_max_send_capped_at_8
  • ExpertReplicationConfigTest::test_flags_parse
  • ExpertReplicationConfigTest::test_invalid_bounds_rejected
  • RoutedMoEReplicationWiringTest::test_end_to_end_sparse_matmul_parity_and_required_rbf
  • RoutedMoEReplicationWiringTest::test_guardrails_raise_on_unsupported_configs
  • RoutedMoEReplicationWiringTest::test_permute_with_physical_route_fn
  • RoutedMoEReplicationWiringTest::test_remat_and_multi_branch_trace_parity_with_r0
  • RoutedMoEReplicationWiringTest::test_route_metadata_with_replica_plan_is_valid_pytree
  • RoutedMoEReplicationWiringTest::test_routed_bias_updates_match_r0

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests locally and provided test artifacts above, if applicable.
  • I have made or will make corresponding changes to the doc if needed.

Default off. When on, each MoE layer prints the ragged_buffer_factor it needs to avoid dropping tokens (max tokens received by an EP shard / tokens per shard).
@github-actions github-actions Bot added the skip-ci-tests Add this label to the PR to skip running CI tests label Nov 12, 2026
Pure-jnp planners (greedy, vectorized rounds, and lagged placement) that choose which hot experts get replica slots on which EP devices and reroute whole source shards to them. Not wired into the MoE layer yet.
Custom-VJP ops that move replica expert weights to their receiving EP devices with one ragged all-to-all per weight array (gather, direct and split variants) and add replica gradients back into the home experts. Not wired into the MoE layer yet.
Adds the expert_replica_* flags (expert_replicas_per_device=0, i.e. off, by default) and uses moe_replica_planner and moe_replica_weights in RoutedMoE: tokens are remapped to physical (home or replica) slots before the dispatch sort, replica weights are moved per layer, and replica slots are computed by the expert GMMs.
@Shuwen-Fang
Shuwen-Fang force-pushed the moe-expert-replication-wiring branch from 1242c48 to eed7de2 Compare November 13, 2026 00:37
@Shuwen-Fang Shuwen-Fang changed the title [MoE replication 5/5] Wire online hot-expert replication into the ragged all-to-all MoE path [MoE replication 4/4] Wire online hot-expert replication into the ragged all-to-all MoE path Nov 13, 2026
@codecov

codecov Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.13514% with 9 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/maxtext/layers/moe.py 65.38% 7 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

@Shuwen-Fang

Copy link
Copy Markdown
Collaborator Author

@gemini-cli /review

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 I'm sorry @Shuwen-Fang, but I was unable to process your request. Please see the logs for more details.

@Shuwen-Fang

Copy link
Copy Markdown
Collaborator Author

@gemini-cli /review

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: Spec met across all acceptance criteria with robust test coverage; no significant issues found.

Inferred spec & validation
  • Sources: PR description (#6302), linked PRs (#6299, #6300, #6301), code before/after across src/maxtext/layers/moe.py, src/maxtext/layers/moe_replica_planner.py, src/maxtext/layers/moe_replica_weights.py, and src/maxtext/configs/.
  • Problem: Token routing hotspots in MoE overload individual expert-parallel devices on the ragged all-to-all path, forcing high ragged buffer factors and stranding idle compute on cold devices.
Spec item Implementation Validating test
R1 (stated): Online hot-expert replication is wired into the use_ring_of_experts=False ragged all-to-all path (ra2a_and_route) when expert_replicas_per_device > 0, lowering required buffer factors under skewed traffic. ✅ met moe_expert_replication_test.py::RoutedMoEReplicationWiringTest::test_end_to_end_sparse_matmul_parity_and_required_rbf ✅
R2 (stated): Default configuration (expert_replicas_per_device=0) remains a zero-overhead no-op with bitwise numerical parity. ✅ met moe_expert_replication_test.py::ExpertReplicationConfigTest::test_defaults_off ✅, moe_replica_weights_test.py::ExtendWithReplicasTest::test_noop_when_zero_replicas_or_single_device ✅
R3 (stated): ReplicaPlan threads through RouteMetadata as a registered Flax PyTree, safely crossing JIT compilation, jax.checkpoint remat, and jax.lax.cond. ✅ met moe_expert_replication_test.py::RoutedMoEReplicationWiringTest::test_route_metadata_with_replica_plan_is_valid_pytree ✅, test_remat_and_multi_branch_trace_parity_with_r0 ✅
R4 (stated): Gradients of replica weights on receiver devices return to home devices via custom VJP and accumulate into home expert gradients in master weight dtype. ✅ met moe_replica_weights_test.py::ExtendWithReplicasTest::test_forward_and_backward_sum_multiple_replicas_to_home ✅, test_slot0_with_unused_send_slots_padded_to_zero ✅, test_mixed_precision_fp32_master_weights_to_bf16_compute ✅
R5 (stated): Config validation rejects incompatible MoE features (Ring of Experts, FP8/quantized MoE, dropless fallback, biases, embedding chunking, EP=1) fail-fast. ✅ met moe_expert_replication_test.py::ExpertReplicationConfigTest::test_config_validator_rejects_unsupported_combinations ✅, test_guardrails_raise_on_unsupported_configs ✅
C1 (stated): Preserves backward compatibility of existing checkpoints and YAML configs when replication is disabled. ✅ preserved moe_expert_replication_test.py::ExpertReplicationConfigTest::test_defaults_off ✅
C2 (stated/inferred): Zero host-device synchronization, tracer leaks, or dynamic control flow inside JIT/shard_map hot paths. ✅ preserved moe_expert_replication_test.py::RoutedMoEReplicationWiringTest::test_end_to_end_sparse_matmul_parity_and_required_rbf ✅
C3 (inferred): Gradient accumulation on home devices uses XLA-fused elementwise loops without materializing intermediate dynamic scatter-add copies in HBM. ✅ preserved moe_replica_weights_test.py::ExtendWithReplicasTest::test_forward_and_backward_sum_multiple_replicas_to_home ✅

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @Shuwen-Fang, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates online hot-expert replication into the ragged all-to-all MoE path in MaxText. By allowing hot experts to be replicated across devices, the system can better balance token loads during training. The implementation includes a planner to compute replication plans, weight management logic to handle replicated expert weights, and strict guardrails to ensure compatibility with existing MoE configurations. The changes are designed to be opt-in via configuration flags and have been validated for performance and correctness on TPU hardware.

Highlights

  • Online Hot-Expert Replication: Introduced online hot-expert replication for the ragged all-to-all MoE path, allowing load balancing by replicating hot experts onto light devices.
  • Configuration Flags: Added three default-off configuration flags: expert_replicas_per_device, expert_replica_max_send, and expert_replica_board_size to control replication behavior.
  • Architecture & Safety: Implemented explicit RouteMetadata.replica_plan PyTree threading and added comprehensive config-time and runtime validation to prevent unsupported MoE configurations.
  • Performance: Validated performance improvements on SFS 4x4x1 configurations, showing significant reductions in step time (up to ~13%) and HBM usage.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: No significant issues found.

Inferred spec & validation
  • Sources: PR description, code before/after.
  • Problem: Need to wire online hot-expert replication into the ragged all-to-all MoE path to improve load balancing for models with skewed routing.
Spec item Implementation Validating test
R1 (stated): Rejects unsupported config combinations at init ✅ met moe_expert_replication_test.py::test_config_validator_rejects_unsupported_combinations ✅
R2 (stated): Validates runtime preconditions (num_ep > 1, is_batch_sharded_by_expert=True) in ra2a_and_route ✅ met moe_expert_replication_test.py::test_guardrails_raise_on_unsupported_configs ✅
R3 (stated): PyTree-safely threads replica_plan through RouteMetadata into _moe_body for weight extension ✅ met moe_expert_replication_test.py::test_route_metadata_with_replica_plan_is_valid_pytree ✅
C1 (stated): Off by default with zero graph/HLO change ✅ preserved moe_expert_replication_test.py::test_defaults_off ✅
C2 (stated): At most one replica per logical expert per step ✅ preserved moe_replica_planner_test.py::test_plan_satisfies_weight_move_invariants ✅

@Shuwen-Fang

Copy link
Copy Markdown
Collaborator Author

Split into #6374 (config) and #6375 (wiring) for easier review. Branch kept for history.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-ci-tests Add this label to the PR to skip running CI tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant