Repository navigation
[MoE replication 2/4] Add online hot-expert replica planner - #6300
Shuwen-Fang wants to merge 1 commit into
Conversation
4f1ea19 to
6234bca
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
@gemini-cli /review |
|
🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
|
@gemini-cli /review |
|
🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request implements the second phase of the hot-expert replication stack for MoE models. It provides a JAX-based planner that dynamically computes expert replication plans to alleviate load imbalances in expert-parallel setups. The solution focuses on performance by using vectorized operations to match hot experts with available replica slots on lighter devices, while respecting board-level constraints and communication invariants. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
🤖 Hi @Shuwen-Fang, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
|
@gemini-cli /review |
|
🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
There was a problem hiding this comment.
📋 Review Summary
Verdict: Spec partially met; the planner gets stuck on unsplittable experts.
Top issues:
- 🔴 [Bug] Unsplittable experts cause an infinite loop in the planner.
sent[h, loc]is only updated ifokis True, so if an expert's source shards are too large to move, the sender gets stuck retrying the same expert in all subsequent rounds. → Updatesentunconditionally so the planner moves on to the next hottest expert. - 🟠 [Alignment]
ReplicaPlanis defined as a plain Python class. While this works for local variables, passing it throughjax.lax.condorjax.lax.scanwill crash with aTypeError. → ConvertReplicaPlanto a registered PyTree (e.g.@flax.struct.dataclass).
Inferred spec & validation
- Sources: PR description, code after.
- Problem: The ragged all-to-all MoE path lacks a planner for online hot-expert replication.
| Spec item | Implementation | Validating test |
|---|---|---|
| R1 (stated): at most one replica per (home, receiver) pair | ✅ met | test_plan_satisfies_weight_move_invariants ✅ |
| R2 (stated): distinct valid home IDs across each receiver's R slots | ✅ met | test_plan_satisfies_weight_move_invariants ✅ |
| R3 (stated): valid sends compacted in ascending receiver order | ✅ met | test_plan_satisfies_weight_move_invariants ✅ |
| R4 (stated): loop-free vectorized planner matching hottest to coldest | test_plan_reduces_max_load_vs_greedy_oracle |
|
| R5 (stated): Redirects whole source shards, preferring same-board sources | ✅ met | test_same_board_sources_preferred ✅ |
C1 (inferred): enabled=False returns identity plan |
✅ preserved | test_disabled_plan_is_identity ✅ |
| C2 (inferred): Empty slots do not clobber expert 0 | ✅ preserved | test_expert_zero_not_clobbered_by_empty_slots ✅ |
| import jax.numpy as jnp | ||
|
|
||
|
|
||
| class ReplicaPlan: |
There was a problem hiding this comment.
🟠 [Alignment] ReplicaPlan is explicitly documented as 'not a pytree'. In JAX/Flax codebases like MaxText, data structures holding jax.Arrays that are passed between layer components should be registered PyTrees. While a plain class works as a local variable, if this plan is eventually passed through jax.lax.cond (e.g., to conditionally enable replication) or jax.lax.scan, JAX will crash with a TypeError: Cannot return a non-PyTree. Convert it to a @flax.struct.dataclass (or typing.NamedTuple) to avoid this.
There was a problem hiding this comment.
Fixed in 822ea14: converted ReplicaPlan to @flax.struct.dataclass with num_slots and num_replicas marked pytree_node=False and array fields as PyTree children, covered by test_replica_plan_pytree_jit_and_cond.
| rep_mask = rep_mask.at[r, slot].set(jnp.where(ok[:, None], take, rep_mask[r, slot])) | ||
| free = free.at[r].add(-ok.astype(jnp.int32)) | ||
| send_cnt = send_cnt.at[h].add(ok.astype(jnp.int32)) | ||
| pair_used = pair_used.at[h, r].set(pair_used[h, r] | ok) |
There was a problem hiding this comment.
🔴 [Bug] If a hot expert's source shards are fundamentally unsplittable (i.e. the first source shard > half the load gap to the receiver), take evaluates to False and moved == 0. ok becomes False, so sent[h, loc] remains False. The sender will repeatedly select this exact same unsplittable expert in every subsequent round, failing each time, and completely starving its other experts of replication opportunities.
| pair_used = pair_used.at[h, r].set(pair_used[h, r] | ok) | |
| sent = sent.at[h, loc].set(True) |
There was a problem hiding this comment.
Fixed in 822ea14 with a conservative starvation heuristic: instead of unconditionally setting sent[h, loc] = True (which would prematurely skip an expert that merely paired with a warm partner or full receiver on round j before shift=True rotates it to a colder receiver on round j+1), unmovable marks sent[h, loc] when all positive non-home shards exceed half the current global max-to-min load gap (or there are zero non-home tokens), covered by test_unmovable_hottest_expert_does_not_starve_second_hottest.
|
@gemini-cli /review |
|
🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
There was a problem hiding this comment.
📋 Review Summary
Verdict: No significant issues found.
Inferred spec & validation
- Sources: PR description; code before/after.
- Problem: Online hot-expert replication planner is needed for the ragged all-to-all MoE path to balance load across devices without intra-shard token splitting.
| Spec item | Implementation | Validating test |
|---|---|---|
| R1 (stated): Loop-free vectorized planner executing 2 * R unrolled rounds of hottest-to-coldest device matching. | ✅ | ReplicaPlannerTest::test_plan_reduces_max_load_vs_greedy_oracle0 ✅ |
| R2 (stated): Redirects whole source shards, preferring same-board sources first, never the home device's own shard. | ✅ | ReplicaPlannerTest::test_same_board_sources_preferred ✅ |
| R3 (stated): Starvation heuristic skips experts whose non-home shards exceed half the global max-to-min load gap. | ✅ | ReplicaPlannerTest::test_unmovable_hottest_expert_does_not_starve_second_hottest ✅ |
C1 (stated): At most one replica per (home, receiver) device pair. |
✅ preserved | ReplicaPlannerTest::test_plan_satisfies_weight_move_invariants ✅ |
C2 (stated): Valid sends are compacted at send_slots[h, :k] in ascending receiver order. |
✅ preserved | ReplicaPlannerTest::test_plan_satisfies_weight_move_invariants ✅ |
C3 (stated): Empty replica slots (-1) never clobber expert 0 in the route table. |
✅ preserved | ReplicaPlannerTest::test_expert_zero_not_clobbered_by_empty_slots ✅ |
| """Online hot-expert replication planner for the ragged all-to-all MoE path. | ||
| Each expert-parallel (EP) device owns L = E / EP "home" experts and R extra | ||
| replica slots, so the physical slot space has EP * (L + R) slots. Each step, |
There was a problem hiding this comment.
Are all R extra replica slots always filled?
There was a problem hiding this comment.
No. R is an upper bound on replica slots per device, not a fill count. A slot is filled only when the planner finds a useful offload (positive token gain, free slot on the receiver, sender under expert_replica_max_send). Unfilled slots keep rep_home = -1, get no tokens in route_table, and receive no weights in extend_with_replicas. test_disabled_plan_is_identity and test_expert_zero_not_clobbered_by_empty_slots cover this.
822ea14 to
894680c
Compare
Description
Adds the pure-JAX online hot-expert replication planner (
src/maxtext/layers/moe_replica_planner.py, 176 lines) for the ragged all-to-all MoE path (part 2 of the hot-expert replication stack; no callers wired yet):ReplicaPlan:@flax.struct.dataclassPyTree holdingroute_table([EP, E]),rep_home([EP, R]),rep_slot([EP, R]),send_slots([EP, C]), andsend_peer_index([EP, EP]) as array children andnum_slots/num_replicasas static fields (pytree_node=False), documenting the three sender/receiver invariants relied on bymoe_replica_weights.extend_with_replicas(at most one replica per(home, receiver)pair, distinct valid home IDs across each receiver'sRslots, and valid sends compacted atsend_slots[h, :k]in ascending receiver order).compute_plan_iter: loop-free vectorized planner executing2 * Runrolled rounds of hottest-to-coldest device matching (at most one replica per logical expert per step). Redirects whole source shards (same-board sources first, never the home device's own shard) so no intra-shard token split or extra sort is needed, with a conservative starvation heuristic skipping experts whose positive non-home shards all exceed half the global max-to-min load gap.identity_route_table&_finalize_plan: constructs the physical slot routing table and sender/receiver metadata using order-independent.maxreductions so empty replica slots (-1) never clobber expert0.Tests
JAX_PLATFORMS=cpu pytest -v tests/unit/moe_replica_planner_test.py(14 passed):ReplicaPlannerTest::test_disabled_plan_is_identityReplicaPlannerTest::test_expert_zero_not_clobbered_by_empty_slotsReplicaPlannerTest::test_identity_route_tableReplicaPlannerTest::test_invalid_arguments_raiseReplicaPlannerTest::test_more_replica_slots_than_experts_supportedReplicaPlannerTest::test_plan_is_validReplicaPlannerTest::test_plan_reduces_max_load_vs_greedy_oracle0..test_plan_reduces_max_load_vs_greedy_oracle3ReplicaPlannerTest::test_plan_satisfies_weight_move_invariantsReplicaPlannerTest::test_replica_plan_pytree_jit_and_condReplicaPlannerTest::test_same_board_sources_preferredReplicaPlannerTest::test_unmovable_hottest_expert_does_not_starve_second_hottestChecklist
Before submitting this PR, please make sure (put X in square brackets):