Skip to content

[MoE replication 4a/5] Add online hot-expert replication config flags and validation - #6374

Open
Shuwen-Fang wants to merge 2 commits into
AI-Hypercomputer:mainfrom
Shuwen-Fang:moe-expert-replication-config
Open

Shuwen-Fang wants to merge 2 commits into
AI-Hypercomputer:mainfrom
Shuwen-Fang:moe-expert-replication-config

Conversation

@Shuwen-Fang

@Shuwen-Fang Shuwen-Fang commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Adds the three configuration flags for online hot-expert replication on the ragged all-to-all MoE path (expert_replicas_per_device=0 default off, expert_replica_max_send=4, expert_replica_board_size=8) in base.yml and MoEGeneral, plus MaxTextConfig.validate_expert_replication() to fail fast before compilation on unsupported combinations (Ring-of-Experts, te_moe_block, moe_use_fused_gmm3d, embedding chunking, MoE biases, chunked wgrad accumulation, moe_dropless_fallback, EP=1, and quantized/FP8 MoE GEMMs).

No runtime MoE behavior changes in this PR.

BUGS: b/585678451

Stack

# PR Scope
1/5 #6299 log_required_ragged_buffer_factor on ragged all-to-all
2/5 #6300 Online hot-expert replica planner (moe_replica_planner.py)
3/5 #6301 Custom-VJP SparseCore weight replication (moe_replica_weights.py)
4a/5 #6374 (this PR) Config flags & pre-compile validation (expert_replicas_per_device=0)
4b/5 #6375 RoutedMoE wiring & CPU end-to-end parity tests
Bench #6298 Standalone synthetic routing replay benchmark

Tests

Tested in tests/unit/moe_expert_replication_config_test.py (JAX_PLATFORMS=cpu pytest tests/unit/moe_expert_replication_config_test.py, 20 passed) covering defaults, flag parsing, expert_replica_max_send <= 8 bounds, and every validator rejection rule.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed.

@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces the necessary configuration infrastructure and validation logic to support online hot-expert replication on the ragged all-to-all MoE path. By adding explicit configuration flags and a robust pre-compilation validation step, the changes ensure that users can safely configure replication while preventing invalid or unsupported model configurations from proceeding to the compilation stage.

Highlights

  • Configuration Flags: Added three new configuration flags (expert_replicas_per_device, expert_replica_max_send, and expert_replica_board_size) to base.yml and MoEGeneral to support online hot-expert replication.
  • Pre-compile Validation: Implemented MaxTextConfig.validate_expert_replication() to ensure compatibility by failing fast if unsupported features like Ring-of-Experts, FP8/quantized MoE, or specific chunking methods are enabled.
  • Unit Testing: Added tests/unit/moe_expert_replication_config_test.py to verify flag parsing, bounds checking, and validation logic for various configuration combinations.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @Shuwen-Fang, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: Spec met; one logic defect in config validation.

Top issues:

  1. [Bug] bool() evaluation of self.quantization causes a false positive ValueError when it is set to "te_no_quant". → Check against the specific unquantized Enum values instead.
Inferred spec & validation
  • Sources: PR description; code before/after.
  • Problem: Online hot-expert replication needs config flags and validation.
Spec item Implementation Validating test
R1 (stated): Add 3 flags to base.yml and types.py ✅ moe_expert_replication_config_test.py::test_defaults_off ✅
R2 (stated): Add validate_expert_replication() to reject unsupported combinations ⚠️ partial moe_expert_replication_config_test.py::test_config_validator_rejects_unsupported_combinations ✅
C1 (stated): No runtime behavior changes ✅ preserved ❌ none

Comment thread src/maxtext/configs/types.py Outdated
"expert_replicas_per_device > 0 does not support moe_dropless_fallback "
"(the overflow check uses pre-replication logical shard counts)."
)
if self.fp8_moe or self.fp8_moe_fake_quant or bool(self.quantization) or bool(self.te_gmm_quantization):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 bool(self.quantization) evaluates to True for explicitly unquantized values like QuantizationType.TE_NO_QUANT ("te_no_quant"). This causes a false positive ValueError rejecting valid configs that explicitly disable quantization. Check against the Enum values instead.

Suggested change
if self.fp8_moe or self.fp8_moe_fake_quant or bool(self.quantization) or bool(self.te_gmm_quantization):
if (
self.fp8_moe
or self.fp8_moe_fake_quant
or self.quantization not in (None, QuantizationType.NONE, QuantizationType.TE_NO_QUANT)
or self.te_gmm_quantization not in (None, TEGroupedGemmQuantizationType.EMPTY, TEGroupedGemmQuantizationType.TE_NO_QUANT)
):

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b146f04: TE_NO_QUANT is now treated as unquantized; added test.

@codecov

codecov Bot commented Nov 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-ci-tests Add this label to the PR to skip running CI tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant