Repository navigation
[MoE replication 4a/5] Add online hot-expert replication config flags and validation - #6374
Shuwen-Fang wants to merge 2 commits into
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces the necessary configuration infrastructure and validation logic to support online hot-expert replication on the ragged all-to-all MoE path. By adding explicit configuration flags and a robust pre-compilation validation step, the changes ensure that users can safely configure replication while preventing invalid or unsupported model configurations from proceeding to the compilation stage. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
🤖 Hi @Shuwen-Fang, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
There was a problem hiding this comment.
📋 Review Summary
Verdict: Spec met; one logic defect in config validation.
Top issues:
- [Bug]
bool()evaluation ofself.quantizationcauses a false positiveValueErrorwhen it is set to"te_no_quant". → Check against the specific unquantized Enum values instead.
Inferred spec & validation
- Sources: PR description; code before/after.
- Problem: Online hot-expert replication needs config flags and validation.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): Add 3 flags to base.yml and types.py |
✅ | moe_expert_replication_config_test.py::test_defaults_off ✅ |
R2 (stated): Add validate_expert_replication() to reject unsupported combinations |
moe_expert_replication_config_test.py::test_config_validator_rejects_unsupported_combinations ✅ |
|
| C1 (stated): No runtime behavior changes | ✅ preserved | ❌ none |
| "expert_replicas_per_device > 0 does not support moe_dropless_fallback " | ||
| "(the overflow check uses pre-replication logical shard counts)." | ||
| ) | ||
| if self.fp8_moe or self.fp8_moe_fake_quant or bool(self.quantization) or bool(self.te_gmm_quantization): |
There was a problem hiding this comment.
🔴 bool(self.quantization) evaluates to True for explicitly unquantized values like QuantizationType.TE_NO_QUANT ("te_no_quant"). This causes a false positive ValueError rejecting valid configs that explicitly disable quantization. Check against the Enum values instead.
| if self.fp8_moe or self.fp8_moe_fake_quant or bool(self.quantization) or bool(self.te_gmm_quantization): | |
| if ( | |
| self.fp8_moe | |
| or self.fp8_moe_fake_quant | |
| or self.quantization not in (None, QuantizationType.NONE, QuantizationType.TE_NO_QUANT) | |
| or self.te_gmm_quantization not in (None, TEGroupedGemmQuantizationType.EMPTY, TEGroupedGemmQuantizationType.TE_NO_QUANT) | |
| ): |
There was a problem hiding this comment.
Fixed in b146f04: TE_NO_QUANT is now treated as unquantized; added test.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Description
Adds the three configuration flags for online hot-expert replication on the ragged all-to-all MoE path (
expert_replicas_per_device=0default off,expert_replica_max_send=4,expert_replica_board_size=8) inbase.ymlandMoEGeneral, plusMaxTextConfig.validate_expert_replication()to fail fast before compilation on unsupported combinations (Ring-of-Experts,te_moe_block,moe_use_fused_gmm3d, embedding chunking, MoE biases, chunked wgrad accumulation,moe_dropless_fallback,EP=1, and quantized/FP8 MoE GEMMs).No runtime MoE behavior changes in this PR.
BUGS: b/585678451
Stack
log_required_ragged_buffer_factoron ragged all-to-allmoe_replica_planner.py)moe_replica_weights.py)expert_replicas_per_device=0)RoutedMoEwiring & CPU end-to-end parity testsTests
Tested in
tests/unit/moe_expert_replication_config_test.py(JAX_PLATFORMS=cpu pytest tests/unit/moe_expert_replication_config_test.py, 20 passed) covering defaults, flag parsing,expert_replica_max_send <= 8bounds, and every validator rejection rule.Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.