Repository navigation
Add synthetic MoE routing benchmark harness under benchmarks/moe_synthetic_routing - #6298
Shuwen-Fang wants to merge 1 commit into
Conversation
22555a7 to
8d98378
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
@gemini-cli /review |
|
🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
There was a problem hiding this comment.
📋 Review Summary
Verdict: Spec met. High-quality, self-contained benchmark implementation with a minor memory leak logic flaw.
Top issues:
- [Bug] The
pools_by_loaderdictionary holds a strong reference toDataLoaderinstances in its values(self, sr), creating a memory leak. → Use aweakref.WeakKeyDictionaryto safely associate the synthetic routing state with theDataLoaderwithout preventing its garbage collection (see inline comment).
Inferred spec & validation
- Sources: PR description, code after.
- Problem: Need a reproducible, steady-state MoE routing distribution (lognormal or balanced) that injects
forced_routed_expertsbatches without modifying MaxText core config/schemas.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): Generate forced_routed_experts [batch, seq, num_layers, top_k] in lognormal and balanced modes. |
✅ | test_lognormal_shape_dtype_and_range ✅ |
R2 (stated): Transparent CLI parser that intercepts synthetic_routing args. |
✅ | test_flag_and_kv_parsing_and_pyconfig_compat ✅ |
R3 (stated): Install runtime hooks on DataLoader.load_next_batch and maxtext_utils.get_shaped_batch. |
✅ | test_runtime_hooks_patch_and_restore ✅ |
R4 (stated): Zero changes to src/maxtext/. |
✅ | (No tests required) |
| C1 (inferred): Deterministic and exactly identical across runs when seeded. | ✅ preserved | test_lognormal_is_deterministic ✅ |
|
|
||
| @functools.wraps(orig_load_next_batch) | ||
| def wrapped_load_next_batch(self, *args, **kwargs): | ||
| example_batch = orig_load_next_batch(self, *args, **kwargs) | ||
| entry = pools_by_loader.get(id(self)) | ||
| if entry is None or entry[0] is not self: | ||
| sr = synthetic_routing.SyntheticRouting(self.config, self.input_data_shardings, syn_cfg=syn_cfg) | ||
| pools_by_loader[id(self)] = (self, sr) | ||
| else: | ||
| sr = entry[1] | ||
| example_batch = dict(example_batch) |
There was a problem hiding this comment.
🟠 The pools_by_loader dictionary maps id(self) to (self, sr). Since the value contains a strong reference to self (the DataLoader instance), the DataLoader will never be garbage collected. Since instances of DataLoader don't override __slots__ and are hashable by identity, you can safely use a weakref.WeakKeyDictionary to associate state with the instance without preventing its collection or tracking IDs manually.
| @functools.wraps(orig_load_next_batch) | |
| def wrapped_load_next_batch(self, *args, **kwargs): | |
| example_batch = orig_load_next_batch(self, *args, **kwargs) | |
| entry = pools_by_loader.get(id(self)) | |
| if entry is None or entry[0] is not self: | |
| sr = synthetic_routing.SyntheticRouting(self.config, self.input_data_shardings, syn_cfg=syn_cfg) | |
| pools_by_loader[id(self)] = (self, sr) | |
| else: | |
| sr = entry[1] | |
| example_batch = dict(example_batch) | |
| import weakref | |
| pools_by_loader = weakref.WeakKeyDictionary() | |
| @functools.wraps(orig_load_next_batch) | |
| def wrapped_load_next_batch(self, *args, **kwargs): | |
| example_batch = orig_load_next_batch(self, *args, **kwargs) | |
| sr = pools_by_loader.get(self) | |
| if sr is None: | |
| sr = synthetic_routing.SyntheticRouting(self.config, self.input_data_shardings, syn_cfg=syn_cfg) | |
| pools_by_loader[self] = sr |
There was a problem hiding this comment.
Fixed in 858075b: replaced dict[int, tuple[object, SyntheticRouting]] with weakref.WeakKeyDictionary and added a GC weakref check in test_runtime_hooks_patch_and_restore.
|
@gemini-cli /review |
|
🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
|
@gemini-cli /review |
|
🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
|
@gemini-cli /review |
|
🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
|
@gemini-cli /review |
|
🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
|
🤖 I'm sorry @Shuwen-Fang, but I was unable to process your request. Please see the logs for more details. |
|
@gemini-cli /review |
|
🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
There was a problem hiding this comment.
📋 Review Summary
Verdict: No significant issues found.
Inferred spec & validation
- Sources: PR description, code before/after.
- Problem: Benchmarking MoE load-imbalance requires a controlled, reproducible expert distribution without intrusive modifications to core MaxText config/modules.
| Spec item | Implementation | Validating test |
|---|---|---|
| R1 (stated): Add a standalone synthetic-routing benchmark wrapper. | ✅ | test_readme_defaults_match_synthetic_routing_cfg ✅ |
R2 (stated): Support generating lognormal and balanced forced_routed_experts. |
✅ | test_balanced_is_exactly_balanced ✅ |
R3 (stated): Hook into get_shaped_batch and load_next_batch without modifying core schemas. |
✅ | test_runtime_hooks_patch_and_restore ✅ |
| R4 (stated): Include overhead control ablation wrapper. | ✅ | test_overhead_control_hook_zeroes_replica_routing_and_restores ✅ |
C1 (inferred): Must not leak hooks into main or after execution. |
✅ preserved | test_runtime_hooks_patch_and_restore ✅ |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request adds a flexible, standalone synthetic routing benchmark harness for Mixture-of-Experts (MoE) models. By injecting deterministic or balanced expert routing assignments directly into the data pipeline, it allows for precise load-balancing and overhead-control experiments without requiring changes to the core MaxText architecture or configuration schemas. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
🤖 Hi @Shuwen-Fang, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the |
There was a problem hiding this comment.
📋 Review Summary
Verdict: Clean, well-tested MoE routing benchmark harness; address boolean CLI flag parsing and minor edge cases before merge.
Top issues:
- [Bug]
parse_synthetic_routing_argsconsumes the subsequent CLI argument when boolean options like--synthetic_routing_overhead_controlare passed without=true, crashing withValueError: Expected boolean for synthetic routing optionon valid CLI invocations. → Support standalone boolean switch flags by settingTruewithout consumingargv[idx + 1](see inline comment onsynthetic_routing.py). - [Bug]
depth_map_profilesassumes empirical.npzprofiles are strictly 2D[P, E], raisingAxisErroralongaxis=1if provided with a 1D vector[E]. → Wrapprofileswithnp.atleast_2dto support both 1D and 2D profiles cleanly (see inline comment onsynthetic_routing.py). - [Complexity] Hook installation in
install_synthetic_routingoccurs outside thetry:block, risking leaked monkey patches if initialization fails. → Move hook setup inside thetry:block to ensurefinally:restoration (see inline comment ontrain.py).
Inferred spec & validation
- Sources: PR #6298 description;
forced_routed_expertsintegration inmaxtext.layers.nnx_decodersandmaxtext.layers.moe;FORCED_ROUTING_SUPPORTED_DECODER_BLOCKSinmaxtext.configs.types. - Problem: Benchmarking MoE load balancing and ragged buffer factors on real weights/data couples routing behavior to model state and dataset, preventing reproducible ablation of routing imbalance without modifying core MaxText code or schemas.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): Deterministic top-k expert assignment generator producing [batch, seq, num_layers, top_k] int32 batches |
✅ met | moe_synthetic_routing_test.py::SyntheticRoutingGeneratorTest::test_lognormal_* ✅ |
| R2 (stated): Uniform balanced routing baseline where every expert receives the exact mean token count | ✅ met | moe_synthetic_routing_test.py::SyntheticRoutingGeneratorTest::test_balanced_is_exactly_balanced ✅ |
R3 (stated): Lognormal routing with per-layer sigma/sigmas, document affinity jitter (row_tau, row_len), and optional .npz profile |
✅ met | moe_synthetic_routing_test.py::SyntheticRoutingGeneratorTest::test_profile_file_depth_mapping ✅ |
R4 (stated): Benchmark entrypoint wrapping get_shaped_batch and DataLoader.load_next_batch without core schema changes |
✅ met | moe_synthetic_routing_test.py::SyntheticRoutingWrapperTest::test_runtime_hooks_patch_and_restore, test_forced_routing_reaches_routed_moe_and_changes_group_sizes ✅ |
R5 (stated): Hot-expert replication overhead-control ablation hook (--synthetic_routing_overhead_control=true) |
✅ met | moe_synthetic_routing_test.py::SyntheticRoutingWrapperTest::test_overhead_control_hook_zeroes_replica_routing_and_restores, test_overhead_control_missing_planner_raises_clear_import_error ✅ |
C1 (stated): Zero modification to core src/maxtext/ modules or config schemas |
✅ preserved | moe_synthetic_routing_test.py::SyntheticRoutingWrapperTest::test_flag_and_kv_parsing_and_pyconfig_compat ✅ |
| C2 (inferred): Clean restoration of monkey-patched methods on context exit | ✅ preserved | moe_synthetic_routing_test.py::SyntheticRoutingWrapperTest::test_runtime_hooks_patch_and_restore ✅ |
| C3 (inferred): No memory leaks or per-step host syncs in hot dispatch loops | ✅ preserved | moe_synthetic_routing_test.py::SyntheticRoutingWrapperTest::test_runtime_hooks_patch_and_restore (weakref GC check) ✅ |
| if "=" in stripped: | ||
| raw_val = stripped.split("=", 1)[1] | ||
| elif arg.startswith("--") and idx + 1 < len(argv): | ||
| idx += 1 | ||
| raw_val = argv[idx] | ||
| else: | ||
| raise ValueError(f"Missing value for synthetic routing option: {arg!r}") | ||
| caster = _CFG_FIELD_TYPES[key] | ||
| if key == "synthetic_routing" and raw_val.strip().lower() in ("none", "null", ""): | ||
| parsed[key] = "none" | ||
| elif caster is bool: | ||
| parsed[key] = _parse_bool(raw_val) | ||
| else: | ||
| parsed[key] = caster(raw_val) |
There was a problem hiding this comment.
🟠 When --synthetic_routing_overhead_control is passed as a boolean flag without =true (standard CLI boolean switch syntax), the parser unconditionally consumes the next argument as its value via argv[idx + 1]. If followed by another CLI argument (such as a config path), it attempts to parse that argument as a bool and crashes with ValueError: Expected boolean for synthetic routing option; if passed at the end of argv, it fails with Missing value.
| if "=" in stripped: | |
| raw_val = stripped.split("=", 1)[1] | |
| elif arg.startswith("--") and idx + 1 < len(argv): | |
| idx += 1 | |
| raw_val = argv[idx] | |
| else: | |
| raise ValueError(f"Missing value for synthetic routing option: {arg!r}") | |
| caster = _CFG_FIELD_TYPES[key] | |
| if key == "synthetic_routing" and raw_val.strip().lower() in ("none", "null", ""): | |
| parsed[key] = "none" | |
| elif caster is bool: | |
| parsed[key] = _parse_bool(raw_val) | |
| else: | |
| parsed[key] = caster(raw_val) | |
| caster = _CFG_FIELD_TYPES[key] | |
| if caster is bool and "=" not in stripped: | |
| if idx + 1 < len(argv) and argv[idx + 1].strip().lower() in ("true", "false", "1", "0", "yes", "no"): | |
| idx += 1 | |
| parsed[key] = _parse_bool(argv[idx]) | |
| else: | |
| parsed[key] = True | |
| else: | |
| if "=" in stripped: | |
| raw_val = stripped.split("=", 1)[1] | |
| elif arg.startswith("--") and idx + 1 < len(argv): | |
| idx += 1 | |
| raw_val = argv[idx] | |
| else: | |
| raise ValueError(f"Missing value for synthetic routing option: {arg!r}") | |
| if key == "synthetic_routing" and raw_val.strip().lower() in ("none", "null", ""): | |
| parsed[key] = "none" | |
| elif caster is bool: | |
| parsed[key] = _parse_bool(raw_val) | |
| else: | |
| parsed[key] = caster(raw_val) |
There was a problem hiding this comment.
Fixed in fae56dfc9: parse_synthetic_routing_args now handles standalone boolean switch flags (e.g. --synthetic_routing_overhead_control without =true) without consuming the next non-boolean CLI argument, covered by test_bare_boolean_switch_flag_parsing.
| """Maps `[P, E_in]` log-profiles onto `[num_layers, num_experts]` sorted descending.""" | ||
| p = -np.sort(-np.asarray(profiles, dtype=np.float64), axis=1) | ||
| if p.shape[1] != num_experts: | ||
| q_in = (np.arange(p.shape[1]) + 0.5) / p.shape[1] |
There was a problem hiding this comment.
🟡 depth_map_profiles assumes profiles is strictly 2D [P, E], raising AxisError along axis=1 if an empirical .npz profile provides a 1D vector [E]. Wrapping profiles with np.atleast_2d allows 1D profiles to be promoted to [1, E] and broadcast across layers cleanly via the existing depth mapping.
| q_in = (np.arange(p.shape[1]) + 0.5) / p.shape[1] | |
| p = -np.sort(-np.atleast_2d(np.asarray(profiles, dtype=np.float64)), axis=1) |
There was a problem hiding this comment.
Fixed in fae56dfc9: wrapped profiles with np.atleast_2d in depth_map_profiles and added test_profile_file_1d_vector_supported.
| try: | ||
| from maxtext.layers import moe_replica_planner as mrp_module # pylint: disable=import-outside-toplevel | ||
| except ImportError as exc: | ||
| raise ImportError( | ||
| "--synthetic_routing_overhead_control=true requires maxtext.layers.moe_replica_planner " | ||
| "(PRs #6300/#6302); on standalone main, moe_replica_planner is not present." | ||
| ) from exc | ||
|
|
||
| orig_compute_plan_iter = mrp_module.compute_plan_iter | ||
|
|
||
| @functools.wraps(orig_compute_plan_iter) | ||
| def overhead_control_plan_iter(counts_src, num_ep, num_replicas, *args, **kwargs): | ||
| plan = orig_compute_plan_iter(counts_src, num_ep, num_replicas, *args, **kwargs) | ||
| plan.route_table = mrp_module.identity_route_table(num_ep, counts_src.shape[1], num_replicas) | ||
| return plan | ||
|
|
||
| mrp_module.compute_plan_iter = overhead_control_plan_iter | ||
|
|
||
| if syn_cfg.enabled: | ||
|
|
||
| @functools.wraps(orig_get_shaped_batch) | ||
| def wrapped_get_shaped_batch(config, batch_sharding=None, is_eval=False): | ||
| shaped_batch = orig_get_shaped_batch(config, batch_sharding=batch_sharding, is_eval=is_eval) | ||
| if not is_eval: | ||
| routing_shape = shaped_batch["inputs"].shape + ( | ||
| int(config.num_decoder_layers), | ||
| int(config.num_experts_per_tok), | ||
| ) | ||
| shaped_batch = dict(shaped_batch) | ||
| shaped_batch["forced_routed_experts"] = jax.ShapeDtypeStruct(routing_shape, jnp.int32, sharding=batch_sharding) | ||
| return shaped_batch | ||
|
|
||
| pools_by_loader: weakref.WeakKeyDictionary[object, synthetic_routing.SyntheticRouting] = weakref.WeakKeyDictionary() | ||
|
|
||
| @functools.wraps(orig_load_next_batch) | ||
| def wrapped_load_next_batch(self, *args, **kwargs): | ||
| example_batch = orig_load_next_batch(self, *args, **kwargs) | ||
| sr = pools_by_loader.get(self) | ||
| if sr is None: | ||
| sr = synthetic_routing.SyntheticRouting(self.config, self.input_data_shardings, syn_cfg=syn_cfg) | ||
| pools_by_loader[self] = sr | ||
| example_batch = dict(example_batch) | ||
| example_batch["forced_routed_experts"] = sr.next() | ||
| return example_batch | ||
|
|
||
| maxtext_utils.get_shaped_batch = wrapped_get_shaped_batch | ||
| data_loader.DataLoader.load_next_batch = wrapped_load_next_batch | ||
|
|
||
| try: | ||
| yield | ||
| finally: | ||
| maxtext_utils.get_shaped_batch = orig_get_shaped_batch | ||
| data_loader.DataLoader.load_next_batch = orig_load_next_batch | ||
| if mrp_module is not None and orig_compute_plan_iter is not None: |
There was a problem hiding this comment.
🟡 If an exception is raised while setting up the data loader hooks, mrp_module.compute_plan_iter will remain monkey-patched because hook installation occurs before the try: block. Enclosing all patching operations inside the try: block guarantees that previously applied hooks are always restored on error.
| try: | |
| from maxtext.layers import moe_replica_planner as mrp_module # pylint: disable=import-outside-toplevel | |
| except ImportError as exc: | |
| raise ImportError( | |
| "--synthetic_routing_overhead_control=true requires maxtext.layers.moe_replica_planner " | |
| "(PRs #6300/#6302); on standalone main, moe_replica_planner is not present." | |
| ) from exc | |
| orig_compute_plan_iter = mrp_module.compute_plan_iter | |
| @functools.wraps(orig_compute_plan_iter) | |
| def overhead_control_plan_iter(counts_src, num_ep, num_replicas, *args, **kwargs): | |
| plan = orig_compute_plan_iter(counts_src, num_ep, num_replicas, *args, **kwargs) | |
| plan.route_table = mrp_module.identity_route_table(num_ep, counts_src.shape[1], num_replicas) | |
| return plan | |
| mrp_module.compute_plan_iter = overhead_control_plan_iter | |
| if syn_cfg.enabled: | |
| @functools.wraps(orig_get_shaped_batch) | |
| def wrapped_get_shaped_batch(config, batch_sharding=None, is_eval=False): | |
| shaped_batch = orig_get_shaped_batch(config, batch_sharding=batch_sharding, is_eval=is_eval) | |
| if not is_eval: | |
| routing_shape = shaped_batch["inputs"].shape + ( | |
| int(config.num_decoder_layers), | |
| int(config.num_experts_per_tok), | |
| ) | |
| shaped_batch = dict(shaped_batch) | |
| shaped_batch["forced_routed_experts"] = jax.ShapeDtypeStruct(routing_shape, jnp.int32, sharding=batch_sharding) | |
| return shaped_batch | |
| pools_by_loader: weakref.WeakKeyDictionary[object, synthetic_routing.SyntheticRouting] = weakref.WeakKeyDictionary() | |
| @functools.wraps(orig_load_next_batch) | |
| def wrapped_load_next_batch(self, *args, **kwargs): | |
| example_batch = orig_load_next_batch(self, *args, **kwargs) | |
| sr = pools_by_loader.get(self) | |
| if sr is None: | |
| sr = synthetic_routing.SyntheticRouting(self.config, self.input_data_shardings, syn_cfg=syn_cfg) | |
| pools_by_loader[self] = sr | |
| example_batch = dict(example_batch) | |
| example_batch["forced_routed_experts"] = sr.next() | |
| return example_batch | |
| maxtext_utils.get_shaped_batch = wrapped_get_shaped_batch | |
| data_loader.DataLoader.load_next_batch = wrapped_load_next_batch | |
| try: | |
| yield | |
| finally: | |
| maxtext_utils.get_shaped_batch = orig_get_shaped_batch | |
| data_loader.DataLoader.load_next_batch = orig_load_next_batch | |
| if mrp_module is not None and orig_compute_plan_iter is not None: | |
| try: | |
| if syn_cfg.synthetic_routing_overhead_control: | |
| try: | |
| from maxtext.layers import moe_replica_planner as mrp_module # pylint: disable=import-outside-toplevel | |
| except ImportError as exc: | |
| raise ImportError( | |
| "--synthetic_routing_overhead_control=true requires maxtext.layers.moe_replica_planner " | |
| "(PRs #6300/#6302); on standalone main, moe_replica_planner is not present." | |
| ) from exc | |
| orig_compute_plan_iter = mrp_module.compute_plan_iter | |
| @functools.wraps(orig_compute_plan_iter) | |
| def overhead_control_plan_iter(counts_src, num_ep, num_replicas, *args, **kwargs): | |
| plan = orig_compute_plan_iter(counts_src, num_ep, num_replicas, *args, **kwargs) | |
| plan.route_table = mrp_module.identity_route_table(num_ep, counts_src.shape[1], num_replicas) | |
| return plan | |
| mrp_module.compute_plan_iter = overhead_control_plan_iter | |
| if syn_cfg.enabled: | |
| @functools.wraps(orig_get_shaped_batch) | |
| def wrapped_get_shaped_batch(config, batch_sharding=None, is_eval=False): | |
| shaped_batch = orig_get_shaped_batch(config, batch_sharding=batch_sharding, is_eval=is_eval) | |
| if not is_eval: | |
| routing_shape = shaped_batch["inputs"].shape + ( | |
| int(config.num_decoder_layers), | |
| int(config.num_experts_per_tok), | |
| ) | |
| shaped_batch = dict(shaped_batch) | |
| shaped_batch["forced_routed_experts"] = jax.ShapeDtypeStruct(routing_shape, jnp.int32, sharding=batch_sharding) | |
| return shaped_batch | |
| pools_by_loader: weakref.WeakKeyDictionary[object, synthetic_routing.SyntheticRouting] = weakref.WeakKeyDictionary() | |
| @functools.wraps(orig_load_next_batch) | |
| def wrapped_load_next_batch(self, *args, **kwargs): | |
| example_batch = orig_load_next_batch(self, *args, **kwargs) | |
| sr = pools_by_loader.get(self) | |
| if sr is None: | |
| sr = synthetic_routing.SyntheticRouting(self.config, self.input_data_shardings, syn_cfg=syn_cfg) | |
| pools_by_loader[self] = sr | |
| example_batch = dict(example_batch) | |
| example_batch["forced_routed_experts"] = sr.next() | |
| return example_batch | |
| maxtext_utils.get_shaped_batch = wrapped_get_shaped_batch | |
| data_loader.DataLoader.load_next_batch = wrapped_load_next_batch | |
| yield | |
| finally: |
There was a problem hiding this comment.
Fixed in fae56dfc9: moved all hook installation inside the try: block (and switched overhead_control_plan_iter to dataclasses.replace(plan, ...) for frozen @flax.struct.dataclass ReplicaPlan compatibility), covered by test_partial_hook_install_failure_restores_all_originals and test_overhead_control_hook_with_real_replica_planner.
|
@gemini-cli /review |
|
🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
|
@gemini-cli /review |
|
🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details. |
There was a problem hiding this comment.
📋 Review Summary
Verdict: Spec largely met; very high quality, but contains a critical JAX PyTree mismatch bug on evaluation runs.
Top issues:
- 🔴 [Bug]
wrapped_get_shaped_batchskips addingforced_routed_expertsto the signature during evaluation (is_eval=True), butwrapped_load_next_batchunconditionally injects the key into all data batches. This causes a JAX PyTree mismatch when evaluation runs, crashing the step, andSyntheticRoutingassumes the training batch size, which would mismatch the eval batch size. → Unconditionally add the key inwrapped_get_shaped_batch, and pass the actualbatch_sizefrom the data loader batch toSyntheticRouting. - 🟠 [Alignment] Calling
np.load()on.npzfiles directly via index extraction leaves the internalZipFileunclosed, which can raise aResourceWarningduring garbage collection. → Use awith np.load(...) as data:context manager.
Inferred spec & validation
- Sources: PR description, code before/after.
- Problem: Need a way to run controlled MoE load-imbalance experiments via synthetic routing assignments (lognormal/balanced) without modifying core MaxText code, plus an overhead-control flag.
| Spec item | Implementation | Validating test |
|---|---|---|
R1 (stated): Generate synthetic routing matrices and inject them as forced_routed_experts via hooks. |
✅ | test_forced_routing_reaches_routed_moe_and_changes_group_sizes ✅ |
R2 (stated): Do this without altering core src/maxtext/ code. |
✅ | test_flag_and_kv_parsing_and_pyconfig_compat ✅ |
R3 (stated): Provide --synthetic_routing_overhead_control to override replica routing to identity. |
✅ | test_overhead_control_hook_with_real_replica_planner ✅ |
| C1 (inferred): Synthetic routing injection must not crash evaluation steps if they run. | ❌ violated | test_runtime_hooks_patch_and_restore is_eval signature logic in isolation but does not simulate a full evaluation PyTree match). |
| argv: Sequence[str], | ||
| ) -> tuple[SyntheticRoutingCfg, list[str]]: | ||
| """Extracts `synthetic_routing*` options (both `--k=v` / `--k v` and `k=v`) from `argv`.""" | ||
| if not argv: |
There was a problem hiding this comment.
🟠 Calling np.load() on a file object without a context manager leaves the returned NpzFile (and its internal ZipFile) unclosed. Since ZipFile creates additional file-like objects, failing to close it properly can raise ResourceWarnings on garbage collection.
| argv: Sequence[str], | |
| ) -> tuple[SyntheticRoutingCfg, list[str]]: | |
| """Extracts `synthetic_routing*` options (both `--k=v` / `--k v` and `k=v`) from `argv`.""" | |
| if not argv: | |
| if "://" in prof_path or prof_path.startswith("/cns/"): | |
| from etils import epath # pylint: disable=import-outside-toplevel | |
| with epath.Path(prof_path).open("rb") as f: | |
| with np.load(f) as data: | |
| profile = data["log_profile"] |
There was a problem hiding this comment.
The log_profile array is eagerly read and materialized inside the with open(...) / with epath.Path(...).open(...) context before the underlying file handle closes, so no code change is needed here.
|
|
||
|
|
||
| def _parse_bool(raw_val: str) -> bool: | ||
| lowered = raw_val.strip().lower() | ||
| if lowered in ("true", "1", "yes"): | ||
| return True |
There was a problem hiding this comment.
🔴 Accept the actual batch_size to support evaluation runs (which use config.eval_per_device_batch_size), preventing shape mismatches in train.py.
| def _parse_bool(raw_val: str) -> bool: | |
| lowered = raw_val.strip().lower() | |
| if lowered in ("true", "1", "yes"): | |
| return True | |
| def __init__(self, config, sharding, syn_cfg: SyntheticRoutingCfg | None = None, batch_size: int | None = None): | |
| import jax # pylint: disable=import-outside-toplevel | |
| opts = syn_cfg if syn_cfg is not None else config | |
| self.mode = opts.synthetic_routing | |
| batch = int(batch_size if batch_size is not None else config.global_batch_size_to_load) |
| sr = synthetic_routing.SyntheticRouting(self.config, self.input_data_shardings, syn_cfg=syn_cfg) | ||
| pools_by_loader[self] = sr | ||
| example_batch = dict(example_batch) |
There was a problem hiding this comment.
🔴 Evaluation batch sizes (eval_per_device_batch_size) often differ from training batch sizes. Extract the actual batch size from the data loader's batch here to ensure forced_routed_experts matches the inputs batch dimension for eval runs.
| sr = synthetic_routing.SyntheticRouting(self.config, self.input_data_shardings, syn_cfg=syn_cfg) | |
| pools_by_loader[self] = sr | |
| example_batch = dict(example_batch) | |
| if sr is None: | |
| batch_size = example_batch["inputs"].shape[0] | |
| sr = synthetic_routing.SyntheticRouting(self.config, self.input_data_shardings, syn_cfg=syn_cfg, batch_size=batch_size) | |
| pools_by_loader[self] = sr |
| return SyntheticRoutingCfg(), [] | ||
| remaining: list[str] = [argv[0]] | ||
| parsed: dict[str, Any] = {} | ||
| idx = 1 |
There was a problem hiding this comment.
🟠 Use a context manager to ensure the NpzFile is closed immediately.
| return SyntheticRoutingCfg(), [] | |
| remaining: list[str] = [argv[0]] | |
| parsed: dict[str, Any] = {} | |
| idx = 1 | |
| else: | |
| with open(prof_path, "rb") as f: | |
| with np.load(f) as data: | |
| profile = data["log_profile"] |
5807f8b to
082c0e4
Compare
Description
Adds a standalone synthetic-routing benchmark wrapper under
benchmarks/moe_synthetic_routing/(synthetic_routing.py,train.py,README.md) to drive controlled MoE load-imbalance experiments viaforced_routed_expertswithout modifying any coresrc/maxtext/modules or config schemas.Key Components
benchmarks/moe_synthetic_routing/synthetic_routing.py:lognormal_routing: deterministic per-(seed, step, layer) top-kexpert assignment generator supporting per-layersigma, per-document affinity jitter (row_tau,row_len), and optional empirical.npzlog-popularity profiles (local,gs://, or/cns/, 1D[E]or 2D[P, E]).balanced_routing: exact uniform control where every expert receives the exact mean token count (required_rbf = 1.0).SyntheticRouting+parse_synthetic_routing_args: host batch pool and CLI flag extractor (--synthetic_routing=*orsynthetic_routing*=*), including--synthetic_routing_overhead_control/--synthetic_routing_overhead_control=truefor replica weight-move overhead ablation (requires#6300/#6302; on standalonemain, raises a descriptiveImportError).benchmarks/moe_synthetic_routing/train.py:install_synthetic_routing) wrappingmaxtext_utils.get_shaped_batchandDataLoader.load_next_batch(using aWeakKeyDictionaryper loader instance) insidetry/finallyand restoring originals on exit or setup error, plus optionalmoe_replica_planner.compute_plan_iterwrapper viadataclasses.replace(plan, route_table=...)when--synthetic_routing_overhead_controlis set (requires#6300/#6302).benchmarks/moe_synthetic_routing/README.md:FORCED_ROUTING_SUPPORTED_DECODER_BLOCKS(qwen3_5,mixtral, unscannedgemma4; pure-NNX decoder without pipeline parallelism and withmtp_num_layers=0), matches everySyntheticRoutingCfgdefault (verified bytest_readme_defaults_match_synthetic_routing_cfg), and provides a runnable 8-chip TPU v6e command.Tests
1. Automated CPU Unit Tests (
tests/unit/moe_synthetic_routing_test.py— 21 passed, 1 skipped on standalonemain; 22 passed with#6300/#6302, 5 subtests passed)SyntheticRoutingGeneratorTest:test_balanced_is_exactly_balanced,test_lognormal_depends_on_seed_and_step,test_lognormal_is_deterministic,test_lognormal_shape_dtype_and_range,test_lognormal_threads_match_serial,test_lognormal_topk_distinct_per_token,test_profile_file_1d_vector_supported,test_profile_file_depth_mappingSyntheticRoutingPoolTest:test_pool_cycles,test_unknown_mode_raisesSyntheticRoutingWrapperTest:test_bare_boolean_switch_flag_parsing,test_default_args_off,test_flag_and_kv_parsing_and_pyconfig_compat,test_forced_routing_reaches_routed_moe_and_changes_group_sizes,test_invalid_cli_options_rejected,test_main_delegates_with_stripped_argv,test_overhead_control_hook_with_real_replica_planner,test_overhead_control_hook_zeroes_replica_routing_and_restores,test_overhead_control_missing_planner_raises_clear_import_error,test_partial_hook_install_failure_restores_all_originals,test_readme_defaults_match_synthetic_routing_cfg,test_runtime_hooks_patch_and_restore2. TPU v6e-8 Verification
README.md8-chipra2acommand (EP=8, 4-layer scaledqwen3.5-397b-a17b,per_device_batch_size=2,max_target_length=1024,use_ring_of_experts=false):0.067s/stepsteady state;loss: 10.900 -> 10.554), and--synthetic_routing=balancedran cleanly (0.063s/step).REQUIRED_RBFverification on#6298(EP=8, 2-layerqwen3.5-397b-a17b,base_emb_dim=2048,use_ring_of_experts=true use_ragged_sort=true log_required_ragged_buffer_factor=true):--synthetic_routing=balanced:REQUIRED_RBF step=0..2 max=1.0000 per_layer=[1.0000,1.0000]--synthetic_routing=lognormal --synthetic_routing_sigmas=0.9 --synthetic_routing_row_taus=1.3 --synthetic_routing_row_len=512(L=64experts/shard):REQUIRED_RBF step=0 max=1.3367 per_layer=[1.1254,1.3367]--synthetic_routing=lognormal --synthetic_routing_sigma=2.0 --synthetic_routing_row_taus=1.3 --synthetic_routing_row_len=512(L=64experts/shard):REQUIRED_RBF step=0 max=2.0321 per_layer=[1.3970,2.0321]#6302@fa4b588e6, 4 steps,EP=8,num_experts=128,--synthetic_routing=lognormal --synthetic_routing_sigma=1.5):expert_replicas_per_device=4with--synthetic_routing_overhead_controlran 4 steps cleanly (0.000e+00diff onloss,raw_grad_norm,param_normvsR=0) and produced identicalREQUIRED_RBFtoR=0across all 4 steps (confirming replicas move while zero tokens are rerouted to replica slots).Checklist
Before submitting this PR, please make sure (put X in square brackets):