Skip to content

Add synthetic MoE routing benchmark harness under benchmarks/moe_synthetic_routing - #6298

Open
Shuwen-Fang wants to merge 1 commit into
AI-Hypercomputer:mainfrom
Shuwen-Fang:moe-synthetic-routing
Open

Shuwen-Fang wants to merge 1 commit into
AI-Hypercomputer:mainfrom
Shuwen-Fang:moe-synthetic-routing

Conversation

@Shuwen-Fang

@Shuwen-Fang Shuwen-Fang commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Adds a standalone synthetic-routing benchmark wrapper under benchmarks/moe_synthetic_routing/ (synthetic_routing.py, train.py, README.md) to drive controlled MoE load-imbalance experiments via forced_routed_experts without modifying any core src/maxtext/ modules or config schemas.

Key Components

  • benchmarks/moe_synthetic_routing/synthetic_routing.py:
    • lognormal_routing: deterministic per-(seed, step, layer) top-k expert assignment generator supporting per-layer sigma, per-document affinity jitter (row_tau, row_len), and optional empirical .npz log-popularity profiles (local, gs://, or /cns/, 1D [E] or 2D [P, E]).
    • balanced_routing: exact uniform control where every expert receives the exact mean token count (required_rbf = 1.0).
    • SyntheticRouting + parse_synthetic_routing_args: host batch pool and CLI flag extractor (--synthetic_routing=* or synthetic_routing*=*), including --synthetic_routing_overhead_control / --synthetic_routing_overhead_control=true for replica weight-move overhead ablation (requires #6300/#6302; on standalone main, raises a descriptive ImportError).
  • benchmarks/moe_synthetic_routing/train.py:
    • Context-managed runtime hook (install_synthetic_routing) wrapping maxtext_utils.get_shaped_batch and DataLoader.load_next_batch (using a WeakKeyDictionary per loader instance) inside try/finally and restoring originals on exit or setup error, plus optional moe_replica_planner.compute_plan_iter wrapper via dataclasses.replace(plan, route_table=...) when --synthetic_routing_overhead_control is set (requires #6300/#6302).
  • benchmarks/moe_synthetic_routing/README.md:
    • Documents FORCED_ROUTING_SUPPORTED_DECODER_BLOCKS (qwen3_5, mixtral, unscanned gemma4; pure-NNX decoder without pipeline parallelism and with mtp_num_layers=0), matches every SyntheticRoutingCfg default (verified by test_readme_defaults_match_synthetic_routing_cfg), and provides a runnable 8-chip TPU v6e command.

Tests

1. Automated CPU Unit Tests (tests/unit/moe_synthetic_routing_test.py — 21 passed, 1 skipped on standalone main; 22 passed with #6300/#6302, 5 subtests passed)

  • SyntheticRoutingGeneratorTest: test_balanced_is_exactly_balanced, test_lognormal_depends_on_seed_and_step, test_lognormal_is_deterministic, test_lognormal_shape_dtype_and_range, test_lognormal_threads_match_serial, test_lognormal_topk_distinct_per_token, test_profile_file_1d_vector_supported, test_profile_file_depth_mapping
  • SyntheticRoutingPoolTest: test_pool_cycles, test_unknown_mode_raises
  • SyntheticRoutingWrapperTest: test_bare_boolean_switch_flag_parsing, test_default_args_off, test_flag_and_kv_parsing_and_pyconfig_compat, test_forced_routing_reaches_routed_moe_and_changes_group_sizes, test_invalid_cli_options_rejected, test_main_delegates_with_stripped_argv, test_overhead_control_hook_with_real_replica_planner, test_overhead_control_hook_zeroes_replica_routing_and_restores, test_overhead_control_missing_planner_raises_clear_import_error, test_partial_hook_install_failure_restores_all_originals, test_readme_defaults_match_synthetic_routing_cfg, test_runtime_hooks_patch_and_restore

2. TPU v6e-8 Verification

  1. Verbatim README.md 8-chip ra2a command (EP=8, 4-layer scaled qwen3.5-397b-a17b, per_device_batch_size=2, max_target_length=1024, use_ring_of_experts=false):
    python3 -m benchmarks.moe_synthetic_routing.train src/maxtext/configs/base.yml \
      run_name=moe_routing_bench base_output_directory=/tmp/maxtext_bench \
      dataset_type=synthetic steps=10 log_period=1 enable_checkpointing=false \
      model_name=qwen3.5-397b-a17b override_model_config=true use_multimodal=false \
      base_emb_dim=1024 base_num_query_heads=4 base_num_kv_heads=2 head_dim=256 \
      gdn_num_key_heads=4 gdn_num_value_heads=8 \
      base_moe_mlp_dim=256 base_mlp_dim=256 vocab_size=32768 \
      base_num_decoder_layers=4 \
      per_device_batch_size=2 max_target_length=1024 \
      ici_expert_parallelism=8 ici_fsdp_parallelism=1 use_ring_of_experts=false \
      sparse_matmul=true megablox=true ragged_buffer_factor=1.5 \
      --synthetic_routing=lognormal \
      --synthetic_routing_sigmas=0.9 \
      --synthetic_routing_row_taus=1.3 \
      --synthetic_routing_row_len=512
    Ran 10 steps cleanly on TPU v6e-8 (0.067s/step steady state; loss: 10.900 -> 10.554), and --synthetic_routing=balanced ran cleanly (0.063s/step).
  2. Standalone REQUIRED_RBF verification on #6298 (EP=8, 2-layer qwen3.5-397b-a17b, base_emb_dim=2048, use_ring_of_experts=true use_ragged_sort=true log_required_ragged_buffer_factor=true):
    • --synthetic_routing=balanced: REQUIRED_RBF step=0..2 max=1.0000 per_layer=[1.0000,1.0000]
    • --synthetic_routing=lognormal --synthetic_routing_sigmas=0.9 --synthetic_routing_row_taus=1.3 --synthetic_routing_row_len=512 (L=64 experts/shard): REQUIRED_RBF step=0 max=1.3367 per_layer=[1.1254,1.3367]
    • --synthetic_routing=lognormal --synthetic_routing_sigma=2.0 --synthetic_routing_row_taus=1.3 --synthetic_routing_row_len=512 (L=64 experts/shard): REQUIRED_RBF step=0 max=2.0321 per_layer=[1.3970,2.0321]
  3. Overhead-control smoke test against PR stack (#6302 @ fa4b588e6, 4 steps, EP=8, num_experts=128, --synthetic_routing=lognormal --synthetic_routing_sigma=1.5):
    • expert_replicas_per_device=4 with --synthetic_routing_overhead_control ran 4 steps cleanly (0.000e+00 diff on loss, raw_grad_norm, param_norm vs R=0) and produced identical REQUIRED_RBF to R=0 across all 4 steps (confirming replicas move while zero tokens are rerouted to replica slots).

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests locally and provided test artifacts above, if applicable.
  • I have made or will make corresponding changes to the doc if needed.

@github-actions github-actions Bot added the skip-ci-tests Add this label to the PR to skip running CI tests label Nov 12, 2026
@Shuwen-Fang Shuwen-Fang changed the title [MoE replication 1/5] Add synthetic MoE routing for load-balancing perf experiments Add synthetic MoE routing benchmark harness under benchmarks/moe_synthetic_routing Nov 13, 2026
@Shuwen-Fang
Shuwen-Fang force-pushed the moe-synthetic-routing branch from 22555a7 to 8d98378 Compare November 13, 2026 00:37
@codecov

codecov Bot commented Nov 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@Shuwen-Fang

Copy link
Copy Markdown
Collaborator Author

@gemini-cli /review

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: Spec met. High-quality, self-contained benchmark implementation with a minor memory leak logic flaw.

Top issues:

  1. [Bug] The pools_by_loader dictionary holds a strong reference to DataLoader instances in its values (self, sr), creating a memory leak. → Use a weakref.WeakKeyDictionary to safely associate the synthetic routing state with the DataLoader without preventing its garbage collection (see inline comment).
Inferred spec & validation
  • Sources: PR description, code after.
  • Problem: Need a reproducible, steady-state MoE routing distribution (lognormal or balanced) that injects forced_routed_experts batches without modifying MaxText core config/schemas.
Spec item Implementation Validating test
R1 (stated): Generate forced_routed_experts [batch, seq, num_layers, top_k] in lognormal and balanced modes. ✅ test_lognormal_shape_dtype_and_range ✅
R2 (stated): Transparent CLI parser that intercepts synthetic_routing args. ✅ test_flag_and_kv_parsing_and_pyconfig_compat ✅
R3 (stated): Install runtime hooks on DataLoader.load_next_batch and maxtext_utils.get_shaped_batch. ✅ test_runtime_hooks_patch_and_restore ✅
R4 (stated): Zero changes to src/maxtext/. ✅ (No tests required)
C1 (inferred): Deterministic and exactly identical across runs when seeded. ✅ preserved test_lognormal_is_deterministic ✅

Comment on lines +59 to +69

@functools.wraps(orig_load_next_batch)
def wrapped_load_next_batch(self, *args, **kwargs):
example_batch = orig_load_next_batch(self, *args, **kwargs)
entry = pools_by_loader.get(id(self))
if entry is None or entry[0] is not self:
sr = synthetic_routing.SyntheticRouting(self.config, self.input_data_shardings, syn_cfg=syn_cfg)
pools_by_loader[id(self)] = (self, sr)
else:
sr = entry[1]
example_batch = dict(example_batch)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 The pools_by_loader dictionary maps id(self) to (self, sr). Since the value contains a strong reference to self (the DataLoader instance), the DataLoader will never be garbage collected. Since instances of DataLoader don't override __slots__ and are hashable by identity, you can safely use a weakref.WeakKeyDictionary to associate state with the instance without preventing its collection or tracking IDs manually.

Suggested change
@functools.wraps(orig_load_next_batch)
def wrapped_load_next_batch(self, *args, **kwargs):
example_batch = orig_load_next_batch(self, *args, **kwargs)
entry = pools_by_loader.get(id(self))
if entry is None or entry[0] is not self:
sr = synthetic_routing.SyntheticRouting(self.config, self.input_data_shardings, syn_cfg=syn_cfg)
pools_by_loader[id(self)] = (self, sr)
else:
sr = entry[1]
example_batch = dict(example_batch)
import weakref
pools_by_loader = weakref.WeakKeyDictionary()
@functools.wraps(orig_load_next_batch)
def wrapped_load_next_batch(self, *args, **kwargs):
example_batch = orig_load_next_batch(self, *args, **kwargs)
sr = pools_by_loader.get(self)
if sr is None:
sr = synthetic_routing.SyntheticRouting(self.config, self.input_data_shardings, syn_cfg=syn_cfg)
pools_by_loader[self] = sr

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 858075b: replaced dict[int, tuple[object, SyntheticRouting]] with weakref.WeakKeyDictionary and added a GC weakref check in test_runtime_hooks_patch_and_restore.

@Shuwen-Fang

Copy link
Copy Markdown
Collaborator Author

@gemini-cli /review

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@Shuwen-Fang

Copy link
Copy Markdown
Collaborator Author

@gemini-cli /review

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@Shuwen-Fang

Copy link
Copy Markdown
Collaborator Author

@gemini-cli /review

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@Shuwen-Fang

Copy link
Copy Markdown
Collaborator Author

@gemini-cli /review

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 I'm sorry @Shuwen-Fang, but I was unable to process your request. Please see the logs for more details.

@Shuwen-Fang

Copy link
Copy Markdown
Collaborator Author

@gemini-cli /review

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: No significant issues found.

Inferred spec & validation
  • Sources: PR description, code before/after.
  • Problem: Benchmarking MoE load-imbalance requires a controlled, reproducible expert distribution without intrusive modifications to core MaxText config/modules.
Spec item Implementation Validating test
R1 (stated): Add a standalone synthetic-routing benchmark wrapper. ✅ test_readme_defaults_match_synthetic_routing_cfg ✅
R2 (stated): Support generating lognormal and balanced forced_routed_experts. ✅ test_balanced_is_exactly_balanced ✅
R3 (stated): Hook into get_shaped_batch and load_next_batch without modifying core schemas. ✅ test_runtime_hooks_patch_and_restore ✅
R4 (stated): Include overhead control ablation wrapper. ✅ test_overhead_control_hook_zeroes_replica_routing_and_restores ✅
C1 (inferred): Must not leak hooks into main or after execution. ✅ preserved test_runtime_hooks_patch_and_restore ✅

@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request adds a flexible, standalone synthetic routing benchmark harness for Mixture-of-Experts (MoE) models. By injecting deterministic or balanced expert routing assignments directly into the data pipeline, it allows for precise load-balancing and overhead-control experiments without requiring changes to the core MaxText architecture or configuration schemas.

Highlights

  • Synthetic MoE Routing Harness: Introduced a standalone benchmark wrapper under benchmarks/moe_synthetic_routing/ to enable controlled MoE load-imbalance experiments via forced_routed_experts without modifying core MaxText modules.
  • Routing Modes: Implemented lognormal routing for realistic expert popularity distributions and balanced routing for uniform expert assignment, both supporting per-layer configurations and empirical profile mapping.
  • Runtime Injection: Added a context-managed runtime hook in train.py that intercepts DataLoader and get_shaped_batch to inject synthetic routing data, with optional overhead-control ablation support.
  • Testing: Included comprehensive CPU unit tests covering generator logic, pool cycling, CLI flag parsing, and runtime hook lifecycle management.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @Shuwen-Fang, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: Clean, well-tested MoE routing benchmark harness; address boolean CLI flag parsing and minor edge cases before merge.

Top issues:

  1. [Bug] parse_synthetic_routing_args consumes the subsequent CLI argument when boolean options like --synthetic_routing_overhead_control are passed without =true, crashing with ValueError: Expected boolean for synthetic routing option on valid CLI invocations. → Support standalone boolean switch flags by setting True without consuming argv[idx + 1] (see inline comment on synthetic_routing.py).
  2. [Bug] depth_map_profiles assumes empirical .npz profiles are strictly 2D [P, E], raising AxisError along axis=1 if provided with a 1D vector [E]. → Wrap profiles with np.atleast_2d to support both 1D and 2D profiles cleanly (see inline comment on synthetic_routing.py).
  3. [Complexity] Hook installation in install_synthetic_routing occurs outside the try: block, risking leaked monkey patches if initialization fails. → Move hook setup inside the try: block to ensure finally: restoration (see inline comment on train.py).
Inferred spec & validation
  • Sources: PR #6298 description; forced_routed_experts integration in maxtext.layers.nnx_decoders and maxtext.layers.moe; FORCED_ROUTING_SUPPORTED_DECODER_BLOCKS in maxtext.configs.types.
  • Problem: Benchmarking MoE load balancing and ragged buffer factors on real weights/data couples routing behavior to model state and dataset, preventing reproducible ablation of routing imbalance without modifying core MaxText code or schemas.
Spec item Implementation Validating test
R1 (stated): Deterministic top-k expert assignment generator producing [batch, seq, num_layers, top_k] int32 batches ✅ met moe_synthetic_routing_test.py::SyntheticRoutingGeneratorTest::test_lognormal_* ✅
R2 (stated): Uniform balanced routing baseline where every expert receives the exact mean token count ✅ met moe_synthetic_routing_test.py::SyntheticRoutingGeneratorTest::test_balanced_is_exactly_balanced ✅
R3 (stated): Lognormal routing with per-layer sigma/sigmas, document affinity jitter (row_tau, row_len), and optional .npz profile ✅ met moe_synthetic_routing_test.py::SyntheticRoutingGeneratorTest::test_profile_file_depth_mapping ✅
R4 (stated): Benchmark entrypoint wrapping get_shaped_batch and DataLoader.load_next_batch without core schema changes ✅ met moe_synthetic_routing_test.py::SyntheticRoutingWrapperTest::test_runtime_hooks_patch_and_restore, test_forced_routing_reaches_routed_moe_and_changes_group_sizes ✅
R5 (stated): Hot-expert replication overhead-control ablation hook (--synthetic_routing_overhead_control=true) ✅ met moe_synthetic_routing_test.py::SyntheticRoutingWrapperTest::test_overhead_control_hook_zeroes_replica_routing_and_restores, test_overhead_control_missing_planner_raises_clear_import_error ✅
C1 (stated): Zero modification to core src/maxtext/ modules or config schemas ✅ preserved moe_synthetic_routing_test.py::SyntheticRoutingWrapperTest::test_flag_and_kv_parsing_and_pyconfig_compat ✅
C2 (inferred): Clean restoration of monkey-patched methods on context exit ✅ preserved moe_synthetic_routing_test.py::SyntheticRoutingWrapperTest::test_runtime_hooks_patch_and_restore ✅
C3 (inferred): No memory leaks or per-step host syncs in hot dispatch loops ✅ preserved moe_synthetic_routing_test.py::SyntheticRoutingWrapperTest::test_runtime_hooks_patch_and_restore (weakref GC check) ✅

Comment on lines +117 to +130
if "=" in stripped:
raw_val = stripped.split("=", 1)[1]
elif arg.startswith("--") and idx + 1 < len(argv):
idx += 1
raw_val = argv[idx]
else:
raise ValueError(f"Missing value for synthetic routing option: {arg!r}")
caster = _CFG_FIELD_TYPES[key]
if key == "synthetic_routing" and raw_val.strip().lower() in ("none", "null", ""):
parsed[key] = "none"
elif caster is bool:
parsed[key] = _parse_bool(raw_val)
else:
parsed[key] = caster(raw_val)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 When --synthetic_routing_overhead_control is passed as a boolean flag without =true (standard CLI boolean switch syntax), the parser unconditionally consumes the next argument as its value via argv[idx + 1]. If followed by another CLI argument (such as a config path), it attempts to parse that argument as a bool and crashes with ValueError: Expected boolean for synthetic routing option; if passed at the end of argv, it fails with Missing value.

Suggested change
if "=" in stripped:
raw_val = stripped.split("=", 1)[1]
elif arg.startswith("--") and idx + 1 < len(argv):
idx += 1
raw_val = argv[idx]
else:
raise ValueError(f"Missing value for synthetic routing option: {arg!r}")
caster = _CFG_FIELD_TYPES[key]
if key == "synthetic_routing" and raw_val.strip().lower() in ("none", "null", ""):
parsed[key] = "none"
elif caster is bool:
parsed[key] = _parse_bool(raw_val)
else:
parsed[key] = caster(raw_val)
caster = _CFG_FIELD_TYPES[key]
if caster is bool and "=" not in stripped:
if idx + 1 < len(argv) and argv[idx + 1].strip().lower() in ("true", "false", "1", "0", "yes", "no"):
idx += 1
parsed[key] = _parse_bool(argv[idx])
else:
parsed[key] = True
else:
if "=" in stripped:
raw_val = stripped.split("=", 1)[1]
elif arg.startswith("--") and idx + 1 < len(argv):
idx += 1
raw_val = argv[idx]
else:
raise ValueError(f"Missing value for synthetic routing option: {arg!r}")
if key == "synthetic_routing" and raw_val.strip().lower() in ("none", "null", ""):
parsed[key] = "none"
elif caster is bool:
parsed[key] = _parse_bool(raw_val)
else:
parsed[key] = caster(raw_val)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in fae56dfc9: parse_synthetic_routing_args now handles standalone boolean switch flags (e.g. --synthetic_routing_overhead_control without =true) without consuming the next non-boolean CLI argument, covered by test_bare_boolean_switch_flag_parsing.

"""Maps `[P, E_in]` log-profiles onto `[num_layers, num_experts]` sorted descending."""
p = -np.sort(-np.asarray(profiles, dtype=np.float64), axis=1)
if p.shape[1] != num_experts:
q_in = (np.arange(p.shape[1]) + 0.5) / p.shape[1]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 depth_map_profiles assumes profiles is strictly 2D [P, E], raising AxisError along axis=1 if an empirical .npz profile provides a 1D vector [E]. Wrapping profiles with np.atleast_2d allows 1D profiles to be promoted to [1, E] and broadcast across layers cleanly via the existing depth mapping.

Suggested change
q_in = (np.arange(p.shape[1]) + 0.5) / p.shape[1]
p = -np.sort(-np.atleast_2d(np.asarray(profiles, dtype=np.float64)), axis=1)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in fae56dfc9: wrapped profiles with np.atleast_2d in depth_map_profiles and added test_profile_file_1d_vector_supported.

Comment on lines +52 to +105
try:
from maxtext.layers import moe_replica_planner as mrp_module # pylint: disable=import-outside-toplevel
except ImportError as exc:
raise ImportError(
"--synthetic_routing_overhead_control=true requires maxtext.layers.moe_replica_planner "
"(PRs #6300/#6302); on standalone main, moe_replica_planner is not present."
) from exc

orig_compute_plan_iter = mrp_module.compute_plan_iter

@functools.wraps(orig_compute_plan_iter)
def overhead_control_plan_iter(counts_src, num_ep, num_replicas, *args, **kwargs):
plan = orig_compute_plan_iter(counts_src, num_ep, num_replicas, *args, **kwargs)
plan.route_table = mrp_module.identity_route_table(num_ep, counts_src.shape[1], num_replicas)
return plan

mrp_module.compute_plan_iter = overhead_control_plan_iter

if syn_cfg.enabled:

@functools.wraps(orig_get_shaped_batch)
def wrapped_get_shaped_batch(config, batch_sharding=None, is_eval=False):
shaped_batch = orig_get_shaped_batch(config, batch_sharding=batch_sharding, is_eval=is_eval)
if not is_eval:
routing_shape = shaped_batch["inputs"].shape + (
int(config.num_decoder_layers),
int(config.num_experts_per_tok),
)
shaped_batch = dict(shaped_batch)
shaped_batch["forced_routed_experts"] = jax.ShapeDtypeStruct(routing_shape, jnp.int32, sharding=batch_sharding)
return shaped_batch

pools_by_loader: weakref.WeakKeyDictionary[object, synthetic_routing.SyntheticRouting] = weakref.WeakKeyDictionary()

@functools.wraps(orig_load_next_batch)
def wrapped_load_next_batch(self, *args, **kwargs):
example_batch = orig_load_next_batch(self, *args, **kwargs)
sr = pools_by_loader.get(self)
if sr is None:
sr = synthetic_routing.SyntheticRouting(self.config, self.input_data_shardings, syn_cfg=syn_cfg)
pools_by_loader[self] = sr
example_batch = dict(example_batch)
example_batch["forced_routed_experts"] = sr.next()
return example_batch

maxtext_utils.get_shaped_batch = wrapped_get_shaped_batch
data_loader.DataLoader.load_next_batch = wrapped_load_next_batch

try:
yield
finally:
maxtext_utils.get_shaped_batch = orig_get_shaped_batch
data_loader.DataLoader.load_next_batch = orig_load_next_batch
if mrp_module is not None and orig_compute_plan_iter is not None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 If an exception is raised while setting up the data loader hooks, mrp_module.compute_plan_iter will remain monkey-patched because hook installation occurs before the try: block. Enclosing all patching operations inside the try: block guarantees that previously applied hooks are always restored on error.

Suggested change
try:
from maxtext.layers import moe_replica_planner as mrp_module # pylint: disable=import-outside-toplevel
except ImportError as exc:
raise ImportError(
"--synthetic_routing_overhead_control=true requires maxtext.layers.moe_replica_planner "
"(PRs #6300/#6302); on standalone main, moe_replica_planner is not present."
) from exc
orig_compute_plan_iter = mrp_module.compute_plan_iter
@functools.wraps(orig_compute_plan_iter)
def overhead_control_plan_iter(counts_src, num_ep, num_replicas, *args, **kwargs):
plan = orig_compute_plan_iter(counts_src, num_ep, num_replicas, *args, **kwargs)
plan.route_table = mrp_module.identity_route_table(num_ep, counts_src.shape[1], num_replicas)
return plan
mrp_module.compute_plan_iter = overhead_control_plan_iter
if syn_cfg.enabled:
@functools.wraps(orig_get_shaped_batch)
def wrapped_get_shaped_batch(config, batch_sharding=None, is_eval=False):
shaped_batch = orig_get_shaped_batch(config, batch_sharding=batch_sharding, is_eval=is_eval)
if not is_eval:
routing_shape = shaped_batch["inputs"].shape + (
int(config.num_decoder_layers),
int(config.num_experts_per_tok),
)
shaped_batch = dict(shaped_batch)
shaped_batch["forced_routed_experts"] = jax.ShapeDtypeStruct(routing_shape, jnp.int32, sharding=batch_sharding)
return shaped_batch
pools_by_loader: weakref.WeakKeyDictionary[object, synthetic_routing.SyntheticRouting] = weakref.WeakKeyDictionary()
@functools.wraps(orig_load_next_batch)
def wrapped_load_next_batch(self, *args, **kwargs):
example_batch = orig_load_next_batch(self, *args, **kwargs)
sr = pools_by_loader.get(self)
if sr is None:
sr = synthetic_routing.SyntheticRouting(self.config, self.input_data_shardings, syn_cfg=syn_cfg)
pools_by_loader[self] = sr
example_batch = dict(example_batch)
example_batch["forced_routed_experts"] = sr.next()
return example_batch
maxtext_utils.get_shaped_batch = wrapped_get_shaped_batch
data_loader.DataLoader.load_next_batch = wrapped_load_next_batch
try:
yield
finally:
maxtext_utils.get_shaped_batch = orig_get_shaped_batch
data_loader.DataLoader.load_next_batch = orig_load_next_batch
if mrp_module is not None and orig_compute_plan_iter is not None:
try:
if syn_cfg.synthetic_routing_overhead_control:
try:
from maxtext.layers import moe_replica_planner as mrp_module # pylint: disable=import-outside-toplevel
except ImportError as exc:
raise ImportError(
"--synthetic_routing_overhead_control=true requires maxtext.layers.moe_replica_planner "
"(PRs #6300/#6302); on standalone main, moe_replica_planner is not present."
) from exc
orig_compute_plan_iter = mrp_module.compute_plan_iter
@functools.wraps(orig_compute_plan_iter)
def overhead_control_plan_iter(counts_src, num_ep, num_replicas, *args, **kwargs):
plan = orig_compute_plan_iter(counts_src, num_ep, num_replicas, *args, **kwargs)
plan.route_table = mrp_module.identity_route_table(num_ep, counts_src.shape[1], num_replicas)
return plan
mrp_module.compute_plan_iter = overhead_control_plan_iter
if syn_cfg.enabled:
@functools.wraps(orig_get_shaped_batch)
def wrapped_get_shaped_batch(config, batch_sharding=None, is_eval=False):
shaped_batch = orig_get_shaped_batch(config, batch_sharding=batch_sharding, is_eval=is_eval)
if not is_eval:
routing_shape = shaped_batch["inputs"].shape + (
int(config.num_decoder_layers),
int(config.num_experts_per_tok),
)
shaped_batch = dict(shaped_batch)
shaped_batch["forced_routed_experts"] = jax.ShapeDtypeStruct(routing_shape, jnp.int32, sharding=batch_sharding)
return shaped_batch
pools_by_loader: weakref.WeakKeyDictionary[object, synthetic_routing.SyntheticRouting] = weakref.WeakKeyDictionary()
@functools.wraps(orig_load_next_batch)
def wrapped_load_next_batch(self, *args, **kwargs):
example_batch = orig_load_next_batch(self, *args, **kwargs)
sr = pools_by_loader.get(self)
if sr is None:
sr = synthetic_routing.SyntheticRouting(self.config, self.input_data_shardings, syn_cfg=syn_cfg)
pools_by_loader[self] = sr
example_batch = dict(example_batch)
example_batch["forced_routed_experts"] = sr.next()
return example_batch
maxtext_utils.get_shaped_batch = wrapped_get_shaped_batch
data_loader.DataLoader.load_next_batch = wrapped_load_next_batch
yield
finally:

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in fae56dfc9: moved all hook installation inside the try: block (and switched overhead_control_plan_iter to dataclasses.replace(plan, ...) for frozen @flax.struct.dataclass ReplicaPlan compatibility), covered by test_partial_hook_install_failure_restores_all_originals and test_overhead_control_hook_with_real_replica_planner.

@Shuwen-Fang

Copy link
Copy Markdown
Collaborator Author

@gemini-cli /review

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@Shuwen-Fang

Copy link
Copy Markdown
Collaborator Author

@gemini-cli /review

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: Spec largely met; very high quality, but contains a critical JAX PyTree mismatch bug on evaluation runs.

Top issues:

  1. 🔴 [Bug] wrapped_get_shaped_batch skips adding forced_routed_experts to the signature during evaluation (is_eval=True), but wrapped_load_next_batch unconditionally injects the key into all data batches. This causes a JAX PyTree mismatch when evaluation runs, crashing the step, and SyntheticRouting assumes the training batch size, which would mismatch the eval batch size. → Unconditionally add the key in wrapped_get_shaped_batch, and pass the actual batch_size from the data loader batch to SyntheticRouting.
  2. 🟠 [Alignment] Calling np.load() on .npz files directly via index extraction leaves the internal ZipFile unclosed, which can raise a ResourceWarning during garbage collection. → Use a with np.load(...) as data: context manager.
Inferred spec & validation
  • Sources: PR description, code before/after.
  • Problem: Need a way to run controlled MoE load-imbalance experiments via synthetic routing assignments (lognormal/balanced) without modifying core MaxText code, plus an overhead-control flag.
Spec item Implementation Validating test
R1 (stated): Generate synthetic routing matrices and inject them as forced_routed_experts via hooks. ✅ test_forced_routing_reaches_routed_moe_and_changes_group_sizes ✅
R2 (stated): Do this without altering core src/maxtext/ code. ✅ test_flag_and_kv_parsing_and_pyconfig_compat ✅
R3 (stated): Provide --synthetic_routing_overhead_control to override replica routing to identity. ✅ test_overhead_control_hook_with_real_replica_planner ✅
C1 (inferred): Synthetic routing injection must not crash evaluation steps if they run. ❌ violated test_runtime_hooks_patch_and_restore ⚠️ superficial (checks is_eval signature logic in isolation but does not simulate a full evaluation PyTree match).

Comment on lines +104 to +107
argv: Sequence[str],
) -> tuple[SyntheticRoutingCfg, list[str]]:
"""Extracts `synthetic_routing*` options (both `--k=v` / `--k v` and `k=v`) from `argv`."""
if not argv:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Calling np.load() on a file object without a context manager leaves the returned NpzFile (and its internal ZipFile) unclosed. Since ZipFile creates additional file-like objects, failing to close it properly can raise ResourceWarnings on garbage collection.

Suggested change
argv: Sequence[str],
) -> tuple[SyntheticRoutingCfg, list[str]]:
"""Extracts `synthetic_routing*` options (both `--k=v` / `--k v` and `k=v`) from `argv`."""
if not argv:
if "://" in prof_path or prof_path.startswith("/cns/"):
from etils import epath # pylint: disable=import-outside-toplevel
with epath.Path(prof_path).open("rb") as f:
with np.load(f) as data:
profile = data["log_profile"]

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The log_profile array is eagerly read and materialized inside the with open(...) / with epath.Path(...).open(...) context before the underlying file handle closes, so no code change is needed here.

Comment on lines +92 to +97


def _parse_bool(raw_val: str) -> bool:
lowered = raw_val.strip().lower()
if lowered in ("true", "1", "yes"):
return True

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Accept the actual batch_size to support evaluation runs (which use config.eval_per_device_batch_size), preventing shape mismatches in train.py.

Suggested change
def _parse_bool(raw_val: str) -> bool:
lowered = raw_val.strip().lower()
if lowered in ("true", "1", "yes"):
return True
def __init__(self, config, sharding, syn_cfg: SyntheticRoutingCfg | None = None, batch_size: int | None = None):
import jax # pylint: disable=import-outside-toplevel
opts = syn_cfg if syn_cfg is not None else config
self.mode = opts.synthetic_routing
batch = int(batch_size if batch_size is not None else config.global_batch_size_to_load)

Comment on lines +96 to +98
sr = synthetic_routing.SyntheticRouting(self.config, self.input_data_shardings, syn_cfg=syn_cfg)
pools_by_loader[self] = sr
example_batch = dict(example_batch)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Evaluation batch sizes (eval_per_device_batch_size) often differ from training batch sizes. Extract the actual batch size from the data loader's batch here to ensure forced_routed_experts matches the inputs batch dimension for eval runs.

Suggested change
sr = synthetic_routing.SyntheticRouting(self.config, self.input_data_shardings, syn_cfg=syn_cfg)
pools_by_loader[self] = sr
example_batch = dict(example_batch)
if sr is None:
batch_size = example_batch["inputs"].shape[0]
sr = synthetic_routing.SyntheticRouting(self.config, self.input_data_shardings, syn_cfg=syn_cfg, batch_size=batch_size)
pools_by_loader[self] = sr

Comment on lines +108 to +111
return SyntheticRoutingCfg(), []
remaining: list[str] = [argv[0]]
parsed: dict[str, Any] = {}
idx = 1

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Use a context manager to ensure the NpzFile is closed immediately.

Suggested change
return SyntheticRoutingCfg(), []
remaining: list[str] = [argv[0]]
parsed: dict[str, Any] = {}
idx = 1
else:
with open(prof_path, "rb") as f:
with np.load(f) as data:
profile = data["log_profile"]

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-ci-tests Add this label to the PR to skip running CI tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant