Skip to content

Extend log_required_ragged_buffer_factor to the ragged all-to-all path (no new flag) - #6299

Open
Shuwen-Fang wants to merge 1 commit into
AI-Hypercomputer:mainfrom
Shuwen-Fang:moe-ra2a-required-rbf-log
Open

Shuwen-Fang wants to merge 1 commit into
AI-Hypercomputer:mainfrom
Shuwen-Fang:moe-ra2a-required-rbf-log

Conversation

@Shuwen-Fang

@Shuwen-Fang Shuwen-Fang commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Extend log_required_ragged_buffer_factor to the ragged all-to-all path (no new flag).

  • Reuses existing log_required_ragged_buffer_factor flag & structured logger: Populates RouteOutput.required_rbf inside RoutedMoE.ra2a_and_route (use_ring_of_experts=False) via self.required_ragged_buffer_factor(...) (mesh-wide jax.lax.pmax over tuple(self.mesh.axis_names)), emitting the standard REQUIRED_RBF step=... max=... per_layer=[...] log line once per step in train.py with zero device-to-host sync inside the layer.
  • Scoped strictly to active ragged_all_to_all buffers: In ra2a_and_route, only num_ep > 1 with is_batch_sharded_by_expert allocates a ragged_buffer_factor-sized receive buffer (get_ragged_buffer_size). When num_ep == 1 or is_batch_sharded_by_ep(inputs) is False (batch size 1 decode), no ragged_all_to_all receive buffer exists (ragged_buffer_factor cannot truncate tokens), so required_rbf stays None and "moe_required_rbf" is not sown.
  • Config validation (src/maxtext/configs/types.py): Relaxes log_required_ragged_buffer_factor=True to allow use_ring_of_experts=False whenever sparse_matmul=True (while still requiring use_ragged_sort=True when use_ring_of_experts=True).

Validation

1. TPU v6e-8 Probe Parity & 128-Row-Aligned Drop Boundary (85f7c74d2)

(a) Exact parity with prior log_ra2a_required_rbf debug probe (17e928f11)

Verified over 4 steps × 4 layers (EP=8, per_device_batch_size=2, max_target_length=1024, top_k=10 → balanced_size = 20,480 rows/shard, use_ring_of_experts=false, seed=0) using #6298 synthetic routing:

  • 128 experts (L=16/shard), lognormal (sigma=0.9, row_tau=1.3, row_len=768): all 16 (step, layer) values match identically (step=0: [1.3676, 1.5240, 1.2921, 1.2648], step=1: [1.2732, 1.3465, 1.2262, 1.1531], step=2: [1.1669, 1.6920, 1.2893, 1.1545], step=3: [1.1877, 1.4992, 1.3093, 1.2880]).
  • 512 experts (L=64/shard), empirical Qwen3.5 profile (qwen397b_routing_profile.npz, row_tau=1.3, row_len=768): all 16 (step, layer) values match identically (step=0: [1.3205, 1.0804, 1.2630, 1.3212], step=1: [1.2775, 1.1416, 1.2175, 1.3247], step=2: [1.3638, 1.0943, 1.3478, 1.2827], step=3: [1.2427, 1.0994, 1.1789, 1.3839]; at EP=32 on SFS this same profile gives 2.4823).

(b) Exact 128-row-aligned drop-prediction boundary on the ra2a path

On TPU v6e-8 (EP=8, 2-layer qwen3.5-397b-a17b, base_emb_dim=2048, num_experts=128, use_ring_of_experts=false, use_ragged_sort=true, balanced_size = 20,480 rows/shard), the probe reports:

  • step=0: [1.5240, 1.2921], step=1: [1.3465, 1.2262], step=2: [1.6920, 1.2893] (1.6920 × 20,480 = 34,652 rows on layer 0), step=3: [1.4992, 1.3093].
  • Smallest 128-row-aligned capacity >= 34,652: 34,688 rows (ragged_buffer_factor = 1.69376221).
  • Next 128-row-aligned capacity < 34,652: 34,560 rows (ragged_buffer_factor = 1.68751221, 128 rows lower).

| Step | Force-dropless (rbf=-1.0) loss | rbf = 1.69376221 (34,688 rows, >= probe) |Δloss| / |Δgrad_norm| / fallback_layers | rbf = 1.68751221 (34,560 rows, < probe) |Δloss| / |Δgrad_norm| / fallback_layers |
|---|---|---|---|
| 0 | 10.89864540 | 0.000e+00 / 0.000e+00 / 0.0 | 0.000e+00 / 0.000e+00 / 0.0 |
| 1 | 10.77676964 | 0.000e+00 / 0.000e+00 / 0.0 | 0.000e+00 / 0.000e+00 / 0.0 |
| 2 | 10.68296623 | 0.000e+00 / 0.000e+00 / 0.0 | 7.629e-06 / 2.980e-07 / 1.0 (layer 0 overflows) |
| 3 | 10.64385128 | 0.000e+00 / 0.000e+00 / 0.0 | 2.956e-05 / 2.325e-06 / 0.0 |

(c) Flag-off bitwise parity vs main (561de23e8)

8-step TPU v6e-8 training (log_required_ragged_buffer_factor=false): 0.000e+00 diff on loss, raw_grad_norm, and param_norm across all 8 steps.

2. Automated CPU Unit Tests (tests/unit/moe_test.py::RequiredRaggedBufferFactorTest — 5 passed)

  • test_matches_buffer_size_boundary
  • test_return_ra2a_zeros_dropped_token_rows
  • test_ra2a_sows_mesh_max_required_rbf
  • test_ra2a_ep1_and_unsharded_batch_do_not_sow_required_rbf
  • test_validator_rules_for_log_required_ragged_buffer_factor

(Note on CI codecov/patch: fork PRs upload coverage tokenlessly across 52 matrix sessions; on prior run 78a8c583c, Codecov ingested 38/52 sessions even though all 4 cpu-unit shards passed, which can cause spurious partial-upload codecov/patch reports on large files like moe.py.)

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests locally and provided test artifacts above, if applicable.
  • I have made or will make corresponding changes to the doc if needed.

@codecov

codecov Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/maxtext/layers/moe.py 0.00% 4 Missing ⚠️

📢 Thoughts on this report? Let us know!

@Shuwen-Fang Shuwen-Fang changed the title [MoE replication 2/5] Add log_ra2a_required_rbf debug flag Add log_ra2a_required_rbf debug flag for ragged all-to-all MoE Nov 13, 2026
@Shuwen-Fang
Shuwen-Fang marked this pull request as ready for review November 13, 2026 00:06
@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request adds a diagnostic tool for the ragged all-to-all Mixture-of-Experts (MoE) path. By introducing a new configuration flag, users can now identify the optimal buffer size required to avoid token drops, facilitating better benchmarking of routing skew and load-balancing strategies without altering the default execution behavior.

Highlights

  • New Debug Flag: Introduced the log_ra2a_required_rbf configuration flag to assist in monitoring ragged all-to-all MoE performance.
  • Diagnostic Logging: When enabled, the system now logs the per-layer minimum ragged_buffer_factor required to prevent token dropping during expert-parallel routing.
  • Configuration Updates: Updated base configuration and type definitions to include the new flag, defaulting it to false to ensure no impact on existing compiled HLO paths.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @Shuwen-Fang, I'm running an automatic MaxText review of this pull request. You can track my progress in the logs. To request another review later, add the gemini-review label or comment @gemini-cli /review.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: Bypasses existing MoE probe architecture, introduces uncoordinated host print storm in shard_map, and lacks automated tests.

Top issues:

  1. [Alignment] MaxText already implements log_required_ragged_buffer_factor (in configs/types.py, base.yml, and layers/moe.py) with complete plumbing for intermediate sowing and structured per-step logging in train.py. Introducing log_ra2a_required_rbf creates a redundant second configuration flag and duplicates this diagnostic concept. → Delete log_ra2a_required_rbf, populate RouteOutput.required_rbf inside ra2a_and_route under log_required_ragged_buffer_factor, and relax the config validator in types.py (line 4665) to permit ragged all-to-all.
  2. [Perf] Calling jax.debug.print with ordered=False inside shard_map forces every device across the mesh to execute host callbacks on every layer and step, generating a severe I/O storm on large meshes. Furthermore, omitting step number and layer metadata causes outputs from all devices and layers to interleave as scrambled, anonymous floats on stdout. → Remove jax.debug.print and route the computed required factor through RouteOutput.required_rbf to leverage self.sow(nnx.Intermediate, "moe_required_rbf", ...) and _maybe_log_required_rbf in train.py.
  3. [Bug] all_shards_group_sizes is all-gathered across batch_axis only; on topologies with orthogonal mesh axes (such as data or FSDP parallelism), _req is not reduced globally across replicas. Different devices will compute and report conflicting required buffer factors for the same layer. → Apply jax.lax.pmax(required, tuple(self.mesh.axis_names)) when self.mesh is present, consistent with RoutedMoE.required_ragged_buffer_factor.
  4. [Tests] The PR adds no unit or integration tests for the required buffer factor computation or config plumbing. → Add a unit test in tests/unit/moe_test.py that passes an imbalanced expert assignment and verifies the computed required RBF matches the expected traffic peak.
Inferred spec & validation
  • Sources: PR description; code before/after (src/maxtext/configs/base.yml, src/maxtext/configs/types.py, src/maxtext/layers/moe.py).
  • Problem: The ragged all-to-all MoE path lacked probe logging for the minimum ragged_buffer_factor required to prevent token drops under routing skew.
Spec item Implementation Validating test
R1 (stated): Calculate per-layer minimum ragged_buffer_factor to avoid token drops on ragged all-to-all path (num_ep > 1, is_batch_sharded_by_expert=True) ⚠️ partial (lacks mesh pmax across non-EP axes) ❌ none
R2 (stated): Report or log the per-layer required RBF when enabled 🩹 symptom only (emits via uncoordinated jax.debug.print in shard_map) ❌ none
R3 (stated): When disabled (false, default), model execution and compiled HLO remain unchanged ✅ met ⚠️ superficial (manual verification only; no PR test)
C1 (inferred): Architectural alignment with existing MaxText probe infrastructure (log_required_ragged_buffer_factor) ❌ violated (duplicates flag and bypasses logging pipeline) ❌ none
C2 (inferred): Multi-device execution safety in shard_map hot path without host callback storms or interleaved stdout ❌ violated (jax.debug.print executed per core on every layer/step) ❌ none

Comment thread src/maxtext/layers/moe.py Outdated
Comment on lines +3157 to +3159
all_shards_group_sizes = jax.lax.all_gather(reshaped_group_sizes, axis_name=batch_axis)
if self.config.log_ra2a_required_rbf:
_req = jnp.max(jnp.sum(all_shards_group_sizes, axis=0)) / jnp.shape(x)[0]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 This implementation diverges from the existing log_required_ragged_buffer_factor infrastructure (RouteOutput.required_rbf, self.sow('moe_required_rbf', ...), and _maybe_log_required_rbf in train.py). Additionally, executing jax.debug.print with ordered=False inside shard_map triggers uncoordinated host callbacks across all mesh devices on every layer and step, scrambling stdout and degrading performance. Please remove jax.debug.print, reduce across self.mesh.axis_names with jax.lax.pmax, and assign to RouteOutput.required_rbf under self.config.log_required_ragged_buffer_factor.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 78a8c58: removed jax.debug.print and log_ra2a_required_rbf, and populated RouteOutput.required_rbf in ra2a_and_route via self.required_ragged_buffer_factor(...) (with mesh-wide jax.lax.pmax) under self.config.log_required_ragged_buffer_factor, plus added multi-device unit tests in tests/unit/moe_test.py.

Comment thread src/maxtext/configs/types.py Outdated
Comment on lines +1199 to +1201
log_ra2a_required_rbf: bool = Field(
False, description="Debug: print the per-layer ragged_buffer_factor needed to avoid drops (ragged a2a path)."
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Redundant configuration flag. MoEGeneral already defines log_required_ragged_buffer_factor (line 1225) for this diagnostic purpose. Instead of adding log_ra2a_required_rbf, remove this field and update the validator in types.py (line 4665) to permit log_required_ragged_buffer_factor on ragged all-to-all configurations.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 78a8c58: deleted log_ra2a_required_rbf from base.yml and types.py and relaxed the log_required_ragged_buffer_factor validator in types.py to accept ragged all-to-all (use_ring_of_experts=False, sparse_matmul=True).

@Shuwen-Fang Shuwen-Fang changed the title Add log_ra2a_required_rbf debug flag for ragged all-to-all MoE Extend log_required_ragged_buffer_factor probe to ragged all-to-all MoE Nov 13, 2026
@Shuwen-Fang

Copy link
Copy Markdown
Collaborator Author

@gemini-cli /review

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@Shuwen-Fang

Copy link
Copy Markdown
Collaborator Author

@gemini-cli /review

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

Verdict: No significant issues found.

Inferred spec & validation
  • Sources: PR description; code before/after.
  • Problem: log_required_ragged_buffer_factor probe is not supported for the ragged all-to-all MoE routing path (use_ring_of_experts=False).
Spec item Implementation Validating test
R1 (stated): Allow log_required_ragged_buffer_factor=True for ragged all-to-all with sparse_matmul=True. ✅ test_validator_rules_for_log_required_ragged_buffer_factor ✅
R2 (stated): Compute exact pre-truncation peak demand and sow as "moe_required_rbf". ✅ test_ra2a_sows_mesh_max_required_rbf ✅
R3 (stated): Support both num_ep == 1 and num_ep > 1 cases (including FSDP-only sharding). ✅ test_ra2a_ep1_and_unsharded_batch_required_rbf ✅
C1 (stated): Zero overhead when disabled (log_required_ragged_buffer_factor=False). ✅ preserved ❌ none (verified by manual E2E parity run)
C2 (inferred): Maintain existing config restrictions for ring of experts (e.g. use_ragged_sort). ✅ preserved test_validator_rules_for_log_required_ragged_buffer_factor ✅

@Shuwen-Fang Shuwen-Fang changed the title Extend log_required_ragged_buffer_factor probe to ragged all-to-all MoE Extend log_required_ragged_buffer_factor to the ragged all-to-all path (no new flag) Nov 13, 2026
@Shuwen-Fang

Copy link
Copy Markdown
Collaborator Author

@gemini-cli /review

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Hi @Shuwen-Fang, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants