Nemo Mbridge Perf Moe Hardware Configs

NVIDIAOfficialPublished by NVIDIA

Skill
Guidelines
Data and AI
Apache-2.0
Medium risk

From the author

Representative, point-in-time MoE training playbooks by hardware and model family. Use them as candidate seeds, then revalidate the exact runtime, semantics, topology, and steady-state throughput.

— the description by NVIDIA

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured reference for Mixture-of-Experts (MoE) training configurations. It outlines representative playbooks, performance bands, and tuning strategies for different hardware (H100, B200, GB200, GB300) and model families (DSV3, Qwen3). It emphasizes that these configurations serve as candidate seeds for performance optimization rather than absolute defaults, highlighting the importance of revalidation through profile-driven testing, dispatcher selection, and environment tuning.

What it does

Provides candidate hardware-specific MoE training configurations.
Defines performance bands and representative shapes for planning.
Lists critical environment variables for performance tuning.
Outlines recompute strategies based on memory pressure.
Details CPU-side tuning requirements for GB200/GB300 systems.
Identifies common pitfalls in MoE performance benchmarking.

When to use it

When planning MoE training runs on H100, B200, or GB200/300 hardware.
When seeking candidate seeds for model parallelism and dispatcher settings.
When troubleshooting performance regressions in MoE training pipelines.
When establishing a baseline for throughput and memory usage.

Advantages

Reduces trial-and-error by providing validated starting configurations.
Encourages rigorous performance validation through A/B testing.
Highlights the impact of software stack and container quality on throughput.
Provides clear guidance on balancing memory headroom and throughput.

Usage examples

“What is a recommended configuration for training DSV3 on H100 hardware?”: The agent provides the DeepEP dispatcher configuration with specific TP, EP, PP, and VPP settings, noting the importance of communication overlap.
“How should I tune GB200 systems for MoE training?”: The agent suggests using HybridEP, focusing on CPU-side tuning, graph-friendly static shapes, and specific environment variables like NCCL_GRAPH_REGISTER=0.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/nvidia/nemo-mbridge-perf-moe-hardware-configs/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip nemo-mbridge-perf-moe-hardware-configs.zip -d .agents/skills

Documentation by NVIDIA

MoE Hardware Configuration Reference

Stable docs: @docs/training/moe-optimization.md Card: @skills/nemo-mbridge-perf-moe-hardware-configs/card.yaml

Quick Platform Playbook

These rows are search seeds, not hardware defaults or throughput promises.

Platform	Candidates to screen after `alltoall` bring-up	What usually matters most
H100	DeepEP or HybridEP, explicit overlap, supported FP8 modes	communication overlap, dispatcher/runtime compatibility, and PP efficiency
B200	DeepEP or HybridEP, supported FP8 modes, careful PP layout	container quality and tuned communication settings
GB200	HybridEP, then profile-driven graphs and CPU cleanup	host overhead, topology-aware dispatch, memory headroom
GB300	HybridEP and the target container's lower-precision/kernel stack	the same system interactions as GB200, with remeasurement required

First Answer Checklist

For hardware playbook questions, answer from these canonical rows before adding throughput caveats:

Workload	Hardware	Dispatcher	Layout
DSV3	H100	DeepEP	TP=2, EP=64, PP=8, VPP=4
DSV3	GB200/GB300	HybridEP	TP=1, EP=64, PP=4, VPP=4
Qwen3 235B	H100	`alltoall` + overlap in the current canonical recipe	TP=2, EP=32, PP=8, VPP=4
Qwen3 235B	GB200	HybridEP	TP=1 or 2, EP=32-64, PP=4, VPP=unspecified
Qwen3 30B	16×H100	HybridEP	TP=1, EP=16, PP=1, plain EP overlap

For Qwen3 235B on GB200, explicitly say VPP=unspecified; do not invent or extrapolate VPP=12 unless a measured row provides it. Treat TE-scoped CUDA graph scopes (attn, moe_router, moe_preprocess) as profile-driven candidates, CUDA_DEVICE_MAX_CONNECTIONS selection, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, NCCL_GRAPH_REGISTER=0, GB200/GB300 CPU-side tuning, and the warning not to cargo-cult tracker rows.

Rounded Performance Bands

These are intentionally rounded so the document stays durable as the tracker moves. Treat them as planning ranges, not exact promises.

Workload family	Hardware	Typical band	Representative shape
DSV3, large-scale	H100	low-to-mid hundreds TFLOPS/GPU, high-teens MFU	TP2, EP64, PP8, DeepEP
DSV3, large-scale	B200	high-hundreds TFLOPS/GPU, mid-teens MFU	TP1, EP32, PP8, DeepEP
DSV3, large-scale	GB200	around 1K TFLOPS/GPU, low-20s MFU	TP1, EP64, PP4, HybridEP
DSV3, large-scale	GB300	above the GB200 band, often mid-20s MFU	TP1, EP64, PP4, HybridEP
Qwen3 235B	H100	historical low-300s snapshots; remeasure the current recipe	TP2, EP32, PP8; current recipe uses `alltoall` + overlap
Qwen3 235B	GB200	high-hundreds TFLOPS/GPU in tuned runs	TP1 or TP2, EP32-64, PP4, HybridEP
Qwen3 30B	H100	about 300 TFLOPS/GPU on the validated 16-GPU shape	TP1, EP16, PP1, HybridEP + EP overlap
Qwen3-Next 80B	GB200	low-300s TFLOPS/GPU in BF16-class runs	TP1, EP32, PP2, HybridEP

Representative Config Families

DSV3 on H100

Dispatcher: DeepEP
TP=2  EP=64  PP=8  VPP=4
Routing: force balance
Recompute: light-to-moderate selective recompute
Priority: overlap communication and keep PP efficient

DSV3 on B200

Dispatcher: DeepEP
TP=1  EP=32  PP=8  VPP=2 or similar
Precision: MXFP8-class
Recompute: selective recompute around MLA up-projection and MLP-side modules
Priority: container quality, PP layout, and DeepEP SMS tuning

DSV3 on GB200 or GB300

Dispatcher: HybridEP
TP=1  EP=64  PP=4  VPP=4
Precision: MXFP8-class
CUDA Graph: attn + moe_router + moe_preprocess
Priority: HybridEP, CPU optimization, and graph-friendly static shapes

Qwen3 235B on H100

Dispatcher: alltoall in the current canonical recipe; re-screen flex backends on the target stack
TP=2  EP=32  PP=8  VPP=4
Recompute: none in the current canonical recipe
Priority: communication overlap and router-path cleanup

Qwen3 235B on GB200

Dispatcher: HybridEP
TP=1 or 2  EP=32 to 64  PP=4  VPP=unspecified unless measured
CUDA Graph: attn + moe_router + moe_preprocess
Recompute: moe_act, mlp, or norm depending on memory pressure
Priority: balance throughput against memory headroom

Qwen3 30B-A3B on 16 H100

Dispatcher: HybridEP
TP=1  EP=16  PP=1  CP=1
Precision: BF16
Sequence: 4096
Batch: MBS1 GBS1024
Routing: force balance
EP overlap: enabled
Delayed wgrad: disabled
CUDA Graph: moe_router + moe_preprocess
HybridEP: permute fusion, 32 SMs, 64-token combine chunks
Measured: 20.14729s/step, 299.352 model TFLOPS/GPU over iterations 41-50
Rank-0 peak allocated memory: 62.166 GiB

The current number is the final multi-knob canonical recipe result. An earlier matched A/B isolated plain EP overlap: 244.039 to 287.305 TFLOPS/GPU, with communication hidden by GEMM/attention increasing from 0.11% to 36.55%. Do not attribute the later 299.352 result entirely to overlap.

Qwen3-Next 80B on GB200

Dispatcher: HybridEP
TP=1  EP=32  PP=2  VPP around 4
CUDA Graph: attn + moe_router + moe_preprocess
Priority: pipeline layout and grouped GEMM quality

Cross-Cutting Patterns

PP layout

E = embedding
t = transformer
m = MTP
L = loss
| = stage boundary

The biggest platform difference is usually not just the dispatcher. It is the combination of dispatcher, PP shape, and whether VPP keeps each stage balanced.

Recompute strategy

Memory pressure	Starting point
low	none or a very narrow selective set
moderate	`moe_act`, `mlp`, `norm`, or similar selective modules
high	model-specific up-projection plus selective MoE and MLP modules
extreme or long-context	full recompute only if the selective path still does not fit

Environment variables

CUDA_DEVICE_MAX_CONNECTIONS=1
CUDA_DEVICE_MAX_CONNECTIONS=32   # common when EP overlap and CUDA graphs are combined
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
NCCL_GRAPH_REGISTER=0

CPU-side tuning

On GB200 and GB300, CPU affinity and general host-overhead cleanup can move the needle almost as much as a dispatcher swap. Treat them as first-class tuning work, not as afterthoughts.

Pitfalls

Do not cargo-cult a tracker row: the winning config usually depends on routing mode, container, and PP layout as much as on hardware name.
Container quality matters: large regressions can come from the software stack rather than the model recipe.
VPP must be intentional: a bad VPP split can erase the gain from a better dispatcher.
Compare absolute throughput, not only MFU: MFU can mislead when switching between BF16, FP8, and other precision modes.
Force-balance routing is benchmark-only: it can control routing variance, but it changes semantics. Keep routing fixed within an A/B and validate natural routing separately for training acceptance.
Do not treat the dispatcher table as a hard platform rule: HybridEP is the validated winner for the canonical 16×H100 Qwen3 30B shape, while the current 256×H100 Qwen3 235B recipe uses alltoall. Benchmark backend compatibility and throughput in the production container.
Separate screening, causality, and acceptance: short runs reject weak candidates, matched one-variable A/Bs explain a mechanism, and a 50-step final run validates the complete winner.

Last signature refresh: 2026-08-03.

Files

ATTRIBUTION.mdAdded by AutoGineer875 B
BENCHMARK.md4.8 KiB
LICENSELicence file added by AutoGineer11.2 KiB
SKILL.md7.8 KiB
card.yaml6.8 KiB
evals/evals.json2.2 KiB
skill-card.md4.0 KiB
skill.oms.sig4.7 KiB

card.yaml

title: moe_hardware_configs
validated_on: "2026-07-31"
summary: >
  Representative point-in-time MoE configurations by model family and hardware.
  Rows are candidate seeds, not platform defaults or throughput promises, and
  must be revalidated with fixed semantics, topology, runtime, and timing windows.

validation_status:
  dsv3_h100:
    - measured
  dsv3_b200:
    - measured
  dsv3_gb200:
    - measured
  dsv3_gb300:
    - measured
  qwen3_235b_h100:
    - current_recipe_seed
    - historical_measurements_not_attached_to_current_dispatcher
  qwen3_235b_b200:
    - measured
  qwen3_235b_gb200:
    - measured
  qwen3_30b_h100:
    - measured
    - profile_validated
  qwen3_next_80b_b200:
    - measured
  qwen3_next_80b_gb200:
    - measured
  qwen3_next_397b_gb200:
    - measured

best_configs:
  dsv3_685b_h100_1024:
    model: "DSV3 685B w/ MTP"
    hardware: "1024×H100"
    precision: FP8-Block
    dispatcher: DeepEP
    tflops: ~370
    mfu: "~19%"
    parallelism: "TP2 EP64 PP8 VPP4"
    batch: "MBS1 GBS8192"
    routing: force_balance
    recompute: "up_proj, mlp"
    pp_layout: "Et*3|(tt|)*29m|L"
    overlap_1f1b: true

  dsv3_685b_b200_256_bf16:
    model: "DSV3 685B w/ MTP"
    hardware: "256×B200"
    precision: BF16
    dispatcher: DeepEP
    tflops: ~520
    mfu: "~21%"
    parallelism: "TP1 EP32 PP8 EDP2 VPP4"
    batch: "MBS1 GBS4096"
    routing: force_balance
    recompute: "mla_up_proj, mlp, moe_act, layernorm"

  dsv3_685b_b200_256_fp8mx:
    model: "DSV3 685B w/ MTP"
    hardware: "256×B200"
    precision: FP8-MX
    dispatcher: DeepEP
    tflops: ~800
    mfu: "~16%"
    parallelism: "TP1 EP32 PP8 VPP2"
    batch: "MBS1 GBS8192"
    routing: force_balance
    overlap_1f1b: true
    note: "--moe-deepep-num-sms 24"

  dsv3_671b_gb200_256:
    model: "DSV3 671B no MTP"
    hardware: "256×GB200"
    precision: FP8-MX
    dispatcher: HybridEP
    tflops: ~1000
    mfu: "~19%"
    parallelism: "TP1 EP32 PP8 VPP4"
    batch: "MBS1 GBS2048"
    routing: force_balance
    cuda_graph: "attn, moe_router, moe_preprocess"

  dsv3_685b_gb200_256:
    model: "DSV3 685B w/ MTP"
    hardware: "256×GB200"
    precision: FP8-MX
    dispatcher: HybridEP
    tflops: ~1100
    mfu: "~22%"
    parallelism: "TP1 EP64 PP4 VPP4"
    batch: "MBS1 GBS8192"
    routing: force_balance
    overlap_1f1b: true
    cuda_graph: "attn, moe_router, moe_preprocess"

  dsv3_685b_gb300_256:
    model: "DSV3 685B w/ MTP"
    hardware: "256×GB300"
    precision: FP8-MX
    dispatcher: HybridEP
    tflops: ~1200
    mfu: "~25%"
    parallelism: "TP1 EP64 PP4 VPP4"
    batch: "MBS1 GBS8192"
    routing: force_balance
    overlap_1f1b: true
    cuda_graph: "attn, moe_router, moe_preprocess"

  qwen3_235b_h100_256_current_recipe_seed:
    model: "Qwen3 235B-A22B"
    hardware: "256×H100"
    precision: BF16
    dispatcher: alltoall
    parallelism: "TP2 EP32 PP8 VPP4"
    batch: "MBS1 GBS8192"
    routing: force_balance
    moe_a2a_overlap: true
    ep_overlap: true
    delay_wgrad_compute: true
    recompute: none
    measurement_status: "recipe seed; remeasure on the target stack"
    note: "Do not attach historical DeepEP throughput to this current alltoall configuration"

  qwen3_235b_b200_128:
    model: "Qwen3 235B-A22B"
    hardware: "128×B200"
    precision: BF16
    dispatcher: DeepEP
    tflops: ~590
    mfu: "~26%"
    parallelism: "TP2 EP16 PP4 EDP2 VPP12"
    batch: "MBS2 GBS2048"
    routing: force_balance
    overlap_1f1b: true

  qwen3_235b_gb200_128:
    model: "Qwen3 235B-A22B"
    hardware: "128×GB200"
    precision: BF16
    dispatcher: HybridEP
    tflops: ~700
    mfu: "~28%"
    parallelism: "TP1 EP32 PP4 VPP12"
    batch: "MBS1 GBS1024"
    routing: force_balance
    cuda_graph: "attn, moe_router, moe_preprocess"

  qwen3_30b_h100_32:
    model: "Qwen3 30B-A3B"
    hardware: "32×H100"
    precision: BF16
    dispatcher: DeepEP
    tflops: ~210
    mfu: "~21%"
    parallelism: "TP1 EP8 PP1 EDP4"
    batch: "MBS4 GBS256"
    routing: force_balance
    recompute: full

  qwen3_30b_h100_16:
    model: "Qwen3 30B-A3B"
    hardware: "16×H100"
    precision: BF16
    dispatcher: HybridEP
    tflops: 299.352
    step_time_ms: 20147.290
    parallelism: "TP1 EP16 PP1 CP1"
    sequence_length: 4096
    batch: "MBS1 GBS1024"
    routing: force_balance
    ep_overlap: true
    delay_wgrad_compute: false
    cuda_graph: "moe_router, moe_preprocess"
    hybridep_permute_fusion: true
    hybridep_num_sms: 32
    combine_chunk_tokens: 64
    rank0_peak_allocated_gib: 62.166
    earlier_overlap_ab:
      baseline_tflops: 244.039
      overlap_tflops: 287.305
      throughput_gain_percent: 17.729
      profile_comm_hidden_percent: "0.11 off, 36.55 on"

  qwen3_next_80b_gb200_64:
    model: "Qwen3-Next 80B-A3B"
    hardware: "64×GB200"
    precision: BF16
    dispatcher: HybridEP
    tflops: ~330
    parallelism: "TP1 EP32 PP2 VPP4"
    batch: "MBS2 GBS1024"
    routing: force_balance
    cuda_graph: "attn, moe_router, moe_preprocess"
    note: "native CE, cutlass grouped gemm"

  qwen3_next_397b_gb200_128:
    model: "Qwen3.5-397B-A17B"
    hardware: "128×GB200"
    precision: BF16
    dispatcher: HybridEP
    tflops: ~550
    parallelism: "TP1 EP32 PP4 VPP4"
    batch: "MBS1 GBS4096"
    routing: force_balance
    cuda_graph: "attn, moe_router, moe_preprocess"
    note: "native CE, cutlass grouped gemm"

optimization_patterns:
  - name: ep_comm_overlap
    impact: "+17.729% throughput on measured Qwen3 30B 16×H100 HybridEP shape"
  - name: force_balance_routing
    impact: "benchmark-only control; validate natural routing separately"
  - name: 1f1b_overlap
    impact: "candidate; measure on the exact PP/VPP and dispatcher stack"
  - name: cuda_graphs
    impact: "candidate only when host/launch overhead is visible; capture can be neutral or slower"
  - name: cpu_binding
    impact: "profile-driven host-overhead candidate"
  - name: mbs_tuning
    impact: "shape-dependent; sweep without changing global token semantics"

evidence:
  - "examples/model_verification_cards/qwen3-30b-a3b/card.yaml"
  - "src/megatron/bridge/perf_recipes/qwen/h100/qwen3_moe.py"
  - "[MoE Perf] MCore Release Performance Tracker - DeepSeek-V3"
  - "[MoE Perf] MCore Release Performance Tracker - Qwen3"
  - "[MoE Perf] MCore Release Performance Tracker - Qwen3-Next"
  - "[MoE Perf] MCore Release Performance Tracker - DSV3 Long-Context"

known_limitations:
  - Container versions can cause significant performance regressions
  - Dispatcher recommendations are starting points; the validated Qwen3 30B H100 winner used HybridEP rather than DeepEP
  - VPP must divide transformer layers evenly across PP stages
  - FP8 MFU numbers use higher peak FLOPS denominator than BF16
  - Configs are point-in-time measurements; mcore/TE updates may change results
  - Forced-routing benchmark rows are not training-equivalent acceptance evidence
  - Earlier one-variable A/B gains must not be attributed to a later multi-knob winner

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Security

What our scanner noticed· Medium risk

Large base64-encoded blobs

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

NVIDIA

Publisher

NVIDIA (Official)

Source

NVIDIA/skills · skills/nemo-mbridge-perf-moe-hardware-configs · commit fce3cca

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE-APACHE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download nemo-mbridge-perf-moe-hardware-configs.zip