Nemo Mbridge Perf Moe Hardware Configs
NVIDIAOfficialPublished by NVIDIA
- Skill
- Guidelines
- Data and AI
- Apache-2.0
- Medium risk
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
This skill provides a structured reference for Mixture-of-Experts (MoE) training configurations. It outlines representative playbooks, performance bands, and tuning strategies for different hardware (H100, B200, GB200, GB300) and model families (DSV3, Qwen3). It emphasizes that these configurations serve as candidate seeds for performance optimization rather than absolute defaults, highlighting the importance of revalidation through profile-driven testing, dispatcher selection, and environment tuning.
What it does
- Provides candidate hardware-specific MoE training configurations.
- Defines performance bands and representative shapes for planning.
- Lists critical environment variables for performance tuning.
- Outlines recompute strategies based on memory pressure.
- Details CPU-side tuning requirements for GB200/GB300 systems.
- Identifies common pitfalls in MoE performance benchmarking.
When to use it
- When planning MoE training runs on H100, B200, or GB200/300 hardware.
- When seeking candidate seeds for model parallelism and dispatcher settings.
- When troubleshooting performance regressions in MoE training pipelines.
- When establishing a baseline for throughput and memory usage.
Advantages
- Reduces trial-and-error by providing validated starting configurations.
- Encourages rigorous performance validation through A/B testing.
- Highlights the impact of software stack and container quality on throughput.
- Provides clear guidance on balancing memory headroom and throughput.
Usage examples
- “What is a recommended configuration for training DSV3 on H100 hardware?”
- The agent provides the DeepEP dispatcher configuration with specific TP, EP, PP, and VPP settings, noting the importance of communication overlap.
- “How should I tune GB200 systems for MoE training?”
- The agent suggests using HybridEP, focusing on CPU-side tuning, graph-friendly static shapes, and specific environment variables like NCCL_GRAPH_REGISTER=0.
Install
Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/nvidia/nemo-mbridge-perf-moe-hardware-configs/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip nemo-mbridge-perf-moe-hardware-configs.zip -d .agents/skills
Documentation by NVIDIA
MoE Hardware Configuration Reference
Stable docs: @docs/training/moe-optimization.md Card: @skills/nemo-mbridge-perf-moe-hardware-configs/card.yaml
Quick Platform Playbook
These rows are search seeds, not hardware defaults or throughput promises.
| Platform | Candidates to screen after alltoall bring-up | What usually matters most |
|---|---|---|
| H100 | DeepEP or HybridEP, explicit overlap, supported FP8 modes | communication overlap, dispatcher/runtime compatibility, and PP efficiency |
| B200 | DeepEP or HybridEP, supported FP8 modes, careful PP layout | container quality and tuned communication settings |
| GB200 | HybridEP, then profile-driven graphs and CPU cleanup | host overhead, topology-aware dispatch, memory headroom |
| GB300 | HybridEP and the target container's lower-precision/kernel stack | the same system interactions as GB200, with remeasurement required |
First Answer Checklist
For hardware playbook questions, answer from these canonical rows before adding throughput caveats:
| Workload | Hardware | Dispatcher | Layout |
|---|---|---|---|
| DSV3 | H100 | DeepEP | TP=2, EP=64, PP=8, VPP=4 |
| DSV3 | GB200/GB300 | HybridEP | TP=1, EP=64, PP=4, VPP=4 |
| Qwen3 235B | H100 | alltoall + overlap in the current canonical recipe | TP=2, EP=32, PP=8, VPP=4 |
| Qwen3 235B | GB200 | HybridEP | TP=1 or 2, EP=32-64, PP=4, VPP=unspecified |
| Qwen3 30B | 16×H100 | HybridEP | TP=1, EP=16, PP=1, plain EP overlap |
For Qwen3 235B on GB200, explicitly say VPP=unspecified; do not invent or
extrapolate VPP=12 unless a measured row provides it. Treat TE-scoped CUDA
graph scopes (attn, moe_router, moe_preprocess) as profile-driven
candidates,
CUDA_DEVICE_MAX_CONNECTIONS selection,
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, NCCL_GRAPH_REGISTER=0,
GB200/GB300 CPU-side tuning, and the warning not to cargo-cult tracker rows.
Rounded Performance Bands
These are intentionally rounded so the document stays durable as the tracker moves. Treat them as planning ranges, not exact promises.
| Workload family | Hardware | Typical band | Representative shape |
|---|---|---|---|
| DSV3, large-scale | H100 | low-to-mid hundreds TFLOPS/GPU, high-teens MFU | TP2, EP64, PP8, DeepEP |
| DSV3, large-scale | B200 | high-hundreds TFLOPS/GPU, mid-teens MFU | TP1, EP32, PP8, DeepEP |
| DSV3, large-scale | GB200 | around 1K TFLOPS/GPU, low-20s MFU | TP1, EP64, PP4, HybridEP |
| DSV3, large-scale | GB300 | above the GB200 band, often mid-20s MFU | TP1, EP64, PP4, HybridEP |
| Qwen3 235B | H100 | historical low-300s snapshots; remeasure the current recipe | TP2, EP32, PP8; current recipe uses alltoall + overlap |
| Qwen3 235B | GB200 | high-hundreds TFLOPS/GPU in tuned runs | TP1 or TP2, EP32-64, PP4, HybridEP |
| Qwen3 30B | H100 | about 300 TFLOPS/GPU on the validated 16-GPU shape | TP1, EP16, PP1, HybridEP + EP overlap |
| Qwen3-Next 80B | GB200 | low-300s TFLOPS/GPU in BF16-class runs | TP1, EP32, PP2, HybridEP |
Representative Config Families
DSV3 on H100
Dispatcher: DeepEP
TP=2 EP=64 PP=8 VPP=4
Routing: force balance
Recompute: light-to-moderate selective recompute
Priority: overlap communication and keep PP efficientDSV3 on B200
Dispatcher: DeepEP
TP=1 EP=32 PP=8 VPP=2 or similar
Precision: MXFP8-class
Recompute: selective recompute around MLA up-projection and MLP-side modules
Priority: container quality, PP layout, and DeepEP SMS tuningDSV3 on GB200 or GB300
Dispatcher: HybridEP
TP=1 EP=64 PP=4 VPP=4
Precision: MXFP8-class
CUDA Graph: attn + moe_router + moe_preprocess
Priority: HybridEP, CPU optimization, and graph-friendly static shapesQwen3 235B on H100
Dispatcher: alltoall in the current canonical recipe; re-screen flex backends on the target stack
TP=2 EP=32 PP=8 VPP=4
Recompute: none in the current canonical recipe
Priority: communication overlap and router-path cleanupQwen3 235B on GB200
Dispatcher: HybridEP
TP=1 or 2 EP=32 to 64 PP=4 VPP=unspecified unless measured
CUDA Graph: attn + moe_router + moe_preprocess
Recompute: moe_act, mlp, or norm depending on memory pressure
Priority: balance throughput against memory headroomQwen3 30B-A3B on 16 H100
Dispatcher: HybridEP
TP=1 EP=16 PP=1 CP=1
Precision: BF16
Sequence: 4096
Batch: MBS1 GBS1024
Routing: force balance
EP overlap: enabled
Delayed wgrad: disabled
CUDA Graph: moe_router + moe_preprocess
HybridEP: permute fusion, 32 SMs, 64-token combine chunks
Measured: 20.14729s/step, 299.352 model TFLOPS/GPU over iterations 41-50
Rank-0 peak allocated memory: 62.166 GiBThe current number is the final multi-knob canonical recipe result. An earlier matched A/B isolated plain EP overlap: 244.039 to 287.305 TFLOPS/GPU, with communication hidden by GEMM/attention increasing from 0.11% to 36.55%. Do not attribute the later 299.352 result entirely to overlap.
Qwen3-Next 80B on GB200
Dispatcher: HybridEP
TP=1 EP=32 PP=2 VPP around 4
CUDA Graph: attn + moe_router + moe_preprocess
Priority: pipeline layout and grouped GEMM qualityCross-Cutting Patterns
PP layout
E= embeddingt= transformerm= MTPL= loss|= stage boundary
The biggest platform difference is usually not just the dispatcher. It is the combination of dispatcher, PP shape, and whether VPP keeps each stage balanced.
Recompute strategy
| Memory pressure | Starting point |
|---|---|
| low | none or a very narrow selective set |
| moderate | moe_act, mlp, norm, or similar selective modules |
| high | model-specific up-projection plus selective MoE and MLP modules |
| extreme or long-context | full recompute only if the selective path still does not fit |
Environment variables
CUDA_DEVICE_MAX_CONNECTIONS=1
CUDA_DEVICE_MAX_CONNECTIONS=32 # common when EP overlap and CUDA graphs are combined
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
NCCL_GRAPH_REGISTER=0CPU-side tuning
On GB200 and GB300, CPU affinity and general host-overhead cleanup can move the needle almost as much as a dispatcher swap. Treat them as first-class tuning work, not as afterthoughts.
Pitfalls
Do not cargo-cult a tracker row: the winning config usually depends on routing mode, container, and PP layout as much as on hardware name.
Container quality matters: large regressions can come from the software stack rather than the model recipe.
VPP must be intentional: a bad VPP split can erase the gain from a better dispatcher.
Compare absolute throughput, not only MFU: MFU can mislead when switching between BF16, FP8, and other precision modes.
Force-balance routing is benchmark-only: it can control routing variance, but it changes semantics. Keep routing fixed within an A/B and validate natural routing separately for training acceptance.
Do not treat the dispatcher table as a hard platform rule: HybridEP is the validated winner for the canonical 16×H100 Qwen3 30B shape, while the current 256×H100 Qwen3 235B recipe uses
alltoall. Benchmark backend compatibility and throughput in the production container.Separate screening, causality, and acceptance: short runs reject weak candidates, matched one-variable A/Bs explain a mechanism, and a 50-step final run validates the complete winner.
Last signature refresh: 2026-08-03.
Files
- ATTRIBUTION.mdAdded by AutoGineer875 B
- BENCHMARK.md4.8 KiB
- LICENSELicence file added by AutoGineer11.2 KiB
- SKILL.md7.8 KiB
- card.yaml6.8 KiB
- evals/evals.json2.2 KiB
- skill-card.md4.0 KiB
- skill.oms.sig4.7 KiB
skill-card.md
Description: <br>
Representative, point-in-time MoE training playbooks by hardware and model family. Use them as candidate seeds, then revalidate the exact runtime, semantics, topology, and steady-state throughput. <br>
This skill is ready for commercial/non-commercial use. <br>
Owner
NVIDIA <br>
License/Terms of Use: <br>
Apache 2.0 <br>
Use Case: <br>
Developers and performance engineers selecting MoE parallelism configurations and dispatcher strategies for large-scale training on NVIDIA hardware (H100, B200, GB200, GB300). <br>
Deployment Geography for Use: <br>
Global <br>
Requirements / Dependencies: <br>
Requires API Key or External Credential: [No] <br> Credential Type(s): [None] <br>
Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate. <br>
Known Risks and Mitigations: <br>
Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. <br> Mitigation: Review and scan skill before deployment. <br>
Reference(s): <br>
Skill Output: <br>
Output Type(s): [Configuration instructions, Analysis] <br> Output Format: [Markdown with inline code blocks and tables] <br> Output Parameters: [1D] <br> Other Properties Related to Output: [None] <br>
Evaluation Agents Used: <br>
Claude Code (
aws/anthropic/bedrock-claude-opus-4-8) <br>Codex (
openai/openai/gpt-5.5) <br>
Evaluation Tasks: <br>
Evaluated against 1 task (1 positive) in isolated k8s-sandbox pods with dataset digest sha256:bfc3173e. <br>
Evaluation Metrics Used: <br>
Reported benchmark dimensions: <br>
Security: Whether the skill is safe to use (unsafe operations, secret leakage, unauthorized access). <br>
Correctness: Whether the answer is correct against the reference answer. <br>
Discoverability: Whether the right skill was loaded and activated when needed. <br>
Effectiveness: Whether the skill helped complete the user's goal and expected workflow. <br>
Efficiency: Whether the skill avoided wasted tool or skill usage. <br>
Underlying evaluation signals used in this run: <br>
security: Unsafe operations, secret leakage, and unauthorized access. <br>skill_execution: Whether the expected skill was found and executed. <br>skill_efficiency: Routing quality, workspace-aware skill reads, and productive tool use. <br>accuracy: Final-answer correctness against the reference answer. <br>goal_accuracy: Whether the user's goal was achieved. <br>behavior_check: Whether the expected workflow behavior was followed. <br>
Evaluation Results: <br>
| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) |
|---|---|---|
| Overall | 32% → 96% (+64 points) | 64% → 96% (+32 points) |
| Security | 100% → 100% (±0 points) | 100% → 100% (±0 points) |
| Correctness | 0% → 100% (+100 points) | 80% → 100% (+20 points) |
| Discoverability | 50% → 100% (+50 points) | 50% → 94% (+44 points) |
| Effectiveness | 12% → 80% (+68 points) | 80% → 86% (+6 points) |
| Efficiency | 0% → 100% (+100 points) | 8% → 100% (+92 points) |
Skill Version(s): <br>
2f0c6a87 (source: git SHA, committed 2026-08-03) <br>
Ethical Considerations: <br>
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>
(For Release on NVIDIA Platforms Only) <br> Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns here. <br>
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Security
What our scanner noticed· Medium risk
- Large base64-encoded blobs
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Published by
- NVIDIA
- Publisher
- NVIDIA (Official)
- Source
- NVIDIA/skills · skills/nemo-mbridge-perf-moe-hardware-configs · commit fce3cca
- Licence
- Apache License 2.0 (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE-APACHE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
AutoGineer is not affiliated with or endorsed by NVIDIA.
Related skills
Pytorch Lightning
K-Dense· Community
A framework for organizing PyTorch code to streamline neural network training, scaling, and experiment tracking.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreMITLow riskHas scriptsGitHub stars of the source repository56.4k
Aeon
K-Dense· Community
A scikit-learn compatible Python toolkit for comprehensive time series machine learning tasks.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreMITLow riskGitHub stars of the source repository56.4k
Umap Learn
K-Dense· Community
Integrate UMAP-learn for non-linear dimensionality reduction, visualization, and clustering preprocessing.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreMITLow riskGitHub stars of the source repository56.4k
Scikit Learn
K-Dense· Community
Comprehensive guidance for classical machine learning workflows using the scikit-learn Python library.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreMITLow riskHas scriptsGitHub stars of the source repository56.4k
Shap
K-Dense· Community
Explain and audit machine-learning model predictions using the SHAP framework for feature attribution and visualization.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreMITLow riskHas scriptsGitHub stars of the source repository56.4k
Pufferlib
K-Dense· Community
Version-aware guidance for PufferLib reinforcement learning environments, vectorization, policies, and training.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreMITLow riskHas scriptsGitHub stars of the source repository56.4k
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download nemo-mbridge-perf-moe-hardware-configs.zip