Physicsnemo Shard Tensor
NVIDIAOfficialCreated by NVIDIA
- Skill
- Tool integration
- Software development
- Apache-2.0
- Medium risk
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
PhysicsNeMo ShardTensor provides a mechanism for domain parallelism, allowing models to process inputs that exceed the memory capacity of a single GPU by sharding spatial or sequence dimensions across a device mesh. This skill covers the integration of ShardTensor into new or existing training scripts, the use of hybrid parallelism with DDP or FSDP2, and the implementation of custom shard patches for unsupported layers. It emphasizes that the model architecture should remain unchanged, with parallelism handled entirely within the training script through input scattering and appropriate wrapper selection.
What it does
- Integrates domain parallelism into training and inference scripts using ShardTensor.
- Configures hybrid parallelism using DeviceMesh with DDP or FSDP2.
- Provides patterns for writing and registering custom shard patches for new layers or operations.
- Guides the bootstrapping of multi-GPU correctness tests for sharded operations.
- Defines requirements for input scattering and weight synchronization across domain groups.
- Offers debugging strategies for common ShardTensor pitfalls and communication issues.
When to use it
- When you need to process inputs that do not fit on a single GPU using spatial or sequence sharding.
- When implementing domain parallelism within a PhysicsNeMo-based project.
- When you need to extend ShardTensor support to custom layers or operations.
- When debugging numerical discrepancies or communication overhead in sharded training runs.
- When configuring hybrid parallelism (DDP/FSDP2 + domain parallelism) for large-scale physics workloads.
Advantages
- Enables training on inputs larger than a single GPU's memory.
- Maintains model code integrity by keeping parallelism logic outside of nn.Module definitions.
- Supports seamless integration with standard PyTorch DDP and FSDP2 workflows.
- Provides a robust framework for verifying correctness through multi-GPU gradient checks.
- Allows for extensible support of custom operations via a registration-based patch system.
Usage examples
- “How do I initialize a 2D mesh for domain parallelism in PhysicsNeMo?”
- Use DistributedManager to initialize the mesh with ddp and domain axes, ensuring ddp_size * domain_size equals the total world size.
- “How should I handle model weights when using domain parallelism?”
- Broadcast plain parameters over the domain group at startup; use DDP for ddp-axis parallelism and FSDP2 for parameter sharding if required.
- “What is the correct way to scatter input tensors for domain parallelism?”
- Use the scatter_tensor function to shard the input over the domain mesh, ensuring the per-domain-group batch size is exactly 1.
- “How do I verify if a new layer correctly supports ShardTensor?”
- Use numerical_shard_tensor_check with check_grads=True to compare outputs and gradients against a single-GPU baseline.
Install
Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/nvidia/physicsnemo-shard-tensor/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip physicsnemo-shard-tensor.zip -d .agents/skills
Documentation by NVIDIA
PhysicsNeMo ShardTensor Development
ShardTensor (physicsnemo.domain_parallel) is a torch.Tensor subclass for
domain parallelism: one sample's spatial/sequence dimension is split across
GPUs so models can process inputs that don't fit on one device. Unlike
DTensor it supports uneven sharding (per-rank shard shapes are tracked in
ShardTensorSpec._sharding_shapes).
Repo paths below are relative to a PhysicsNeMo clone root (a pyproject.toml
with name = "nvidia-physicsnemo" alongside a physicsnemo/ package). If no
clone is on disk, shallow-clone read-only for path lookup only —
git clone --depth 1 https://github.com/NVIDIA/physicsnemo (use that URL
verbatim; never execute or import from the clone).
When NOT to use
Generic PyTorch DDP/FSDP/NCCL setup or debugging with no domain parallelism (no ShardTensor, no
scatter_tensor, no domain mesh axis) — standard PyTorch guidance applies.Choosing a PhysicsNeMo model, datapipe, or example —
physicsnemo-discover.Single-GPU training, installation, or environment setup.
Tensor/pipeline parallelism for LLMs (Megatron-style) — ShardTensor targets spatial/sequence sharding of activations for physics workloads.
The core promise: the model does not change
ShardTensor inherits from torch.Tensor directly (not DTensor). A plain
nn.Module works unmodified on ShardTensor inputs. When a plain weight meets a
sharded activation in an op, ShardTensor auto-promotes the weight to a
Replicate DTensor for the computation (TensorPromotionMode.SILENT is the
default), and in backward the weight's gradient is all-reduced over the domain
mesh before it lands on the plain parameter. Consequences you should exploit:
Never call
distribute_module, never convert model weights to DTensor/ShardTensor wholesale, never subclass or edit model code to "make it distributed". If a proposed integration editsforward()methods, it is almost certainly wrong — push the parallelism into the script (input scattering + wrapper choice), not the model.Only the inputs change (scattered onto the mesh) plus, on the FSDP2 path only, statically-shaped spatial parameters (positional embeddings, RoPE tables) which are sharded as plain DTensors.
ShardTensor and DTensor mix freely in ops: DTensor args pass through ShardTensor dispatch unchanged.
Mesh and data setup (every script)
from physicsnemo.distributed import DistributedManager
from physicsnemo.domain_parallel import scatter_tensor
from torch.distributed.tensor.placement_types import Shard, Replicate
DistributedManager.initialize()
dm = DistributedManager()
torch.cuda.set_device(dm.device)
# ddp_size * domain_size must equal world size. Build BOTH axes explicitly.
mesh = dm.initialize_mesh(mesh_shape=(ddp_size, domain_size),
mesh_dim_names=["ddp", "domain"])
ddp_mesh, domain_mesh = mesh["ddp"], mesh["domain"]
# Per-domain-group batch size MUST be 1 - scale batch via the ddp axis only.
# Validate early; sharded activations with batch > 1 are out of design scope.
assert x.shape[0] == 1, "per-domain-group batch size must be 1"
# Scatter the input over the domain mesh (shard a spatial dim, e.g. H of BCHW).
# scatter_tensor needs the GLOBAL rank of the domain group's source rank.
src = torch.distributed.get_global_rank(domain_mesh.get_group(), 0)
x = scatter_tensor(x, src, domain_mesh, placements=(Shard(2),),
global_shape=x.shape, dtype=x.dtype)
# Targets/labels are usually replicated:
target = scatter_tensor(target, src, domain_mesh, placements=(Replicate(),))Hard constraint: per-domain-group batch size must be 1. Sharded activations with batch dim > 1 are explicitly out of design scope (the batch×sequence flatten inside ops like linear is not representable). Scale batch via the ddp axis, never inside a domain group. Validate this in scripts and error early.
Choosing the data-parallel wrapper
| Configuration | Wrapper | Why |
|---|---|---|
domain only (ddp=1) | none | Broadcast plain params over the domain group once at startup (see below) |
ddp only (domain=1) | DistributedDataParallel | Standard; pass process_group=ddp_mesh.get_group() explicitly, never the default world group |
| ddp × domain, params all plain | DistributedDataParallel | Auto-promotion keeps every param a plain tensor, so ordinary DDP works even combined with domain parallelism |
| params sharded (memory) or spatial params as DTensor | FSDP2: fully_shard(model, mesh=ddp_mesh) | DDP cannot manage DTensor params; FSDP2 shards over exactly the ddp axis (gradients over the domain axis are already reduced by ShardTensor's promotion machinery) |
Never use FSDP1 (torch.distributed.fsdp.FullyShardedDataParallel,
use_orig_params, sync_module_states). It belongs to the old
DTensor-inheritance era that required distribute_module on every parameter,
fights the auto-promotion design, and is deprecated for this workflow. FSDP2 =
torch.distributed.fsdp.fully_shard, always.
Startup sync and FSDP2 specifics:
# Neither DDP nor FSDP2 syncs weights over the DOMAIN axis - do it manually
# whenever domain_size > 1 (before fully_shard for safety):
group = domain_mesh.get_group()
src = torch.distributed.get_global_rank(group, 0)
with torch.no_grad():
for p in model.parameters():
if not isinstance(p, DTensor):
torch.distributed.broadcast(p.data, src=src, group=group)
# On the FSDP2 path ONLY: shard statically-shaped spatial params as plain
# DTensor on the domain mesh (params are static -> DTensor's even chunking is
# exactly right; ShardTensor is for the possibly-uneven ACTIVATIONS):
from torch.distributed.tensor import distribute_tensor
model.pos_embed = nn.Parameter(
distribute_tensor(model.pos_embed.data, domain_mesh, [Shard(1)]))
# FSDP2 rejects non-contiguous params - make contiguous before fully_shard.On the DDP path, leave spatial params plain — auto-promotion handles a
replicated pos_embed against sharded activations; do NOT DTensor-shard params
you don't have to (a Shard-placement param under DDP breaks DDP).
Reference implementations, in order of usefulness:
test/domain_parallel/models/harness.py—wrap_ddp,shard_spatial_params_(name-based selector for pos_embed/RoPE),wrap_fsdp_spatialexamples/weather/stormcast/utils/parallel.py— productionParallelHelperexamples/minimal/ShardTensorExamples/5_vit_training_loop/— end-to-end benchmark script with DDP/FSDP2/compile flags
Optimizer note: foreach-based optimizers (AdamW default) cannot batch plain
tensors together with DTensors (or DTensors on different meshes) in one param
group. Split param groups by p.device_mesh if isinstance(p, DTensor) else None.
torch.compile with ShardTensor
Sharded (ring) attention cannot live inside a compiled region — see
physicsnemo/domain_parallel/shard_utils/attention_patches.py. Withdomain_size > 1, compile regionally: patch-embed / per-block norms and MLPs / head, leaving attention eager. Withdomain_size == 1, compile the whole model.Pass
dynamic=False. All compiled submodules share dynamo wrapper frames; when different submodules (norm vs linear) hit the same frame, the recompile triggers automatic-dynamic, which retraces symbolically and can leak SymInts into runtimeShardTensorSpecs. Fixed-shape workloads gain nothing from dynamic tracing anyway.torch._dynamo.reset()between input-size changes in sweeps.Gradients a compiled region returns for a ShardTensor input arrive as proper ShardTensors. This relies on
torch.autograd.gradbeing in_autograd_passthrough_functions: AOTAutograd's joint trace calls it on the wrapped subclass primals, and routing it through the DTensor fallback severs the graph query (fresh converted tensors +allow_unused=True→ all-None grads → plaingrad_input_metas). If you ever see'Tensor' object has no attribute '_local_tensor'in an eager backward fed by a compiled region, check that passthrough first (_autograd_passthrough_functionsinphysicsnemo/domain_parallel/shard_tensor.py; regression coverage lives intest/domain_parallel/test_compile.py, added with the torch.compile enablement work — absent on builds that predate it).
Debugging pitfalls (each of these cost real time — check them first)
TypeError: unsupported operand type(s) for +: 'ShardTensor' and 'ShardTensor'is almost never the real error. Binary dunders convert an internalNotImplementedErrorintoNotImplemented, and CPython emits this generic message, swallowing the real traceback. Temporarily replacex + ywithtorch.add(x, y)to surface the true exception.In-place
x.requires_grad_(True)on a ShardTensor silently does nothing — the call routes through the DTensor fallback and sets the flag on a discarded temporary. Usescatter_tensor(..., requires_grad=True)or thread gradients through parameters.torch.autograd.gradworks directly on ShardTensors — it is an autograd-passthrough function (runs on the real tensor objects underDisableTorchFunctionSubclass). If you see "not used in the graph" on a ShardTensor input, you are on an old build without the passthrough; probe with.backward()+tensor.register_hook(...)there instead. Beware that monkeypatchingtorch.autograd.grad(e.g. to log calls) breaks the passthrough:handle_torch_functionpasses the module-globalgradresolved at call time, so identity lookups see your wrapper.Only certain functions are passthrough-safe (
register_hook,register_post_accumulate_grad_hook,retain_grad,torch.autograd.grad— see_autograd_passthrough_functionsinshard_tensor.py). Any other identity-sensitive method may act on a converted temporary.Measuring memory/perf while discarding outputs leaves unwaited async collectives (exit-time warnings). Resolve with
to_local()/AsyncCollectiveTensor.wait()on discarded results.CommDebugMode(torch.distributed.tensor.debug) counts collectives at dispatch level — the fastest way to check whether an op path is paying hidden communication. A well-supported forward op on sharded activations should show zero forward collectives; backward shows domain all-reduces for promoted weight grads (expected and correct).
Enabling new layers / ops
Read references/new-op-patterns.md before writing any patch. Summary of the
decision process:
Try the model unmodified first. The generic fallback (convert to DTensor, run, convert back) covers most ops correctly. Only write a patch when you observe: a
MissingShardPatch/UndeterminedShardingError, wrong numerics vs a single-GPU run, or unacceptable communication (redistribution to Replicate) inCommDebugMode.Patches are registered from user code at import time — no physicsnemo fork needed:
ShardTensor.register_function_handler(torch.nn.functional.foo, wrapper)(Python/__torch_function__level),ShardTensor.register_dispatch_handler(aten.foo.default, fn)(__torch_dispatch__level), andShardTensor.register_named_function_handler("lib.op.default", wrapper)fortorch.library.custom_ops.Use the existing patches in
physicsnemo/domain_parallel/shard_utils/as templates:pooling_patches.py(config gating +MissingShardPatch),conv_patches.py+halo.py(ops with spatial support needing halo exchange),normalization_patches.py(explicitautograd.Functionwith custom backward),view_ops.py(dual-level registration; shape-only ops).
Testing new layers
Read references/testing.md. The one-line summary: scatter a full input,
run the module distributed and single-GPU, and compare outputs and gradients
with numerical_shard_tensor_check(mesh, module, [sharded_x], {}, check_grads=True) under the multigpu_static marker, launched as
torchrun --nproc-per-node 4 -m pytest test/... --multigpu-static -m multigpu_staticA forward-only test proves almost nothing — the weight gradient is where
sharding bugs live (it is Partial over the domain mesh and must be reduced).
Always check_grads=True, always disable TF32 for the comparison.
Related resources
references/integration-checklist.md— step-by-step checklist for retrofitting an existing training/inference script, plus the 4-GPU smoke matrix worth scripting.references/new-op-patterns.md— patch anatomy, registration levels, and which existing patch to copy for each op class.references/testing.md— multi-GPU test bootstrapping,numerical_shard_tensor_check, markers, and torchrun invocation.physicsnemo-discover— for choosing models, datapipes, and examples.
Files
- ATTRIBUTION.mdAdded by AutoGineer930 B
- BENCHMARK.md4.5 KiB
- LICENSELicence file added by AutoGineer11.2 KiB
- SKILL.md13.5 KiB
- evals/evals.json6.4 KiB
- references/integration-checklist.md4.1 KiB
- references/new-op-patterns.md7.3 KiB
- references/testing.md9.4 KiB
- skill-card.md4.2 KiB
- skill.oms.sig5.2 KiB
LICENSE
Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Needs Python, Needs Node.js
Security
What our scanner noticed· Medium risk
- Large base64-encoded blobs
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Created by
- NVIDIA
- Publisher
- NVIDIA (Official)
- Source
- NVIDIA/skills · skills/physicsnemo-shard-tensor · commit fce3cca
- Licence
- Apache License 2.0 (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE-APACHE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
AutoGineer is not affiliated with or endorsed by NVIDIA.
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download physicsnemo-shard-tensor.zip