Generated banner for Hyperpod Performance Debugger
Generated logo for Hyperpod Performance Debugger

Hyperpod Performance Debugger

AWS LabsOfficialPublished by AWS Labs

  • Skill
  • Research and analysis
  • Cloud and DevOps
  • Apache-2.0
  • Low risk
  • Has scripts

From the author

Diagnose performance issues on Amazon SageMaker HyperPod clusters — uneven NCCL bandwidth across nodes and poor filesystem throughput. Read-only. Surfaces host-side signals (Xid, ECC, NVLink, EFA reachability, FSx saturation) and routes to the appropriate sibling skill (hyperpod-node-debugger, hyperpod-nccl, hyperpod-version-checker, hyperpod-issue-report) for any remediation. Triggers on uneven NCCL across nodes, straggler node, FSx slow, checkpoint slow, dataloader slow, filesystem bottleneck, FSx throughput, cross-AZ latency, topology mismatch.
— the description by AWS Labs

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The HyperPod Performance Debugger is a read-only diagnostic tool designed to identify performance degradation in Amazon SageMaker HyperPod clusters. It focuses on two primary issues: uneven NCCL bandwidth across nodes and poor filesystem performance. The skill collects host-side telemetry, including GPU health, EFA connectivity, and filesystem utilization, to provide actionable insights. It acts as a triage layer, surfacing critical signals and routing complex hardware or configuration issues to specialized sibling skills for remediation.

What it does

  • Collects host-side performance snapshots including GPU, EFA, and filesystem metrics.
  • Identifies straggler nodes and uneven NCCL bandwidth across cluster nodes.
  • Detects filesystem bottlenecks and high I/O wait times on FSx mounts.
  • Surfaces hardware-level signals like Xid errors, ECC counts, and NVLink status.
  • Provides clear routing paths to specialized debugging skills for hardware, versioning, or configuration issues.
  • Recommends built-in HyperPod health features for ongoing cluster monitoring.

When to use it

  • When training jobs exhibit uneven performance across different nodes.
  • When data loading or checkpointing operations are causing significant training stalls.
  • When you suspect network topology mismatches or cross-AZ latency issues.
  • When you need to verify host-side health signals before escalating to support.

Advantages

  • Provides a standardized, read-only diagnostic workflow for complex cluster environments.
  • Reduces mean time to resolution by automatically routing issues to the correct specialized skill.
  • Helps distinguish between application-level I/O patterns and infrastructure-level bottlenecks.
  • Integrates with native HyperPod health features to prevent future regressions.

Usage examples

“My training job is running slower on some nodes than others. How do I check for NCCL issues?”
The agent will guide you to run the performance snapshot script and analyze the output for pairwise bandwidth variance or straggler nodes.
“Training is stalling during checkpointing. Is my FSx filesystem saturated?”
The agent will help you collect CloudWatch metrics and host-side iostat data to determine if the filesystem throughput is the bottleneck.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/awslabs/hyperpod-performance-debugger/download
  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip hyperpod-performance-debugger.zip -d .agents/skills

Documentation by AWS Labs

HyperPod Performance Debugger

  1. Uneven NCCL performance across nodes — workload faster on some node sets than others, pairwise bandwidth variance, suspected straggler.

  2. Poor filesystem performance — training stalled on data loading, checkpoint save/load dominating step time, FSx throughput saturated.

Scope and delegation

Route findings outside the two in-scope scenarios to the owner skill below.

Concern observedRoute to
GPU hardware fault, ECC, NVLink, Xid, DCGM diagnostics, drain/replacehyperpod-node-debugger (§ F Hardware/Auto-Repair, § G GPU)
Cannot allocate memory at os.fork(), root volume exhaustedhyperpod-node-debugger (§ I Resource Exhaustion)
NCCL timeouts, hangs, AllReduce stalls, EFA TCP fallback, RDMA memlockhyperpod-nccl
EFA / NCCL / CUDA / NVIDIA driver version drift across nodeshyperpod-version-checker
EFA self-referencing security-group rule missing — single nodehyperpod-node-debugger § A (EFA / Security Group)
EFA self-referencing security-group rule missing — cluster-widehyperpod-cluster-debugger § A (EFA Health Checks)
Slurm node state changes (drain / resume / reboot)hyperpod-slurm-debugger
Diagnostic bundle for AWS Supporthyperpod-issue-report
Shell access on a nodehyperpod-ssm

Operating policy

  • Read-only. Print commands the customer runs; do not execute commands that modify state.

  • Container vs host version comparisons go through hyperpod-version-checker.

  • Xid lines, ECC counts, NVLink lane state, and thermal readings get surfaced; the catalog and verdict live in hyperpod-node-debugger § G.

Workflow

  1. Confirm the symptom is uneven NCCL or poor filesystem performance. If neither, route to the matching sibling skill above.

  2. Run scripts/perf-snapshot.sh (read-only) to gather host-side signals for the suspect node and FSx filesystems mounted on it.

  3. For each [CONCERN] line in the script output, open the matching section below and read the supporting reference.

  4. After the per-incident diagnosis, recommend the HyperPod platform health features in § Continuous health coverage so the customer gets ongoing protection.

Step 1: Run the snapshot

bash scripts/perf-snapshot.sh --cluster <CLUSTER_NAME_OR_ARN> --region <REGION>

# Scope to one suspect node:
bash scripts/perf-snapshot.sh --cluster <C> --region <R> --node <INSTANCE_ID>

The script samples one node by default. It collects host-side data via hyperpod-ssm: nvidia-smi output (temperature, SM clocks, PCIe link width, ECC, NVLink, topo -m), recent dmesg Xid lines, EFA port state and fi_info provider visibility, EFA installer + kernel module versions, CPU governor, NVL72 Fabric Manager state, FSx CloudWatch utilization, df -h / lfs df -h per mount, host iowait, /dev/shm size, and root-volume usage. All read-only.

Tags: [OK] healthy · [CONCERN] signal worth investigating (carries a → pointer to the owner skill) · [INFO] informational.

Host vs container scope. The script runs on the host via SSM and reports host-scope values. Many setups ship the EFA / libfabric / OFI-NCCL / CUDA stack inside the training container by design — a host value of unknown is not by itself a defect. What matters for performance is the stack the workload actually uses. Verify versions inside the container (and across nodes) via hyperpod-version-checker before drawing conclusions.

Step 2: Match signal → section

ObservationSection
Pairwise NCCL bandwidth varies across node pairs / suspected stragglerA: Uneven NCCL Performance
Nodes spread across AZs / network-node-layer labels / UltraServer boundariesA
EFA port not ACTIVE on a node, missing OFI plugin, or FI provider not visibleA + route to hyperpod-node-debugger § A; hyperpod-version-checker for cross-node version compare
iostat shows high iowait, FSx CloudWatch utilization sustained near 100%B: Poor Filesystem Performance
DataLoader stalls, checkpoint dominates step timeB
Xid line in dmesg, uncorrectable ECC, inactive NVLink lane, GPU ≥ 88°CRoute to hyperpod-node-debugger § G
Container vs host version drift suspectedRoute to hyperpod-version-checker
Cannot allocate memory at os.fork(), root volume full, OOM eventsRoute to hyperpod-node-debugger § I
NCCL timeout, hang, TCP fallback (NET/OFI Using TCP), RDMA memlockRoute to hyperpod-nccl

A: Uneven NCCL Performance

The customer reports identical training jobs running with different step times on different node sets, pairwise bandwidth variance, or some allocations consistently slower than others despite identical code.

Per the official troubleshooting guide, the common contributing factors are network topology differences between nodes (cross-AZ, cross-rack, cross-UltraServer), degraded EFA performance on some nodes, mixed instance types or generations within an instance group, and CPU frequency scaling differences.

Diagnostic pass (read-only)

The host-side data points — GPU thermal/ECC/PCIe/clocks, Xid, NVLink lanes, EFA port state and provider visibility, CPU governor, EFA/OFI/driver versions, nvidia-smi topo -m — are all collected by scripts/perf-snapshot.sh (Step 1 above). The script tags [CONCERN] with thresholds and emits routing pointers; rerun it per suspect node via --node <INSTANCE_ID>.

For driver / CUDA / NCCL / EFA / OFI version drift across nodes, run hyperpod-version-checker skill.

Pairwise NCCL bandwidth test

Run the standard nccl-tests recipes from awslabs/awsome-distributed-training. For an N-node cluster, run all-reduce across every pair and record busbw for each pair. Pairs more than ~5% below the run mean (the threshold the AWS validation script flags) are problematic candidates.

Expected busbw per SKU is published in the AI-on-HyperPod NCCL test guide. Benchmark the specific instance type before relying on a number.

Pairwise scripts, HyperPod topology surfaces (HyperPod API, EKS labels, Slurm topology.conf), and GB200 NVL72 specifics are in references/perf-details.md § Uneven NCCL.

Topology verification

HyperPod exposes topology through three operator-visible surfaces:

  • HyperPod API: aws sagemaker describe-cluster-node returns NodeDetails.Placement.AvailabilityZone / AvailabilityZoneId and NodeDetails.UltraServerInfo.Id (UltraServer SKUs only).

  • EKS labels: topology.kubernetes.io/zone, topology.k8s.aws/network-node-layer-{1,2,3} (highest-numbered = closest to instance), topology.k8s.aws/ultraserver-id.

  • Slurm: HyperPod auto-generates topology.conf. Inspect via scontrol show topology.

Tightly coupled work shares the same AZ, the same highest-numbered network-node-layer label (EKS) or the same Slurm topology block, and — for NVL72 jobs — the same UltraServerInfo.Id / topology.k8s.aws/ultraserver-id. If the cluster is spread across AZs or layers, topology must be re-established at provisioning time. Route provisioning changes to hyperpod-cluster-debugger § B (Capacity & AZ).


B: Poor Filesystem Performance

The customer reports training bottlenecked on data loading, checkpoint save/load dominating step time, executables/scripts loading slowly, or iowait high.

Per the official troubleshooting guide, the resolution path follows this order:

  1. Check CloudWatch metrics on the filesystem.

  2. Check the provisioned performance configuration against workload requirements.

  3. Investigate which operations are causing the I/O — workload demand vs inefficient pattern.

  4. Consider upgrading provisioned performance.

  5. Choose the filesystem type that matches the I/O pattern.

This skill covers steps 1–3. Steps 4–5 are customer decisions; surface the data and let the customer pick.

Diagnostic pass (read-only)

scripts/perf-snapshot.sh (Step 1 above) covers the on-node side of this pass: it discovers FSx mounts, calls aws cloudwatch get-metric-statistics on DataReadBytes and (for OpenZFS) FileServerDiskIopsUtilization, prints df -h for /fsx /opt/dlami/nvme /opt/sagemaker, runs lfs df -h per Lustre mount, and reports iostat iowait. It tags [CONCERN] when OpenZFS IOPS utilization sustains ≥ 80% or iowait > 20%.

For longer windows or additional metrics (DataWriteBytes, Lustre DiskIopsUtilization, OpenZFS FileServerDiskThroughputUtilization), drive the query directly:

aws cloudwatch get-metric-statistics --region <REGION> \
  --namespace AWS/FSx --metric-name DataReadBytes \
  --dimensions Name=FileSystemId,Value=<FSID> \
  --start-time "$(date -u -d '3 hours ago' +%Y-%m-%dT%H:%M:%S)" \
  --end-time   "$(date -u +%Y-%m-%dT%H:%M:%S)" \
  --period 60 --statistics Sum Maximum

The full per-filesystem-type metric catalog is in references/perf-details.md § Filesystem.

Branches

Provisioned capacity is saturated. CloudWatch utilization sustained near 100% across the workload window. Customer decision: scale up the filesystem.

  • FSx for Lustre throughput scales with StorageCapacity × PerUnitStorageThroughput; capacity changes are non-disruptive.

  • FSx for OpenZFS — increase provisioned IOPS or throughput.

I/O pattern is inefficient. CloudWatch shows headroom but the workload is still I/O-bound. Customer decision: change the application.

  • DataLoader: raise num_workers, set pin_memory=True, persistent_workers=True.

  • Checkpointing: use async + sharded (torch.distributed.checkpoint.async_save plus FSDP SHARDED_STATE_DICT). FULL_STATE_DICT serializes through rank 0 and is a frequent root cause.

  • Small-file workloads: Lustre is optimized for large sequential I/O. For millions of small files, use WebDataset / tar shards, FSx for OpenZFS, or NVMe scratch.

Filesystem-selection guidance and the async-checkpoint pattern are in references/perf-details.md § Filesystem.


Continuous health coverage

Once the immediate incident is diagnosed, recommend HyperPod's built-in health features so problems are caught before the next training run rather than after another customer-reported regression.

  • Enable NodeRecovery=Automatic on the cluster. The Health Monitoring Agent (HMA) continuously monitors GPU- and Trainium-based instances and marks instances unhealthy on detected failure. With auto-recovery enabled, HyperPod reboots or replaces the node — no operator intervention.

  • Enable OnStartDeepHealthChecks on every GPU instance group with both check categories:

    • InstanceStress — stress-ng on CPU/memory/disk, GPU and PCI device count verification, DCGM level-4 diagnostics (memory test included), and EFA loopback bandwidth/latency.

    • InstanceConnectivity — multi-node NCCL all-reduce.

    Every newly provisioned or auto-replaced node passes the same hardware bar before accepting jobs.

  • Run on-demand deep health checks when this skill or any sibling surfaces a hardware concern but the cluster is mid-workload. aws sagemaker start-cluster-health-check runs the same checks against a specific instance group; nodes are placed in a Slurm maintenance reservation and the check is queued until any running job completes (not preempted). Console: HyperPod → Clusters → Instances → Run deep health checks.

    Not supported when NodeProvisioningMode=Continuous; one on-demand request per cluster at a time. Requires the latest AMI — run UpdateClusterSoftware first.

Logs land in CloudWatch at /aws/sagemaker/Clusters/<cluster_name>/<cluster_id> under DeepHealthCheckResults/<log_stream_id>, and on each node at /var/log/aws/clusters/sagemaker-deep-health-check.log.

References

  • references/perf-details.md — pairwise NCCL test recipes, HyperPod topology check, GB200 NVL72 placement; CloudWatch metric catalog per filesystem type, async-checkpoint pattern, filesystem selection guide.

External:

Files

LICENSE


                                 Apache License
                           Version 2.0, January 2004
                        http://www.apache.org/licenses/

   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

   1. Definitions.

      "License" shall mean the terms and conditions for use, reproduction,
      and distribution as defined by Sections 1 through 9 of this document.

      "Licensor" shall mean the copyright owner or entity authorized by
      the copyright owner that is granting the License.

      "Legal Entity" shall mean the union of the acting entity and all
      other entities that control, are controlled by, or are under common
      control with that entity. For the purposes of this definition,
      "control" means (i) the power, direct or indirect, to cause the
      direction or management of such entity, whether by contract or
      otherwise, or (ii) ownership of fifty percent (50%) or more of the
      outstanding shares, or (iii) beneficial ownership of such entity.

      "You" (or "Your") shall mean an individual or Legal Entity
      exercising permissions granted by this License.

      "Source" form shall mean the preferred form for making modifications,
      including but not limited to software source code, documentation
      source, and configuration files.

      "Object" form shall mean any form resulting from mechanical
      transformation or translation of a Source form, including but
      not limited to compiled object code, generated documentation,
      and conversions to other media types.

      "Work" shall mean the work of authorship, whether in Source or
      Object form, made available under the License, as indicated by a
      copyright notice that is included in or attached to the work
      (an example is provided in the Appendix below).

      "Derivative Works" shall mean any work, whether in Source or Object
      form, that is based on (or derived from) the Work and for which the
      editorial revisions, annotations, elaborations, or other modifications
      represent, as a whole, an original work of authorship. For the purposes
      of this License, Derivative Works shall not include works that remain
      separable from, or merely link (or bind by name) to the interfaces of,
      the Work and Derivative Works thereof.

      "Contribution" shall mean any work of authorship, including
      the original version of the Work and any modifications or additions
      to that Work or Derivative Works thereof, that is intentionally
      submitted to Licensor for inclusion in the Work by the copyright owner
      or by an individual or Legal Entity authorized to submit on behalf of
      the copyright owner. For the purposes of this definition, "submitted"
      means any form of electronic, verbal, or written communication sent
      to the Licensor or its representatives, including but not limited to
      communication on electronic mailing lists, source code control systems,
      and issue tracking systems that are managed by, or on behalf of, the
      Licensor for the purpose of discussing and improving the Work, but
      excluding communication that is conspicuously marked or otherwise
      designated in writing by the copyright owner as "Not a Contribution."

      "Contributor" shall mean Licensor and any individual or Legal Entity
      on behalf of whom a Contribution has been received by Licensor and
      subsequently incorporated within the Work.

   2. Grant of Copyright License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      copyright license to reproduce, prepare Derivative Works of,
      publicly display, publicly perform, sublicense, and distribute the
      Work and such Derivative Works in Source or Object form.

   3. Grant of Patent License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      (except as stated in this section) patent license to make, have made,
      use, offer to sell, sell, import, and otherwise transfer the Work,
      where such license applies only to those patent claims licensable
      by such Contributor that are necessarily infringed by their
      Contribution(s) alone or by combination of their Contribution(s)
      with the Work to which such Contribution(s) was submitted. If You
      institute patent litigation against any entity (including a
      cross-claim or counterclaim in a lawsuit) alleging that the Work
      or a Contribution incorporated within the Work constitutes direct
      or contributory patent infringement, then any patent licenses
      granted to You under this License for that Work shall terminate
      as of the date such litigation is filed.

   4. Redistribution. You may reproduce and distribute copies of the
      Work or Derivative Works thereof in any medium, with or without
      modifications, and in Source or Object form, provided that You
      meet the following conditions:

      (a) You must give any other recipients of the Work or
          Derivative Works a copy of this License; and

      (b) You must cause any modified files to carry prominent notices
          stating that You changed the files; and

      (c) You must retain, in the Source form of any Derivative Works
          that You distribute, all copyright, patent, trademark, and
          attribution notices from the Source form of the Work,
          excluding those notices that do not pertain to any part of
          the Derivative Works; and

      (d) If the Work includes a "NOTICE" text file as part of its
          distribution, then any Derivative Works that You distribute must
          include a readable copy of the attribution notices contained
          within such NOTICE file, excluding those notices that do not
          pertain to any part of the Derivative Works, in at least one
          of the following places: within a NOTICE text file distributed
          as part of the Derivative Works; within the Source form or
          documentation, if provided along with the Derivative Works; or,
          within a display generated by the Derivative Works, if and
          wherever such third-party notices normally appear. The contents
          of the NOTICE file are for informational purposes only and
          do not modify the License. You may add Your own attribution
          notices within Derivative Works that You distribute, alongside
          or as an addendum to the NOTICE text from the Work, provided
          that such additional attribution notices cannot be construed
          as modifying the License.

      You may add Your own copyright statement to Your modifications and
      may provide additional or different license terms and conditions
      for use, reproduction, or distribution of Your modifications, or
      for any such Derivative Works as a whole, provided Your use,
      reproduction, and distribution of the Work otherwise complies with
      the conditions stated in this License.

   5. Submission of Contributions. Unless You explicitly state otherwise,
      any Contribution intentionally submitted for inclusion in the Work
      by You to the Licensor shall be under the terms and conditions of
      this License, without any additional terms or conditions.
      Notwithstanding the above, nothing herein shall supersede or modify
      the terms of any separate license agreement you may have executed
      with Licensor regarding such Contributions.

   6. Trademarks. This License does not grant permission to use the trade
      names, trademarks, service marks, or product names of the Licensor,
      except as required for reasonable and customary use in describing the
      origin of the Work and reproducing the content of the NOTICE file.

   7. Disclaimer of Warranty. Unless required by applicable law or
      agreed to in writing, Licensor provides the Work (and each
      Contributor provides its Contributions) on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
      implied, including, without limitation, any warranties or conditions
      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
      PARTICULAR PURPOSE. You are solely responsible for determining the
      appropriateness of using or redistributing the Work and assume any
      risks associated with Your exercise of permissions under this License.

   8. Limitation of Liability. In no event and under no legal theory,
      whether in tort (including negligence), contract, or otherwise,
      unless required by applicable law (such as deliberate and grossly
      negligent acts) or agreed to in writing, shall any Contributor be
      liable to You for damages, including any direct, indirect, special,
      incidental, or consequential damages of any character arising as a
      result of this License or out of the use or inability to use the
      Work (including but not limited to damages for loss of goodwill,
      work stoppage, computer failure or malfunction, or any and all
      other commercial damages or losses), even if such Contributor
      has been advised of the possibility of such damages.

   9. Accepting Warranty or Additional Liability. While redistributing
      the Work or Derivative Works thereof, You may choose to offer,
      and charge a fee for, acceptance of support, warranty, indemnity,
      or other liability obligations and/or rights consistent with this
      License. However, in accepting such obligations, You may act only
      on Your own behalf and on Your sole responsibility, not on behalf
      of any other Contributor, and only if You agree to indemnify,
      defend, and hold each Contributor harmless for any liability
      incurred by, or claims asserted against, such Contributor by reason
      of your accepting any such warranty or additional liability.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs a command-line tool, Needs Node.js

Security

What our scanner noticed· Low risk

  • Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
AWS Labs
Publisher
AWS Labs (Official)
Source
awslabs/agent-plugins · plugins/sagemaker-ai/skills/hyperpod-performance-debugger · commit fbefb37
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • NOTICE (added by AutoGineer from NOTICE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by AWS Labs.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download hyperpod-performance-debugger.zip