Hyperpod Version Checker

From the author

Check and compare software component versions on SageMaker HyperPod cluster nodes - NVIDIA drivers, CUDA toolkit, cuDNN, NCCL, EFA, AWS OFI NCCL, GDRCopy, MPI, Neuron SDK (Trainium/Inferentia), Python, and PyTorch. Use when checking component versions, verifying CUDA/driver compatibility, detecting version mismatches across nodes, planning upgrades, documenting cluster configuration, or troubleshooting version-related issues on HyperPod. Triggers on requests about versions, compatibility, component checks, or upgrade planning for HyperPod clusters.

— the description by AWS Labs

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The HyperPod Version Checker is a diagnostic tool designed to audit the software stack on SageMaker HyperPod cluster nodes. It systematically detects versions of critical components including NVIDIA drivers, CUDA toolkits, deep learning libraries like NCCL and PyTorch, networking stacks like EFA, and AWS Neuron SDKs for Trainium/Inferentia instances. By providing both human-readable reports and machine-parseable JSON output, it enables administrators to verify cluster-wide consistency, validate compatibility between drivers and toolkits, and troubleshoot environment-related issues during distributed training or inference workloads.

What it does

Detects versions for NVIDIA drivers, CUDA, cuDNN, NCCL, EFA, and MPI.
Identifies versions for AWS Neuron SDK, Python, PyTorch, and container runtimes.
Generates detailed text reports or structured JSON output for automated analysis.
Provides built-in compatibility reference tables for drivers, CUDA, EFA, and NCCL.
Supports multi-node comparison workflows when executed across a cluster.
Allows for custom output file paths and logging-friendly console output.

When to use it

When verifying software consistency across all nodes in a HyperPod cluster.
During troubleshooting of version-related errors in distributed training jobs.
When planning upgrades for drivers, CUDA toolkits, or AWS Neuron SDKs.
To document the current configuration state of a cluster for audit purposes.
When validating compatibility between installed GPU drivers and CUDA versions.

Advantages

Reduces manual effort in checking complex software stacks on individual nodes.
Facilitates easy diffing of node configurations using JSON output.
Includes reference data to quickly spot incompatible driver and library combinations.
Supports both interactive debugging and automated logging workflows.

Usage examples

“Check the current software versions on this HyperPod node.”: The script executes and outputs a formatted text report listing all detected versions for drivers, CUDA, and ML libraries.
“Generate a JSON report of all installed components for comparison.”: The script outputs structured JSON to stdout, which can be piped to a file or a comparison tool to identify version mismatches across the cluster.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/awslabs/hyperpod-version-checker/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip hyperpod-version-checker.zip -d .agents/skills

Documentation by AWS Labs

Upload to cluster nodes via hyperpod-ssm skill, then execute.

Usage

# Text report to console + file
bash hyperpod_check_versions.sh

# JSON only to stdout (text report still saved to file) — best for piping/parsing
bash hyperpod_check_versions.sh --json

# Custom output file
bash hyperpod_check_versions.sh --output /tmp/versions.txt

# No color (for logging)
bash hyperpod_check_versions.sh --no-color

Output file: component_versions_<hostname>_<timestamp>.txt (default)

What It Checks

Component	Detection Method	Applicable When
NVIDIA Driver	`nvidia-smi`	GPU instances (p3/p4/p5/g5)
CUDA Toolkit	`nvcc`, `/usr/local/cuda` symlink	GPU instances
cuDNN	Header file, packages	GPU instances doing deep learning
NCCL	Library filename, header, packages	Distributed GPU training
EFA	`/opt/amazon/efa_installed_packages`, `fi_info`	EFA-capable instances (p4d/p4de/p5/trn1/trn2)
AWS OFI NCCL	`efa_installed_packages`, library search	EFA + NCCL workloads
GDRCopy	rpm/dpkg, kernel module	GPU instances with RDMA (p4d+/p5)
MPI	`mpirun`, `/opt/amazon/openmpi`	Distributed training
Neuron SDK	`neuronx-cc`, `neuron-ls`, packages	Trainium/Inferentia (trn1/trn2/inf1/inf2)
Python/PyTorch	`python3`, `torch` import	ML workloads
Container runtime	`docker`, `containerd`, `kubectl`, `nvidia-ctk`	EKS clusters

Multi-Node Comparison

Run on each node individually via the hyperpod-ssm skill. With --json, stdout is clean JSON for easy diffing.

Compatibility Reference

The script automatically analyzes CUDA/driver compatibility. For reference:

Driver Series	Supported CUDA
580+	13.x, 12.x, 11.x
570+	12.8+ (Blackwell), 12.x, 11.x
545+	12.3-12.7, 11.x
525-535	12.0-12.2, 11.x
450+	11.x only

NCCL: Use 2.18+ for CUDA 12.x, 2.12+ for CUDA 11.x. Must be consistent across all nodes.

EFA Installer	AWS OFI NCCL
1.29+	v1.7.3+ (recommended)
1.26-1.28	v1.7.0-v1.7.2
1.20-1.25	v1.6.0+

Files

ATTRIBUTION.mdAdded by AutoGineer932 B
LICENSELicence file added by AutoGineer9.9 KiB
NOTICELicence file added by AutoGineer89 B
SKILL.md3.6 KiB
scripts/hyperpod_check_versions.shScript20.3 KiB

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python, Needs Node.js

Security

What our scanner noticed· Low risk

Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Spark Memory Thermal Ops
Seth Hobson· Community
Manage unified memory and thermal constraints for long-running machine learning jobs on NVIDIA DGX Spark.
Cloud and DevOpsClaude CodeClaude appsOpenAI Codex+14 more
MITLow riskHas scriptsGitHub stars of the source repository42.3k
Observability And Instrumentation
Addy Osmani· Community
A systematic approach to instrumenting production code with structured logs, metrics, and traces for effective system observability.
Cloud and DevOpsClaude CodeClaude appsOpenAI Codex+14 more
MITGitHub stars of the source repository113.9k
Gke Alert Configuration
GoogleOfficial
Configure GKE alerting policies in Terraform using PromQL and Google Cloud Managed Service for Prometheus.
Cloud and DevOpsClaude CodeClaude appsOpenAI Codex+14 more
Apache-2.0Low riskHas scriptsGitHub stars of the source repository23.2k
Application Performance Observability Engineer
Seth Hobson· Community
Expert observability engineer for designing production-grade monitoring, logging, tracing, and reliability systems.
Cloud and DevOpsClaude Code
MITGitHub stars of the source repository42.3k
Distributed Debugging Devops Troubleshooter
Seth Hobson· Community
An expert DevOps troubleshooter for incident response, system reliability, and complex distributed system debugging.
Cloud and DevOpsClaude Code
MITGitHub stars of the source repository42.3k
Distributed Tracing
Seth Hobson· Community
Implement distributed tracing using Jaeger and Tempo to monitor request flows and performance across microservices.
Cloud and DevOpsClaude CodeClaude appsOpenAI Codex+14 more
MITGitHub stars of the source repository42.3k