Hyperpod Issue Report

From the author

Generate comprehensive issue reports from HyperPod clusters (EKS and Slurm) by collecting diagnostic logs and configurations for troubleshooting and AWS Support cases. Use when users need to collect diagnostics from HyperPod cluster nodes, generate issue reports for AWS Support, investigate node failures or performance problems, document cluster state, or create diagnostic snapshots. Triggers on requests involving issue reports, diagnostic collection, support case preparation, or cluster troubleshooting that requires gathering logs and system information from multiple nodes.

— the description by AWS Labs

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The HyperPod Issue Report skill provides a structured workflow to gather comprehensive diagnostic data from EKS and Slurm-based HyperPod clusters. It utilizes SSM to collect system logs, GPU status, and configuration files from cluster nodes in parallel, storing the results in an S3 bucket. The tool automatically detects the cluster type and supports targeted collection by instance group or specific node IDs, facilitating efficient troubleshooting and preparation for AWS Support cases.

What it does

Collects diagnostic logs and system configurations from HyperPod nodes via SSM.
Supports both EKS and Slurm cluster architectures with automatic detection.
Aggregates cluster-wide resource information and node-specific logs into S3.
Generates a summary report of collection success and failure statistics.
Provides EKS-specific cluster resource snapshots using kubectl.
Offers options to tune performance for large clusters by adjusting worker concurrency.

When to use it

When investigating node failures or performance degradation in a HyperPod cluster.
When preparing diagnostic data for an AWS Support case.
When you need to document the current state of a cluster for auditing or troubleshooting.
When you need to collect logs from multiple nodes simultaneously to identify patterns.

Advantages

Reduces manual effort by automating log collection across large clusters.
Ensures consistent diagnostic data capture for both EKS and Slurm environments.
Provides configurable parallel execution to handle clusters of varying sizes.
Centralizes diagnostic artifacts in S3 for easy sharing and analysis.

Usage examples

“Generate an issue report for my HyperPod cluster named my-cluster in us-west-2 and save it to s3://my-diagnostic-bucket/reports.”: The agent will verify the cluster, ensure the S3 bucket exists, and run the collection script to gather logs from all nodes, outputting a summary and the S3 path.
“Collect diagnostics only for the worker-group instance group in my cluster.”: The agent will execute the collection script with the --instance-groups flag, limiting the diagnostic gathering to nodes within that specific group.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/awslabs/hyperpod-issue-report/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip hyperpod-issue-report.zip -d .agents/skills

Documentation by AWS Labs

Collect diagnostic logs from HyperPod cluster nodes via SSM, store results in S3. Supports both EKS and Slurm clusters with auto-detection. Uses the bundled scripts/hyperpod_issue_report.py for reliable parallel collection.

Prerequisites

AWS CLI configured with permissions: sagemaker:DescribeCluster, sagemaker:ListClusterNodes, ssm:StartSession, s3:PutObject, s3:GetObject, eks:DescribeCluster
Python 3.8+ and uv (see uv installation docs for install options)
SSM Agent running on target nodes; node IAM roles need s3:GetObject/s3:PutObject on the report bucket
For EKS clusters: kubectl installed and configured (see Workflow step 2)

Workflow

1. Gather Information

Collect from the user:

Cluster identifier (required): accepts cluster name or full cluster ARN (e.g., arn:aws:sagemaker:us-west-2:123456789012:cluster/abc123)
AWS region (required unless extractable from ARN)
S3 path for report storage (required, e.g. s3://bucket/prefix). If the user doesn't have a bucket, create one (e.g., s3://hyperpod-diagnostics-<account-id>-<region>)
Issue description (optional)
Target scope: all nodes, specific instance groups, or specific node IDs (optional)

2. Verify Environment

aws sts get-caller-identity
aws sagemaker describe-cluster --cluster-name <name-or-arn> --region <region>

If the S3 bucket doesn't exist, create it:

aws s3 mb s3://<bucket-name> --region <region>

For EKS clusters (check Orchestrator.Eks in describe-cluster output):

Ensure kubectl is installed (which kubectl). If missing, install it for the current platform.
Configure kubeconfig using the EKS cluster name from the describe-cluster response:
```
aws eks update-kubeconfig --name <eks-cluster-name> --region <region>
```

3. Run the Collection Script

uv run scripts/hyperpod_issue_report.py \
  --cluster <cluster-name-or-arn> \
  --region <region> \
  --s3-path s3://<bucket>[/prefix]

Use --help for all options including --instance-groups, --nodes, --max-workers, and --debug. Note: --instance-groups and --nodes are mutually exclusive. Node identifiers accept instance IDs (i-*), EKS names (hyperpod-i-*), or Slurm names (ip-*).

4. Present Results

After collection, the script shows statistics and offers interactive download. Report the S3 location and offer to:

Download the report locally
Help analyze collected diagnostics (see references/collection-details.md for what's in each file)
Prepare a summary for AWS Support

Troubleshooting

See references/troubleshooting.md for error handling, large cluster tuning, and known limitations.

Files

ATTRIBUTION.mdAdded by AutoGineer926 B
LICENSELicence file added by AutoGineer9.9 KiB
NOTICELicence file added by AutoGineer89 B
SKILL.md3.5 KiB
references/collection-details.md3.6 KiB
references/troubleshooting.md3.5 KiB
scripts/hyperpod_issue_report.pyScript65.3 KiB

SKILL.md