Hyperpod Cluster Debugger

AWS LabsOfficialPublished by AWS Labs

Skill
Review and audit
Cloud and DevOps
Apache-2.0
Low risk
Has scripts

From the author

Diagnose and remediate cluster-wide HyperPod (EKS or Slurm) problems — creation / deployment failures (CloudFormation, EFA health check, lifecycle scripts, capacity), EKS access, node replacement, CloudFormation nested-stack errors, post-maintenance rollback state, dangling nodes, autoscaler conflicts. Includes `--validate` pre-flight. Read-only.

— the description by AWS Labs

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The HyperPod Cluster Debugger is a read-only diagnostic tool designed to identify and resolve common failures in HyperPod clusters, including deployment errors, EFA health checks, lifecycle script failures, and capacity issues. It provides a structured workflow to collect cluster state via AWS APIs and SSM, mapping findings to specific remediation runbooks. The tool emphasizes safe operations by providing suggested commands for users to execute manually, ensuring that IaC-managed environments are not inadvertently drifted.

What it does

Collects cluster state via AWS APIs and SSM for comprehensive diagnostic reporting.
Validates pre-flight requirements including SGs, subnets, IAM, and capacity.
Maps detected failures to specific remediation runbooks for EFA, EKS, SSM, and CloudFormation.
Provides actionable, read-only diagnostic commands to identify root causes.
Offers guidance on distinguishing between manual fixes and IaC-managed configurations.
Generates diagnostic bundles to streamline escalation to AWS Support.

When to use it

When a HyperPod cluster fails during creation or deployment.
When diagnosing EKS access, node replacement, or autoscaler conflicts.
When troubleshooting lifecycle script timeouts or failures.
When verifying capacity availability in specific AZs before cluster creation.
When investigating post-maintenance rollback states or dangling nodes.

Advantages

Reduces manual troubleshooting time by automating the collection of cluster state.
Provides clear, categorized remediation steps for complex HyperPod error signals.
Promotes safe practices by preventing unintended configuration drift in IaC environments.
Standardizes the diagnostic output for efficient escalation to AWS Support.
Includes pre-flight validation to catch configuration errors before cluster deployment.

Usage examples

“Diagnose my existing HyperPod cluster named my-cluster in us-west-2.”: The tool runs the diagnostic script, collects state, and outputs a list of [PASS], [FAIL], or [WARN] items with pointers to specific remediation documentation.
“Validate my network and IAM configuration before creating a new HyperPod cluster.”: The tool performs a pre-flight check using the provided SG IDs and subnet IDs, verifying capacity and permissions to ensure a successful cluster creation.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/awslabs/hyperpod-cluster-debugger/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip hyperpod-cluster-debugger.zip -d .agents/skills

Documentation by AWS Labs

HyperPod Cluster Debugger

Operating policy. Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a Suggested command (run this yourself) block and wait for the customer to run it. Destructive order: investigate → reboot → replace (replace destroys root + secondary volumes; not supported on Slurm controller nodes).

Before any state-changing CLI: ask if it's IaC-managed. HyperPod clusters, SGs, EKS access entries, and IAM are usually provisioned via CloudFormation / CDK / Terraform. If yes, the fix belongs in IaC — running the CLI will drift and the next deploy reverts it. Use the CLI only when IaC is unavailable (locked out, predates IaC, mid-review).

scripts/diagnose-cluster.sh is read-only: it collects state via AWS APIs (and SSM for Slurm controller health) and prints each issue as [FAIL] ... → references/<file>.md § <section>.

Reference	Open when
cluster-diagnostics-detail.md	Per-finding remediation runbook (§ A–L)
cluster-operations.md	Operational deep-dives (EFA SG, EKS access, SSM, Slurm, filesystem)
cloudformation-errors.md	§ H needs the full per-resource CFN error catalog
capacity-planning.md	§ B or `--validate` flags capacity / subnet sizing
lifecycle-scripts.md	§ C points at a specific lifecycle failure
iam-permissions.md	Full IAM policy for the diagnostic

Workflow

Collect HyperPod cluster name (not EKS name), region, exact error string.
Run scripts/diagnose-cluster.sh (or --validate for pre-create).
For every [FAIL] line, Read the referenced section.
Present finding, root cause, and the Suggested-command block verbatim. Wait for customer approval.
Re-run the diagnostic to confirm.

Step 1: Run diagnostics

# Diagnose an existing cluster:
bash scripts/diagnose-cluster.sh --cluster <CLUSTER_NAME_OR_ARN> --region <REGION>

# Pre-flight (no cluster needed) — validates SGs, subnets, IAM, VPC endpoints,
# optionally S3 lifecycle scripts and per-AZ capacity:
bash scripts/diagnose-cluster.sh --validate --region <REGION> \
  --sg-ids <sg-1,sg-2> --subnet-ids <sub-1,sub-2> [--iam-role <role-arn>] \
  [--s3-uri s3://<BUCKET>/path/] [--instance-type ml.p5.48xlarge]

Pass --instance-type when the target instance type is known — enables the per-AZ capacity check (warns if none of the provided subnets are in an AZ that offers that type, which causes insufficient-capacity failures at creation time).

Tags: [PASS] · [FAIL] (counted, has → references/... pointer) · [WARN] · [INFO]. Priorities: P0 blocks operation · P1 degraded · P2 informational.

Step 2: Match signal → section

Error messages / events:

Signal	Section
`"EFA health checks did not run successfully"` (public-doc verbatim signal)	A: EFA Health Checks
Insufficient-capacity or AZ-mismatch failure at creation	B: Capacity & AZ
Lifecycle-script failure or timeout during provisioning	C: Lifecycle Scripts
kubectl auth error (server asks for credentials / no API group list)	D: EKS Access
`InService` but not all instances visible	E: Cluster Provisioning
`"Target is not connected"` / SSM errors	F: SSM Connectivity
Node replacement not happening / `batch-replace` not working	G: Node Replacement
`"Embedded stack failed"` / any CloudFormation error	H: CloudFormation Errors
`UpdateClusterSoftware` failed or cluster in post-maintenance rollback state	J: AMI & Cluster Updates
Dangling / orphaned nodes in EKS vs `list-cluster-nodes`	K: Dangling Nodes & Cleanup
Cluster Autoscaler breaks after HyperPod attached	L: Autoscaler Compatibility
Slow I/O, FSx throughput saturated	cluster-operations.md § 9
Slurm node name → instance ID lookup	I: Utilities

A: EFA Health Checks

SG missing self-reference. Add inbound + outbound self-ref to every SG on the cluster, plus least-privilege egress for the AWS APIs the node needs (HTTPS 443 to S3 / ECR / SageMaker / SSM / STS / CloudWatch Logs — via VPC-endpoint prefix-lists when possible). Full procedure: cluster-diagnostics-detail.md § A.

B: Capacity & AZ

Instance type unavailable in the requested AZ. Verify with describe-instance-type-offerings, then change AZ, use Flexible Training Plans, or request ODCR. Full: § B · strategy: capacity-planning.md.

C: Lifecycle Scripts

Script failed or timed out during provisioning. Read CloudWatch under /aws/sagemaker/Clusters/<name>/<id> — common causes: missing S3 VPC endpoint, IAM gap, CRLF line endings, instance-group name mismatch. Full: § C · layout: lifecycle-scripts.md.

D: EKS Access / kubectl

IAM identity not in EKS access entries. Verify with sts get-caller-identity, create an access entry with admin policy, update kubeconfig. Full: § D.

E: Cluster Provisioning

InService without all instances is expected under Continuous Provisioning — failures surface as events, not cluster errors. For stuck Creating/Updating/Deleting: check CFN nested stacks (§ H), IAM, capacity, events; if stuck Deleting check VPC ENI dependencies. Full: § E.

F: SSM Connectivity

Target is not connected: use sagemaker-cluster:<CLUSTER_ID>_<GROUP>-<INSTANCE_ID> format (not raw EC2 ID), install session-manager-plugin, confirm node Running. Check IAM + VPC endpoints on timeouts. Full: § F.

G: Node Replacement

Auto-repair: confirm NodeRecovery=Automatic, check Health Monitoring Agent (HMA) logs + node labels / Slurm reason, confirm capacity. Manual: reboot first, replace only if reboot fails. Replace requires the cluster to have been patched via UpdateClusterSoftware at least once and cannot target a Slurm controller node. Full: § G.

H: CloudFormation Errors

Embedded stack failed hides the real error. Drill into nested stacks via Events tab (filter Failed) until you reach a non-stack resource. CLI: describe-stack-events --query 'StackEvents[?ResourceStatus==\CREATE_FAILED`]'`. Also covers SLR creation failures and permission-boundary denials. Full: § H · catalog: cloudformation-errors.md.

I: Utilities

Map Slurm node names (ip-10-x-y-z) to HyperPod instance IDs via list-cluster-nodes or on-node /opt/ml/config/resource_config.json. Full: § I.

J: AMI & Cluster Updates

UpdateClusterSoftware fails and rolls back, or the cluster stays in a post-maintenance rollback state. Common causes: lifecycle script incompatible with new AMI, HMA version too old, insufficient rolling-update capacity. If the cluster has active nodes, collect diagnostics and escalate rather than delete-and-recreate. Full: § J.

K: Dangling Nodes & Cleanup

Nodes in kubectl get nodes but not in list-cluster-nodes (ghost EKS nodes), or the inverse (HyperPod nodes that never registered kubelet). Script flags both. Full: § K.

L: Autoscaler Compatibility

Cluster Autoscaler errors on HyperPod provider IDs and breaks autoscaling for all node groups. No officially endorsed workaround — escalate to AWS Support. Karpenter does not conflict with HyperPod nodes by default. Full: § L.

Prerequisites

aws CLI v2.13+ authenticated to the cluster's account
jq, python3, bash 4.2+
kubectl authenticated to the EKS cluster (EKS checks skipped if absent)
session-manager-plugin (Slurm controller health checks only)

IAM policy: references/iam-permissions.md.

Defaults

Region — required: pass --region or set $AWS_DEFAULT_REGION.
Mode — --cluster <NAME> (diagnose) or --validate (pre-create).
Event window — up to 500 most recent events (5 × 100, paginated).
Colors — auto-disabled on non-TTY; --no-color to force off.

Error handling

Failure	Script	Tell the customer
`aws sts get-caller-identity` fails	Exit 1	"Fix AWS credentials and rerun."
Cluster not found	Exit 1 after listing region's clusters	"Confirm HyperPod cluster name (not EKS) and region."
`sagemaker:` / `ec2:` / `eks:` / `logs:` denied	Warn, add `Missing IAM permission for <API>`, continue	"Grant the listed IAM action and rerun."
`kubectl` absent or unauthenticated	Skip EKS checks (access entries, add-ons, aws-auth, nodes)	"Install/authenticate kubectl."
`session-manager-plugin` absent (Slurm)	Skip Slurm controller probe	"Install session-manager-plugin."
SSM throttled / times out (180s)	Retry with backoff; warn and continue if still failing	"Rerun later — script is idempotent."
CloudWatch log group not found	Skip CloudWatch check	"CloudWatch not configured on this cluster."

Exit codes: 0 no critical failures · 1 one or more critical failures (cluster not found, fatal prerequisite missing, or any [FAIL] in diagnose or --validate mode). [WARN] lines do not affect the exit code.

Skill delegation

Need	Use
Shell on nodes	`hyperpod-ssm`
Version comparison across nodes	`hyperpod-version-checker`

Escalate to AWS Support

Escalate when:

EFA health checks fail despite correct SG rules.
Capacity errors persist despite a valid Flexible Training Plan / ODCR.
Node replacement fails repeatedly without clear events / log signal.
Cluster stuck in a non-terminal state (Creating, Updating, or a post-maintenance rollback state) for an extended period.
CloudFormation root-cause is an internal service error.

Before opening the case

Run these commands and attach the output. Goal: AWS Support has everything at case open.

# 1. Cluster identity + status (confirms region, ARN, orchestrator, instance groups)
aws sagemaker describe-cluster --cluster-name <CLUSTER> --region <REGION>

# 2. Full cluster-level diagnostic bundle
bash scripts/diagnose-cluster.sh --cluster <CLUSTER> --region <REGION> > diag.txt

# 3. Per-node log/config bundle to S3 (delegates to hyperpod-issue-report skill)
#    See skills/hyperpod-issue-report/SKILL.md for the exact invocation.

Include in the case

Cluster name + ARN (or ClusterId suffix) and AWS region
ClusterStatus + FailureMessage from describe-cluster
Timestamp window (UTC start / end) of the failure
Exact error strings observed (copy verbatim from events / logs / console)
Affected instance IDs / NodeLogicalIds / instance group names
diag.txt from step 2 above
S3 URI of the hyperpod-issue-report bundle from step 3

Files

ATTRIBUTION.mdAdded by AutoGineer934 B
LICENSELicence file added by AutoGineer9.9 KiB
NOTICELicence file added by AutoGineer89 B
SKILL.md14.1 KiB
references/capacity-planning.md4.5 KiB
references/cloudformation-errors.md5.8 KiB
references/cluster-diagnostics-detail.md23.3 KiB
references/cluster-operations.md11.5 KiB
references/iam-permissions.md1.2 KiB
references/lifecycle-scripts.md5.1 KiB
scripts/diagnose-cluster.shScript69.0 KiB

references/lifecycle-scripts.md

Lifecycle Script Reference

Companion to SKILL.md § C and cluster-operations.md § 3. Lifecycle scripts run on each node during provisioning. A failure here blocks the node — and often the entire cluster — from reaching InService.

Layout

Default AWS-published lifecycle scripts (commonly called "base-config") handle provisioning for Slurm and EKS. Before deep debugging, compare the customer's in-use scripts against the latest published version — upstream fixes often resolve the failure.

Slurm entry point (typical base-config layout)

on_create.sh → lifecycle_script.py for orchestration (detects node type from /opt/ml/config/resource_config.json and runs per-type steps). Controller nodes provision first; compute / login nodes wait for the controller to write slurm.conf to shared storage. Customer-forked pipelines may differ — read on_create.sh on the affected node to confirm.

Controller failure cascades to all compute nodes — if the controller's lifecycle script fails, compute nodes cannot find slurm.conf and also fail.

EKS entry point

on_create.sh → on_create_main.sh (configures containerd storage, kubelet, FSx client, EFA).

S3 URI validation

SourceS3Uri starts with s3://
OnCreate filename matches an S3 key in that prefix
Execution role has s3:GetObject and s3:ListBucket on the bucket

Common errors

S3 access

Timeout reaching S3 from the lifecycle script (e.g. Connect timeout on endpoint URL: s3://...) → no S3 VPC endpoint; node cannot reach S3 from a private subnet.

Fix — add an S3 Gateway endpoint

Customer-run. Gateway endpoint type is free; Interface endpoints are billed per-hour.

aws ec2 create-vpc-endpoint \
  --vpc-id <VPC_ID> \
  --service-name com.amazonaws.<REGION>.s3 \
  --route-table-ids <ROUTE_TABLE_ID> \
  --vpc-endpoint-type Gateway

Caution: routes S3 traffic for every resource using the listed route tables through the VPC endpoint. Can break workloads that rely on going to S3 via public DNS + NAT with custom endpoint policies. Review the VPC's default endpoint policy (or set --policy-document) before creating.

AccessDenied / 403 Forbidden on GetObject — add s3:GetObject + s3:ListBucket on the lifecycle bucket to the execution role.

Script execution

Symptom	Cause	Fix
`No such file or directory` on entry script	`OnCreate` name doesn't match S3 key	`aws s3 ls s3://<BUCKET>/ \| grep on_create` to verify
`\r: command not found` / CRLF terminators	Edited on Windows	`dos2unix on_create.sh` or `sed -i 's/\r$//' on_create.sh`
Script hangs (lifecycle timeout)	Blocking op, infinite loop, waiting for absent resource	Add `set -euo pipefail`, add network timeouts
`provisioning_parameters.json` KeyError	Instance group name mismatch	`InstanceGroupName` in API call must match group key in JSON

Slurm

Compute nodes fail because slurm.conf not found — controller's lifecycle failed. Fix the controller first.

slurmctld: error ... — check /var/log/slurmctld.log on controller via SSM. Common causes: wrong SlurmctldHost, partition/node definition errors, missing MUNGE key.

FSx

mount.lustre: ... Connection timed out — FSx in different VPC/AZ, or SG doesn't allow Lustre traffic. FSx and HyperPod nodes must share a VPC; SG must allow TCP 988 and 1018-1023 between nodes and FSx. Verify FSx is AVAILABLE.

Reading logs

CloudWatch (from workstation)

CLUSTER_ID=$(aws sagemaker describe-cluster --cluster-name <NAME> --region <R> \
  --query 'ClusterArn' --output text | cut -d/ -f2)
LOG_GROUP="/aws/sagemaker/Clusters/<CLUSTER_NAME>/${CLUSTER_ID}"

# List lifecycle log streams:
aws logs describe-log-streams \
  --log-group-name "$LOG_GROUP" --region <R> \
  --query 'logStreams[?starts_with(logStreamName,`LifecycleConfig`)].{Stream:logStreamName,LastEvent:lastEventTimestamp}' \
  --output table

# Read a specific stream:
aws logs get-log-events \
  --log-group-name "$LOG_GROUP" \
  --log-stream-name "LifecycleConfig/<GROUP>/<INSTANCE_ID>" \
  --region <R> --limit 100 \
  --query 'events[*].message' --output text

On-node (via SSM)

cat /var/log/provision/provisioning.log      # full provisioning log
cat /opt/ml/config/resource_config.json      # node topology
cat /opt/slurm/etc/slurm.conf                # Slurm config (if generated)
cat /opt/ml/metadata/resource-metadata.json  # node metadata

Test locally

file on_create.sh         # must not say "with CRLF line terminators"
head -1 on_create.sh      # must start with #!/bin/bash
bash -n on_create.sh      # syntax check
shellcheck on_create.sh   # optional lint

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs a command-line tool, Needs Python, Needs Node.js

Security

What our scanner noticed· Low risk

Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

AWS Labs

Publisher

AWS Labs (Official)

Source

awslabs/agent-plugins · plugins/sagemaker-ai/skills/hyperpod-cluster-debugger · commit fbefb37

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
NOTICE (added by AutoGineer from NOTICE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by AWS Labs.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download hyperpod-cluster-debugger.zip