Hyperpod Cluster Debugger

AWS LabsOfficialPublished by AWS Labs

Skill
Review and audit
Cloud and DevOps
Apache-2.0
Low risk
Has scripts

From the author

Diagnose and remediate cluster-wide HyperPod (EKS or Slurm) problems — creation / deployment failures (CloudFormation, EFA health check, lifecycle scripts, capacity), EKS access, node replacement, CloudFormation nested-stack errors, post-maintenance rollback state, dangling nodes, autoscaler conflicts. Includes `--validate` pre-flight. Read-only.

— the description by AWS Labs

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The HyperPod Cluster Debugger is a read-only diagnostic tool designed to identify and resolve common failures in HyperPod clusters, including deployment errors, EFA health checks, lifecycle script failures, and capacity issues. It provides a structured workflow to collect cluster state via AWS APIs and SSM, mapping findings to specific remediation runbooks. The tool emphasizes safe operations by providing suggested commands for users to execute manually, ensuring that IaC-managed environments are not inadvertently drifted.

What it does

Collects cluster state via AWS APIs and SSM for comprehensive diagnostic reporting.
Validates pre-flight requirements including SGs, subnets, IAM, and capacity.
Maps detected failures to specific remediation runbooks for EFA, EKS, SSM, and CloudFormation.
Provides actionable, read-only diagnostic commands to identify root causes.
Offers guidance on distinguishing between manual fixes and IaC-managed configurations.
Generates diagnostic bundles to streamline escalation to AWS Support.

When to use it

When a HyperPod cluster fails during creation or deployment.
When diagnosing EKS access, node replacement, or autoscaler conflicts.
When troubleshooting lifecycle script timeouts or failures.
When verifying capacity availability in specific AZs before cluster creation.
When investigating post-maintenance rollback states or dangling nodes.

Advantages

Reduces manual troubleshooting time by automating the collection of cluster state.
Provides clear, categorized remediation steps for complex HyperPod error signals.
Promotes safe practices by preventing unintended configuration drift in IaC environments.
Standardizes the diagnostic output for efficient escalation to AWS Support.
Includes pre-flight validation to catch configuration errors before cluster deployment.

Usage examples

“Diagnose my existing HyperPod cluster named my-cluster in us-west-2.”: The tool runs the diagnostic script, collects state, and outputs a list of [PASS], [FAIL], or [WARN] items with pointers to specific remediation documentation.
“Validate my network and IAM configuration before creating a new HyperPod cluster.”: The tool performs a pre-flight check using the provided SG IDs and subnet IDs, verifying capacity and permissions to ensure a successful cluster creation.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/awslabs/hyperpod-cluster-debugger/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip hyperpod-cluster-debugger.zip -d .agents/skills

Documentation by AWS Labs

HyperPod Cluster Debugger

Operating policy. Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a Suggested command (run this yourself) block and wait for the customer to run it. Destructive order: investigate → reboot → replace (replace destroys root + secondary volumes; not supported on Slurm controller nodes).

Before any state-changing CLI: ask if it's IaC-managed. HyperPod clusters, SGs, EKS access entries, and IAM are usually provisioned via CloudFormation / CDK / Terraform. If yes, the fix belongs in IaC — running the CLI will drift and the next deploy reverts it. Use the CLI only when IaC is unavailable (locked out, predates IaC, mid-review).

scripts/diagnose-cluster.sh is read-only: it collects state via AWS APIs (and SSM for Slurm controller health) and prints each issue as [FAIL] ... → references/<file>.md § <section>.

Reference	Open when
cluster-diagnostics-detail.md	Per-finding remediation runbook (§ A–L)
cluster-operations.md	Operational deep-dives (EFA SG, EKS access, SSM, Slurm, filesystem)
cloudformation-errors.md	§ H needs the full per-resource CFN error catalog
capacity-planning.md	§ B or `--validate` flags capacity / subnet sizing
lifecycle-scripts.md	§ C points at a specific lifecycle failure
iam-permissions.md	Full IAM policy for the diagnostic

Workflow

Collect HyperPod cluster name (not EKS name), region, exact error string.
Run scripts/diagnose-cluster.sh (or --validate for pre-create).
For every [FAIL] line, Read the referenced section.
Present finding, root cause, and the Suggested-command block verbatim. Wait for customer approval.
Re-run the diagnostic to confirm.

Step 1: Run diagnostics

# Diagnose an existing cluster:
bash scripts/diagnose-cluster.sh --cluster <CLUSTER_NAME_OR_ARN> --region <REGION>

# Pre-flight (no cluster needed) — validates SGs, subnets, IAM, VPC endpoints,
# optionally S3 lifecycle scripts and per-AZ capacity:
bash scripts/diagnose-cluster.sh --validate --region <REGION> \
  --sg-ids <sg-1,sg-2> --subnet-ids <sub-1,sub-2> [--iam-role <role-arn>] \
  [--s3-uri s3://<BUCKET>/path/] [--instance-type ml.p5.48xlarge]

Pass --instance-type when the target instance type is known — enables the per-AZ capacity check (warns if none of the provided subnets are in an AZ that offers that type, which causes insufficient-capacity failures at creation time).

Tags: [PASS] · [FAIL] (counted, has → references/... pointer) · [WARN] · [INFO]. Priorities: P0 blocks operation · P1 degraded · P2 informational.

Step 2: Match signal → section

Error messages / events:

Signal	Section
`"EFA health checks did not run successfully"` (public-doc verbatim signal)	A: EFA Health Checks
Insufficient-capacity or AZ-mismatch failure at creation	B: Capacity & AZ
Lifecycle-script failure or timeout during provisioning	C: Lifecycle Scripts
kubectl auth error (server asks for credentials / no API group list)	D: EKS Access
`InService` but not all instances visible	E: Cluster Provisioning
`"Target is not connected"` / SSM errors	F: SSM Connectivity
Node replacement not happening / `batch-replace` not working	G: Node Replacement
`"Embedded stack failed"` / any CloudFormation error	H: CloudFormation Errors
`UpdateClusterSoftware` failed or cluster in post-maintenance rollback state	J: AMI & Cluster Updates
Dangling / orphaned nodes in EKS vs `list-cluster-nodes`	K: Dangling Nodes & Cleanup
Cluster Autoscaler breaks after HyperPod attached	L: Autoscaler Compatibility
Slow I/O, FSx throughput saturated	cluster-operations.md § 9
Slurm node name → instance ID lookup	I: Utilities

A: EFA Health Checks

SG missing self-reference. Add inbound + outbound self-ref to every SG on the cluster, plus least-privilege egress for the AWS APIs the node needs (HTTPS 443 to S3 / ECR / SageMaker / SSM / STS / CloudWatch Logs — via VPC-endpoint prefix-lists when possible). Full procedure: cluster-diagnostics-detail.md § A.

B: Capacity & AZ

Instance type unavailable in the requested AZ. Verify with describe-instance-type-offerings, then change AZ, use Flexible Training Plans, or request ODCR. Full: § B · strategy: capacity-planning.md.

C: Lifecycle Scripts

Script failed or timed out during provisioning. Read CloudWatch under /aws/sagemaker/Clusters/<name>/<id> — common causes: missing S3 VPC endpoint, IAM gap, CRLF line endings, instance-group name mismatch. Full: § C · layout: lifecycle-scripts.md.

D: EKS Access / kubectl

IAM identity not in EKS access entries. Verify with sts get-caller-identity, create an access entry with admin policy, update kubeconfig. Full: § D.

E: Cluster Provisioning

InService without all instances is expected under Continuous Provisioning — failures surface as events, not cluster errors. For stuck Creating/Updating/Deleting: check CFN nested stacks (§ H), IAM, capacity, events; if stuck Deleting check VPC ENI dependencies. Full: § E.

F: SSM Connectivity

Target is not connected: use sagemaker-cluster:<CLUSTER_ID>_<GROUP>-<INSTANCE_ID> format (not raw EC2 ID), install session-manager-plugin, confirm node Running. Check IAM + VPC endpoints on timeouts. Full: § F.

G: Node Replacement

Auto-repair: confirm NodeRecovery=Automatic, check Health Monitoring Agent (HMA) logs + node labels / Slurm reason, confirm capacity. Manual: reboot first, replace only if reboot fails. Replace requires the cluster to have been patched via UpdateClusterSoftware at least once and cannot target a Slurm controller node. Full: § G.

H: CloudFormation Errors

Embedded stack failed hides the real error. Drill into nested stacks via Events tab (filter Failed) until you reach a non-stack resource. CLI: describe-stack-events --query 'StackEvents[?ResourceStatus==\CREATE_FAILED`]'`. Also covers SLR creation failures and permission-boundary denials. Full: § H · catalog: cloudformation-errors.md.

I: Utilities

Map Slurm node names (ip-10-x-y-z) to HyperPod instance IDs via list-cluster-nodes or on-node /opt/ml/config/resource_config.json. Full: § I.

J: AMI & Cluster Updates

UpdateClusterSoftware fails and rolls back, or the cluster stays in a post-maintenance rollback state. Common causes: lifecycle script incompatible with new AMI, HMA version too old, insufficient rolling-update capacity. If the cluster has active nodes, collect diagnostics and escalate rather than delete-and-recreate. Full: § J.

K: Dangling Nodes & Cleanup

Nodes in kubectl get nodes but not in list-cluster-nodes (ghost EKS nodes), or the inverse (HyperPod nodes that never registered kubelet). Script flags both. Full: § K.

L: Autoscaler Compatibility

Cluster Autoscaler errors on HyperPod provider IDs and breaks autoscaling for all node groups. No officially endorsed workaround — escalate to AWS Support. Karpenter does not conflict with HyperPod nodes by default. Full: § L.

Prerequisites

aws CLI v2.13+ authenticated to the cluster's account
jq, python3, bash 4.2+
kubectl authenticated to the EKS cluster (EKS checks skipped if absent)
session-manager-plugin (Slurm controller health checks only)

IAM policy: references/iam-permissions.md.

Defaults

Region — required: pass --region or set $AWS_DEFAULT_REGION.
Mode — --cluster <NAME> (diagnose) or --validate (pre-create).
Event window — up to 500 most recent events (5 × 100, paginated).
Colors — auto-disabled on non-TTY; --no-color to force off.

Error handling

Failure	Script	Tell the customer
`aws sts get-caller-identity` fails	Exit 1	"Fix AWS credentials and rerun."
Cluster not found	Exit 1 after listing region's clusters	"Confirm HyperPod cluster name (not EKS) and region."
`sagemaker:` / `ec2:` / `eks:` / `logs:` denied	Warn, add `Missing IAM permission for <API>`, continue	"Grant the listed IAM action and rerun."
`kubectl` absent or unauthenticated	Skip EKS checks (access entries, add-ons, aws-auth, nodes)	"Install/authenticate kubectl."
`session-manager-plugin` absent (Slurm)	Skip Slurm controller probe	"Install session-manager-plugin."
SSM throttled / times out (180s)	Retry with backoff; warn and continue if still failing	"Rerun later — script is idempotent."
CloudWatch log group not found	Skip CloudWatch check	"CloudWatch not configured on this cluster."

Exit codes: 0 no critical failures · 1 one or more critical failures (cluster not found, fatal prerequisite missing, or any [FAIL] in diagnose or --validate mode). [WARN] lines do not affect the exit code.

Skill delegation

Need	Use
Shell on nodes	`hyperpod-ssm`
Version comparison across nodes	`hyperpod-version-checker`

Escalate to AWS Support

Escalate when:

EFA health checks fail despite correct SG rules.
Capacity errors persist despite a valid Flexible Training Plan / ODCR.
Node replacement fails repeatedly without clear events / log signal.
Cluster stuck in a non-terminal state (Creating, Updating, or a post-maintenance rollback state) for an extended period.
CloudFormation root-cause is an internal service error.

Before opening the case

Run these commands and attach the output. Goal: AWS Support has everything at case open.

# 1. Cluster identity + status (confirms region, ARN, orchestrator, instance groups)
aws sagemaker describe-cluster --cluster-name <CLUSTER> --region <REGION>

# 2. Full cluster-level diagnostic bundle
bash scripts/diagnose-cluster.sh --cluster <CLUSTER> --region <REGION> > diag.txt

# 3. Per-node log/config bundle to S3 (delegates to hyperpod-issue-report skill)
#    See skills/hyperpod-issue-report/SKILL.md for the exact invocation.

Include in the case

Cluster name + ARN (or ClusterId suffix) and AWS region
ClusterStatus + FailureMessage from describe-cluster
Timestamp window (UTC start / end) of the failure
Exact error strings observed (copy verbatim from events / logs / console)
Affected instance IDs / NodeLogicalIds / instance group names
diag.txt from step 2 above
S3 URI of the hyperpod-issue-report bundle from step 3

Files

ATTRIBUTION.mdAdded by AutoGineer934 B
LICENSELicence file added by AutoGineer9.9 KiB
NOTICELicence file added by AutoGineer89 B
SKILL.md14.1 KiB
references/capacity-planning.md4.5 KiB
references/cloudformation-errors.md5.8 KiB
references/cluster-diagnostics-detail.md23.3 KiB
references/cluster-operations.md11.5 KiB
references/iam-permissions.md1.2 KiB
references/lifecycle-scripts.md5.1 KiB
scripts/diagnose-cluster.shScript69.0 KiB

references/cluster-operations.md

Cluster Operations Reference

Operational deep-dives for the hyperpod-cluster-debugger skill. See SKILL.md for the workflow entry points.

1. EFA Security Group (multi-SG clusters)

The EFA health check runs during instance provisioning, before lifecycle scripts execute. If it fails, lifecycle scripts never run and CloudWatch lifecycle logs are empty — the cluster event will say "EFA health checks did not run successfully".

When a cluster uses multiple security groups, all SGs must have the self-referencing rules. Check each:

for SG in $(aws sagemaker describe-cluster --cluster-name <C> --region <R> \
  --query 'VpcConfig.SecurityGroupIds[]' --output text); do
  echo "=== $SG ==="
  aws ec2 describe-security-groups --group-ids $SG --region <R> \
    --query 'SecurityGroups[0].{In:IpPermissions,Out:IpPermissionsEgress}'
done

Fix commands are in cluster-diagnostics-detail.md § A.

2. Capacity

See capacity-planning.md.

3. Lifecycle scripts

See lifecycle-scripts.md.

4. EKS access control

Authentication modes

Access entries require API or API_AND_CONFIG_MAP. If the cluster is on CONFIG_MAP only, aws eks list-access-entries returns nothing useful; verify the mode with describe-cluster --query 'cluster.accessConfig.authenticationMode' and consult the EKS access-entries documentation for the switching procedure.

Access policies (EKS-native)

Policy	Scope	Use case
`AmazonEKSClusterAdminPolicy`	Cluster-wide	Full admin (debugging)
`AmazonEKSAdminPolicy`	Namespace	Namespace admin (multi-tenant)
`AmazonEKSEditPolicy`	Namespace	Read/write workloads
`AmazonEKSViewPolicy`	Namespace	Read-only

Troubleshooting kubectl auth

aws sts get-caller-identity            # your identity
kubectl config current-context         # which cluster kubeconfig points at
kubectl cluster-info                   # API server reachable?

If using an assumed role: access entries reference the IAM role ARN, not the assumed-role session ARN.

Role ARN: arn:aws:iam::123456789012:role/MyRole
Session ARN: arn:aws:sts::123456789012:assumed-role/MyRole/session-name

5. Continuous Provisioning (EKS only)

The cluster transitions to InService once the control plane is ready; instances are created asynchronously and failures are reported as events, not cluster failures. Failed instances can be individually replaced.

# Poll instance creation:
watch -n 30 "aws sagemaker describe-cluster --cluster-name <C> --region <R> \
  --query 'InstanceGroups[*].{Name:InstanceGroupName,Current:CurrentCount,Target:InstanceCount}' --output table"

# Poll cluster events:
watch -n 30 "aws sagemaker list-cluster-events --cluster-name <C> --region <R> \
  --query 'ClusterEventSummaries[0:5].{Time:EventTime,Msg:Message}' --output table"

Nodes in `list-cluster-nodes` but not in `kubectl get nodes`

Check lifecycle script logs — it registers the node with EKS
Verify the EKS endpoint is reachable from worker subnets
Check kubelet on the node via SSM
Verify the node's IAM role has AmazonEKSWorkerNodePolicy

Cluster events are emitted for HyperPod EKS. For HyperPod Slurm, events are not yet surfaced — use CloudWatch logs and list-cluster-nodes instead.

6. SSM target format

See the hyperpod-ssm skill's SKILL.md for the target format (sagemaker-cluster:<CLUSTER_ID>_<GROUP>-<INSTANCE_ID>), prerequisites, and manual-command examples. HyperPod requires start-session — not send-command against raw instance IDs.

7. Node replacement (batch APIs)

Full Suggested-command blocks with preconditions + blast radius are in cluster-diagnostics-detail.md § G.2. Summary:

Cluster must be InService
Batch limit: 1-25 node IDs per call for both APIs
batch-replace-cluster-nodes destroys root + secondary volumes and is not supported on Slurm controller nodes — back up first
Monitor with list-cluster-events after the call
Prefer batch APIs over legacy paths (Slurm reason fields, K8s labels)

8. Slurm — controller operations

The per-node Slurm operations (resuming a single node, fixing a single Slurm state) live in the hyperpod-node-debugger skill. This section is controller-level only.

Diagnose controller health (via SSM on the controller)

scontrol ping                                     # slurmctld responsive?
systemctl status slurmctld                        # service state
systemctl is-active munge && systemctl status munge   # auth daemon (required)
systemctl is-active slurmdbd                      # accounting DB (if used)

slurmctld down

journalctl -u slurmctld --since "1 hour ago" --no-pager | tail -100
tail -200 /var/log/slurm/slurmctld.log

Common causes and fixes:

OOM on controller: restart the service; investigate the job scale that triggered it.
Munge auth failure (Invalid authentication credential): munge key mismatch. Re-sync /etc/munge/munge.key to every node, restart munge + slurmctld.
Accounting DB unreachable (slurmdbd + MariaDB / RDS): check network path and credentials. slurmctld won't start if accounting is required but unreachable.
Config error in slurm.conf: slurmctld -D -vvv (foreground) prints the parse error. Roll back to the last known-good config.

Fix — restart slurmctld

Customer-run on the Slurm controller (via SSM) after the root cause is diagnosed. Running jobs, pending queue, and node states are preserved; caches and resource calculations reset. Brief scheduler pause.

sudo systemctl restart slurmctld
scontrol ping   # expect "Slurmctld(primary) is UP"

If slurm.conf is broken the service will not return — roll back the config first.

munge inactive

Diagnose:

systemctl status munge
ls -l /etc/munge/munge.key   # expect munge:munge, mode 0400
sudo md5sum /etc/munge/munge.key   # must match on controller + every compute node

Fix — start munge

Customer-run. Safe when munge is inactive and the key file is present and matches other nodes.

sudo systemctl start munge

If md5 mismatches another node, jobs will still fail auth — re-distribute the controller's key cluster-wide and restart munge on every node.

Stuck jobs (PENDING / COMPLETING / CONFIGURING)

squeue -o "%i %j %T %R %N" --noheader | grep -iE "COMPLETING|CONFIGURING|PENDING"
scontrol show job <JOBID>
scancel <JOBID>               # if safe to cancel

Common reason codes:

(Resources) — waiting for free nodes. Check sinfo -o "%P %a %l %D %T".
(AssocGrpNodeLimit) / (QOSMaxJobsPerUserLimit) — quota-related. sacctmgr show assoc.
(NodeDown) — partition has no healthy nodes. Use the hyperpod-node-debugger skill.
(BeginTime) — scheduled for a future start time.

Restarting slurmctld to clear stuck-job symptoms uses the same Suggested-command block as above (§ slurmctld down).

Verify after remediation

scontrol ping                                   # "Slurmctld(primary) is UP"
sinfo                                            # no "down*" or "drain" states
systemctl is-active slurmctld munge
scontrol show config | grep StateSaveLocation   # must be persistent + writable

9. Filesystem performance

Symptom: training bottlenecked by data loading, checkpoint save / load, or slow executable / script loading.

Diagnose on the node

mount | grep -E "fsx|nfs|lustre|ebs|nvme"
df -hT
iostat -x 1 5                 # per-device throughput / IOPS / utilization

# FSx for Lustre:
lfs df -h                     # per-OST utilization (uneven = hotspot)
lfs getstripe <path>          # striping config; wider = more parallelism

# FSx for OpenZFS / NFS:
nfsstat -m                    # per-mount retransmissions / wait times
nfsiostat 5                   # ops/s, throughput, RTT

# EBS:
lsblk -o NAME,TYPE,SIZE,MOUNTPOINT

CloudWatch (from your workstation)

# FSx for Lustre throughput saturation:
aws cloudwatch get-metric-statistics \
  --namespace AWS/FSx --metric-name DataReadBytes \
  --dimensions Name=FileSystemId,Value=<FSxId> \
  --statistics Sum --period 300 \
  --start-time "$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --end-time   "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --region <REGION>
# Also: DataWriteBytes, FreeDataStorageCapacity, MetadataOperations

# EBS throughput / IOPS:
aws cloudwatch get-metric-statistics \
  --namespace AWS/EBS --metric-name VolumeReadOps \
  --dimensions Name=VolumeId,Value=<vol-id> \
  --statistics Sum --period 60 \
  --start-time "$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --end-time   "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --region <REGION>
# Also: VolumeWriteOps, VolumeReadBytes, VolumeWriteBytes, BurstBalance

Interpret

Signal	Interpretation	Action
FSx Lustre `DataReadBytes` sustained at the ceiling	Throughput ceiling hit	Increase throughput-per-TiB or grow storage (throughput scales with size)
FSx Lustre metadata ops saturated	Small-file workload on Lustre	Move small-file traffic to FSx for OpenZFS; keep Lustre for large sequential I/O
FSx OpenZFS `TotalIOps` near provisioned IOPS	IOPS ceiling hit	Increase provisioned IOPS
EBS `BurstBalance` draining to 0 on `gp2`	Baseline IOPS insufficient	Migrate to `gp3` or `io2` with provisioned IOPS / throughput
`iostat %util` > 90% on a mount device	Local device saturated	If NVMe instance store: at hardware ceiling, change data layout
Slow only at checkpoint time	Write amplification (many small files)	Consolidate checkpoints; rank-0 writer patterns

Choose the right filesystem

Workload	Best fit
Large sequential reads (datasets >> 1 MiB), many-reader training	FSx for Lustre
Small-file / metadata-heavy / mixed random I/O	FSx for OpenZFS
Single-instance scratch	EBS `gp3` or `io2`
Highest per-GPU throughput, ephemeral	NVMe instance store (`/opt/dlami/nvme`)

For HyperPod Slurm, the default lifecycle script supports FSx for OpenZFS for /home — evaluate it if home is on Lustre and you see metadata-op saturation.

Verify after remediation

CloudWatch: throughput / IOPS climbs past the old flat-line
Training step time drops; data-loading fraction of step time drops
iostat %util stays below 80% under sustained load

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs a command-line tool, Needs Python, Needs Node.js

Security

What our scanner noticed· Low risk

Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

AWS Labs

Publisher

AWS Labs (Official)

Source

awslabs/agent-plugins · plugins/sagemaker-ai/skills/hyperpod-cluster-debugger · commit fbefb37

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
NOTICE (added by AutoGineer from NOTICE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by AWS Labs.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download hyperpod-cluster-debugger.zip

From the author

AutoGineer summary

What it does

When to use it

Advantages

Usage examples

Install

Documentation by AWS Labs

HyperPod Cluster Debugger

Workflow

Step 1: Run diagnostics

Step 2: Match signal → section

A: EFA Health Checks

B: Capacity & AZ

C: Lifecycle Scripts

D: EKS Access / kubectl

E: Cluster Provisioning

F: SSM Connectivity

G: Node Replacement

H: CloudFormation Errors

I: Utilities

J: AMI & Cluster Updates

K: Dangling Nodes & Cleanup

L: Autoscaler Compatibility

Prerequisites

Defaults

Error handling

Skill delegation

Escalate to AWS Support

Before opening the case

Include in the case

Files

references/cluster-operations.md

Cluster Operations Reference

1. EFA Security Group (multi-SG clusters)

2. Capacity

3. Lifecycle scripts

4. EKS access control

Authentication modes

Access policies (EKS-native)

Troubleshooting kubectl auth

5. Continuous Provisioning (EKS only)

Nodes in list-cluster-nodes but not in kubectl get nodes

6. SSM target format

7. Node replacement (batch APIs)

8. Slurm — controller operations

Diagnose controller health (via SSM on the controller)

slurmctld down

Fix — restart slurmctld

munge inactive

Fix — start munge

Stuck jobs (PENDING / COMPLETING / CONFIGURING)

Verify after remediation

9. Filesystem performance

Diagnose on the node

CloudWatch (from your workstation)

Interpret

Choose the right filesystem

Verify after remediation

Requirements and compatibility

Security

Attribution and licence

Related skills

Modal

Cloud Infrastructure Network Engineer

Deployment Validation Cloud Architect

Airflow Dag Patterns

Cost Optimization

Hybrid Cloud Networking

Download ZIP

Nodes in `list-cluster-nodes` but not in `kubectl get nodes`