Hyperpod Issue Report

Collection Details

What Gets Collected

Common (Both EKS and Slurm)

• nvidia_smi.txt — GPU status, utilization, memory, temperature

• resource_config.json — HyperPod resource config from /opt/ml/config/resource_config.json

• cluster_logs/ — Contents of /var/log/aws/clusters/

• systemd_services.txt — All systemd service statuses

• disk_usage.txt — df output

• hostname.txt, instance_group.txt, instance_id.txt, cluster_type.txt, timestamp.txt

EKS-Specific (Per-Node)

• containerd_status.txt — systemctl status containerd

• kubelet_status.txt — systemctl status kubelet

• eks-log-collector-output.txt — EKS log collector execution log

• eks-logs/ — EKS log collector output subdirectories:

  • cni/ — CNI plugin logs and config

  • containerd/ — Runtime logs, config, version, images, containers, tasks, plugins

  • docker/ — Docker logs (if present)

  • gpu/ — GPU diagnostics

  • ipamd/ — AWS VPC CNI IPAMD logs

  • kernel/ — dmesg output, uname info

  • kubelet/ — Kubelet logs and config

  • modinfo/ — Kernel module info (lustre, ip_vs, etc.)

  • networking/ — Network config, iptables, routes, interfaces

  • nodeadm/ — Node administration logs

  • sandbox-image/ — Sandbox image info

  • storage/ — Mounts, inodes, lsblk, LVM, fstab, XFS, pod local storage

  • sysctls/ — Kernel parameters

  • system/ — Services, systemd-analyze, top, ps, netstat, CPU/IO throttling

  • var_log/ — System logs from /var/log

EKS-Specific (kubectl — Collected Locally)

Packaged as kubectl_resources.tar.gz, collected from the local machine (not from nodes).

High Priority:

• nodes_describe.txt — Detailed node descriptions (capacity, conditions, running pods)

• pods_all_namespaces.txt / pods_describe_all_namespaces.txt — All pods with details

• events_all_namespaces.txt — Cluster events sorted by timestamp

• pvcs_all_namespaces.txt / pvcs_describe_all_namespaces.txt — PersistentVolumeClaims

• services_all_namespaces.txt / services_describe_all_namespaces.txt — Network endpoints

Medium Priority:

• deployments_all_namespaces.txt, statefulsets_all_namespaces.txt, daemonsets_all_namespaces.txt

• configmaps_all_namespaces.txt, secrets_all_namespaces.txt (metadata only)

• resourcequotas_all_namespaces.txt, networkpolicies_all_namespaces.txt

Slurm-Specific

• sinfo.txt — Node and partition information

• sinfo_R.txt — Reasons for node down/drain states

• slurmctld_status.txt — Slurm controller daemon status

• slurmd_status.txt — Slurm compute node daemon status

• opt_slurm_etc/ — Slurm configuration from /opt/slurm/etc/

• nvidia-bug-report.log.gz — NVIDIA bug report (compressed)

• syslog, kern.log — System logs

• dmesg_T.txt — Kernel ring buffer with timestamps

• var_log_slurm/ — Slurm logs from /var/log/slurm/

Report Output Structure

s3://bucket/prefix/cluster-name/YYYYMMDD_HHMMSS/
├── collector_script.sh
├── summary.json
├── kubectl_resources.tar.gz      # EKS only
└── instances/
    ├── worker1_i-abc123.tar.gz
    └── worker2_i-abc124.tar.gz

Tarball filename format: {instance-group}_{instance-id}.tar.gz

Summary JSON Format

{
  "cluster_name": "my-cluster",
  "cluster_id": "abc123",
  "report_id": "20260126_143022",
  "timestamp": "2026-01-26T14:30:22.123456",
  "total_nodes": 8,
  "successful": 7,
  "failed": 1,
  "results": [
    {
      "InstanceId": "i-0123456789abcdef0",
      "NodeGroup": "worker-group",
      "Success": true,
      "Output": "...",
      "ElapsedTime": 45.2
    }
  ]
}