AutoGineer
Generated banner for Aks Gpu Inference
Generated logo for Aks Gpu Inference

Aks Gpu Inference

MicrosoftOfficialCreated by Microsoft

  • Skill
  • Review and audit
  • Cloud and DevOps
  • MIT
  • No issues found
Download ZIP

From the author

Diagnose Day-2 AKS GPU and KAITO incidents using profile-aware, read-only evidence. WHEN: 'Insufficient nvidia.com/gpu', GPU pod Pending, model-load OOM, DCGM/VRAM, KAITO Workspace not ready, or GPU autoscaling. DO NOT USE FOR: setup (airunway-aks-setup), non-GPU incidents (aks-troubleshooting), standalone VM quota (azure-quotas), or generic cost (cost-analysis or cost-optimization from the optional azure-cost plugin).
— the description by Microsoft

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured workflow for diagnosing operational issues in Azure Kubernetes Service (AKS) GPU-enabled clusters and KAITO workspaces. It focuses on identifying the root causes of common failures such as pod scheduling delays, VRAM exhaustion, OOM errors, and workspace readiness issues. The skill emphasizes a read-only approach, requiring explicit user authorization before performing any state-changing operations like scaling or node maintenance.

What it does

  • Analyzes GPU pod scheduling failures and pending states.
  • Correlates container OOM events with device-level VRAM telemetry.
  • Validates KAITO workspace readiness and configuration.
  • Inspects GPU profiles and autoscaling settings.
  • Provides a framework for gathering evidence from DCGM exporters and Kubernetes status.
  • Ensures all diagnostic actions comply with host-level security and authorization policies.

When to use it

  • When encountering 'Insufficient nvidia.com/gpu' errors.
  • When GPU-enabled pods remain in a Pending state.
  • When diagnosing model-load OOM or CUDA device-allocation failures.
  • When investigating KAITO workspace not-ready incidents.
  • When troubleshooting GPU autoscaling or node-pool capacity issues.

Advantages

  • Provides a systematic, profile-aware diagnostic path for complex GPU workloads.
  • Distinguishes between container-level memory issues and GPU-specific device exhaustion.
  • Maintains strict read-only safety by requiring authorization for any mutation.
  • Integrates telemetry from DCGM and Kubernetes events for precise incident correlation.

Usage examples

“My GPU pod is stuck in Pending, help me diagnose.”
The agent will bind to the cluster, capture pod events and the current gpuProfile, and investigate scheduling constraints or node capacity issues.
“Why is my model failing with a CUDA OOM error?”
The agent will correlate the model-load timestamp with DCGM metrics like FB_USED and FB_FREE to determine if the failure was caused by VRAM exhaustion.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/microsoft/aks-gpu-inference/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip aks-gpu-inference.zip -d .agents/skills

Documentation by Microsoft

AKS GPU Inference Day-2

Quick Reference

PropertyValue
Best forAKS GPU/KAITO failures
EvidenceBound target, events/status, gpuProfile, DCGM

When to Use This Skill

Use for scheduling, VRAM/OOM, KAITO readiness, and workload scaling. Route exclusions as described above.

MCP Tools

Use a fitting host-advertised Azure read or the references' read-only queries.

Host Capability Gate

Before executing any referenced pipeline, verify that the host authorizes the required shell and kubectl/az commands against the bound target. File-backed collection also requires approved artifact storage and access to any bundled files it uses. A governed Azure CLI tool alone does not establish these capabilities. Equivalent host reads may replace commands only where their advertised schemas provide the required evidence.

If execution is unavailable or prohibited, say so and analyze supplied or redacted events, status, logs, and metrics, or give the operator a scoped collection plan. State which reads did not run and leave conclusions requiring missing evidence unconfirmed. Never route kubectl through Azure MCP or bypass host policy. The authorization requirements below still apply.

Workflow

  1. Bind subscription, cluster, kube context, pool, and affected resource.

  2. Capture exact events/status and observed gpuProfile; state which reads ran.

  3. Route to scheduling, observability, KAITO, or scaling.

  4. Separate container/host OOM from device-allocation failures. For OOMKilled, correlate container memory limits/usage and node conditions. For Managed+Install device-memory evidence, use the exporter on port 19400 and incident-window FB_USED/FB_FREE; sizing tables do not establish a cause.

  5. For KAITO not-ready, warn that Workspace deletion leaves its GPU pools; cleanup is separate and needs explicit authorization.

  6. Report evidence, confidence, missing evidence, and owner handoff.

Require explicit authorization before scaling, cordon/drain, deletion, add-on enablement, or monitoring mutation.

Error Handling

ConditionResponse
Missing/contradictory evidenceMark unconfirmed; request the owner/read
Unknown profilePreserve evidence; do not prescribe stack repair
Mutation requiredPropose separately and wait for authorization

Files

references/gpu-scheduling.md

GPU scheduling, stack, quota, and capacity

Use this flow for a GPU pod that is Pending, especially with Insufficient nvidia.com/gpu. Keep every read bound to the named subscription, cluster, context, pool, namespace, and pod.

Capture the four walls

az account show --query '{subscription:id,tenant:tenantId}' -o json
az aks show --subscription <subscription> -g <resource-group> -n <cluster> \
  --query '{id:id,fqdn:fqdn,location:location}' -o json
kubectl config current-context
kubectl describe pod -n <namespace> <pod>
kubectl get node <gpu-node> \
  -o jsonpath='{.metadata.name}{"\n"}{.spec.taints}{"\n"}{.status.capacity}{"\n"}{.status.allocatable}{"\n"}'
az aks nodepool show --subscription <subscription> -g <resource-group> \
  --cluster-name <cluster> -n <pool> \
  --query '{id:id,vmSize:vmSize,gpuProfile:gpuProfile,enableAutoScaling:enableAutoScaling,provisioningState:provisioningState}' -o json

  1. Regional and family quota: use the failed operation's region and exact GPU VM family. Read az vm list-usage --location <region> -o json; compare that family's current value and limit. Do not claim a universal default.

  2. Capacity and SKU eligibility: retain the exact allocation error, then inspect az vm list-skus --location <region> --resource-type virtualMachines --all -o json for the selected SKU and restrictions. Quota, regional/zone capacity, and SKU eligibility are separate conclusions.

  3. Scheduling contract: compare node taints with pod tolerations and verify the container requests or limits nvidia.com/gpu. A selector must also match an eligible node. Quote the mismatched fields.

  4. Install profile: branch only on the observed gpuProfile.

Observed profileAKS responsibilityDay-2 interpretation
managementMode: Managed, driver: InstallDriver, device plugin, DCGM exporter, GPU healthThe profile establishes AKS ownership, but missing nvidia.com/gpu remains an unconfirmed incident cause until component status and node conditions support a managed-stack or node-health fault
managementMode: Unmanaged, driver: InstallDriver onlyVerify the customer-managed device plugin; the driver alone does not advertise nvidia.com/gpu
managementMode: Unmanaged, driver: NoneNoneVerify the BYO driver and device plugin owners and status
Missing or contradictoryUnknownDo not guess; preserve outputs and hand off

managementMode, migStrategy, and driver are creation-time settings. Changing profile means a new pool and requires authorization. Managed GPU pools are preview and do not support cluster autoscaler; never recommend cluster-autoscaler scale-to-zero for them. For a non-managed profile, inspect the actual autoscaler setting before discussing node scaling.

Treat the managed-pool autoscaler restriction as a supported platform fact, not as the observed cause of a Pending pod. It may be a relevant configuration hypothesis, but the incident cause stays unconfirmed until managed component status and node-condition evidence support it.

Sources: AKS-managed GPU node pools, self-managed NVIDIA GPUs.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Node.js

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
Microsoft
Publisher
Microsoft (Official)
Source
microsoft/skills · .github/plugins/aks-skills/skills/aks-gpu-inference · commit 0696600
Licence
MIT License (redistribution allowed)
  • LICENSE (added by AutoGineer from .github/plugins/aks-skills/LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by Microsoft.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download aks-gpu-inference.zip