Generated banner for AWS AI ML
Generated logo for AWS AI ML

AWS AI ML

AWSOfficialPublished by AWS

  • Skill
  • Tool integration
  • Cloud and DevOps
  • Apache-2.0
  • Low risk
  • Has scripts

From the author

Selects, deploys, and customizes AI models on Amazon SageMaker. Training or Processing jobs, fine-tuning (SFT/DPO/RLVR/RLAIF), model selection, dataset preparation, evaluation, SageMaker or Bedrock deployment, inference optimization and endpoint diagnostics. Covers the full lifecycle from planning through production. Use when fine-tuning models on SageMaker, choosing which base model to customize, fine-tune, or deploy from SageMaker JumpStart or Hub, SageMakerPublicHub, or the SageMaker public model catalog, transforming or validating training data, evaluating model quality, deploying or optimizing endpoints, configuring IAM/S3 for training, or managing SageMaker Managed MLflow. Use for endpoint health, failures, latency, logs, metrics, errors. Covers Serverless Model Customization, Nova and OSS deployment paths, and PySDK v3. NOT for Ground Truth labeling, Feature Store, or general-purpose AWS infrastructure.
— the description by AWS

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured framework for managing the Amazon SageMaker model lifecycle. It covers planning, dataset preparation, fine-tuning, model evaluation, and endpoint deployment. The skill assists users in navigating SageMaker-specific workflows, including selecting base models from catalogs, configuring training and processing jobs, and diagnosing endpoint health. It supports various fine-tuning techniques and deployment pathways, ensuring users follow validated procedures for their machine learning projects on AWS.

What it does

  • Guides users through the end-to-end lifecycle of SageMaker model customization.
  • Assists in selecting and evaluating base models from SageMaker JumpStart and public hubs.
  • Provides workflows for validating and transforming training and evaluation datasets.
  • Supports configuration of fine-tuning techniques like SFT, DPO, RLVR, and RLAIF.
  • Facilitates the deployment of models to SageMaker endpoints or Bedrock.
  • Diagnoses endpoint issues including latency, errors, and container logs.

When to use it

  • When planning a new model customization or deployment project on SageMaker.
  • When you need to validate or transform datasets for training or evaluation.
  • When selecting a base model or choosing a fine-tuning technique.
  • When you need to configure training, processing, or evaluation jobs.
  • When deploying a model to a SageMaker endpoint or troubleshooting an existing one.
  • When setting up or managing a SageMaker Managed MLflow environment.

Advantages

  • Provides structured, step-by-step workflows for complex SageMaker tasks.
  • Ensures dataset formatting meets specific SageMaker requirements.
  • Offers clear routing to relevant documentation based on user intent.
  • Helps resolve configuration issues for IAM, S3, and SDK settings.
  • Simplifies the diagnosis of endpoint performance and failure logs.

Usage examples

“I need to fine-tune a model on SageMaker. Where do I start?”
The agent will initiate the planning workflow, prompting you to define your use case and requirements before guiding you through model selection and dataset preparation.
“My SageMaker endpoint is returning errors. Can you help?”
The agent will route you to the endpoint diagnostics workflow to collect logs, check instance status, and analyze latency or failure metrics.
“How do I convert my CSV training data for SageMaker?”
The agent will guide you through the dataset transformation workflow, recommending the appropriate tools to convert your data into the required JSONL format.
“Which base model should I use for a text summarization task?”
The agent will trigger the model selection workflow to help you evaluate available models and resolve the correct Hub model ID for your specific requirements.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/aws/aws-ai-ml/download
  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip aws-ai-ml.zip -d .agents/skills

Documentation by AWS

AWS AI/ML Model Customization

Domain expertise for fine-tuning and deploying models on Amazon SageMaker. Covers the full model customization lifecycle from planning through production deployment.

Routing

Match the user's intent to the appropriate reference folder and load only that content.

User intentReferenceWhen to use
Plan a model customization project, discover scope of work, resume or modify a planreferences/planning/User's request relates to model customization or deployment (fine-tuning, training, building, customizing, reviewing data, deploying or standing up a model — including selecting or deploying an off-the-shelf or base model with no training — or getting advice on approach). Always co-activate with other intents to discover full scope. Load this reference FIRST when the request matches multiple rows in this table — read its plan templates before routing to a single-action reference.
Define the business problem, success criteria, or use case specreferences/use-case-specification/User says "define my use case", "capture requirements", "what should I decide up front", or as default first step in any plan. Skip only if user explicitly declines.
Select or change a base modelreferences/model-selection/User asks which model to use, mentions a model name or family, or wants to evaluate what's available. Always activate model-selection even for known model names because the exact Hub model ID must be resolved. Recommended: route to use-case-specification first to capture requirements — this produces better filtering results. Routing to use-case-specification first is not required if user provides a specific model name/ID or declines. If intent is ambiguous (fine-tune vs deploy as-is), model-selection MUST confirm which path before proceeding. Base model filtering for deployment MUST go through select-for-deployment.md and its scripts for any final recommendation.
Choose a fine-tuning technique (SFT, DPO, RLVR, RLAIF)references/finetuning-technique/User has decided to fine-tune and needs to choose a technique, or technique needs validation against the selected model's recipes. Requires a base model to be selected first.
Validate dataset quality and formatreferences/dataset-evaluation/User says "is my dataset okay", "check my training data", "I have my own data", or before starting any fine-tuning job.
Transform or convert a dataset between formatsreferences/dataset-transformation/User says "transform", "convert", "reformat", or dataset schema needs to change. Always use this rather than writing inline transformation code.
Generate fine-tuning code and start trainingreferences/finetuning/User says "start training", "fine-tune my model", "I'm ready to train", or plan reaches the finetuning step. Supports SFT, DPO, RLVR, RLAIF trainers.
Train a model with a SageMaker Training jobreferences/training-jobs/Use when the goal is to fit or update model parameters and produce model artifacts or checkpoints. Covers recipes, JumpStart traditional ML, custom training scripts or containers, distributed training, and training-job diagnosis.
Process data with a SageMaker Processing jobreferences/processing-jobs/Use when the goal is preprocessing, feature engineering, evaluation, transformation, or ETL without model training as the primary operation. Covers scripts, framework processors, Spark, custom containers, Athena, Redshift, and processing-job diagnosis.
Evaluate or benchmark a trained modelreferences/model-evaluation/User says "evaluate my model", "run a benchmark", "test model performance", "compare models". Supports LLM-as-Judge and Custom Scorer.
Deploy, benchmark, or optimize a model on an endpoint or Bedrockreferences/model-deployment/User says "deploy my model", "create an endpoint", or "make it available" (plain deploy) — or, for the inference-optimization sub-workflows on SageMaker Real-Time Endpoints only, "benchmark my endpoint" / "compare benchmark runs" (benchmarking), or states a performance/cost/latency/throughput goal for a new deployment such as "find the cheapest instance" (recommendations). Handles Nova vs OSS deployment pathways.
Set up IAM roles, S3 buckets, SDK configurationreferences/sdk-getting-started/User says "set up", "getting started", "check my environment", "configure SDK", or as first step in any plan involving SageMaker training/evaluation/deployment.
Manage project directory and artifactsreferences/directory-management/Starting a new project, resuming existing one, or when PLAN.md needs to be associated with a project directory.
Set up, update, or delete a SageMaker Managed MLflow appreferences/manage-mlflow/User says "set up MLflow", "create MLflow app", "update my MLflow app", "delete my MLflow app", "I need an MLflow server", asks "what is SageMaker MLflow", or a workflow needs an MLflow backend and none is connected.
Diagnose a failing or unhealthy SageMaker endpointreferences/endpoint-diagnostics/User reports endpoint errors, latency, inference failures, or a deployment that failed. "What's the status of my endpoint?", "Is my endpoint erroring?", "My endpoint failed — why?", "How many instances are running behind my endpoint?", "Is the latency my model or SageMaker?", "Show me the container logs for my endpoint." NOT for training-job issues, endpoint deletion, scaling changes, or new deployments.

Rules

  • Service boundary. This skill covers SageMaker, including deployment of a customized model to Bedrock. Bedrock-native work (invoking Bedrock's managed foundation models, Bedrock Agents, Knowledge Bases, Guardrails, AgentCore) is supported by AWS but has no validated workflow in this skill: let the user know and proceed with best-effort guidance using general AWS knowledge. If the user asks for a model on AWS without naming a service, compare SageMaker and Bedrock or ask one service-choice question. Do not select a service by default. The two services have different model catalogs, deployment paths, and pricing, and a default selection can send the user into an incorrect workflow.

  • Progressive disclosure. Load only the reference folder relevant to the current user intent. Do not load all references at once.

  • Best-effort help. If the user's request falls outside this skill's references, do not dead-end the conversation. Help them using general AWS knowledge and documentation, and inform the user that the guidance is not covered by this skill's validated workflows.

  • Usage attribution. Before running any AWS CLI command or packaged script, set export AWS_SDK_UA_APP_ID=AWSSkill-SageMaker.

Files

references/training-jobs/overview.md

Training Jobs

Launch, monitor, and diagnose SageMaker training jobs with SageMaker Python SDK v3. This capability covers serverful recipes, zero-config JumpStart traditional ML, BYO source/container/algorithm jobs, and training-job debugging.

When to use

Use this capability when the user asks to:

  • fine-tune with a pre-built/custom recipe, reserved capacity/training plan, or distributed compute

  • train XGBoost, LightGBM, CatBoost, sklearn, or another JumpStart traditional ML model

  • run train.py, a custom ECR image, or a Marketplace algorithm on managed training compute

  • diagnose a failed, stopped, or stuck SageMaker training job

For ordinary managed serverless SFT/DPO/RLVR/RLAIF requests without those signals, use finetuning.

Training surface decision

Intended outcomeUse
Managed serverless SFT/DPO/RLVR/RLAIF with SageMaker's customization experiencefinetuning and its SFTTrainer/technique-specific flow
A serverful recipe that controls launcher/distribution/runtime on explicit training instancesThis reference with ModelTrainer.from_recipe
Traditional ML from JumpStart, or BYO script/image/algorithmThis reference

Both SFTTrainer and recipe training can accept a Compute object. The deciding factor is the requested workflow and artifact contract, not the presence of Compute: use SFTTrainer for the managed customization workflow and ModelTrainer.from_recipe when the user explicitly needs a published/custom serverful recipe or its distributed runtime.

Operating assumptions

AreaAssumption
RuntimePython supported by sagemaker>=3.22.0,<4.0; dependencies are installed by the generated template
CredentialsAmbient AWS credentials plus a SageMaker execution role; caller needs sagemaker:CreateTrainingJob, sagemaker:DescribeTrainingJob, sagemaker:ListTrainingJobs, S3/ECR access as applicable, and iam:PassRole
StorageSmall source/config files may be local under the project directory; datasets, checkpoints, recipes, and model artifacts use S3
RegionResolve from the current AWS session; verify model/recipe, compute, quota, image, and S3 resources in that region
NetworkPyPI access is needed for installation; the job needs the AWS endpoints and any image/package endpoints its code uses

Decision flow

  1. Check conversation and project files before asking for values.

  2. Choose exactly one launch branch:

    • Recipe/serverful LLM → code_templates/recipe_training.py

    • JumpStart traditional ML → code_templates/jumpstart_training.py

    • BYO source, image, or algorithm → code_templates/custom_training.py

    • Existing job failure/status → references/troubleshooting.md

  3. For JumpStart, hand off model discovery to model-selection; use the exact Hub model ID/version it returns. If direct CLI verification is needed, use aws sagemaker list-hub-contents and aws sagemaker describe-hub-content. Never invent IDs or silently select a different version.

  4. For recipes, search the SageMaker HyperPod recipes catalog or use a user-provided YAML. Inspect the selected schema and call get_resolved_recipe() before approval; never invent recipe names or override keys.

  5. For custom jobs, require exactly one of a versioned/digest-pinned training image URI or an algorithm name/ARN.

Recipe configuration precedence

ModelTrainer.from_recipe loads the base recipe, applies recipe_overrides, and derives source code, image, distributed settings, compute details, and hyperparameters. The explicit Compute argument is required, but only INSTANCE_TYPE must be supplied by this template. Leave INSTANCE_COUNT and VOLUME_SIZE_GB as None to preserve the selected recipe/SDK defaults; set them only from confirmed user inputs when an explicit override is required. TRAINING_PLAN_ARN, when set, is also an explicit compute override. Direct constructor inputs such as networking, stopping condition, output, checkpoint, role, and base job name are then applied to the trainer. Review get_resolved_recipe() output before launch so the user can see the final merged configuration.

General published recipes may resolve their image. Nova and LLM fine-tuning recipe families require an explicit, verified training_image; use a supported, versioned URI from the selected recipe/provider documentation rather than latest. Verify the selected recipe's requirements because recipe contracts may change.

Gather inputs

Use cascading fallback: conversation → project files → ask one unresolved question at a time.

InputRecipeJumpStartCustom
Recipe name/YAMLRequired——
JumpStart model ID/version—Required—
Training image or algorithmRecipe-dependentResolved by configRequired
Source directory/entry scriptOptionalOptional overrideOptional
Training S3 URIIf recipe needs itRequiredRequired if algorithm needs it
Instance type/countRequiredOptional overrideRequired
Training plan ARNOptionalOptionalOptional
Role and output S3 URIRequiredRequiredRequired
Hyperparameters/recipe overridesOptionalOptionalOptional
Checkpoint, VPC, encryptionOptional, only when requested/requiredOptionalOptional
Max runtimeRequired spend boundRequired spend boundRequired spend bound

A code-level accept_eula switch exists only on JumpStartConfig. Set it to True only after the user explicitly accepts the gated model license. Recipe licenses and prerequisites are recipe/provider-specific; surface and confirm them before launch rather than claiming that an EULA flag is propagated by from_recipe.

Preflight and confirmation

Before generating code or submitting a job:

  1. Verify region, role trust, S3 region, image/algorithm/recipe existence, and service quota for requested compute.

  2. If quota is insufficient, report the exact requested/current values and explain how to request an increase without submitting one: open Service Quotas, choose the target Region, find the SageMaker quota matching the job type and instance type, select Request increase, enter the required value and use-case rationale, then monitor request history. For CLI users, first discover the matching quota code with aws service-quotas list-service-quotas --service-code sagemaker --region <region>; only draft request-service-quota-increase after the user confirms the quota code and desired value. Do not submit a request, silently switch compute, or downsize without approval.

  3. Validate optional checkpoint, VPC/security groups, output KMS key permissions, network isolation implications, and inter-container encryption only when selected.

  4. Summarize model/recipe/image, input/output, instance type/count, training plan, max runtime, optional advanced settings, and wait behavior.

  5. Link to SageMaker pricing rather than calculating a dollar estimate.

  6. Ask for explicit approval. Stop until approved.

Generate and launch

  1. Read references/configuration.md and the selected template.

  2. Write the customized output under the project directory in notebook or script mode.

  3. Keep WAIT_FOR_COMPLETION=False unless the user asks to block.

  4. Templates create a collision-resistant job prefix, launch asynchronously, and resolve only that submission through public ListTrainingJobs/DescribeTrainingJob APIs. They do not use the SDK's private _latest_training_job state.

  5. Report output model artifacts only after the job reaches Completed.

Diagnose an existing job

Read references/troubleshooting.md. Start with DescribeTrainingJob, then inspect only the job's CloudWatch stream and relevant metrics. Return the failure signature, root cause category, evidence, and concrete fix; state what was not checked.

Out of Scope

The following are supported by SageMaker and AWS but do not have a validated workflow in this reference. If the user's request matches one of these, let them know and proceed with best-effort guidance using general AWS knowledge:

  • HyperPod job submission and cluster lifecycle

  • Autopilot/AutoML job creation

  • distributed configuration that requires custom launcher code beyond a published recipe

  • every optional CreateTrainingJob field; templates intentionally expose only verified checkpoint, VPC, output-KMS, network-isolation, and traffic-encryption settings

References

  • code_templates/recipe_training.py — serverful pre-built/custom recipe

  • code_templates/jumpstart_training.py — JumpStart zero-config traditional ML

  • code_templates/custom_training.py — BYO source/image/algorithm

  • references/configuration.md — placeholder and branch configuration

  • references/troubleshooting.md — status and failure classification

  • SageMaker HyperPod recipes

  • SageMaker Python SDK image URI utility

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Low risk

  • Instructions that use the network
  • Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
AWS
Publisher
AWS (Official)
Source
aws/agent-toolkit-for-aws · skills/core-skills/aws-ai-ml · commit 0a218f8
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • NOTICE (added by AutoGineer from NOTICE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced
Also distributed in
aws/agent-toolkit-for-aws

AutoGineer is not affiliated with or endorsed by AWS.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download aws-ai-ml.zip