AutoGineer
Generated banner for AWS AI ML
Generated logo for AWS AI ML

AWS AI ML

AWSOfficialPublished by AWS

  • Skill
  • Tool integration
  • Cloud and DevOps
  • Apache-2.0
  • Low risk
  • Has scripts
Download ZIP

From the author

Selects, deploys, and customizes AI models on Amazon SageMaker. Training or Processing jobs, fine-tuning (SFT/DPO/RLVR/RLAIF), model selection, dataset preparation, evaluation, SageMaker or Bedrock deployment, inference optimization and endpoint diagnostics. Covers the full lifecycle from planning through production. Use when fine-tuning models on SageMaker, choosing which base model to customize, fine-tune, or deploy from SageMaker JumpStart or Hub, SageMakerPublicHub, or the SageMaker public model catalog, transforming or validating training data, evaluating model quality, deploying or optimizing endpoints, configuring IAM/S3 for training, or managing SageMaker Managed MLflow. Use for endpoint health, failures, latency, logs, metrics, errors. Covers Serverless Model Customization, Nova and OSS deployment paths, and PySDK v3. NOT for Ground Truth labeling, Feature Store, or general-purpose AWS infrastructure.
— the description by AWS

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured framework for managing the Amazon SageMaker model lifecycle. It covers planning, dataset preparation, fine-tuning, model evaluation, and endpoint deployment. The skill assists users in navigating SageMaker-specific workflows, including selecting base models from catalogs, configuring training and processing jobs, and diagnosing endpoint health. It supports various fine-tuning techniques and deployment pathways, ensuring users follow validated procedures for their machine learning projects on AWS.

What it does

  • Guides users through the end-to-end lifecycle of SageMaker model customization.
  • Assists in selecting and evaluating base models from SageMaker JumpStart and public hubs.
  • Provides workflows for validating and transforming training and evaluation datasets.
  • Supports configuration of fine-tuning techniques like SFT, DPO, RLVR, and RLAIF.
  • Facilitates the deployment of models to SageMaker endpoints or Bedrock.
  • Diagnoses endpoint issues including latency, errors, and container logs.

When to use it

  • When planning a new model customization or deployment project on SageMaker.
  • When you need to validate or transform datasets for training or evaluation.
  • When selecting a base model or choosing a fine-tuning technique.
  • When you need to configure training, processing, or evaluation jobs.
  • When deploying a model to a SageMaker endpoint or troubleshooting an existing one.
  • When setting up or managing a SageMaker Managed MLflow environment.

Advantages

  • Provides structured, step-by-step workflows for complex SageMaker tasks.
  • Ensures dataset formatting meets specific SageMaker requirements.
  • Offers clear routing to relevant documentation based on user intent.
  • Helps resolve configuration issues for IAM, S3, and SDK settings.
  • Simplifies the diagnosis of endpoint performance and failure logs.

Usage examples

“I need to fine-tune a model on SageMaker. Where do I start?”
The agent will initiate the planning workflow, prompting you to define your use case and requirements before guiding you through model selection and dataset preparation.
“My SageMaker endpoint is returning errors. Can you help?”
The agent will route you to the endpoint diagnostics workflow to collect logs, check instance status, and analyze latency or failure metrics.
“How do I convert my CSV training data for SageMaker?”
The agent will guide you through the dataset transformation workflow, recommending the appropriate tools to convert your data into the required JSONL format.
“Which base model should I use for a text summarization task?”
The agent will trigger the model selection workflow to help you evaluate available models and resolve the correct Hub model ID for your specific requirements.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/aws/aws-ai-ml/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip aws-ai-ml.zip -d .agents/skills

Documentation by AWS

AWS AI/ML Model Customization

Domain expertise for fine-tuning and deploying models on Amazon SageMaker. Covers the full model customization lifecycle from planning through production deployment.

Routing

Match the user's intent to the appropriate reference folder and load only that content.

User intentReferenceWhen to use
Plan a model customization project, discover scope of work, resume or modify a planreferences/planning/User's request relates to model customization or deployment (fine-tuning, training, building, customizing, reviewing data, deploying or standing up a model — including selecting or deploying an off-the-shelf or base model with no training — or getting advice on approach). Always co-activate with other intents to discover full scope. Load this reference FIRST when the request matches multiple rows in this table — read its plan templates before routing to a single-action reference.
Define the business problem, success criteria, or use case specreferences/use-case-specification/User says "define my use case", "capture requirements", "what should I decide up front", or as default first step in any plan. Skip only if user explicitly declines.
Select or change a base modelreferences/model-selection/User asks which model to use, mentions a model name or family, or wants to evaluate what's available. Always activate model-selection even for known model names because the exact Hub model ID must be resolved. Recommended: route to use-case-specification first to capture requirements — this produces better filtering results. Routing to use-case-specification first is not required if user provides a specific model name/ID or declines. If intent is ambiguous (fine-tune vs deploy as-is), model-selection MUST confirm which path before proceeding. Base model filtering for deployment MUST go through select-for-deployment.md and its scripts for any final recommendation.
Choose a fine-tuning technique (SFT, DPO, RLVR, RLAIF)references/finetuning-technique/User has decided to fine-tune and needs to choose a technique, or technique needs validation against the selected model's recipes. Requires a base model to be selected first.
Validate dataset quality and formatreferences/dataset-evaluation/User says "is my dataset okay", "check my training data", "I have my own data", or before starting any fine-tuning job.
Transform or convert a dataset between formatsreferences/dataset-transformation/User says "transform", "convert", "reformat", or dataset schema needs to change. Always use this rather than writing inline transformation code.
Generate fine-tuning code and start trainingreferences/finetuning/User says "start training", "fine-tune my model", "I'm ready to train", or plan reaches the finetuning step. Supports SFT, DPO, RLVR, RLAIF trainers.
Train a model with a SageMaker Training jobreferences/training-jobs/Use when the goal is to fit or update model parameters and produce model artifacts or checkpoints. Covers recipes, JumpStart traditional ML, custom training scripts or containers, distributed training, and training-job diagnosis.
Process data with a SageMaker Processing jobreferences/processing-jobs/Use when the goal is preprocessing, feature engineering, evaluation, transformation, or ETL without model training as the primary operation. Covers scripts, framework processors, Spark, custom containers, Athena, Redshift, and processing-job diagnosis.
Evaluate or benchmark a trained modelreferences/model-evaluation/User says "evaluate my model", "run a benchmark", "test model performance", "compare models". Supports LLM-as-Judge and Custom Scorer.
Deploy, benchmark, or optimize a model on an endpoint or Bedrockreferences/model-deployment/User says "deploy my model", "create an endpoint", or "make it available" (plain deploy) — or, for the inference-optimization sub-workflows on SageMaker Real-Time Endpoints only, "benchmark my endpoint" / "compare benchmark runs" (benchmarking), or states a performance/cost/latency/throughput goal for a new deployment such as "find the cheapest instance" (recommendations). Handles Nova vs OSS deployment pathways.
Set up IAM roles, S3 buckets, SDK configurationreferences/sdk-getting-started/User says "set up", "getting started", "check my environment", "configure SDK", or as first step in any plan involving SageMaker training/evaluation/deployment.
Manage project directory and artifactsreferences/directory-management/Starting a new project, resuming existing one, or when PLAN.md needs to be associated with a project directory.
Set up, update, or delete a SageMaker Managed MLflow appreferences/manage-mlflow/User says "set up MLflow", "create MLflow app", "update my MLflow app", "delete my MLflow app", "I need an MLflow server", asks "what is SageMaker MLflow", or a workflow needs an MLflow backend and none is connected.
Diagnose a failing or unhealthy SageMaker endpointreferences/endpoint-diagnostics/User reports endpoint errors, latency, inference failures, or a deployment that failed. "What's the status of my endpoint?", "Is my endpoint erroring?", "My endpoint failed — why?", "How many instances are running behind my endpoint?", "Is the latency my model or SageMaker?", "Show me the container logs for my endpoint." NOT for training-job issues, endpoint deletion, scaling changes, or new deployments.

Rules

  • Service boundary. This skill covers SageMaker, including deployment of a customized model to Bedrock. Bedrock-native work (invoking Bedrock's managed foundation models, Bedrock Agents, Knowledge Bases, Guardrails, AgentCore) is supported by AWS but has no validated workflow in this skill: let the user know and proceed with best-effort guidance using general AWS knowledge. If the user asks for a model on AWS without naming a service, compare SageMaker and Bedrock or ask one service-choice question. Do not select a service by default. The two services have different model catalogs, deployment paths, and pricing, and a default selection can send the user into an incorrect workflow.

  • Progressive disclosure. Load only the reference folder relevant to the current user intent. Do not load all references at once.

  • Best-effort help. If the user's request falls outside this skill's references, do not dead-end the conversation. Help them using general AWS knowledge and documentation, and inform the user that the guidance is not covered by this skill's validated workflows.

  • Usage attribution. Before running any AWS CLI command or packaged script, set export AWS_SDK_UA_APP_ID=AWSSkill-SageMaker.

Files

references/model-evaluation/references/evaluation-type-guide.md

Evaluation Type Guide

Help the user decide which evaluation type to use based on their goals and constraints.

Evaluation types at a glance

TypeWhat it doesEval dataset?CostSupported models
LLM-as-JudgeAn LLM scores your model's responses on subjective qualities like helpfulness, correctness, coherence, and safety.YesHigherOSS only
Custom ScorerYour own scoring logic (or a built-in scorer) evaluates outputs programmatically — exact/near match, pattern checks.YesLowerAll

When to use which — in short:

  • To assess subjective qualities like tone, helpfulness, coherence, or faithfulness → LLM-as-Judge

  • When a programmatic approach can give a meaningful signal about output quality → Custom Scorer

Decision flow

Work through the steps below in order. For each, use what you already know from conversation history, plan.md, workflow_state.json, or other files you've read. Only ask the user if you genuinely don't know.

Step 1: Check for evaluation dataset

For this step, you need to know: whether the user has an evaluation dataset.

If you don't know from previous context, ask:

"Do you have an eval dataset?"

⏸ Wait for user.

If the user does not have one:

"All supported evaluation types require an evaluation dataset. Unfortunately, this skill can't help with model evaluation without one."

Stop here. Do not offer to help create or find a dataset, since our references do not support this.

If the user has an evaluation dataset, continue.

Step 2: Check model compatibility

For this step, you need to know: what type of model is being evaluated (open source or Nova).

If you don't already know from conversation context, try to determine it:

  1. If you have the training job name or ARN, run aws sagemaker list-tags --resource-arn <training-job-arn> and look for the sagemaker-studio:jumpstart-model-id tag.

    • Contains "nova" (e.g., nova-micro, nova-lite, nova-pro) → Nova

    • Anything else (Llama, Mistral, Qwen, GPT-OSS, DeepSeek, etc.) → OSS

  2. If you have a Model Package ARN, run aws sagemaker describe-model-package --model-package-name <arn> and check the model description or source tags for the same model ID.

  3. If neither is available, ask the user:

    "What model are you evaluating — is it a Nova model or an open-source model (like Llama, Mistral, Qwen, etc.)?"

If the model is Nova, LLM-as-Judge is not supported. Tell the user:

"Unfortunately, LLM-as-Judge isn't available for Nova models. Can we use Custom Scorer instead?"

If Nova -> Skip to Step 4 (Recommend); the choice is already determined.

Else -> Continue to Step 3 with the remaining options.

Step 3: Gather information and assess signal strength

This step has four substeps. Substeps 3a and 3b are questions you ask the user; substeps 3c and 3d are silent reasoning you do on your own. Do each substep on its own — never collapse them, and never skip an ASK substep because the answer "seems obvious." You may only skip an ASK substep if you already have a clear, explicit answer from earlier in the conversation.

Step 3a: Understand the task and data — [ASK THE USER]

You need to understand: what the model does, what the evaluation data looks like, and — critically — what the model's outputs and the reference answers look like (e.g., a single exact value, a short label, or free-form paragraph-length text).

If you don't already have a clear picture from conversation context, ask:

"Can you tell me about the task you're focused on? What does your model do, what does your evaluation dataset look like, and what form do the answers take — exact values, short labels, or longer free-form text?"

The answer-format detail matters: it drives the signal-strength assessment in Step 3c and the caveats in Step 4. If the user's answer is vague about output/answer format, ask a follow-up before moving on.

⏸ Wait for user.

Step 3b: Check cost sensitivity — [ASK THE USER]

LLM-as-Judge invokes a model to score each sample, which adds cost. Custom Scorer runs your own code, which is cheaper.

You MUST ask this question unless the user has already given an explicit cost priority earlier in the conversation. Do not infer or assume cost sensitivity from the task, the budget, or anything else — get it directly from the user:

"On a scale of 1-5, how important is it to you to keep evaluation costs low, even if it means less nuanced results? 5 means prioritize budget above all else."

⏸ Wait for user.

Step 3c: Assess Custom Scorer signal strength — [THINK — do not message the user]

Based on the task and data (especially the answer format from Step 3a), decide how strong a signal a custom (programmatic) scorer could give about task success. Rate it strong, medium, or weak:

  • Strong: A programmatic check can reliably tell you whether the output is correct. Examples include math problems with numerical answers, classification tasks with labels, or extraction tasks with exact ground truths.

  • Medium: The task has reference answers, and programmatic comparison gives a useful but imperfect signal. Examples include summarization or Q&A where the answers are free-form / paragraph-length, so text comparison against references captures something meaningful but misses semantically-equivalent phrasings, or format compliance checks where you can verify structure even if you can't verify content quality.

  • Weak: What matters about the output is hard to capture in code. There may be no reliable reference answer to compare against, or the reference doesn't capture what the user actually cares about.

Think broadly — even tasks that seem subjective may have a programmatic angle. Note your rating; you'll use it in Step 4, and if it is medium or weak you must surface that limitation as a caveat.

Step 3d: Assess LLM-as-Judge signal strength — [THINK — do not message the user]

Based on the task and data, decide how strong a signal an LLM judge could give about task success. Rate it strong, medium, or weak:

  • Strong: Key model quality metrics are inherently subjective — helpfulness, coherence, tone, etc.

  • Medium: The task has some subjective element, but also a clear factual or structural component that a programmatic approach could partially cover.

  • Weak: The task has a single objectively correct answer, and a judge model carries the risk of hallucinating, while a programmatic check would be more reliable.

Think broadly — LLM-as-Judge can surface issues that are hard to anticipate with code, but it's not always the best tool for the job.

Step 4: Recommend an evaluation type

Gate — before you recommend, confirm you have every input below. If the model is Nova, the gate is already satisfied (Custom Scorer is the only option) — skip the checklist and proceed directly to the Nova recommendation. Otherwise, do not present a recommendation until each of these is known. If any is missing, go back and complete the corresponding step first:

  • [ ] Evaluation dataset confirmed (Step 1)

  • [ ] Model type known — Nova or OSS (Step 2)

  • [ ] Task + answer format understood (Step 3a)

  • [ ] Cost sensitivity obtained directly from the user, or already stated explicitly earlier (Step 3b)

  • [ ] Custom Scorer signal strength rated (Step 3c) and LLM-as-Judge signal strength rated (Step 3d)

Once all inputs are in hand, use the signal strength assessments and cost sensitivity to make a recommendation:

  • Nova model → recommend Custom Scorer (only available option).

  • Custom Scorer signal is strong → recommend Custom Scorer. It's deterministic, reproducible, and cost-effective. A programmatic approach gives you a reliable signal for this task.

  • Cost sensitivity is very high (4-5) and Custom Scorer signal is weak but not totally absent → recommend Custom Scorer, but be upfront that the programmatic signal may be limited for this task. A partial signal at low cost may be preferable to a richer signal at higher cost.

  • LLM-as-Judge signal is strong and Custom Scorer signal is weak → recommend LLM-as-Judge. The task needs the kind of nuanced judgment that only an LLM can provide.

  • Both have medium or strong signal → Carefully weigh the customer's cost concerns with the benefits of each eval type. Recommend the one that you think fits all of their needs the best.

Always state caveats. Whenever the Custom Scorer signal you rated in Step 3c is medium or weak (e.g., QA or summarization with free-form / paragraph-length reference answers), and you recommend Custom Scorer, you MUST tell the user that the programmatic signal is limited — semantically-equivalent but differently-worded answers may score poorly, so the result is an approximate signal, not a precise measure of quality.

Present your recommendation with a brief reason:

"Based on what you've told me, I'd recommend [evaluation type] — [one sentence explaining why]. Want to go with that?"

⏸ Wait for user to confirm.

Once the user confirms, return to the main SKILL.md workflow (Step 2: Validate and hand off).

Custom Scorer: choosing the right scorer type

If the user chose Custom Scorer, use this logic to recommend the specific scorer:

ScorerRecommend when
Prime MathTask involves mathematical reasoning with verifiable numeric/symbolic answers
Prime CodeTask involves code generation that can be tested against input/output pairs
Custom LambdaAny task with custom scoring logic that doesn't fit Prime Math or Prime Code

Decision logic:

  1. If task is math with verifiable answers → recommend Prime Math.

  2. If task is code generation with testable I/O → recommend Prime Code.

  3. Otherwise → recommend Custom Lambda.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Low risk

  • Instructions that use the network
  • Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
AWS
Publisher
AWS (Official)
Source
aws/agent-toolkit-for-aws · skills/core-skills/aws-ai-ml · commit 0a218f8
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • NOTICE (added by AutoGineer from NOTICE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced
Also distributed in
aws/agent-toolkit-for-aws

AutoGineer is not affiliated with or endorsed by AWS.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download aws-ai-ml.zip