AWS AI ML
AWSOfficialPublished by AWS
- Skill
- Tool integration
- Cloud and DevOps
- Apache-2.0
- Low risk
- Has scripts
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
This skill provides a structured framework for managing the Amazon SageMaker model lifecycle. It covers planning, dataset preparation, fine-tuning, model evaluation, and endpoint deployment. The skill assists users in navigating SageMaker-specific workflows, including selecting base models from catalogs, configuring training and processing jobs, and diagnosing endpoint health. It supports various fine-tuning techniques and deployment pathways, ensuring users follow validated procedures for their machine learning projects on AWS.
What it does
- Guides users through the end-to-end lifecycle of SageMaker model customization.
- Assists in selecting and evaluating base models from SageMaker JumpStart and public hubs.
- Provides workflows for validating and transforming training and evaluation datasets.
- Supports configuration of fine-tuning techniques like SFT, DPO, RLVR, and RLAIF.
- Facilitates the deployment of models to SageMaker endpoints or Bedrock.
- Diagnoses endpoint issues including latency, errors, and container logs.
When to use it
- When planning a new model customization or deployment project on SageMaker.
- When you need to validate or transform datasets for training or evaluation.
- When selecting a base model or choosing a fine-tuning technique.
- When you need to configure training, processing, or evaluation jobs.
- When deploying a model to a SageMaker endpoint or troubleshooting an existing one.
- When setting up or managing a SageMaker Managed MLflow environment.
Advantages
- Provides structured, step-by-step workflows for complex SageMaker tasks.
- Ensures dataset formatting meets specific SageMaker requirements.
- Offers clear routing to relevant documentation based on user intent.
- Helps resolve configuration issues for IAM, S3, and SDK settings.
- Simplifies the diagnosis of endpoint performance and failure logs.
Usage examples
- “I need to fine-tune a model on SageMaker. Where do I start?”
- The agent will initiate the planning workflow, prompting you to define your use case and requirements before guiding you through model selection and dataset preparation.
- “My SageMaker endpoint is returning errors. Can you help?”
- The agent will route you to the endpoint diagnostics workflow to collect logs, check instance status, and analyze latency or failure metrics.
- “How do I convert my CSV training data for SageMaker?”
- The agent will guide you through the dataset transformation workflow, recommending the appropriate tools to convert your data into the required JSONL format.
- “Which base model should I use for a text summarization task?”
- The agent will trigger the model selection workflow to help you evaluate available models and resolve the correct Hub model ID for your specific requirements.
Install
Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/aws/aws-ai-ml/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip aws-ai-ml.zip -d .agents/skills
Documentation by AWS
AWS AI/ML Model Customization
Domain expertise for fine-tuning and deploying models on Amazon SageMaker. Covers the full model customization lifecycle from planning through production deployment.
Routing
Match the user's intent to the appropriate reference folder and load only that content.
| User intent | Reference | When to use |
|---|---|---|
| Plan a model customization project, discover scope of work, resume or modify a plan | references/planning/ | User's request relates to model customization or deployment (fine-tuning, training, building, customizing, reviewing data, deploying or standing up a model — including selecting or deploying an off-the-shelf or base model with no training — or getting advice on approach). Always co-activate with other intents to discover full scope. Load this reference FIRST when the request matches multiple rows in this table — read its plan templates before routing to a single-action reference. |
| Define the business problem, success criteria, or use case spec | references/use-case-specification/ | User says "define my use case", "capture requirements", "what should I decide up front", or as default first step in any plan. Skip only if user explicitly declines. |
| Select or change a base model | references/model-selection/ | User asks which model to use, mentions a model name or family, or wants to evaluate what's available. Always activate model-selection even for known model names because the exact Hub model ID must be resolved. Recommended: route to use-case-specification first to capture requirements — this produces better filtering results. Routing to use-case-specification first is not required if user provides a specific model name/ID or declines. If intent is ambiguous (fine-tune vs deploy as-is), model-selection MUST confirm which path before proceeding. Base model filtering for deployment MUST go through select-for-deployment.md and its scripts for any final recommendation. |
| Choose a fine-tuning technique (SFT, DPO, RLVR, RLAIF) | references/finetuning-technique/ | User has decided to fine-tune and needs to choose a technique, or technique needs validation against the selected model's recipes. Requires a base model to be selected first. |
| Validate dataset quality and format | references/dataset-evaluation/ | User says "is my dataset okay", "check my training data", "I have my own data", or before starting any fine-tuning job. |
| Transform or convert a dataset between formats | references/dataset-transformation/ | User says "transform", "convert", "reformat", or dataset schema needs to change. Always use this rather than writing inline transformation code. |
| Generate fine-tuning code and start training | references/finetuning/ | User says "start training", "fine-tune my model", "I'm ready to train", or plan reaches the finetuning step. Supports SFT, DPO, RLVR, RLAIF trainers. |
| Train a model with a SageMaker Training job | references/training-jobs/ | Use when the goal is to fit or update model parameters and produce model artifacts or checkpoints. Covers recipes, JumpStart traditional ML, custom training scripts or containers, distributed training, and training-job diagnosis. |
| Process data with a SageMaker Processing job | references/processing-jobs/ | Use when the goal is preprocessing, feature engineering, evaluation, transformation, or ETL without model training as the primary operation. Covers scripts, framework processors, Spark, custom containers, Athena, Redshift, and processing-job diagnosis. |
| Evaluate or benchmark a trained model | references/model-evaluation/ | User says "evaluate my model", "run a benchmark", "test model performance", "compare models". Supports LLM-as-Judge and Custom Scorer. |
| Deploy, benchmark, or optimize a model on an endpoint or Bedrock | references/model-deployment/ | User says "deploy my model", "create an endpoint", or "make it available" (plain deploy) — or, for the inference-optimization sub-workflows on SageMaker Real-Time Endpoints only, "benchmark my endpoint" / "compare benchmark runs" (benchmarking), or states a performance/cost/latency/throughput goal for a new deployment such as "find the cheapest instance" (recommendations). Handles Nova vs OSS deployment pathways. |
| Set up IAM roles, S3 buckets, SDK configuration | references/sdk-getting-started/ | User says "set up", "getting started", "check my environment", "configure SDK", or as first step in any plan involving SageMaker training/evaluation/deployment. |
| Manage project directory and artifacts | references/directory-management/ | Starting a new project, resuming existing one, or when PLAN.md needs to be associated with a project directory. |
| Set up, update, or delete a SageMaker Managed MLflow app | references/manage-mlflow/ | User says "set up MLflow", "create MLflow app", "update my MLflow app", "delete my MLflow app", "I need an MLflow server", asks "what is SageMaker MLflow", or a workflow needs an MLflow backend and none is connected. |
| Diagnose a failing or unhealthy SageMaker endpoint | references/endpoint-diagnostics/ | User reports endpoint errors, latency, inference failures, or a deployment that failed. "What's the status of my endpoint?", "Is my endpoint erroring?", "My endpoint failed — why?", "How many instances are running behind my endpoint?", "Is the latency my model or SageMaker?", "Show me the container logs for my endpoint." NOT for training-job issues, endpoint deletion, scaling changes, or new deployments. |
Rules
Service boundary. This skill covers SageMaker, including deployment of a customized model to Bedrock. Bedrock-native work (invoking Bedrock's managed foundation models, Bedrock Agents, Knowledge Bases, Guardrails, AgentCore) is supported by AWS but has no validated workflow in this skill: let the user know and proceed with best-effort guidance using general AWS knowledge. If the user asks for a model on AWS without naming a service, compare SageMaker and Bedrock or ask one service-choice question. Do not select a service by default. The two services have different model catalogs, deployment paths, and pricing, and a default selection can send the user into an incorrect workflow.
Progressive disclosure. Load only the reference folder relevant to the current user intent. Do not load all references at once.
Best-effort help. If the user's request falls outside this skill's references, do not dead-end the conversation. Help them using general AWS knowledge and documentation, and inform the user that the guidance is not covered by this skill's validated workflows.
Usage attribution. Before running any AWS CLI command or packaged script, set
export AWS_SDK_UA_APP_ID=AWSSkill-SageMaker.
Files
- ATTRIBUTION.mdAdded by AutoGineer883 B
- LICENSELicence file added by AutoGineer9.9 KiB
- NOTICELicence file added by AutoGineer67 B
- SKILL.md8.1 KiB
- references/dataset-evaluation/overview.md5.4 KiB
- references/dataset-evaluation/references/custom-scorer-evaluation-dataset-formats.md3.6 KiB
- references/dataset-evaluation/references/strategy_data_requirements.md4.8 KiB
- references/dataset-evaluation/scripts/format_detector.pyScript26.7 KiB
- references/dataset-transformation/code_templates/transformation.pyScript1.5 KiB
- references/dataset-transformation/overview.md13.3 KiB
- references/dataset-transformation/references/code_output_guide.md2.8 KiB
- references/dataset-transformation/references/dataset_transformation_code.md5.2 KiB
- references/dataset-transformation/references/sagemaker_dataset_formats.md2.9 KiB
- references/dataset-transformation/scripts/transformation_tools.pyScript4.8 KiB
- references/directory-management/overview.md1.9 KiB
- references/endpoint-diagnostics/overview.md3.5 KiB
- references/endpoint-diagnostics/references/minimum_iam_policy.md1.2 KiB
- references/endpoint-diagnostics/scripts/collect_diagnostics.pyScript12.8 KiB
- references/finetuning-technique/overview.md2.1 KiB
- references/finetuning-technique/references/finetune_technique_selection_guide.md2.3 KiB
- references/finetuning-technique/scripts/get_recipes.pyScript817 B
- references/finetuning/code_templates/dpo.pyScript6.4 KiB
- references/finetuning/code_templates/rlaif_builtin.pyScript5.8 KiB
- references/finetuning/code_templates/rlaif_custom_prompt.pyScript6.3 KiB
- references/finetuning/code_templates/rlvr.pyScript6.5 KiB
- references/finetuning/code_templates/sft.pyScript6.1 KiB
- references/finetuning/overview.md11.4 KiB
- references/finetuning/references/code_output_guide.md3.2 KiB
- references/finetuning/references/continuous_customization.md8.4 KiB
- references/finetuning/references/eula_links.md7.3 KiB
- references/finetuning/references/rlaif_guide.md3.8 KiB
- references/finetuning/references/rlvr_reward_function.md10.0 KiB
- references/finetuning/scripts/mlflow_reference.pyScript841 B
- references/finetuning/templates/nova_rlvr_reward_function_source_template.pyScript12.8 KiB
- references/finetuning/templates/rlvr_reward_function_source_template.pyScript9.1 KiB
- references/manage-mlflow/overview.md4.8 KiB
- references/manage-mlflow/references/app-creation-workflow.md3.2 KiB
- references/manage-mlflow/references/app-deletion-workflow.md2.3 KiB
- references/manage-mlflow/references/app-discovery-workflow.md1.5 KiB
- references/manage-mlflow/references/app-update-workflow.md1.7 KiB
- references/manage-mlflow/references/sm-mlflow-guide.md1.3 KiB
- references/model-deployment/code_templates/deploy-jumpstart-sagemaker.pyScript7.5 KiB
- references/model-deployment/code_templates/deploy-nova-bedrock.pyScript2.0 KiB
- references/model-deployment/code_templates/deploy-nova-sagemaker.pyScript1.9 KiB
- references/model-deployment/code_templates/deploy-oss-bedrock.pyScript4.0 KiB
- references/model-deployment/code_templates/deploy-oss-sagemaker.pyScript2.6 KiB
- references/model-deployment/code_templates/optimize-benchmark.pyScript4.6 KiB
- references/model-deployment/code_templates/optimize-huggingface-to-s3.pyScript2.0 KiB
- references/model-deployment/code_templates/optimize-recommendation.pyScript6.4 KiB
- references/model-deployment/overview.md16.6 KiB
- references/model-deployment/references/code_output_guide.md3.2 KiB
- references/model-deployment/references/deploy-jumpstart-sagemaker.md14.3 KiB
- references/model-deployment/references/deploy-nova-bedrock.md4.4 KiB
- references/model-deployment/references/deploy-nova-sagemaker.md5.8 KiB
- references/model-deployment/references/deploy-oss-bedrock.md5.2 KiB
- references/model-deployment/references/deploy-oss-sagemaker.md12.9 KiB
- references/model-deployment/references/model-licenses.md11.9 KiB
- references/model-deployment/references/optimization/benchmark-workflow.md12.7 KiB
- references/model-deployment/references/optimization/benchmarking-guidance.md8.9 KiB
- references/model-deployment/references/optimization/huggingface-to-s3.md3.1 KiB
- references/model-deployment/references/optimization/interpreting-results.md8.5 KiB
- references/model-deployment/references/optimization/recommendation-deploy.md1.5 KiB
- references/model-deployment/references/optimization/recommendation-options.md3.1 KiB
- references/model-deployment/references/optimization/recommendation-workflow.md10.3 KiB
- references/model-evaluation/code_templates/custom_scorer_evaluator.pyScript2.5 KiB
- references/model-evaluation/code_templates/llmaaj_evaluator.pyScript2.8 KiB
- references/model-evaluation/overview.md5.2 KiB
- references/model-evaluation/references/code_output_guide.md3.3 KiB
- references/model-evaluation/references/create-reward-function.md3.0 KiB
- references/model-evaluation/references/custom-lambda-scorer.md4.4 KiB
- references/model-evaluation/references/custom-scorer-evaluation.md10.5 KiB
- references/model-evaluation/references/evaluation-type-guide.md10.1 KiB
- references/model-evaluation/references/llmaaj-builtin-evaluation.md5.0 KiB
- references/model-evaluation/references/llmaaj-custom-evaluation.md2.5 KiB
- references/model-evaluation/references/llmaaj-evaluation.md14.6 KiB
- references/model-evaluation/references/supported-judge-models.md2.9 KiB
- references/model-evaluation/scripts/nova_reward_function_source_template.pyScript13.1 KiB
- references/model-evaluation/scripts/reward_function_source_template.pyScript9.2 KiB
- references/model-evaluation/scripts/validate_custom_metrics.pyScript3.5 KiB
- references/model-selection/overview.md10.3 KiB
- references/model-selection/references/benchmarks/agentic-index.md2.8 KiB
- references/model-selection/references/benchmarks/coding-index.md2.8 KiB
- references/model-selection/references/benchmarks/gpqa.md2.8 KiB
- references/model-selection/references/benchmarks/hle.md2.8 KiB
- references/model-selection/references/benchmarks/ifbench.md2.8 KiB
- references/model-selection/references/benchmarks/intelligence-index.md2.8 KiB
- references/model-selection/references/benchmarks/mmmu-pro.md3.0 KiB
- references/model-selection/references/benchmarks/tau2.md2.8 KiB
- references/model-selection/references/model-licenses.md7.3 KiB
- references/model-selection/references/model-selection.md8.2 KiB
- references/model-selection/references/select-for-deployment.md19.8 KiB
- references/model-selection/scripts/filter_deployable_models.pyScript10.1 KiB
- references/model-selection/scripts/get_deployable_models.pyScript21.7 KiB
- references/model-selection/scripts/get_model_names.pyScript1.4 KiB
- references/model-selection/scripts/search_deployable_models.pyScript34.9 KiB
- references/planning/overview.md10.3 KiB
- references/planning/references/deploy-base-model-plan.md699 B
- references/planning/references/evaluate-first-plan.md1.9 KiB
- references/planning/references/input-output-contracts.md3.1 KiB
- references/planning/references/model-customization-plan.md1.5 KiB
- references/planning/references/region-availability.md1.2 KiB
- references/planning/references/skill-routing-constraints.md2.7 KiB
- references/processing-jobs/code_templates/dataset_definition_processing.pyScript4.1 KiB
- references/processing-jobs/code_templates/framework_processing.pyScript3.0 KiB
- references/processing-jobs/code_templates/script_processing.pyScript2.9 KiB
- references/processing-jobs/code_templates/spark_processing.pyScript3.7 KiB
- references/processing-jobs/overview.md6.7 KiB
- references/processing-jobs/references/configuration.md4.6 KiB
- references/processing-jobs/references/troubleshooting.md2.4 KiB
- references/sdk-getting-started/overview.md643 B
- references/sdk-getting-started/references/execution-role-setup.md2.4 KiB
- references/sdk-getting-started/references/sagemaker-python-sdk-setup.md2.1 KiB
- references/training-jobs/code_templates/custom_training.pyScript6.2 KiB
- references/training-jobs/code_templates/jumpstart_training.pyScript5.6 KiB
- references/training-jobs/code_templates/recipe_training.pyScript5.7 KiB
- references/training-jobs/overview.md9.1 KiB
- references/training-jobs/references/configuration.md4.0 KiB
- references/training-jobs/references/troubleshooting.md4.4 KiB
- references/use-case-specification/overview.md2.0 KiB
- references/use-case-specification/references/spec-for-deployment.md4.0 KiB
- references/use-case-specification/references/spec-for-finetuning.md2.1 KiB
references/model-selection/references/model-selection.md
Model Selection
Help the user choose a base model for finetuning. This reference applies to both Nova and OSS model paths — benchmark data helps compare models regardless of family.
Select the most relevant benchmark(s), then provide the user with a list of available models listed in performance order for that benchmark.
Understanding the use case
Use what you already know from the conversation — the user may have described their task, domain, data, or goals. Make sure you're familiar with use_case_spec.md if it exists. Be sure to think about the success criteria from the user, if any has been documented. If you still don't have enough to map to relevant benchmarks, ask the user to describe their use case and success criteria in more detail.
Making a recommendation
Selecting relevant benchmark(s)
Read the benchmark descriptions below and identify which 2-3 benchmarks are most relevant to the user's task.
Understand the usecase: Model Selection is an essential step in model customization. You MUST make sure you understand the customer's use case well enough to select the proper benchmark. If you do not understand the usecase in enough detail, you MUST ask follow-up questions.
Criteria for Benchmark Selection: Picking the most relevant benchmarks is essential, because the user bases their model selection on this. You must think carefully about which benchmarks are really the most fitting. Frequently, intelligence index is the best, so make sure there is a good reason to rank anything else above that. Think about which benchmarks are testing model functionality that is the most relevant to what the user is trying to do.
Data Source: Artificial Analysis (artificialanalysis.ai). Independent evaluator, consistent methodology across all models.
Intelligence Index
Artificial Analysis's composite overall quality score, combining multiple benchmarks into a single ranking. It's an index on AA's own scale where higher is better.
Consider this: As a default ranking, when no other benchmark seems to fit. This MUST always be in the top 2-3 recommended benchmarks, because it's relevant to all tasks.
GPQA
The hardest 198 questions from the GPQA benchmark — graduate-level multiple-choice in biology, physics, and chemistry, written by PhD-level domain experts. Questions are "Google-proof" — skilled non-experts with unrestricted web access score only 34%, so the benchmark tests genuine scientific reasoning rather than information retrieval.
Consider this: For tasks related to scientific reasoning, particularly in the fields of biology, physics, and chemistry. Could be a good proxy for general scientific knowledge or the ability to think critically or logically.
IF-Bench
Tests precise control over text output — the model must satisfy unusual, mechanically verifiable constraints like placing a specific word at an exact position in a sentence, using exactly N numbers, or ensuring no two consecutive words share the same first letter. 58 novel constraint types designed to be harder than standard instruction-following tests.
Consider this: For any task requiring tight control over model output, where the user has very specific requirements about features that must be present in the model output. Could be a good proxy for instruction following, although the benchmark is really about precise instruction following in unusually difficult contexts
MMMU-Pro
College-level multimodal questions requiring both vision and reasoning across six disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering. Image types include charts, diagrams, tables, maps, and chemical structures. The "Pro" version filters out questions answerable without the image, so it specifically tests integrated visual-textual understanding.
Consider this: For any task combining visual and textual inputs — chart/diagram interpretation, visual reasoning, document understanding. Also relevant for comparing overall model capability, since VLMs that score well here tend to perform well on text-only tasks too.
τ²-bench
Simulates multi-turn customer service conversations where both the agent and a simulated user actively modify a shared state (e.g., databases, account records). The agent must follow domain-specific policies, use API tools, and resolve customer requests. Scores are from the telecom domain.
Consider this: For situations where accurate tool use is important, for multi-turn conversation, and situations where instruction following or adherence to business logic is important. Be careful not to over-index on this for customer service conversations.
HLE (Humanity's Last Exam)
2,500 expert-written questions across dozens of subjects including mathematics, humanities, and natural sciences, designed to be at the frontier of human knowledge. Questions can't be answered via internet retrieval. Scores are very low — most models cluster in the single digits.
Consider this: For tasks where reasoning, the ability to think logically, and the ability to draw connections between different concepts to form an analysis or conclusion is important. This benchmark may not show a large difference between most models.
Coding Index
Artificial Analysis's composite coding score, combining Terminal-Bench Hard (agentic software engineering, system administration, and data processing in terminal environments) and SciCode (scientist-curated coding problems across 16 scientific disciplines).
Consider this: For software engineering tasks, scientific/research computing, system administration automation, any task where the model needs to write and/or execute code .
Agentic Index
Artificial Analysis's composite agentic score, combining GDPval-AA (real-world tasks across 44 occupations and 9 industries, with shell access and web browsing in an agentic loop) and τ²-bench Telecom.
Consider this: For autonomous agents, workflow automation, tool-using assistants, any use case where the model needs to independently use tools, browse the web, or carry out multi-step tasks.
Presenting relevant benchmarks
After picking 2-3 relevant benchmarks from the above list (including Intelligence Index), present them in ranked order to the user, with a short explanation of what the benchmark is, and why you think it's relevant:
"In order to select a model, it's helpful to look at performance on public benchmarks. After considering several benchmarks, I think these are the most relevant to your task: " "1. [benchmark]: [short description of benchmark]. [Why this is relevant] "2. [benchmark]: [short description of benchmark]. [Why this is relevant]
(Include a 3rd only if you think there are 3 relevant benchmarks)
"Which of these do you think are relevant? After you pick, I'll show you the model rankings."
'' wait for user
Presenting Models for Selection
After the user tells you which benchmarks they think are relevant, you need to present the models from the corresponding benchmark file in /benchmarks. Present the table exactly as it appears in the benchmark table, skipping any rows for models that are not in the user's list of available models. VLMs MUST be included in all benchmark tables where they have scores — do not filter them out based on the user's task being text-only. Double check your work to avoid hallucinations:
"[Benchmark 1:] "[table from benchmark file]
"========================="
"[Benchmark 2:] "[table from benchmark file]
Give a 1-2 sentence analysis of what these benchmarks tell us. Keep in mind that most users balance performance and cost requirements.
Ask the user to select a model:
"Given this information, which model would you like to select?"
''wait for user
If the user has any questions about model selection, answer them to the best of your ability, leaning on the benchmarks as much as possible and being transparent about your knowledge gaps and confidence.
Once the user has chosen a model, consider that the chosen base model. This workflow is complete.
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Needs Python
Security
What our scanner noticed· Low risk
- Instructions that use the network
- Scripts that your agent may run
This skill includes scripts that your agent may run. Read them before you use it.
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Published by
- AWS
- Publisher
- AWS (Official)
- Source
- aws/agent-toolkit-for-aws · skills/core-skills/aws-ai-ml · commit 0a218f8
- Licence
- Apache License 2.0 (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE)
- NOTICE (added by AutoGineer from NOTICE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
- Also distributed in
- aws/agent-toolkit-for-aws
AutoGineer is not affiliated with or endorsed by AWS.
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download aws-ai-ml.zip