AutoGineer
Generated banner for Model Evaluation
Generated logo for Model Evaluation

Model Evaluation

AWS LabsOfficialPublished by AWS Labs

  • Skill
  • Workflow
  • Software development
  • Apache-2.0
  • Low risk
  • Has scripts
Download ZIP

From the author

Generates python code that evaluates SageMaker models. Supports two evaluation types: LLM-as-Judge and Custom Scorer. Use when the user says "evaluate my model", "run a benchmark", "test model performance", "how did my model perform", "compare models", or other similar requests.
— the description by AWS Labs

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured workflow for evaluating SageMaker models, supporting both open-source and Nova models. It guides users through selecting an evaluation strategy—either LLM-as-Judge for qualitative assessment or a Custom Scorer for programmatic validation—and generates the necessary Python code or Jupyter notebooks to execute these benchmarks within the SageMaker environment.

What it does

  • Determines the appropriate evaluation type based on model architecture and user requirements.
  • Validates compatibility between the chosen evaluation method and the target model.
  • Generates Python scripts or Jupyter notebooks for model evaluation.
  • Supports LLM-as-Judge for OSS models.
  • Facilitates the creation and registration of Lambda-based custom scorers.
  • Integrates with SageMaker SDK for streamlined model benchmarking.

When to use it

  • When you need to benchmark the performance of a fine-tuned or base SageMaker model.
  • When you want to use an LLM to grade model outputs.
  • When you require custom programmatic logic to score model responses.
  • When you need to compare different model versions or configurations.

Advantages

  • Ensures consistent evaluation workflows through guided steps.
  • Prevents incompatible configurations by validating model types early.
  • Automates the generation of boilerplate code for evaluation tasks.
  • Provides flexibility by supporting both automated LLM grading and custom logic.

Usage examples

“Evaluate my model.”
The agent will initiate the evaluation workflow, asking you to choose between LLM-as-Judge or Custom Scorer, and verify your model type.
“How did my model perform?”
The agent will guide you through setting up a benchmark or custom scorer to generate performance metrics.
“Run a benchmark on my fine-tuned Llama model.”
The agent will confirm the model type and guide you through the LLM-as-Judge or Custom Scorer setup process.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/awslabs/model-evaluation/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip model-evaluation.zip -d .agents/skills

Documentation by AWS Labs

Model Evaluation

Generate code that evaluates a SageMaker model.

Prerequisites

  • The SDK environment has been verified (SDK version, region, execution role). If not done, activate the sdk-getting-started skill first.

Principles

  1. One thing at a time. Each response advances exactly one decision. Never combine multiple questions in a single turn.

  2. Confirm before proceeding. Wait for the user to agree before moving to the next step.

  3. Don't read files until you need them. Only read reference files when you've reached the step that requires them.

  4. Don't ask what you already know. If the answer is in conversation history, workflow_state.json, plan.md, or any file you've already read — use it. Confirm if unsure, but don't re-ask.

  5. No narration. Share outcomes and ask questions. Keep responses short.

  6. No repetition. If you said something before a tool call, don't repeat it after.

Scope

This skill supports the evaluation feature for SageMaker Serverless Model Customization. It can evaluate any base or fine-tuned model supported by SageMaker serverless model customization — both OSS models (Llama, Mistral, Qwen, etc.) and Nova models.

Tell the user when the skill is activated:

"I can help evaluate any base or fine-tuned model supported by SageMaker serverless model customization."

If the user requests help evaluating a model that isn't supported by SageMaker serverless model customization, explain that it is not supported by this skill.

Evaluation Types

There are two evaluation types:

  • LLM-as-Judge — an LLM grades your model's responses. (OSS models only — not supported for Nova.)

  • Custom Scorer — programmatic evaluation via Lambda function (includes built-in math and code scorers). Works with both OSS and Nova models.

Workflow

Step 1: Determine evaluation type

Do you already know which evaluation type to use?

Check conversation history, plan.md, workflow_state.json, or anything else you've already read.

If yes: confirm with the user.

"It sounds like you want to run [evaluation type]. Is that right?"

⏸ Wait for confirmation. If confirmed → go to Step 2.

If no: ask.

"What kind of evaluation would you like to run? I support:

  1. LLM-as-Judge — an LLM grades your model's responses

  2. Custom Scorer — programmatic scoring (math, code, or your own logic)

Pick one, or say 'help me decide' if you're not sure."

⏸ Wait for user.

  • If user picks one → go to Step 2.

  • If user indicates uncertainty, by saying something like "help me decide," "whatever you think," "I'm not sure" → read references/evaluation-type-guide.md and follow its instructions. It will guide the user to a choice and then return here. You MUST NEVER make a recommendation to the user on eval type without reading references/evaluation-type-guide.md.

Step 2: Validate and hand off to evaluation workflow

Before reading the reference file, validate that the chosen evaluation type is compatible with the user's situation. You may already know these answers from conversation context — don't ask if you don't need to.

LLM-as-Judge validation

  1. What model type are we evaluating? LLM-as-Judge is not supported for Nova models. To determine model type (if you don't already know it):

    • If you have the training job name or ARN, use the AWS MCP tool list-tags on the training job ARN and look for the sagemaker-studio:jumpstart-model-id tag. Contains "nova" → Nova. Anything else → OSS.

    • If you have a Model Package ARN, use the AWS MCP tool describe-model-package and check the model description or source tags.

    • If neither is available, ask the user.

  2. Does the user have an evaluation dataset? LLM-as-Judge requires one.

Custom Scorer validation

  1. Does the user have an evaluation dataset? Custom Scorer requires one. (Works with both OSS and Nova models, though for Nova only custom lambdas are supported.)

If validation fails, tell the user which requirement(s) aren't met and offer alternatives:

"[Evaluation type] won't work because [reason]."

If the failure reason was lack of an eval dataset, there's nothing we can do. Inform the user:

"Unfortunately all of the supported eval types require an eval dataset. I can't help you with model evaluation."

If the failure reason is something else, offer to help them pick a different evaluation type.

⏸ Wait for user.

If they say they do want help choosing a different eval type → read references/evaluation-type-guide.md.

If validation passes, read the corresponding reference file:

User choseRead
LLM-as-Judgereferences/llmaaj-evaluation.md
Custom Scorerreferences/custom-scorer-evaluation.md

Follow the reference file's instructions from the beginning.

Files

references/llmaaj-builtin-evaluation.md

LLM-as-Judge with Built-in Metrics: Alignment Guide

This file describes the process for aligning on built-in metrics to use for model evaluations with LLMaaJ.

Select Metrics

Refer to the metrics tables below for the full list of metrics with descriptions and common combinations.

Based on the user's task and data, recommend specific metrics with reasoning:

"Based on your [task], I recommend these metrics:

  • [metric1]: [why it matters for this task]

Does this look good, or do you want to consider other metrics?"

⏸ Wait for user to confirm.

Tips:

  • Start with the common combinations from the metrics file as a baseline

  • Adjust based on what you know about the user's task and data

  • If the user pushes back, understand why and adjust — don't just agree

LLM-as-Judge Built-in Metrics

SageMaker provides 11 built-in metrics for LLM-as-Judge evaluation, organized into Quality and Responsible AI categories.

Quality Metrics

MetricDescriptionWhen to Use
CorrectnessMeasures if the model's response to the prompt is correct. If a reference response (ground truth) is provided in the dataset, the evaluator considers this when scoring.QA, math problems, factual tasks
CompletenessMeasures how well the model's response answers every question in the prompt. If a reference response is provided, the evaluator considers this when scoring.Multi-part questions, comprehensive answers, summarization
FaithfulnessIdentifies whether the response contains information not found in the prompt to measure how faithful the response is to the available context.RAG applications, context-grounded responses
HelpfulnessMeasures how helpful the model's response is using factors including whether it follows instructions, is sensible and coherent, and anticipates implicit needs.General assistance, customer service, broad evaluation
CoherenceMeasures how coherent the response is by identifying logical gaps, inconsistencies, and contradictions.Long-form content, reasoning tasks, explanations
RelevanceMeasures how relevant the answer is to the prompt.All tasks - commonly used baseline metric
FollowingInstructionsMeasures how well the model's response respects the exact directions found in the prompt.Instruction-following tasks, structured outputs, specific formatting
ProfessionalStyleAndToneMeasures how appropriate the response's style, formatting, and tone is for a professional setting.Business communications, formal writing

Responsible AI Metrics

MetricDescriptionWhen to Use
HarmfulnessEvaluates whether the response contains harmful content.Safety evaluation, content moderation
StereotypingEvaluates whether content in the response contains stereotypes of any kind (either positive or negative).Fairness evaluation, bias detection
RefusalDetermines if the response directly declines to answer the prompt or rejects the request by providing reasons.Safety evaluation, understanding model boundaries

Usage in Code

In code, these metrics are specified as Builtin.Correctness, Builtin.Completeness, etc. When discussing with users, use natural language names.

Common Metric Combinations

  • QA/Math tasks → Correctness, Completeness, Faithfulness, Relevance

  • Summarization → Completeness, Coherence, Relevance

  • General assistance → Helpfulness, Relevance, FollowingInstructions

  • Safety evaluation → Harmfulness, Stereotyping, Refusal

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Low risk

  • Instructions that use the network
  • Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
AWS Labs
Publisher
AWS Labs (Official)
Source
awslabs/agent-plugins · plugins/sagemaker-ai/skills/model-evaluation · commit fbefb37
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • NOTICE (added by AutoGineer from NOTICE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by AWS Labs.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download model-evaluation.zip