AutoGineer
Generated banner for Model Evaluation
Generated logo for Model Evaluation

Model Evaluation

AWS LabsOfficialPublished by AWS Labs

  • Skill
  • Workflow
  • Software development
  • Apache-2.0
  • Low risk
  • Has scripts
Download ZIP

From the author

Generates python code that evaluates SageMaker models. Supports two evaluation types: LLM-as-Judge and Custom Scorer. Use when the user says "evaluate my model", "run a benchmark", "test model performance", "how did my model perform", "compare models", or other similar requests.
— the description by AWS Labs

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured workflow for evaluating SageMaker models, supporting both open-source and Nova models. It guides users through selecting an evaluation strategy—either LLM-as-Judge for qualitative assessment or a Custom Scorer for programmatic validation—and generates the necessary Python code or Jupyter notebooks to execute these benchmarks within the SageMaker environment.

What it does

  • Determines the appropriate evaluation type based on model architecture and user requirements.
  • Validates compatibility between the chosen evaluation method and the target model.
  • Generates Python scripts or Jupyter notebooks for model evaluation.
  • Supports LLM-as-Judge for OSS models.
  • Facilitates the creation and registration of Lambda-based custom scorers.
  • Integrates with SageMaker SDK for streamlined model benchmarking.

When to use it

  • When you need to benchmark the performance of a fine-tuned or base SageMaker model.
  • When you want to use an LLM to grade model outputs.
  • When you require custom programmatic logic to score model responses.
  • When you need to compare different model versions or configurations.

Advantages

  • Ensures consistent evaluation workflows through guided steps.
  • Prevents incompatible configurations by validating model types early.
  • Automates the generation of boilerplate code for evaluation tasks.
  • Provides flexibility by supporting both automated LLM grading and custom logic.

Usage examples

“Evaluate my model.”
The agent will initiate the evaluation workflow, asking you to choose between LLM-as-Judge or Custom Scorer, and verify your model type.
“How did my model perform?”
The agent will guide you through setting up a benchmark or custom scorer to generate performance metrics.
“Run a benchmark on my fine-tuned Llama model.”
The agent will confirm the model type and guide you through the LLM-as-Judge or Custom Scorer setup process.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/awslabs/model-evaluation/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip model-evaluation.zip -d .agents/skills

Documentation by AWS Labs

Model Evaluation

Generate code that evaluates a SageMaker model.

Prerequisites

  • The SDK environment has been verified (SDK version, region, execution role). If not done, activate the sdk-getting-started skill first.

Principles

  1. One thing at a time. Each response advances exactly one decision. Never combine multiple questions in a single turn.

  2. Confirm before proceeding. Wait for the user to agree before moving to the next step.

  3. Don't read files until you need them. Only read reference files when you've reached the step that requires them.

  4. Don't ask what you already know. If the answer is in conversation history, workflow_state.json, plan.md, or any file you've already read — use it. Confirm if unsure, but don't re-ask.

  5. No narration. Share outcomes and ask questions. Keep responses short.

  6. No repetition. If you said something before a tool call, don't repeat it after.

Scope

This skill supports the evaluation feature for SageMaker Serverless Model Customization. It can evaluate any base or fine-tuned model supported by SageMaker serverless model customization — both OSS models (Llama, Mistral, Qwen, etc.) and Nova models.

Tell the user when the skill is activated:

"I can help evaluate any base or fine-tuned model supported by SageMaker serverless model customization."

If the user requests help evaluating a model that isn't supported by SageMaker serverless model customization, explain that it is not supported by this skill.

Evaluation Types

There are two evaluation types:

  • LLM-as-Judge — an LLM grades your model's responses. (OSS models only — not supported for Nova.)

  • Custom Scorer — programmatic evaluation via Lambda function (includes built-in math and code scorers). Works with both OSS and Nova models.

Workflow

Step 1: Determine evaluation type

Do you already know which evaluation type to use?

Check conversation history, plan.md, workflow_state.json, or anything else you've already read.

If yes: confirm with the user.

"It sounds like you want to run [evaluation type]. Is that right?"

⏸ Wait for confirmation. If confirmed → go to Step 2.

If no: ask.

"What kind of evaluation would you like to run? I support:

  1. LLM-as-Judge — an LLM grades your model's responses

  2. Custom Scorer — programmatic scoring (math, code, or your own logic)

Pick one, or say 'help me decide' if you're not sure."

⏸ Wait for user.

  • If user picks one → go to Step 2.

  • If user indicates uncertainty, by saying something like "help me decide," "whatever you think," "I'm not sure" → read references/evaluation-type-guide.md and follow its instructions. It will guide the user to a choice and then return here. You MUST NEVER make a recommendation to the user on eval type without reading references/evaluation-type-guide.md.

Step 2: Validate and hand off to evaluation workflow

Before reading the reference file, validate that the chosen evaluation type is compatible with the user's situation. You may already know these answers from conversation context — don't ask if you don't need to.

LLM-as-Judge validation

  1. What model type are we evaluating? LLM-as-Judge is not supported for Nova models. To determine model type (if you don't already know it):

    • If you have the training job name or ARN, use the AWS MCP tool list-tags on the training job ARN and look for the sagemaker-studio:jumpstart-model-id tag. Contains "nova" → Nova. Anything else → OSS.

    • If you have a Model Package ARN, use the AWS MCP tool describe-model-package and check the model description or source tags.

    • If neither is available, ask the user.

  2. Does the user have an evaluation dataset? LLM-as-Judge requires one.

Custom Scorer validation

  1. Does the user have an evaluation dataset? Custom Scorer requires one. (Works with both OSS and Nova models, though for Nova only custom lambdas are supported.)

If validation fails, tell the user which requirement(s) aren't met and offer alternatives:

"[Evaluation type] won't work because [reason]."

If the failure reason was lack of an eval dataset, there's nothing we can do. Inform the user:

"Unfortunately all of the supported eval types require an eval dataset. I can't help you with model evaluation."

If the failure reason is something else, offer to help them pick a different evaluation type.

⏸ Wait for user.

If they say they do want help choosing a different eval type → read references/evaluation-type-guide.md.

If validation passes, read the corresponding reference file:

User choseRead
LLM-as-Judgereferences/llmaaj-evaluation.md
Custom Scorerreferences/custom-scorer-evaluation.md

Follow the reference file's instructions from the beginning.

Files

references/custom-lambda-scorer.md

Custom Lambda Scorer

This file guides you through resolving a custom Lambda scorer (evaluator) for use with Custom Scorer evaluation.

Resolve evaluator

For this step, you need: the evaluator ARN of a registered reward function.

Check if you already know this from conversation context (e.g., the user mentioned a reward function ARN, or one was used in a previous evaluation). If so, confirm and return to the main workflow.

If not, ask:

"Do you have an existing reward function registered in SageMaker? If so, what's the evaluator ARN?"

If the user has an ARN, validate it:

  • It should look like: arn:aws:sagemaker:REGION:ACCOUNT:hub-content/.../JsonDoc/NAME/VERSION

  • Validate by splitting the part after hub-content/ into HUB_NAME/JsonDoc/CONTENT_NAME/VERSION and calling:

    aws sagemaker describe-hub-content --hub-name HUB_NAME --hub-content-type JsonDoc --hub-content-name CONTENT_NAME --hub-content-version VERSION --region REGION

    If the call succeeds, HubContentStatus is Available, and HubContentSearchKeywords includes @evaluatortype:rewardfunction, the evaluator is valid.

If validation fails, tell the user what went wrong:

  • API call errors → "That ARN doesn't seem to exist. Could you double-check it?"

  • Status is not Available → "That evaluator exists but isn't ready (status: [status]). It may still be provisioning."

  • Missing @evaluatortype:rewardfunction → "That resource exists but doesn't appear to be a reward function evaluator. Could you verify you have the right ARN?"

In any failure case, offer to re-enter the ARN or fall back to a built-in scorer.

If the user doesn't have one:

"You don't have a registered reward function yet. I can help you create one — I'll provide a template with your scoring logic and register it as a SageMaker Hub Evaluator. Or you can use a built-in scorer instead.

  1. Create a new reward function — I'll walk you through it

  2. Use a built-in scorer — Prime Math or Prime Code

Which would you prefer?"

  • If create new → read references/create-reward-function.md and follow its instructions. It will produce an evaluator ARN. Once complete, return here and proceed to "After resolution".

  • If built-in → return to the main Custom Scorer workflow and switch to the built-in scorer path.

After resolution

Once you have the evaluator ARN, return to the main Custom Scorer workflow.

Lambda input/output contracts

Lambda return format

The return format depends on the model type:

For OSS models:

# <RETURN_FORMAT> — OSS models
return {
    "statusCode": 200,
    "headers": {"Content-Type": "application/json"},
    "body": json.dumps([result])  # body is a JSON STRING
}

For Nova models:

# <RETURN_FORMAT> — Nova models
return {
    "statusCode": 200,
    "headers": {"Content-Type": "application/json"},
    "body": [result]  # body is a PARSED LIST (not json.dumps)
}

Each result object has the shape:

{
  "id": "sample_id",
  "aggregate_reward_score": 0.85,
  "metrics_list": [{ "name": "metric_name", "value": 0.75, "type": "Metric" }]
}
Lambda input format

The input format depends on the model type:

For OSS models (gen_qa path):

[{
  "id": "hash",
  "model_response": "model's generated text",
  "query": "the prompt",
  "response": "the gold answer from dataset",
  "reference_answer": { "text": "the gold answer from dataset" },
  "metadata": {},
  "processor_config": {}
}]

For Nova models (rft_eval path):

[{
  "id": "sample_id",
  "messages": [
    { "role": "user", "content": "the prompt" },
    { "role": "assistant", "content": "model's generated output" }
  ],
  "reference_answer": "the gold answer from dataset"
}]

To extract the model response from Nova input: read the last message with role: "assistant".

Evaluator registration

CustomScorerEvaluator requires a Hub Content ARN (registered via Evaluator.create()), NOT a raw Lambda ARN.

from sagemaker.ai_registry.evaluator import Evaluator
from sagemaker.ai_registry.air_constants import REWARD_FUNCTION

evaluator = Evaluator.create(
    name="my-reward-function",
    source="path/to/reward_function.py",
    type=REWARD_FUNCTION
)
# Use evaluator.arn as the evaluator parameter

Using a raw Lambda ARN (e.g., arn:aws:lambda:...) will fail with Invalid HubContentArn format.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Low risk

  • Instructions that use the network
  • Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
AWS Labs
Publisher
AWS Labs (Official)
Source
awslabs/agent-plugins · plugins/sagemaker-ai/skills/model-evaluation · commit fbefb37
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • NOTICE (added by AutoGineer from NOTICE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by AWS Labs.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download model-evaluation.zip