Dataset Evaluation

AWS LabsOfficialPublished by AWS Labs

Skill
Review and audit
Software development
Apache-2.0
Low risk
Has scripts

From the author

Validates dataset formatting and quality for SageMaker model fine-tuning (SFT, DPO, or RLVR). Use when the user says "is my dataset okay", "evaluate my data", "check my training data", "I have my own data", or before starting any fine-tuning job. Detects file format, checks schema compliance against the selected model and technique, and reports whether the data is ready for training or evaluation.

— the description by AWS Labs

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a systematic workflow to verify that your training or evaluation datasets meet the specific schema and formatting requirements for Amazon SageMaker model fine-tuning (SFT, DPO, RLVR) or custom scorer evaluation. It automates the detection of file formats and ensures compliance with the selected model family and training strategy, helping you identify potential issues before initiating expensive training jobs.

What it does

Locates and verifies access to local or S3-based datasets.
Detects file formats and validates them against strategy-specific requirements.
Checks schema compliance for training datasets (SFT, DPO, RLVR).
Validates evaluation datasets for Prime Math, Prime Code, or Custom Lambda scorers.
Identifies mismatches between dataset structure and the intended fine-tuning strategy.
Provides guidance on necessary transformations if the data format is invalid.

When to use it

Before starting any model fine-tuning job on SageMaker.
When you need to confirm if your custom dataset is correctly formatted for a specific training strategy.
When preparing evaluation datasets for custom scorers like Prime Math or Prime Code.
When you are unsure if your data meets the requirements for Nova or Open Weights models.

Advantages

Prevents failed training jobs by catching formatting errors early.
Ensures alignment between data structure and the chosen fine-tuning strategy.
Supports both training and evaluation dataset validation workflows.
Provides clear feedback on whether data is ready for production use.

Usage examples

“Is my dataset okay for SFT training?”: The agent will locate your dataset, verify it against the SFT schema for your selected model, and confirm if it is ready for training.
“Check my training data for a DPO job.”: The agent will validate your file structure to ensure it contains the required prompt, chosen, and rejected fields.
“I have my own data for Prime Math evaluation, can you check it?”: The agent will verify your dataset against the Prime Math schema, ensuring it contains the correct query and response fields.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/awslabs/dataset-evaluation/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip dataset-evaluation.zip -d .agents/skills

Documentation by AWS Labs

Workflow Instruction

Follow the workflow shown below. Locate the dataset, check the file type, and resolve any issues with missing files or wrong file types. Determine the fine-tuning model and fine-tuning strategy. Run the appropriate validation based on the model family. Summarize the results: is the dataset ready for fine-tuning?

Prerequisites

The SDK environment has been verified (SDK version, region, execution role). If not done, activate the sdk-getting-started skill first.

Workflow

Locate Dataset:
- The full path may be a local file path, or an S3 URI
- Resolve the full path to the dataset file, make sure read permissions are available, and help the user if the file is not found
Determine strategy and model:
- File formatting depends on the currently selected fine-tuning strategy and fine-tuning base model.
- If the strategy and model are already known from the conversation context (e.g., selected via the model-selection and finetuning-technique skills), use them.
- If not available in context, activate the model-selection and/or finetuning-technique skills to determine them before proceeding.
- Exception: If the user is validating an evaluation dataset (not a training dataset), neither model nor technique is required — the format detector can validate eval format (query/response structure) independently. Do not block on model-selection or finetuning-technique for eval dataset validation.
Check File Formatting: Run the tool format_detector.py to make sure the file conforms to formatting requirements.
- Send the full path directly to the format_detector script as an argument
- Do not send the model and strategy as arguments
- Do not download data from S3
- Do not make local copies of data
Summarize Results: Tell the user if their data is ready
- Examine the output of format_detector and compare to the known strategy and model
- Important: training datasets and evaluation datasets have different format requirements.
  - Training datasets must match the fine-tuning strategy format per references/strategy_data_requirements.md
  - Evaluation datasets (for model evaluation) must match one of the SageMaker evaluation dataset formats.
  - Custom Scorer evaluation datasets have scorer-specific requirements. If the dataset is intended for Custom Scorer evaluation (Prime Math, Prime Code, or Custom Lambda), read references/custom-scorer-evaluation-dataset-formats.md and validate against the scorer-specific schema. The scorer type should be known from conversation context (determined in the model-evaluation skill).
- Report back to the user if their current dataset is valid for its intended purpose
- Warn the user if their dataset is valid, but for a different strategy or model
- Warn the user if their dataset is not valid for any strategy/model pair
- If the user plans to finetune a model with the evaluated dataset, it needs to be uploaded to an S3 bucket in the same region as the planned training job (usually the default region). Warn the user if this is NOT the case.
- If the dataset is NOT in the necessary format, recommend transforming it using the dataset-transformation skill, wait for user confirmation, and update the plan based on their response

Messages to the User

Introduction: "This skill checks the structure of your dataset for model fine-tuning."
File types: This skill applies to files that are formatted according to the Amazon SageMaker AI Developer Guide

Resources

scripts/format_detector.py is self-contained format validation script that can be run independently
model-selection and finetuning-technique skills should have already determined the base model and fine-tuning strategy
references/strategy_data_requirements.md contains data format requirements per strategy

Script Details

scripts/format_detector.py is self-contained format validation script that can be run independently:

# With the file path argument identified in workflow step 1
python scripts/format_detector.py local_path/to/dataset

References

scripts/format_detector.py — Self-contained format validation script
references/strategy_data_requirements.md — Data format requirements per strategy

Files

ATTRIBUTION.mdAdded by AutoGineer920 B
LICENSELicence file added by AutoGineer9.9 KiB
NOTICELicence file added by AutoGineer89 B
SKILL.md4.9 KiB
references/custom-scorer-evaluation-dataset-formats.md4.1 KiB
references/strategy_data_requirements.md4.8 KiB
scripts/format_detector.pyScript25.0 KiB

references/strategy_data_requirements.md

Finetuning Strategy Data Requirements

Critical Nova models have a different set of formats than open weights models. Make sure you refer to the right section based on the user's base model.

Open Weights Models Data Format by Strategy (Llama, Qwen, GPT-OSS, etc.)

SFT (Supervised Fine-Tuning)

Required format:

{
  "prompt": "",
  "completion": ""
}

What it needs:

Input-output pairs
Single "correct" response per input
Consistent quality across examples

DPO (Direct Preference Optimization)

Required format:

{
  "prompt": "",
  "chosen": "",
  "rejected": ""
}

What it needs:

Input with two responses: preferred (chosen) and dispreferred (rejected)
Clear preference signal between responses
Both responses should be plausible but one is better
Avoiding unintentional length bias

RLVR (Reinforcement Learning from Verifiable Rewards)

Required format:

{
  "data_source": "",
  "prompt": [
    {
      "content": "",
      "role": ""
    }
  ],
  "ability": "",
  "reward_model": {
    "ground_truth": "",
    "style": ""
  }
}

What it needs:

user prompt
Ground truth responses in reward_model.ground_truth field (leave empty if user data does not have responses)

How it works:

Model generates response for input
Lambda receives full user prompt + reward model fields
Lambda computes reward (uses ground_truth if included in verification logic)
Model learns to maximize rewards

RLAIF (Reinforcement Learning from AI Feedback)

RLAIF uses the same base schema as RLVR. The ability and reward_model.style fields determine which evaluator is used.

Base schema:

{
  "data_source": "",
  "prompt": [
    {
      "role": "",
      "content": ""
    }
  ],
  "ability": "",
  "reward_model": {
    "style": "",
    "ground_truth": ""
  }
}

Built-in Evaluators

`ability`	`reward_model.style`	Use case
`pairwise-judging`	`llmj`	Compare two model responses and pick the better one
`chain-of-thought`	`llmj-cot`	Evaluate quality of step-by-step reasoning
`faithfulness`	`llmj-faithfulness`	Check if response stays grounded in provided context
`summarization`	`llmj-summarization`	Evaluate quality of a generated summary

pairwise-judging — prompt must include both responses to compare; ground_truth is the preferred response index + reasoning.

chain-of-thought / faithfulness / summarization — prompt contains the task; ground_truth is the reference answer or source text.

Custom Evaluator

Set reward_model.style to llmj-custom and supply a Jinja2 prompt template. The template receives {{ prompt }}, {{ response }}, and optional {{ ground_truth }} as variables. The LLM judge must return a JSON object with a score field (0.0–1.0).

{
  "data_source": "",
  "prompt": [
    {
      "role": "user",
      "content": ""
    }
  ],
  "ability": "chain-of-thought",
  "reward_model": {
    "style": "llmj-custom",
    "ground_truth": ""
  }
}

The custom Jinja prompt is provided separately at training time (not embedded in the dataset). It must instruct the judge to return exactly: {"score": <0.0-1.0>, ...}.

Nova Models Data Format by Strategy

SFT (Supervised Fine-Tuning)

{
  "schemaVersion": "bedrock-conversation-2024",
  "system": [
    {
      "text": ""
    }
  ],
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "text": ""
        }
      ]
    },
    {
      "role": "assistant",
      "content": [
        {
          "text": ""
        }
      ]
    }
  ]
}

DPO (Direct Preference Optimization)

The format is the same as SFT for the first N-1 turns. The final assistant turn uses candidates with preferenceLabel instead of regular content.

{
  "schemaVersion": "bedrock-conversation-2024",
  "system": [
    {
      "text": ""
    }
  ],
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "text": ""
        }
      ]
    },
    {
      "role": "assistant",
      "candidates": [
        {
          "content": [
            {
              "text": ""
            }
          ],
          "preferenceLabel": "preferred"
        },
        {
          "content": [
            {
              "text": ""
            }
          ],
          "preferenceLabel": "non-preferred"
        }
      ]
    }
  ]
}

RLVR

{
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant."
    },
    {
      "role": "user",
      "content": "Hello!"
    }
  ],
  "reference_answer": {
    "answer": "49"
  }
}

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Low risk

Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

AWS Labs

Publisher

AWS Labs (Official)

Source

awslabs/agent-plugins · plugins/sagemaker-ai/skills/dataset-evaluation · commit fbefb37

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
NOTICE (added by AutoGineer from NOTICE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by AWS Labs.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download dataset-evaluation.zip

From the author

AutoGineer summary

What it does

When to use it

Advantages

Usage examples

Install

Documentation by AWS Labs

Workflow Instruction

Prerequisites

Workflow

Messages to the User

Resources

Script Details

References

Files

references/strategy_data_requirements.md

Finetuning Strategy Data Requirements

Open Weights Models Data Format by Strategy (Llama, Qwen, GPT-OSS, etc.)

SFT (Supervised Fine-Tuning)

DPO (Direct Preference Optimization)

RLVR (Reinforcement Learning from Verifiable Rewards)

RLAIF (Reinforcement Learning from AI Feedback)

Built-in Evaluators

Custom Evaluator

Nova Models Data Format by Strategy

SFT (Supervised Fine-Tuning)

DPO (Direct Preference Optimization)

RLVR

Requirements and compatibility

Security

Attribution and licence

Related skills

Migrate To Shoehorn

Tdd

Test Driven Development

Webapp Testing

Test Driven Development

Browser Testing With Devtools

Download ZIP