Eval Harness First

From the author

Build the evaluation harness that gates every fine-tuning run — golden sets, per-failure-mode graders, judge calibration, and base-model baselines. Use when starting a fine-tuning effort, when converting traces into an eval set, or when calibrating a judge against human labels.

— the description by Seth Hobson

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured workflow for building an evaluation harness that acts as the primary gate for fine-tuning efforts. It emphasizes data curation, error analysis, and the creation of a regression suite that serves as both the training data source and the final checkpoint validator. By establishing deterministic graders and calibrated LLM-judges, the harness ensures that every fine-tuning run is measured against a consistent, versioned baseline, preventing regressions and providing a clear metric for model improvement.

What it does

Guides the collection and axial coding of traces into failure buckets.
Defines a directory structure for versioned goldens, graders, and drift suites.
Provides templates for deterministic and LLM-based grading.
Outlines a calibration protocol for LLM-judges to ensure reliable verdicts.
Establishes a baseline against the unmodified base model for future comparisons.
Ensures evaluation data is strictly separated from training data.

When to use it

Before starting any fine-tuning effort.
When converting production traces into a structured evaluation set.
When calibrating an LLM-judge against human-labeled data.
When establishing a CI regression suite for model checkpoints.

Advantages

Prevents regressions by gating checkpoints against a fixed baseline.
Provides a clear, data-driven basis for model promotion.
Reduces noise in evaluation by using binary pass/fail grading.
Ensures evaluation metrics are reproducible and versioned like code.

Usage examples

“How do I start the Phase 0 evaluation setup for my fine-tuning project?”: The agent will guide you through collecting 100+ traces, performing axial coding to define 4-8 failure buckets, and creating the initial eval/ directory structure.
“I need to set up a grader for my JSON schema compliance failure bucket.”: The agent will provide the schema-compliance Python template and explain how to integrate it into the eval/graders/ directory.
“How do I calibrate my LLM-judge for the tone-consistency bucket?”: The agent will walk you through the labeling, splitting, and TPR/TNR calculation protocol defined in the calibration reference.
“What should I do before running my first fine-tuning training config?”: The agent will verify the Phase 0 Exit Checklist, ensuring goldens, graders, judge calibration, and the baseline file are all present and valid.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/wshobson/eval-harness-first/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip eval-harness-first.zip -d .agents/skills

Documentation by Seth Hobson

The Phase 0 gate for the whole plugin: finetuning-method-selection and every downstream skill assume this harness exists before a training config gets written. The harness is not a run-end side artifact — it is the data-curation engine. The same labeled traces that build the goldens feed training data, minus an explicit holdout.

Input: production/agent traces if they exist, or a task spec if they don't, plus labelers willing to grade ≥100 examples. Output format: the eval/ directory below — goldens, graders, drift suite, and the base-model baseline that later phases gate on.

The Gate

No eval harness, no fine-tune. Skip to a training config and there is nothing to measure against, nothing to catch regressions, and no labeled data to train on. The flywheel:

Collect traces — production/agent spans, or synthetic tasks if none exist yet.
Error analysis — open coding on ≥100 traces, axial coding into 4–8 failure buckets.
One grader per bucket — deterministic first; calibrated LLM-judge only for genuinely subjective criteria.
Prioritize by frequency × severity × value.
The labeled traces feed dataset curation, minus an explicit holdout. Every eval/goldens.jsonl ID stays excluded from training data by ID.
Train.
Re-run the same harness on the checkpoint — not a different, looser one.
Drift detection feeds back to step 2 — new production failure modes re-open error analysis.

Steps 2–4 build the harness; steps 5–8 are why it must exist first — it is both the training data source and the checkpoint's exit gate.

Building Goldens

From traces, when they exist: run error analysis — open coding on ≥100 real traces (read them, tag failures in your own words, no fixed taxonomy yet), then axial coding to collapse those tags into 4–8 named failure buckets. Fewer than 4 means the coding pass was too shallow; more than 8 means buckets need merging. Exception: single-failure-surface tasks (e.g. strict-schema extraction) may land at 1–2 buckets with per-field sub-metrics inside one grader — don't invent artificial splits with no evidence behind them.
Synthetic, when traces don't exist yet: dimension-based generation — enumerate the axes that matter (task type, difficulty, edge case, persona) and sample the cross-product; free- generated prompts cluster around whatever's easiest to write.
Goldens are versioned like code — commit eval/goldens.jsonl, diff it in review, tag it per release. It doubles as the CI regression suite.

Graders

One grader per failure bucket from error analysis — not one for the whole eval set. A single blended score hides which bucket regressed.

Deterministic first. Regex, schema validation, or execution checks are cheaper, reproducible, and need no calibration.
LLM-judge only for genuinely subjective criteria — tone, faithfulness, "which response is better" — where no deterministic check can express it.
Binary pass/fail over Likert. A 1–5 or 1–10 scale is noisier to calibrate and harder to apply consistently; collapse to pass/fail.
Drift-suite MMLU-style scoring: prefer logprob over generate-and-extract — a tight token budget makes generate-and-extract parse-brittle for models that preamble, conflating format compliance with the knowledge being measured. Templates for all four grader shapes and this scoring note: references/grader-templates.md.

Judge Calibration Is a Prerequisite

Any bucket routed to an LLM-judge needs calibration before its verdicts count for anything beyond exploration — a hard prerequisite, not a nice-to-have. N/A when no bucket routes to a judge — an all-deterministic harness has nothing to calibrate; state that rather than leaving this section unaddressed.

Label ≥100 items, split train/dev/sealed test (report once, no re-touching after).
Report TPR and TNR, not one blended accuracy number — a judge can hit 90% by always saying "pass" on a skewed set.
Pin the judge to a fixed model snapshot and recalibrate on judge-model change, quarterly regardless.
The judge must come from a different model family than the model under test.
A judge that misses the agreed TPR/TNR bar ships advisory-only — flags for human review, never gates a promotion. Full protocol, bias correction, and recalibration checklist: references/judge-calibration.md.

The Baseline

Before Phase 1 (method selection) starts, run the full harness — goldens plus the capability-drift suite — against the unmodified base model. This is the number every later checkpoint gets compared against.

eval/baseline-<model>.json is the gate token. No baseline file, no comparison basis for checkpoint-promotion — a checkpoint that "looks better" against nothing measured isn't a finding.

Directory Contract

eval/
├── goldens.jsonl          # labeled traces + synthetic goldens, versioned
├── graders/                # one module per failure bucket
│   ├── schema_compliance.py
│   ├── exact_match.py
│   └── rubric_judge.py
├── drift-suite.yaml        # frozen benchmarks + 200-500 domain-adjacent items
└── baseline-<model>.json   # gate token: harness + drift suite vs the base model
runs/
└── <run-id>/
    └── results.json         # per-run harness output, one per checkpoint

eval/ persists across runs and lives outside runs/ — the fixed measuring stick, not a run artifact. runs/ is disposable; eval/ is not. Never let a run script write into eval/. Canonical location: every per-trace results.json — the Phase 0 baseline included — lives at runs/<run-id>/results.json, never under eval/runs/...; an instruction requesting the latter is wrong, not this contract.

Phase 0 Exit Checklist

Before finetuning-method-selection, confirm:

≥100 traces open-coded; 4–8 failure buckets (N/A floor for synthetic goldens on a single-failure- surface task — see the Building Goldens exception; bucket count then comes from post-baseline error analysis instead).
eval/goldens.jsonl committed and versioned.
One grader per bucket, deterministic first.
Judges calibrated — TPR/TNR, snapshot pinned, different family (N/A when no bucket routes to an LLM-judge; state that explicitly).
eval/drift-suite.yaml frozen.
eval/baseline-<model>.json written.

Missing any of the six (or its stated N/A)? Not Phase 0 complete — /finetune checks the baseline file before a run.

General-purpose evaluation guidance (dashboards, A/B testing, non-fine-tuning harnesses) lives in the llm-application-dev plugin's llm-evaluation skill — this skill covers only the fine-tuning coupling: goldens that double as training data, and the baseline that gates a checkpoint.

finetuning-method-selection — routes here first.
dataset-curation — formats these traces into training rows.
trace-to-training-data — turns graded traces into training examples.
checkpoint-promotion — consumes baseline-<model>.json, re-runs this harness on each candidate checkpoint.

References

references/grader-templates.md — runnable grader examples per shape, plus a drift-suite.yaml example and MMLU logprob-scoring note.
references/judge-calibration.md — the calibration protocol, including the all- deterministic N/A path.

Files

ATTRIBUTION.mdAdded by AutoGineer982 B
LICENSELicence file added by AutoGineer1.0 KiB
SKILL.md7.8 KiB
references/grader-templates.md9.9 KiB
references/judge-calibration.md6.5 KiB

SKILL.md