AutoGineer
Generated banner for Eval Driven Dev
Generated logo for Eval Driven Dev

Eval Driven Dev

GitHubOfficialFirst committed by yiouliDistributed by GitHub

  • Skill
  • Workflow
  • Software development
  • MIT
  • Low risk
  • Has scripts
Download ZIP

From the author

Improve AI application with evaluation-driven development. Define eval criteria, instrument the application, build golden datasets, observe and evaluate application runs, analyze results, and produce a concrete action plan for improvements. ALWAYS USE THIS SKILL when the user asks to set up QA, add tests, add evals, evaluate, benchmark, fix wrong behaviors, improve quality, or do quality assurance for any Python project that calls an LLM model.
— the description by yiouli

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured, six-step procedure to build an automated evaluation pipeline for Python-based AI applications. It focuses on testing the application's end-to-end logic—including request handling, context assembly, and routing—rather than just the LLM itself. By instrumenting the application, creating golden datasets, and using evaluators to score outputs, developers can move from anecdotal testing to data-driven quality assurance. The workflow emphasizes using real LLM calls and production-like data to ensure that evaluation results are meaningful and actionable.

What it does

  • Analyzes project capabilities and defines specific evaluation criteria based on real-world failure modes.
  • Instruments application data boundaries using wrap() to enable controlled input injection and output capture.
  • Implements a Runnable interface to execute the application end-to-end without mocking core components.
  • Captures reference traces to inform the creation of diverse, realistic test datasets.
  • Executes automated tests via pixie test and generates structured, data-backed analysis reports.
  • Produces a prioritized action plan for improving application quality based on evaluation outcomes.

When to use it

  • When you need to set up QA or benchmarking for a Python project that integrates an LLM.
  • When you need to fix inconsistent behaviors or improve the quality of an AI application.
  • When you want to transition from manual testing to a repeatable, automated evaluation pipeline.
  • When you need to validate that application logic, prompt assembly, and routing are performing as expected.

Advantages

  • Ensures evaluation is performed on the actual application code path, not mocked versions.
  • Provides a systematic, repeatable process for identifying and addressing specific failure modes.
  • Bridges the gap between raw test scores and actionable development tasks.
  • Standardizes the approach to quality assurance across different LLM-powered projects.

Usage examples

“Set up an evaluation pipeline for my customer support chatbot to ensure it handles account lookup failures correctly.”
The agent will analyze the chatbot's entry points, define evaluation criteria for account lookup, instrument the code, and create a dataset of scenarios to benchmark the bot's performance.
“My research agent is hallucinating citations. Can you help me add tests to fix this?”
The agent will identify the research synthesis capability, define a factuality evaluation criterion, and build a test suite that specifically targets citation accuracy to produce an improvement plan.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/github/eval-driven-dev/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip eval-driven-dev.zip -d .agents/skills

Documentation by yiouli

Eval-Driven Development for Python LLM Applications

You're building an automated evaluation pipeline that tests a Python-based AI application end-to-end — running it the same way a real user would, with real inputs — then scoring the outputs using evaluators and producing pass/fail results via pixie test.

What you're testing is the app itself — its request handling, context assembly (how it gathers data, builds prompts, manages conversation state), routing, and response formatting. The app uses an LLM, which makes outputs non-deterministic — that's why you use evaluators (LLM-as-judge, similarity scores) instead of assertEqual — but the thing under test is the app's code, not the LLM.

During evaluation, the app's own code runs for real — routing, prompt assembly, LLM calls, response formatting — nothing is mocked or stubbed. But the data the app reads from external sources (databases, caches, third-party APIs, voice streams) is replaced with test-specified values via instrumentations. This means each test case controls exactly what data the app sees, while still exercising the full application code path.

Rule: The app's LLM calls must go to a real LLM. Do not replace, mock, stub, or intercept the LLM with a fake implementation. The LLM is the core value-generating component — replacing it makes the eval tautological (you control both inputs and outputs, so scores are meaningless). If the project's test suite contains LLM mocking patterns, those are for the project's own unit tests — do NOT adopt them for the eval Runnable.

The deliverable is a working pixie test run with real scores — not a plan, not just instrumentation, not just a dataset.

This skill is about doing the work, not describing it. Read code, edit files, run commands, produce a working pipeline.

Before you start

First, activate the virtual environment. Identify the correct virtual environment for the project and activate it. After the virtual environment is active, run the setup.sh included in the skill's resources. The script updates the eval-driven-dev skill and pixie-qa python package to latest version, initialize the pixie working directory if it's not already initialized, and start a web server in the background to show user updates.

Setup error handling — what you can skip vs. what must succeed:

  • Skill update fails → OK to continue. The existing skill version is sufficient.

  • pixie-qa upgrade fails but was already installed → OK to continue with the existing version.

  • pixie-qa is NOT installed and installation fails → STOP. Ask the user for help. The workflow cannot proceed without the pixie package.

  • pixie init fails → STOP. Ask the user for help.

  • pixie start (web server) fails → STOP. Ask the user for help. Check server.log in the pixie root directory for diagnostics. Common causes: port conflict, missing dependency, slow environment. Do NOT proceed without the web server — the user needs it to see eval results.

The workflow

Follow Steps 1–6 straight through without stopping. Do not ask the user for confirmation at intermediate steps — verify each step yourself and continue.

How to work — read this before doing anything else:

  • One step at a time. Read only the current step's instructions. Do NOT read Steps 2–6 while working on Step 1.

  • Read references only when a step tells you to. Each step names a specific reference file. Read it when you reach that step — not before.

  • Create artifacts immediately. After reading code for a sub-step, write the output file for that sub-step before moving on. Don't accumulate understanding across multiple sub-steps before writing anything.

  • Verify, then move on. Each step has a checkpoint. Verify it, then proceed to the next step. Don't plan future steps while verifying the current one.

When to stop and ask for help:

Some blockers cannot and should not be worked around. When you encounter any of the following, stop immediately and ask the user for help — do not attempt workarounds:

  • Application won't run due to missing environment variables or configuration: The app requires environment variables or configuration that are not set and cannot be inferred. Do NOT work around this by mocking, faking, or replacing application components — the eval must exercise real production code. Ask the user to fix the environment setup.

  • App import failures that indicate a broken project: If the app's core modules cannot be imported due to missing system dependencies or incompatible Python versions (not just missing pip packages you can install), ask the user to fix the project setup.

  • Ambiguous entry point: If the app has multiple equally plausible entry points and the project analysis doesn't clarify which one matters most, ask the user which to target.

Blockers you SHOULD resolve yourself (do not ask): missing Python packages (install them), missing pixie package (install it), port conflicts (pick a different port), file permission issues (fix them).

Run Steps 1–6 in sequence. If the user's prompt makes it clear that earlier steps are already done (e.g., "run the existing tests", "re-run evals"), skip to the appropriate step. When in doubt, start from Step 1.

Step 1: Understand the app and define eval criteria

First, check the user's prompt for specific requirements. Before reading app code, examine what the user asked for:

  • Referenced documents or specs: Does the prompt mention a file to follow (e.g., "follow the spec in EVAL_SPEC.md", "use the methodology in REQUIREMENTS.md")? If so, read that file first — it may specify datasets, evaluation dimensions, pass criteria, or methodology that override your defaults.

  • Specified datasets or data sources: Does the prompt reference specific data files (e.g., "use questions from eval_inputs/research_questions.json", "use the scenarios in call_scenarios.json")? If so, read those files — you must use them as the basis for your eval dataset, not fabricate generic alternatives.

  • Specified evaluation dimensions: Does the prompt name specific quality aspects to evaluate (e.g., "evaluate on factuality, completeness, and bias", "test identity verification and tool call correctness")? If so, every named dimension must have a corresponding evaluator in your test file.

If the prompt specifies any of the above, they take priority. Read and incorporate them before proceeding.

Step 1 has three sub-steps. Each reads its own reference file and produces its own output file. Complete each sub-step fully before starting the next.

Sub-step 1a: Project analysis

Reference: Read references/1-a-project-analysis.md now.

Before looking at code structure or entry points, understand what this software does in the real world — its purpose, its users, the complexity of real inputs, and where it fails. This understanding drives every downstream decision: which entry points matter most, what eval criteria to define, what trace inputs to use, and what dataset entries to create. Write the detailed context file before moving on. Note: the project may contain tests/, fixtures/, examples/, mock servers, and documentation — these are the project's own development infrastructure, NOT data sources for your eval pipeline. Ignore them when sourcing trace inputs and dataset content.

Checkpoint: pixie_qa/00-project-analysis.md written — covering what the software does, target users, capability inventory (at least 3 capabilities if the project has them), realistic input characteristics, and hard problems / failure modes (at least 2).

Sub-step 1b: Entry point & execution flow

Reference: Read references/1-b-entry-point.md now.

Read the source code to understand how the app starts and how a real user invokes it. Use the capability inventory from pixie_qa/00-project-analysis.md to prioritize entry points — focus on the entry point(s) that exercise the most valuable capabilities, not just the first one found. Write the detailed context file before moving on.

Checkpoint: pixie_qa/01-entry-point.md written — covering entry point, execution flow, user-facing interface, and env requirements.

Sub-step 1c: Eval criteria

Reference: Read references/1-c-eval-criteria.md now.

Define the app's use cases and eval criteria. Derive use cases from the capability inventory in pixie_qa/00-project-analysis.md. Derive eval criteria from the hard problems / failure modes — not generic quality dimensions. Use cases drive dataset creation (Step 4); eval criteria drive evaluator selection (Step 3). Write the detailed context file before moving on.

Checkpoint: pixie_qa/02-eval-criteria.md written — covering use cases, eval criteria, and their applicability scope. Do NOT read Step 2 instructions yet.

Step 2: Instrument, run application, and capture a reference trace

Step 2 has three sub-steps. Each reads its own reference file. Complete each sub-step before starting the next.

Sub-step 2a: Instrument with wrap

Reference: Read references/2a-instrumentation.md now.

Add wrap() calls at the app's data boundaries so the eval harness can inject controlled inputs and capture outputs. This makes the app testable without changing its logic.

Checkpoint: wrap() calls added at all data boundaries. Every eval criterion from pixie_qa/02-eval-criteria.md has a corresponding data point.

Sub-step 2b: Implement the Runnable

Reference: Read references/2b-implement-runnable.md now.

Write a Runnable class that lets the eval harness invoke the app exactly as a real user would. The Runnable should be simple — it just wires up the app's real entry point to the harness interface. If it's getting complicated, something is wrong.

Checkpoint: pixie_qa/run_app.py written. The Runnable calls the app's real entry point with real LLM configuration — no mocking, no faking, no component replacement.

Sub-step 2c: Capture and verify a reference trace

Reference: Read references/2c-capture-and-verify-trace.md now.

Run the app through the Runnable and capture a trace. The trace proves instrumentation and the Runnable are working correctly, and provides the data shapes needed for dataset creation in Step 4.

Checkpoint: pixie_qa/reference-trace.jsonl exists. All expected wrap entries and llm_span entries appear. pixie format shows all data points needed for evaluation. Do NOT read Step 3 instructions yet.

Step 3: Define evaluators

Reference: Read references/3-define-evaluators.md now for the detailed sub-steps.

Goal: Turn the qualitative eval criteria from Step 1c into concrete, runnable scoring functions. Each criterion maps to either a built-in evaluator, an agent evaluator (the default for any semantic or qualitative criterion), or a manual custom function (only for mechanical/deterministic checks like regex or field existence). The evaluator mapping artifact bridges between criteria and the dataset, ensuring every quality dimension has a scorer. Select evaluators that measure the hard problems identified in pixie_qa/00-project-analysis.md — not just generic quality dimensions.

Checkpoint: All evaluators implemented. pixie_qa/03-evaluator-mapping.md written with criterion-to-evaluator mapping and decision rationale. Do NOT read Step 4 instructions yet.

Step 4: Build the dataset

Reference: Read references/4-build-dataset.md now for the detailed sub-steps.

Goal: Create the test scenarios that tie everything together — the runnable (Step 2), the evaluators (Step 3), and the use cases (Step 1c). Each dataset entry defines what to send to the app, what data the app should see from external services, and how to score the result. Use the reference trace from Step 2 as the source of truth for data shapes and field names. Cover entries from the capability inventory in pixie_qa/00-project-analysis.md and include entries targeting the failure modes identified there. Do NOT use the project's own test fixtures, mock servers, or example data as dataset eval_input content — source real-world data instead. Every wrap(purpose="input") in the app must have pre-captured content in each entry's eval_input — do NOT leave eval_input empty when the app has input wraps.

Checkpoint: Dataset JSON created at pixie_qa/datasets/<name>.json with diverse entries covering all use cases. Dataset realism audit passed — entries use real-world data at representative scale, no project test fixtures contamination, at least one entry targets a failure mode with uncertain outcome, and every eval_input has captured content for all input wraps. Do NOT read Step 5 instructions yet.

Step 5: Run pixie test and fix mechanical issues

Reference: Read references/5-run-tests.md now for the detailed sub-steps.

Goal: Execute the full pipeline end-to-end and get it running without mechanical errors. This step is strictly about fixing setup and data issues in the pixie QA components (dataset, runnable, custom evaluators) — NOT about fixing the application itself or evaluating result quality. Once pixie test completes without errors and produces real evaluator scores for every entry, this step is done.

Checkpoint: pixie test runs to completion. Every dataset entry has evaluator scores (real EvaluationResult or PendingEvaluation). No setup errors, no import failures, no data validation errors.

If the test errors out, that's a mechanical bug in your QA components — fix and re-run. But once tests produce scores, move on. Do NOT assess result quality here — that's Step 6.

Always proceed to Step 6 after tests produce scores. Analysis is the essential final step — without it, pending evaluations are never completed and the user gets uninterpreted raw scores with no actionable insights. Do NOT stop here and ask the user whether to continue.

Cycle rule for iterative runs: Every successful pixie test invocation creates a concrete pixie_qa/results/<test_id> directory and starts a new analysis cycle. Before you edit application code, prompts, datasets, evaluators, or rerun pixie test, complete Step 6 for that exact results directory. Do not skip earlier cycles and analyze only the last run.

Step 6: Analyze outcomes

Reference: Read references/6-analyze-outcomes.md now — it has the complete three-phase analysis process, writing guidelines, and output format requirements.

Goal: Analyze pixie test results in a structured, data-driven process to produce actionable insights on test case quality, evaluator quality, and application quality. This step completes pending evaluations, writes per-entry and per-dataset analysis, and produces a prioritized action plan. Every statement must be backed by concrete data from the evaluation run — no speculation, no hand-waving.

Persisted analysis artifacts: In this trimmed workflow, persist analysis only at the dataset level and test-run level. Those artifacts still use a detailed version (for agent consumption: data points, evidence trails, reasoning chains) plus a summary version (for human review: concise TLDR readable in under 2 minutes). Do not create per-entry analysis files.

Hard completion gate: Step 6 is not complete until all of the following are true:

  • Every "status": "pending" entry in every pixie_qa/results/<test_id>/dataset-*/entry-*/evaluations.jsonl has been replaced with a scored result containing score and reasoning.

  • Every dataset directory has analysis.md and analysis-summary.md.

  • The test run root has action-plan.md and action-plan-summary.md.

  • You have run the Step 6 verifier script from this skill's resources/ directory against pixie_qa/results/<test_id>, and it reports success.

Explicitly not sufficient:

  • Writing a single top-level file such as pixie_qa/06-analysis.md

  • Saying pending evaluations are for the user to review in the web UI

  • Saying an entry "likely passes" without updating evaluations.jsonl

Web Server Management

pixie-qa runs a web server in the background for displaying context, traces, and eval results to the user. It's automatically started by the setup script (via pixie start, which launches a detached background process and returns immediately).

When the user is done with the eval-driven-dev workflow, inform them the web server is still running and you can clean it up with:

pixie stop

IMPORTANT: after the web server is stopped, the web UI becomes inaccessible. So only stop the server if the user confirms they're done with all web UI features. If they want to keep using the web UI, do NOT stop the server.

And whenever you restart the workflow, always run the setup.sh script in resources again to ensure the web server is running:

Files

references/6-analyze-outcomes.md

Step 6: Analyze Outcomes

Why this step: pixie test produced raw scores. Now you analyze those results to understand what they mean — completing pending evaluations, identifying patterns, validating hypotheses, and producing an actionable improvement plan. The analysis is structured in three phases that build on each other: entry-level → dataset-level → action plan.

Result directory structure

After pixie test, the result directory looks like:

{PIXIE_ROOT}/results/<test_id>/
  meta.json
  dataset-{idx}/
    metadata.json
    entry-{idx}/
      config.json              # evaluators, description, expectation
      eval-input.jsonl         # input data fed to evaluators
      eval-output.jsonl        # output data captured from app
      evaluations.jsonl        # scored + pending evaluations
      trace.jsonl              # LLM call traces

Read meta.json to find the <test_id>. All the data you need for analysis is in this directory.

Hard completion gate

You are the grader for Step 6. Pending evaluations are not a handoff to the user, and the web UI is not a substitute for grading. You may use the web UI to browse traces and outputs, but completion happens by writing files on disk.

Step 6 is incomplete until all of the following are true:

  • Every "status": "pending" entry in every evaluations.jsonl has been replaced with a scored entry that contains both score and reasoning.

  • Every dataset directory contains analysis.md and analysis-summary.md.

  • The test run root contains action-plan.md and action-plan-summary.md.

  • The verifier script in this skill's resources/ directory passes for the target results directory.

Forbidden shortcuts:

  • Leaving any "status": "pending" entries in place

  • Telling the user to review pending evaluations in the web UI

  • Writing a single top-level substitute file such as pixie_qa/06-analysis.md

  • Writing phrases like "likely passes" or "probably fails" without scoring the evaluation and updating evaluations.jsonl

If you do any of the above, Step 6 is not done.

Iteration rule

If you are iterating across multiple fix/test cycles, every successful pixie test run creates a new pixie_qa/results/<test_id> directory and a new Step 6 obligation. The moment that directory exists, it becomes the analysis target for the current cycle.

Before you edit application code, prompts, datasets, evaluators, or rerun pixie test, complete Step 6 for that exact results directory. Do not skip earlier cycles and analyze only the last run.

Additional forbidden shortcut:

  • Do not create a newer pixie_qa/results/<test_id> and leave an older one from the same task without Step 6 artifacts.

Writing principles

Every analysis detailed artifact you produce must follow these principles:

  • Data-driven: Every opinion or statement must be backed by concrete data from the evaluation run. Quote scores, cite entry indices, reference specific eval input/output content. No hand-waving. It is better to write nothing than to write something unsubstantiated.

  • Evidence-first: Present the raw data and evidence before drawing conclusions. The reader (another coding agent) should be able to independently verify your conclusions from the evidence you cite.

  • Traceable: For every conclusion, provide the chain: data source → observation → reasoning → conclusion. Another agent should be able to follow this chain backward to verify or challenge any claim.

  • No selling: Do not advocate, promote, or use value-laden language ("excellent", "robust", "impressive", "well-designed"). State what the data shows and what actions it implies. Let the reader form quality judgments.

  • Action-oriented: Every analysis should contribute to the end goal of concrete improvements to the evaluation pipeline or application. Do not write observations that don't lead somewhere.

Every persisted analysis summary artifact must follow these principles:

  • Concise: The human reader should be able to understand the key findings and actions in under 2 minutes for any single artifact.

  • Conclusions-first: Lead with what the reader needs to know (results, findings, actions), not with methodology or background.

  • Plain language: Avoid jargon. A non-technical stakeholder should be able to follow the summary.

  • Consistent: Summary conclusions must match the detailed version's evidence. Never add claims in the summary that aren't supported in the detailed version.

Dual-variant pattern

Every persisted analysis artifact in this step has two files:

ArtifactDetailed file (for agent)Summary file (for human)
Dataset analysisdataset-{idx}/analysis.mddataset-{idx}/analysis-summary.md
Action planaction-plan.mdaction-plan-summary.md

Always write the detailed version first, then derive the summary from it. The summary is a strict subset of the detailed version's content — it should never contain claims or conclusions not present in the detailed version.

Phase 1: Entry-level grading pass

Process each dataset entry individually. For each dataset-{idx}/entry-{idx}/:

1a. Read the entry data

Read these files for the entry:

  • config.json — what evaluators were configured, the description, the expectation

  • eval-input.jsonl — what data was fed to the app/evaluators

  • eval-output.jsonl — what the app produced

  • evaluations.jsonl — current evaluation results (scored and pending)

  • trace.jsonl — what LLM calls the app made (if available)

1b. Complete pending evaluations

If evaluations.jsonl contains entries with "status": "pending", you must grade them:

  1. Read the criteria field of the pending evaluation

  2. Apply the criteria to the entry's eval input, eval output, and trace data

  3. Assign a score between 0.0 and 1.0:

    • 1.0 — fully meets the criteria

    • 0.5–0.9 — partially meets criteria (explain what's missing)

    • 0.0–0.4 — does not meet criteria

  4. Write a reasoning string (1–3 sentences citing specific evidence from the output or trace)

  5. Replace the pending entry in evaluations.jsonl with the scored result. Do not append a second row and leave the pending row in place. Overwrite the pending row itself.

Before (pending):

{
  "evaluator": "ResponseQuality",
  "status": "pending",
  "criteria": "The response should..."
}

After (scored):

{
  "evaluator": "ResponseQuality",
  "score": 0.85,
  "reasoning": "Response addresses the main question but omits..."
}

Grading guidelines:

  • Be evidence-based — every score must reference specific output or trace content

  • Use the criteria literally — do not expand or reinterpret beyond what's written

  • Consider the trace — distinguish between app logic problems and LLM quality issues

  • Be calibrated — reserve 1.0 for outputs that genuinely satisfy criteria fully

  • Do not penalize LLM non-determinism — different phrasing of a correct answer is not a failure

  • Do not defer to the user — if the evidence is sufficient to write "likely passes", it is sufficient to assign a score and update evaluations.jsonl

1c. Do not persist entry-level analysis files

In this trimmed workflow, do not write entry-{idx}/analysis.md or entry-{idx}/analysis-summary.md. Phase 1 is only for reading evidence and converting every pending evaluation into a scored row in evaluations.jsonl.

You may take temporary scratch notes while reasoning, but they are not deliverables. Persist only:

  • updated evaluations.jsonl in each entry directory

  • dataset-level analysis files in Phase 2

  • run-level action plan files in Phase 3

Phase 2: Dataset-level analysis

After all entries in a dataset are analyzed, produce the dataset-level analysis. Write analysis.md in the dataset directory (dataset-{idx}/analysis.md).

2a. Aggregate the data

Summarize across all entries in the dataset:

  • Pass/fail counts and overall pass rate

  • Per-evaluator statistics (pass rate, min/max/mean scores)

  • Which entries failed which evaluators (failure clusters)

2b. Form and validate hypotheses

Come up with exactly 3 high-confidence hypotheses across these three dimensions:

  1. Test cases quality — Does the set of test cases sufficiently and efficiently verify the application's capabilities? Does it cover the important failure modes? Are there blind spots?

  2. Evaluation criteria/evaluator quality — Do the evaluators have proper granularity and grading to catch real issues? Are there rubber-stamp evaluators (all 1.0)? Are there flaky evaluators (high variance without code changes)? Are criteria too vague or too strict?

  3. Application quality — Based on the evaluation results, what are the application's strengths and weaknesses? Where does it produce high-quality output? Where does it fail?

For each hypothesis:

  • State the hypothesis clearly in one sentence

  • Cite the evidence — entry indices, evaluator names, scores, reasoning quotes, trace data

  • Validate or invalidate — look at the actual eval input/output data and code to confirm or refute

  • Conclusion — what action does this hypothesis imply?

It is always possible to produce 3 hypotheses even when the data is limited. If the evaluation data doesn't give a conclusive answer on application quality, that itself is a signal about test case or evaluator gaps.

2c. Write the dataset analysis (two files)

Produce two files for the dataset analysis. Write the detailed version first, then derive the summary.

Detailed version: dataset-{idx}/analysis.md

This file is for agent consumption — it provides the complete data aggregation, hypothesis formation with evidence chains, and validated conclusions that a coding agent can act on directly.

Writing principles:

  • Show all the data before interpreting it. Start with the raw aggregation (pass/fail, per-evaluator stats, failure clusters) before any hypotheses. The data should stand on its own.

  • For each hypothesis, present: data → reasoning → conclusion. The reader should be able to follow your logic step by step and arrive at the same conclusion independently.

  • Cross-reference raw entry evidence directly. When citing evidence, reference the specific entry index and the underlying files/data points (for example: entry-3/evaluations.jsonl, entry-3/eval-output.jsonl, or entry-3/trace.jsonl).

  • Distinguish correlation from causation. If two entries fail the same evaluator, that's a pattern. But the root cause might differ — verify by checking the actual output data, don't assume.

  • Do not speculate without marking it. If a conclusion is uncertain, say "Hypothesis (unvalidated): ..." and explain what additional data would confirm or refute it.

Content:

  1. Overview — dataset name, entry count, overall pass rate

  2. Raw aggregation data

    • Per-evaluator statistics table (pass rate, score range, mean, standard deviation)

    • Failure matrix: entries × evaluators showing scores, highlighting failures

    • Failure clusters: entries grouped by shared failed evaluators

  3. Hypothesis 1: Test cases — hypothesis statement, evidence with entry/evaluator references, validation steps taken, conclusion with specific action

  4. Hypothesis 2: Evaluators — same structure

  5. Hypothesis 3: Application — same structure

  6. Open questions — anything the data doesn't conclusively answer, with suggestions for what additional data would help

Summary version: dataset-{idx}/analysis-summary.md

This file is for human review — a scannable overview of the dataset results, key findings, and recommended actions.

Template:

# Dataset Analysis — Summary

**Dataset**: <name> | **Entries**: <N> | **Pass rate**: <X/N (Y%)>

## Results at a glance

| Evaluator | Pass rate | Avg score | Notes                  |
| --------- | --------- | --------- | ---------------------- |
| ...       | ...       | ...       | <one-liner if notable> |

## Key findings

1. <Finding>: <1-2 sentences with the conclusion and its implication>
2. ...
3. ...

## Recommended actions (priority order)

1. <Action>: <what to do and expected impact, 1-2 sentences>
2. ...
3. ...

Maximum ~40 lines for the summary.

Phase 3: Action plan (two files)

After all datasets are analyzed, produce the action plan. Write two files at the test run root. Write the detailed version first, then derive the summary.

Detailed version: {PIXIE_ROOT}/results/<test_id>/action-plan.md

This file is for agent consumption — it provides specific, implementable improvement items with full evidence trails, so a coding agent can pick up any item and execute it without additional context-gathering.

Writing principles:

  • Each item must be self-contained. A coding agent reading just one priority item should have enough context (evidence references, file paths, expected changes) to implement it.

  • Trace every item back to evidence. Each priority must reference: which hypothesis (from which dataset analysis), which entries/evaluators provided the evidence, and what the specific data showed.

  • Be concrete about "How". Don't say "improve the prompt" — say "In scrapegraphai/prompts/generate_answer.py line 45, add instruction: '...'". The more specific, the more actionable.

  • Do not include speculative items. Every item must have validated evidence. If an item is based on an unvalidated hypothesis, either validate it first or exclude it.

Structure:

# Action Plan (Detailed)

## Summary

- X datasets analyzed, Y total entries, Z% overall pass rate
- [1-2 sentence high-level assessment]

## Priority 1: [Most impactful improvement]

- **What**: [specific change to make]
- **Why**: [which hypothesis from which dataset analysis, with entry/evaluator references]
- **Evidence**: [specific scores, output excerpts, trace data that support this]
- **Expected impact**: [which entries/evaluators this will improve, and predicted score change]
- **How**: [concrete implementation steps with file paths and line numbers]
- **Verification**: [how to verify the fix worked — which entries to re-run, what scores to expect]

## Priority 2: ...

...
Summary version: {PIXIE_ROOT}/results/<test_id>/action-plan-summary.md

This file is for human review — a prioritized list of improvements that a human can understand and approve in under 2 minutes.

Template:

# Action Plan — Summary

**Overall**: <X entries, Y% pass rate. 1-sentence assessment.>

## Actions (priority order)

1. **<Action title>**: <What to change and why, 2-3 sentences. Expected impact.>
2. **<Action title>**: <What to change and why, 2-3 sentences. Expected impact.>
3. ...

Maximum ~30 lines for the summary.

Prioritization criteria:

  • Systemic issues (affecting multiple entries/datasets) before isolated ones

  • Issues with clear, validated evidence before speculative ones

  • Application quality gaps before evaluator refinements before test case additions

  • Quick fixes before large refactors

The action plan should have 3–5 items. Each must trace back to a validated hypothesis from Phase 2. Do not include items that are speculative or lack evidence.

Process summary

  1. Phase 1 (per entry): Read data → grade pending evaluations → update evaluations.jsonl

  2. Phase 2 (per dataset): Aggregate → form 3 hypotheses → validate → write dataset-{idx}/analysis.md + dataset-{idx}/analysis-summary.md

  3. Phase 3 (per test run): Synthesize → prioritize → write action-plan.md + action-plan-summary.md

Process entries within a dataset concurrently (using subagents if available). Process phases sequentially — Phase 2 depends on Phase 1 outputs, Phase 3 depends on Phase 2 outputs.

Final verification

Before you end your turn, run the Step 6 verifier script that ships beside setup.sh in this skill's resources/ directory against the exact test run directory you analyzed.

Example shape:

python /path/to/eval-driven-dev/resources/verify_step6_completion.py pixie_qa/results/<test_id>

If the verifier reports any error, keep working. Step 6 is not complete until the verifier passes.

Requirements and compatibility

Written for: GitHub Copilot

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

The author says: Python 3.10+

Security

What our scanner noticed· Low risk

  • Phrasing that asks for actions without confirmation
  • Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

First committed by
GitHub avatar of yiouliyiouli
Publisher
GitHub (Official)
Distributed by
GitHub
Source
github/awesome-copilot · skills/eval-driven-dev · commit 797337f
Licence
MIT License (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced
Contributors
yiouli

AutoGineer is not affiliated with or endorsed by yiouli or GitHub.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download eval-driven-dev.zip