Phoenix Evals

From the author

Build and run evaluators for AI/LLM applications using Phoenix.

— the description by jimbobbennett

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

Phoenix Evals provides a structured approach to building, running, and validating evaluators for AI and LLM applications. It supports both Python and TypeScript, offering tools for code-based and LLM-based evaluation, experiment management, and error analysis. The framework emphasizes a code-first approach, encouraging developers to define deterministic evaluations before moving to LLM-based judges, and provides mechanisms to validate evaluator accuracy against human-labeled data.

What it does

Builds custom code-based and LLM-based evaluators for AI applications.
Supports batch evaluation of datasets using structured dataframes.
Facilitates experiment management, including dataset creation and synthetic data generation.
Provides tools for error analysis, including axial coding and failure taxonomy creation.
Enables validation of evaluator accuracy against human-provided labels.
Integrates with tracing and observability to sample and review application traces.

When to use it

When you need to systematically evaluate the performance of LLM application outputs.
When building RAG systems that require specific retrieval and faithfulness metrics.
When you need to categorize and analyze failure patterns in AI agent workflows.
When transitioning from manual testing to automated, repeatable evaluation experiments.
When you need to validate that your automated judges align with human expectations.

Advantages

Supports a code-first approach to evaluation for deterministic results.
Provides a clear path for validating judge model accuracy.
Offers structured methods for building failure taxonomies.
Enables consistent evaluation across both Python and TypeScript environments.
Integrates directly with tracing data for context-aware evaluation.

Usage examples

“How do I build a custom LLM evaluator in Python?”: The skill provides guidance on using the LLM class and custom templates to define evaluators that assess specific model outputs.
“How can I categorize failures in my AI agent?”: The skill outlines an axial coding process to group failure notes into structured taxonomies like planning, tool selection, or execution errors.
“How do I run a batch evaluation on my dataset?”: The skill demonstrates using the evaluate_dataframe function to process a dataframe and return scores for defined metrics.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/github/phoenix-evals/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip phoenix-evals.zip -d .agents/skills

Documentation by jimbobbennett

Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans.

Quick Reference

Task	Files
Setup	setup-python, setup-typescript
Decide what to evaluate	evaluators-overview
Choose a judge model	fundamentals-model-selection
Use pre-built evaluators	evaluators-pre-built
Build code evaluator	evaluators-code-python, evaluators-code-typescript
Build LLM evaluator	evaluators-llm-python, evaluators-llm-typescript, evaluators-custom-templates
Batch evaluate DataFrame	evaluate-dataframe-python
Understand experiments	experiments-overview
Run experiment	experiments-running-python, experiments-running-typescript
Create dataset	experiments-datasets-python, experiments-datasets-typescript
Generate synthetic data	experiments-synthetic-python, experiments-synthetic-typescript
Validate evaluator accuracy	validation, validation-evaluators-python, validation-evaluators-typescript
Sample traces for review	observe-sampling-python, observe-sampling-typescript
Analyze errors	error-analysis, error-analysis-multi-turn, axial-coding
RAG evals	evaluators-rag
Avoid common mistakes	common-mistakes-python, fundamentals-anti-patterns
Production	production-overview, production-guardrails, production-continuous

Workflows

Starting Fresh: observe-tracing-setup → error-analysis → axial-coding → evaluators-overview

Building Evaluator: fundamentals → common-mistakes-python → evaluators-{code|llm}-{python|typescript} → validation-evaluators-{python|typescript}

RAG Systems: evaluators-rag → evaluators-code-* (retrieval) → evaluators-llm-* (faithfulness)

Production: production-overview → production-guardrails → production-continuous

Reference Categories

Prefix	Description
`fundamentals-*`	Types, scores, anti-patterns
`observe-*`	Tracing, sampling
`error-analysis-*`	Finding failures
`axial-coding-*`	Categorizing failures
`evaluators-*`	Code, LLM, RAG evaluators
`experiments-*`	Datasets, running experiments
`validation-*`	Validating evaluator accuracy against human labels
`production-*`	CI/CD, monitoring

Key Principles

Principle	Action
Error analysis first	Can't automate what you haven't observed
Custom > generic	Build from your failures
Code first	Deterministic before LLM
Validate judges	>80% TPR/TNR
Binary > Likert	Pass/fail, not 1-5

Files

ATTRIBUTION.mdAdded by AutoGineer931 B
LICENSEAdded by AutoGineer10.0 KiB
SKILL.md4.5 KiB
references/axial-coding.md2.3 KiB
references/common-mistakes-python.md7.0 KiB
references/error-analysis-multi-turn.md1.3 KiB
references/error-analysis.md4.2 KiB
references/evaluate-dataframe-python.md4.4 KiB
references/evaluators-code-python.md3.0 KiB
references/evaluators-code-typescript.md1.2 KiB
references/evaluators-custom-templates.md1.2 KiB
references/evaluators-llm-python.md2.6 KiB
references/evaluators-llm-typescript.md1.4 KiB
references/evaluators-overview.md1.1 KiB
references/evaluators-pre-built.md2.4 KiB
references/evaluators-rag.md3.1 KiB
references/experiments-datasets-python.md4.8 KiB
references/experiments-datasets-typescript.md2.7 KiB
references/experiments-overview.md1.7 KiB
references/experiments-running-python.md2.9 KiB
references/experiments-running-typescript.md3.1 KiB
references/experiments-synthetic-python.md1.8 KiB
references/experiments-synthetic-typescript.md2.2 KiB
references/fundamentals-anti-patterns.md1.6 KiB
references/fundamentals-model-selection.md1.4 KiB
references/fundamentals.md1.8 KiB
references/observe-sampling-python.md2.4 KiB
references/observe-sampling-typescript.md3.9 KiB
references/observe-tracing-setup.md3.3 KiB
references/production-continuous.md3.6 KiB
references/production-guardrails.md1.3 KiB
references/production-overview.md2.3 KiB
references/setup-python.md1.9 KiB
references/setup-typescript.md981 B
references/validation-evaluators-python.md1.0 KiB
references/validation-evaluators-typescript.md4.4 KiB
references/validation.md1.7 KiB

references/experiments-running-python.md

Experiments: Running Experiments in Python

Execute experiments with run_experiment.

Basic Usage

from phoenix.client import Client
from phoenix.client.experiments import run_experiment

client = Client()
dataset = client.datasets.get_dataset(name="qa-test-v1")

def my_task(example):
    return call_llm(example.input["question"])

def exact_match(output, expected):
    return 1.0 if output.strip().lower() == expected["answer"].strip().lower() else 0.0

experiment = run_experiment(
    dataset=dataset,
    task=my_task,
    evaluators=[exact_match],
    experiment_name="qa-experiment-v1",
)

Task Functions

# Basic task
def task(example):
    return call_llm(example.input["question"])

# With context (RAG)
def rag_task(example):
    return call_llm(f"Context: {example.input['context']}\nQ: {example.input['question']}")

Evaluator Parameters

Parameter	Access
`output`	Task output
`expected`	Example expected output
`input`	Example input
`metadata`	Example metadata

Options

experiment = run_experiment(
    dataset=dataset,
    task=my_task,
    evaluators=evaluators,
    experiment_name="my-experiment",
    dry_run=3,       # Test with 3 examples
    repetitions=3,   # Run each example 3 times
)

Results

print(experiment.aggregate_scores)
# {'accuracy': 0.85, 'faithfulness': 0.92}

for run in experiment.runs:
    print(run.output, run.scores)

Stability

Single-run scores are noisy when either the task or the evaluator is non-deterministic — an LLM call, tool use, streaming output, an LLM-as-judge. On a small dataset, that per-run noise can swamp the signal from a prompt change.

Averaging over repetitions lets the score you report reflect the prompt rather than the sampling noise:

run_experiment(
    # ...
    repetitions=3,
)

Things to consider:

Reach for repetitions when the task or the evaluator is an LLM call and the dataset is small.
Prefer repetitions when per-example cost is low and you mostly want to settle the score; prefer growing the dataset when you also need to cover more behaviors.
Skip repetitions when both the task and the evaluator are deterministic (e.g. string comparison against a ground truth) — a single run is the answer.

Consider adding stability when:

Repeat runs of the same experiment drift in ways that feel larger than the differences you're trying to measure.
A prompt change flips example labels in ways that don't track with how the outputs actually changed.
The judge's reasoning on the same output reads differently from one run to the next.

Repetitions are also what repetitions=1 (default) silently relies on — don't trust a tuning decision based on a single 10-example run.

Add Evaluations Later

from phoenix.client.experiments import evaluate_experiment

evaluate_experiment(experiment=experiment, evaluators=[new_evaluator])

Requirements and compatibility

Written for: GitHub Copilot

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

The author says: Requires Phoenix server. Python skills need phoenix and openai packages; TypeScript skills need @arizeai/phoenix-client.

Security

What our scanner noticed· Low risk

Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Migrate To Shoehorn
Matt Pocock· Community
Automate the migration of TypeScript test assertions from unsafe as-casting to the type-safe shoehorn library.
Software developmentClaude CodeClaude appsOpenAI Codex+14 more
MITGitHub stars of the source repository317.5k
Tdd
Matt Pocock· Community
A reference guide for implementing the red-green-refactor loop in test-driven development.
Software developmentClaude CodeClaude appsOpenAI Codex+14 more
MITGitHub stars of the source repository317.5k
Test Driven Development
Jesse Vincent (obra)· Community
A rigorous framework for Test-Driven Development, mandating that failing tests precede all production code.
Software developmentClaude CodeClaude appsOpenAI Codex+14 more
MITGitHub stars of the source repository314k
Webapp Testing
AnthropicOfficial
A toolkit for automating local web application testing using Playwright and server management helpers.
Software developmentClaude CodeClaude appsOpenAI Codex+14 more
Apache-2.0Low riskHas scriptsGitHub stars of the source repository188.6k
Test Driven Development
Addy Osmani· Community
A systematic approach to software development using the red-green-refactor cycle to ensure code reliability and maintainability.
Software developmentClaude CodeClaude appsOpenAI Codex+14 more
MITGitHub stars of the source repository113.9k
Browser Testing With Devtools
Addy Osmani· Community
Integrate Chrome DevTools to enable live browser inspection, debugging, and automated UI verification for your AI agent.
Software developmentClaude CodeClaude appsOpenAI Codex+14 more
MITGitHub stars of the source repository113.9k

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download phoenix-evals.zip