Phoenix Evals

GitHubOfficialFirst committed by jimbobbennettDistributed by GitHub

Skill
Tool integration
Software development
Apache-2.0
Low risk

Download ZIP

From the author

Build and run evaluators for AI/LLM applications using Phoenix.

— the description by jimbobbennett

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

Phoenix Evals provides a structured approach to building, running, and validating evaluators for AI and LLM applications. It supports both Python and TypeScript, offering tools for code-based and LLM-based evaluation, experiment management, and error analysis. The framework emphasizes a code-first approach, encouraging developers to define deterministic evaluations before moving to LLM-based judges, and provides mechanisms to validate evaluator accuracy against human-labeled data.

What it does

Builds custom code-based and LLM-based evaluators for AI applications.
Supports batch evaluation of datasets using structured dataframes.
Facilitates experiment management, including dataset creation and synthetic data generation.
Provides tools for error analysis, including axial coding and failure taxonomy creation.
Enables validation of evaluator accuracy against human-provided labels.
Integrates with tracing and observability to sample and review application traces.

When to use it

When you need to systematically evaluate the performance of LLM application outputs.
When building RAG systems that require specific retrieval and faithfulness metrics.
When you need to categorize and analyze failure patterns in AI agent workflows.
When transitioning from manual testing to automated, repeatable evaluation experiments.
When you need to validate that your automated judges align with human expectations.

Advantages

Supports a code-first approach to evaluation for deterministic results.
Provides a clear path for validating judge model accuracy.
Offers structured methods for building failure taxonomies.
Enables consistent evaluation across both Python and TypeScript environments.
Integrates directly with tracing data for context-aware evaluation.

Usage examples

“How do I build a custom LLM evaluator in Python?”: The skill provides guidance on using the LLM class and custom templates to define evaluators that assess specific model outputs.
“How can I categorize failures in my AI agent?”: The skill outlines an axial coding process to group failure notes into structured taxonomies like planning, tool selection, or execution errors.
“How do I run a batch evaluation on my dataset?”: The skill demonstrates using the evaluate_dataframe function to process a dataframe and return scores for defined metrics.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/github/phoenix-evals/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip phoenix-evals.zip -d .agents/skills

Documentation by jimbobbennett

Phoenix Evals

Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans.

Quick Reference

Task	Files
Setup	setup-python, setup-typescript
Decide what to evaluate	evaluators-overview
Choose a judge model	fundamentals-model-selection
Use pre-built evaluators	evaluators-pre-built
Build code evaluator	evaluators-code-python, evaluators-code-typescript
Build LLM evaluator	evaluators-llm-python, evaluators-llm-typescript, evaluators-custom-templates
Batch evaluate DataFrame	evaluate-dataframe-python
Understand experiments	experiments-overview
Run experiment	experiments-running-python, experiments-running-typescript
Create dataset	experiments-datasets-python, experiments-datasets-typescript
Generate synthetic data	experiments-synthetic-python, experiments-synthetic-typescript
Validate evaluator accuracy	validation, validation-evaluators-python, validation-evaluators-typescript
Sample traces for review	observe-sampling-python, observe-sampling-typescript
Analyze errors	error-analysis, error-analysis-multi-turn, axial-coding
RAG evals	evaluators-rag
Avoid common mistakes	common-mistakes-python, fundamentals-anti-patterns
Production	production-overview, production-guardrails, production-continuous

Workflows

Starting Fresh: observe-tracing-setup → error-analysis → axial-coding → evaluators-overview

Building Evaluator: fundamentals → common-mistakes-python → evaluators-{code|llm}-{python|typescript} → validation-evaluators-{python|typescript}

RAG Systems: evaluators-rag → evaluators-code-* (retrieval) → evaluators-llm-* (faithfulness)

Production: production-overview → production-guardrails → production-continuous

Reference Categories

Prefix	Description
`fundamentals-*`	Types, scores, anti-patterns
`observe-*`	Tracing, sampling
`error-analysis-*`	Finding failures
`axial-coding-*`	Categorizing failures
`evaluators-*`	Code, LLM, RAG evaluators
`experiments-*`	Datasets, running experiments
`validation-*`	Validating evaluator accuracy against human labels
`production-*`	CI/CD, monitoring

Key Principles

Principle	Action
Error analysis first	Can't automate what you haven't observed
Custom > generic	Build from your failures
Code first	Deterministic before LLM
Validate judges	>80% TPR/TNR
Binary > Likert	Pass/fail, not 1-5

Files

ATTRIBUTION.mdAdded by AutoGineer931 B
LICENSEAdded by AutoGineer10.0 KiB
SKILL.md4.5 KiB
references/axial-coding.md2.3 KiB
references/common-mistakes-python.md7.0 KiB
references/error-analysis-multi-turn.md1.3 KiB
references/error-analysis.md4.2 KiB
references/evaluate-dataframe-python.md4.4 KiB
references/evaluators-code-python.md3.0 KiB
references/evaluators-code-typescript.md1.2 KiB
references/evaluators-custom-templates.md1.2 KiB
references/evaluators-llm-python.md2.6 KiB
references/evaluators-llm-typescript.md1.4 KiB
references/evaluators-overview.md1.1 KiB
references/evaluators-pre-built.md2.4 KiB
references/evaluators-rag.md3.1 KiB
references/experiments-datasets-python.md4.8 KiB
references/experiments-datasets-typescript.md2.7 KiB
references/experiments-overview.md1.7 KiB
references/experiments-running-python.md2.9 KiB
references/experiments-running-typescript.md3.1 KiB
references/experiments-synthetic-python.md1.8 KiB
references/experiments-synthetic-typescript.md2.2 KiB
references/fundamentals-anti-patterns.md1.6 KiB
references/fundamentals-model-selection.md1.4 KiB
references/fundamentals.md1.8 KiB
references/observe-sampling-python.md2.4 KiB
references/observe-sampling-typescript.md3.9 KiB
references/observe-tracing-setup.md3.3 KiB
references/production-continuous.md3.6 KiB
references/production-guardrails.md1.3 KiB
references/production-overview.md2.3 KiB
references/setup-python.md1.9 KiB
references/setup-typescript.md981 B
references/validation-evaluators-python.md1.0 KiB
references/validation-evaluators-typescript.md4.4 KiB
references/validation.md1.7 KiB

references/experiments-datasets-python.md

Experiments: Datasets in Python

Creating and managing evaluation datasets.

Creating Datasets

create_dataset() upserts: if a dataset with the same name already exists it is updated in-place; re-running with identical inputs is a no-op.

from phoenix.client import Client

client = Client()

# From examples
dataset = client.datasets.create_dataset(
    name="qa-test-v1",
    examples=[
        {
            "input": {"question": "What is 2+2?"},
            "output": {"answer": "4"},
            "metadata": {"category": "math"},
        },
    ],
)

# With stable example IDs for targeted updates across uploads
dataset = client.datasets.create_dataset(
    name="qa-test-v1",
    examples=[
        {
            "id": "q-001",                      # stable ID — server updates this row, not inserts
            "input": {"question": "What is 2+2?"},
            "output": {"answer": "4"},
            "metadata": {"category": "math"},
        },
    ],
)

# From DataFrame
dataset = client.datasets.create_dataset(
    dataframe=df,
    name="qa-test-v1",
    input_keys=["question"],
    output_keys=["answer"],
    metadata_keys=["category"],
    split_key="split",        # single split column (use this instead of deprecated split_keys)
    example_id_key="id",      # column containing stable example IDs
)

From Production Traces

spans_df = client.spans.get_spans_dataframe(project_identifier="my-app")

dataset = client.datasets.create_dataset(
    dataframe=spans_df[["input.value", "output.value"]],
    name="production-sample-v1",
    input_keys=["input.value"],
    output_keys=["output.value"],
)

Retrieving Datasets

dataset = client.datasets.get_dataset(name="qa-test-v1")
df = dataset.to_dataframe()

Key Parameters

Parameter	Description
`input_keys`	Columns for task input
`output_keys`	Columns for expected output
`metadata_keys`	Additional context
`example_id_key`	Column with stable example IDs; server updates the matching row instead of inserting
`split_key`	Single column for split assignment (replaces deprecated `split_keys`)
`split_keys`	Deprecated — use `split_key` (singular) instead

Using Evaluators in Experiments

Evaluators as experiment evaluators

Pass phoenix-evals evaluators directly to run_experiment as the evaluators argument:

from functools import partial
from phoenix.client import AsyncClient
from phoenix.evals import ClassificationEvaluator, LLM, bind_evaluator

# Define an LLM evaluator
refusal = ClassificationEvaluator(
    name="refusal",
    prompt_template="Is this a refusal?\nQuestion: {{query}}\nResponse: {{response}}",
    llm=LLM(provider="openai", model="gpt-4o"),
    choices={"refusal": 0, "answer": 1},
)

# Bind to map dataset columns to evaluator params
refusal_evaluator = bind_evaluator(refusal, {"query": "input.query", "response": "output"})

# Define experiment task
async def run_rag_task(input, rag_engine):
    return rag_engine.query(input["query"])

# Run experiment with the evaluator
experiment = await AsyncClient().experiments.run_experiment(
    dataset=ds,
    task=partial(run_rag_task, rag_engine=query_engine),
    experiment_name="baseline",
    evaluators=[refusal_evaluator],
    concurrency=10,
)

Evaluators as the task (meta evaluation)

Use an LLM evaluator as the experiment task to test the evaluator itself against human annotations:

from phoenix.evals import create_evaluator

# The evaluator IS the task being tested
def run_refusal_eval(input, evaluator):
    result = evaluator.evaluate(input)
    return result[0]

# A simple heuristic checks judge vs human agreement
@create_evaluator(name="exact_match")
def exact_match(output, expected):
    return float(output["score"]) == float(expected["refusal_score"])

# Run: evaluator is the task, exact_match evaluates it
experiment = await AsyncClient().experiments.run_experiment(
    dataset=annotated_dataset,
    task=partial(run_refusal_eval, evaluator=refusal),
    experiment_name="judge-v1",
    evaluators=[exact_match],
    concurrency=10,
)

This pattern lets you iterate on evaluator prompts until they align with human judgments. See tutorials/evals/evals-2/evals_2.0_rag_demo.ipynb for a full worked example.

Best Practices

Upsert by default: Re-upload to the same name to update in-place; use example_id_key so the server targets specific rows instead of treating every upload as new data
Versioning: Version with tags or new names (e.g., qa-test-v2) when you want a clean snapshot, not just incremental edits
Metadata: Track source, category, difficulty
Balance: Ensure diverse coverage across categories
Avoid split_keys: Pass split_key (singular) — split_keys is deprecated and emits a DeprecationWarning

Requirements and compatibility

Written for: GitHub Copilot

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

The author says: Requires Phoenix server. Python skills need phoenix and openai packages; TypeScript skills need @arizeai/phoenix-client.

Security

What our scanner noticed· Low risk

Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

First committed by

jimbobbennett

Publisher

GitHub (Official)

Distributed by

GitHub

Source

github/awesome-copilot · skills/phoenix-evals · commit 797337f

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (written by AutoGineer)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

Contributors

jimbobbennett, stbenjam

AutoGineer is not affiliated with or endorsed by jimbobbennett or GitHub.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download phoenix-evals.zip

From the author

AutoGineer summary

What it does

When to use it

Advantages

Usage examples

Install

Documentation by jimbobbennett

Phoenix Evals

Quick Reference

Workflows

Reference Categories

Key Principles

Files

references/experiments-datasets-python.md

Experiments: Datasets in Python

Creating Datasets

From Production Traces

Retrieving Datasets

Key Parameters

Using Evaluators in Experiments

Evaluators as experiment evaluators

Evaluators as the task (meta evaluation)

Best Practices

Requirements and compatibility

Security

Attribution and licence

Related skills

Migrate To Shoehorn

Tdd

Test Driven Development

Webapp Testing

Test Driven Development

Browser Testing With Devtools

Download ZIP