Logfire Evals

Evaluate AI code with Logfire

How This Works

Python's pydantic_evals and Node.js's logfire/evals run a task against cases, apply evaluators, and return a report. An active Logfire or OpenTelemetry provider may export inputs and outputs even if this skill did not configure it. Intentional upload needs logfire.configure() in Python or a configured Node.js exporter.

Span-based evaluators inspect the task's OpenTelemetry span tree. Without working Logfire instrumentation, Python reports "No span tree available"; Node.js HasMatchingSpan can produce no evaluator result at all. Treat either signal as a setup failure, not evidence about the agent.

Step 1: Check for an Existing Braintrust Suite First

Cheap check, before anything else: does this repo already have an existing Braintrust suite — actual Eval(...) calls or from braintrust import Eval in source, not just a braintrust dependency listed without any real usage? This path needs no CLI auth at all — don't run Step 2 for it. The compatibility endpoint is documented only for Logfire Cloud US and EU; for any other supplied origin, use the native evaluation path instead.

Keep the existing Eval() code (Python braintrust>=0.30.1 / TypeScript braintrust>=3.24.0 — verified versions) and redirect its next run to Logfire by changing environment variables only, no pydantic_evals involved:

export BRAINTRUST_APP_URL="https://logfire-us.pydantic.dev/v1/braintrust"  # EU: logfire-eu.pydantic.dev
export BRAINTRUST_API_KEY="<logfire-project-api-key>"                     # Settings -> API Keys
unset BRAINTRUST_API_URL BRAINTRUST_PROXY_URL  # these override the endpoint above if set — the #1 "it still hit Braintrust" cause

The destination project's API key needs project:write_otlp and project:read_datasets: the SDK writes the run, then reads experiment metadata for its summary. An ingest-only write token fails with 403.

This compatibility preview covers inline/callable data, local tasks and scorers, multiple scores, one label per name, and normal summary finalization. It excludes Braintrust-hosted resources, BTQL, the model proxy, server-side scoring, and post-finalization feedback. Rust, summarize_scores=False, and manual flush() without comparison do not request the summary needed for projection. See the full coverage and concept mapping.

Skip straight to Step 5 (Verify) — the SDK's own printed result URL also opens directly in Logfire, and nothing else here (auth, dataset definition) applies to this path.

No existing Braintrust suite? Continue to Step 2 now, before the more detailed identification in Step 3 — nothing past this point requires knowing the function/agent or dataset shape yet.

Step 2: Authenticate When the Run Needs Logfire

For an explicitly local-only run without span evaluators, use a fresh process that neither preloads nor imports application telemetry; omit Python's logfire.configure() and any Node.js exporter bootstrap. If the task configures an exporter and has no documented disable switch, stop rather than claiming local-only. Uploading, hosted datasets, and span evaluators require Logfire authentication to the exact project.

For a Logfire-backed run, use Authenticate and Select the Exact Project to derive the CLI target from the supplied Logfire URL and run its target-aware whoami check with a verified CLI path — for JS/TS projects without uv, use the external-prefix npm fallback instead of plain npx, which can execute a repository-local binary. Skip to Step 3 if that already reports the right project and resolved --region or --base-url target; otherwise, continue through the full authentication and project-selection sequence there. This CLI flow is for logfire.configure(); Step 3's hosted-dataset operations use a separate API key with different scopes.

Step 3: Detect What to Evaluate

Identify the real task and any dataset. Follow repository package, test, and dependency conventions. For a first eval, prefer 3-5 cases from tests, schemas, examples, or synthetic fixtures, with deterministic checks for defined behavior. Do not copy the example unless it fits, replace an eval framework, or refactor unrelated code. Without a runnable task or safe expected behavior, ask one focused question instead of inventing either.

• In-code dataset: a Python module using pydantic_evals, or a Node.js module using logfire/evals. This is the default for an agent-driven workflow.

• Hosted/managed dataset: cases live in the Logfire UI, edited by non-engineers, pulled/pushed via a separate LogfireAPIClient (from logfire.experimental.api_client import LogfireAPIClient). Hosted inputs and outputs must be JSON objects; scalar roots accepted by code-defined datasets are rejected. client.get_dataset(name) with no type arguments returns a raw dict, not something push_dataset or .evaluate_sync() can take — pass the input/output (and metadata, if used) types to get back a real pydantic_evals.Dataset: client.get_dataset(name, MyInputType, MyOutputType). If the stored dataset contains custom evaluators, also pass their classes with custom_evaluator_types=[MyEvaluator] (and custom report evaluators with custom_report_evaluator_types=[...]) so they can be deserialized. Push with client.push_dataset(dataset). This needs its own API key from Settings → API Keys (scoped project:read_datasets/project:write_datasets), not Step 2's CLI auth flow. Only relevant if the user specifically wants case editing outside code.

Step 4: Define the Dataset and Run It

Use the repository's existing package manager and lockfile. Install only the missing integration for its language.

Python

Add pydantic-evals[logfire] with the detected Python manager: uv add, poetry add, or pdm add. For a pip/requirements project, update its declared requirements and install from that file; do not introduce a second manager or lockfile.

import logfire
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EqualsExpected, IsInstance

logfire.configure()  # omit only in the isolated local-only process described above

def classify_sentiment(text: str) -> str:
    return 'positive' if 'love' in text else 'negative'


dataset = Dataset[str, str, None](
    name='sentiment-eval',
    cases=[
        Case(name='positive', inputs='I love this', expected_output='positive'),
        Case(name='negative', inputs='This is terrible', expected_output='negative'),
    ],
    evaluators=[EqualsExpected(), IsInstance(type_name='str')],
)

report = dataset.evaluate_sync(classify_sentiment)  # or `await dataset.evaluate(...)`
report.print(include_input=True, include_output=True)

Use logfire[datasets] instead only when the task specifically needs the hosted-dataset API from Step 3.

JavaScript or TypeScript on Node.js

Do not apply this section to Deno, Bun, browsers, or workers; their exporter setup is not validated by this skill.

Add logfire and @pydantic/logfire-node with the manager selected by the existing lockfile: pnpm add, yarn add, bun add, or npm install. Do not introduce a second lockfile.

Configure Logfire before loading the task. Reuse an existing instrumentation entry point rather than configuring it twice.

import * as logfire from '@pydantic/logfire-node'
import { Case, Dataset, EqualsExpected, renderReport } from 'logfire/evals'

logfire.configure()

const classifySentiment = (text: string) => (text.includes('love') ? 'positive' : 'negative')
const dataset = new Dataset<string, string>({
  name: 'sentiment-eval',
  cases: [new Case({ name: 'positive', inputs: 'I love this', expectedOutput: 'positive' })],
  evaluators: [new EqualsExpected()],
})

await dataset.evaluate(classifySentiment).then((report) => {
  console.log(renderReport(report, { includeInput: true, includeOutput: true }))
}).finally(() => logfire.shutdown({ timeoutMillis: 5000 }))

Other built-ins include Equals, Contains, IsInstance, MaxDuration, HasMatchingSpan, and LLMJudge. Node.js custom evaluators extend Evaluator, and LLMJudge needs a judge callback. Use @pydantic/logfire-node/datasets only for hosted datasets.

Smoke test before a paid or full run

Before running the full dataset, run a smoke test on 2-3 cases if the dataset is large or uses LLMJudge or any evaluator that makes billed model calls. This catches setup errors before they multiply cost across the dataset.

Python:

smoke = Dataset(
    name=dataset.name,
    cases=dataset.cases[:3],
    evaluators=dataset.evaluators,
    report_evaluators=dataset.report_evaluators,
)
smoke_report = smoke.evaluate_sync(classify_sentiment)
smoke_report.print(include_input=True, include_output=True)

Node.js:

const smoke = new Dataset({
  name: dataset.name,
  cases: dataset.cases.slice(0, 3),
  evaluators: dataset.evaluators,
  reportEvaluators: dataset.reportEvaluators,
})
await smoke.evaluate(classifySentiment).then((report) => {
  console.log(renderReport(report, { includeInput: true, includeOutput: true }))
}).finally(() => logfire.shutdown({ timeoutMillis: 5000 }))

Confirm the smoke run has zero unexpected errors and the assertions that should pass do. Then, if the full dataset is large or uses paid model calls, tell the user the case count and which evaluators will make model calls, and get explicit confirmation before running the full dataset — don't run an expensive full pass on the strength of a clean smoke test alone without saying so.

The remaining details in this section are Python-specific. Custom evaluators inherit Evaluator and implement evaluate; use @dataclass for configurable fields and portable serialization. Case names must be unique within a dataset. The evaluators reached for most:

Also available: Contains, MaxDuration, TrajectoryMatch, ArgumentCorrectness, MaxToolCalls, MaxModelRequests — same span-tree dependency as ToolCorrectness for the tool/trajectory ones; see pydantic_evals.evaluators for the full set. These five agentic (span-based) evaluators need pydantic-evals>=2.4.0 — on an older pin, check pyproject.toml/uv.lock and upgrade before reaching for them, since the import itself is what fails, not a silent no-op.

The Python evaluator (arbitrary code execution) was removed for security reasons — don't reach for it even if an older example references it.

If editing a hosted dataset: client.push_dataset(dataset) overwrites server-side evaluators on every push, including removing ones you deleted locally — don't push a stale local copy over a dataset others have edited in the UI.

Step 5: Verify

A report printing to the terminal isn't proof it reached Logfire — confirm the run actually landed. Never report a case as passed, a score, or a run as complete without having actually checked it in this session — if a run fails, cancels, or produces no scores, report that failure plainly; never substitute an invented score or a manual guess at what the result "should" be.

Came from the Step 1 Braintrust path (Step 2 skipped)? There's no whoami-resolved project to look up here — use the SDK's own printed result URL instead, which already opens directly in the right Logfire project. Confirm the same things below (completion, pass mix, case detail) from that page rather than searching by name.

1. Query for the run directly, if a Logfire MCP server or API is connected — the root span for a run is named evaluate {name} and carries gen_ai.operation.name = 'experiment', dataset_name, and task_name attributes; find the most recent one matching your dataset's name and confirm logfire.experiment.metadata shows the case count and pass rate you expect. Otherwise, open AI Evaluations → Datasets & Experiments → Experiments in Logfire for the exact project from Step 2, and find the run by name/timestamp.

2. Read the Overview tab (or the queried metadata) first: completion count, assertion pass mix, task errors, average duration. If completion says "Not reported," the run sent case data but never signaled it finished — treat that as a broken run, not a passing one.

3. Open the Cases tab, starting from Needs Review / Failed / Errors, not the full list.

4. Drill into a failing case's trace in Live view for the actual evidence, rather than trusting the summary score alone.

5. Fix and re-run until the cases that should pass do, and any tool-call/trajectory checks show real span data, not "No span tree available."

Close with a final report built from what you just confirmed — the run name, exact case count and pass rate you queried, and which evaluators ran — not a template. Include the direct link to this experiment (the SDK's own printed result URL, or the Datasets & Experiments page you opened it from), so the user can see the run without having to ask where to look.