RAG Eval

NVIDIAOfficialCreated by NVIDIA RAG

Skill
Review and audit
Software development
Apache-2.0
Medium risk

From the author

Filesystem RAG benchmarks: corpus/, train.json, evaluate_rag.py (RAGAS quality). Not for prod monitoring, latency/throughput benchmarking (use rag-perf), or evals outside this repo layout.

— the description by NVIDIA RAG

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured approach to evaluating RAG (Retrieval-Augmented Generation) pipeline quality using the NVIDIA RAG Blueprint framework. It utilizes a specific filesystem layout consisting of a corpus directory and a train.json file to run automated evaluations via the evaluate_rag.py script. The process leverages RAGAS metrics to assess accuracy, context relevance, and groundedness, allowing developers to tune retrieval and generation parameters for optimal performance.

What it does

Automates RAG quality evaluation using RAGAS metrics.
Supports tuning of retrieval parameters like top_k and vdb_top_k.
Enables configuration of generation settings such as temperature and max-tokens.
Provides structured JSON output for headline metrics and per-query forensics.
Includes troubleshooting guidance for common ingestion and evaluation errors.
Facilitates comparative analysis of different pipeline configurations.

When to use it

When you need to measure the quality of your RAG pipeline's responses.
When comparing different retrieval or generation settings for your RAG system.
When you have prepared a dataset following the required corpus and train.json structure.
When you need to perform forensic analysis on specific query failures.

Advantages

Standardized evaluation workflow within the NVIDIA RAG Blueprint ecosystem.
Clear separation between quality evaluation and performance benchmarking.
Detailed per-query data for deep-dive analysis of retrieval and generation.
Flexible configuration for testing various RAG pipeline parameters.

Usage examples

“Run a full RAG evaluation on my dataset located at /data/my_rag_test.”: The agent will execute the evaluation script using the provided dataset path, host, and port, generating RAGAS metrics and summary reports in the results directory.
“How can I compare the impact of different top_k settings on my RAG quality?”: The agent will guide you to adjust the --top_k and --vdb_top_k flags in the evaluation command to perform comparative runs and analyze the resulting RAGAS scores.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/nvidia/rag-eval/download
Unzip it into the cross-client skills folder of your project:
```
mkdir -p .agents/skills && unzip rag-eval.zip -d .agents/skills
```

Documentation by NVIDIA RAG

On-disk RAG evaluation (`corpus/` + `train.json`)

Purpose

Guide agents through NVIDIA RAG Blueprint filesystem benchmarks: preparing corpus/ and train.json, running scripts/eval/evaluate_rag.py, tuning retrieval and generation flags for quality comparisons, interpreting RAGAS JSON outputs, and triaging failures (HTTP/stream errors, empty contexts, collection mismatch, judge API).

For latency, throughput, and load testing, use the rag-perf skill (scripts/rag-perf, docs/performance-benchmarking.md) — not this skill.

When not to use

Do not use this skill for: deploying or repairing services (use rag-blueprint); evaluating APIs without the corpus/ + train.json layout; general ML experimentation unrelated to this evaluator; production monitoring/alerting; or latency/throughput benchmarking (use rag-perf).

Prerequisites

Repo cloned; run commands from repo root (imports and paths assume this).
Python 3.11+ and uv; eval deps: uv sync --project scripts/eval.
Reachable RAG server and ingestor (defaults often localhost:8081 / 8082).
NVIDIA_API_KEY for RAGAS (see credential hygiene); optional RAG_EVAL_JUDGE_MODEL.
Dataset roots passed to --dataset-paths each contain corpus/ and train.json.

Instructions

Prepare data — Ensure each dataset directory matches the layout and train.json rules in references/dataset-and-conversion.md. When sources arrive as public links (sites or dataset pages), materialize documents under corpus/—prefer PDF for multimodal content so images stay embedded; convert CSV/JSONL/etc. using the patterns there.
Run eval — uv run --project scripts/eval python scripts/eval/evaluate_rag.py with --dataset-paths, --host, and --port. See references/benchmark-execution.md for command examples, outputs, and errors. Use references/evaluate-rag-cli.md for flag-level detail.
Tune quality — Adjust --top_k / --vdb_top_k, reranker and query-rewriting toggles, and generation overrides (--temperature, --top-p, --max-tokens) as documented in references/benchmark-execution.md when comparing retrieval/generation configs for RAGAS scores.
Analyze results — Use references/result-analysis.md for scripts; scan rag_*_evaluation_summary.json for headline RAGAS metrics.
Triage errors — Use the error signal table and the Troubleshooting section below.

Examples

Set API key without putting secrets in shell history (preferred patterns): load from a gitignored env file or secrets manager; avoid committing .env; rotate keys if exposed. Details: references/benchmark-execution.md#credential-hygiene-nvidia_api_key.

Minimal eval (key already in environment):

uv sync --project scripts/eval
uv run --project scripts/eval python scripts/eval/evaluate_rag.py \
  --dataset-paths /path/to/my_dataset \
  --host localhost \
  --port 8081

Pretty-print summary JSON:

python3 -m json.tool results/my_dataset/rag_my_dataset_evaluation_summary.json

More examples (skip ingestion, quality sweeps): references/benchmark-execution.md.

Limitations

Evaluator behavior is fixed to the filesystem contract and evaluate_rag.py; it does not substitute for custom offline judges or non-RAG benchmarks.
Vector DB / embedding choices follow deployed ingestor and RAG env — not overridden by this CLI alone.
Scores depend on retrieval quality, judge model availability, and NVIDIA_API_KEY; empty contexts yield partial RAGAS metrics (see references).
Large procedural detail lives under references/ to keep routing concise; read those files when the user needs step-by-step conversion, full flags, or error tables.

Troubleshooting

Error / signal	Likely cause	What to do
Immediate exit mentioning `NVIDIA_API_KEY`	Missing or invalid key	Set key via secure channel; see credential hygiene in `references/benchmark-execution.md`.
`train.json must be a JSON array`	Wrong JSON shape	Top-level array of objects; validate per `references/dataset-and-conversion.md`.
Fewer rows in `evaluation_data.json` than `train.json`	Per-query failures	Check stderr: network or stream JSON errors; see error table in benchmark-execution.
Empty `generated_contexts` everywhere	Retrieval gap	Verify collection, ingestion, `top_k` / `vdb_top_k`, and `ingestor_server_url` without `/v1` suffix.
Ingestor 404 on upload	Bad ingestor base URL	Pass `http://host:port` only — code appends `/v1/`.

Full signal table: references/benchmark-execution.md#common-error-cases-and-signals.

Gotchas

Run from repo root: paths and imports in scripts/eval/evaluate_rag.py assume this; a wrong directory silently breaks imports.
--ingestor_server_url: pass http://host:port without /v1—the code appends /v1/ automatically. Including /v1 causes 404s on ingestor calls.
Vector DB / embedding settings: not set by this CLI; configure via the deployed ingestor and RAG server env vars (e.g. APP_VECTORSTORE_URL, embedding model).
--model / --llm_endpoint: forwarded verbatim only when explicitly set; omit to keep the server's configured LLM.
Stale collections: a previous run's ingested data persists unless you use --force_ingestion. Use --collection with a unique name when comparing quality across isolated runs.
Empty context metrics: if all generated_contexts are empty, RAGAS scores only nv_accuracy and leaves the other two metrics blank—this is not a silent success.

Source of truth

Piece	Location
Driver	`scripts/eval/evaluate_rag.py` (`CORPUS_DIRECTORY` = `corpus`, `EVAL_DATA` = `train.json`)
Human README (always in-repo)	`scripts/eval/README.md`
Full CLI (flags, defaults)	`scripts/eval/evaluate_rag.py --help`; `references/evaluate-rag-cli.md`
Dataset / conversion	`references/dataset-and-conversion.md`
Runs, outputs, errors	`references/benchmark-execution.md`
Result analysis scripts	`references/result-analysis.md`
Latency / throughput	rag-perf skill, `docs/performance-benchmarking.md`

Agent playbook

Run eval — uv sync --project scripts/eval then uv run --project scripts/eval python scripts/eval/evaluate_rag.py with required --dataset-paths, --host, and --port (and env NVIDIA_API_KEY). Argument --ingestor_server_url is optional (defaults to http://localhost:8082); pass it only when overriding the ingestor endpoint.
Quality tuning — See references/benchmark-execution.md: --top_k/--vdb_top_k, reranker and query-rewriting toggles, --temperature, --top-p, --max-tokens.
Data conversion — Follow references/dataset-and-conversion.md.
Analyze results — references/result-analysis.md; quick scan: python3 -m json.tool results/<dataset>/rag_<dataset>_evaluation_summary.json.
Error triage — references/benchmark-execution.md#common-error-cases-and-signals.

Files

ATTRIBUTION.mdAdded by AutoGineer886 B
BENCHMARK.md2.8 KiB
LICENSELicence file added by AutoGineer11.2 KiB
SKILL.md9.2 KiB
eval/h100.json2.8 KiB
eval/nvidia_hosted.json1.9 KiB
references/benchmark-execution.md8.1 KiB
references/dataset-and-conversion.md8.2 KiB
references/evaluate-rag-cli.md4.2 KiB
references/result-analysis.md2.8 KiB
skill-card.md2.6 KiB
skill.oms.sig5.7 KiB

references/benchmark-execution.md

Benchmark runs, outputs, and error signals

Load this for full command examples, artifact descriptions, quality interpretation, retrieval/generation flags, and the error-signal table.

For latency, throughput, and load testing, use the rag-perf skill — not this document.

Credential hygiene (`NVIDIA_API_KEY`)

Prefer a secrets manager or a sourced env file that is not committed; ensure .env and key files are in .gitignore.
Shell history may record export ... lines — avoid pasting real keys on the command line; rotate the key if it was exposed.
Do not hardcode API keys in scripts or commit them to version control.

After the key is available in the environment, run commands from the repo root.

Output artifacts

Under --output_dir (default results), each dataset gets a subdirectory named after the dataset directory basename. Files share the same <label> (the dataset folder name):

File	Purpose
`rag_<label>_evaluation_data.json`	Per query: `question`, `answer`, `generated_answer`, `generated_contexts`, `retrieved_docs`. Written before RAGAS. Use for forensics and failure patterns.
`rag_<label>_evaluation_summary.json`	Headline means: `nv_accuracy_mean`, `nv_context_relevance_mean`, `nv_response_groundedness_mean`. Fast pass/fail.
`rag_<label>_evaluation_results.json`	RAGAS vectors: per-sample score lists under `nv_accuracy`, `nv_context_relevance`, `nv_response_groundedness`.
`rag_<label>_evaluation_metrics.json`	Structured roll-up: `ingestion_metrics_list`, `evaluation_metrics` (model dump of `RagEvaluationMetrics`).

Analysis tips: If evaluation_data has fewer rows than train.json, some queries failed (exceptions print during the run). After drops, use id / query_id to align rows rather than positional index. For "worst questions," pair index i in evaluation_results score lists with the ith object in evaluation_data.

Interpreting RAGAS quality metrics

nv_accuracy — answer accuracy (LLM judge vs ground-truth answer).
nv_context_relevance and nv_response_groundedness — scored when retrieved contexts exist.
If no non-empty generated_contexts are present across the run, the code scores answer accuracy only—do not treat empty context metrics as a silent success.

Running the benchmark

Set NVIDIA_API_KEY (see credential hygiene above). Optionally set RAG_EVAL_JUDGE_MODEL for the RAGAS judge LLM id. Then from repo root:

Minimal full-run example

uv run --project scripts/eval python scripts/eval/evaluate_rag.py \
  --dataset-paths /path/to/my_dataset \
  --host localhost \
  --port 8081 \
  --ingestor_server_url http://localhost:8082 \
  --output_dir results

(NVIDIA_API_KEY must already be exported or injected by your environment.)

Skip ingestion (collection already populated)

uv run --project scripts/eval python scripts/eval/evaluate_rag.py \
  --dataset-paths /path/to/my_dataset \
  --host localhost \
  --port 8081 \
  --ingestor_server_url http://localhost:8082 \
  --skip_ingestion

Ingestion only (no RAGAS scoring)

uv run --project scripts/eval python scripts/eval/evaluate_rag.py \
  --dataset-paths /path/to/my_dataset \
  --host localhost \
  --port 8081 \
  --ingestor_server_url http://localhost:8082 \
  --skip_evaluation

Force re-ingest (delete existing collection first)

uv run --project scripts/eval python scripts/eval/evaluate_rag.py \
  --dataset-paths /path/to/my_dataset \
  --host localhost --port 8081 \
  --ingestor_server_url http://localhost:8082 \
  --force_ingestion

Retrieval and generation options (quality comparisons)

Use these flags when comparing pipeline configs for RAGAS scores. Omit any flag to leave the RAG server default.

Retrieval depth

--top_k 5          # sent as reranker_top_k to the generate endpoint
--vdb_top_k 20     # vector DB candidate pool size

Toggle pipeline stages

--enable-reranker          # send enable_reranker=true on /v1/generate
--disable-reranker         # send enable_reranker=false
--enable-query-rewriting   # send enable_query_rewriting=true
--disable-query-rewriting  # send enable_query_rewriting=false

Omitting these flags does not send the field—the RAG server uses its own configured default. --enable-reranker and --disable-reranker are mutually exclusive; same for the query-rewriting pair.

Generation parameters

--temperature 0.0    # deterministic output for repeatable benchmarks
--top-p 0.95
--max-tokens 512     # cap answer length

These are forwarded verbatim to /v1/generate; omit to use the server default.

Example: quality comparison across configs

uv run --project scripts/eval python scripts/eval/evaluate_rag.py \
  --dataset-paths /path/to/my_dataset \
  --host localhost --port 8081 \
  --ingestor_server_url http://localhost:8082 \
  --skip_ingestion \
  --disable-reranker \
  --disable-query-rewriting \
  --temperature 0.0 \
  --max-tokens 512 \
  --output_dir results/baseline_no_rerank

Use a distinct --collection or --force_ingestion when you need an isolated corpus for each config.

Result analysis

For ready-to-run Python scripts, read result-analysis.md. It contains: per-query worst-accuracy table, CSV export, and markdown report table.

Quick headline scan:

python3 -m json.tool results/my_dataset/rag_my_dataset_evaluation_summary.json

Rows with has_context=N and low nv_accuracy signal retrieval problems (ingestion gap or collection mismatch), not generation problems.

Common error cases and signals

Signal	What it usually means	What to check
Script exits immediately on `NVIDIA_API_KEY`	Judge cannot run	Export a valid key; optional `RAG_EVAL_JUDGE_MODEL` for an available catalog model.
`train.json must be a JSON array` / validation errors	Bad JSON shape	Top-level array of objects, not a single object or multiline records without array wrapper.
Fewer rows in `evaluation_data.json` than in `train.json`	Per-query exception	Stderr during run: network or JSON decode on stream.
Row has `generated_answer: ""` and `generated_contexts: []`	RAG returned no content	Retrieval returned nothing: collection exists and is populated? `top_k`/`vdb_top_k` too low?
`Response contained error message` / answers matching the server's error sentinel	RAG returned an error string	RAG server logs, collection existence, `collection_names` vs ingested data.
`Failed to get response from rag-server`	HTTP or network	`--host`/`--port`, firewall, RAG server health and logs.
Ingestor or collection errors	4xx/5xx on ingestor	`ingestor_server_url` base without `/v1`, credentials, disk, ingestor logs.
`nv_context_relevance` / `nv_response_groundedness` empty with empty `generated_contexts`	No usable retrieved text for context metrics	Ingestion, `collection_name` alignment, `top_k` / retrieval config.
>50% failures warning in stdout	`error_count` high	Systematic config issue (wrong collection, RAG down, or streaming parse errors).
Citation / filename mismatch in metrics	Names do not line up	`corpus/` file basenames vs citation `document_name` patterns.
Stale collection from a previous run tainting results	Unexpectedly high or low accuracy	Use `--force_ingestion` to delete and re-ingest, or `--collection` to isolate.

Pre-flight checklist

Each dataset root: corpus/ + train.json (corpus/ preferably PDF, including sources where the upstream link does not name a file explicitly).
train.json: top-level array of objects (dict-shaped root is rejected). Run the quick validation in dataset-and-conversion.md after any conversion.
Rows include question and answer for meaningful RAGAS scores.
NVIDIA_API_KEY available before invoking the script (optional RAG_EVAL_JUDGE_MODEL if not using the default judge).
For config comparisons: use a distinct --collection or --force_ingestion / --skip_ingestion so each run sees the intended corpus state.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Declares allowed tools, Needs an API key, Needs Python

The author says: Repository checkout with uv; Python 3.11+; run from repo root; uv sync --project scripts/eval (eval deps live in scripts/eval/pyproject.toml); network to RAG, ingestor, and vdb endpoints; NVIDIA_API_KEY for RAGAS; optional RAG_EVAL_JUDGE_MODEL (default mistralai/mixtral-8x22b-instruct-v0.1).

Security

What our scanner noticed· Medium risk

Large base64-encoded blobs

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

NVIDIA RAG

Publisher

NVIDIA (Official)

Source

NVIDIA/skills · skills/rag-eval · commit fce3cca

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE-APACHE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by NVIDIA RAG or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download rag-eval.zip