RAG Perf

NVIDIAOfficialCreated by NVIDIA RAG

Skill
Tool integration
Software development
Apache-2.0
Medium risk

From the author

Performance benchmarking for a deployed NVIDIA RAG Blueprint server: profiling pass + aiperf load test driven by a single YAML config. Not for accuracy / RAGAS scoring (use rag-eval) or for deploying / repairing services (use rag-blueprint).

— the description by NVIDIA RAG

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

rag-perf is a command-line tool designed to benchmark the performance of deployed NVIDIA RAG Blueprint servers. It executes a two-phase process: a server-side profiling pass to analyze stage-specific timing and bottlenecks, followed by an optional aiperf-based load test to measure throughput, latency, and error rates. The tool is entirely driven by YAML configuration files, allowing users to define concurrency levels, retrieval parameters, and synthetic query generation. It provides comprehensive reporting, including per-point analysis and aggregate sweep comparisons, making it suitable for regression testing and performance tuning of RAG pipelines.

What it does

Performs server-side profiling to identify bottlenecks and stage-specific latency.
Executes aiperf load tests to measure TTFT, throughput, and error rates.
Supports multi-axis sweeps for concurrency, vdb_top_k, and reranker_top_k parameters.
Generates unified reports including summary tables and side-by-side comparisons.
Validates configurations and provides reproducible output via resolved YAML dumps.
Integrates with synthetic query generation for automated load testing.

When to use it

When you need to profile the latency and performance of a deployed RAG server.
When conducting regression checks on RAG pipeline configurations.
When performing concurrency sweeps to determine optimal throughput.
When tuning retrieval and reranker parameters for performance optimization.

Advantages

Config-driven approach ensures reproducible benchmarking runs.
Provides granular stage-by-stage timing breakdowns.
Automates complex multi-axis performance sweeps.
Includes built-in support for synthetic query generation.
Produces clear, actionable reports with bottleneck identification.

Usage examples

“Run a quick performance profile of my RAG server.”: The agent executes the quick_profile.yaml preset, generating a profile report without running a full load test.
“Benchmark my RAG server with a concurrency sweep.”: The agent runs the sweep.yaml configuration, producing a nested directory structure with aggregate results comparing different concurrency and top-k settings.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/nvidia/rag-perf/download
Unzip it into the cross-client skills folder of your project:
```
mkdir -p .agents/skills && unzip rag-perf.zip -d .agents/skills
```

Documentation by NVIDIA RAG

RAG-Perf — config-driven perf benchmark CLI

Purpose

Drive a deployed NVIDIA RAG Blueprint server with a YAML config, run a server-side profiling pass (per-stage timing, citation quality, bottleneck inference) and an optional aiperf load test (TTFT / E2E / token & request throughput / error rate), and write a unified report. The CLI is intentionally minimal: rag-perf -c <config> plus --help / --version. Behaviour is fully config-driven; field variations belong in YAML.

Scope

Accuracy / RAGAS scoring of answer quality → use the rag-eval skill.
Deploying, repairing, or configuring services (compose, helm, NIM env vars) → use the rag-blueprint skill.
Production monitoring / alerting — rag-perf is a one-shot benchmark tool.
Runtime requirement: a deployed RAG server reachable on the network.

Prerequisites

Repo cloned; run commands from the repo root (config paths in the presets are repo-root-relative).
Python 3.11+ and uv on PATH.
Install rag-perf into its own uv-managed venv: uv sync --project scripts/rag-perf.
For unit tests: install dev extras as well — uv sync --project scripts/rag-perf --extra dev (otherwise pytest-asyncio is missing and async tests error out at collection time).
A reachable RAG server (default http://localhost:8081). For the aiperf phase, the bundled nvidia_rag endpoint plugin must be installed — pip install -e ./scripts/rag-perf registers it via the aiperf.plugins entry point.
For synthetic queries: an OpenAI-compatible chat-completions endpoint reachable at synthetic.llm_url (default http://localhost:8999/v1/chat/completions).
rag-perf itself runs without NVIDIA_API_KEY (unlike rag-eval). The synthetic LLM endpoint may require its own auth — that's the deployment's concern.

Instructions

Pick a preset. The three under scripts/rag-perf/configs/ are:
- quick_profile.yaml — profile-only, ~30 s. Skips load test. For fast iteration on retrieval / reranker tuning.
- single_run.yaml — one concurrency level, profiling + aiperf, ~2 min. Regression checks.
- sweep.yaml — multi-axis sweep. load.concurrency, rag.vdb_top_k, rag.reranker_top_k are all int | list[int]; any of them as a list becomes a sweep axis (Cartesian product).
Edit the preset. Required: replace rag.collection_names: ["<collection_name>"] with a real collection on the deployed ingestor server. Verify the collection exists via GET /v1/collections on the ingestor. The placeholder <collection_name> validates fine but every request will fail at retrieval. Use a copied YAML preset for variants; the CLI surface is intentionally config-only.
Run. From repo root:
```
uv run --project scripts/rag-perf rag-perf -c scripts/rag-perf/configs/single_run.yaml
```
Same form for the other presets. The CLI accepts only -c / --config (required), --help, --version.
Read stdout. Every invocation prints, in order: a startup banner, a one-line summary, the fully resolved config as YAML (so the run is reproducible from terminal output), per-grid-point progress with the shlex-joined aiperf command in copy-pastable form, a rich per-point summary table (stage breakdown with bars, citation quality, bottleneck, load-test block), and finally a side-by-side comparison table auto-labelled by whichever axis varied. See references/output-and-analysis.md.
Inspect artifacts. Layout depends on run shape — flat for single-point + iterations=1, nested under iter_<i>/<point>/... otherwise. See references/output-and-analysis.md for the full directory tree, file purposes, and how to parse results.json / results.csv / report.md.
Summarise for the user. When reporting back, follow the playbook in references/output-and-analysis.md#summarising-results-to-the-user: pick the canonical result file for the run shape, build a headline table (concurrency × top-k axes × TTFT × throughput × bottleneck × citation quality), compute scaling efficiency on sweeps, always flag zero citations / non-zero error rate / suspect llm_ttft_ms / small-sample p99, and propose a concrete next-experiment YAML.
Tune. Schema is fully documented in docs/performance-benchmarking.md and the deeper-dive references below. Common knobs: turn aiperf.enabled: false for profile-only mode, increase load.iterations for variance estimation, set load.sleep_between_points_s: 60 for overnight Cartesian sweeps.

Examples

Profile-only (quickest signal on retrieval / reranker tuning):

uv run --project scripts/rag-perf rag-perf -c scripts/rag-perf/configs/quick_profile.yaml

Output: rag-perf-results/quick_profile/run_<ts>/{profile_report.md, profile_results.json, profiling/}. The aiperf_rag_on/ directory is omitted. Filenames are profile_* because aiperf.enabled: false.

Single benchmark point with full report:

uv run --project scripts/rag-perf rag-perf -c scripts/rag-perf/configs/single_run.yaml

Output: flat run_<ts>/{report.md, results.json, results.csv, profiling/, aiperf_rag_on/}.

Concurrency sweep:

uv run --project scripts/rag-perf rag-perf -c scripts/rag-perf/configs/sweep.yaml

Output: nested run_<ts>/iter_1/<CR:_VDB-K:_RERANKER-K:_…>/{profiling,aiperf_rag_on}/ per point, plus aggregate report.md / results.json / results.csv at the run root.

Run unit tests:

uv sync --project scripts/rag-perf --extra dev   # one-time, installs pytest-asyncio
uv run --project scripts/rag-perf python -m pytest tests/unit/test_rag_perf/

Limitations

The CLI is config-only: author or copy YAML to vary a parameter.
load.concurrency / rag.vdb_top_k / rag.reranker_top_k accept int | list[int]; the validator requires unique list values because each value names a unique point dir.
input.file and input.synthetic follow an XOR rule — both set fails validation. When neither is set, synthetic auto-fills with defaults so a bare config still validates.
File-based input format is inferred from extension only (.jsonl or .csv); other extensions are rejected.
Synthetic generation streams each query to disk as it completes (failure-resilient) but fails fast on the first LLM error — partial JSONL is preserved. Re-run after fixing the endpoint.
Reasoning models (Nemotron Omni, Qwen-Reasoning) require synthetic.disable_thinking: true (the default). Without it the model exhausts the token budget on chain-of-thought and content returns empty — the generator now raises with a clear message instead of substituting reasoning_content for the answer.
aiperf-specific knobs outside the YAML surface (request rate distribution, GPU telemetry config, etc.) require editing AiperfRunner._base_aiperf_cmd in scripts/rag-perf/rag_perf/runner.py.
Procedural detail lives under references/ to keep this file concise.

Troubleshooting

Error / signal	Likely cause	What to do
`Configuration errors in <yaml>: • input — ... XOR rule`	Both `input.file` and `input.synthetic` set	Pick one. The XOR validator runs at YAML load time.
`input.file must end in .jsonl or .csv`	Extension other than `.jsonl` / `.csv`	Rename or convert.
`load.concurrency has duplicate values`	e.g. `[2, 2, 4]`	Each concurrency maps to a unique point dir; dedupe.
`warmup_requests must be >= 1`	YAML had `warmup_requests: 0`	aiperf rejects warmup=0; minimum is 1.
`LLM returned empty content (reasoning_content was populated — model exhausted its budget on chain-of-thought; raise min_query_tokens or set synthetic.disable_thinking=true).`	Reasoning model used CoT and ran out of tokens	Set `synthetic.disable_thinking: true` (the default) or raise `min_query_tokens`.
`✗ All N profiling requests failed across M point(s).` + exit 1	Bad URL, server down, wrong collection	Verify `target.url`, `rag.collection_names` (the `<collection_name>` placeholder will hit this).
Per-iteration `⚠ N profiling requests failed` warning, run continues	Some requests timed out / errored mid-run	Check rag-server logs, raise `target.timeout_s`, drop concurrency.
`RuntimeError: Random synthetic query generation failed at query N: ...`	LLM endpoint rejected a request mid-generation	Partial JSONL is at `synthetic.jsonl_output_path`; fix endpoint and re-run with reduced `num_queries`, or point `input.file` at the partial file.
`Citation count (mean): 0` and `Citation relevance score: N/A` for a non-empty deployment	Collection mismatch between `rag.collection_names` and what's actually ingested	Run `curl -s http://<ingestor>:8082/v1/collections` to list real collections.
Tests error with `ModuleNotFoundError: No module named 'pytest_asyncio'`	Dev extras missing	`uv sync --project scripts/rag-perf --extra dev`.
CI: `ModuleNotFoundError: No module named 'ruamel'` from `tests/unit/test_rag_perf/`	rag-perf package missing from CI venv	Add `uv pip install -e ./scripts/rag-perf` after the top-level install in the unit-tests job.

Gotchas

Run from repo root. Preset configs reference scripts/rag-perf/examples/queries.jsonl and scripts/rag-perf/prompts/default_prompts.yaml with repo-root-relative paths. Running from inside scripts/rag-perf/ will fail those file lookups.
CLI is config-only. Edit the YAML or copy a preset for URL, concurrency, collection, and similar fields.
Always edit rag.collection_names before the first run. The presets ship with ["<collection_name>"] as a deliberate placeholder. Validation passes, retrieval fails silently for every request — manifests as Citation count (mean): 0 everywhere.
load.concurrency_list, rag.vdb_top_k_list, rag.reranker_top_k_list are read-only properties that normalise scalar-or-list to a list. Use them when reasoning about the grid; the underlying YAML field is whatever the user wrote.
aiperf.enabled: false changes filenames. The top-level outputs become profile_report.md / profile_results.json / profile_results.csv. The aggregate sweep table also suppresses load-test rows and the "Optimal throughput" footer.
Resolved-config dump is verbose (50+ lines) — expected. It's what makes terminal output a self-contained reproducer; don't filter it out in scripts.
The aiperf shell command is logged before each subprocess. Look for \n $ python -m aiperf profile -m ... --endpoint-type nvidia_rag ... in stdout — copy-paste runnable for reproducing a single point outside rag-perf.
--endpoint-type nvidia_rag comes from the bundled plugin at scripts/rag-perf/rag_perf/plugin/nvidia_rag.py. It teaches aiperf about the RAG /v1/generate request shape and parses citations + per-stage metrics out of the SSE stream. If aiperf can't resolve nvidia_rag, rag-perf needs editable installation in the venv — re-run uv sync --project scripts/rag-perf (or uv pip install -e ./scripts/rag-perf).
Sweep-mode point-name collision. When two points differ only in concurrency (e.g. [1, 4] × single vdb_top_k), the dir name encodes everything: CR:1_ISL:50_OSL:512_VDB-K:20_RERANKER-K:4_Model:.... Cluster / GPU / experiment_name (output.cluster, output.gpu, output.experiment_name) are appended too — useful for diff-friendly artifact paths across machines.
load.iterations > 1 repeats the entire grid. Each repetition writes to its own iter_<i>/. Aggregate CSV row count = n_points × iterations.

Source of truth

Piece	Location
Driver	`scripts/rag-perf/rag_perf/cli.py` (`main` is the single Click command)
Schema	`scripts/rag-perf/rag_perf/config.py` (`RunConfig` and sub-models)
Orchestrator	`scripts/rag-perf/rag_perf/runner.py` (`BenchmarkRunner.run`, `RagProfiler`, `AiperfRunner`)
aiperf plugin	`scripts/rag-perf/rag_perf/plugin/nvidia_rag.py`
User-facing doc	`docs/performance-benchmarking.md`
Presets	`scripts/rag-perf/configs/{quick_profile,single_run,sweep}.yaml`
Sample queries	`scripts/rag-perf/examples/queries.jsonl`
Synthetic prompts	`scripts/rag-perf/prompts/default_prompts.yaml`
Config schema details	`references/config-schema.md`
Synthetic-query generation	`references/synthetic-generation.md`
Output layout & metric semantics	`references/output-and-analysis.md`

Agent playbook

Sync deps: uv sync --project scripts/rag-perf (one-time per checkout).
Pick & customise a preset: copy scripts/rag-perf/configs/<preset>.yaml if you want a variant; always set rag.collection_names to a real collection.
Run: uv run --project scripts/rag-perf rag-perf -c <config> from repo root.
Read the per-point + aggregate tables on stdout. Bottleneck inference is in the per-point profiling section; comparison across points is the final aggregate table.
Parse artifacts under output.dir/run_<ts>/ — see references/output-and-analysis.md. For multi-point runs, results.csv has one row per (point × iteration).
Summarise for the user using the playbook in references/output-and-analysis.md#summarising-results-to-the-user — headline table, scaling-efficiency math for sweeps, mandatory flags for zero citations / non-zero errors / suspect llm_ttft_ms / low sample size, and a concrete next-experiment YAML.
Tune retrieval / reranker: flip to quick_profile.yaml or aiperf.enabled: false for fast iteration, then return to single_run.yaml / sweep.yaml when characterising under load.
Triage failures: see Troubleshooting above and references/output-and-analysis.md for empty-citation / bottleneck=N/A patterns.

Files

ATTRIBUTION.mdAdded by AutoGineer886 B
BENCHMARK.md2.6 KiB
LICENSELicence file added by AutoGineer11.2 KiB
SKILL.md16.0 KiB
eval/h100.json2.4 KiB
eval/nvidia_hosted.json1.8 KiB
references/config-schema.md7.6 KiB
references/output-and-analysis.md12.8 KiB
references/synthetic-generation.md7.2 KiB
skill-card.md2.7 KiB
skill.oms.sig5.4 KiB

references/output-and-analysis.md

Output layout and result analysis

Load this when the user asks where artifacts went, how to interpret a metric, or what a column in results.csv means. Driver code: scripts/rag-perf/rag_perf/runner.py (BenchmarkRunner.run, _write_aggregate_outputs) and scripts/rag-perf/rag_perf/reporting.py (MetricsAggregator, Reporter, RagMetricsSummary).

Stdout sequence (in order)

Banner: ASCII "RAG PERF" logo + version.
Run-info summary: target URL, collection, vdb_top_k / reranker_top_k, input source, concurrency, total_requests, aiperf on/off. One-line per field, ~7 lines.
Resolved configuration: the full RunConfig dumped as YAML via RunConfig.to_yaml_str(). Verbose (~50 lines) by design — makes terminal output a self-contained reproducer. Don't strip in scripts.
Per grid point:
- Section rule: ─── Point N/M: conc=... vdb_top_k=... rr_top_k=... ───
- → Running profiling pass (collecting server-side metrics)...
- → Running aiperf load test (concurrency=..., requests=...)... (only when aiperf.enabled: true)
- aiperf's own per-iteration log lines (logger.INFO output from the subprocess)
- Copy-pastable shell command: \n $ python -m aiperf profile -m ... --endpoint-type nvidia_rag ...\n — useful for reproducing a single point outside rag-perf
- aiperf summary (its own table)
Per-point summary table (rich format, after each point completes in multi-point mode): "RAG-Perf Results — conc=N vdb_top_k=N rr_top_k=N" with stage breakdown bars, citation quality, bottleneck, load-test block.
Aggregate sweep table (multi-point only): "RAG-Perf Sweep — <varying axis>" side-by-side comparison. Auto-detects which axes vary; column header reflects the varying axis (concurrency / vdb_top_k / reranker_top_k / iter#). Footer: Optimal throughput: <axis>=<value> (X req/s) and Best p99 TTFT < 30s: <axis>=<value>.

If aiperf.enabled: false, the load-test rows in step 5/6 are suppressed and the optimal-throughput footer is hidden.

On-disk layout

Top level always: output.dir/run_<ts>/ (UTC timestamp YYYYMMDDTHHMMSS).

Single point + `iterations=1` + `aiperf.enabled=true`

run_<ts>/
├── report.md            # markdown summary of this point
├── results.csv          # one-row CSV
├── results.json         # single RagMetricsSummary dict
├── profiling/
│   └── profiler_records.jsonl
└── aiperf_rag_on/
    ├── inputs.json
    ├── profile_export_aiperf.csv
    ├── profile_export_aiperf.json
    ├── profile_export.jsonl
    └── logs/aiperf.log

Single point + `iterations=1` + `aiperf.enabled=false`

run_<ts>/
├── profile_report.md
├── profile_results.json
├── (no profile_results.csv if "csv" not in output.formats)
└── profiling/
    └── profiler_records.jsonl

No aiperf_rag_on/. profile_* filename prefix is the visual indicator.

Multi-point or `iterations > 1`

run_<ts>/
├── report.md            # aggregate, summarises all points
├── results.csv          # one row per (point × iteration)
├── results.json         # list of RagMetricsSummary dicts (or single dict if N=1)
└── iter_<i>/
    └── CR:<conc>_ISL:<isl>_OSL:<osl>_VDB-K:<vdb>_RERANKER-K:<rr>_Model:<model_clean>[_Cluster:<x>][_GPU:<y>][_Experiment:<z>]/
        ├── profiling/
        │   └── profiler_records.jsonl
        └── aiperf_rag_on/
            └── ... (same files as above)

<isl> is synthetic.min_query_tokens for synthetic mode, literal var for file-based mode (where ISL varies per query). <osl> is generation.max_tokens. <model_clean> is model_name with / replaced by -.

`RagMetricsSummary` fields (results.json / results.csv)

Defined in scripts/rag-perf/rag_perf/reporting.py.

Stage breakdown (profiling pass)

Field	Source	Notes
`stage_breakdown.rag_ttft_ms`	`metrics.rag_ttft_ms` from final SSE chunk	Total server-side TTFT
`stage_breakdown.retrieval_ms`	`metrics.retrieval_time_ms`	Vector DB retrieval
`stage_breakdown.reranking_ms`	`metrics.context_reranker_time_ms`	Reranker stage
`stage_breakdown.llm_ttft_ms`	`metrics.llm_ttft_ms`	LLM time-to-first-token
`stage_breakdown.llm_generation_ms`	`metrics.llm_generation_time_ms`	LLM full generation
`stage_breakdown.{retrieval,reranking,llm}_frac`	derived	Each stage as fraction of `rag_ttft_ms`
`stage_breakdown.bottleneck`	`argmax(retrieval_ms, reranking_ms, llm_ttft_ms)`	Stage name string

Citation quality

Field	Source
`citation_quality.mean_count`	Mean number of citations across requests
`citation_quality.{mean,p50,p90}_score`	Aggregations of per-citation `score` field

Citations land on the first SSE chunk. The profiler latches them on the first non-empty citations.results payload (server attaches them alongside the initial empty content delta, not the final chunk). Don't change this.

Client-side timing (profiling pass)

Field	Notes
`profile_client_ttft_p50_ms`, `_p90_ms`	Client-observed TTFT — includes network round-trip
`profile_client_e2e_p50_ms`	End-to-end latency for the profiling-pass requests

aiperf load-test fields

Field	Notes
`load_ttft_{mean,p50,p90,p99}_ms`	TTFT distribution under load
`load_e2e_{mean,p90,p99}_ms`	End-to-end latency under load
`load_throughput_tok_s`	Output-token throughput
`load_request_throughput`	Requests per second
`load_error_rate`	Failed / total

All None when aiperf.enabled: false (suppressed in tables).

Run metadata

Field	Notes
`concurrency`, `vdb_top_k`, `reranker_top_k`	Identifying axes — populated up-front in `_run_point`, before aiperf branches
`collection_names`, `total_requests`	Echoed from config
`profile_requests_failed`, `profile_requests_total`	If equal across all points → cli exits 1 (CI safety)

Quick analysis recipes

Pretty-print a single-point summary:

python3 -m json.tool rag-perf-results/<dir>/run_<ts>/results.json

One-row-per-point view of a sweep:

column -ts',' rag-perf-results/<dir>/run_<ts>/results.csv | less -S

Compare two sweep runs:

diff <(cat rag-perf-results/before/run_*/results.csv) \
     <(cat rag-perf-results/after/run_*/results.csv)

Replay a single aiperf invocation outside rag-perf: copy the \n $ python -m aiperf profile ... line from rag-perf's stdout — it's a self-contained shlex-joined shell command using the same temp queries JSONL.

Summarising results to the user

After a run finishes, follow this playbook to produce a tight report instead of dumping raw JSON.

1. Locate the canonical result file

Depends on run shape:

Shape	Read first	Then
Single point + aiperf	`run_<ts>/results.json` (single dict)	`run_<ts>/report.md` for the rendered tables
Single point + profile-only	`run_<ts>/profile_results.json`	`run_<ts>/profile_report.md`
Multi-point or `iterations>1`	`run_<ts>/results.csv` (one row per point × iter)	`run_<ts>/results.json` (list of dicts) for nested fields the CSV flattens away

Discover the latest run dir with:

ls -td rag-perf-results/<preset>/run_* | head -1

2. Extract the headline numbers

For each point pull these into a table:

Column	Path in `RagMetricsSummary`
Concurrency	`concurrency`
`vdb_top_k`, `reranker_top_k`	(same names, top-level)
Server RAG TTFT (mean)	`stage_breakdown.rag_ttft_ms`
Retrieval / Reranking / LLM TTFT	`stage_breakdown.{retrieval_ms, reranking_ms, llm_ttft_ms}`
Bottleneck	`stage_breakdown.bottleneck`
TTFT p50 / p99	`load_ttft_p50_ms`, `load_ttft_p99_ms`
E2E p99	`load_e2e_p99_ms`
Throughput (req/s, tok/s)	`load_request_throughput`, `load_throughput_tok_s`
Error rate	`load_error_rate`
Citation count / score (mean)	`citation_quality.mean_count`, `citation_quality.mean_score`
Profile-pass success ratio	`1 - profile_requests_failed / profile_requests_total`

If aiperf.enabled: false, load_* are all None — note "profile-only run" and skip the load-test column group.

3. Compute the unaccounted-time gap

unaccounted = rag_ttft_ms − (retrieval_ms + reranking_ms + llm_ttft_ms)

If unaccounted > a stage's reported time, the breakdown isn't telling the whole story (most often: llm_ttft_ms is mismeasured server-side and reads near zero, leaving most of the TTFT unattributed). Mention this in the summary as a caveat — don't let the user infer "the LLM is free."

4. Compute scaling efficiency (sweeps only)

For a concurrency sweep, compute throughput ratio vs concurrency ratio between the lowest and highest points:

scaling_efficiency = (req/s_max / req/s_min) / (concurrency_max / concurrency_min)

Linear scaling = 1.0; sub-linear < 1.0 indicates saturation. Pair with TTFT p99 ratio — >2× p99 worsening for <1.5× throughput gain is the canonical congestion signature; flag the knee location.

5. Signals worth calling out

Always flag in the summary, not just in passing:

Citation count (mean): 0 everywhere — collection mismatch. Suggest verifying with curl http://<ingestor>:8082/v1/collections.
load_error_rate > 0 — non-zero error rate in a benchmark is a finding, not a footnote. State the absolute count and the likely cause (saturation? timeouts?).
stage_breakdown.llm_ttft_ms < 1 ms — almost certainly a measurement bug, not a real number. Caveat any LLM-stage conclusions.
profile_requests_failed > 0 — partial profiling pass; the per-stage means may be skewed if the failures clustered.
Bottleneck stays constant across the sweep — informative: tells the user that scaling that axis doesn't shift the bottleneck (e.g. reranker stays dominant whether vdb_top_k=20 or 100 → reranker model is the real cost, not the chunk-count).
Tail-latency p99 from very low total_requests (< 50) — explicitly note that the tail is not statistically robust at that sample size; recommend bumping total_requests for follow-up.

6. Suggest concrete next experiments

Tie suggestions to the data, not generic advice. Examples:

"Reranker is the bottleneck at 23% of TTFT — try enable_reranker: false as a baseline to see how much accuracy you'd give up to drop that 164 ms."
"Throughput plateau between conc=4 and conc=8 — add concurrency: [1, 2, 4, 6, 8] to find the knee precisely."
"TTFT p99 jumps 3× for 1.7× throughput gain at conc=4 — the system is saturating; back off to conc=2 for SLA-bound traffic and use conc≥4 only when batched throughput matters more than tail latency."
"Citation score mean 0.58 with p90 0.80 is fine; if you want higher precision try reranker_top_k=2 and watch the per-citation score change."

7. Format the summary

Use a small fixed structure:

Run shape — preset, point count, iterations, profile-only or full.
Headline table — one row per point, columns from §2.
Findings — 3–5 bullets pointing at numbers in the table (cite the column).
Caveats — sample size, suspect metrics, anything in §5.
Recommended next config — concrete YAML diff or a "try this preset" line.

Aim for ~30 lines total. Long-form interpretation belongs in a follow-up if the user asks; the first response should be scannable.

Common patterns in results

Pattern	Likely cause
`Citation count (mean): 0` everywhere	Collection mismatch (placeholder `<collection_name>` left in config, or wrong collection name); verify with `curl http://<ingestor>:8082/v1/collections`.
`Citation relevance score: N/A` while count > 0	Citations returned without `score` field — server-side issue; check rag-server build.
`LLM TTFT: 0.4 ms`	Suspiciously low — likely a server-side metric measurement bug, not a real number. Don't infer optimisation conclusions from this stage alone.
Bottleneck stays at "RERANKING" across vdb_top_k sweep	Reranker is the dominant cost regardless of input fan-out at this scale. Try `enable_reranker: false` as a baseline.
TTFT p99 grows >2× while throughput grows <1.5× across concurrency	System saturation between those two concurrency levels. Add intermediate values to find the knee.
Sub-linear throughput scaling with high error rate	Server overloaded; lower concurrency or raise `total_requests` to get past warmup-noise.
`WARNING: usage was empty` (only in older outputs)	Pre-fix behaviour. Current build always populates usage from aiperf. If you see this on a current run, file a bug.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Declares allowed tools, Needs network access, Needs an API key, Needs Python

The author says: Repository checkout with uv; Python 3.11+; run from repo root; uv sync --project scripts/rag-perf (perf deps live in scripts/rag-perf/pyproject.toml); reachable RAG server (default http://localhost:8081); for synthetic queries an OpenAI-compatible chat-completions endpoint is required (default http://localhost:8999/v1/chat/completions); aiperf load-test phase uses the bundled nvidia_rag endpoint plugin, registered automatically when rag-perf is installed editable.

Security

What our scanner noticed· Medium risk

Large base64-encoded blobs
Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

NVIDIA RAG

Publisher

NVIDIA (Official)

Source

NVIDIA/skills · skills/rag-perf · commit fce3cca

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE-APACHE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by NVIDIA RAG or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download rag-perf.zip