Warp Eval

NVIDIAOfficialCreated by NVIDIA Corporation

Skill
Workflow
Software development
Apache-2.0
Medium risk
Has scripts

From the author

Evaluate whether an existing hot path is a credible NVIDIA Warp candidate. Use for irregular or spatial queries, particle or geometry simulation, branch-heavy loops, many small launches, host fallbacks, or large intermediates. CPU-only code and absent GPU dependencies are normal unless NVIDIA is prohibited. Exclude required cross-vendor or CPU-only deployment, vendor-lowered dense or NN layers, general Warp API questions, and already-selected Warp kernels. Contribution policy alone is not exclusion.

— the description by NVIDIA Corporation

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The warp-eval skill provides a structured, evidence-based methodology to determine if specific hot paths in an existing codebase are viable candidates for NVIDIA Warp. It emphasizes objective reporting over recommendations, requiring users to define clear performance metrics and semantic contracts before any prototyping occurs. The process includes static screening, rigorous baseline comparisons, and authorized benchmarking, ensuring that Warp is only considered after in-project alternatives have been exhausted. The skill enforces strict gates to abort evaluations that do not meet technical or policy requirements, ensuring that only credible candidates proceed to measurement.

What it does

Screens code bottlenecks against specific technical gates to determine Warp suitability.
Requires the establishment of a semantic contract and performance baseline before prototyping.
Enforces a ladder of in-project optimizations to be exhausted before introducing Warp.
Produces a standardized evaluation report containing measurements, evidence, and caveats.
Ensures all performance claims are backed by synchronized, end-to-end measurements.
Provides scripts for synchronized timing, memory profiling, and report validation.

When to use it

When investigating performance bottlenecks involving irregular queries, simulations, or branch-heavy loops.
When you need to objectively assess if Warp provides a measurable benefit over existing in-project baselines.
When you require a reproducible, audit-ready report on potential GPU acceleration candidates.
When you need to determine if a performance issue is material enough to warrant a new dependency.

Advantages

Prevents premature optimization by requiring algorithmic improvements first.
Provides a clear, objective framework to stop evaluations that are not technically viable.
Ensures performance measurements include realistic overheads like JIT, transfers, and synchronization.
Maintains production code integrity by working in isolated, independent patches.
Standardizes the reporting of facts, hypotheses, and unknowns.

Usage examples

“Evaluate if the particle simulation loop in my project is a good candidate for Warp.”: The agent will identify the candidate seam, derive the contract, check initial gates, and provide a resource preview before asking for authorization to proceed with profiling.
“I have a performance bottleneck in my geometry processing code; can you run a warp-eval?”: The agent will initiate the screening process, request the target repository, and begin identifying candidate patterns to determine if a formal evaluation is warranted.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/nvidia/warp-eval/download
Unzip it into the cross-client skills folder of your project:
```
mkdir -p .agents/skills && unzip warp-eval.zip -d .agents/skills
```

Documentation by NVIDIA Corporation

Warp evaluation

Purpose

Collect reproducible evidence about how a narrow seam in an existing codebase would behave in NVIDIA Warp. Report facts; the user decides.

Name Warp as the option under evaluation in the first line, state that no adoption recommendation will follow, and do not treat the triggering performance request as authorization to experiment.

Measured evaluations produce warp-evaluation-report/: the report, one independently applicable diff per solution, the drivers, and raw results. Never modify production code. Exits before measured work create no directory.

Hard rules

These override any local reasoning.

Default objectives are latency, throughput, and peak or retained memory. Count maintainability, ergonomics, packaging, extensibility, autodiff or new functionality only when the user names it; otherwise report them as constraints or costs, not benefits.
Correctness is a gate. If the incumbent is buggy or its contract unclear, abort that comparison until an independent oracle or clarified contract exists. Report the defect without prescribing a response.
Never predict savings from source shape or another workload. Materialized memory comes from profiler or allocator evidence, not source a compiler may fuse.
"It can be written in Warp" is a hypothesis, never a proven opportunity.
Gates are exact, never adjacent or analogous. Fire a gate only when every condition in its definition is established. Gate D requires a maintained implementation confirmed to execute with CUDA on an NVIDIA GPU; a fast native CPU library is a baseline, not Gate D.
Label every claim as observed fact, measurement, hypothesis or unknown.
Time the user-visible stage, not the kernel. Include Warp's cold import/init/JIT in the real process regime, transfers, launches, Python launch loops, structure build/refit, allocation, conversion, validation, compaction and synchronization. Report each cost and the end-to-end difference.
No measured work without explicit authorization. After stage 1, stop and ask before profiling, environment changes, prototyping, benchmarking or GPU use.
Authorized measured work includes Warp. If profiling exposes no gate, continue through the strongest in-project baseline, minimum Warp prototype and end-to-end comparison. If Warp is out of scope, stop before profiling.
Never infer environment intent. NVIDIA deployment and ownership of an optional compiled dependency are product decisions. Abort only on a stated constraint; otherwise ask once and stop (AWAITING INTENT).
Warp measurements are CUDA-only and synchronized. Resolve an explicit NVIDIA CUDA device; discard CPU resolution and dispatch-only timing.
Never recommend or rank adoption options. Report facts, measurements, hypotheses and unknowns per seam and regime.
Stop at the report. No rollout or production edits.

Requirements

Static screening requires the target repository and its stated product constraints. Measured work additionally requires explicit authorization, an NVIDIA CUDA GPU, target-project dependencies and a representative workload.

Limitations

Warp CPU kernels are serial and outside this skill's evaluation scope. "Run it on the CPU instead" is not a performance fallback.
Mesh and geometry queries compute in float32/int32 even behind a float64 public API. Absolute error grows with coordinate magnitude.
Warp does not follow semver. Feature releases can break APIs, only the newest feature line is maintained, deprecations run roughly four monthly releases.
latest docs track development, not the shipped release. Pin to the target project's own Warp version; if it has none, use the current stable and say which.
Kernel-only builtins are not resolvable from Python — hasattr(wp, "mesh_query_point") is False on a release that has it. Probe the stub file or that version's docs before concluding a builtin is absent.
enable_backward=False (kernel, module or global) removes adjoint codegen. If nothing differentiates through the seam, set it before measuring compile cost.

Output Format

State	Reached when	Report directory
`ABORT`	Any gate establishes that the evaluated seam cannot satisfy the stated scope	No if nothing was measured; otherwise preserve the evidence already collected
`AWAITING INTENT`	The environment gates turn on a fact only the user has	No — one question, both branches concrete
`AWAITING AUTHORIZATION`	A candidate pattern survives stage 1	No — early findings plus scope/resource preview
`INCOMPLETE`	Authorized work cannot obtain representative evidence required for the scoped evaluation	Yes — preserve collected evidence and name the one missing artifact
Report delivered	Authorized work produced measurements, or stopped after the report directory existed	Yes — facts per seam and regime, with missing evidence explicit

Use this exact shape for an early exit: ABORT — Gate <letter>: <cited fact>; <why the scoped Warp evaluation cannot proceed>. Do not name a preferred alternative. Reporting rules: references/evidence-and-reporting.md. A delivered directory follows the template's fixed order: schema and provenance, authorization and evaluation state, stage census, one B<n> evidence section per seam/regime, caveats, then environment/reproduction. solutions/, benchmarks/ and results/ contain every linked artifact.

Examples

Gate exit: ABORT — Gate A: deployment.md requires one implementation with AMD, Apple and NVIDIA parity; a Warp-specific path cannot satisfy this scope.

Surviving candidate: Name the seam and pattern, label inferred facts as assumptions, state that no gate has fired, preview the profile/baseline/Warp prototype/benchmark scope and its cost, then ask the separate intent and authorization questions from references/authorization-checkpoint.md.

Inputs

Required: the target repository and a performance, memory or scale problem with a candidate seam. Optional: explicit deployment/packaging constraints, existing profiles or logs, representative datasets and acceptance criteria. Prompt constraints take precedence over repository policy/configuration, then existing logs. User corrections override inference. Never substitute an assumption for a stated fact or measurement.

Available scripts

Script	Purpose	Arguments
`scripts/driver-template.py`	Copy once per bottleneck; define workloads and variants	Edit placeholders, then run the copied driver
`scripts/measure.py`	Import from drivers for synchronized timing, memory and isolated cases	Python API; do not execute directly
`scripts/validate_report_schema.py`	Validate the delivered report and evidence links	`<report-directory>`

Use run_script("scripts/validate_report_schema.py", args=["warp-evaluation-report"]) when supported; otherwise invoke the script with Python and the report directory.

Troubleshooting

No representative workload: mark the scope INCOMPLETE and name the missing artifact; do not invent data or fire Gate F.
Warp resolves to CPU or no CUDA device: discard the run and stop before correctness or timing claims.
Report validation fails: fix the report or referenced artifact; never waive the schema error.

Instructions

Every stage before the last can end the evaluation. Stop as soon as a gate fires; do not gather evidence that cannot change the scoped facts.

1. Read the code, derive the contract, check the gates

Identify a candidate and its metric with references/target-patterns.md.
Derive devices/residency, dtypes/shapes, sizes, frequency, process lifetime, gradients and packaging from the repository. Infer before asking.
State the inferred contract in one line and invite correction. Unknown hardware, counts, sizes and tolerances are assumptions, never measurements. Every inference remains open to correction and cannot satisfy a gate that requires a stated fact or measurement.
Check Gates A–E before profiling. Check Gate F now only if representative evidence already exists; otherwise carry it into stage 2. Every gate uses only the exact boundaries in references/rejection-gates.md.

Gate	Fires when
A	Production is stated CPU-only or to need non-NVIDIA portability, with no acceptable optional CUDA path
B	Data must cross the host/device boundary per small or infrequent call and the boundary cannot be widened
C	The region is dense tensor algebra already mapped to a tuned framework or vendor library
D	A mature CUDA implementation already meets the contract, and no non-performance objective was requested
E	A stated policy blocks Warp's dependency, compilation, cache or fallback obligations
F	Representative evidence proves the region too small a share of its requested metric for any backend to move it

Gate F can fire in stage 1 only from representative evidence that already exists — a supplied profile, structural bound, or arithmetic on figures the user quoted. If that evidence does not exist, Gate F remains open until stage 2 profiling; inferred values never fire it.
Gates A and E need a stated constraint. A CPU implementation, another accelerator, no Warp dependency, or a small dependency list proves nothing.
When A/E are unresolved and a pattern survives, ask whether an optional NVIDIA path is acceptable: named extra, soft import, existing fallback, default install unchanged. Every affirmative answer must say explicitly that Warp will be prototyped and benchmarked; conditions constrain only that Warp scope. A negative or undecided answer means ABORT.
Do not ask when another gate fired, the repository answers, or no pattern matched. No pattern means no profiling.
If a candidate survives, combine any intent question with the authorization checkpoint — early findings, exact scope, stages, resource cost — then stop. Stage 2 requires both settled intent and explicit authorization.

2. Profile the real application

Requires explicit authorization and a settled intent question.

Profile with the project's own profiler and representative entry points.
Measure synchronized end-to-end stage time and peak memory before choosing a backend. Report which entry points were profiled and which a gate screened.
Prioritize further measurement by observed cost, not source appearance.
Name each measurement by the public method and variant actually invoked. A fallback is an execution regime of that public seam, not a different operation, and a cheaper sibling method cannot screen out the named method.
Confirm the timed branch ran on the intended device. Unchanged cost and near-zero device allocation between host and device inputs exposes a host fallback.
Measure the stage's free-stage ceiling and stubbed floor through the public boundary. Do not subtract per-op timings.
If the measured ceiling proves the candidate cannot move its own metric, ABORT the affected scope under Gate F, preserve the evidence already collected, and stop. The existing authorization already covered this materiality check; do not ask for authorization again.
If representative coverage is unavailable — no representative dataset, runnable entry point or production distribution — record the single missing artifact, mark the affected scope INCOMPLETE, and stop. This is missing evidence, not Gate F and not ABORT. An invented workload cannot prove materiality.

Protocol: references/benchmark-protocol.md.

3. Form falsifiable hypotheses

Record per candidate: source, bottleneck evidence, objective, narrow seam, mechanism Warp could change, strongest incumbent, risks, acceptance threshold, cheapest falsifying experiment. Screen against references/target-patterns.md; if none survives, write the report and stop.

4. Write the contract before the prototype

Define values, dtypes, shapes, devices, errors, mutation, ordering, ties, capacity/overflow, topology/degeneracy, tolerances, required gradients, streams, ownership, aliasing, invalidation, concurrency, capture, teardown and fallback.

Fix tolerances before seeing Warp output. Never weaken a contract after a mismatch.
ABORT before prototyping if the proposed seam cannot satisfy a required contract.
Pre-register, before timing: workload provenance, the state variable and production range controlling cost, tuning knobs, incumbent run-to-run spread, and the oracle applied to every implementation.

Hazards and adversarial checks: references/semantic-contract.md.

5. Improve the baseline first

Algorithm before backend: (1) a better or output-sensitive algorithm; (2) chunking, tiling, sparse output, layout, rematerialization; (3) the incumbent framework's compiler and native primitives; (4) what the project already depends on — its own accelerator backend, a parallel idiom it ships but leaves off, or a capability an existing dependency exposes and nobody wired up; (5) only then narrow Warp.

Compare only in-scope options: the improved incumbent, capabilities reachable through current dependencies, and Warp. Do not add unrelated libraries.
Search declared dependencies for dormant backends, flags and bindings before calling a route absent.
Keep the improved incumbent as the oracle. A result against an untuned, incorrect or asymptotically inferior baseline does not establish a backend comparison.

Close every in-project route before prototyping Warp:

State	What it takes to claim it
measured	timed through the same boundary as the baseline
absent	a cited declaration, symbol table or missing flag proves it is unavailable
waived	you asked the user and they chose to skip it; record their words

A capability present but unbound is reachable, not absent. If exposing it costs no more than the planned Warp seam, measure it first. Ladder details: references/baselines.md. Waived routes do not block stage 6; every route not explicitly waived must be measured or evidenced absent before the Warp prototype begins.

6. Prototype the minimum unit

Work in a separate copy, production unchanged, integration seam off by default. Prototype only enough to test the hypothesis.
Select an explicit CUDA device for every Warp prototype and verify that Warp resolves it as CUDA before correctness or performance work. Never exercise or report Warp's CPU backend.
Size the unit by shared data and structure lifetime, not function boundaries: build/refit/query costs that amortize together are one unit.
Name variants before timing. Preserve each solution as an independent patch against the pinned baseline, and prove it applies cleanly and reproduces the measured result.
Surface unsupported cases and overflow before timing.
Run the adversarial contract checks, audit correctness at production scale, and deliberately break a branch to prove the tests fail. Any failed required check returns ABORT for the affected scope.
Before treating a disappointing compute-bound or reuse-heavy result as representative, consider the stronger formulations in references/target-patterns.md. One naive kernel does not bound Warp's potential.

7. Benchmark the whole boundary

Copy scripts/driver-template.py once per bottleneck and measure through scripts/measure.py. Do not hand-roll timing or memory.
Warp launches are asynchronous. The measurement helper must synchronize the selected device immediately before starting and after enqueueing every timed region, before stopping its wall timer.
Include every rule-7 cost, the public call, the immediate downstream stage, realistic sizes, and cold/warm process regimes.
Transcribe every report cell from emitted JSON; absent records read not measured.
Report null_test, and one-time costs both separately and amortized. Serialize GPU measurements under an exclusive device lock. Below 1.5× is no measured difference.
ABORT for a seam and regime whose predeclared end-to-end performance or memory requirement fails, after preserving the measurements.

Full protocol: references/benchmark-protocol.md.

8. Summarize the evidence

Per seam and execution regime, report semantic results, strongest-baseline measurements, workload provenance, time, memory, lifecycle, portability and ownership facts.
Use pass, fail, not measured, not available, no representative data, unknown or n/a only where a stated criterion makes that status objective. Missing evidence remains missing.
Mark the report complete only when every evaluated seam and regime includes an end-to-end Warp measurement. Use aborted — <gate and scope> when a gate fired, or incomplete — <missing evidence and scope> when representative evidence was unavailable. Preserve everything collected. Never deliver an incumbent-only report as a completed warp-eval.
Give every stage worth ≥10 % of the measured total a census row with a status and a one-line reason, including the screened-out stages.
Check each gate against the metric that candidate's pattern exhausts, not the study's headline objective.

9. Hand back

Use assets/warp-evaluation-report-template.md unchanged in schema. Record authorization and scope.
One table per bottleneck: baseline, current-dependency solutions, then Warp; absolute time and peak memory, ratios, contract status, evidence gaps.
Include prose only where it explains the measurements or their bounds. Working artifacts belong in results/.

uv run python scripts/validate_report_schema.py <report-directory>

Run the validator once the first measurement establishes a census/table, and again before delivery. Fix every error.
Verify builds and imports from artifacts, not exit status. Ship the exact drivers run. Then stop.

Files

ATTRIBUTION.mdAdded by AutoGineer897 B
BENCHMARK.md4.3 KiB
LICENSELicence file added by AutoGineer11.2 KiB
SKILL.md19.2 KiB
assets/warp-evaluation-report-template.md5.8 KiB
config/skillspector-baseline.yaml475 B
evals/evals.json22.3 KiB
evals/files/gate-a-required-portability/deployment.md638 B
evals/files/gate-a-required-portability/neighbors.pyScript872 B
evals/files/gate-b-unwidenable-host-boundary/architecture.md794 B
evals/files/gate-b-unwidenable-host-boundary/scoring.pyScript880 B
evals/files/gate-c-dense-vendor-lowered/feature_stage.pyScript615 B
evals/files/gate-d-cpu-incumbent-not-cuda/baseline.md472 B
evals/files/gate-d-cpu-incumbent-not-cuda/operation_api.pyScript440 B
evals/files/gate-d-mature-incumbent/baseline.md960 B
evals/files/gate-d-mature-incumbent/contact_api.pyScript948 B
evals/files/gate-e-runtime-obligations-forbidden/candidates.pyScript735 B
evals/files/gate-e-runtime-obligations-forbidden/packaging-policy.md845 B
evals/files/gate-f-immaterial-stage/profile.json603 B
evals/files/gate-f-immaterial-stage/spatial_filter.pyScript646 B
evals/files/pattern-fused-edge-intermediate/context.md742 B
evals/files/pattern-fused-edge-intermediate/message_passing.pyScript521 B
evals/files/pattern-sequential-recurrence/adaptive_filter.pyScript462 B
evals/files/pattern-sequential-recurrence/context.md596 B
evals/files/proceed-persistent-mesh-query/context.md767 B
evals/files/proceed-persistent-mesh-query/queries.pyScript804 B
references/authorization-checkpoint.md3.8 KiB
references/baselines.md12.2 KiB
references/benchmark-protocol.md16.3 KiB
references/evidence-and-reporting.md7.2 KiB
references/rejection-gates.md8.2 KiB
references/semantic-contract.md13.3 KiB
references/target-patterns.md16.3 KiB
scripts/driver-template.pyScript3.4 KiB
scripts/measure.pyScript20.8 KiB
scripts/validate_report_schema.pyScript13.5 KiB
skill-card.md4.4 KiB
skill.oms.sig12.5 KiB

references/semantic-contract.md

Semantic contract and correctness hazards

Write the contract before building anything; the hazard catalog is what adversarial testing must cover once you do. Each hazard here has, on its own, decided a migration.

Write the contract before the prototype

Tolerances chosen after seeing Warp's output are not tolerances. Write down first:

Observable behavior — values, dtypes, shapes, devices, error conditions, what may be mutated; ordering, tie-breaking, capacities, overflow, partitions, topology, degeneracy; which outputs are semantic and which are implementation artifacts (arbitrary IDs may be free to vary, a partition or a selected face may not); numerical, geometric and task-level tolerances, each with a number; gradients only where already required or explicitly requested.

Lifecycle — which stream work launches on and who synchronizes; who owns inputs, outputs, caches and acceleration structures; whether any returned array views internal storage; what mutation triggers refit versus rebuild; how many builders/caches may exist per device, stream and thread, and what concurrent misuse does; what must stay static and alive across graph capture and replay; what happens when the GPU is absent, compilation fails, or an unsupported dtype/device arrives.

Two rules over both:

For an API that already ships, its docstrings and tests are the contract. Read them as binding rather than inferring a looser one the prototype happens to satisfy.
Correctness and required gradients veto performance. Never weaken a contract after a mismatch. If the mismatch is acceptable, that is a product decision someone makes explicitly — record it as one.

Hazard catalog

Cutoff ties and float classification. Specify < versus <= and the arithmetic precision exactly. Test a point exactly on the cutoff and one representable step either side. Expect a handful of disagreements per millions of pairs, and expect two implementations of the same incumbent framework to disagree with each other. Decide up front whether production needs bit-identical sets, a geometric tolerance, or only invariant downstream quantities.

Truncation and overflow. A capacity-sized count cannot distinguish "exactly full" from "truncated" — return the unclamped count or an explicit overflow bit. Traversal order is usually unspecified, so above capacity test only that retained entries are valid unique members of the true set; do not promise nearest or reference ordering unless you implemented it. Size capacity from clustered and adversarial distributions plus telemetry, never one average-density sample.

Face and distance-type ties. Equal distances can select different valid faces. Harmless for a scalar distance; fatal when the selected face and a multi-way distance type feed the backward pass and are user-visible. Test exact duplicate faces, neighboring coplanar faces, shared edges/vertices and degenerate triangles, and re-test on every version upgrade — tie order is rarely a documented guarantee.

Measure how often the ambiguity is reachable before judging a mismatch rate. On a closed mesh the closest point lands on a shared edge or vertex far more often than in a face interior, so most face indices can be legitimately ambiguous — while the same comparison on synthetic triangle sets runs orders of magnitude cleaner. Report the reachability statistic next to the mismatch count, or the count means nothing.
A raw disagreement rate measures the implementation you built, not the seam's potential: taking whatever face the acceleration structure returns will disagree constantly, while a kernel that reproduces the incumbent's branch order can drive the same comparison near zero. Say which you measured.

Gradients at edges, vertices, seams and medial axes. The closest face and its normal are not unique at nonsmooth locations. An all-point maximum gradient difference is usually dominated entirely by the nondifferentiable region and says nothing about smooth-region accuracy — report the two separately. Use finite differences away from topology changes. Test repeated backward and every gradient operand. A shared existing backward is safe only if the IDs and types you feed it satisfy its original assumptions.

Topology drift. Tiny field differences near zero or near sampling boundaries change marching-cubes face indices, so a result can preserve exact vertex, face and component counts and exact bounds and still not be byte-identical. Validate the downstream artifact: component counts, largest component, surface area, bounds, watertightness, bidirectional geometric distance.

Float32 scale behavior. Warp mesh queries operate in float32/int32 even behind a float64 public API. Relative error stays tiny while absolute error grows with coordinate magnitude — at large scales, float32 spacing alone can put absolute point error in the tens at a relative error still around 1e-7. Sweep realistic and extreme scales, state the supported coordinate envelope, and never silently downcast a float64 contract.

Nondeterministic atomics and reductions. Atomic append or union order changes edge/label order, which changes downstream floating-point rounding. Compare semantic sets and partitions, and deterministic public labels where the contract requires them. Define a physical-invariant tolerance rather than demanding bit equality.

Measure the incumbent's own run-to-run spread first. Call it repeatedly on identical inputs and record the spread of every output: that is the floor for any equality contract. A tolerance tighter than the floor describes a test the incumbent itself fails.
"Bitwise equal" looks like the safest thing to pre-register and is not: a backward kernel accumulating with atomics can be reproducible only to a few parts in 1e5, making a bitwise contract unsatisfiable by anything, including itself. Establishing the floor measures the baseline, so it belongs before the contract rather than compromising it.
When both implementations select identical indices, any residual difference in an atomically-accumulated output is ordering noise by construction: assert the index equality, then bound the value difference by the measured floor.

Output aliasing. A builder returning a view into internal storage overwrites previous outputs on the next call. Either name the alias as a borrowed view or return fresh framework-owned arrays. Test two live outputs across a subsequent call and a garbage collection.

Mutable external state. A functional framework model does not make an external hash grid immutable. A side-effect token orders calls within one executable; it does not serialize independent host threads or executables. Scope builders per stream/executable, document non-shareability, and either test or reject concurrent use.

Stream and lifetime ordering. Zero-copy pointers need both storage lifetime and CUDA ordering. Record a readiness event after structure construction when another stream may query. Keep input storage, Warp objects, cache wrappers, captured graphs and output owners alive through final stream completion. Test non-default streams, producer/consumer events, deletion before completion, and repeated create/destroy.

Pointer-keyed graph caches. Graph capture can be keyed by call descriptor and buffer pointers. Capture counts usually stabilize after a few distinct pointer sets, but allocator churn can exceed the graph-cache limit and cause repeated recapture and eviction — measure capture count and latency under realistic churn, since stable warm samples hide it.

Probe retention explicitly; it is not visible any other way. Take a weak reference to the acceleration structure and its owning cache, capture the graph, drop every strong reference the caller would plausibly drop, force a collection, and assert the weak references are still alive. A captured graph can retain raw pointers without retaining the Python object that owns the underlying structure, so a replay after the caller released its cache runs against a freed handle — while every eager test passes and every warm replay looks correct. Do this even when the seam does not itself capture graphs: a persistent-index API invites a caller to capture around it. If a convenience path builds a cache internally and does not return it, say plainly that the path is not capture-safe.

Bounds, pruning and early exit. Any argument that lets an implementation skip work — an admissible lower bound, a monotonicity claim, a conservative filter, a hierarchical cull, an early exit — is a correctness proof obligation, not an optimization detail. An inadmissible bound produces a plausible, fast, wrong answer that will look like your best-performing variant, and it is impossible to catch by inspection or by timing. Before benchmarking such a variant, brute-force it against the unpruned implementation on randomized inputs at scale and confirm it never selects a worse result under the contract's own metric. State the bound in the report so a reader can check the algebra. An unvalidated pruned variant must not be quoted as a performance result.

Optional dependencies and version drift. Pin or qualify Warp and every compiled dependency across the supported matrix. Lazy imports must preserve the default path when Warp is absent. Re-run tie, graph, stream and build tests on every Warp/framework/CUDA upgrade.

Validate the oracle, not just the prototype

Every check compares the prototype against the incumbent, which is evidence only if the incumbent is the real one, running correctly, in the configuration you think.

Assert an independent invariant on the incumbent's own output — a quantity derivable without either implementation (in-range row count, a conservation law, a known total). Print it next to the comparison.
Prefer a third, independent reference. Agreement between two implementations tells you nothing when both share a wrong assumption. When the third disagrees with both accelerated ones, suspect the reference first — unfused host and fused device arithmetic legitimately disagree near boundaries.
Re-derive the incumbent from the project rather than transcribing it, and check you extracted the configuration you meant: a kernel builder emitting both CPU and device variants will hand you the device variant carrying the host reduction body, which silently loses work under contention and makes any prototype look correct and fast.
A mutation test on the prototype does not cover this — the mutant dies against a broken oracle exactly as against a good one.
Score every implementation against the oracle, including the incumbent, and report their errors separately. Scoring only the candidate turns shared benign numerical noise into an apparent defect of the new code: a float64 oracle disagrees with a float32 candidate on near-tied points, which reads as a semantic failure until the incumbent is scored against the same oracle and disagrees on the identical points. differs on N is not a finding; "candidate wrong on N₁, incumbent wrong on N₂" is.

Adversarial test checklist

Run before any performance measurement:

empty, singleton, and all-identical inputs;
duplicates and exact ties;
values exactly at, one step inside, and one step outside every threshold;
capacity below, equal to, and above demand; overflow surfaced correctly;
degenerate geometry and exact degeneracies;
extreme scales, very small and very large;
noncontiguous arrays and unsupported dtypes hitting the fallback;
tracked mutation, then refit and rebuild;
non-default streams; repeated calls; repeated backward;
graph capture, replay with changed input, and recapture;
a large random audit at production scale, sized to the defect rate you need to detect;
every bound, pruning rule or early exit brute-forced against the unpruned path before it is benchmarked;
the incumbent's own run-to-run spread, before any equality tolerance is fixed;
both implementations scored against the oracle, errors reported separately;
weak-reference retention across graph capture and caller release;
the unchanged upstream suite — and check what it covers, not just that it passes. A suite whose seam tests are all float64 and a few dozen elements routes every case to the fallback and exercises none of a float32 fast path, protecting nothing while appearing green.

Never write a test that asserts a hazard reproduces

"The stale cache must return wrong answers" makes a failure mode the pass condition, and fails whenever the implementation is better than you assumed or the race does not fire. Characterize the frequency instead (0/20 runs disagreed) and record it as an observation.

A hazard that does not reproduce is not cleared, it is unquantified. Report the trial count and keep the mitigation.
A safety property stronger than you assumed is a finding, not a failed test. An admissibility check that refuses unproven candidates and falls through to the exact path converts cache invalidation from a correctness contract into a performance one — a materially safer API that a suite insisting the hazard fire would have recorded as a failure.

Finally: deliberately break a key branch and confirm the suite catches it. A suite that cannot fail has validated nothing.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

The author says: Screening, static evaluation and reporting need no GPU. Measuring Warp requires an NVIDIA CUDA GPU, the target project's dependencies and a representative workload; without them, abort before profiling.

Security

What our scanner noticed· Medium risk

Large base64-encoded blobs
Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

NVIDIA Corporation

Publisher

NVIDIA (Official)

Source

NVIDIA/skills · skills/warp-eval · commit fce3cca

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE-APACHE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by NVIDIA Corporation or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download warp-eval.zip