Tao Analyze Gaps Od Map

NVIDIAOfficialCreated by NVIDIA Corporation

Skill
Review and audit
Software development
Apache-2.0
Medium risk
Has scripts

From the author

Run TAO Data Services object-detection gap analysis from ground-truth and inference annotations. Use when an object detection workflow needs to identify weak images by comparing model predictions against ground truth using per-class recall, precision, and AP50 thresholds. Use when the user asks to "analyze OD gaps", "find weak OD images", or "run mAP gap analysis".

— the description by NVIDIA Corporation

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The TAO Analyze Gaps OD mAP skill identifies weak images in an object detection dataset by evaluating model performance against ground truth. It computes metrics such as TP, FP, FN, and AP50 on a per-image and per-class basis. By applying configurable thresholds, the skill flags images where model performance is insufficient, providing actionable data for downstream mining and model improvement workflows. It requires pre-existing inference annotations and ground truth data to function.

What it does

Compares ground-truth and inference annotations to identify model performance gaps.
Calculates per-image and per-class metrics including recall, precision, and AP50.
Generates a list of weak images based on user-defined performance thresholds.
Produces detailed parquet reports for box-level gaps and image-level metrics.
Validates configuration specifications to ensure correct input formats and pathing.
Outputs a JSON gap report summarizing FP/FN counts and run settings.

When to use it

When you need to identify images where an object detection model is underperforming.
When preparing data for active learning or image mining workflows.
When you need to audit model predictions against ground truth annotations.
When you need to generate a quantitative report on model weaknesses per class.

Advantages

Enables targeted data collection by identifying specific weak images.
Supports both KITTI and COCO annotation formats.
Provides granular metrics for per-class performance analysis.
Ensures reproducibility by linking output artifacts with the specific experiment configuration.

Usage examples

“Analyze OD gaps for my current model predictions.”: The agent will guide you to configure the object_detection.yaml spec with your annotation paths and run the gap analysis container to identify weak images.
“Find weak OD images where car recall is below 0.9.”: The agent will help you set the weak_thresholds in your spec to include a car recall threshold of 0.9 and execute the analysis to filter the dataset.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/nvidia/tao-analyze-gaps-od-map/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip tao-analyze-gaps-od-map.zip -d .agents/skills

Documentation by NVIDIA Corporation

TAO Analyze Gaps OD mAP

Use this skill to run TAO Data Services object-detection gap analysis. The skill compares ground-truth and inference annotations, computes per-image per-class TP/FP/FN/AP50 metrics, and identifies weak images where any class metric falls below its threshold. It does not run inference; upstream steps must produce the inference annotations first.

The container entrypoint is:

gap_analysis object_detection -e /absolute/path/to/object_detection.yaml

Inputs

Required spec fields:

Field	Meaning
`ground_truth_ann_path`	KITTI label directory or COCO `.json` with ground-truth boxes.
`inference_ann_path`	KITTI label directory or COCO `.json` with model predictions.
`images_dir`	Root image directory. Establishes the full image universe including unannotated images.
`results_dir`	Output directory for all artifacts.
`kpi`	Identifier tag written to every output row.
`input_format`	`kitti` or `coco`. Must be declared explicitly; never inferred from the path.

Common optional fields:

Field	Default	Meaning
`iou_threshold`	`0.5`	IoU at or above which a prediction is accepted as a true positive.
`conf_threshold`	`0.0`	Predictions below this confidence are dropped before matching.
`min_area`	`0`	Boxes whose pixel area (w × h) is strictly below this value are discarded.
`class_mapping`	`{}`	Maps raw annotation label strings to canonical class names. Absent labels are kept as-is.
`weak_thresholds`	`{}`	Per-class thresholds as `{class_name: {recall, precision, ap50}}`. Absent keys fall back to the `default_*_threshold` values. Reference ITS defaults: `car 0.99`, `bicycle 0.7`, `person 0.7` — a strict gate on the abundant, well-learned class and looser gates on the rare ones the loop exists to improve.
`default_recall_threshold`	`0.5`	Fallback recall threshold for classes not listed in `weak_thresholds`.
`default_precision_threshold`	`0.0`	Fallback precision threshold. Set to `0.0` to disable precision-based weak selection.
`default_ap50_threshold`	`0.5`	Fallback for classes absent from `weak_thresholds`. Set `0.0` so unlisted classes never mark an image weak — the reference filter had no fallback, and leaving TAO DS's `0.5` in place silently gates every class you did not list.

Do not hand-write the spec. Copy the template and fill in the nulls — every tuning value it already carries is the one this stage wants — then validate:

cp skills/data/tao-analyze-gaps-od-map/assets/default_object_detection.yaml "$SPEC"
# fill ground_truth_ann_path, inference_ann_path, images_dir, results_dir, kpi, input_format
python3 skills/data/tao-analyze-gaps-od-map/scripts/verify_object_detection_spec.py --spec "$SPEC"

verify rejects the spellings that fail — uppercase input_format, relative or missing paths, a weak_thresholds entry that is a bare number rather than a mapping — and reports every gated class plus the default_* fallbacks, so the selection criteria behind a weak set are recoverable from the run's output. It warns when a fallback is above zero, since that gates classes you did not list.

Quick Start

Run from the tao-skill-bank repo root.

Write the spec into the results directory. The run emits four artifacts and does not retain the spec, so a completed gap analysis otherwise cannot tell you which thresholds produced its weak set — and that weak set sizes the mining budget downstream. Keeping them together makes the selection criteria recoverable from the run alone.

RESULTS_DIR=/absolute/path/for/this/run          # results_dir in the spec
SPEC="$RESULTS_DIR/object_detection.yaml"        # spec lives beside its outputs
RUN_ROOT=/absolute/path/that/contains/annotations/images/and/results
GPU_COUNT=1

DS_IMAGE="$(scripts/resolve_versions_key.py images.tao_toolkit.data_services)"

RUNTIME_HOME="$RUN_ROOT/.tao-runtime/home"
mkdir -p "$RUNTIME_HOME/.cache"

docker run --rm --gpus "$GPU_COUNT" --shm-size=8g --network=host \
  --user "$(id -u):$(id -g)" \
  -e USER="$(id -un)" -e LOGNAME="$(id -un)" \
  -e HOME="$RUNTIME_HOME" -e XDG_CACHE_HOME="$RUNTIME_HOME/.cache" \
  -v "$RUN_ROOT:$RUN_ROOT" \
  -w "$RUN_ROOT" \
  "$DS_IMAGE" \
  gap_analysis object_detection -e "$SPEC"

Keep --user and the identity env block above. TAO DS images call getpass.getuser() at startup; without USER/LOGNAME they fail with KeyError: 'getpwuid(): uid not found: <uid>' because the UID has no /etc/passwd entry. Do not work around it by running the container as root; that leaves root-owned outputs. This follows the launch contract in tao-run-on-docker.

Preflight

Verify Docker access:

docker info > /dev/null

Resolve and pull the data-services image if needed:

DS_IMAGE="$(scripts/resolve_versions_key.py images.tao_toolkit.data_services)"
docker image inspect "$DS_IMAGE" > /dev/null || docker pull "$DS_IMAGE"

Confirm RUN_ROOT contains the spec, both annotation sources, and the image directory. Mount RUN_ROOT to the same absolute path inside Docker.

Outputs

Artifact	Location	Contents
FP/FN box gaps	`results_dir/box_gaps.parquet`	One row per unmatched box: `kpi`, `image_id`, `filepath`, `class`, `gap_type` (FP/FN), `bbox`, `confidence`, `best_iou`.
Per-image metrics	`results_dir/image_metrics.parquet`	Per-image per-class: `tp`, `fp`, `fn`, `precision`, `recall`, `ap50`.
Weak images	`results_dir/weak_images.parquet`	Images where any class metric falls below threshold: `filepath`, `weak_classes`, `weak_recall`, `weak_precision`, `weak_ap50`. Feed this into `tao-mine-od-images`.
Gap report	`results_dir/gap_report.json`	FP/FN counts by type and class, plus run settings.

All four artifacts are always written, even when no gaps are found.

Troubleshooting

The subtask object_detection requires -e/--experiment_spec_file: rerun with gap_analysis object_detection -e "$SPEC".

Input path not found inside Docker: use a RUN_ROOT mount where host and container paths are identical.

input_format error: set input_format: kitti or input_format: coco explicitly — it is never inferred from the path.

weak_images.parquet is empty: all class metrics are above their thresholds. Lower default_recall_threshold / default_ap50_threshold or add per-class entries to weak_thresholds.

Output directory not writable after Docker exits: the container ran without --user, so it wrote as root. Relaunch with the --user and identity env block shown above.

Files

ATTRIBUTION.mdAdded by AutoGineer939 B
BENCHMARK.md6.9 KiB
LICENSELicence file added by AutoGineer11.2 KiB
SKILL.md7.2 KiB
assets/default_object_detection.yaml1.2 KiB
config/skillspector-baseline.yaml1.3 KiB
evals/evals.json948 B
references/skill_info.yaml660 B
scripts/verify_object_detection_spec.pyScript8.8 KiB
skill-card.md4.9 KiB
skill.oms.sig5.5 KiB

BENCHMARK.md

Skill Benchmark: tao-analyze-gaps-od-map

✅ Overall verdict: PASS — Recommended for publication

Publication Recommendation

Recommended for publication based on the completed evaluation evidence in this report.

Evaluation Metadata

Skill: tao-analyze-gaps-od-map
Evaluation date: 2026-10-16
Evaluator version: 1.5.6
Agents: Claude Code (aws/anthropic/bedrock-claude-opus-4-8), Codex (openai/openai/gpt-5.5)
Tasks: 1 evaluation tasks (1 positive)
Dataset digest: sha256:201d53df8972c71637b4d01e11df157458227c64bee4b81840057d3aa1144da7 (skill-evaluator-dataset-snapshot/1)
Attempts per task: 1
Environment: k8s-sandbox
Tier 2 evidence: required for publication
Tier 3 evidence: required for publication

Each task attempt ran in its own isolated sandbox pod.

What This Report Answers

The three-tier evaluation checks whether the skill:

is safe to use;
produces correct answers;
is discovered and activated when needed;
helps the agent complete the user's goal and expected workflow; and
avoids wasted skill and tool usage.

Results at a Glance

Measure	Claude Code (Baseline → Skill Uplift)	Codex (Baseline → Skill Uplift)
Overall	92.1% — baseline ran, but no comparable score was available; uplift unavailable	96.2% — baseline ran, but no comparable score was available; uplift unavailable
Security	100.0% → 100.0% (±0.0 points)	100.0% → 100.0% (±0.0 points)
Correctness	0.0% → 100.0% (+100.0 points)	20.0% → 100.0% (+80.0 points)
Discoverability	100.0% — baseline ran, but no comparable score was available; uplift unavailable	95.0% — baseline ran, but no comparable score was available; uplift unavailable
Effectiveness	12.5% → 87.5% (+75.0 points)	40.0% → 87.5% (+47.5 points)
Efficiency	73.1% — baseline ran, but no comparable score was available; uplift unavailable	98.5% — baseline ran, but no comparable score was available; uplift unavailable

How to read this table: baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points.

Example: 47.0% → 92.0% (+45.0 points) means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline.

Token Usage

Actual Tier 3 execution usage is reported for every observed agent/case pair and both conditions.

Agent	Dataset case	With skill	Without skill	Delta	Change	Coverage
claude-code	All cases	95,041	31,429	+63,612	+202.40%	skill 1/1; base 1/1
claude-code	tao-analyze-gaps-od-map-basic	95,041	31,429	+63,612	+202.40%	skill 1/1; base 1/1
codex	All cases	29,851	13,508	+16,343	+120.99%	skill 1/1; base 1/1
codex	tao-analyze-gaps-od-map-basic	29,851	13,508	+16,343	+120.99%	skill 1/1; base 1/1
ALL AGENTS	Dataset aggregate	124,892	44,937	+79,955	+177.93%	skill 2/2; base 2/2

Prompt tokens include cached reads, so total tokens are prompt + completion (cached is not added twice). The Efficiency score uses (prompt - cached) + completion. N/A means the relevant trajectory counters were not available; coverage is never estimated.

Tier Status

Tier	Purpose	Status	Evidence
Tier 1	Static validation	PASSED WITH OBSERVATIONS	11 validator(s); 19 finding(s)
Tier 2	Semantic deduplication	PASSED	2 validator(s); 0 finding(s)
Tier 3	Live agent evaluation	PASS	2 agent(s); 1 task(s)

Findings and Observations

<details> <summary>Show detailed findings and successful checks</summary>

MEDIUM QUALITY/quality_correctness: No documented scripts in table format (skills/data/tao-analyze-gaps-od-map/SKILL.md)
MEDIUM QUALITY/quality_correctness: Instructions don't mention 'run_script' (skills/data/tao-analyze-gaps-od-map/SKILL.md)
MEDIUM QUALITY/quality_correctness: SKILL_SPEC recommended field missing: 'metadata.tags' (skills/data/tao-analyze-gaps-od-map/SKILL.md)
MEDIUM SCHEMA/frontmatter_field_placement: Root field 'tags' is ignored; use 'metadata.tags' (skills/data/tao-analyze-gaps-od-map/SKILL.md)
MEDIUM SCHEMA/folder_hierarchy: Unexpected nesting depth for general skill (skills/data/tao-analyze-gaps-od-map)
14 additional finding(s) are available in the full evaluation artifacts.

</details>

Scoring Methodology

<details> <summary>Show dimension definitions, source signals, and thresholds</summary>

Dimension	Question	Scored signals
Security	Is it safe to use?	`security` (100%)
Correctness	Is the answer correct?	`accuracy` (100%)
Discoverability	Was the right skill loaded when needed?	`skill_execution` (100%)
Effectiveness	Did the skill help complete the task?	`goal_accuracy` (50%) + `behavior_check` (50%)
Efficiency	Did it avoid wasted tool calls and token usage?	`skill_efficiency` (50%) + `token_efficiency` (50%)

Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%.
Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL.
Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate.
The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold.
Effectiveness is the equal-weight mean of goal completion (goal_accuracy) and expected workflow adherence (behavior_check).
Efficiency is 50% tool-call productivity (the backward-compatible skill_efficiency wire id) and 50% token_efficiency. Positive-case skill routing is scored under Discoverability, not Efficiency; a negative case without a routing target is N/A. N/A sources are omitted, remaining weights are renormalized, and the dimension is marked partial.

Signals present in this run:

security (Security): unsafe operations, secret leakage, and unauthorized access.
skill_execution (Skill Execution): whether the expected skill was selected, decoys were avoided, and the workflow executed.
skill_efficiency (Tool Productivity): tool-call productivity (legacy wire id; routing is scored under Discoverability).
accuracy (Accuracy): final-answer correctness against the reference answer.
goal_accuracy (Goal Accuracy): whether the user's goal was achieved.
behavior_check (Behavior Check): whether the expected workflow behavior was followed.
token_efficiency (Token Efficiency): actual uncached prompt plus completion usage (50% of Efficiency).

</details>

Freshness

Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Declares allowed tools, Needs a command-line tool, Needs Python

The author says: Requires docker, nvidia-container-toolkit, and the TAO data-services container pinned in versions.yaml.

Security

What our scanner noticed· Medium risk

Large base64-encoded blobs
References to credential files
Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

NVIDIA Corporation

Publisher

NVIDIA (Official)

Source

NVIDIA/skills · skills/tao-analyze-gaps-od-map · commit fce3cca

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE-APACHE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by NVIDIA Corporation or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download tao-analyze-gaps-od-map.zip