AutoGineer
Generated banner for Tao Run Deft Object Detection
Generated logo for Tao Run Deft Object Detection

Tao Run Deft Object Detection

NVIDIAOfficialCreated by NVIDIA Corporation

  • Skill
  • Workflow
  • Software development
  • Apache-2.0
  • Medium risk
  • Has scripts
Download ZIP

From the author

Run the full DEFT smart-data-augmentation loop for NVIDIA TAO Grounding DINO object detection: zero-shot baseline inference, KPI analysis, per-class gap analysis, SigLIP embedding of weak images, unique-neighbor mining against a source pool, ODVG dataset staging, and retraining — repeated for a fixed number of iterations. Also prepares the source pool the loop mines from, as a separate run: Co-DETR pseudo-labeling, folding to the target classes, KITTI→COCO→ODVG conversion, and embedding. Use for prompts like "run the DEFT OD loop", "run smart data augmentation for grounding dino", "mine and retrain my detection model", "improve OD mAP with gap analysis and mining", "prep the source pool", or "pseudo-label my unlabeled images for mining"; do not use for standalone TAO training, one-off inference, or gap analysis alone.
— the description by NVIDIA Corporation

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill executes a multi-stage, iterative workflow designed to improve object detection model performance through smart data augmentation. It manages a complete loop including zero-shot baseline inference, KPI analysis, per-class gap analysis, SigLIP-based embedding, unique-neighbor mining from a source pool, dataset staging, and retraining. The skill also provides a separate invocation for preparing the source pool, which involves pseudo-labeling raw images with Co-DETR, remapping classes, and converting data formats. It operates as a state-managed loop, ensuring consistency across iterations and providing automated reporting.

What it does

  • Performs zero-shot baseline inference and KPI analysis on Grounding DINO models.
  • Conducts per-class gap analysis to identify model weaknesses.
  • Mines unique, high-value training samples from a prepared source pool using SigLIP embeddings.
  • Stages mined data into the ODVG format for iterative model retraining.
  • Prepares source pools by pseudo-labeling raw images and converting them to the required ODVG format.
  • Maintains a persistent state machine to track progress and ensure auditability across training iterations.

When to use it

  • When you need to improve the mAP of a Grounding DINO model through iterative data mining.
  • When you have a large pool of unlabeled images that need to be pseudo-labeled and mined for training.
  • When you want to automate the gap analysis, mining, and retraining cycle.
  • When you need to prepare a source pool of images for use in a DEFT-based detection workflow.

Advantages

  • Automates complex, multi-step data augmentation and training loops.
  • Reduces manual effort in identifying and labeling high-value training data.
  • Ensures consistent data processing and state management across iterations.
  • Provides clear visibility into model performance trends through automated KPI reporting.

Usage examples

“run the DEFT OD loop”
The agent initiates the iterative mining and retraining process, starting with baseline inference and gap analysis.
“prep the source pool”
The agent runs the Co-DETR pseudo-labeling and embedding pipeline to prepare your raw images for future mining.
“mine and retrain my detection model”
The agent performs gap analysis, mines new samples from the source pool, stages the data, and triggers a retraining iteration.
“improve OD mAP with gap analysis and mining”
The agent executes the full DEFT loop to identify performance gaps and iteratively grow the training dataset to boost mAP.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/nvidia/tao-run-deft-object-detection/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip tao-run-deft-object-detection.zip -d .agents/skills

Documentation by NVIDIA Corporation

Skill: tao-run-deft-object-detection

Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).

Execution Contract

Treat this as a disk-backed state machine, not as a prose recipe.

  1. Preserve every explicit user value. epoch 1 means train.num_epochs=1; a spec value or documented default applies only when the user did not supply that parameter. Show the source of every run parameter (user, spec, or default) in the Pre-Flight Summary.

  2. After the user approves the Summary, initialize deft_state.json once with scripts/init_deft_state.py. Never hand-author or reinitialize it on resume.

  3. Run every bundled or inline host-Python command through scripts/deft_python.sh. On startup, after context compaction, before every stage, and before any completion claim, run:

    <skill_root>/scripts/deft_python.sh \
  <skill_root>/scripts/audit_deft_run.py --results-dir "${RESULTS_DIR}"

    If it prints DEFT_RUN_STATUS=INVALID, stop and repair the listed disk inconsistency; do not launch another stage. Read the path printed as read_before_action before continuing.

  4. Invoke the mapped underlying skill after reading the DEFT overlay. Do not replace a missing or unread stage reference, or a failed skill call, with guessed shell commands, inline Python, a different output tree, or fabricated data.

  5. Commit every stage with scripts/commit_stage.py; it verifies artifacts, updates deft_state.json, appends exactly one ordered loop_log.jsonl event, and rolls back if its audit fails.

  6. Claim the loop complete only when this exits zero:

    <skill_root>/scripts/deft_python.sh \
  <skill_root>/scripts/audit_deft_run.py \
  --results-dir "${RESULTS_DIR}" --require-complete

Context Discipline

  • Load references just in time. Run the audit, read only its read_before_action file and the current stage's named section, then act. Never preload all references.

  • Redirect verbose train, inference, and Docker output to files. Inspect at most the final 40 lines or a one-line artifact check; never print a full spec, state file, or loop log into the conversation.

  • A Skill-tool call loads stage instructions; it does not start a background orchestrator. Continue the documented stage in the parent immediately after it returns.

When to Use This Skill

Use this skill when the user wants an agent to run the full smart-data-augmentation loop for a TAO Grounding DINO detection model: zero-shot baseline, gap analysis, mining, dataset growth, and retraining across N iterations.

  • "Run the DEFT OD loop"

  • "Run smart data augmentation for grounding dino"

  • "Mine more training data for my detection model and retrain"

  • "Improve detection mAP with gap analysis and unique-neighbor mining"

Also use it to prepare the source pool, which is a separate run that completes before the loop launches (see ## Two Invocations: Prep, Then Loop):

  • "Prep the source pool"

  • "Pseudo-label my unlabeled images for mining"

  • "Build the mining pool from these raw images"

Do not use this skill for a single standalone TAO training run, one-off inference, or gap analysis alone. Invoke the relevant leaf skill directly instead.

Scope: Grounding DINO + ODVG

This loop targets Grounding DINO with ODVG training annotations (tmm_odvg.jsonl + labelmap.json), matching the reference pipeline. dataset.train_data_sources is a list; each iteration appends one new ODVG source rather than rewriting a combined CSV. DINO and RT-DETR (COCO) are not supported by this workflow — use the leaf skills directly for those.

The loop does not train at baseline. It evaluates the supplied zero-shot / pretrained checkpoint as iteration 0 and only trains from iteration 1 onward, once mining has produced data to add.

Two Invocations: Prep, Then Loop

The source pool is prepared by its own run, before the loop launches. They are separate because a pool is prepped once and then serves many loop runs — coupling them would re-label and re-embed the same images on every launch.

InvocationDoesProduces
"Prep the source pool"Co-DETR pseudo-labels raw pool images, folds to the target classes, converts KITTI→COCO→ODVG, verifies, embedscoco.json, odvg/, source_embeddings.parquet, pool_report.json
"Run the DEFT loop"baseline → iterationscheckpoints, KPI, the mAP trend

Follow references/prep-source-pool.md for the first. The loop then takes those four paths as inputs; Pre-Flight validates them and init_deft_state.py pins them, so a run cannot reach mine with no corpus to search.

Launch Intake

Do not ask which platform to run on. This workflow supports one: every stage launches with docker run, and its compatibility line requires docker plus the nvidia-container-toolkit. The bank's rule in AGENTS.md is to treat a single supported platform as the selection and only ask when an application supports several — asking here would offer a choice that nothing downstream can act on.

Read tao-run-on-docker's ## Credentials section.

Never ask for or read credential values. Check only whether the required environment variable is set; if unset, tell the user which variable to export.

Agent Behavior

There is exactly one user gate: pre-flight confirmation. Print the Pre-Flight Summary (see references/preflight.md), then STOP and wait for explicit approval ("go", "yes", "looks good"). Do not launch any side-effecting step before that approval.

After the gate, the skill is fully autonomous. Run the entire loop without asking for confirmation. Only stop if a step fails with an unrecoverable error or a hard-stop gate fires. Print a one-line status update at each stage milestone.

Auto-mode required. The post-gate loop fires constant side-effecting calls; without auto-accept mode it stalls on the first prompt. Remind the user at the Pre-Flight Summary to enable auto-mode (shift+tab) before approving.

Non-zero command rule. Never repeat an unchanged failed command. Read the final error block, map it to the loaded stage reference, make one evidence-based correction, and rerun its documented verification. If the reference does not cover the failure, commit status=error and halt.

Revised plan. If any run parameter changes after the Summary was shown, re-run Pre-Flight and show an updated Summary before proceeding.

Workflow

Full detail in references/pipeline-and-state.md.

  1. Pre-Flight. Run every check in references/preflight.md. Resolve workspace, specs, annotations, the zero-shot checkpoint, the source-pool embedding parquet, and container images. Hard stop only on missing input you cannot resolve yourself.

  2. Prep (once, before baseline). If the source pool is not already labeled and embedded, pseudo-label it with Co-DETR, fold the predictions onto the user's target classes, convert KITTI→COCO→ODVG, and embed the pool. Skip any step whose output already exists — no script does this for you, so check before each step. See references/prep-source-pool.md.

  3. Baseline (iter_0) — no training. Run inference with the supplied zero-shot / pretrained checkpoint, then kpi_analyze. Seed train_grounding_dino.yaml from the user's template for later iterations to extend.

  4. Iterate. For each iteration 1..max_iterations, run the seven stages in order: gap_analysis → embed → mine → stage → train → inference → kpi_analyze. Each iteration's gap_analysis consumes the previous phase's inference labels. Between stages run the audit and follow its one-line disk-backed next action.

  5. Stop when max_iterations is reached or a hard-stop gate fires. mAP is reported, not gated — the loop does not early-exit on a metric target.

  6. The report renders itself. results/DEFT_Loop_Report.md is written by init_deft_state.py and re-rendered by every accepted commit_stage.py call. Do not spawn a subagent for it and do not compose it by hand; to refresh it out of band, run scripts/render_report.py --results-dir "${RESULTS_DIR}".

All stages run inline in the parent context. Prefer invoking the underlying tao-skill-bank:* skills via the Skill tool, layering loop conventions on top via the matching references/*.md overlay.

Using Bundled Scripts

Run bundled scripts through <skill_root>/scripts/deft_python.sh. Resolve every path argument to an absolute host path first. Use commit_stage.py for all state and log writes. See references/scripts-and-agents.md.

Stage Reference Modules

Each stage maps to one underlying skill or to bundled glue. Read only the current stage's overlay, then invoke.

The overlays are written against two loop variables that Pre-Flight sets and every stage uses: N, the current iteration number, and PHASE, which is baseline or iter${N}. They are stated here because a stage overlay is read on its own — see references/preflight.md for the full set.

If an overlay is missing, stop and ask the user to reinstall the plugin — the loop cannot run a stage whose settings it does not have. If a mapped skill is unavailable, do not stop and do not improvise: fall back to the overlay's documented docker run as described in references/scripts-and-agents.md, and commit that stage with --execution-path direct-container instead of the documented skill. The overlay carries everything the invocation needs, so the fallback produces the same artifacts.

On a runtime without a Skill tool (Codex, for example), the mapped skills are still on disk under the plugin root: read skills/<layer>/<name>/SKILL.md there for the invocation contract, then run it through the direct-container fallback above, committing the stage with --execution-path direct-container. On such a runtime that fallback is the expected route for every stage, not an exception.

StageOverlayUnderlying skill
prep (once)references/prep-source-pool.mdtao-skill-bank:tao-train-codetr + tao-generate-image-embeddings (+ bundled glue)
gap_analysisreferences/tao-analyze-gaps-od-map.mdtao-skill-bank:tao-analyze-gaps-od-map
embedreferences/tao-generate-image-embeddings.mdtao-skill-bank:tao-generate-image-embeddings
minereferences/tao-mine-od-images.mdtao-skill-bank:tao-mine-od-images
stagereferences/stage-mined-data.md(bundled glue — no leaf skill)
train, inferencereferences/grounding-dino.mdtao-skill-bank:tao-train-grounding-dino
kpi_analyzereferences/tao-analyze-detection-kpi.mdtao-skill-bank:tao-analyze-detection-kpi

Path rule (invariant). Record absolute host paths under ${RESULTS_DIR}. Mount "$WORKSPACE:$WORKSPACE" with identical host and container paths. TAO's update_results_dir appends the task name to results_dir, so passing results_dir=X to train writes X/train/ and to inference writes X/inference/. Never append the subdirectory yourself.

Data, Pre-Flight, Pipeline, and State References

TopicReference
Data contract, ODVG layout, source pool, output treereferences/data-layout.md
One-time source-pool prep (pseudo-label, remap, convert, embed)references/prep-source-pool.md
Pre-Flight checks, defaults, Summary templatereferences/preflight.md
Pipeline stages, state schema, loop-end sequencereferences/pipeline-and-state.md
Bundled scripts, glue, the report renderer, stage tablereferences/scripts-and-agents.md

max_iterations defaults to 1 — one mine, train and score pass, the smallest run that yields a comparison against the baseline. Confirm it with the user when they have not said how many iterations they want; an unattended run takes the default rather than stopping to ask.

Gating

Run the full Pre-Flight, print the Summary, then STOP at the one user gate. After approval, run the baseline and the seven-stage iteration pipeline.

Hard-stop and never auto-retry on: any stage status=error; a missing or zero-row source-pool embedding parquet; a zero-row mining result when weak images were present; a missing ODVG annotation source; an image/annotation mismatch after staging; or a train exit that emits no new iteration checkpoint. The loop stops when max_iterations is reached or an unrecoverable gate fires. Each terminal path commits loop_stop through commit_stage.py, then follows the loop-end sequence in references/pipeline-and-state.md.

Files

scripts/prepare_class_mappings_for_mining_data_prep.py

#!/usr/bin/env python3
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""Translate one ``classes.yaml`` into the two mappings TAO needs. Nothing else.

The fold from a detector's vocabulary down to the target classes is TAO's job,
in two places, and this script only feeds them.

**Co-DETR does the fold** via ``inference.category_mapping`` — an ordered mapping
from output category to the original classmap names merged into it. From
``category_mapping.py``: unmapped originals are dropped, a name claimed by two
groups keeps the first with a warning, names absent from the classmap are warned
about and ignored, and an empty remap raises. Crucially it then runs
``apply_category_mapping_groupnms`` — per-output-category soft-NMS *after* the
merge. That matters: one object detected as both ``truck`` and ``car`` becomes two
boxes of the same class the moment they fold together, and only a post-fold NMS
removes the duplicate. Renaming labels afterwards cannot.

**annotations convert does the KITTI->COCO fold** via ``kitti.mapping``::

    labels2cat = {label: k for k, v in cat_map.items() for label in v}
    mapped = labels2cat.get(row_p[0], None)
    if not mapped:
        continue

Because Co-DETR already emitted target names, the mapping handed to the converter
is an identity map over the targets. It is still written explicitly rather than
omitted: with no ``mapping`` the converter auto-derives one by scanning the label
directory, so the COCO category IDs would follow filesystem discovery order and a
target absent from this particular pool would vanish from ``categories``. Those
IDs travel into the ODVG labelmap and then into training, so they need to be
stable across runs.

What is left here is a format translation, and it is worth a script only because
doing it by hand is how ``- car: car`` gets written — a bare string where a list
belongs. Both consumers iterate the value, so a string iterates character by
character, yielding class names ``c``, ``a``, ``r``. Nothing matches, every box is
dropped, and ``annotations convert`` still prints ``Execution status: PASS`` and
exits 0.

Whether the fold actually worked is ``validate_pool_coco.py``'s question, asked of
the COCO that comes out.

Stdlib only apart from PyYAML.
"""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

import yaml


def load_classes(path: Path) -> dict[str, list[str]]:
    """Return {target: [source aliases]} in the order the user declared them.

    Order is preserved deliberately: Co-DETR assigns output category IDs
    ``0..K-1`` in exactly this iteration order.
    """
    with path.open("r", encoding="utf-8") as fh:
        data = yaml.safe_load(fh) or {}

    classes = data.get("classes", data) if isinstance(data, dict) else data
    if classes is None:
        raise ValueError(f"{path}: no 'classes' key and file is not a bare list.")

    fold: dict[str, list[str]] = {}
    if isinstance(classes, list):
        # Identity form: keep these names as-is, drop everything else.
        for name in classes:
            fold[str(name)] = [str(name)]
    elif isinstance(classes, dict):
        for target, sources in classes.items():
            target = str(target)
            if sources is None:
                sources = [target]
            if isinstance(sources, str):
                sources = [sources]
            fold[target] = [str(s) for s in sources]
    else:
        raise ValueError(f"{path}: 'classes' must be a list or a mapping.")

    if not fold:
        raise ValueError(f"{path}: no target classes defined.")

    # Co-DETR resolves this by keeping the first group and logging a warning, which
    # is easy to miss in a long inference log. Refuse it here instead: which target
    # a shared source belongs to changes the pseudo-labels, so it is the user's call.
    owner: dict[str, str] = {}
    for target, sources in fold.items():
        for source in sources:
            if source in owner and owner[source] != target:
                raise ValueError(
                    f"{path}: source class '{source}' is mapped to both "
                    f"'{owner[source]}' and '{target}'. Co-DETR would silently keep "
                    f"'{owner[source]}' — pick one explicitly."
                )
            owner[source] = target

    # Target names become COCO categories, then ODVG labels, then the Grounding DINO
    # caption list, and finally the class field of KITTI inference labels. KITTI is
    # space-delimited, so a multi-word target makes those labels unparseable.
    spaced = [t for t in fold if len(t.split()) != 1]
    if spaced:
        raise ValueError(
            f"{path}: target class name(s) contain whitespace: {spaced}. "
            "Targets end up as KITTI class names at inference time and KITTI is "
            "space-delimited. Use an underscore, e.g. 'road_sign'. Source names may "
            "contain spaces — COCO has 'traffic light'."
        )
    return fold


def emit_yaml_mapping(fold: dict[str, list[str]], style: str) -> str:
    """Render a mapping in one of the two shapes TAO expects.

    ``convert`` wants a list of single-key dicts; ``codetr`` wants a plain nested
    mapping. Every scalar goes through ``json.dumps``: YAML is a JSON superset, so
    a quoted scalar is always valid and there is no hand-maintained set of
    characters to escape. Unquoted, a class named ``car,truck`` would emit
    ``[car,truck]`` and parse as two aliases — converting against the wrong
    classes without complaint.
    """
    lines = []
    for target, sources in fold.items():
        prefix = "- " if style == "convert" else "  "
        lines.append(f"{prefix}{json.dumps(target)}:")
        indent = "  " if style == "convert" else "  - "
        for source in sources:
            lines.append(f"{indent}- {json.dumps(source)}" if style == "convert"
                         else f"    - {json.dumps(source)}")
    return "\n".join(lines) + "\n"


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__,
                                     formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument("--classes", required=True, help="Class-mapping YAML.")
    parser.add_argument("--emit-codetr-category-mapping", default=None,
                        help="Write the `category_mapping` block for the Co-DETR inference "
                             "spec — the real fold, applied at detection time.")
    parser.add_argument("--emit-kitti-mapping", default=None,
                        help="Write the `kitti.mapping` YAML for `annotations convert`. "
                             "Identity over the targets, because Co-DETR already folded.")
    parser.add_argument("--emit-classmap", default=None,
                        help="Write the target class list as a newline-separated classmap.txt.")
    return parser.parse_args()


def main() -> int:
    try:
        args = parse_args()
        if not any((args.emit_codetr_category_mapping, args.emit_kitti_mapping,
                    args.emit_classmap)):
            print("ERROR: nothing to do — pass at least one --emit-* flag.", file=sys.stderr)
            return 2

        fold = load_classes(Path(args.classes).expanduser().resolve())
        targets = list(fold)

        def write(path_str: str, text: str) -> Path:
            path = Path(path_str).expanduser().resolve()
            path.parent.mkdir(parents=True, exist_ok=True)
            path.write_text(text, encoding="utf-8")
            return path

        if args.emit_codetr_category_mapping:
            body = emit_yaml_mapping(fold, "codetr")
            path = write(args.emit_codetr_category_mapping, f"category_mapping:\n{body}")
            print(f"codetr category_mapping -> {path}")

        if args.emit_kitti_mapping:
            identity = {t: [t] for t in targets}
            path = write(args.emit_kitti_mapping, emit_yaml_mapping(identity, "convert"))
            print(f"convert kitti.mapping   -> {path}  (identity — Co-DETR already folded)")

        if args.emit_classmap:
            path = write(args.emit_classmap, "\n".join(targets) + "\n")
            print(f"target classmap         -> {path}")

        print()
        for target, sources in fold.items():
            print(f"  {target} <- {', '.join(sources)}")
        print(
            "\nBoth consumers match source names EXACTLY, including case: Co-DETR checks "
            "`orig_name not in name_to_id`\nand the converter does `labels2cat.get(row_p[0])`. "
            "Copy names verbatim from the detector's classmap.\nA name that does not match is "
            "a warning in the log, not an error — run validate_pool_coco.py to catch it."
        )
        return 0
    except Exception as exc:  # noqa: BLE001
        print(f"ERROR: {exc}", file=sys.stderr)
        return 1


if __name__ == "__main__":
    raise SystemExit(main())

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Declares allowed tools, Needs Python

The author says: Requires docker + nvidia-container-toolkit and one or more CUDA GPUs. Workflows declare additional requirements.

Security

What our scanner noticed· Medium risk

  • Large base64-encoded blobs
  • References to credential files
  • Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
NVIDIA Corporation
Publisher
NVIDIA (Official)
Source
NVIDIA/skills · skills/tao-run-deft-object-detection · commit fce3cca
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE-APACHE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by NVIDIA Corporation or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download tao-run-deft-object-detection.zip