Generated banner for Tao Run Deft Object Detection
Generated logo for Tao Run Deft Object Detection

Tao Run Deft Object Detection

NVIDIAOfficialCreated by NVIDIA Corporation

  • Skill
  • Workflow
  • Software development
  • Apache-2.0
  • Medium risk
  • Has scripts

From the author

Run the full DEFT smart-data-augmentation loop for NVIDIA TAO Grounding DINO object detection: zero-shot baseline inference, KPI analysis, per-class gap analysis, SigLIP embedding of weak images, unique-neighbor mining against a source pool, ODVG dataset staging, and retraining — repeated for a fixed number of iterations. Also prepares the source pool the loop mines from, as a separate run: Co-DETR pseudo-labeling, folding to the target classes, KITTI→COCO→ODVG conversion, and embedding. Use for prompts like "run the DEFT OD loop", "run smart data augmentation for grounding dino", "mine and retrain my detection model", "improve OD mAP with gap analysis and mining", "prep the source pool", or "pseudo-label my unlabeled images for mining"; do not use for standalone TAO training, one-off inference, or gap analysis alone.
— the description by NVIDIA Corporation

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill executes a multi-stage, iterative workflow designed to improve object detection model performance through smart data augmentation. It manages a complete loop including zero-shot baseline inference, KPI analysis, per-class gap analysis, SigLIP-based embedding, unique-neighbor mining from a source pool, dataset staging, and retraining. The skill also provides a separate invocation for preparing the source pool, which involves pseudo-labeling raw images with Co-DETR, remapping classes, and converting data formats. It operates as a state-managed loop, ensuring consistency across iterations and providing automated reporting.

What it does

  • Performs zero-shot baseline inference and KPI analysis on Grounding DINO models.
  • Conducts per-class gap analysis to identify model weaknesses.
  • Mines unique, high-value training samples from a prepared source pool using SigLIP embeddings.
  • Stages mined data into the ODVG format for iterative model retraining.
  • Prepares source pools by pseudo-labeling raw images and converting them to the required ODVG format.
  • Maintains a persistent state machine to track progress and ensure auditability across training iterations.

When to use it

  • When you need to improve the mAP of a Grounding DINO model through iterative data mining.
  • When you have a large pool of unlabeled images that need to be pseudo-labeled and mined for training.
  • When you want to automate the gap analysis, mining, and retraining cycle.
  • When you need to prepare a source pool of images for use in a DEFT-based detection workflow.

Advantages

  • Automates complex, multi-step data augmentation and training loops.
  • Reduces manual effort in identifying and labeling high-value training data.
  • Ensures consistent data processing and state management across iterations.
  • Provides clear visibility into model performance trends through automated KPI reporting.

Usage examples

“run the DEFT OD loop”
The agent initiates the iterative mining and retraining process, starting with baseline inference and gap analysis.
“prep the source pool”
The agent runs the Co-DETR pseudo-labeling and embedding pipeline to prepare your raw images for future mining.
“mine and retrain my detection model”
The agent performs gap analysis, mines new samples from the source pool, stages the data, and triggers a retraining iteration.
“improve OD mAP with gap analysis and mining”
The agent executes the full DEFT loop to identify performance gaps and iteratively grow the training dataset to boost mAP.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/nvidia/tao-run-deft-object-detection/download
  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip tao-run-deft-object-detection.zip -d .agents/skills

Documentation by NVIDIA Corporation

Skill: tao-run-deft-object-detection

Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).

Execution Contract

Treat this as a disk-backed state machine, not as a prose recipe.

  1. Preserve every explicit user value. epoch 1 means train.num_epochs=1; a spec value or documented default applies only when the user did not supply that parameter. Show the source of every run parameter (user, spec, or default) in the Pre-Flight Summary.

  2. After the user approves the Summary, initialize deft_state.json once with scripts/init_deft_state.py. Never hand-author or reinitialize it on resume.

  3. Run every bundled or inline host-Python command through scripts/deft_python.sh. On startup, after context compaction, before every stage, and before any completion claim, run:

    <skill_root>/scripts/deft_python.sh \
      <skill_root>/scripts/audit_deft_run.py --results-dir "${RESULTS_DIR}"

    If it prints DEFT_RUN_STATUS=INVALID, stop and repair the listed disk inconsistency; do not launch another stage. Read the path printed as read_before_action before continuing.

  4. Invoke the mapped underlying skill after reading the DEFT overlay. Do not replace a missing or unread stage reference, or a failed skill call, with guessed shell commands, inline Python, a different output tree, or fabricated data.

  5. Commit every stage with scripts/commit_stage.py; it verifies artifacts, updates deft_state.json, appends exactly one ordered loop_log.jsonl event, and rolls back if its audit fails.

  6. Claim the loop complete only when this exits zero:

    <skill_root>/scripts/deft_python.sh \
      <skill_root>/scripts/audit_deft_run.py \
      --results-dir "${RESULTS_DIR}" --require-complete

Context Discipline

  • Load references just in time. Run the audit, read only its read_before_action file and the current stage's named section, then act. Never preload all references.

  • Redirect verbose train, inference, and Docker output to files. Inspect at most the final 40 lines or a one-line artifact check; never print a full spec, state file, or loop log into the conversation.

  • A Skill-tool call loads stage instructions; it does not start a background orchestrator. Continue the documented stage in the parent immediately after it returns.

When to Use This Skill

Use this skill when the user wants an agent to run the full smart-data-augmentation loop for a TAO Grounding DINO detection model: zero-shot baseline, gap analysis, mining, dataset growth, and retraining across N iterations.

  • "Run the DEFT OD loop"

  • "Run smart data augmentation for grounding dino"

  • "Mine more training data for my detection model and retrain"

  • "Improve detection mAP with gap analysis and unique-neighbor mining"

Also use it to prepare the source pool, which is a separate run that completes before the loop launches (see ## Two Invocations: Prep, Then Loop):

  • "Prep the source pool"

  • "Pseudo-label my unlabeled images for mining"

  • "Build the mining pool from these raw images"

Do not use this skill for a single standalone TAO training run, one-off inference, or gap analysis alone. Invoke the relevant leaf skill directly instead.

Scope: Grounding DINO + ODVG

This loop targets Grounding DINO with ODVG training annotations (tmm_odvg.jsonl + labelmap.json), matching the reference pipeline. dataset.train_data_sources is a list; each iteration appends one new ODVG source rather than rewriting a combined CSV. DINO and RT-DETR (COCO) are not supported by this workflow — use the leaf skills directly for those.

The loop does not train at baseline. It evaluates the supplied zero-shot / pretrained checkpoint as iteration 0 and only trains from iteration 1 onward, once mining has produced data to add.

Two Invocations: Prep, Then Loop

The source pool is prepared by its own run, before the loop launches. They are separate because a pool is prepped once and then serves many loop runs — coupling them would re-label and re-embed the same images on every launch.

InvocationDoesProduces
"Prep the source pool"Co-DETR pseudo-labels raw pool images, folds to the target classes, converts KITTI→COCO→ODVG, verifies, embedscoco.json, odvg/, source_embeddings.parquet, pool_report.json
"Run the DEFT loop"baseline → iterationscheckpoints, KPI, the mAP trend

Follow references/prep-source-pool.md for the first. The loop then takes those four paths as inputs; Pre-Flight validates them and init_deft_state.py pins them, so a run cannot reach mine with no corpus to search.

Launch Intake

Do not ask which platform to run on. This workflow supports one: every stage launches with docker run, and its compatibility line requires docker plus the nvidia-container-toolkit. The bank's rule in AGENTS.md is to treat a single supported platform as the selection and only ask when an application supports several — asking here would offer a choice that nothing downstream can act on.

Read tao-run-on-docker's ## Credentials section.

Never ask for or read credential values. Check only whether the required environment variable is set; if unset, tell the user which variable to export.

Agent Behavior

There is exactly one user gate: pre-flight confirmation. Print the Pre-Flight Summary (see references/preflight.md), then STOP and wait for explicit approval ("go", "yes", "looks good"). Do not launch any side-effecting step before that approval.

After the gate, the skill is fully autonomous. Run the entire loop without asking for confirmation. Only stop if a step fails with an unrecoverable error or a hard-stop gate fires. Print a one-line status update at each stage milestone.

Auto-mode required. The post-gate loop fires constant side-effecting calls; without auto-accept mode it stalls on the first prompt. Remind the user at the Pre-Flight Summary to enable auto-mode (shift+tab) before approving.

Non-zero command rule. Never repeat an unchanged failed command. Read the final error block, map it to the loaded stage reference, make one evidence-based correction, and rerun its documented verification. If the reference does not cover the failure, commit status=error and halt.

Revised plan. If any run parameter changes after the Summary was shown, re-run Pre-Flight and show an updated Summary before proceeding.

Workflow

Full detail in references/pipeline-and-state.md.

  1. Pre-Flight. Run every check in references/preflight.md. Resolve workspace, specs, annotations, the zero-shot checkpoint, the source-pool embedding parquet, and container images. Hard stop only on missing input you cannot resolve yourself.

  2. Prep (once, before baseline). If the source pool is not already labeled and embedded, pseudo-label it with Co-DETR, fold the predictions onto the user's target classes, convert KITTI→COCO→ODVG, and embed the pool. Skip any step whose output already exists — no script does this for you, so check before each step. See references/prep-source-pool.md.

  3. Baseline (iter_0) — no training. Run inference with the supplied zero-shot / pretrained checkpoint, then kpi_analyze. Seed train_grounding_dino.yaml from the user's template for later iterations to extend.

  4. Iterate. For each iteration 1..max_iterations, run the seven stages in order: gap_analysis → embed → mine → stage → train → inference → kpi_analyze. Each iteration's gap_analysis consumes the previous phase's inference labels. Between stages run the audit and follow its one-line disk-backed next action.

  5. Stop when max_iterations is reached or a hard-stop gate fires. mAP is reported, not gated — the loop does not early-exit on a metric target.

  6. The report renders itself. results/DEFT_Loop_Report.md is written by init_deft_state.py and re-rendered by every accepted commit_stage.py call. Do not spawn a subagent for it and do not compose it by hand; to refresh it out of band, run scripts/render_report.py --results-dir "${RESULTS_DIR}".

All stages run inline in the parent context. Prefer invoking the underlying tao-skill-bank:* skills via the Skill tool, layering loop conventions on top via the matching references/*.md overlay.

Using Bundled Scripts

Run bundled scripts through <skill_root>/scripts/deft_python.sh. Resolve every path argument to an absolute host path first. Use commit_stage.py for all state and log writes. See references/scripts-and-agents.md.

Stage Reference Modules

Each stage maps to one underlying skill or to bundled glue. Read only the current stage's overlay, then invoke.

The overlays are written against two loop variables that Pre-Flight sets and every stage uses: N, the current iteration number, and PHASE, which is baseline or iter${N}. They are stated here because a stage overlay is read on its own — see references/preflight.md for the full set.

If an overlay is missing, stop and ask the user to reinstall the plugin — the loop cannot run a stage whose settings it does not have. If a mapped skill is unavailable, do not stop and do not improvise: fall back to the overlay's documented docker run as described in references/scripts-and-agents.md, and commit that stage with --execution-path direct-container instead of the documented skill. The overlay carries everything the invocation needs, so the fallback produces the same artifacts.

On a runtime without a Skill tool (Codex, for example), the mapped skills are still on disk under the plugin root: read skills/<layer>/<name>/SKILL.md there for the invocation contract, then run it through the direct-container fallback above, committing the stage with --execution-path direct-container. On such a runtime that fallback is the expected route for every stage, not an exception.

StageOverlayUnderlying skill
prep (once)references/prep-source-pool.mdtao-skill-bank:tao-train-codetr + tao-generate-image-embeddings (+ bundled glue)
gap_analysisreferences/tao-analyze-gaps-od-map.mdtao-skill-bank:tao-analyze-gaps-od-map
embedreferences/tao-generate-image-embeddings.mdtao-skill-bank:tao-generate-image-embeddings
minereferences/tao-mine-od-images.mdtao-skill-bank:tao-mine-od-images
stagereferences/stage-mined-data.md(bundled glue — no leaf skill)
train, inferencereferences/grounding-dino.mdtao-skill-bank:tao-train-grounding-dino
kpi_analyzereferences/tao-analyze-detection-kpi.mdtao-skill-bank:tao-analyze-detection-kpi

Path rule (invariant). Record absolute host paths under ${RESULTS_DIR}. Mount "$WORKSPACE:$WORKSPACE" with identical host and container paths. TAO's update_results_dir appends the task name to results_dir, so passing results_dir=X to train writes X/train/ and to inference writes X/inference/. Never append the subdirectory yourself.

Data, Pre-Flight, Pipeline, and State References

TopicReference
Data contract, ODVG layout, source pool, output treereferences/data-layout.md
One-time source-pool prep (pseudo-label, remap, convert, embed)references/prep-source-pool.md
Pre-Flight checks, defaults, Summary templatereferences/preflight.md
Pipeline stages, state schema, loop-end sequencereferences/pipeline-and-state.md
Bundled scripts, glue, the report renderer, stage tablereferences/scripts-and-agents.md

max_iterations defaults to 1 — one mine, train and score pass, the smallest run that yields a comparison against the baseline. Confirm it with the user when they have not said how many iterations they want; an unattended run takes the default rather than stopping to ask.

Gating

Run the full Pre-Flight, print the Summary, then STOP at the one user gate. After approval, run the baseline and the seven-stage iteration pipeline.

Hard-stop and never auto-retry on: any stage status=error; a missing or zero-row source-pool embedding parquet; a zero-row mining result when weak images were present; a missing ODVG annotation source; an image/annotation mismatch after staging; or a train exit that emits no new iteration checkpoint. The loop stops when max_iterations is reached or an unrecoverable gate fires. Each terminal path commits loop_stop through commit_stage.py, then follows the loop-end sequence in references/pipeline-and-state.md.

Files

references/preflight.md

Pre-Flight Checks and Summary

Resolve everything you can before asking the user. Parameter precedence is strict: explicit user value → workspace spec → documented default. Never replace an explicit value with a recommendation. Record the winning value and its source in the Summary.

Checks, in order

  1. Workspace and run directory. Resolve to an absolute path (WORKSPACE=$(realpath -m <workspace>)); never hand a quoted ~/... path to Python. Derive RESULTS_DIR=${WORKSPACE}/results/run_$(date +%Y%m%d_%H%M%S). Do not create it before the user gate. If resuming, set RESULTS_DIR to the existing run directory (detect via results/run_*/deft_state.json).

  2. Host Python. Probe with the bundled launcher:

    <skill_root>/scripts/deft_python.sh -c "import pandas,numpy,matplotlib,pyarrow,PIL,yaml"

    All six. deft_python.sh selects an interpreter only when every one of pandas numpy matplotlib pyarrow PIL yaml imports; a host missing just matplotlib makes it exit 2 with no interpreter at all, and every bundled script becomes unrunnable. Probing a shorter list will pass here and then fail at the first real call. If the probe fails, the host is not provisioned. Installing packages is out of scope for this skill — deft_python.sh never installs. Report what is missing and stop.

    deft_python.sh finds $WORKSPACE/.venv only when WORKSPACE (or WORKSPACE_DIR) is exported in the calling shell — otherwise pass the interpreter as DEFT_PYTHON.

    deft_python.sh auto-selects $WORKSPACE/.venv/bin/python once it exists. Pre-Flight is incomplete until the probe exits zero.

  3. Credentials. Check presence only; never print a value.

    for var in NGC_KEY HF_TOKEN; do
      [ -n "${!var:-}" ] && printf '%s SET\n' "$var" || printf '%s UNSET\n' "$var"
    done
    VariableRequired for
    NGC_KEYConditionally — only when an nvcr.io image has to be pulled, or the Grounding DINO checkpoint downloaded. The Co-DETR checkpoint is public HuggingFace and needs no credential. Both images already local and an ngc CLI holding its own credentials is a complete run with this unset. See check 5.
    HF_TOKENConditionally — only when the encoder resolves to a HuggingFace id rather than a local snapshot directory. See check 10.

    Defer both verdicts. Check 5 sets WILL_PULL_AFTER_APPROVAL, and only then does an unset NGC_KEY stop the run; check 10 resolves the encoder, and a local snapshot needs no HuggingFace access at all. Stopping a run whose inputs are all present is the failure to avoid here.

    fetch_gdino_checkpoint.py shells out to the ngc CLI, which is not always on PATH — check for it and add its directory before deciding a checkpoint cannot be fetched.

  4. Resolve and export the version-managed container image env vars. The rest of this skill — the Pre-Flight Summary's docker image inspect line, every stage launch, and every references/*.md — reads two env vars. Resolve both from the installed skill bank's versions.yaml; never copy a tag into this document or carry one over from an earlier run.

    TAO_PYT_IMAGE=$(
      <skill_root>/scripts/deft_python.sh \
        <skill_bank>/scripts/resolve_versions_key.py images.tao_toolkit.pyt
    )
    TAO_DS_IMAGE=$(
      <skill_root>/scripts/deft_python.sh \
        <skill_bank>/scripts/resolve_versions_key.py images.tao_toolkit.data_services
    )
    : "${TAO_PYT_IMAGE:?versions key images.tao_toolkit.pyt did not resolve}"
    : "${TAO_DS_IMAGE:?versions key images.tao_toolkit.data_services did not resolve}"
    export TAO_PYT_IMAGE TAO_DS_IMAGE

    resolve_versions_key.py reads the versions.yaml of the bank it is part of, so this works from a git clone and from a plugin install alike. Set TAO_SKILL_BANK_PATH only to point at a different bank than the one the script lives in. The :? guards matter: an unset image variable makes docker image inspect fail in a way that reads as a missing image rather than a missing resolution.

    Env varversions-keyUsed by
    TAO_PYT_IMAGEimages.tao_toolkit.pyttrain, inference
    TAO_DS_IMAGEimages.tao_toolkit.data_servicesgap_analysis, embed, mine, kpi_analyze

    versions.yaml is the single place either URI is written, and step 4 resolves both from it, so a bump lands in one file and no document can drift from it.

  5. Image presence. docker image inspect "$TAO_PYT_IMAGE" "$TAO_DS_IMAGE". Record anything missing as WILL_PULL_AFTER_APPROVAL; do not pull before the gate.

  6. Zero-shot checkpoint — pull it from NGC unless the user supplied one. The baseline scores this checkpoint without training and every iteration fine-tunes from it.

    The user's own path always wins. When they did not give one, fetch the published checkpoint rather than asking:

    Pre-Flight resolves the path only. --plan reports whether the checkpoint is already present or would be downloaded, and writes nothing — the download is a side effect and belongs after the gate, with the container pulls:

    # --plan reports what it would fetch; it prints a sentence, not a path.
    <skill_root>/scripts/deft_python.sh \
      <skill_root>/scripts/fetch_gdino_checkpoint.py --plan \
      --dest "$WORKSPACE/checkpoints/gdino"
    
    # After approval, the same command without --plan downloads and prints the path.
    ZERO_SHOT_CHECKPOINT=$(<skill_root>/scripts/deft_python.sh \
      <skill_root>/scripts/fetch_gdino_checkpoint.py \
      --dest "$WORKSPACE/checkpoints/gdino")

    Capture the variable from the real run, not from --plan: under --plan the script prints WILL DOWNLOAD after approval: ... -> <dir>, so capturing it puts an English sentence in a variable every later stage uses as a path.

    That resolves nvidia/tao/grounding_dino:grounding_dino_swin_tiny_commercial_trainable_v1.1 (1.93 GB) and prints the checkpoint path on stdout. It is idempotent — an existing download is reused, so a resumed run re-costs nothing. Use a trainable release; the sibling deployable one is for TensorRT export and cannot be fine-tuned.

    Requires the ngc CLI and a configured account (check 3). On an air-gapped host, or any time the download cannot run, the user supplies --zero-shot-checkpoint directly — the script says so rather than failing obscurely.

    Record in the Summary which source won (user or NGC <version>), and hard-stop if the resolved path does not exist.

    Then confirm the spec matches the checkpoint's architecture. model.backbone, num_queries, enc_layers, dec_layers, num_feature_levels, and especially class_embed_bias must all agree, or the run dies at load time — after the container has started, with an error that reads like a bad checkpoint rather than a bad spec.

    class_embed_bias is the one that actually bites: it defaults to False, is absent from the shipped infer.yaml template, and a checkpoint trained with it True fails with only class_embed.*.bias keys reported unexpected. Read the values out of the checkpoint and set them explicitly — see references/grounding-dino.md.

    The checkpoint is coupled to the container, not just the spec. Record which TAO PyTorch image the checkpoint was trained with in the Summary; the pinned image is not automatically the right one.

  7. Co-DETR checkpoint — only when prep will run; fetch it unless the user supplied one. Prep pseudo-labels the whole pool with Co-DETR, and assets/overlays/codetr_inference.yaml pins the ViT-L/16 geometry that exactly one published checkpoint loads.

    The user's own path always wins, as for the zero-shot checkpoint. It is how an air-gapped host runs prep at all, and a checkpoint someone already staged is not replaced. Hard-stop if the path does not exist. Its size should be 2,934,763,233 bytes, the published checkpoint's; flag a different size in the Summary rather than stopping, since a compatible fine-tune can differ, but say that a checkpoint the pinned architecture cannot load empties every pseudo-label and verify_pseudo_labels.py catches it only after the labelling pass. A path outside $WORKSPACE needs its directory in $EXTRA_MOUNTS, or the container cannot see it. Record the source (user) in the Summary and skip the fetcher entirely.

    When the user gave no path, fetch the published checkpoint rather than asking: a run should not depend on a file somebody staged by hand. --dest belongs under $WORKSPACE. The Co-DETR container mounts $WORKSPACE and $EXTRA_MOUNTS and nothing else, so a checkpoint anywhere else needs an extra -v on every launch that reads it.

    # Report first; --plan prints a sentence, not a path.
    <skill_root>/scripts/deft_python.sh \
      <skill_root>/scripts/fetch_codetr_checkpoint.py --plan \
      --dest "$WORKSPACE/checkpoints/codetr"
    
    # After approval, the same command without --plan prints the path.
    CODETR_CHECKPOINT=$(<skill_root>/scripts/deft_python.sh \
      <skill_root>/scripts/fetch_codetr_checkpoint.py \
      --dest "$WORKSPACE/checkpoints/codetr")

    Skip this on a run whose pool is already labelled and embedded — prep does not run, so nothing reads the checkpoint. On a run that will prep, a missing checkpoint is a hard stop here rather than 2.5 hours in: the Co-DETR pass is the longest stage in the workflow and it is the first thing prep does.

    The download is plain HTTPS over stdlib, so it needs nothing installed. Size and SHA-256 are verified before the file is put in place, because a checkpoint that does not match the pinned architecture loads no weights, exits 0, prints Execution status: PASS and writes one empty label file per image.

    A checkpoint already at --dest is reused after a size check, which catches the truncated file an interrupted transfer leaves. Add --verify to check its SHA-256 as well when the file is suspect — copied from another host, or left by a run whose pseudo-labels came out empty. It is off by default because it reads the full 2.8 GiB. --plan applies the same checks, so a truncated file fails Pre-Flight instead of appearing in the Summary as ALREADY PRESENT and failing after approval.

  8. Train-spec template. Must exist and parse as YAML, and dataset.train_data_sources must be a list (Grounding DINO ODVG shape). A mapping there means the spec is COCO-shaped and this workflow cannot append to it.

    Seed training data is optional. Unlike the AOI loop — where ChangeNet must learn the task from a mandatory seed set — Grounding DINO is zero-shot capable and can start cold. Inspect the list and branch:

    • Non-empty → validate every entry's image_dir, json_file, and label_map resolve on disk. Report the source count and total ODVG record count in the Summary. Iteration 1 appends to what is already there.

    • Empty or absent → note in the Summary that iteration 1 trains on mined data alone, and that the first iteration's dataset will be small. Not an error.

    Either way the source pool (check 9) stays mandatory — without it there is nothing to mine and the loop cannot add data at all.

  9. Source pool. Two artifacts, both required:

    • source_pool_embeddings parquet — must be non-empty and carry filepath and embedding.

    • source_pool_annotations — an ODVG tree containing *.jsonl records keyed by file_name, and ideally a *labelmap.json. Staging synthesizes a labelmap from observed categories when none is found, but an explicit one is preferred.

    Hard-stop if either is missing or the parquet has zero rows. The loop consumes a prepared pool; it does not build one. Preparing the pool is its own run, completed before the loop launches — see references/prep-source-pool.md. init_deft_state.py refuses to write state without these paths, because the alternative is discovering the corpus is absent at mine, six stages and a training run later.

    Pass --pool-report with the prep run's pool_report.json; init refuses an existing pool without one, under every allocation policy. It records which classes the pool actually holds annotations for, and init cross-checks that against the target classes.

    A pool prepared elsewhere may not carry pool_report.json. Do not re-label to get one — generate it from the pool's COCO with validate_pool_coco.py --record (references/prep-source-pool.md). A pool prepared for a different class set does not make mining fail — it makes mining return neighbours of something else, and the affected class simply never improves.

  10. Resolve the encoder — local snapshot first, never an implicit online default.

The encoder that embeds each iteration's weak images must be the same one that produced the source-pool parquet. A mismatch is silent: mining succeeds and returns confidently wrong neighbours. Record the resolved values as config.embedding_model / config.embedding_model_path and reuse them verbatim on every iteration.

Resolution order — keep an already-set SIGLIP_MODEL_PATH only when it is a directory containing config.json; otherwise search the known cache roots:

resolved_siglip=""
if [ -n "${SIGLIP_MODEL_PATH:-}" ] && [ -f "$SIGLIP_MODEL_PATH/config.json" ]; then
  resolved_siglip="$SIGLIP_MODEL_PATH"
else
  for cache_root in \
    "${HF_HOME:+$HF_HOME/hub}" \
    "${HUGGINGFACE_HUB_CACHE:-}" \
    "$HOME/.cache/huggingface/hub" \
    "$WORKSPACE/source_pool/hf_cache/hub" \
    "$WORKSPACE/source_pool/hf_cache"; do
    [ -n "$cache_root" ] || continue
    for snapshot in "$cache_root/models--google--siglip-base-patch16-224/snapshots/"*; do
      if [ -f "$snapshot/config.json" ]; then
        resolved_siglip="$snapshot"
        break 2
      fi
    done
  done
fi
[ -n "$resolved_siglip" ] && export SIGLIP_MODEL_PATH="$(realpath "$resolved_siglip")"
# init takes it as --embedding-model-path; bind it here or the resolution is lost.
export EMBEDDING_MODEL_PATH="${SIGLIP_MODEL_PATH:-}"

Rules:

  • Select a snapshot only from models--google--siglip-base-patch16-224/snapshots/* and verify its config.json. Never substitute a DINO, C-RADIO, or other model that happens to be cached — it will embed successfully and produce garbage matches.

  • HF_HOME may point outside the workspace, so do not limit the search to the workspace tree.

  • Fall back to the bare HuggingFace id google/siglip-base-patch16-224 only after outbound HuggingFace access has been verified, and only then require HF_TOKEN. Never let the embed stage pick an online default implicitly.

  • If no local snapshot resolves and outbound access cannot be verified, hard-stop. Report the cache roots searched.

Record the resolved snapshot path (or the verified HF id) in the Summary. If the user cannot say which encoder produced the source pool, surface that as an explicit risk row rather than assuming SigLIP.

  1. KPI inputs. Image directory, ground-truth KITTI label directory, and class-mapping YAML must all exist. image_dir must not end in / — kpi_analyze derives its Sequence Name from the second-to-last path component.

  2. Class thresholds and mining config. Per-class AP50 thresholds and the mining multiplier both have reference defaults — do not interrogate the user for them. Omitting --ap50-thresholds-json gates each target class at the reference ITS value (car 0.99, bicycle 0.7, person 0.7) and any other target class at 0.7; --multiplier defaults to 3. Surface the defaulted values in the Pre-Flight Summary so the user can override them, and treat a class gated by assumption as worth flagging: too loose a gate marks no image weak and the iteration mines nothing. If rare classes are configured, also require source_detection_file and target_detection_file as COCO JSONs — class_stratified mining needs them and TAO DS will not infer the format.

  3. GPU count.

    if command -v nvidia-smi >/dev/null 2>&1; then nvidia-smi --list-gpus | wc -l
    else docker run --rm --gpus all "$TAO_PYT_IMAGE" python3 -c 'import torch; print(torch.cuda.device_count())'
    fi
  4. Spec sanity. train.checkpoint_interval must be <= train.num_epochs. prepare_spec_for_train.py lowers it automatically when an explicit epoch override would violate this, but flag the adjustment in the Summary so it is not a surprise.

max_iterations defaults to 1. Confirm it with the user when they have not said how many iterations they want, but do not block on it: an unattended run takes the default.

Defaults

  • train.num_epochs — from the train-spec template

  • train.optim.lr — from the train-spec template

  • multiplier — 3

  • max_iterations — 1

  • allocation_policy — class_stratified when rare classes are given, else global

  • rare_class_list — every target class holding a below-mean share of the pool's annotations. It is a property of the pool, so on a run that still has to prep it is left unset at init and the prep commit derives it from pool_report.json. Supply --rare-class-list only to override that.

  • distance_metric — euclidean

  • candidate_expansion_factor — 5

  • embedding_model — SigLIP; embedding_model_path is resolved, not defaulted (check 10)

  • iou_threshold — 0.5

  • kpi.conf_threshold — 0.0 (frozen as config.kpi_conf_threshold; every phase is scored at it)

  • workspace root — user prompt, else ~/workspace

Container identity

Every launch in this skill passes --user "$(id -u):$(id -g)" $DOCKER_IDENTITY. A uid that has no /etc/passwd entry inside the image — which is every uid but 1000 on the TAO images — then has no name, and getpass.getuser() falls back to pwd.getpwuid() and raises. torch calls it while setting up the inductor cache directory, so the container dies at import with KeyError: 'getpwuid(): uid not found: <uid>' before the requested action starts. $HOME is unset for the same reason and defaults to /, which is not writable, so matplotlib and cupy fail on their cache directories.

Export the identity beside the mounts and pass it to every launch:

DEFT_HOME="$WORKSPACE/.deft_home"
mkdir -p "$DEFT_HOME"
DOCKER_IDENTITY="-e USER=deft -e HOME=$DEFT_HOME"
export DOCKER_IDENTITY

Both are required, and neither is sufficient alone. USER satisfies getpass.getuser(), which reads it before consulting pwd; any non-empty value works, and a literal keeps container logs identical whoever runs the workflow. HOME defaults to /, which is not writable, so without it matplotlib and cupy fail on their cache directories even once the uid has a name.

HOME points into the workspace rather than at /tmp so the caches are owned by the caller and removed with the workspace.

This is invisible on a uid-1000 workstation and fires on every shared or cloud host, so a run that works locally proves nothing about it. Dropping --user is not the alternative: see references/pipeline-and-state.md.

Container mounts

Export them once, right after init, and use the variable in every launch:

EXTRA_MOUNTS=$(<skill_root>/scripts/deft_python.sh -c 'import json,sys
m=json.load(open(sys.argv[1]))["config"].get("extra_container_mounts") or []
print(" ".join(f"-v {p}:{p}" for p in m))' "${RESULTS_DIR}/deft_state.json") || exit 1
export EXTRA_MOUNTS
echo "EXTRA_MOUNTS=$EXTRA_MOUNTS"

Echo it and read the result. A command substitution that fails leaves EXTRA_MOUNTS empty and every later launch still runs, mounting nothing extra — the same silent failure this block exists to prevent.

init_deft_state.py derives this list from the run's own inputs — the checkpoints, the KPI images and labels, the class mapping, the encoder repo, the pool, and both COCO detection files. A path that none of those name is invisible to it: pass --extra-mount PATH (repeatable) at init and it joins the list. Files are mounted by their parent directory.

Every docker run in this skill is written as -v "$WORKSPACE:$WORKSPACE" $EXTRA_MOUNTS -w "$WORKSPACE". With inputs inside the workspace it expands to nothing and the command is unchanged; with KPI data or checkpoints outside it, it is the difference between a working stage and No .txt label files found.

A HuggingFace snapshot directory is a tree of symlinks into a sibling blobs/, so the mount must be the repo root (.../models--org--name/), not the snapshot. Mounting the snapshot alone makes the loader report a missing model file for a model that is present. init_deft_state.py handles this when it derives the list.

init_deft_state.py records config.extra_container_mounts: the directories that inputs live in outside $WORKSPACE. Containers see only "$WORKSPACE:$WORKSPACE", so every docker run in this skill must add a -v "$m:$m" for each entry, in addition to the workspace mount. KPI images, ground truth and checkpoints commonly sit outside the workspace, and a container cannot read what is not mounted.

Pre-Flight Summary

Print this and STOP — wait for explicit approval. This is the only user gate.

## DEFT OD Loop — Pre-Flight Summary

### Run config
| Field                  | Value                                          | Source            |
| ---------------------- | ---------------------------------------------- | ----------------- |
| Model                  | Grounding DINO (ODVG)                          | workflow          |
| Max iterations         | N                                              | user              |
| Stop condition         | max_iterations reached, or zero weak images.
                          mAP is reported, not gated — no target.         | workflow          |
| Epochs / LR            | N / X                                          | user/spec         |
| Encoder                | <model> @ <resolved snapshot or verified HF id>| resolved          |
| Allocation policy      | class_stratified / global                      | user/default      |
| Rare classes           | <list or none>                                 | user              |
| Mining multiplier      | N (budget = iter1 weak count x N)              | user/default      |
| AP50 thresholds        | {"car": 0.99, ...}                             | user/default      |
| GPUs                   | N                                              | detected          |
| Resuming               | yes — iter N complete / no                     | disk              |

### Inputs
| Field                     | Value                                        |
| ------------------------- | -------------------------------------------- |
| Zero-shot checkpoint      | <path>                                       |
| Train spec template       | <path> (N base source(s); 0 = mined-only)    |
| Source pool embeddings    | <path> (N rows, encoder: <model>)            |
| Source pool annotations   | <path> (N jsonl, labelmap: found/synthesized)|
| KPI images                | <path>                                       |
| KPI ground truth          | <path> (N label files)                       |
| Class mapping             | <path>                                       |

### Docker images
| Env var         | Image              | Status     |
| --------------- | ------------------ | ---------- |
| `TAO_PYT_IMAGE` | `<$TAO_PYT_IMAGE>` | OK/MISSING |
| `TAO_DS_IMAGE`  | `<$TAO_DS_IMAGE>`  | OK/MISSING |

### Per-iteration stages
gap_analysis -> embed -> mine -> stage -> train -> inference -> kpi_analyze
(baseline runs inference -> kpi_analyze only; no training)

Remind the user to enable auto-mode (shift+tab) before approving — the post-gate loop is continuously side-effecting.

Run variables

Every stage reference launches containers with shell variables. They are set once here and used unchanged for the rest of the run; nothing later re-derives them. Export them before the first stage:

# Resolved at the gate above
export WORKSPACE RESULTS_DIR MAX_ITERATIONS NUM_GPUS NUM_EPOCHS LEARNING_RATE
export MULTIPLIER ALLOCATION_POLICY AP50_THRESHOLDS_JSON
export ZERO_SHOT_CHECKPOINT TRAIN_SPEC_TEMPLATE
export SOURCE_POOL_EMBEDDINGS SOURCE_POOL_ANNOTATIONS
export EMBEDDING_MODEL EMBEDDING_MODEL_PATH
export KPI_IMAGES_DIR GROUND_TRUTH_LABELS_DIR CLASS_MAPPING
export TAO_PYT_IMAGE TAO_DS_IMAGE EXTRA_MOUNTS DOCKER_IDENTITY

# Prep runs only
export POOL_IMAGES CODETR_CHECKPOINT CODETR_CLASSMAP CLASSES_YAML

# Read by the init block below; unset ones simply drop out of it
export TARGET_CLASSES KPI_CONF_THRESHOLD
export SOURCE_DETECTION_FILE TARGET_DETECTION_FILE   # class_stratified only
export POOL_REPORT RARE_CLASS_LIST                   # an already-prepared pool only
export EXTRA_MOUNTS_IN                               # space-separated extra mounts, if any

Two more are set per iteration rather than once, and every stage reference uses them:

N=<current iteration number>       # 1, 2, ... — the loop counter; paths read iter${N}
PHASE=<baseline|iter${N}>          # which phase's output tree this stage writes to

N is the iteration being run. PHASE is baseline for the zero-shot pass and iter${N} inside the loop; the two differ only because the baseline is not an iteration. Re-export both at the top of each iteration.

An unset variable is not an error in a docker run: it expands to empty, so -e "" reaches TAO as a blank spec path and the failure reads as a spec problem rather than a missing export. Set them all, and prefer set -u in the shell driving the run so a typo fails at the point of use.

Each stage's own spec path is bound in that stage's reference, next to the path it documents — $CODETR_SPEC, $INFER_SPEC, $OD_GAP_SPEC, $EMBED_SPEC, $MINE_SPEC and $KPI_SPEC.

Immediately After Approval

Perform the planned pulls and directory creation, then initialize state once: --pool-report is the exception: omit it on a prep run. Unlike the pool artifacts and --source-detection-file, a --pool-report path that does not exist yet is a hard error, not a warning — validate_pool_coco.py writes it during prep, so pass the flag only when the pool already exists. Omitting it is what downgrades the requirement.

Order: init_deft_state.py runs first and prep is the run's first committed stage. On a pool that still needs preparing, the pool artifacts and --source-detection-file do not exist yet — pass them anyway, as the paths prep will write, together with prep's inputs (--pool-images, --codetr-checkpoint, --codetr-classmap). Init reports them as warnings rather than errors, because prep produces them. Supplying neither the artifacts nor prep's inputs is still an error: nothing would create them.

Three groups of flags apply only to some runs. Build them first so the command below stays one shape:

# class_stratified only. Both are mandatory under that policy — init refuses without
# them, so a run that omits them here fails at `mine`, several stages later.
STRATIFIED_FLAGS=""
if [ "$ALLOCATION_POLICY" = class_stratified ]; then
  STRATIFIED_FLAGS="--source-detection-file $SOURCE_DETECTION_FILE"
  STRATIFIED_FLAGS="$STRATIFIED_FLAGS --target-detection-file $TARGET_DETECTION_FILE"
fi

# Prep runs only: the pool does not exist yet and these are what builds it.
PREP_FLAGS=""
if [ -n "${POOL_IMAGES:-}" ]; then
  PREP_FLAGS="--pool-images $POOL_IMAGES --codetr-checkpoint $CODETR_CHECKPOINT"
  PREP_FLAGS="$PREP_FLAGS --codetr-classmap $CODETR_CLASSMAP"
fi

# A pool that already exists must declare what it holds: prep is what would otherwise
# produce the report, and a prep that is not going to run cannot. The report is
# required under every policy; under class_stratified the rare classes are too, since
# prep would otherwise derive them from it.
EXISTING_POOL_FLAGS=""
if [ -z "${POOL_IMAGES:-}" ]; then
  EXISTING_POOL_FLAGS="--pool-report $POOL_REPORT"
  if [ "$ALLOCATION_POLICY" = class_stratified ]; then
    EXISTING_POOL_FLAGS="$EXISTING_POOL_FLAGS --rare-class-list $RARE_CLASS_LIST"
  fi
fi

# Any path the run's own inputs do not name. Repeatable; omit when there are none.
EXTRA_MOUNT_FLAGS=""
for path in ${EXTRA_MOUNTS_IN:-}; do
  EXTRA_MOUNT_FLAGS="$EXTRA_MOUNT_FLAGS --extra-mount $path"
done
<skill_root>/scripts/deft_python.sh <skill_root>/scripts/init_deft_state.py \
  --workspace "$WORKSPACE" \
  --results-dir "$RESULTS_DIR" \
  --max-iterations "$MAX_ITERATIONS" \
  --num-gpus "$NUM_GPUS" \
  --num-epochs "$NUM_EPOCHS" \
  --learning-rate "$LEARNING_RATE" \
  --zero-shot-checkpoint "$ZERO_SHOT_CHECKPOINT" \
  --train-spec-template "$TRAIN_SPEC_TEMPLATE" \
  --source-pool-embeddings "$SOURCE_POOL_EMBEDDINGS" \
  --source-pool-annotations "$SOURCE_POOL_ANNOTATIONS" \
  --embedding-model "$EMBEDDING_MODEL" \
  --embedding-model-path "$EMBEDDING_MODEL_PATH" \
  --kpi-images-dir "$KPI_IMAGES_DIR" \
  --ground-truth-labels-dir "$GROUND_TRUTH_LABELS_DIR" \
  --class-mapping "$CLASS_MAPPING" \
  --ap50-thresholds-json "$AP50_THRESHOLDS_JSON" \
  --multiplier "$MULTIPLIER" \
  --allocation-policy "$ALLOCATION_POLICY" \
  --target-classes "$TARGET_CLASSES" \
  --kpi-conf-threshold "$KPI_CONF_THRESHOLD" \
  $STRATIFIED_FLAGS $PREP_FLAGS $EXISTING_POOL_FLAGS $EXTRA_MOUNT_FLAGS

<skill_root>/scripts/deft_python.sh <skill_root>/scripts/audit_deft_run.py \
  --results-dir "$RESULTS_DIR"

Then copy the train-spec template to ${RESULTS_DIR}/train_grounding_dino.yaml and begin the baseline.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Declares allowed tools, Needs Python

The author says: Requires docker + nvidia-container-toolkit and one or more CUDA GPUs. Workflows declare additional requirements.

Security

What our scanner noticed· Medium risk

  • Large base64-encoded blobs
  • References to credential files
  • Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
NVIDIA Corporation
Publisher
NVIDIA (Official)
Source
NVIDIA/skills · skills/tao-run-deft-object-detection · commit fce3cca
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE-APACHE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by NVIDIA Corporation or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download tao-run-deft-object-detection.zip