Tao Run Deft Object Detection
NVIDIAOfficialCreated by NVIDIA Corporation
- Skill
- Workflow
- Software development
- Apache-2.0
- Medium risk
- Has scripts
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
This skill executes a multi-stage, iterative workflow designed to improve object detection model performance through smart data augmentation. It manages a complete loop including zero-shot baseline inference, KPI analysis, per-class gap analysis, SigLIP-based embedding, unique-neighbor mining from a source pool, dataset staging, and retraining. The skill also provides a separate invocation for preparing the source pool, which involves pseudo-labeling raw images with Co-DETR, remapping classes, and converting data formats. It operates as a state-managed loop, ensuring consistency across iterations and providing automated reporting.
What it does
- Performs zero-shot baseline inference and KPI analysis on Grounding DINO models.
- Conducts per-class gap analysis to identify model weaknesses.
- Mines unique, high-value training samples from a prepared source pool using SigLIP embeddings.
- Stages mined data into the ODVG format for iterative model retraining.
- Prepares source pools by pseudo-labeling raw images and converting them to the required ODVG format.
- Maintains a persistent state machine to track progress and ensure auditability across training iterations.
When to use it
- When you need to improve the mAP of a Grounding DINO model through iterative data mining.
- When you have a large pool of unlabeled images that need to be pseudo-labeled and mined for training.
- When you want to automate the gap analysis, mining, and retraining cycle.
- When you need to prepare a source pool of images for use in a DEFT-based detection workflow.
Advantages
- Automates complex, multi-step data augmentation and training loops.
- Reduces manual effort in identifying and labeling high-value training data.
- Ensures consistent data processing and state management across iterations.
- Provides clear visibility into model performance trends through automated KPI reporting.
Usage examples
- “run the DEFT OD loop”
- The agent initiates the iterative mining and retraining process, starting with baseline inference and gap analysis.
- “prep the source pool”
- The agent runs the Co-DETR pseudo-labeling and embedding pipeline to prepare your raw images for future mining.
- “mine and retrain my detection model”
- The agent performs gap analysis, mines new samples from the source pool, stages the data, and triggers a retraining iteration.
- “improve OD mAP with gap analysis and mining”
- The agent executes the full DEFT loop to identify performance gaps and iteratively grow the training dataset to boost mAP.
Install
Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/nvidia/tao-run-deft-object-detection/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip tao-run-deft-object-detection.zip -d .agents/skills
Documentation by NVIDIA Corporation
Skill: tao-run-deft-object-detection
Standalone install? If this session was not initialized by the TAO skill bank plugin, run the
tao-setupskill first (host preflight, credentials, cross-skill discovery).
Execution Contract
Treat this as a disk-backed state machine, not as a prose recipe.
Preserve every explicit user value.
epoch 1meanstrain.num_epochs=1; a spec value or documented default applies only when the user did not supply that parameter. Show the source of every run parameter (user,spec, ordefault) in the Pre-Flight Summary.After the user approves the Summary, initialize
deft_state.jsononce withscripts/init_deft_state.py. Never hand-author or reinitialize it on resume.Run every bundled or inline host-Python command through
scripts/deft_python.sh. On startup, after context compaction, before every stage, and before any completion claim, run:<skill_root>/scripts/deft_python.sh \ <skill_root>/scripts/audit_deft_run.py --results-dir "${RESULTS_DIR}"If it prints
DEFT_RUN_STATUS=INVALID, stop and repair the listed disk inconsistency; do not launch another stage. Read the path printed asread_before_actionbefore continuing.Invoke the mapped underlying skill after reading the DEFT overlay. Do not replace a missing or unread stage reference, or a failed skill call, with guessed shell commands, inline Python, a different output tree, or fabricated data.
Commit every stage with
scripts/commit_stage.py; it verifies artifacts, updatesdeft_state.json, appends exactly one orderedloop_log.jsonlevent, and rolls back if its audit fails.Claim the loop complete only when this exits zero:
<skill_root>/scripts/deft_python.sh \ <skill_root>/scripts/audit_deft_run.py \ --results-dir "${RESULTS_DIR}" --require-complete
Context Discipline
Load references just in time. Run the audit, read only its
read_before_actionfile and the current stage's named section, then act. Never preload all references.Redirect verbose train, inference, and Docker output to files. Inspect at most the final 40 lines or a one-line artifact check; never print a full spec, state file, or loop log into the conversation.
A Skill-tool call loads stage instructions; it does not start a background orchestrator. Continue the documented stage in the parent immediately after it returns.
When to Use This Skill
Use this skill when the user wants an agent to run the full smart-data-augmentation loop for a TAO Grounding DINO detection model: zero-shot baseline, gap analysis, mining, dataset growth, and retraining across N iterations.
"Run the DEFT OD loop"
"Run smart data augmentation for grounding dino"
"Mine more training data for my detection model and retrain"
"Improve detection mAP with gap analysis and unique-neighbor mining"
Also use it to prepare the source pool, which is a separate run that completes before the
loop launches (see ## Two Invocations: Prep, Then Loop):
"Prep the source pool"
"Pseudo-label my unlabeled images for mining"
"Build the mining pool from these raw images"
Do not use this skill for a single standalone TAO training run, one-off inference, or gap analysis alone. Invoke the relevant leaf skill directly instead.
Scope: Grounding DINO + ODVG
This loop targets Grounding DINO with ODVG training annotations (tmm_odvg.jsonl + labelmap.json), matching the reference pipeline. dataset.train_data_sources is a list; each iteration appends one new ODVG source rather than rewriting a combined CSV. DINO and RT-DETR (COCO) are not supported by this workflow — use the leaf skills directly for those.
The loop does not train at baseline. It evaluates the supplied zero-shot / pretrained checkpoint as iteration 0 and only trains from iteration 1 onward, once mining has produced data to add.
Two Invocations: Prep, Then Loop
The source pool is prepared by its own run, before the loop launches. They are separate because a pool is prepped once and then serves many loop runs — coupling them would re-label and re-embed the same images on every launch.
| Invocation | Does | Produces |
|---|---|---|
| "Prep the source pool" | Co-DETR pseudo-labels raw pool images, folds to the target classes, converts KITTI→COCO→ODVG, verifies, embeds | coco.json, odvg/, source_embeddings.parquet, pool_report.json |
| "Run the DEFT loop" | baseline → iterations | checkpoints, KPI, the mAP trend |
Follow references/prep-source-pool.md for the first. The loop then takes those four paths as
inputs; Pre-Flight validates them and init_deft_state.py pins them, so a run cannot reach
mine with no corpus to search.
Launch Intake
Do not ask which platform to run on. This workflow supports one: every stage launches
with docker run, and its compatibility line requires docker plus the
nvidia-container-toolkit. The bank's rule in AGENTS.md is to treat a single supported
platform as the selection and only ask when an application supports several — asking here
would offer a choice that nothing downstream can act on.
Read tao-run-on-docker's ## Credentials section.
Never ask for or read credential values. Check only whether the required environment variable is set; if unset, tell the user which variable to export.
Agent Behavior
There is exactly one user gate: pre-flight confirmation. Print the Pre-Flight Summary (see
references/preflight.md), then STOP and wait for explicit approval ("go", "yes", "looks good"). Do not launch any side-effecting step before that approval.After the gate, the skill is fully autonomous. Run the entire loop without asking for confirmation. Only stop if a step fails with an unrecoverable error or a hard-stop gate fires. Print a one-line status update at each stage milestone.
Auto-mode required. The post-gate loop fires constant side-effecting calls; without auto-accept mode it stalls on the first prompt. Remind the user at the Pre-Flight Summary to enable auto-mode (shift+tab) before approving.
Non-zero command rule. Never repeat an unchanged failed command. Read the final error block, map it to the loaded stage reference, make one evidence-based correction, and rerun its documented verification. If the reference does not cover the failure, commit
status=errorand halt.Revised plan. If any run parameter changes after the Summary was shown, re-run Pre-Flight and show an updated Summary before proceeding.
Workflow
Full detail in references/pipeline-and-state.md.
Pre-Flight. Run every check in
references/preflight.md. Resolve workspace, specs, annotations, the zero-shot checkpoint, the source-pool embedding parquet, and container images. Hard stop only on missing input you cannot resolve yourself.Prep (once, before baseline). If the source pool is not already labeled and embedded, pseudo-label it with Co-DETR, fold the predictions onto the user's target classes, convert KITTI→COCO→ODVG, and embed the pool. Skip any step whose output already exists — no script does this for you, so check before each step. See
references/prep-source-pool.md.Baseline (iter_0) — no training. Run
inferencewith the supplied zero-shot / pretrained checkpoint, thenkpi_analyze. Seedtrain_grounding_dino.yamlfrom the user's template for later iterations to extend.Iterate. For each iteration 1..
max_iterations, run the seven stages in order:gap_analysis→embed→mine→stage→train→inference→kpi_analyze. Each iteration'sgap_analysisconsumes the previous phase's inference labels. Between stages run the audit and follow its one-line disk-backed next action.Stop when
max_iterationsis reached or a hard-stop gate fires. mAP is reported, not gated — the loop does not early-exit on a metric target.The report renders itself.
results/DEFT_Loop_Report.mdis written byinit_deft_state.pyand re-rendered by every acceptedcommit_stage.pycall. Do not spawn a subagent for it and do not compose it by hand; to refresh it out of band, runscripts/render_report.py --results-dir "${RESULTS_DIR}".
All stages run inline in the parent context. Prefer invoking the underlying tao-skill-bank:* skills via the Skill tool, layering loop conventions on top via the matching references/*.md overlay.
Using Bundled Scripts
Run bundled scripts through <skill_root>/scripts/deft_python.sh. Resolve every path argument to an absolute host path first. Use commit_stage.py for all state and log writes. See references/scripts-and-agents.md.
Stage Reference Modules
Each stage maps to one underlying skill or to bundled glue. Read only the current stage's overlay, then invoke.
The overlays are written against two loop variables that Pre-Flight sets and every stage uses: N, the current iteration number, and PHASE, which is baseline or iter${N}. They are stated here because a stage overlay is read on its own — see references/preflight.md for the full set.
If an overlay is missing, stop and ask the user to reinstall the plugin — the loop cannot run a stage whose settings it does not have. If a mapped skill is unavailable, do not stop and do not improvise: fall back to the overlay's documented docker run as described in references/scripts-and-agents.md, and commit that stage with --execution-path direct-container instead of the documented skill. The overlay carries everything the invocation needs, so the fallback produces the same artifacts.
On a runtime without a Skill tool (Codex, for example), the mapped skills are still on disk under the plugin root: read skills/<layer>/<name>/SKILL.md there for the invocation contract, then run it through the direct-container fallback above, committing the stage with --execution-path direct-container. On such a runtime that fallback is the expected route for every stage, not an exception.
| Stage | Overlay | Underlying skill |
|---|---|---|
prep (once) | references/prep-source-pool.md | tao-skill-bank:tao-train-codetr + tao-generate-image-embeddings (+ bundled glue) |
gap_analysis | references/tao-analyze-gaps-od-map.md | tao-skill-bank:tao-analyze-gaps-od-map |
embed | references/tao-generate-image-embeddings.md | tao-skill-bank:tao-generate-image-embeddings |
mine | references/tao-mine-od-images.md | tao-skill-bank:tao-mine-od-images |
stage | references/stage-mined-data.md | (bundled glue — no leaf skill) |
train, inference | references/grounding-dino.md | tao-skill-bank:tao-train-grounding-dino |
kpi_analyze | references/tao-analyze-detection-kpi.md | tao-skill-bank:tao-analyze-detection-kpi |
Path rule (invariant). Record absolute host paths under ${RESULTS_DIR}. Mount "$WORKSPACE:$WORKSPACE" with identical host and container paths. TAO's update_results_dir appends the task name to results_dir, so passing results_dir=X to train writes X/train/ and to inference writes X/inference/. Never append the subdirectory yourself.
Data, Pre-Flight, Pipeline, and State References
| Topic | Reference |
|---|---|
| Data contract, ODVG layout, source pool, output tree | references/data-layout.md |
| One-time source-pool prep (pseudo-label, remap, convert, embed) | references/prep-source-pool.md |
| Pre-Flight checks, defaults, Summary template | references/preflight.md |
| Pipeline stages, state schema, loop-end sequence | references/pipeline-and-state.md |
| Bundled scripts, glue, the report renderer, stage table | references/scripts-and-agents.md |
max_iterations defaults to 1 — one mine, train and score pass, the smallest run that yields a comparison against the baseline. Confirm it with the user when they have not said how many iterations they want; an unattended run takes the default rather than stopping to ask.
Gating
Run the full Pre-Flight, print the Summary, then STOP at the one user gate. After approval, run the baseline and the seven-stage iteration pipeline.
Hard-stop and never auto-retry on: any stage status=error; a missing or zero-row source-pool embedding parquet; a zero-row mining result when weak images were present; a missing ODVG annotation source; an image/annotation mismatch after staging; or a train exit that emits no new iteration checkpoint. The loop stops when max_iterations is reached or an unrecoverable gate fires. Each terminal path commits loop_stop through commit_stage.py, then follows the loop-end sequence in references/pipeline-and-state.md.
Files
- ATTRIBUTION.mdAdded by AutoGineer957 B
- BENCHMARK.md8.0 KiB
- LICENSELicence file added by AutoGineer11.2 KiB
- SKILL.md13.7 KiB
- agents/reporter.md1.4 KiB
- assets/gap_analysis_object_detection.yaml1.9 KiB
- assets/image_embeddings.yaml790 B
- assets/overlays/coco_to_odvg.yaml334 B
- assets/overlays/codetr_inference.yaml5.0 KiB
- assets/overlays/grounding_dino_inference.yaml1.3 KiB
- assets/overlays/kitti_to_coco.yaml500 B
- assets/overlays/kpi_analyze.yaml2.0 KiB
- assets/tmm_unique_neighbor_matching.yaml1.7 KiB
- assets/train_grounding_dino.yaml2.6 KiB
- config/skillspector-baseline.yaml4.6 KiB
- evals/evals.json5.6 KiB
- references/data-layout.md7.7 KiB
- references/example_classes_its.yaml2.0 KiB
- references/grounding-dino.md16.7 KiB
- references/pipeline-and-state.md15.4 KiB
- references/preflight.md29.4 KiB
- references/prep-source-pool.md27.3 KiB
- references/scripts-and-agents.md15.2 KiB
- references/stage-mined-data.md6.3 KiB
- references/tao-analyze-detection-kpi.md11.4 KiB
- references/tao-analyze-gaps-od-map.md8.6 KiB
- references/tao-generate-image-embeddings.md4.0 KiB
- references/tao-mine-od-images.md8.2 KiB
- scripts/apply_spec_overrides.pyScript16.0 KiB
- scripts/audit_deft_run.pyScript39.2 KiB
- scripts/await_stage.pyScript7.5 KiB
- scripts/commit_stage.pyScript52.8 KiB
- scripts/deft_python.shScript2.5 KiB
- scripts/deft_stages.pyScript8.4 KiB
- scripts/emit_default_spec.pyScript9.1 KiB
- scripts/fetch_codetr_checkpoint.pyScript10.4 KiB
- scripts/fetch_gdino_checkpoint.pyScript7.4 KiB
- scripts/init_deft_state.pyScript44.8 KiB
- scripts/prepare_budget_for_mining.pyScript7.1 KiB
- scripts/prepare_class_mappings_for_mining_data_prep.pyScript8.9 KiB
- scripts/prepare_exclude_for_mining.pyScript4.7 KiB
- scripts/prepare_input_for_image_embeddings.pyScript4.3 KiB
- scripts/prepare_mapping_for_kpi_analyze.pyScript5.1 KiB
- scripts/prepare_spec_for_train.pyScript11.8 KiB
- scripts/prepare_thresholds_for_gap_analysis.pyScript5.9 KiB
- scripts/prepare_val_split_for_train.pyScript5.5 KiB
- scripts/render_report.pyScript33.7 KiB
- scripts/stage_mined_odvg.pyScript10.3 KiB
- scripts/summarize_kpi.pyScript9.1 KiB
- scripts/tests/test_state_machine.shScript178.2 KiB
- scripts/validate_odvg_images.pyScript4.6 KiB
- scripts/validate_pool_coco.pyScript13.0 KiB
- scripts/verify_class_contract.pyScript8.0 KiB
- scripts/verify_embed_encoder.pyScript3.8 KiB
- scripts/verify_pseudo_labels.pyScript7.5 KiB
- skill-card.md5.5 KiB
- skill.oms.sig16.6 KiB
references/preflight.md
Pre-Flight Checks and Summary
Resolve everything you can before asking the user. Parameter precedence is strict: explicit user value → workspace spec → documented default. Never replace an explicit value with a recommendation. Record the winning value and its source in the Summary.
Checks, in order
Workspace and run directory. Resolve to an absolute path (
WORKSPACE=$(realpath -m <workspace>)); never hand a quoted~/...path to Python. DeriveRESULTS_DIR=${WORKSPACE}/results/run_$(date +%Y%m%d_%H%M%S). Do not create it before the user gate. If resuming, setRESULTS_DIRto the existing run directory (detect viaresults/run_*/deft_state.json).Host Python. Probe with the bundled launcher:
<skill_root>/scripts/deft_python.sh -c "import pandas,numpy,matplotlib,pyarrow,PIL,yaml"All six.
deft_python.shselects an interpreter only when every one ofpandas numpy matplotlib pyarrow PIL yamlimports; a host missing justmatplotlibmakes it exit 2 with no interpreter at all, and every bundled script becomes unrunnable. Probing a shorter list will pass here and then fail at the first real call. If the probe fails, the host is not provisioned. Installing packages is out of scope for this skill —deft_python.shnever installs. Report what is missing and stop.deft_python.shfinds$WORKSPACE/.venvonly whenWORKSPACE(orWORKSPACE_DIR) is exported in the calling shell — otherwise pass the interpreter asDEFT_PYTHON.deft_python.shauto-selects$WORKSPACE/.venv/bin/pythononce it exists. Pre-Flight is incomplete until the probe exits zero.Credentials. Check presence only; never print a value.
for var in NGC_KEY HF_TOKEN; do [ -n "${!var:-}" ] && printf '%s SET\n' "$var" || printf '%s UNSET\n' "$var" doneVariable Required for NGC_KEYConditionally — only when an nvcr.ioimage has to be pulled, or the Grounding DINO checkpoint downloaded. The Co-DETR checkpoint is public HuggingFace and needs no credential. Both images already local and anngcCLI holding its own credentials is a complete run with this unset. See check 5.HF_TOKENConditionally — only when the encoder resolves to a HuggingFace id rather than a local snapshot directory. See check 10. Defer both verdicts. Check 5 sets
WILL_PULL_AFTER_APPROVAL, and only then does an unsetNGC_KEYstop the run; check 10 resolves the encoder, and a local snapshot needs no HuggingFace access at all. Stopping a run whose inputs are all present is the failure to avoid here.fetch_gdino_checkpoint.pyshells out to thengcCLI, which is not always onPATH— check for it and add its directory before deciding a checkpoint cannot be fetched.Resolve and export the version-managed container image env vars. The rest of this skill — the Pre-Flight Summary's
docker image inspectline, every stage launch, and everyreferences/*.md— reads two env vars. Resolve both from the installed skill bank'sversions.yaml; never copy a tag into this document or carry one over from an earlier run.TAO_PYT_IMAGE=$( <skill_root>/scripts/deft_python.sh \ <skill_bank>/scripts/resolve_versions_key.py images.tao_toolkit.pyt ) TAO_DS_IMAGE=$( <skill_root>/scripts/deft_python.sh \ <skill_bank>/scripts/resolve_versions_key.py images.tao_toolkit.data_services ) : "${TAO_PYT_IMAGE:?versions key images.tao_toolkit.pyt did not resolve}" : "${TAO_DS_IMAGE:?versions key images.tao_toolkit.data_services did not resolve}" export TAO_PYT_IMAGE TAO_DS_IMAGEresolve_versions_key.pyreads theversions.yamlof the bank it is part of, so this works from a git clone and from a plugin install alike. SetTAO_SKILL_BANK_PATHonly to point at a different bank than the one the script lives in. The:?guards matter: an unset image variable makesdocker image inspectfail in a way that reads as a missing image rather than a missing resolution.Env var versions-key Used by TAO_PYT_IMAGEimages.tao_toolkit.pyttrain,inferenceTAO_DS_IMAGEimages.tao_toolkit.data_servicesgap_analysis,embed,mine,kpi_analyzeversions.yamlis the single place either URI is written, and step 4 resolves both from it, so a bump lands in one file and no document can drift from it.Image presence.
docker image inspect "$TAO_PYT_IMAGE" "$TAO_DS_IMAGE". Record anything missing asWILL_PULL_AFTER_APPROVAL; do not pull before the gate.Zero-shot checkpoint — pull it from NGC unless the user supplied one. The baseline scores this checkpoint without training and every iteration fine-tunes from it.
The user's own path always wins. When they did not give one, fetch the published checkpoint rather than asking:
Pre-Flight resolves the path only.
--planreports whether the checkpoint is already present or would be downloaded, and writes nothing — the download is a side effect and belongs after the gate, with the container pulls:# --plan reports what it would fetch; it prints a sentence, not a path. <skill_root>/scripts/deft_python.sh \ <skill_root>/scripts/fetch_gdino_checkpoint.py --plan \ --dest "$WORKSPACE/checkpoints/gdino" # After approval, the same command without --plan downloads and prints the path. ZERO_SHOT_CHECKPOINT=$(<skill_root>/scripts/deft_python.sh \ <skill_root>/scripts/fetch_gdino_checkpoint.py \ --dest "$WORKSPACE/checkpoints/gdino")Capture the variable from the real run, not from
--plan: under--planthe script printsWILL DOWNLOAD after approval: ... -> <dir>, so capturing it puts an English sentence in a variable every later stage uses as a path.That resolves
nvidia/tao/grounding_dino:grounding_dino_swin_tiny_commercial_trainable_v1.1(1.93 GB) and prints the checkpoint path on stdout. It is idempotent — an existing download is reused, so a resumed run re-costs nothing. Use atrainablerelease; the siblingdeployableone is for TensorRT export and cannot be fine-tuned.Requires the
ngcCLI and a configured account (check 3). On an air-gapped host, or any time the download cannot run, the user supplies--zero-shot-checkpointdirectly — the script says so rather than failing obscurely.Record in the Summary which source won (
userorNGC <version>), and hard-stop if the resolved path does not exist.Then confirm the spec matches the checkpoint's architecture.
model.backbone,num_queries,enc_layers,dec_layers,num_feature_levels, and especiallyclass_embed_biasmust all agree, or the run dies at load time — after the container has started, with an error that reads like a bad checkpoint rather than a bad spec.class_embed_biasis the one that actually bites: it defaults toFalse, is absent from the shippedinfer.yamltemplate, and a checkpoint trained with itTruefails with onlyclass_embed.*.biaskeys reported unexpected. Read the values out of the checkpoint and set them explicitly — seereferences/grounding-dino.md.The checkpoint is coupled to the container, not just the spec. Record which TAO PyTorch image the checkpoint was trained with in the Summary; the pinned image is not automatically the right one.
Co-DETR checkpoint — only when
prepwill run; fetch it unless the user supplied one. Prep pseudo-labels the whole pool with Co-DETR, andassets/overlays/codetr_inference.yamlpins the ViT-L/16 geometry that exactly one published checkpoint loads.The user's own path always wins, as for the zero-shot checkpoint. It is how an air-gapped host runs prep at all, and a checkpoint someone already staged is not replaced. Hard-stop if the path does not exist. Its size should be 2,934,763,233 bytes, the published checkpoint's; flag a different size in the Summary rather than stopping, since a compatible fine-tune can differ, but say that a checkpoint the pinned architecture cannot load empties every pseudo-label and
verify_pseudo_labels.pycatches it only after the labelling pass. A path outside$WORKSPACEneeds its directory in$EXTRA_MOUNTS, or the container cannot see it. Record the source (user) in the Summary and skip the fetcher entirely.When the user gave no path, fetch the published checkpoint rather than asking: a run should not depend on a file somebody staged by hand.
--destbelongs under$WORKSPACE. The Co-DETR container mounts$WORKSPACEand$EXTRA_MOUNTSand nothing else, so a checkpoint anywhere else needs an extra-von every launch that reads it.# Report first; --plan prints a sentence, not a path. <skill_root>/scripts/deft_python.sh \ <skill_root>/scripts/fetch_codetr_checkpoint.py --plan \ --dest "$WORKSPACE/checkpoints/codetr" # After approval, the same command without --plan prints the path. CODETR_CHECKPOINT=$(<skill_root>/scripts/deft_python.sh \ <skill_root>/scripts/fetch_codetr_checkpoint.py \ --dest "$WORKSPACE/checkpoints/codetr")Skip this on a run whose pool is already labelled and embedded — prep does not run, so nothing reads the checkpoint. On a run that will prep, a missing checkpoint is a hard stop here rather than 2.5 hours in: the Co-DETR pass is the longest stage in the workflow and it is the first thing prep does.
The download is plain HTTPS over stdlib, so it needs nothing installed. Size and SHA-256 are verified before the file is put in place, because a checkpoint that does not match the pinned architecture loads no weights, exits 0, prints
Execution status: PASSand writes one empty label file per image.A checkpoint already at
--destis reused after a size check, which catches the truncated file an interrupted transfer leaves. Add--verifyto check its SHA-256 as well when the file is suspect — copied from another host, or left by a run whose pseudo-labels came out empty. It is off by default because it reads the full 2.8 GiB.--planapplies the same checks, so a truncated file fails Pre-Flight instead of appearing in the Summary asALREADY PRESENTand failing after approval.Train-spec template. Must exist and parse as YAML, and
dataset.train_data_sourcesmust be a list (Grounding DINO ODVG shape). A mapping there means the spec is COCO-shaped and this workflow cannot append to it.Seed training data is optional. Unlike the AOI loop — where ChangeNet must learn the task from a mandatory seed set — Grounding DINO is zero-shot capable and can start cold. Inspect the list and branch:
Non-empty → validate every entry's
image_dir,json_file, andlabel_mapresolve on disk. Report the source count and total ODVG record count in the Summary. Iteration 1 appends to what is already there.Empty or absent → note in the Summary that iteration 1 trains on mined data alone, and that the first iteration's dataset will be small. Not an error.
Either way the source pool (check 9) stays mandatory — without it there is nothing to mine and the loop cannot add data at all.
Source pool. Two artifacts, both required:
source_pool_embeddingsparquet — must be non-empty and carryfilepathandembedding.source_pool_annotations— an ODVG tree containing*.jsonlrecords keyed byfile_name, and ideally a*labelmap.json. Staging synthesizes a labelmap from observed categories when none is found, but an explicit one is preferred.
Hard-stop if either is missing or the parquet has zero rows. The loop consumes a prepared pool; it does not build one. Preparing the pool is its own run, completed before the loop launches — see
references/prep-source-pool.md.init_deft_state.pyrefuses to write state without these paths, because the alternative is discovering the corpus is absent atmine, six stages and a training run later.Pass
--pool-reportwith the prep run'spool_report.json; init refuses an existing pool without one, under every allocation policy. It records which classes the pool actually holds annotations for, and init cross-checks that against the target classes.A pool prepared elsewhere may not carry
pool_report.json. Do not re-label to get one — generate it from the pool's COCO withvalidate_pool_coco.py --record(references/prep-source-pool.md). A pool prepared for a different class set does not make mining fail — it makes mining return neighbours of something else, and the affected class simply never improves.Resolve the encoder — local snapshot first, never an implicit online default.
The encoder that embeds each iteration's weak images must be the same one that produced the source-pool parquet. A mismatch is silent: mining succeeds and returns confidently wrong neighbours. Record the resolved values as config.embedding_model / config.embedding_model_path and reuse them verbatim on every iteration.
Resolution order — keep an already-set SIGLIP_MODEL_PATH only when it is a directory containing config.json; otherwise search the known cache roots:
resolved_siglip=""
if [ -n "${SIGLIP_MODEL_PATH:-}" ] && [ -f "$SIGLIP_MODEL_PATH/config.json" ]; then
resolved_siglip="$SIGLIP_MODEL_PATH"
else
for cache_root in \
"${HF_HOME:+$HF_HOME/hub}" \
"${HUGGINGFACE_HUB_CACHE:-}" \
"$HOME/.cache/huggingface/hub" \
"$WORKSPACE/source_pool/hf_cache/hub" \
"$WORKSPACE/source_pool/hf_cache"; do
[ -n "$cache_root" ] || continue
for snapshot in "$cache_root/models--google--siglip-base-patch16-224/snapshots/"*; do
if [ -f "$snapshot/config.json" ]; then
resolved_siglip="$snapshot"
break 2
fi
done
done
fi
[ -n "$resolved_siglip" ] && export SIGLIP_MODEL_PATH="$(realpath "$resolved_siglip")"
# init takes it as --embedding-model-path; bind it here or the resolution is lost.
export EMBEDDING_MODEL_PATH="${SIGLIP_MODEL_PATH:-}"Rules:
Select a snapshot only from
models--google--siglip-base-patch16-224/snapshots/*and verify itsconfig.json. Never substitute a DINO, C-RADIO, or other model that happens to be cached — it will embed successfully and produce garbage matches.HF_HOMEmay point outside the workspace, so do not limit the search to the workspace tree.Fall back to the bare HuggingFace id
google/siglip-base-patch16-224only after outbound HuggingFace access has been verified, and only then requireHF_TOKEN. Never let the embed stage pick an online default implicitly.If no local snapshot resolves and outbound access cannot be verified, hard-stop. Report the cache roots searched.
Record the resolved snapshot path (or the verified HF id) in the Summary. If the user cannot say which encoder produced the source pool, surface that as an explicit risk row rather than assuming SigLIP.
KPI inputs. Image directory, ground-truth KITTI label directory, and class-mapping YAML must all exist.
image_dirmust not end in/—kpi_analyzederives itsSequence Namefrom the second-to-last path component.Class thresholds and mining config. Per-class AP50 thresholds and the mining
multiplierboth have reference defaults — do not interrogate the user for them. Omitting--ap50-thresholds-jsongates each target class at the reference ITS value (car 0.99,bicycle 0.7,person 0.7) and any other target class at0.7;--multiplierdefaults to3. Surface the defaulted values in the Pre-Flight Summary so the user can override them, and treat a class gated by assumption as worth flagging: too loose a gate marks no image weak and the iteration mines nothing. If rare classes are configured, also requiresource_detection_fileandtarget_detection_fileas COCO JSONs —class_stratifiedmining needs them and TAO DS will not infer the format.GPU count.
if command -v nvidia-smi >/dev/null 2>&1; then nvidia-smi --list-gpus | wc -l else docker run --rm --gpus all "$TAO_PYT_IMAGE" python3 -c 'import torch; print(torch.cuda.device_count())' fiSpec sanity.
train.checkpoint_intervalmust be<= train.num_epochs.prepare_spec_for_train.pylowers it automatically when an explicit epoch override would violate this, but flag the adjustment in the Summary so it is not a surprise.
max_iterations defaults to 1. Confirm it with the user when they have not said how many iterations they want, but do not block on it: an unattended run takes the default.
Defaults
train.num_epochs— from the train-spec templatetrain.optim.lr— from the train-spec templatemultiplier—3max_iterations—1allocation_policy—class_stratifiedwhen rare classes are given, elseglobalrare_class_list— every target class holding a below-mean share of the pool's annotations. It is a property of the pool, so on a run that still has to prep it is left unset at init and the prep commit derives it frompool_report.json. Supply--rare-class-listonly to override that.distance_metric—euclideancandidate_expansion_factor—5embedding_model—SigLIP;embedding_model_pathis resolved, not defaulted (check 10)iou_threshold—0.5kpi.conf_threshold—0.0(frozen asconfig.kpi_conf_threshold; every phase is scored at it)workspace root — user prompt, else
~/workspace
Container identity
Every launch in this skill passes --user "$(id -u):$(id -g)" $DOCKER_IDENTITY. A uid that has no
/etc/passwd entry inside the image — which is every uid but 1000 on the TAO images —
then has no name, and getpass.getuser() falls back to pwd.getpwuid() and raises.
torch calls it while setting up the inductor cache directory, so the container dies at
import with KeyError: 'getpwuid(): uid not found: <uid>' before the requested action
starts. $HOME is unset for the same reason and defaults to /, which is not writable,
so matplotlib and cupy fail on their cache directories.
Export the identity beside the mounts and pass it to every launch:
DEFT_HOME="$WORKSPACE/.deft_home"
mkdir -p "$DEFT_HOME"
DOCKER_IDENTITY="-e USER=deft -e HOME=$DEFT_HOME"
export DOCKER_IDENTITYBoth are required, and neither is sufficient alone. USER satisfies
getpass.getuser(), which reads it before consulting pwd; any non-empty value works,
and a literal keeps container logs identical whoever runs the workflow. HOME defaults
to /, which is not writable, so without it matplotlib and cupy fail on their cache
directories even once the uid has a name.
HOME points into the workspace rather than at /tmp so the caches are owned by the
caller and removed with the workspace.
This is invisible on a uid-1000 workstation and fires on every shared or cloud host, so a
run that works locally proves nothing about it. Dropping --user is not the alternative:
see references/pipeline-and-state.md.
Container mounts
Export them once, right after init, and use the variable in every launch:
EXTRA_MOUNTS=$(<skill_root>/scripts/deft_python.sh -c 'import json,sys
m=json.load(open(sys.argv[1]))["config"].get("extra_container_mounts") or []
print(" ".join(f"-v {p}:{p}" for p in m))' "${RESULTS_DIR}/deft_state.json") || exit 1
export EXTRA_MOUNTS
echo "EXTRA_MOUNTS=$EXTRA_MOUNTS"Echo it and read the result. A command substitution that fails leaves EXTRA_MOUNTS
empty and every later launch still runs, mounting nothing extra — the same silent
failure this block exists to prevent.
init_deft_state.py derives this list from the run's own inputs — the checkpoints, the
KPI images and labels, the class mapping, the encoder repo, the pool, and both COCO
detection files. A path that none of those name is invisible to it: pass
--extra-mount PATH (repeatable) at init and it joins the list. Files are mounted by
their parent directory.
Every docker run in this skill is written as
-v "$WORKSPACE:$WORKSPACE" $EXTRA_MOUNTS -w "$WORKSPACE". With inputs inside the
workspace it expands to nothing and the command is unchanged; with KPI data or
checkpoints outside it, it is the difference between a working stage and
No .txt label files found.
A HuggingFace snapshot directory is a tree of symlinks into a sibling blobs/, so the
mount must be the repo root (.../models--org--name/), not the snapshot. Mounting the
snapshot alone makes the loader report a missing model file for a model that is present.
init_deft_state.py handles this when it derives the list.
init_deft_state.py records config.extra_container_mounts: the directories that
inputs live in outside $WORKSPACE. Containers see only "$WORKSPACE:$WORKSPACE",
so every docker run in this skill must add a -v "$m:$m" for each entry, in
addition to the workspace mount. KPI images, ground truth and checkpoints commonly
sit outside the workspace, and a container cannot read what is not mounted.
Pre-Flight Summary
Print this and STOP — wait for explicit approval. This is the only user gate.
## DEFT OD Loop — Pre-Flight Summary
### Run config
| Field | Value | Source |
| ---------------------- | ---------------------------------------------- | ----------------- |
| Model | Grounding DINO (ODVG) | workflow |
| Max iterations | N | user |
| Stop condition | max_iterations reached, or zero weak images.
mAP is reported, not gated — no target. | workflow |
| Epochs / LR | N / X | user/spec |
| Encoder | <model> @ <resolved snapshot or verified HF id>| resolved |
| Allocation policy | class_stratified / global | user/default |
| Rare classes | <list or none> | user |
| Mining multiplier | N (budget = iter1 weak count x N) | user/default |
| AP50 thresholds | {"car": 0.99, ...} | user/default |
| GPUs | N | detected |
| Resuming | yes — iter N complete / no | disk |
### Inputs
| Field | Value |
| ------------------------- | -------------------------------------------- |
| Zero-shot checkpoint | <path> |
| Train spec template | <path> (N base source(s); 0 = mined-only) |
| Source pool embeddings | <path> (N rows, encoder: <model>) |
| Source pool annotations | <path> (N jsonl, labelmap: found/synthesized)|
| KPI images | <path> |
| KPI ground truth | <path> (N label files) |
| Class mapping | <path> |
### Docker images
| Env var | Image | Status |
| --------------- | ------------------ | ---------- |
| `TAO_PYT_IMAGE` | `<$TAO_PYT_IMAGE>` | OK/MISSING |
| `TAO_DS_IMAGE` | `<$TAO_DS_IMAGE>` | OK/MISSING |
### Per-iteration stages
gap_analysis -> embed -> mine -> stage -> train -> inference -> kpi_analyze
(baseline runs inference -> kpi_analyze only; no training)Remind the user to enable auto-mode (shift+tab) before approving — the post-gate loop is continuously side-effecting.
Run variables
Every stage reference launches containers with shell variables. They are set once here and used unchanged for the rest of the run; nothing later re-derives them. Export them before the first stage:
# Resolved at the gate above
export WORKSPACE RESULTS_DIR MAX_ITERATIONS NUM_GPUS NUM_EPOCHS LEARNING_RATE
export MULTIPLIER ALLOCATION_POLICY AP50_THRESHOLDS_JSON
export ZERO_SHOT_CHECKPOINT TRAIN_SPEC_TEMPLATE
export SOURCE_POOL_EMBEDDINGS SOURCE_POOL_ANNOTATIONS
export EMBEDDING_MODEL EMBEDDING_MODEL_PATH
export KPI_IMAGES_DIR GROUND_TRUTH_LABELS_DIR CLASS_MAPPING
export TAO_PYT_IMAGE TAO_DS_IMAGE EXTRA_MOUNTS DOCKER_IDENTITY
# Prep runs only
export POOL_IMAGES CODETR_CHECKPOINT CODETR_CLASSMAP CLASSES_YAML
# Read by the init block below; unset ones simply drop out of it
export TARGET_CLASSES KPI_CONF_THRESHOLD
export SOURCE_DETECTION_FILE TARGET_DETECTION_FILE # class_stratified only
export POOL_REPORT RARE_CLASS_LIST # an already-prepared pool only
export EXTRA_MOUNTS_IN # space-separated extra mounts, if anyTwo more are set per iteration rather than once, and every stage reference uses them:
N=<current iteration number> # 1, 2, ... — the loop counter; paths read iter${N}
PHASE=<baseline|iter${N}> # which phase's output tree this stage writes toN is the iteration being run. PHASE is baseline for the zero-shot pass and
iter${N} inside the loop; the two differ only because the baseline is not an
iteration. Re-export both at the top of each iteration.
An unset variable is not an error in a docker run: it expands to empty, so
-e "" reaches TAO as a blank spec path and the failure reads as a spec problem rather
than a missing export. Set them all, and prefer set -u in the shell driving the run so
a typo fails at the point of use.
Each stage's own spec path is bound in that stage's reference, next to the path it
documents — $CODETR_SPEC, $INFER_SPEC, $OD_GAP_SPEC, $EMBED_SPEC, $MINE_SPEC
and $KPI_SPEC.
Immediately After Approval
Perform the planned pulls and directory creation, then initialize state once:
--pool-report is the exception: omit it on a prep run. Unlike the pool artifacts
and --source-detection-file, a --pool-report path that does not exist yet is a hard
error, not a warning — validate_pool_coco.py writes it during prep, so pass the flag
only when the pool already exists. Omitting it is what downgrades the requirement.
Order: init_deft_state.py runs first and prep is the run's first committed
stage. On a pool that still needs preparing, the pool artifacts and
--source-detection-file do not exist yet — pass them anyway, as the paths prep
will write, together with prep's inputs (--pool-images, --codetr-checkpoint,
--codetr-classmap). Init reports them as warnings rather than errors, because prep
produces them. Supplying neither the artifacts nor prep's inputs is still an error:
nothing would create them.
Three groups of flags apply only to some runs. Build them first so the command below stays one shape:
# class_stratified only. Both are mandatory under that policy — init refuses without
# them, so a run that omits them here fails at `mine`, several stages later.
STRATIFIED_FLAGS=""
if [ "$ALLOCATION_POLICY" = class_stratified ]; then
STRATIFIED_FLAGS="--source-detection-file $SOURCE_DETECTION_FILE"
STRATIFIED_FLAGS="$STRATIFIED_FLAGS --target-detection-file $TARGET_DETECTION_FILE"
fi
# Prep runs only: the pool does not exist yet and these are what builds it.
PREP_FLAGS=""
if [ -n "${POOL_IMAGES:-}" ]; then
PREP_FLAGS="--pool-images $POOL_IMAGES --codetr-checkpoint $CODETR_CHECKPOINT"
PREP_FLAGS="$PREP_FLAGS --codetr-classmap $CODETR_CLASSMAP"
fi
# A pool that already exists must declare what it holds: prep is what would otherwise
# produce the report, and a prep that is not going to run cannot. The report is
# required under every policy; under class_stratified the rare classes are too, since
# prep would otherwise derive them from it.
EXISTING_POOL_FLAGS=""
if [ -z "${POOL_IMAGES:-}" ]; then
EXISTING_POOL_FLAGS="--pool-report $POOL_REPORT"
if [ "$ALLOCATION_POLICY" = class_stratified ]; then
EXISTING_POOL_FLAGS="$EXISTING_POOL_FLAGS --rare-class-list $RARE_CLASS_LIST"
fi
fi
# Any path the run's own inputs do not name. Repeatable; omit when there are none.
EXTRA_MOUNT_FLAGS=""
for path in ${EXTRA_MOUNTS_IN:-}; do
EXTRA_MOUNT_FLAGS="$EXTRA_MOUNT_FLAGS --extra-mount $path"
done<skill_root>/scripts/deft_python.sh <skill_root>/scripts/init_deft_state.py \
--workspace "$WORKSPACE" \
--results-dir "$RESULTS_DIR" \
--max-iterations "$MAX_ITERATIONS" \
--num-gpus "$NUM_GPUS" \
--num-epochs "$NUM_EPOCHS" \
--learning-rate "$LEARNING_RATE" \
--zero-shot-checkpoint "$ZERO_SHOT_CHECKPOINT" \
--train-spec-template "$TRAIN_SPEC_TEMPLATE" \
--source-pool-embeddings "$SOURCE_POOL_EMBEDDINGS" \
--source-pool-annotations "$SOURCE_POOL_ANNOTATIONS" \
--embedding-model "$EMBEDDING_MODEL" \
--embedding-model-path "$EMBEDDING_MODEL_PATH" \
--kpi-images-dir "$KPI_IMAGES_DIR" \
--ground-truth-labels-dir "$GROUND_TRUTH_LABELS_DIR" \
--class-mapping "$CLASS_MAPPING" \
--ap50-thresholds-json "$AP50_THRESHOLDS_JSON" \
--multiplier "$MULTIPLIER" \
--allocation-policy "$ALLOCATION_POLICY" \
--target-classes "$TARGET_CLASSES" \
--kpi-conf-threshold "$KPI_CONF_THRESHOLD" \
$STRATIFIED_FLAGS $PREP_FLAGS $EXISTING_POOL_FLAGS $EXTRA_MOUNT_FLAGS
<skill_root>/scripts/deft_python.sh <skill_root>/scripts/audit_deft_run.py \
--results-dir "$RESULTS_DIR"Then copy the train-spec template to ${RESULTS_DIR}/train_grounding_dino.yaml and begin the baseline.
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Declares allowed tools, Needs Python
The author says: Requires docker + nvidia-container-toolkit and one or more CUDA GPUs. Workflows declare additional requirements.
Security
What our scanner noticed· Medium risk
- Large base64-encoded blobs
- References to credential files
- Scripts that your agent may run
This skill includes scripts that your agent may run. Read them before you use it.
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Created by
- NVIDIA Corporation
- Publisher
- NVIDIA (Official)
- Source
- NVIDIA/skills · skills/tao-run-deft-object-detection · commit fce3cca
- Licence
- Apache License 2.0 (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE-APACHE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
AutoGineer is not affiliated with or endorsed by NVIDIA Corporation or NVIDIA.
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download tao-run-deft-object-detection.zip