AutoGineer
Generated banner for Cdf Auto Labeling
Generated logo for Cdf Auto Labeling

Cdf Auto Labeling

NVIDIAOfficialCreated by NVIDIA

  • Skill
  • Workflow
  • Automation and integrations
  • Apache-2.0
  • Medium risk
Download ZIP

From the author

Use when a user needs to get started with CDF Auto-Labeling, plan a scenario, run or debug a shipped cookbook, author prompts or cookbooks, migrate a pipeline, configure a stage, or pull the prebuilt stage containers from NGC (nvcr.io). Confirm critical inputs (data path, output path, endpoints) and ask when any are missing. This is a router: read the matching reference instead of inventing a workflow.
— the description by NVIDIA

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

CDF Auto-Labeling is a central routing skill designed to guide users through the lifecycle of automated data labeling tasks within the Cosmos Data Factory ecosystem. It provides a structured approach to onboarding, scenario planning, cookbook authoring, and pipeline execution. By acting as a router, it directs users to specialized documentation for specific tasks—such as migrating pipelines, debugging workflows, or managing containerized stages—while enforcing strict input validation and operational guardrails to ensure reproducible results.

What it does

  • Routes users to specific documentation based on their current task or stage in the pipeline.
  • Validates critical run inputs like data paths, output directories, and model endpoints.
  • Provides a standardized workflow for running, debugging, and customizing annotation cookbooks.
  • Facilitates the integration of new stages and the management of prebuilt container images.
  • Enforces operational guardrails to prevent common errors like hardcoding secrets or skipping validation steps.

When to use it

  • When you are a new user needing to get started with CDF Auto-Labeling.
  • When you need to plan a new annotation scenario or adapt an existing cookbook.
  • When you are troubleshooting a failed pipeline run or debugging a specific stage.
  • When you need to migrate an external annotation repository into the CDF framework.
  • When you need to pull or configure prebuilt stage containers from NGC.

Advantages

  • Ensures consistent and reproducible pipeline execution through standardized cookbooks.
  • Prevents configuration errors by mandating input validation before execution.
  • Simplifies complex workflows by breaking them down into manageable, modular tasks.
  • Promotes best practices by delegating specialized tasks to dedicated reference guides.

Usage examples

“I cloned the repo and have my own warehouse-safety video. How do I produce auto-labels for my domain?”
The agent will guide you to verify your environment, run a shipped example to confirm the stack, plan your scenario, adapt a cookbook, and execute a dry-run.
“How do I pull the prebuilt stage containers from NGC?”
The agent will direct you to the ngc-containers reference and provide instructions on using make pull to retrieve the necessary images.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/nvidia/cdf-auto-labeling/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip cdf-auto-labeling.zip -d .agents/skills

Documentation by NVIDIA

CDF Auto-Labeling

Use this skill when a user wants to kick off CDF Auto-Labeling on their own data, domain, or use case, or when the request matches a shipped cookbook, stage, authoring, or migration task. This is a router: sequence the specialized references instead of duplicating their detail.

Routing (Read First)

Request looks likeRead
New user, clean checkout, first validated run, "how do I get started"This file, then the matching reference below
Choose annotation targets / stage subset for a domainreferences/scenario-planning.md
Create, review, or adapt a cookbookreferences/cookbook-authoring.md
Write or adapt VLM/LLM prompts or question banksreferences/prompt-authoring.md
Migrate an existing annotation repo into this onereferences/pipeline-migration.md
Run the video data augmentation cookbookreferences/video-data-augmentation.md
Run or choose an EPAS / PAS cookbookreferences/event-and-person-attribute-search.md
Run event-verification reasoningreferences/event-verification-reasoning.md
Debug an already-integrated workflowreferences/workflow-runner-debugging.md
Pull prebuilt stage images from NGC, log in to nvcr.io, or fix a missing cdf-* imagereferences/ngc-containers.md
Implement or review a new stage or Dockerized servicereferences/workflow-stage-integration.md
Configure or debug one production stageThe matching file under references/stages/

Stage references: super-resolution, detection-and-tracking, captioning, visual-qa, reasoning, person-attribute-search, grounding-2d, referring-expressions, training-export.

Instructions

  1. Confirm the critical run inputs with the user before doing anything else, and ask a concise question whenever one is missing or ambiguous - never guess or silently invent a default. At minimum confirm: input data path, output path, VLM/LLM endpoint URLs and model names, model cache path, GPU ids, and (for reasoning-capable models) the max_tokens cap. Restate the confirmed values back to the user before the first execution.

  2. Verify the environment: repository cloned, make targets available, the model cache path exists, the VLM/LLM endpoints are reachable, and a GPU is available. State any missing prerequisite as a blocker instead of assuming it. If the stage images are not present locally, offer to pull the prebuilt images from NGC with make pull instead of building them. Use ngc-containers.

  3. Run a shipped example first to confirm the stack works end to end before customizing. Pick the closest operator pipeline - video data augmentation, event-and-person-attribute-search, or event-verification-reasoning - and run its committed cookbook. Use the matching operator reference.

  4. Plan the target scenario: define modality, domain, intended consumer, and required annotations, and get a minimal stage subset. Use scenario-planning.

  5. Adapt the closest shipped cookbook to the new domain rather than authoring from scratch. Use cookbook-authoring.

  6. Author the domain prompts and question banks. Use prompt-authoring.

  7. Configure the per-stage settings for the domain (detector classes or SAM3 prompts, endpoints, windowing, max_tokens). Use the relevant stage reference, starting with detection-and-tracking.

  8. Dry-run the adapted cookbook, then execute and validate the outputs. Use workflow-runner-debugging.

Adopting an existing external annotation or dataset-generation repository into CDF instead of starting from a shipped cookbook is a migration task; use pipeline-migration for that path.

Examples

New user, new domain: "I cloned the repo and have my own warehouse-safety video. How do I produce auto-labels for my domain?"

Guided path:

  • Confirm env (model cache, VLM/LLM endpoints, GPU), then prove the stack on a shipped example before customizing:

make run SCRIPT=workflow-runner:main \
  ARGS='--cookbook-file cookbooks/video_data_augmentation/configs/pipeline_video.yaml --container-dry-run'

  • Plan the domain (scenario-planning) -> subset detection_and_tracking -> captioning -> visual_qa -> reasoning -> training_export (add grounding_2d for caption→boxes or referring_expressions for boxes→phrases; use grounding-2d / referring-expressions).

  • Copy the closest cookbook to cookbooks/warehouse_safety/configs/pipeline.yaml and adapt inputs, detector classes/SAM3 prompts, prompts, and question banks.

  • Dry-run the new cookbook, then run for real and validate outputs:

make run SCRIPT=workflow-runner:main \
  ARGS='--cookbook-file cookbooks/warehouse_safety/configs/pipeline.yaml --container-dry-run'

Guardrails

  • Do not guess or fabricate the critical inputs enumerated in step 1; if any is missing or ambiguous, ask the user and confirm before executing.

  • Do not customize a cookbook before a shipped example runs clean; a broken base makes domain debugging ambiguous.

  • Keep the first custom pipeline minimal - only the stages needed for the requested annotations - and expand later.

  • Verify that every selected stage's service package and image exist in the current branch before promising an end-to-end run.

  • Do not put secrets, tokens, or absolute home paths in committed cookbooks; use placeholders such as <model-cache> and env vars for endpoint keys.

  • For reasoning-capable models (for example Gemini 3 Flash), raise max_tokens on the visual_qa and reasoning LLM substages to avoid the thinking-token tax; keep the default cap for non-reasoning models.

  • Do not rely on non-CDF pipelines, commands, or file locations. A first run must be reproducible through workflow-runner:main inside this repo.

Files

references/stages/referring-expressions.md

Referring Expressions stage

Single-stage reference for referring_expressions: known boxes → short discriminative phrases via a VLM (SoM-lite numbered overlays). Inverse of grounding_2d (caption → boxes).

  • When to use / not use

  • Packaging

  • Config

  • Instructions

  • Inputs → Outputs

  • Examples

  • Gotchas

  • Guardrails

When to use / not use

  • Use: wiring boxes→phrases after detection or grounding, tuning VLM/IoU/overlay knobs, or debugging empty/mismatched region descriptions.

  • Do not use: to run a full pipeline end to end, author a whole cookbook, or ground captions into new boxes (that is grounding-2d.md).

Packaging

ImageRole
referring-expressions-serviceSlim VLM product image (no SAM3/CUDA bake)
detection-and-tracking-sam3-serviceUpstream boxes when composing detect → refer

Stages scale independently. Mount SAM3 weights only on the detection stage; the referring stage needs VLM + contextual/objects.json. No product PYTHONPATH mounts.

Read referring-expressions-patterns.md for inputs, type vocabulary, and validation checks.

Config

Cookbook block referring_expressions: (essential):

  • enabled: true

  • VLM via endpoints.vlm

  • Image: container.images.referring_expressions: referring-expressions-service

Common compositions:

# Detect then refer
stages: [detection_and_tracking, referring_expressions]

# Refer only (objects.json already present)
stages: [referring_expressions]

Flat stage_args.referring_expressions knobs:

  • --draw-box-overlay — write SoM overlay under sidecars

  • --min-match-iou — link VLM marks to DAFT boxes

  • --frame-number — frame index in objects.json

  • --max-tokens, --temperature, --top-p, --timeout-s, --retries

  • --force-reprocess

Object type is open vocabulary (no closed traffic enum). Detection prompts belong under stage_args.detection_and_tracking when that stage is included.

Confirm flags with a dry run: make run SCRIPT=workflow-runner:main ARGS='--cookbook-file <config> --container-dry-run'

Instructions

This skill returns referring configuration or debugging guidance; it does not run the pipeline. Provide the config immediately - do not gate it behind execution.

  1. Classify the task. Configuring/tuning referring_expressions, or debugging phrases/overlays. If the request is a full pipeline run, whole-cookbook authoring, or caption→boxes grounding, STOP and hand off.

  2. Gather needs. Identify whether boxes already exist (contextual/objects.json) or detection must run first; VLM endpoint; image paths. If a required input is missing or ambiguous, ask - do not guess.

  3. Return the config. Produce referring_expressions: / stage_args / images per Config. For detect→refer, set SAM3 prompts on detection only; keep referring mounts free of weight trees when objects already exist. Suggest --container-dry-run.

Execution is out of scope. Docker via workflow-runner requires explicit user approval (operator skill).

When debugging. On missing contextual/objects.json, unreachable VLM, or empty sidecars/referring_expressions/referring_expressions.json, report the specific cause; do not fabricate phrases.

Inputs → Outputs

  • Consumes: image media_path; contextual/objects.json (and optional contextual/instances.json) from detection or grounding.

  • Produces: sidecars/referring_expressions/referring_expressions.json, sidecars/referring_expressions/ (step0 + optional marked_boxes.jpg).

Examples

Detect → refer:

stages:
  - detection_and_tracking
  - referring_expressions
container:
  images:
    detection_and_tracking: detection-and-tracking-sam3-service
    referring_expressions: referring-expressions-service
  env:
    NVIDIA_API_KEY: EMPTY
    SAM3_MODEL_PATH: /models/sam3
  mounts:
    - <sam3-weights>:/models/sam3:ro
endpoints:
  vlm:
    url: http://host.docker.internal:8000/v1
    model: Qwen/Qwen3-VL-30B-A3B-Instruct
detection_and_tracking:
  enabled: true
  model: sam3
  tracker: sam3
  classes: [person, vehicle]
referring_expressions:
  enabled: true
stage_args:
  detection_and_tracking: >-
    --sam3-prompts person vehicle
    --sam3-score-threshold-detection 0.5
  referring_expressions: >-
    --draw-box-overlay
    --min-match-iou 0.3
    --frame-number 0

Refer only:

stages:
  - referring_expressions
container:
  images:
    referring_expressions: referring-expressions-service
  env:
    NVIDIA_API_KEY: EMPTY
referring_expressions:
  enabled: true
stage_args:
  referring_expressions: >-
    --draw-box-overlay --min-match-iou 0.3 --frame-number 0

Dry-run:

make run SCRIPT=workflow-runner:main \
  ARGS='--cookbook-file cookbooks/image_spatial_grounding/configs/pipeline_referring.yaml --container-dry-run'

Gotchas

  • Requires upstream boxes; referring does not detect objects itself.

  • Open type strings are expected; do not reintroduce a closed traffic enum.

  • Mark/ID mismatches can still occur when many small boxes crowd the frame — tighten detection thresholds or --min-match-iou before rewriting prompts.

  • Rebuild referring-expressions-service after task/service code changes.

Guardrails

Follow guardrails.md. Referring cookbooks mount SAM3 via <sam3-weights>; this stage does not detect objects itself.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Security

What our scanner noticed· Medium risk

  • Large base64-encoded blobs

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
NVIDIA
Publisher
NVIDIA (Official)
Source
NVIDIA/skills · skills/cdf-auto-labeling · commit fce3cca
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE-APACHE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download cdf-auto-labeling.zip