Cdf Auto Labeling

CDF Auto-Labeling

Use this skill when a user wants to kick off CDF Auto-Labeling on their own data, domain, or use case, or when the request matches a shipped cookbook, stage, authoring, or migration task. This is a router: sequence the specialized references instead of duplicating their detail.

Routing (Read First)

Stage references: super-resolution, detection-and-tracking, captioning, visual-qa, reasoning, person-attribute-search, grounding-2d, referring-expressions, training-export.

Instructions

1. Confirm the critical run inputs with the user before doing anything else, and ask a concise question whenever one is missing or ambiguous - never guess or silently invent a default. At minimum confirm: input data path, output path, VLM/LLM endpoint URLs and model names, model cache path, GPU ids, and (for reasoning-capable models) the max_tokens cap. Restate the confirmed values back to the user before the first execution.

2. Verify the environment: repository cloned, make targets available, the model cache path exists, the VLM/LLM endpoints are reachable, and a GPU is available. State any missing prerequisite as a blocker instead of assuming it. If the stage images are not present locally, offer to pull the prebuilt images from NGC with make pull instead of building them. Use ngc-containers.

3. Run a shipped example first to confirm the stack works end to end before customizing. Pick the closest operator pipeline - video data augmentation, event-and-person-attribute-search, or event-verification-reasoning - and run its committed cookbook. Use the matching operator reference.

4. Plan the target scenario: define modality, domain, intended consumer, and required annotations, and get a minimal stage subset. Use scenario-planning.

5. Adapt the closest shipped cookbook to the new domain rather than authoring from scratch. Use cookbook-authoring.

6. Author the domain prompts and question banks. Use prompt-authoring.

7. Configure the per-stage settings for the domain (detector classes or SAM3 prompts, endpoints, windowing, max_tokens). Use the relevant stage reference, starting with detection-and-tracking.

8. Dry-run the adapted cookbook, then execute and validate the outputs. Use workflow-runner-debugging.

Adopting an existing external annotation or dataset-generation repository into CDF instead of starting from a shipped cookbook is a migration task; use pipeline-migration for that path.

Examples

New user, new domain: "I cloned the repo and have my own warehouse-safety video. How do I produce auto-labels for my domain?"

Guided path:

• Confirm env (model cache, VLM/LLM endpoints, GPU), then prove the stack on a shipped example before customizing:

make run SCRIPT=workflow-runner:main \
  ARGS='--cookbook-file cookbooks/video_data_augmentation/configs/pipeline_video.yaml --container-dry-run'

• Plan the domain (scenario-planning) -> subset detection_and_tracking -> captioning -> visual_qa -> reasoning -> training_export (add grounding_2d for caption→boxes or referring_expressions for boxes→phrases; use grounding-2d / referring-expressions).

• Copy the closest cookbook to cookbooks/warehouse_safety/configs/pipeline.yaml and adapt inputs, detector classes/SAM3 prompts, prompts, and question banks.

• Dry-run the new cookbook, then run for real and validate outputs:

make run SCRIPT=workflow-runner:main \
  ARGS='--cookbook-file cookbooks/warehouse_safety/configs/pipeline.yaml --container-dry-run'

Guardrails

• Do not guess or fabricate the critical inputs enumerated in step 1; if any is missing or ambiguous, ask the user and confirm before executing.

• Do not customize a cookbook before a shipped example runs clean; a broken base makes domain debugging ambiguous.

• Keep the first custom pipeline minimal - only the stages needed for the requested annotations - and expand later.

• Verify that every selected stage's service package and image exist in the current branch before promising an end-to-end run.

• Do not put secrets, tokens, or absolute home paths in committed cookbooks; use placeholders such as <model-cache> and env vars for endpoint keys.

• For reasoning-capable models (for example Gemini 3 Flash), raise max_tokens on the visual_qa and reasoning LLM substages to avoid the thinking-token tax; keep the default cap for non-reasoning models.

• Do not rely on non-CDF pipelines, commands, or file locations. A first run must be reproducible through workflow-runner:main inside this repo.