AutoGineer
Generated banner for Tao Analyze Gaps Vlm Bcq
Generated logo for Tao Analyze Gaps Vlm Bcq

Tao Analyze Gaps Vlm Bcq

NVIDIAOfficialCreated by NVIDIA Corporation

  • Skill
  • Review and audit
  • Software development
  • Apache-2.0
  • Medium risk
  • Has scripts
Download ZIP

From the author

Extract false-positive and false-negative gaps from VLM binary-classification-question (BCQ, yes/no) predictions. Use when the user asks to "analyze VLM BCQ gaps", "extract VLM false positives and false negatives", or identify failure cases from a predictions JSON for DEFT root-cause analysis on a binary-classification VLM workflow.
— the description by NVIDIA Corporation

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill automates the evaluation of Vision Language Model (VLM) binary classification tasks. By comparing model responses against ground truth data, it identifies specific failure cases—false positives and false negatives—and generates structured reports. These outputs are designed to facilitate root-cause analysis and support iterative model improvement workflows.

What it does

  • Parses VLM prediction JSON files for binary yes/no classification results.
  • Compares model responses against provided ground truth labels.
  • Identifies and categorizes false-positive and false-negative prediction errors.
  • Generates a JSONL file containing detailed records of each identified failure case.
  • Produces a human-readable summary report detailing total error counts.
  • Supports relative path resolution for video assets referenced in predictions.

When to use it

  • When you need to audit VLM performance on binary classification tasks.
  • When preparing data for root-cause analysis (RCCA) in a DEFT iteration workflow.
  • When you need to extract specific failure samples from a large set of model predictions.
  • When you need to generate KPI reports for model evaluation cycles.

Advantages

  • Automates the tedious process of manual error identification in VLM outputs.
  • Produces machine-readable output for seamless integration with downstream analysis tools.
  • Provides clear, human-readable summary reports for quick performance assessment.
  • Standardizes the evaluation process for binary classification workflows.

Usage examples

“Analyze my VLM BCQ gaps using the results.json file.”
The skill will process the predictions, compare them against ground truth, and generate a kpi_gaps.jsonl file and a kpi_gaps_report.txt summary.
“Extract false positives and false negatives from my model predictions.”
The agent will identify the failure cases based on the provided predictions JSON and output the specific samples that were misclassified.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/nvidia/tao-analyze-gaps-vlm-bcq/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip tao-analyze-gaps-vlm-bcq.zip -d .agents/skills

Documentation by NVIDIA Corporation

VLM Binary Classification Gap Analysis

Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).

Reads a VLM predictions JSON, compares each model response against ground truth, and writes FP/FN failure cases to a JSONL file with a summary report. Run it with a TAO Data Services spec file; the data-services entrypoint requires -e <spec>.

Purpose

After running a VLM on a binary yes/no evaluation task, the predictions need to be compared against ground truth to identify failure cases. This skill produces a structured list of FP (false positive) and FN (false negative) samples that downstream RCCA stages (e.g., cosmos generation, root cause analysis) consume to drive a DEFT iteration.

Usage

Generate a vlm_bcq_spec.yaml with the bundled helper:

python3 skills/data/tao-analyze-gaps-vlm-bcq/scripts/prepare_vlm_bcq_spec.py \
  --predictions-json /path/to/results.json \
  --videos-dir /path/to/videos/root \
  --results-dir /path/to/output/gaps \
  --output-spec /path/to/output/gaps/vlm_bcq_spec.yaml

Omit --videos-dir when prediction video_id values are already absolute. The generated spec has this shape:

predictions_json: /path/to/results.json
videos_dir: ""
results_dir: /path/to/output/gaps

Set videos_dir when video_id values in the predictions are relative paths:

predictions_json: /path/to/results.json
videos_dir: /path/to/videos/root
results_dir: /path/to/output/gaps

Invoke the vlm_bcq action inside the TAO Toolkit data services container with -e <spec>:

gap_analysis vlm_bcq -e /path/to/vlm_bcq_spec.yaml

Request exactly one GPU from the selected platform (compute_shape.gpus: 1, compute_shape.nodes: 1). VLM BCQ gap analysis does not perform GPU compute, but the Data Services image always calls nvidia-smi and fails when no GPU is visible. One is a GPU count, not a device ID; the platform selects the device.

After the run, surface the FP/FN counts from kpi_gaps_report.txt and point downstream stages at kpi_gaps.jsonl.

Inputs

  • config spec: YAML file passed with -e. Template: assets/default_vlm_bcq.yaml.

  • predictions_json: Path to predictions JSON file. Must be a JSON array where each item has video_id, response, and gt fields. response and gt are parsed with word-boundary matching — 'yes' or 'no' anywhere in the string is recognized. Samples where both or neither are present are skipped with a warning.

  • videos_dir (optional): Base directory for resolving relative video_id paths. If omitted, video_id values are used as absolute paths.

  • results_dir: Output directory for gap-analysis artifacts.

Predictions JSON format:

[
  {
    "video_id": "/path/to/video.mp4",
    "response": "Yes, there is a collision.",
    "gt": "B. No",
    "question": "Is there a collision?"
  }
]

Outputs

  • kpi_gaps.jsonl: One JSON object per line for each FP/FN case. Fields: video_id (absolute path), error_type (FP or FN), question, ground_truth, response.

  • kpi_gaps_report.txt: Human-readable table with total FP/FN counts.

If no gaps are found, no files are written and a message is logged.

Spec Fields

ParameterRequiredDescription
predictions_jsonYesPath to predictions JSON file
results_dirYesOutput directory; created if it does not exist
videos_dirNoBase directory for resolving relative video_id paths

Keep the spec file and every path it references under the bind-mounted workspace so they resolve inside the container. Pass -e <spec> even if you also add Hydra overrides; current TAO Data Services entrypoints hard-require an experiment spec file before processing overrides.

Error Patterns

ErrorCauseFix
FileNotFoundErrorpredictions_json does not existCheck the path
requires the following argument: -e/--experiment_spec_fileThe container was launched without a spec fileWrite vlm_bcq_spec.yaml and pass gap_analysis vlm_bcq -e <spec>
ValueError: must be a JSON arrayPredictions file is not a listWrap predictions in [...]
ValueError: missing 'gt'/'response'/'video_id'A prediction item is missing a required fieldInspect and fix the predictions JSON
Samples silently skippedresponse or gt contains both or neither 'yes'/'no'Check logs for warnings; inspect those samples

Files

skill-card.md

Description: <br>

Extract false-positive and false-negative gaps from VLM binary-classification-question (BCQ, yes/no) predictions. <br>

This skill is ready for commercial/non-commercial use. <br>

Owner

NVIDIA <br>

License/Terms of Use: <br>

Apache 2.0 <br>

Use Case: <br>

Developers and engineers use this skill to compare VLM yes/no predictions against ground truth and extract FP/FN failure cases that downstream DEFT root-cause analysis stages consume on a binary-classification VLM workflow. <br>

Deployment Geography for Use: <br>

Global <br>

Requirements / Dependencies: <br>

Requires API Key or External Credential: [Not Specified] <br> Credential Type(s): [None identified] <br>

Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate. <br>

Known Risks and Mitigations: <br>

Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. <br> Mitigation: Review and scan skill before deployment. <br>

Reference(s): <br>

Skill Output: <br>

Output Type(s): [Analysis, Files, Shell commands] <br> Output Format: [JSONL gap records (kpi_gaps.jsonl) and plain-text summary report (kpi_gaps_report.txt)] <br> Output Parameters: [1D] <br> Other Properties Related to Output: [No files are written when no gaps are found; samples with ambiguous yes/no responses or ground truth are skipped with a warning.] <br>

Evaluation Agents Used: <br>

  • Claude Code (aws/anthropic/bedrock-claude-opus-4-8) <br>

  • Codex (openai/openai/gpt-5.5) <br>

Evaluation Tasks: <br>

1 evaluation task (1 positive), 1 attempt per task, each run in an isolated sandbox pod in a k8s-sandbox environment (evaluator version 1.5.6, evaluated 2026-10-16). <br>

Evaluation Metrics Used: <br>

Reported benchmark dimensions: <br>

  • Security: Is it safe to use? Scored from the security signal. <br>

  • Correctness: Is the answer correct? Scored from the accuracy signal. <br>

  • Discoverability: Was the right skill loaded when needed? Scored from the skill_execution signal. <br>

  • Effectiveness: Did the skill help complete the task? Equal-weight mean of goal_accuracy and behavior_check. <br>

  • Efficiency: Did it avoid wasted tool calls and token usage? 50% skill_efficiency plus 50% token_efficiency. <br>

Underlying evaluation signals used in this run: <br>

  • security: Unsafe operations, secret leakage, and unauthorized access. <br>

  • skill_execution: Whether the expected skill was selected, decoys were avoided, and the workflow executed. <br>

  • skill_efficiency: Tool-call productivity (legacy wire id; routing is scored under Discoverability). <br>

  • accuracy: Final-answer correctness against the reference answer. <br>

  • goal_accuracy: Whether the user's goal was achieved. <br>

  • behavior_check: Whether the expected workflow behavior was followed. <br>

  • token_efficiency: Actual uncached prompt plus completion usage (50% of Efficiency). <br>

Evaluation Results: <br>

MeasureClaude Code (Baseline → Skill Uplift)Codex (Baseline → Skill Uplift)
Overall94.6% — baseline ran, but no comparable score was available; uplift unavailable95.4% — baseline ran, but no comparable score was available; uplift unavailable
Security100.0% → 100.0% (±0.0 points)100.0% → 100.0% (±0.0 points)
Correctness0.0% → 100.0% (+100.0 points)40.0% → 100.0% (+60.0 points)
Discoverability100.0% — baseline ran, but no comparable score was available; uplift unavailable95.0% — baseline ran, but no comparable score was available; uplift unavailable
Effectiveness0.0% → 100.0% (+100.0 points)26.7% → 83.3% (+56.6 points)
Efficiency73.1% — baseline ran, but no comparable score was available; uplift unavailable98.7% — baseline ran, but no comparable score was available; uplift unavailable

Skill Version(s): <br>

0.1.0 (source: frontmatter) <br>

Ethical Considerations: <br>

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>

(For Release on NVIDIA Platforms Only) <br> Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns here. <br>

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Declares allowed tools, Needs Python

The author says: Requires Docker, NVIDIA Container Toolkit, and one visible GPU.

Security

What our scanner noticed· Medium risk

  • Large base64-encoded blobs
  • Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
NVIDIA Corporation
Publisher
NVIDIA (Official)
Source
NVIDIA/skills · skills/tao-analyze-gaps-vlm-bcq · commit fce3cca
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE-APACHE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by NVIDIA Corporation or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download tao-analyze-gaps-vlm-bcq.zip