Generated banner for Tao Generate Video Reasoning Annotations
Generated logo for Tao Generate Video Reasoning Annotations

Tao Generate Video Reasoning Annotations

NVIDIAOfficialCreated by NVIDIA Corporation

  • Skill
  • Workflow
  • Data and AI
  • Apache-2.0
  • Medium risk
  • Has scripts

From the author

Multi-step video annotation pipeline that turns raw videos into Chain-of-Thought training data — multi-level captions, structured descriptions, and QA pairs (MCQ, binary, open-ended) with reasoning traces, via VLM/LLM distillation. Use when the user wants to "create video training data", "generate video QA datasets", "build CoT reasoning traces from videos", "auto-label videos", or run the video_reasoning_annotation pipeline. Triggers include "video annotation", "video CoT", "video QA", "chain-of-thought", "video captioning pipeline", "video distillation".
— the description by NVIDIA Corporation

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The tao-generate-video-reasoning-annotations skill provides a structured pipeline to transform raw video files into annotated datasets suitable for training video understanding models. It utilizes Vision-Language Models (VLMs) for captioning and Large Language Models (LLMs) for synthesizing descriptions and generating various QA pairs, including MCQ, binary, and open-ended questions with reasoning traces. The pipeline is designed to be domain-agnostic, supporting custom prompt modules for specialized use cases like traffic or warehouse monitoring, and includes features for anomaly detection and resume-capable processing.

What it does

  • Filters and classifies videos as normal or anomalous.
  • Generates global, dense, and chunked video captions using VLM distillation.
  • Synthesizes captions into structured narratives using LLMs.
  • Produces diverse QA pairs with step-by-step reasoning traces.
  • Parses outputs into standardized JSON files following the tao-vl-reason-v1.0 format.
  • Supports resume-on-failure to skip already-processed videos.

When to use it

  • When you need to create training data for video understanding models.
  • When you want to generate Chain-of-Thought reasoning traces from video content.
  • When you need to auto-label large collections of video files.
  • When you are working with specific domains like traffic or industrial warehouse monitoring.

Advantages

  • Modular pipeline allows for selecting specific processing steps.
  • Built-in support for domain adaptation through custom prompt modules.
  • Resume capability ensures efficiency during long-running annotation tasks.
  • Supports both Gemini and OpenAI-compatible VLM/LLM backends.
  • Provides structured output compatible with downstream training workflows.

Usage examples

“Run the video annotation pipeline on my traffic footage.”
The agent will guide you through selecting the traffic prompt module, configuring your VLM/LLM endpoints, and executing the pipeline on your video directory.
“How do I create a custom dataset for my warehouse security videos?”
The agent will initiate the domain adaptation workshop to help you define your annotation goals and customize the prompt templates for warehouse-specific events.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/nvidia/tao-generate-video-reasoning-annotations/download
  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip tao-generate-video-reasoning-annotations.zip -d .agents/skills

Documentation by NVIDIA Corporation

Video Reasoning Annotation Pipeline

Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).

Generate Chain-of-Thought training datasets from videos by producing multi-level captions, structured descriptions, and QA pairs (MCQ, binary, open-ended) with step-by-step reasoning traces. Domain-agnostic by default — customize prompts for any video domain.

Purpose

Transform raw videos into CoT Q&A training data for video understanding models. VLMs (e.g., Gemini, Qwen) act as "teacher" annotators: Steps 0–1 require the model to see the video (VLM calls); Steps 2–3 are text-to-text (cheaper LLM calls).

Pipeline architecture

Step 0:  [Optional] Filter & classify videos  → Keep domain-relevant, classify anomaly vs normal
Step 1a: Global + dense captions               → VLM: narrative summary + timestamped events
Step 1b: Chunk captions                         → VLM: fixed-duration segment micro-captions
Step 1c: [Optional, anomaly only] Highlight     → LLM extracts anomaly timestamp, VLM captions clip
Step 2:  Description synthesis                  → LLM: synthesize captions into structured narrative
Step 3:  QA generation                          → LLM: MCQ, binary, open-ended with reasoning
Step 4:  Parse outputs                          → Per-task `tao-vl-reason-v1.0` JSON files

Steps are individually selectable via workflow.steps. The pipeline has built-in resume — each step skips already-processed videos, so re-running after a prompt tweak is safe.

Initial consultation

When the user invokes this skill, walk through these questions in order. Don't skip — getting domain and VLM access right up front prevents wasted runs.

1. Videos
  • Path to the video directory and/or a JSONL with {"video_path": "..."} per line.

  • Confirm format (.mp4 preferred; .avi, .mov, .mkv also walked).

2. Domain — drives prompt selection

Ask the user: "What domain are these videos from?" Choose one of the following branches:

DomainWhat to do
generalUse the default prompts. Set prompts_module: "" (or omit). The built-in nvidia_tao_ds.auto_label.video_reasoning_annotation.prompts covers domain-agnostic content.
traffic (CCTV intersections, highways; dashcam excluded)Use the reference module. Set prompts_module: "nvidia_tao_ds.auto_label.video_reasoning_annotation.prompts_traffic", or copy references/prompts_traffic.py into the user's project and tune for their specific camera angles, then point prompts_module at the copy.
warehouse (industrial site CCTV — safety, operations, security)Same pattern. Set prompts_module: "nvidia_tao_ds.auto_label.video_reasoning_annotation.prompts_warehouse", or copy references/prompts_warehouse.py and tune.
custom (any other domain)Run the workshop in references/domain_adaptation.md. It walks through: Phase 1 — question types the user wants the model to answer; Phase 2 — caption-requirements checklist; Phase 3 — fill the [PLACEHOLDER] markers in nvidia_tao_ds.auto_label.video_reasoning_annotation.prompt_template. The two reference modules above are working examples to model after. Do this before any pipeline runs.
3. Anomaly / normal / mixed
  • Mixed dataset → workflow.mode: "auto" (Step 0 classifies each video).

  • Pre-split anomaly only → workflow.mode: "anomaly", drop Step 0.

  • Pre-split normal only → workflow.mode: "normal", drop Steps 0 and 1c.

4. VLM / LLM endpoint — confirm access before running
  • Gemini (default for both vlm.backend and llm.backend): user needs GOOGLE_API_KEY set, or to put the key in the YAML.

  • OpenAI-compatible (Qwen via vLLM, NIM endpoint, etc.): user provides base_url, model_name, and api_key.

  • Steps 2–3 are text-only — a smaller/cheaper LLM is fine for llm.backend even when vlm.backend is a frontier video model.

If the user has no endpoint at all and wants to self-host, point them at the skills/applications/tao-run-inference-service skill — a workflow that stands up a network-specific TAO inference microservice locally and exposes an OpenAI-compatible endpoint. Should support Cosmos, Qwen, and Gemma. Check skills/applications/tao-run-inference-service/references/service.yaml for the current valid_network_arch_config_basenames list before relying on a specific model.

If the user doesn't have endpoint access ready and isn't ready to set one up, stop here and help them figure it out first.

5. Pilot vs full run
  • Recommend a 5–10 video pilot when domain is custom, when any prompt was edited, or when this is the user's first run.

  • Full-run is fine for general / traffic / warehouse once the user has previously verified output quality on the same data type.

  • The pipeline has built-in resume, so a pilot followed by a full run does not re-process the pilot videos.

Quick start

The pipeline runs inside the TAO Toolkit container via the auto_label CLI:

auto_label generate -e /path/to/spec.yaml \
    results_dir=/results \
    video_reasoning_annotation.data.video_root=/videos \
    video_reasoning_annotation.vlm.gemini.api_key=$GOOGLE_API_KEY \
    video_reasoning_annotation.workflow.mode=auto

Generate a default spec to start from:

auto_label default_specs results_dir=/results module_name=auto_label
# then set:  autolabel_type: "video_reasoning_annotation"

All fields support Hydra dot-notation overrides on the command line. For the full YAML reference (every field, model/endpoint setup, error patterns), see references/configuration.md.

Pilot workflow

Use this when running a 5–10 video pilot:

  1. Run the pipeline on the pilot subset with the chosen prompts_module and workflow.mode.

  2. Inspect results_dir/step_1a_caption/captions.jsonl — captions accurate, capturing the right level of detail?

  3. Inspect results_dir/step_3_qa/qa_output.jsonl — questions meaningful, answers correct, reasoning logical?

  4. If quality is insufficient: adjust the prompts (in prompts_module if domain-customized, or fall back to general if a domain module is over-tuned), and re-run. The pipeline auto-skips already-processed videos.

  5. Once satisfied, scale to the full dataset by pointing data.video_root (or data.input_jsonl_files) at the full set and re-running with the same results_dir (resume) or a fresh one (full re-run).

Quality compounds downstream — bad captions produce bad descriptions which produce bad QA. Focus iteration on Step 1a/1b output first; descriptions and QA usually improve once captions are right.

Configuration summary

Key fields (full reference in references/configuration.md):

FieldDefaultDescription
workflow.steps["0","1a","1b","1c","2","3","4"]Which pipeline steps to execute
workflow.mode"auto""auto", "anomaly", or "normal"
vlm.backend"gemini""gemini" or "openai" (OpenAI-compatible)
llm.backend"gemini"Same options; text-only, cheaper model works
workflow.max_workers4Parallel threads per step (watch API rate limits)
license""Optional: written to metadata.license in step 4 outputs (e.g. "CC-BY-4.0")
description_extra""Optional: extra text appended to per-task descriptions in step 4 metadata
prompts_module""Dotted import path to custom prompts module

Prompts

  • Built-in (general): nvidia_tao_ds.auto_label.video_reasoning_annotation.prompts — domain-agnostic, used by default.

  • Template: nvidia_tao_ds.auto_label.video_reasoning_annotation.prompt_template — same 26 keys with [PLACEHOLDER] markers for domain customization.

  • Reference modules (working examples for the consultation's traffic / warehouse branches): references/prompts_traffic.py, references/prompts_warehouse.py.

  • Custom domains: see references/domain_adaptation.md for the full workshop and placeholder reference.

Inputs

  • video_root: Directory of videos (walked recursively for .mp4, .avi, .mov, .mkv).

  • input_jsonl_files: List of JSONL files with {"video_path": "..."} per line. The video key is also accepted; extra fields are allowed.

  • filter_field: Optional boolean field to filter JSONL entries.

Provide video_root, input_jsonl_files, or both (lists merge).

Outputs

All outputs go to results_dir/ with per-step subdirectories (step_0_filter/, step_1a_caption/, …, step_4_output/):

  • Steps 0–3: JSONL — one JSON object per video per line.

  • Step 4: One <task>.json per non-empty task type, in the tao-vl-reason-v1.0 envelope. Up to 10 files: mcq.json, mcq_openended.json, bcq.json, bcq_openended.json, open_qa.json, causal_linkage.json, temporal_localization.json, temporal_description.json, scene_description.json, video_summarization.json.

Each step 4 file looks like:

{
  "format": "tao-vl-reason-v1.0",
  "metadata": {"type": "annotation", "task": "<task>", "date": "YYYY-MM-DD",
               "description": "<per-task + description_extra>", "license": "<from config>"},
  "media_root": "<data.video_root>" | null,
  "items": [{"video_id": "...", "question": "...", "answer": "...", "reasoning": "..."}, ...]
}

media_root mirrors data.video_root (or null when unset); each item's video_id is the entry's video path with the video_root prefix stripped. Set license and description_extra in the spec to populate the metadata.

Prerequisites

  • Container: nvcr.io/nvidia/tao/tao-toolkit:7.3.0-pyt. <!-- versions-key: images.tao_toolkit.pyt -->

  • ffmpeg / ffprobe: required for chunk captioning (Step 1b) and highlight extraction (Step 1c).

  • VLM endpoint: at least one — Gemini API key or OpenAI-compatible endpoint.

Files

references/prompts_traffic.py

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""Traffic CCTV domain prompts for the video CoT annotation pipeline.

Reference prompts for traffic intersection and highway surveillance footage.
Covers anomaly types: collisions, near-misses, stalled vehicles, and traffic
rule violations (red-light running, illegal turns, wrong-way driving).

USAGE:
    Set ``prompts_module`` in your video_reasoning_annotation YAML config to the dotted import
    path of this file after copying it into your project:
        video_reasoning_annotation:
          prompts_module: "my_package.prompts_traffic"

    Or copy and modify — this file is a starting point, not a final product.
    Tune the prompts based on your specific camera angles, intersection layouts,
    and annotation goals.
"""

PROMPT_TEMPLATES = {

    # =========================================================================
    # VIDEO FILTERING
    # =========================================================================
    "video_filtering": (
        "Determine if this video is traffic CCTV footage.\n\n"
        "A traffic CCTV video should have:\n"
        "- A fixed overhead or elevated camera angle viewing a road, "
        "intersection, or highway\n"
        "- Visible vehicles (cars, trucks, motorcycles, buses) using the "
        "roadway\n"
        "- Road infrastructure (lane markings, traffic signals, signs, "
        "crosswalks, medians)\n\n"
        "Negative examples (NOT traffic CCTV):\n"
        "- Dashcam or in-vehicle POV footage\n"
        "- Indoor surveillance (warehouses, offices, retail stores)\n"
        "- Footage of parking lots with no active traffic flow\n"
        "- Drone footage with rapidly changing altitude or angle\n\n"
        "Answer with ONLY \"Yes\" or \"No\"."
    ),

    # =========================================================================
    # ANOMALY CLASSIFICATION
    # =========================================================================
    "video_anomaly_classification": (
        "You are an expert traffic analyst. Watch this video and classify "
        "whether it contains an anomalous event or only normal traffic "
        "activity.\n\n"
        "An anomaly in traffic CCTV is any event involving a collision, "
        "near-miss, stalled or disabled vehicle, or traffic rule violation "
        "(running red lights, illegal turns, wrong-way driving, failure to "
        "yield).\n\n"
        "Examples of anomalies:\n"
        "- Vehicle runs a red light and collides with cross-traffic\n"
        "- Vehicle stalls or breaks down in a travel lane, creating a hazard\n"
        "- Vehicle makes an illegal U-turn or drives the wrong way\n"
        "- Near-miss where vehicles swerve or brake hard to avoid collision\n\n"
        "Examples of normal activity:\n"
        "- Vehicles waiting at a red light, then proceeding on green\n"
        "- Steady traffic flow with lane discipline and safe following "
        "distances\n"
        "- Pedestrians crossing at marked crosswalks during walk signals\n\n"
        "Near-misses count as anomalies even without contact. Minor "
        "violations count even without consequences. If ambiguous, default "
        "to Normal.\n\n"
        "First, briefly describe what you observe in the video (2-3 "
        "sentences). Then explain your reasoning for the classification "
        "(2-3 sentences). Finally, on the LAST line, write your "
        "classification as exactly one word: \"Anomaly\" or \"Normal\"."
    ),

    # =========================================================================
    # ANOMALY CAPTIONING
    # =========================================================================
    "anomaly_global_caption": (
        "You are an expert traffic analyst. Watch the video carefully.\n\n"
        "Describe this video in detail, addressing the following points in a "
        "cohesive narrative:\n"
        "- **Traffic Signal State and Timing**: What is the current signal "
        "phase (red/green/yellow/flashing)? Which directions have "
        "right-of-way? Do signal transitions occur during the video?\n"
        "- **Vehicle Movements and Positions**: For each distinguishable "
        "vehicle, describe: color, type (sedan/SUV/truck/motorcycle/bus), "
        "direction of travel, lane position, and speed "
        "(stopped/slow/moderate/fast). Note turns, lane changes, and "
        "approach to the intersection.\n"
        "- **The Collision / Incident**: Describe the anomalous event in "
        "detail. What exactly happened? Identify all vehicles involved and "
        "describe their specific actions leading up to, during, and "
        "immediately after the event. Where was the point of impact or "
        "violation?\n"
        "- **Cause and Right-of-Way Analysis**: Which vehicle had "
        "right-of-way? What traffic rule was violated? What was the "
        "apparent root cause? What were the immediate consequences "
        "(damage, disruption, other vehicles' reactions)?\n\n"
        "Provide a holistic summary that captures the complete story and "
        "context. Pay close attention to temporal and spatial details. "
        "Do not hallucinate or make up information."
    ),

    "anomaly_dense_caption": (
        "You are an expert traffic analyst. Provide a dense, event-level "
        "video caption with precise timestamps for a video containing a "
        "traffic anomaly.\n\n"
        "Structure your response as a list of events. For each event, "
        "provide the start and end timestamp in the format "
        "<HH:MM:SS><HH:MM:SS> followed by a detailed description.\n\n"
        "Focus particularly on:\n"
        "1. **Precise Timing**: Accurately capture the start and end of "
        "each distinct action or state change.\n"
        "2. **Anomalous Actions**: Describe in detail the actions of "
        "vehicles involved in the anomaly \u2014 approach speeds, signal "
        "compliance, evasive maneuvers, point of impact, post-collision "
        "movements.\n"
        "3. **Vehicle Identification**: For every key vehicle, specify "
        "color, type (sedan/SUV/truck/motorcycle/bus), lane, and direction "
        "of travel. Track vehicles through the scene.\n"
        "4. **Intersection Layout**: Note lane configuration, signal "
        "positions, turn lanes, crosswalks, medians, and which approaches "
        "are visible.\n"
        "5. **Environmental Factors**: Mention lighting (day/night/dusk), "
        "weather (clear/rain/fog), road surface condition, and visibility "
        "when relevant.\n\n"
        "Format each line as:\n"
        "<Start_Timestamp><End_Timestamp> Description of the event.\n\n"
        "Example:\n"
        "<00:00:00><00:00:03> A four-way signalized intersection viewed "
        "from an overhead camera; eastbound traffic has a green signal and "
        "vehicles are flowing through normally.\n"
        "<00:00:03><00:00:05> A dark sedan approaches from the south at "
        "moderate speed in the left lane; the southbound signal is red.\n"
        "<00:00:05><00:00:07> The dark sedan enters the intersection "
        "against the red signal; a white SUV traveling eastbound on green "
        "is in the collision path.\n"
        "<00:00:07><00:00:12> The sedan strikes the SUV on its passenger "
        "side at the center of the intersection. Both vehicles rotate and "
        "come to rest. Debris scatters across the eastbound lanes. "
        "Surrounding vehicles brake and stop."
    ),

    "anomaly_chunk_caption": (
        "You are an expert traffic analyst. You are viewing a strict "
        "{chunk_duration}-second window of a video that contains a traffic "
        "anomaly somewhere in the full video.\n"
        "Your job is NOT to tell the whole story, but to capture the "
        "**micro-dynamics** and **state changes** within this specific "
        "timeframe.\n\n"
        "Note: This specific chunk may or may not contain the anomaly \u2014 "
        "it covers only a fixed time window. Describe only what you observe "
        "in this clip; do not assume or fabricate anomalous content if it "
        "is not visible.\n\n"
        "Please describe the events in this {chunk_duration}-second window, "
        "focusing on:\n\n"
        "1. **Vehicle Dynamics**:\n"
        "   - Acceleration, deceleration, braking, lane changes, turns\n"
        "   - Approach to intersection, signal compliance, yielding\n"
        "   - For each key vehicle: color, type, lane, direction, speed\n\n"
        "2. **Traffic Signal and Infrastructure**:\n"
        "   - Current signal phase and any transitions\n"
        "   - Lane markings, turn arrows, crosswalk signals\n"
        "   - Visibility conditions (lighting, weather, obstructions)\n\n"
        "3. **Anomaly or Interaction Details**:\n"
        "   - If a collision, near-miss, or violation occurs in this chunk, "
        "describe the sequence precisely: approach, contact/violation, "
        "immediate aftermath.\n"
        "   - If the aftermath is visible (stopped vehicles, debris, "
        "emergency response), describe the post-event state.\n"
        "   - If no anomaly is visible, describe the normal traffic flow.\n\n"
        "Do not speculate on what happened before or after this clip. "
        "Report only what is visually confirmed in these "
        "{chunk_duration} seconds. Be precise and objective."
    ),

    # =========================================================================
    # NORMAL CAPTIONING
    # =========================================================================
    "normal_global_caption": (
        "You are an expert traffic analyst. Watch the video carefully.\n\n"
        "Describe this video in detail, addressing the following points in a "
        "cohesive narrative:\n"
        "- **Traffic Signal State and Timing**: What is the current signal "
        "phase? Which directions have right-of-way? Do signal transitions "
        "occur during the video?\n"
        "- **Vehicle Movements and Positions**: For each distinguishable "
        "vehicle, describe: color, type, direction of travel, lane position, "
        "and speed. Note turns, lane changes, and signal compliance.\n"
        "- **Activities and Traffic Flow**: Describe the overall traffic "
        "pattern \u2014 flow direction, density (light/moderate/heavy), "
        "pedestrian activity, and any notable behaviors.\n"
        "- **Overall Context**: Summarize the intersection or road segment, "
        "time of day, weather, and general traffic conditions.\n\n"
        "Provide a holistic summary that captures the complete picture. "
        "Pay close attention to temporal and spatial details. "
        "Do not hallucinate or make up information."
    ),

    "normal_dense_caption": (
        "You are an expert traffic analyst. Provide a dense, event-level "
        "video caption with precise timestamps for this video showing normal "
        "traffic activity.\n\n"
        "Structure your response as a list of events. For each event, "
        "provide the start and end timestamp in the format "
        "<HH:MM:SS><HH:MM:SS> followed by a detailed description.\n\n"
        "Focus particularly on:\n"
        "1. **Precise Timing**: Accurately capture the start and end of "
        "each distinct action, movement, or state change.\n"
        "2. **Traffic Behaviors**: Describe vehicle movements \u2014 lane "
        "changes, turns, stops at signals, yielding, merging, pedestrian "
        "crossings.\n"
        "3. **Vehicle Identification**: For every key vehicle, specify "
        "color, type, lane, and direction. Track vehicles through the "
        "scene.\n"
        "4. **Intersection Layout**: Note lane configuration, signal "
        "positions, and spatial relationships.\n\n"
        "Format each line as:\n"
        "<Start_Timestamp><End_Timestamp> Description of the event.\n\n"
        "Example:\n"
        "<00:00:00><00:00:04> A T-intersection viewed from an elevated "
        "angle; the northbound signal is red and three vehicles are "
        "waiting in the left-turn lane.\n"
        "<00:00:04><00:00:08> The signal turns green; the first vehicle "
        "(white sedan) begins a left turn while yielding to oncoming "
        "traffic.\n"
        "<00:00:08><00:00:12> The white sedan completes the turn; the "
        "second vehicle (red pickup) follows, also turning left. A "
        "pedestrian waits at the crosswalk.\n"
        "<00:00:12><00:00:16> The pedestrian walk signal activates; "
        "the pedestrian crosses while remaining vehicles wait."
    ),

    "normal_chunk_caption": (
        "You are an expert traffic analyst. You are viewing a strict "
        "{chunk_duration}-second window of a video showing normal traffic "
        "activity with no anomalous events.\n"
        "Your job is NOT to tell the whole story, but to capture the "
        "**micro-dynamics** and **state changes** within this specific "
        "timeframe.\n\n"
        "Please describe the events in this {chunk_duration}-second window, "
        "focusing on:\n\n"
        "1. **Vehicle Dynamics**:\n"
        "   - Movements: acceleration, braking, turns, lane changes, "
        "merging\n"
        "   - Signal compliance: stopping at red, proceeding on green, "
        "yielding\n"
        "   - For each key vehicle: color, type, lane, direction\n\n"
        "2. **Traffic Signal and Infrastructure**:\n"
        "   - Current signal phase and any transitions\n"
        "   - Pedestrian signals and crosswalk activity\n"
        "   - Visibility conditions\n\n"
        "3. **Traffic Flow Patterns**:\n"
        "   - Density and speed of traffic\n"
        "   - Queuing behavior at signals\n"
        "   - Pedestrian and cyclist interactions with vehicles\n\n"
        "Do not speculate on what happened before or after this clip. "
        "Report only what is visually confirmed in these "
        "{chunk_duration} seconds. Be precise and objective."
    ),

    # =========================================================================
    # HIGHLIGHT CHUNK (ANOMALY ONLY)
    # =========================================================================
    "highlight_timestamp_extraction": (
        "You are an expert traffic analyst. Given the captions below for a "
        "video containing a traffic anomaly, identify the EXACT timestamp "
        "(in seconds) when the primary anomaly occurs \u2014 the critical moment "
        "of impact, violation, or dangerous event itself, not the lead-up "
        "behavior before it and not the aftermath.\n\n"
        "Use the chunk captions to cross-reference and narrow down the "
        "precise moment, since they provide detailed observations for "
        "specific time windows.\n\n"
        "Respond with ONLY a single number representing the timestamp in "
        "seconds (e.g., \"20\" or \"12.5\"). Nothing else.\n\n"
        "[Global Caption]\n{global_caption}\n\n"
        "[Dense Caption]\n{dense_caption}\n\n"
        "[Chunk Captions]\n{chunk_captions_str}"
    ),

    "highlight_chunk_caption": (
        "You are an expert traffic analyst. You are viewing a critical "
        "{duration}-second highlight clip extracted from a longer video, "
        "centered on the moment a traffic anomaly occurs.\n\n"
        "This clip spans from {start_time}s to {end_time}s of the "
        "original video. The anomaly is estimated to occur at "
        "approximately {anomaly_time}s.\n\n"
        "Provide an extremely detailed description of what happens in "
        "this clip, with special attention to:\n\n"
        "1. **Pre-Event State** (first 1\u20133 seconds):\n"
        "   - Exact positions, lanes, and speeds of all key vehicles\n"
        "   - Traffic signal state and right-of-way\n"
        "   - Any last-second actions (braking, accelerating, swerving)\n\n"
        "2. **The Critical Moment** (the collision, violation, or "
        "near-miss):\n"
        "   - Exactly what happens \u2014 point of impact, type of violation, "
        "evasive maneuver\n"
        "   - Which vehicles are involved, their colors, types, and "
        "approach directions\n"
        "   - Which vehicle had right-of-way and which violated the rule\n\n"
        "3. **Immediate Aftermath** (last 1\u20133 seconds):\n"
        "   - Where vehicles end up (spun, stopped, continued driving)\n"
        "   - Debris, damage visible\n"
        "   - Other vehicles' reactions (braking, stopping, changing lanes)\n\n"
        "Be extremely precise about identifying vehicles (color, type, "
        "direction) and who does what. Do NOT hallucinate or assume "
        "actions you cannot confirm from the video."
    ),

    # =========================================================================
    # DESCRIPTION GENERATION (TEXT-TO-TEXT)
    # =========================================================================
    "anomaly_description": (
        "You are an expert traffic analyst. You are provided with captions "
        "for a video containing a traffic anomaly:\n"
        "1. Global Caption: General description of the intersection and "
        "events.\n"
        "2. Event-level Dense Caption: Detailed event sequence with "
        "timestamps.\n"
        "3. Chunk Captions: Detailed captions for small time windows.\n\n"
        "Task: Generate a detailed, organized, and cohesive description "
        "of the video with exactly these 3 parts:\n"
        "1. Holistic Scene Description \u2014 Describe the intersection or road "
        "segment: layout (number of lanes, turn lanes, medians), signal "
        "configuration, lighting, weather, road surface, and general "
        "traffic conditions.\n"
        "2. Temporal and Spatial Localization of Anomaly Events \u2014 List "
        "each key event in chronological order using this exact format "
        "(one event per line, timestamp in MM:SS):\n"
        "<start timestamp MM:SS><end timestamp MM:SS> [where in the "
        "intersection] what happened\n"
        "3. Description of the Anomaly \u2014 Covering:\n"
        "   - Category of anomaly (collision, near-miss, stalled vehicle, "
        "rule violation)\n"
        "   - Detailed description of the event\n"
        "   - Time and duration\n"
        "   - Spatial location within the intersection\n"
        "   - Right-of-way analysis: which vehicle had the right-of-way, "
        "which signal phase was active, what rule was violated\n"
        "   - Root cause (red-light running, failure to yield, distraction, "
        "mechanical failure, etc.)\n"
        "   - Consequences (damage, traffic disruption, secondary hazards)\n\n"
        "Instructions for using input data:\n"
        "- Use the [Global Caption] for general scene/event descriptions.\n"
        "- Use the [Dense Caption] to identify timestamps and temporal "
        "sequence of events.\n"
        "- Use [Chunk Captions] for detailed micro-level observations "
        "prior to, during, and after the anomaly.\n"
        "- If a [Highlight Chunk Caption] is provided, treat it as the "
        "most reliable source for the anomaly moment. Prioritize it over "
        "chunk captions for Parts 2 and 3. If no highlight is provided, "
        "rely on the other captions.\n"
        "- Resolve conflicts by using information most captions agree on.\n\n"
        "Output formatting rules (strictly enforced):\n"
        "- NEVER reference the source of information. Do not write "
        "phrases like \"the global caption states\", \"captions indicate\", "
        "or any similar attribution. Write as if you are directly "
        "observing the video.\n"
        "- Refer to vehicles by their visual attributes (color, type, "
        "direction), not by ID numbers or labels.\n\n"
        "Input Captions:\n"
        "Video Length: {video_length}s\n\n"
        "[Global Caption]\n{global_caption}\n\n"
        "{dense_section}\n\n"
        "[Chunk Captions]\n{chunk_captions_str}\n\n"
        "{highlight_section}"
    ),

    "normal_description": (
        "You are an expert traffic analyst. You are provided with captions "
        "for a video showing normal traffic activity.\n\n"
        "Task: Generate a detailed, organized, and cohesive description "
        "of the video with exactly these 3 parts:\n"
        "1. Holistic Scene Description \u2014 Describe the intersection or road "
        "segment: layout, signal configuration, lighting, weather, and "
        "general traffic conditions.\n"
        "2. Temporal and Spatial Localization of Key Events \u2014 List each "
        "notable event in chronological order using this exact format "
        "(one event per line, timestamp in MM:SS):\n"
        "<start timestamp MM:SS><end timestamp MM:SS> [where in the "
        "intersection] what happened\n"
        "3. Event Description \u2014 An activity summary. Structure this "
        "section as bullet points, each in \"Key: value\" format. Cover "
        "the traffic flow patterns, signal compliance, pedestrian "
        "activity, notable vehicle behaviors, and overall conditions.\n\n"
        "Instructions:\n"
        "- Use the [Global Caption] for general scene and traffic "
        "descriptions.\n"
        "- Use the [Dense Caption] for the timestamped sequence of events.\n"
        "- Use [Chunk Captions] for detailed micro-level observations.\n"
        "- Resolve conflicts by using what most sources agree on.\n"
        "- Do not fabricate events not present in the captions.\n"
        "- NEVER reference the source of information. Write as if you "
        "are directly observing the video.\n\n"
        "Input Captions:\n"
        "Video Length: {video_length}s\n\n"
        "[Global Caption]\n{global_caption}\n\n"
        "{dense_section}\n\n"
        "[Chunk Captions]\n{chunk_captions_str}"
    ),

    # =========================================================================
    # QA GENERATION — MCQ
    # =========================================================================
    "anomaly_mcq": (
        "You are an expert traffic analyst. You are provided with captions "
        "and a description for a video containing a traffic anomaly.\n\n"
        "Task:\n"
        "1. Design a multiple-choice question about the traffic anomaly "
        "that requires perception and reasoning to answer.\n"
        "2. The question should focus on one of: the root cause of the "
        "anomaly, right-of-way violation, direct consequence, sequence of "
        "events, which vehicle was at fault, or the type of violation.\n"
        "3. The question must be unambiguous: phrased so there is exactly "
        "one correct interpretation and one correct answer based on the "
        "video.\n"
        "4. Each wrong option must be clearly incorrect based on the "
        "video, not merely less likely. All options should be specific "
        "and parallel in structure.\n"
        "5. Provide step-by-step reasoning to derive the answer. The "
        "reasoning should proceed from observations (signal state, vehicle "
        "positions, movements) through causal inference to the conclusion.\n"
        "6. Treat all captions as a representation of the video itself. "
        "Don't mention \"Global Caption\", \"Dense Caption\", or "
        "\"Chunk Captions\" \u2014 only mention the video.\n"
        "7. Ground your Question, Answer, and Reasoning on the provided "
        "data. Don't hallucinate or make up information.\n\n"
        "Here is an example of the desired content style and depth. "
        "(Note: this is for illustration only \u2014 your output must be "
        "based on the actual content of the video provided, not this "
        "example.)\n"
        "<example>\n"
        "Multiple-Choice Question: What was the primary cause of the "
        "collision at the intersection?\n"
        "A. The dark sedan ran a red light while cross-traffic had a "
        "green signal\n"
        "B. The white SUV made an illegal left turn across oncoming "
        "traffic\n"
        "C. A pedestrian stepped into the roadway causing the sedan to "
        "swerve\n"
        "D. The traffic signal malfunctioned, showing green in both "
        "directions\n"
        "======\n"
        "Answer: A\n"
        "=====\n"
        "Reasoning:\n"
        "The video shows the dark sedan approaching the intersection at "
        "speed from the east while the signal for eastbound traffic is "
        "clearly red. Meanwhile, the white SUV is proceeding through the "
        "intersection on a green light traveling northbound. The sedan "
        "enters the intersection without stopping, striking the SUV on "
        "its passenger side. The traffic signals are functioning normally "
        "\u2014 the eastbound signal is red while northbound has green. There "
        "is no pedestrian involvement and no signal malfunction visible. "
        "The root cause is the sedan's failure to stop at the red light.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Multiple-Choice Question: [Question]\n"
        "A. [Option A]\n"
        "B. [Option B]\n"
        "C. [Option C]\n"
        "D. [Option D]\n"
        "======\n"
        "Answer: [Answer] (should be a single letter choice)\n"
        "=====\n"
        "Reasoning: [Provide the analysis as coherent paragraphs. "
        "Do not use bullet points or numbered lists.]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Global Caption]\n{global_caption}\n\n"
        "[Dense Caption]\n{dense_caption}\n\n"
        "[Chunk Captions]\n{chunk_captions_str}\n\n"
        "[Detailed Video Description]\n{step_2_output}"
    ),

    "normal_mcq": (
        "You are an expert traffic analyst. You are provided with captions "
        "for a video with normal traffic activity.\n\n"
        "Task:\n"
        "1. Design a multiple-choice question that requires perception "
        "and preferably reasoning to answer.\n"
        "2. The question should focus on: traffic flow patterns, signal "
        "compliance, vehicle behaviors, pedestrian interactions, spatial "
        "relationships, or environmental conditions.\n"
        "3. The question must be unambiguous with exactly one correct "
        "answer. Each wrong option must be clearly incorrect, not "
        "merely less likely. All options should be specific and parallel.\n"
        "4. Provide step-by-step reasoning to derive the answer.\n"
        "5. Treat all captions as a representation of the video itself. "
        "Don't mention \"Global Caption\" or \"Dense Caption\" \u2014 only "
        "mention the video.\n"
        "6. Ground your output on the provided data. Don't hallucinate.\n\n"
        "Here is an example of the desired content style and depth. "
        "(Note: this is for illustration only.)\n"
        "<example>\n"
        "Multiple-Choice Question: What happens when the traffic signal "
        "turns green for northbound traffic?\n"
        "A. The white sedan in the left-turn lane yields to oncoming "
        "traffic before completing its turn\n"
        "B. All northbound vehicles proceed straight through the "
        "intersection simultaneously\n"
        "C. A pedestrian begins crossing against the signal, causing "
        "vehicles to stop\n"
        "D. The northbound vehicles remain stopped due to a vehicle "
        "blocking the intersection\n"
        "======\n"
        "Answer: A\n"
        "=====\n"
        "Reasoning:\n"
        "The video shows the northbound signal transition from red to "
        "green at approximately 00:00:06. In the left-turn lane, a white "
        "sedan activates its turn signal and begins to enter the "
        "intersection but pauses to yield to oncoming southbound traffic. "
        "Once the oncoming lane clears, the sedan completes its left "
        "turn at 00:00:12. The straight-through vehicles proceed, but "
        "not simultaneously \u2014 they stagger naturally. No pedestrian "
        "crosses against the signal, and no vehicle blocks the "
        "intersection.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Multiple-Choice Question: [Question]\n"
        "A. [Option A]\n"
        "B. [Option B]\n"
        "C. [Option C]\n"
        "D. [Option D]\n"
        "======\n"
        "Answer: [Answer] (should be a single letter choice)\n"
        "=====\n"
        "Reasoning: [Provide the analysis as coherent paragraphs. "
        "Do not use bullet points or numbered lists.]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Global Caption]\n{global_caption}\n\n"
        "[Dense Caption]\n{dense_caption}"
    ),

    # =========================================================================
    # QA GENERATION — BINARY
    # =========================================================================
    "anomaly_bcq": (
        "You are an expert traffic analyst. You are provided with captions "
        "and a description for a video containing a traffic anomaly.\n\n"
        "Task:\n"
        "1. Design two binary questions, one with answer \"Yes.\" and "
        "one with answer \"No.\". Don't make the questions too "
        "complicated. They should be straightforward but still require "
        "observing and reasoning about the video carefully.\n"
        "2. For each question, provide a step-by-step reasoning of how "
        "to derive the answer based on the video. Start from observation "
        "of the traffic scene and the anomalous event, then "
        "reason/analyze the question.\n"
        "3. Treat the description as a representation of the video "
        "itself. Generate the Question, Answer, and Reasoning as if you "
        "are looking at the video directly. Don't mention the description "
        "in the Question, Answer, or Reasoning. Only mention the video.\n"
        "4. Ground your output on the provided data. Don't hallucinate.\n\n"
        "Here is an example of the desired content style and depth. "
        "(Note: this is for illustration only \u2014 your output must be "
        "based on the actual content of the video provided, not this "
        "example.)\n"
        "<example>\n"
        "Question: Did the dark sedan have a green light when it entered "
        "the intersection?\n"
        "======\n"
        "Answer: No. The dark sedan entered the intersection against a "
        "red signal while cross-traffic had the green light.\n"
        "=====\n"
        "Reasoning:\n"
        "The video shows a signalized intersection. At the time the dark "
        "sedan enters the intersection (approximately 00:00:06), the "
        "signal for its approach direction is clearly red. Meanwhile, "
        "cross-traffic vehicles are proceeding on green. The sedan does "
        "not stop or slow down for the red signal, entering the "
        "intersection illegally and causing the collision.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "1. Question: [Question]\n"
        "======\n"
        "Answer: Yes. [Additional explanation in one sentence]\n"
        "=====\n"
        "Reasoning: [Provide the analysis as coherent paragraphs. "
        "Do not use bullet points or numbered lists.]\n"
        "=====\n"
        "2. Question: [Question]\n"
        "======\n"
        "Answer: No. [Additional explanation in one sentence]\n"
        "=====\n"
        "Reasoning: [Provide the analysis as coherent paragraphs. "
        "Do not use bullet points or numbered lists.]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Global Caption]\n{global_caption}\n\n"
        "[Dense Caption]\n{dense_caption}\n\n"
        "[Chunk Captions]\n{chunk_captions_str}\n\n"
        "[Detailed Video Description]\n{step_2_output}"
    ),

    "normal_bcq": (
        "You are an expert traffic analyst. You are provided with captions "
        "for a video with normal traffic activity.\n\n"
        "Task:\n"
        "1. Design two binary questions, one with answer \"Yes.\" and "
        "one with answer \"No.\". Don't make the questions too "
        "complicated. They should be straightforward but still require "
        "observing and reasoning about the video carefully. Focus on "
        "signal compliance, vehicle behaviors, pedestrian activity, "
        "or traffic flow patterns.\n"
        "2. For each question, provide a step-by-step reasoning of "
        "how to derive the answer based on the video. Start from "
        "observation and description, then reason/analyze.\n"
        "3. Treat the description as a representation of the video "
        "itself. Don't mention the description. Only mention the video.\n"
        "4. Ground your output on the provided data. Don't hallucinate.\n\n"
        "Here is an example of the desired content style and depth. "
        "(Note: this is for illustration only.)\n"
        "<example>\n"
        "Question: Do all vehicles in the video obey the traffic signals?\n"
        "======\n"
        "Answer: Yes. Every vehicle in the video stops at the red light "
        "and proceeds only after the signal turns green.\n"
        "=====\n"
        "Reasoning:\n"
        "The video shows a standard intersection over one full signal "
        "cycle. When the light is red for northbound traffic, three "
        "vehicles wait behind the stop line. When the signal turns green "
        "at 00:00:08, they proceed in order. Eastbound traffic similarly "
        "stops at its red phase and proceeds on green. No vehicle enters "
        "the intersection against a red signal throughout the video.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "1. Question: [Question]\n"
        "======\n"
        "Answer: Yes. [Additional explanation in one sentence]\n"
        "=====\n"
        "Reasoning: [Provide the analysis as coherent paragraphs. "
        "Do not use bullet points or numbered lists.]\n"
        "=====\n"
        "2. Question: [Question]\n"
        "======\n"
        "Answer: No. [Additional explanation in one sentence]\n"
        "=====\n"
        "Reasoning: [Provide the analysis as coherent paragraphs. "
        "Do not use bullet points or numbered lists.]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Global Caption]\n{global_caption}\n\n"
        "[Dense Caption]\n{dense_caption}"
    ),

    # =========================================================================
    # QA GENERATION — OPEN-ENDED
    # =========================================================================
    "anomaly_open_qa": (
        "You are an expert traffic analyst. You are provided with captions "
        "and a description for a video containing a traffic anomaly.\n\n"
        "Task:\n"
        "1. Design an open-ended question about the traffic anomaly that "
        "requires perception and reasoning to answer.\n"
        "2. The question should ask about the root cause, right-of-way "
        "analysis, consequence, sequence of events, or what should have "
        "been done differently to prevent the incident.\n"
        "3. Provide step-by-step reasoning to derive the answer. The "
        "reasoning should proceed from observations (signal state, vehicle "
        "positions, movements) through causal inference to the conclusion.\n"
        "4. Treat all captions as a representation of the video itself. "
        "Don't mention \"Global Caption\", \"Dense Caption\", or "
        "\"Chunk Captions\" \u2014 only mention the video.\n"
        "5. Ground your output on the provided data. Don't hallucinate.\n\n"
        "Here is an example of the desired content style and depth. "
        "(Note: this is for illustration only \u2014 your output must be "
        "based on the actual content of the video provided, not this "
        "example.)\n"
        "<example>\n"
        "Open-ended Question: What is the root cause of the collision "
        "and how could it have been prevented?\n"
        "======\n"
        "Answer: The root cause of the collision is the dark sedan "
        "running a red light at approximately 00:00:06 while cross-traffic "
        "had a green signal and right-of-way. The sedan failed to stop or "
        "even decelerate as it approached the intersection, entering at "
        "speed and striking the white SUV that was lawfully proceeding "
        "through on green. The collision could have been prevented if "
        "the sedan driver had obeyed the red signal and stopped behind "
        "the stop line, as the signal was clearly visible with adequate "
        "approach distance.\n"
        "=====\n"
        "Reasoning:\n"
        "The video shows a standard signalized intersection. Before the "
        "collision, the eastbound signal is red while the northbound "
        "signal is green. The white SUV begins moving northbound on the "
        "fresh green signal. Simultaneously, the dark sedan approaches "
        "from the east at moderate-to-high speed without any sign of "
        "braking or deceleration. The sedan enters the intersection "
        "directly against the red signal and strikes the SUV on the "
        "passenger side.\n\n"
        "The root cause is the sedan's failure to comply with the red "
        "signal. The signal was clearly visible, the road was straight "
        "with good visibility, and there were no obstructions. The sedan "
        "had sufficient distance to stop. The collision was entirely "
        "preventable through basic signal compliance.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Open-ended Question: [Question]\n"
        "======\n"
        "Answer: [Answer] (open-ended, should be a paragraph)\n"
        "=====\n"
        "Reasoning: [Provide the analysis as coherent paragraphs. "
        "Do not use bullet points or numbered lists.]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Global Caption]\n{global_caption}\n\n"
        "[Dense Caption]\n{dense_caption}\n\n"
        "[Chunk Captions]\n{chunk_captions_str}\n\n"
        "[Detailed Video Description]\n{step_2_output}"
    ),

    "normal_open_qa": (
        "You are an expert traffic analyst. You are provided with captions "
        "for a video with normal traffic activity.\n\n"
        "Task:\n"
        "1. Design an open-ended question that requires perception and "
        "preferably reasoning to answer.\n"
        "2. The question should ask about traffic flow patterns, signal "
        "timing effects, how vehicles interact at the intersection, "
        "pedestrian behavior, or what the observable patterns reveal "
        "about the traffic conditions.\n"
        "3. Provide step-by-step reasoning to derive the answer.\n"
        "4. Treat all captions as a representation of the video itself. "
        "Don't mention \"Global Caption\" or \"Dense Caption\" \u2014 only "
        "mention the video.\n"
        "5. Ground your output on the provided data. Don't hallucinate.\n\n"
        "Here is an example of the desired content style and depth. "
        "(Note: this is for illustration only.)\n"
        "<example>\n"
        "Open-ended Question: How does the traffic signal cycle affect "
        "the flow of vehicles through the intersection in this video?\n"
        "======\n"
        "Answer: The signal cycle creates a clear pattern of queuing and "
        "release for each direction. When the northbound signal is red, "
        "vehicles accumulate behind the stop line, forming a queue of "
        "three to four vehicles. When the signal turns green, the queue "
        "clears within approximately 10 seconds as vehicles proceed in "
        "order. During the northbound green phase, eastbound traffic is "
        "held at red, accumulating its own queue. This alternating "
        "pattern ensures orderly flow without conflicts, with each "
        "direction getting approximately equal green time.\n"
        "=====\n"
        "Reasoning:\n"
        "The video covers one full signal cycle at the intersection. "
        "Starting with the northbound red phase, we observe three "
        "vehicles queued. At 00:00:08, the signal turns green and the "
        "queue begins to clear \u2014 the first vehicle moves at 00:00:09, "
        "followed by the second at 00:00:11, and the third at 00:00:13. "
        "By 00:00:18, the intersection is clear. Meanwhile, eastbound "
        "vehicles have been accumulating at their red signal, with four "
        "vehicles visible in the queue by the time their signal turns "
        "green. This demonstrates a balanced signal cycle managing "
        "moderate traffic volume effectively.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Open-ended Question: [Question]\n"
        "======\n"
        "Answer: [Answer] (open-ended, should be a paragraph)\n"
        "=====\n"
        "Reasoning: [Provide the analysis as coherent paragraphs. "
        "Do not use bullet points or numbered lists.]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Global Caption]\n{global_caption}\n\n"
        "[Dense Caption]\n{dense_caption}"
    ),

    "scene_description": (
        "You are an expert traffic safety analyst. You are provided with "
        "a holistic and detailed description for a traffic video. Your "
        "task is to produce a **scene description caption** that captures "
        "the static, spatial layout of the scene — not the events that "
        "unfold over time.\n\n"
        "Task:\n"
        "1. Write a **question** — a natural-language question whose "
        "answer is the scene description caption. The question should ask "
        "about the physical setting, road layout, or environmental "
        "context of the video (e.g., \"What does the scene look like in "
        "this traffic video?\" or \"Describe the road layout and "
        "surroundings visible in the video.\").\n"
        "2. Write an **answer** — a concise yet thorough scene "
        "description covering:\n"
        "   - The camera perspective (overhead, dashboard, fixed CCTV, "
        "etc.).\n"
        "   - Weather, lighting, and approximate time of day.\n"
        "   - Road and lane layout: number of lanes, lane directions (use "
        "frame-relative terms such as \"top to bottom\", \"left to right\"), "
        "median/divider presence, road markings, and lane types "
        "(bus-only, turn-only, shoulder, etc.).\n"
        "   - Traffic infrastructure: traffic signals, signboards, street "
        "lights, crosswalks, sidewalks.\n"
        "   - Stationary objects and environment: buildings, trees, "
        "fences, parked vehicles, text overlays.\n"
        "   - People/objects present and their attributes (color, type, "
        "size) when discernible.\n"
        "   Do NOT describe events, vehicle movements, or collisions — "
        "those belong in the event summary.\n"
        "3. Write a **reasoning** — a coherent paragraph explaining how "
        "observations from the video lead to each element of the answer. "
        "Proceed from what is directly visible to the inferences drawn "
        "(e.g., \"Lane markings and a median divider indicate a divided "
        "four-lane road …\"). Do not use bullet points or numbered lists.\n"
        "4. Treat the description as a representation of the video "
        "itself. Generate the Question, Answer, and Reasoning as if you "
        "are looking at the video directly. Do not mention the "
        "description in the output.\n"
        "5. Ground your output on the video description. Do not "
        "hallucinate or fabricate details.\n\n"
        "Here are examples of the desired answer style and depth. (Note: "
        "these are for illustration only — your output must be based on "
        "the actual content of the video provided, not these examples.)\n"
        "<example>\n"
        "Answer: This is a video recording from an overhead traffic "
        "camera during the daytime, showing a four-lane roadway with one "
        "split lane. Traffic in lanes 1 and 2 flows from top to bottom of "
        "the frame, while traffic in lanes 3 and 4 flows from the bottom "
        "to the top of the frame. The split lane contains stopped "
        "traffic, and one yellow bus moves ahead towards the top of the "
        "frame in the split lane. There is a text overlay on the top "
        "right of the frame that reads \"FOX TV STATIONS\".\n"
        "</example>\n"
        "<example>\n"
        "Answer: The video captures an overhead view from a traffic "
        "signal CCTV camera at a four-way intersection. A white text "
        "overlay shows \"2024-11-04\" with a running timer. Streetlights "
        "are present along the whole intersection, starting from left to "
        "right. Sidewalks are present towards the right and left of the "
        "top lane. Traffic lights are present to the left, right, and top "
        "of the frame. There are three signboards placed on each of the "
        "traffic signal poles, respectively. The road has white markings "
        "showing the intersection clearly. There are trees to the left, "
        "in the top right corner, and in the bottom right corner of the "
        "frame. Distinctly marked road lanes are present, signifying a "
        "signal-regulated junction. There are two houses on the right "
        "side of the frame.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Question: [Question about the scene layout]\n"
        "=====\n"
        "Answer: [Scene description caption]\n"
        "=====\n"
        "Reasoning: [Coherent paragraph deriving each answer element from "
        "video observations]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Detailed Video Description]\n"
        "{step_2_output}"
    ),

    "event_summary": (
        "You are an expert traffic safety analyst. You are provided with "
        "a holistic and detailed description for a traffic video. Your "
        "task is to produce an **event summary caption** that describes "
        "the key events and vehicle actions in the video — not the static "
        "scene layout.\n\n"
        "Task:\n"
        "1. Write a **question** — a natural-language question whose "
        "answer is the event summary caption. The question should ask "
        "about what happens in the video (e.g., \"What are the key events "
        "in this traffic video?\" or \"Summarize the events and any "
        "anomalies observed in the video.\").\n"
        "2. Write an **answer** — a concise yet thorough event summary "
        "covering:\n"
        "   - A brief, high-level summary of all key events in the video.\n"
        "   - For normal traffic: describe the traffic flow, signal "
        "changes, and any notable vehicle maneuvers.\n"
        "   - For anomalous traffic: briefly describe normal traffic, "
        "then describe the anomaly in detail, including the color and "
        "type of the involved vehicles, their driving directions and "
        "lanes, the sequence of events leading to the anomaly, and the "
        "physical aftermath (vehicles stalled, dragged, flipped, etc.).\n"
        "   - **Root cause**: For any anomalous video, explicitly state "
        "the root cause of the incident (e.g., running a red light, "
        "failing to yield, distracted driving).\n"
        "   - Use frame-relative directions (top, bottom, left, right) "
        "and timestamps (MM:SS) where possible.\n"
        "3. Write a **reasoning** — a coherent paragraph explaining the "
        "chain of observations and inferences that support the answer. "
        "Start from what is visible, trace the sequence of events, and "
        "connect them to the stated root cause and outcome. Do not use "
        "bullet points or numbered lists.\n"
        "4. Treat the description as a representation of the video "
        "itself. Generate the Question, Answer, and Reasoning as if you "
        "are looking at the video directly. Do not mention the "
        "description in the output.\n"
        "5. Ground your output on the video description. Do not "
        "hallucinate or fabricate details.\n\n"
        "Here are examples of the desired answer style and depth. (Note: "
        "these are for illustration only — your output must be based on "
        "the actual content of the video provided, not these examples.)\n"
        "<example>\n"
        "Answer: The video shows a blue sedan entering lane 4 at the "
        "bottom of the frame and moving toward the top of the frame. It "
        "collides with a black SUV, which is moving ahead of it. After "
        "the collision, the blue sedan changed lanes to the left and "
        "entered lane 3 without turning on its blinker and became stalled "
        "in lane 3 while the black SUV moved straight in lane 4 toward "
        "the top of the frame and entered the shoulder lane, which is "
        "beside the split lane.\n"
        "</example>\n"
        "<example>\n"
        "Answer: A black sedan enters the frame from the bottom and is "
        "hit on the right side by a rider on a scooter. After the "
        "collision, the black sedan drives ahead towards the right of the "
        "frame, towards a white divider, and the scooter skids, spins, "
        "and moves forward while the rider is flung in the air and lands "
        "near a divider with bushes on it. The bus, cars and motorcycles "
        "behind this collision have come to a halt.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Question: [Question about the events in the video]\n"
        "=====\n"
        "Answer: [Event summary caption]\n"
        "=====\n"
        "Reasoning: [Coherent paragraph tracing the sequence of events "
        "and root cause from video observations]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Detailed Video Description]\n"
        "{step_2_output}"
    ),

    "anomaly_temporal_event_desc": (
        "You are an expert traffic safety analyst and video annotator. "
        "You are provided with a holistic and detailed description for a "
        "traffic video containing an anomaly.\n\n"
        "Task:\n"
        "1. Identify a single anomalous event (e.g., a collision, a "
        "traffic violation, a near-miss) and select [t1, t2] as the tight "
        "temporal window covering just that one event.\n\n"
        "2. Generate a question asking what happened in the video between "
        "[t1] and [t2]. The question must contain ONLY the timestamps and "
        "a neutral phrasing — no vehicles, actions, events, or outcomes.\n\n"
        "3. Provide an Answer as a single coherent paragraph describing "
        "what occurred in that window, including the inferred cause of "
        "the anomaly woven naturally into the description.\n\n"
        "4. Provide a Reasoning trace in 2-3 coherent paragraphs "
        "explaining your observations in addition to what changed and why "
        "during the window itself. Where applicable, include pre-event "
        "context before t1 (the normal baseline state that makes the "
        "anomaly identifiable) and post-event context after t2 (the "
        "resulting state or consequence).\n\n"
        "Constraints:\n"
        "- STRICT ADHERENCE TO TIMESTAMPS: Use only timestamps supported "
        "by the provided captions. Do not hallucinate times.\n"
        "- The Question must contain ONLY the timestamps and a neutral "
        "phrasing — no vehicles, events, outcomes, or causes.\n"
        "- The Answer must be a single coherent paragraph. Do not use "
        "bullet points or numbered lists.\n"
        "- The Reasoning must be coherent paragraphs. Do not use bullet "
        "points or numbered lists.\n"
        "- Treat the description as a representation of the video itself. "
        "Generate the Question, Answer, and Reasoning as if you are "
        "looking at the video directly. Do not mention the description in "
        "the Question, Answer, or Reasoning.\n\n"
        "Here is an example of the desired output:\n"
        "<example>\n"
        "Question: What happened in the video between 00:00:02 and "
        "00:00:06?\n"
        "======\n"
        "Answer: A silver sedan turns left across oncoming traffic "
        "without yielding, causing a T-bone collision with a "
        "straight-moving vehicle. The sedan spins to a halt and a dark "
        "pickup truck is simultaneously struck by a FedEx truck in the "
        "adjacent lane, with all surrounding traffic coming to a full "
        "stop by 00:00:06. The root cause is the sedan driver's failure "
        "to yield, a misjudgment that made the collision unavoidable once "
        "the turn was committed.\n"
        "=====\n"
        "Reasoning:\n"
        "Before 00:00:02, the intersection is operating normally — "
        "oncoming vehicles are flowing straight through with the "
        "right-of-way and the silver sedan is stationary in the left-turn "
        "lane, signaling intent to turn. This baseline of orderly, "
        "rule-compliant traffic is what makes the subsequent deviation "
        "identifiable as an anomaly.\n\n"
        "At 00:00:02, the sedan commits to the left turn without clearing "
        "oncoming traffic, creating an unavoidable conflict. The "
        "straight-moving vehicle has no time to brake, resulting in a "
        "T-bone impact that also displaces the pickup into the path of "
        "the FedEx truck.\n\n"
        "After 00:00:06, all vehicles have come to rest and surrounding "
        "traffic has stopped completely. The intersection, which was "
        "flowing normally moments before, is now fully obstructed. The "
        "root cause is the sedan driver's misjudgment of the available "
        "gap — a failure that made the collision inevitable once the turn "
        "was committed.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Question: [Neutral question with only timestamps]\n"
        "======\n"
        "Answer: [Single paragraph]\n"
        "=====\n"
        "Reasoning: [2-3 coherent paragraphs]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Detailed Video Description]\n"
        "{step_2_output}"
    ),

    "normal_temporal_event_desc": (
        "You are an expert video annotator and traffic analyst. You are "
        "provided with a holistic and detailed description for a normal "
        "traffic video.\n\n"
        "Task:\n"
        "1. Identify a single, self-contained traffic event (e.g., a "
        "vehicle completing a turn, a pedestrian crossing, a lane change, "
        "a signal phase change affecting flow) and select [t1, t2] as the "
        "tight temporal window covering just that one event.\n\n"
        "2. Generate a question asking what happened in the video between "
        "[t1] and [t2]. The question must contain ONLY the timestamps and "
        "a neutral phrasing — no vehicles, actions, events, or outcomes.\n\n"
        "3. Provide an Answer as a single coherent paragraph describing "
        "what occurred in that window and why the events unfolded as they "
        "did (traffic rules, right-of-way, signal phases, vehicle "
        "interactions — woven naturally).\n\n"
        "4. Provide a Reasoning trace in 2-3 coherent paragraphs "
        "explaining your observations in addition to what changed and why "
        "during the window itself. Where applicable, include pre-event "
        "context before t1 (what led to the event or set the conditions "
        "for it) and post-event context after t2 (the resulting state "
        "once the event completed).\n\n"
        "Constraints:\n"
        "- STRICT ADHERENCE TO TIMESTAMPS: Use only timestamps supported "
        "by the provided captions. Do not hallucinate times.\n"
        "- The Question must contain ONLY the timestamps and a neutral "
        "phrasing — no vehicles, events, outcomes, or causes.\n"
        "- The Answer must be a single coherent paragraph. Do not use "
        "bullet points or numbered lists.\n"
        "- The Reasoning must be coherent paragraphs. Do not use bullet "
        "points or numbered lists.\n"
        "- Treat the description as a representation of the video itself. "
        "Generate the Question, Answer, and Reasoning as if you are "
        "looking at the video directly. Do not mention the description in "
        "the Question, Answer, or Reasoning.\n\n"
        "Here is an example of the desired output:\n"
        "<example>\n"
        "Question: What happened in the video between 00:00:08 and "
        "00:00:12?\n"
        "======\n"
        "Answer: A white sedan waiting in the left-turn lane initiates "
        "its turn as a gap opens in oncoming traffic and smoothly "
        "completes the maneuver by 00:00:12. The driver yields until the "
        "last crossing vehicle clears, then commits to the arc and "
        "straightens into the new lane — normal left-turn yielding "
        "behavior executed correctly.\n"
        "=====\n"
        "Reasoning:\n"
        "Before 00:00:08, the white sedan has been stationary in the "
        "left-turn lane with its turn signal active, waiting as a steady "
        "stream of oncoming vehicles passes through the intersection. "
        "This waiting state is the precondition that explains why the "
        "turn begins precisely when it does.\n\n"
        "At 00:00:08, the final oncoming vehicle clears the intersection, "
        "creating a sufficient gap. The sedan enters and follows a "
        "controlled arc across the opposing lanes with no conflicting "
        "traffic, the driver committing only once right-of-way was "
        "available.\n\n"
        "After 00:00:12, the sedan is traveling straight in the new lane "
        "with no further interaction. The turn is fully resolved, and "
        "normal flow resumes — a textbook example of a driver correctly "
        "reading the traffic conditions before acting.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Question: [Neutral question with only timestamps]\n"
        "======\n"
        "Answer: [Single paragraph]\n"
        "=====\n"
        "Reasoning: [2-3 coherent paragraphs]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Detailed Video Description]\n"
        "{step_2_output}"
    ),

    "anomaly_causal_linkage": (
        "You are an expert traffic safety analyst. You are provided with "
        "a holistic and detailed description for a traffic video "
        "containing an anomaly.\n\n"
        "Task:\n"
        "1. Identify two distinct moments:\n"
        "   - [Timestamp A] (Reference Point): The triggering action, "
        "violation, or initial environmental condition.\n"
        "   - [Timestamp B] (Focus Point): The subsequent outcome, final "
        "rest position, or resulting consequence.\n"
        "2. Generate a \"Blind\" Open-ended Question about the traffic "
        "anomaly or any significant event in the video: Ask about the "
        "relationship between the two moments using ONLY their timestamps.\n"
        "   - FORMAT example: \"Explain the relationship between the event "
        "at [Timestamp A] and the situation at [Timestamp B].\"\n"
        "   - CONSTRAINT: Do not mention any specific vehicles, actions, "
        "or outcomes in the question.\n"
        "   - Please design the questions that require perception and "
        "cognition (reasoning) to answer.\n"
        "   - Note: The relationship between the two moments can be "
        "causal (cause/effect), regulatory (rule/violation), or "
        "environmental (condition/reaction).\n"
        "3. Provide a Detailed Answer in a single paragraph:\n"
        "   - Begin by explicitly identifying the specific event "
        "occurring at [Timestamp A] and the specific situation at "
        "[Timestamp B].\n"
        "   - Follow with a narrative explanation of the logical, "
        "physical, or legal connection\n"
        "4. Provide a Reasoning Trace in 2–3 coherent paragraphs:\n"
        "   - Start from observation and description of the video, "
        "identify the events at A and B (what happens at [Timestamp A], "
        "what happens at [Timestamp B]). Describe the 'Normal' baseline "
        "state of the video before [Timestamp A] and identify "
        "environmental clues (traffic lights, signs, road markings) that "
        "define the rules of the scene.\n"
        "   - Trace the \"Logical Bridge\" and relevant events between A "
        "and B. Explain the mechanics (e.g., momentum, reaction time, "
        "obstructed sightlines) that connect the two moments. Highlight "
        "the key events between A and B that are needed to answer the "
        "question.\n"
        "   - Reason through the causal chain, and state the conclusion "
        "that answers the question.\n\n"
        "Constraints:\n"
        "- The question MUST be \"blind\" (refer only to timestamps).\n"
        "- The Answer must be a single, comprehensive narrative paragraph.\n"
        "- The Reasoning must be coherent paragraphs (no bullet points or "
        "lists).\n"
        "- STRICT ADHERENCE TO TIMESTAMPS: Use the exact timestamps from "
        "the provided captions.\n"
        "- Treat the description as a representation of the video itself. "
        "Generate the Question, Answer, and Reasoning as if you are "
        "looking at the video directly. Don't mention the description in "
        "the Question, Answer, or Reasoning. Only mention the video "
        "itself.\n\n"
        "Here is an example of the desired content style and depth:\n"
        "<example>\n"
        "Question: Explain the relationship between the event at 00:00:01 "
        "and the situation at 00:00:05.\n"
        "======\n"
        "Answer: At 00:00:01, a black pickup truck accelerates from "
        "behind a stopped queue to enter the intersection, while at "
        "00:00:05, a dark grey SUV is resting on its roof after a "
        "rollover. The relationship is a direct causal chain where the "
        "pickup truck's high-speed signal violation at 00:00:01 created "
        "an unavoidable T-bone collision with the SUV. This impact "
        "transferred massive kinetic energy to the SUV, causing it to "
        "flip and slide into the far-left corner of the frame, resulting "
        "in the stationary wreckage and roadway obstruction observed at "
        "00:00:05.\n"
        "=====\n"
        "Reasoning:\n"
        "The video at 00:00:01 shows a black pickup truck veering out of "
        "a stationary queue to bypass a red light, while the situation at "
        "00:00:05 shows a post-collision scene with an overturned SUV and "
        "debris. Looking at the wider context of the video, we can see "
        "that a red semi-truck and a white sedan remained stationary at "
        "the off-ramp during this entire window, which serves as an "
        "environmental anchor confirming that the vertical signal phase "
        "was red. This baseline establishes that the pickup's entry at "
        "00:00:01 was a significant departure from safe, predictable "
        "traffic behavior.\n\n"
        "The logical bridge between these timestamps is the physical "
        "impact that occurs at 00:00:02. Because the pickup truck entered "
        "the junction at such a high rate of speed, a \"point of no "
        "return\" was reached where neither driver could perform an "
        "evasive maneuver. The front end of the pickup struck the side of "
        "the SUV, initiating a rollover sequence. Between 00:00:02 and "
        "00:00:05, the SUV's momentum carried it across the asphalt while "
        "it inverted, eventually coming to a stop as its kinetic energy "
        "dissipated.\n\n"
        "We can conclude that the situation at 00:00:05 is the terminal "
        "physical resolution of the violation initiated at 00:00:01. By "
        "tracing the chain of events from the truck's initial "
        "acceleration to the SUV's final resting position, it is clear "
        "that the obstruction at 00:00:05 is the direct result of the "
        "illegal bypass of the traffic queue. The relationship is a "
        "complete causal sequence where the initial decision at 00:00:01 "
        "made the catastrophic outcome at 00:00:05 physically inevitable.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Question: [Question]\n"
        "======\n"
        "Answer: [Answer] (open-ended, should be a paragraph)\n"
        "=====\n"
        "Reasoning: [Provide the analysis as coherent paragraphs. Do not "
        "use bullet points or numbered lists.]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Detailed Video Description]\n"
        "{step_2_output}"
    ),

    "normal_causal_linkage": (
        "You are an expert traffic analyst. You are provided with a "
        "holistic and detailed description for a normal traffic video.\n\n"
        "Task:\n"
        "1. Identify two distinct moments in the video:\n"
        "   - [Timestamp A] (Reference Point): An initial action or "
        "condition (e.g., a traffic signal phase change, a vehicle "
        "beginning a lane change or turn, a pedestrian entering the "
        "crosswalk, a vehicle merging, or the start of a parking "
        "maneuver).\n"
        "   - [Timestamp B] (Focus Point): The subsequent outcome (e.g., "
        "vehicles moving through after the signal change, the vehicle "
        "completing the turn, the pedestrian reaching the other side, "
        "traffic flow resuming).\n"
        "2. Generate a \"Blind\" Open-ended Question about the relationship "
        "between the two moments using ONLY their timestamps:\n"
        "   - FORMAT example: \"Explain the relationship between the event "
        "at [Timestamp A] and the situation at [Timestamp B].\"\n"
        "   - CONSTRAINT: Do not mention any specific vehicles, actions, "
        "or outcomes in the question.\n"
        "   - Design questions that require perception and cognition "
        "(reasoning) to answer.\n"
        "   - The relationship can be causal (cause/effect), sequential "
        "(action/completion), or regulatory (signal/response).\n"
        "3. Provide a Detailed Answer in a single paragraph:\n"
        "   - Begin by explicitly identifying the specific event at "
        "[Timestamp A] and the specific situation at [Timestamp B].\n"
        "   - Follow with a narrative explanation of the logical, "
        "physical, or regulatory connection.\n"
        "4. Provide a Reasoning Trace in 2–3 coherent paragraphs:\n"
        "   - Start from observation and description of the video; "
        "identify the events at A and B (what happens at [Timestamp A], "
        "what happens at [Timestamp B]). Describe the baseline state "
        "before [Timestamp A] and any relevant environmental clues "
        "(traffic lights, signs, road markings).\n"
        "   - Trace the \"Logical Bridge\" and relevant events between A "
        "and B (e.g., gap in traffic, signal phase, vehicle motion). "
        "Highlight the key events between A and B needed to answer the "
        "question.\n"
        "   - Reason through the chain and state the conclusion that "
        "answers the question.\n\n"
        "Constraints:\n"
        "- The question MUST be \"blind\" (refer only to timestamps).\n"
        "- The Answer must be a single, comprehensive narrative paragraph.\n"
        "- The Reasoning must be coherent paragraphs (no bullet points or "
        "lists).\n"
        "- STRICT ADHERENCE TO TIMESTAMPS: Use the exact timestamps from "
        "the provided captions.\n"
        "- Treat the description as a representation of the video itself. "
        "Generate the Question, Answer, and Reasoning as if you are "
        "looking at the video directly. Don't mention the description in "
        "the Question, Answer, or Reasoning. Only mention the video "
        "itself.\n\n"
        "Here is an example of the desired content style and depth:\n"
        "<example>\n"
        "Question: Explain the relationship between the event at 00:00:08 "
        "and the situation at 00:00:12.\n"
        "======\n"
        "Answer: At 00:00:08, a white sedan begins its left turn from the "
        "turn lane after a gap appears in oncoming traffic, and at "
        "00:00:12 the same sedan has fully completed the turn and is "
        "traveling straight in the perpendicular lane. The relationship "
        "is sequential and regulatory: the event at 00:00:08 is the "
        "driver's decision to initiate the turn once it was safe (gap in "
        "traffic, possible signal or yield rule), and the situation at "
        "00:00:12 is the natural completion of that maneuver—the vehicle "
        "has traversed the intersection, straightened its wheels, and "
        "resumed normal flow in the new lane.\n"
        "=====\n"
        "Reasoning:\n"
        "First, from observation of the video: the scene is an "
        "intersection with a left-turn lane. Before 00:00:08, the white "
        "sedan is stationary in that lane with its turn signal on while "
        "oncoming traffic passes. The two timestamps to focus on are "
        "00:00:08 and 00:00:12. At 00:00:08, the key event is the white "
        "sedan beginning its turning motion into the intersection as a "
        "gap appears in the oncoming stream. At 00:00:12, the key "
        "situation is the sedan having completed the turn, with wheels "
        "straightened and the vehicle moving in the perpendicular lane.\n\n"
        "The logical bridge between these two moments is the sedan's arc "
        "through the intersection and the absence of conflicting traffic. "
        "The driver used the gap to cross the path of the former oncoming "
        "flow; the turn was executed in one continuous motion. By "
        "00:00:12 the vehicle has left the intersection and is in a "
        "stable, straight trajectory.\n\n"
        "We can conclude that the situation at 00:00:12 is the direct "
        "result of the decision and action at 00:00:08. The relationship "
        "is one of action and completion: the event at 00:00:08 "
        "(initiating the turn when safe) leads to the situation at "
        "00:00:12 (turn completed, vehicle in new lane). There is no "
        "anomaly; the sequence follows normal traffic behavior and "
        "right-of-way.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Question: [Question]\n"
        "======\n"
        "Answer: [Answer] (open-ended, should be a paragraph)\n"
        "=====\n"
        "Reasoning: [Provide the analysis as coherent paragraphs. Do not "
        "use bullet points or numbered lists.]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Detailed Video Description]\n"
        "{step_2_output}"
    ),

    "anomaly_temporal_localization": (
        "You are an expert video annotator and logical analyst "
        "specializing in temporal localization. You are provided with a "
        "holistic and detailed description for a traffic video containing "
        "an anomaly.\n\n"
        "Task:\n"
        "1. Identify a significant event or anomaly that requires logical "
        "reasoning to localize.\n"
        "2. Generate a Temporal Localization Question in the format: "
        "\"When does [Event X] occur in the video?\"\n"
        "3. Provide the Answer with exact start and end timestamps.\n"
        "4. Provide a detailed Reasoning paragraph following a "
        "three-phase logical trace:\n"
        "   - Describe the state of the video immediately before the "
        "start timestamp. Identify the specific traffic rules or flow "
        "patterns that were being followed to establish why the event had "
        "NOT yet begun.\n"
        "   - Pinpoint the exact visual 'state-change' at the start "
        "timestamp. Explain the root cause (e.g., signal violation, "
        "misjudgment, mechanical failure) that initiated the anomaly.\n"
        "   - Justify the end timestamp by describing the transition into "
        "a new, stable, or static state (e.g., vehicles reaching a full "
        "stop, clearing the intersection, or the camera view changing).\n\n"
        "Constraints:\n"
        "- The reasoning must follow the 'Sandwich Structure': Baseline "
        "-> Trigger -> Resolution.\n"
        "- STRICT ADHERENCE TO TIMESTAMPS: Do not hallucinate times. "
        "Cross-reference all provided captions.\n"
        "- The Reasoning section must be a coherent, dense paragraph. Do "
        "not use bullet points or numbered lists.\n"
        "- Treat the description as a representation of the video itself. "
        "Do not mention the description; describe the video directly.\n\n"
        "Here is an example of the desired content style and depth. You "
        "don't need to follow this example exactly, but you should follow "
        "the same format and the style:\n"
        "<example>\n"
        "Question: When does the side-impact collision between the black "
        "pickup truck and the crossing SUV occur?\n"
        "======\n"
        "Answer: Start_Time: 00:00:01, End_Time: 00:00:05\n"
        "=====\n"
        "Reasoning:\n"
        "The video begins with a stable baseline of urban traffic flow. "
        "Between 00:00:00 and 00:00:01, a dark grey SUV is seen crossing "
        "the intersection from right to left with the right-of-way, while "
        "a queue of vehicles (including a red semi-truck) remains "
        "correctly stationary at the off-ramp signal. The event is "
        "triggered at 00:00:01 when a black pickup truck, bypassing the "
        "stationary queue at high speed, violates the red light and "
        "enters the intersection. This root-cause violation results in an "
        "immediate T-bone impact at 00:00:01, where the pickup strikes "
        "the passenger side of the SUV. The force of the impact causes "
        "the SUV to roll onto its side and slide toward the bottom-left "
        "corner of the frame. The event reaches a resolution at 00:00:05; "
        "by this time, the SUV has slid completely out of the camera's "
        "view, and the black pickup truck has come to a full, static halt "
        "in the center of the intersection with visible front-end damage, "
        "transitioning the scene from an active collision to a stationary "
        "obstruction.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Question: [Question]\n"
        "======\n"
        "Answer: Start_Time: [MM:SS], End_Time: [MM:SS]\n"
        "=====\n"
        "Reasoning: [Provide the analysis as a coherent paragraph. "
        "Describe the 'before-state' that led to the start timestamp, the "
        "visual triggers during the event, and the 'after-state' or "
        "resolution that confirms the event ended at the specific "
        "timestamp.]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Detailed Video Description]\n"
        "{step_2_output}"
    ),

    "normal_temporal_localization": (
        "You are an expert video annotator and logical analyst "
        "specializing in temporal localization. You are provided with a "
        "holistic and detailed description for a normal traffic video.\n\n"
        "Task:\n"
        "1. Identify a significant traffic event that requires logical "
        "reasoning to localize. Focus on regular traffic events such as:\n"
        "   - A vehicle completing a lane change or turn\n"
        "   - A traffic signal phase change affecting vehicle flow\n"
        "   - A pedestrian crossing the street\n"
        "   - A vehicle merging into traffic\n"
        "   - A parking maneuver\n"
        "   - Traffic flow pattern changes (e.g., congestion forming or "
        "clearing)\n\n"
        "2. Generate a Temporal Localization Question in the format: "
        "\"When does [Event X] occur in the video?\"\n\n"
        "3. Provide the Answer with exact start and end timestamps.\n\n"
        "4. Provide a detailed Reasoning paragraph following a "
        "three-phase logical trace:\n"
        "   - Describe the state of the video immediately before the "
        "start timestamp. What was the traffic flow or vehicle position "
        "that preceded this event?\n"
        "   - Pinpoint the exact visual 'state-change' at the start "
        "timestamp. What action or movement initiated the event?\n"
        "   - Justify the end timestamp by describing the transition into "
        "a new, stable state (e.g., vehicle completing the maneuver, "
        "pedestrian reaching the other side, traffic resuming normal "
        "flow).\n\n"
        "Constraints:\n"
        "- The reasoning must follow the 'Sandwich Structure': "
        "Before-State -> Action/Transition -> After-State.\n"
        "- STRICT ADHERENCE TO TIMESTAMPS: Do not hallucinate times. "
        "Cross-reference all provided captions.\n"
        "- The Reasoning section must be a coherent, dense paragraph. Do "
        "not use bullet points or numbered lists.\n"
        "- Treat the description as a representation of the video itself. "
        "Do not mention the description; describe the video directly.\n\n"
        "Here is an example of the desired content style and depth:\n"
        "<example>\n"
        "Question: When does the white sedan complete its left turn at "
        "the intersection?\n"
        "======\n"
        "Answer: Start_Time: 00:00:08, End_Time: 00:00:12\n"
        "=====\n"
        "Reasoning:\n"
        "Prior to 00:00:08, the white sedan is stationary in the "
        "left-turn lane, waiting for oncoming traffic to clear. The "
        "intersection shows a steady flow of vehicles traveling straight "
        "through from the opposite direction, and the white sedan's turn "
        "signal is visible, indicating intent to turn. At 00:00:08, a gap "
        "appears in the oncoming traffic as the last vehicle passes, and "
        "the white sedan begins its turning motion, entering the "
        "intersection and rotating leftward. The vehicle continues its "
        "arc through the intersection, crossing the path where oncoming "
        "traffic was previously flowing. By 00:00:12, the white sedan has "
        "fully completed its turn, now traveling in the perpendicular "
        "lane with its wheels straightened and maintaining a consistent "
        "forward trajectory, marking the conclusion of the turning "
        "maneuver.\n"
        "</example>\n\n"
        "Format:\n"
        "=====\n"
        "Question: [Question]\n"
        "======\n"
        "Answer: Start_Time: [MM:SS], End_Time: [MM:SS]\n"
        "=====\n"
        "Reasoning: [Provide the analysis as a coherent paragraph. "
        "Describe the 'before-state' that led to the start timestamp, the "
        "action/transition during the event, and the 'after-state' that "
        "confirms the event ended at the specific timestamp.]\n\n"
        "Input Data:\n"
        "Video Length: {video_length}s\n\n"
        "[Detailed Video Description]\n"
        "{step_2_output}"
    ),
}


def get_prompt(key, **kwargs):
    """Retrieve and optionally format a prompt template by key."""
    template = PROMPT_TEMPLATES.get(key)
    if template is None:
        raise ValueError(f"No prompt template found for key: {key}")
    if kwargs:
        return template.format(**kwargs)
    return template

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Declares allowed tools, Needs an API key, Needs Python

The author says: Requires docker + nvidia-container-toolkit + at least one VLM endpoint (Gemini API key or OpenAI-compatible).

Security

What our scanner noticed· Medium risk

  • Large base64-encoded blobs
  • Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
NVIDIA Corporation
Publisher
NVIDIA (Official)
Source
NVIDIA/skills · skills/tao-generate-video-reasoning-annotations · commit fce3cca
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE-APACHE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by NVIDIA Corporation or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download tao-generate-video-reasoning-annotations.zip