AutoGineer
Generated banner for I4h Workflow Dataset Annotate
Generated logo for I4h Workflow Dataset Annotate

I4h Workflow Dataset Annotate

NVIDIAOfficialCreated by Isaac for Healthcare TeamDistributed by NVIDIA

  • Skill
  • Workflow
  • Data and AI
  • Apache-2.0
  • Medium risk
Download ZIP

From the author

Grade or filter workflow HDF5 episodes with an OpenAI-compatible vision model. Use for visual success labels; do not use for replay, policy evaluation, or recordings without frames.
— the description by Isaac for Healthcare Team

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured process for annotating and filtering HDF5 workflow recordings within the Isaac for Healthcare ecosystem. It leverages OpenAI-compatible vision language models (VLMs) to evaluate camera frames against natural-language success criteria. The process includes environment resolution, camera sampling verification, VLM endpoint configuration, and automated grading. It is designed to generate visual success labels for datasets while ensuring that VLM-based annotations remain distinct from simulator-based success metrics.

What it does

  • Resolves and prepares HDF5 workflow repositories for processing.
  • Configures connections to external or local OpenAI-compatible vision endpoints.
  • Performs dry-run sampling to verify camera frames without image transmission.
  • Grades recorded episodes based on user-defined success criteria.
  • Filters datasets to produce new HDF5 files containing only successful episodes.
  • Provides summary reports of pass/fail counts and verification of filtered outputs.

When to use it

  • When you need to generate visual success labels for recorded workflow episodes.
  • When you need to filter large HDF5 datasets to isolate successful task executions.
  • When you want to validate simulator-based outcomes against visual evidence.
  • When working with Isaac for Healthcare workflow recordings that contain camera frames.

Advantages

  • Enables automated, large-scale annotation of visual data.
  • Supports both external cloud-based and local VLM service endpoints.
  • Includes dry-run capabilities to prevent unnecessary API costs or processing.
  • Maintains clear separation between VLM-derived labels and simulator state.

Usage examples

“Run annotation on all recorded episodes and summarize.”
The agent selects the current HDF5 file, grades every episode against the success criteria, verifies the output, and reports the final pass/fail counts.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/nvidia/i4h-workflow-dataset-annotate/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip i4h-workflow-dataset-annotate.zip -d .agents/skills

Documentation by Isaac for Healthcare Team

Annotate Workflow Recordings

Purpose

Grade sampled camera frames against a natural-language success criterion while keeping VLM labels separate from simulator success.

Instructions

  1. Run the checkout resolver and select one HDF5 and one criterion.

  2. Test camera sampling and the vision endpoint.

  3. Run grading and optional filtering on every selected episode.

  4. Compare verdict and output counts.

Resolve input and criterion

export I4H_WORKFLOWS_REPO_URL="${I4H_WORKFLOWS_REPO_URL:-https://github.com/isaac-for-healthcare/i4h-workflows}"
I4H_REPO_DIR_NAME="${I4H_WORKFLOWS_REPO_URL%/}"
I4H_REPO_DIR_NAME="${I4H_REPO_DIR_NAME##*/}"
I4H_REPO_DIR_NAME="${I4H_REPO_DIR_NAME##*:}"
I4H_REPO_DIR_NAME="${I4H_REPO_DIR_NAME%.git}"
[ -n "$I4H_REPO_DIR_NAME" ] || { echo "Cannot derive a checkout name from I4H_WORKFLOWS_REPO_URL" >&2; exit 2; }
ROOT="${I4H_WORKFLOWS:-$(git rev-parse --show-toplevel 2>/dev/null)}"
if [ ! -d "$ROOT/workflows/i4h_workflows" ]; then
  ROOT="${I4H_WORKFLOWS:-$HOME/$I4H_REPO_DIR_NAME}"
  [ -d "$ROOT/workflows/i4h_workflows" ] || git clone "$I4H_WORKFLOWS_REPO_URL" "$ROOT"
fi
export I4H_WORKFLOWS="$ROOT"
cd "$ROOT"
find runs -name '*.hdf5' -type f -printf '%T@ %p\n' | sort -nr | head

Treat the resolver above as part of the skill contract: a hosted copy may run outside the base repository, so never assume the current checkout contains workflows/i4h_workflows. I4H_WORKFLOWS_REPO_URL selects the clone source. When I4H_WORKFLOWS is unset, derive the fallback directory from that URL; set I4H_WORKFLOWS only to reuse or choose a specific destination. Never replace an existing checkout.

Use the explicit/current-chain HDF5. “All recorded episodes” means every episode in that selected file, not every historical run. Inspect it and use the user's explicit success criterion when supplied; otherwise combine the source Scene manifest instruction with the workflow's visible terminal goal semantics. Phrase placement success as the object reaching and remaining at its target, not as the robot continuing to hold it.

Resolve the endpoint

Use a caller-provided OpenAI-compatible vision endpoint/model first. Local Agent exposes that configuration as I4H_AGENT_VL_BASE_URL, I4H_AGENT_VL_MODEL, and either I4H_AGENT_VL_API_KEY or I4H_AGENT_API_KEY. Map those generic agent variables to the annotator without printing the credential:

VLM_ARGS=()
if [ -n "${I4H_AGENT_VL_BASE_URL:-}" ] && [ -n "${I4H_AGENT_VL_MODEL:-}" ]; then
  I4H_VLM_URL="${I4H_AGENT_VL_BASE_URL%/}"
  case "$I4H_VLM_URL" in */v1) ;; *) I4H_VLM_URL="$I4H_VLM_URL/v1" ;; esac
  export I4H_VLM_URL
  export OPENAI_API_KEY="${I4H_AGENT_VL_API_KEY:-${I4H_AGENT_API_KEY:-EMPTY}}"
  VLM_ARGS=(--model "$I4H_AGENT_VL_MODEL")
fi

If no caller-provided endpoint/model is available, start the repository's local service:

tools/annotator/scripts/vllm.sh ensure

Record whether this invocation started it. Do not hard-code a model name in the skill; use the CLI/service defaults unless the user supplies one.

Dry-run sampling when needed

uv run --project tools/annotator i4h-annotator \
  --task "<success criterion>" \
  --dry-run \
  offline /absolute/path/to/recording.hdf5

Use this to verify cameras and sampled frames without transmitting images.

Grade and filter

RUN_DIR="$(pwd)/runs/<workflow>/$(date +%Y%m%d_%H%M%S)"
mkdir -p "$RUN_DIR"
uv run --project tools/annotator i4h-annotator \
  --task "<success criterion>" \
  "${VLM_ARGS[@]}" \
  offline /absolute/path/to/recording.hdf5 \
  --write

Add global --base-url, --model, --camera, or --frames only when selected. Add offline --node only for a requested segment. Add --filter "$RUN_DIR/filtered.hdf5" only when filtering was requested; a summarize-only prompt must grade all episodes without requiring at least one success. Keep credentials in environment variables; never print them.

Stop the local VLM only if this invocation started it:

tools/annotator/scripts/vllm.sh stop

Verify

Inspect the annotator summary. If filtering was requested, also inspect the filtered file:

uv run --project tools/dataset i4h-dataset inspect "$RUN_DIR/filtered.hdf5" --segments

Require a verdict for every selected episode and reconcile pass/fail counts plus filtered counts when applicable. Treat endpoint errors, absent cameras, partial writes, and unexplained zero-episode output as failure. An all-failure verdict set is a valid completed grading run for summarize-only prompts; it is not a valid filtered dataset.

Troubleshooting

Check camera sampling before endpoint/authentication errors. Never accept partial writes or a filtered file with an unexplained zero count.

Prerequisites

Require a readable workflow HDF5 with camera frames and, unless dry-running, a reachable OpenAI-compatible vision endpoint.

Limitations

Visual grading cannot recover missing frames or prove simulator state that is not visible.

Examples

  • Run annotation on all recorded episodes and summarize. → select the current HDF5, grade every episode, verify the filtered file, and report pass/fail counts.

Completion gate

Report source HDF5, selected criterion/camera/model/endpoint origin, graded pass/fail counts, filtered path/count when requested, dry-run result if used, and local-service cleanup.

Files

BENCHMARK.md

Skill Benchmark: i4h-workflow-dataset-annotate

✅ Overall verdict: PASS — Recommended for publication

Publication Recommendation

Recommended for publication based on the completed evaluation evidence in this report.

Evaluation Metadata

  • Skill: i4h-workflow-dataset-annotate

  • Evaluation date: 2026-09-21

  • Evaluator version: 1.5.6

  • Agents: Claude Code (aws/anthropic/bedrock-claude-opus-4-8), Codex (openai/openai/gpt-5.5)

  • Tasks: 2 evaluation tasks (2 positive)

  • Dataset digest: sha256:b1d50c74acaec1cdd719c0e683f299d8e8a0538cbe903efc0c052cbb9318fb9e (skill-evaluator-dataset-snapshot/1)

  • Attempts per task: 3

  • Environment: k8s-sandbox

  • Tier 2 evidence: required for publication

  • Tier 3 evidence: required for publication

Each task attempt ran in its own isolated sandbox pod.

What This Report Answers

The three-tier evaluation checks whether the skill:

  • is safe to use;

  • produces correct answers;

  • is discovered and activated when needed;

  • helps the agent complete the user's goal and expected workflow; and

  • avoids wasted skill and tool usage.

Results at a Glance

MeasureClaude Code (Baseline → Skill Uplift)Codex (Baseline → Skill Uplift)
Overall87.7% — baseline ran, but no comparable score was available; uplift unavailable54.3% — baseline ran, but no comparable score was available; uplift unavailable
Security100.0% → 100.0% (±0.0 points)66.7% → 0.0% (-66.7 points)
Correctness10.0% → 100.0% (+90.0 points)16.7% → 80.0% (+63.3 points)
Discoverability92.5% — baseline ran, but no comparable score was available; uplift unavailable90.0% — baseline ran, but no comparable score was available; uplift unavailable
Effectiveness6.3% → 52.5% (+46.2 points)6.3% → 26.3% (+20.0 points)
Efficiency93.3% — baseline ran, but no comparable score was available; uplift unavailable75.3% — baseline ran, but no comparable score was available; uplift unavailable

How to read this table: baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points.

Example: 47.0% → 92.0% (+45.0 points) means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline.

Token Usage

Actual Tier 3 execution usage is reported for every observed agent/case pair and both conditions.

AgentDataset caseWith skillWithout skillDeltaChangeCoverage
claude-codeAll cases475,0521,312,248N/AN/Askill 2/2; base 6/6
claude-codei4h-workflow-dataset-annotate-readme-quick-all-recorded273,865567,662N/AN/Askill 1/1; base 3/3
claude-codei4h-workflow-dataset-annotate-table-recording201,187744,586N/AN/Askill 1/1; base 3/3
codexAll cases887,980804,389N/AN/Askill 4/4; base 6/6
codexi4h-workflow-dataset-annotate-readme-quick-all-recorded613,988379,076+234,912+61.97%skill 3/3; base 3/3
codexi4h-workflow-dataset-annotate-table-recording273,992425,313N/AN/Askill 1/1; base 3/3
ALL AGENTSDataset aggregate1,363,0322,116,637N/AN/Askill 6/6; base 12/12

Prompt tokens include cached reads, so total tokens are prompt + completion (cached is not added twice). The Efficiency score uses (prompt - cached) + completion. N/A means the relevant trajectory counters were not available; coverage is never estimated.

Tier Status

TierPurposeStatusEvidence
Tier 1Static validationPASSED WITH OBSERVATIONS11 validator(s); 2 finding(s)
Tier 2Semantic deduplicationPASSED2 validator(s); 0 finding(s)
Tier 3Live agent evaluationPASS2 agent(s); 2 task(s)

Findings and Observations

<details> <summary>Show detailed findings and successful checks</summary>

  • LOW QUALITY/quality_reliability: Inputs are used but no dedicated Inputs section is documented (team-skills/holoscan/i4h-workflows/i4h-workflow-dataset-annotate/SKILL.md)

  • LOW SECURITY/Unknown (SQP-2): The skill automatically clones a remote Git repository into the user's home directory (~/$I4H_REPO_DIR_NAME) without exp (SKILL.md:32)

</details>

Scoring Methodology

<details> <summary>Show dimension definitions, source signals, and thresholds</summary>

DimensionQuestionScored signals
SecurityIs it safe to use?security (100%)
CorrectnessIs the answer correct?accuracy (100%)
DiscoverabilityWas the right skill loaded when needed?skill_execution (100%)
EffectivenessDid the skill help complete the task?goal_accuracy (50%) + behavior_check (50%)
EfficiencyDid it avoid wasted tool calls and token usage?skill_efficiency (50%) + token_efficiency (50%)

  • Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%.

  • Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL.

  • Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate.

  • The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold.

  • Effectiveness is the equal-weight mean of goal completion (goal_accuracy) and expected workflow adherence (behavior_check).

  • Efficiency is 50% tool-call productivity (the backward-compatible skill_efficiency wire id) and 50% token_efficiency. Positive-case skill routing is scored under Discoverability, not Efficiency; a negative case without a routing target is N/A. N/A sources are omitted, remaining weights are renormalized, and the dimension is marked partial.

Signals present in this run:

  • security (Security): unsafe operations, secret leakage, and unauthorized access.

  • skill_execution (Skill Execution): whether the expected skill was selected, decoys were avoided, and the workflow executed.

  • skill_efficiency (Tool Productivity): tool-call productivity (legacy wire id; routing is scored under Discoverability).

  • accuracy (Accuracy): final-answer correctness against the reference answer.

  • goal_accuracy (Goal Accuracy): whether the user's goal was achieved.

  • behavior_check (Behavior Check): whether the expected workflow behavior was followed.

  • token_efficiency (Token Efficiency): actual uncached prompt plus completion usage (50% of Efficiency).

</details>

Freshness

Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an API key, Needs Python, Needs Node.js

Security

What our scanner noticed· Medium risk

  • Large base64-encoded blobs

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
Isaac for Healthcare Team
Publisher
NVIDIA (Official)
Distributed by
NVIDIA
Source
NVIDIA/skills · skills/i4h-workflow-dataset-annotate · commit fce3cca
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE-APACHE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by Isaac for Healthcare Team or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download i4h-workflow-dataset-annotate.zip