Tao Train Foundation Stereo

NVIDIAOfficialCreated by NVIDIA Corporation

Skill
Tool integration
Data and AI
Apache-2.0
Medium risk

From the author

Stereo depth estimation using FoundationStereo. Predicts disparity maps from stereo image pairs for 3D reconstruction. Use when training, evaluating, exporting, or running inference for a TAO FoundationStereo model. Trigger phrases include "train stereo depth", "FoundationStereo", "stereo disparity estimation", "3D reconstruction from stereo".

— the description by NVIDIA Corporation

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The tao-train-foundation-stereo skill provides a comprehensive interface for the FoundationStereo architecture within the NVIDIA TAO ecosystem. It enables users to perform the full model lifecycle, including training, evaluation, inference, and export, specifically for predicting disparity maps from stereo image pairs. The skill supports advanced features like AutoML for hyperparameter optimization and integrates with TAO Deploy for optimized TensorRT inference. It is designed to handle various dataset formats, including KITTI, Middlebury, and synthetic data, ensuring consistent performance across different stereo vision tasks.

What it does

Trains FoundationStereo models for stereo depth estimation.
Evaluates model performance using stereo-specific metrics like EPE, bp1, and d1.
Exports trained models to ONNX format for deployment.
Supports AutoML for automated hyperparameter tuning during training.
Performs inference on stereo image pairs to generate disparity maps.
Integrates with TAO Deploy for generating optimized TensorRT engines.

When to use it

When you need to perform 3D reconstruction from stereo image pairs.
When training a custom stereo depth estimation model using the FoundationStereo architecture.
When you need to optimize model performance using AutoML.
When preparing a stereo depth model for production deployment via TensorRT.

Advantages

Unified workflow for the entire model lifecycle within the TAO ecosystem.
Supports multiple standard stereo dataset formats.
Provides stereo-specific evaluation metrics for accurate performance assessment.
Includes AutoML capabilities to simplify hyperparameter optimization.

Usage examples

“Train a new FoundationStereo model using my KITTI dataset.”: The agent initiates the training process using the FoundationStereo architecture, configured for the KITTI dataset format, and monitors progress via the specified training metrics.
“Evaluate my trained stereo model on the validation set.”: The agent runs the evaluation action, calculating stereo-specific metrics like EPE and d1 to assess the model's accuracy on the provided validation data.
“Export my FoundationStereo model to ONNX.”: The agent triggers the export workflow, converting the trained PyTorch model into an ONNX file suitable for deployment.
“Run inference on a pair of stereo images.”: The agent processes the input stereo image pair through the model and outputs the predicted disparity map for 3D reconstruction.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/nvidia/tao-train-foundation-stereo/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip tao-train-foundation-stereo.zip -d .agents/skills

Documentation by NVIDIA Corporation

Depth Net Stereo

Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).

Stereo depth estimation using FoundationStereo architecture. Predicts disparity maps from stereo image pairs for 3D reconstruction.

Uses pretrained Depth Anything v2 and EdgeNeXt encoders. Set model.stereo_backbone.depth_anything_v2_pretrained_path and model.stereo_backbone.edgenext_pretrained_path.

The mono and stereo skills both invoke the unified TAO depth_net CLI inside the container; the mono/stereo family is selected via model.model_type (e.g., FoundationStereo).

PyT actions packaged by this model skill: train, evaluate, inference, export, and quantize. The PyT depth_net entrypoint does not accept a gen_trt_engine action in the current TAO image; build TensorRT engines only through the deploy workflow.

For TAO Deploy TensorRT actions (gen_trt_engine, TensorRT evaluate, and TensorRT inference), read references/tao-deploy-foundation-stereo.md first. The deploy spec template lives in this skill's references/spec_template_deploy.yaml.

Train Action Policy

This model is AutoML-enabled at the model layer. Before handling any train-stage request, read references/skill_info.yaml and resolve the run override from either an explicit automl_policy value or the user's workflow request. Use automl_policy: on by default and only expose on / off in new launch prompts. Treat phrases like "turn off AutoML", "disable AutoML", "no HPO", or "plain training" as automl_policy: off for this run only. When automl_policy: on, automl_enabled: true, and both schemas/train.schema.json and references/spec_template_train.yaml are packaged, route the train action through tao-skill-bank:tao-run-automl by default with this model's skill_dir. Preserve workflow/application overrides for datasets, specs, output directories, GPU/platform settings, parent checkpoints, and automl_policy. Use direct model training only when automl_policy: off or the packaged train schema/template is missing; in the missing-schema case, report that AutoML is enabled but not runnable for this model until schemas are generated.

Non-train actions such as evaluate, inference, export, and deploy flows stay in this model skill. The per-run automl_policy override does not change model metadata.

Workflow

Prerequisites — data accessibility

Your dataset (left + right images + GT disparity) must be reachable from inside the container:

SDK runner: place files at the S3 paths the runner resolves (the S3_TRAIN / S3_EVAL placeholders shown in the spec overrides). The runner handles S3 → container-path mounting transparently.
Direct docker run (e.g. local testing): mount the host dataset root read-only at the same in-container path:

docker run ... -v <host_data_root>:<host_data_root>:ro <container> ...

The same accessibility requirement applies to the <output_dir> written by all actions.

Step 1 — Annotation file

Per-line annotation file referenced by data_sources[*].data_file:

Columns	Format	Use
2	`<left> <right>`	Stereo inference (no GT)
3	`<left> <right> <disparity>`	Stereo with GT
4	`<left> <right> <disparity> <occlusion_mask>`	Stereo with GT and occlusion mask

If you already have one, point to it. Otherwise generate via depth_net convert:

depth_net convert -e <convert_spec.yaml>

convert_spec.yaml template (stereo):

results_dir: <directory where generated annotation files are written>
data_root: <directory whose immediate children are scene folders that contain your image+depth files; convert walks data_root recursively but expects per-scene subdirectories at one level below>
image_dir_pattern: [<substring matching left image paths>]
right_dir_pattern: [<substring matching right image paths>]
depth_dir_pattern: [<substring matching GT disparity paths>]
nocc_dir_pattern: []                 # optional, occlusion mask paths
image_extension: '.png'  # always include the leading dot
depth_extension: '.png'  # form must match image_extension (the swap is a substring replace)
nocc_extension: ''
split_ratio: 0.0        # 0.0/1.0 = test-only; 0.8 = 80/20 train+val

convert walks data_root recursively, selects paths whose path-string contains all substrings in image_dir_pattern (AND-filter), then derives right / depth / mask paths by replacing image_dir_pattern[0] with the corresponding pattern's first element plus extension swap. Inspect your dataset's directory layout and identify the substrings distinguishing left, right, and GT (e.g. im0 vs im1 vs disp0GT for Middlebury).

Step 2 — Pair `model_type` and `dataset_name` based on your data

Prefer the dataset-specific class when your layout matches a supported one — it applies class-specific path conventions, evaluation crops, and (where applicable) occlusion-mask handling. Fall back to GenericDataset only for layouts that do not match any registered class.

Data category	`model_type`	`dataset_name`
Middlebury data	`FoundationStereo`	`Middlebury`
KITTI data	`FoundationStereo`	`Kitti`
ETH3D data	`FoundationStereo`	`Eth3d`
FSD synthetic data	`FoundationStereo`	`FSD`
IsaacReal synthetic data	`FoundationStereo`	`IsaacRealDataset`
Crestereo synthetic data	`FoundationStereo`	`Crestereo`
Other / non-canonical layout	`FoundationStereo`	`GenericDataset`

Valid dataset_name values for stereo data_sources (case-insensitive): FSD, IsaacRealDataset, Crestereo, Middlebury, Eth3d, Kitti, GenericDataset.

The same dataset_name value applies across train and evaluate actions (all of which use 3-column or 4-column annotations with GT disparity). The deploy-side evaluate action follows the same rule — see references/tao-deploy-foundation-stereo.md. For inference with 2-column annotations (left + right, no GT), use dataset_name: GenericDataset regardless of data layout — the dataset-specific classes (Middlebury / Kitti / Eth3d / FSD / IsaacRealDataset / Crestereo) require 3-column input and reject 2-column annotations at the dataloader level. For inference with 3-column annotations (left + right + GT), the dataset-specific class is fine.

Step 3 — Write spec yaml from the spec overrides

Copy the action block from references/spec-overrides-foundation-stereo.md. Replace:

model.model_type from Step 2 (typically FoundationStereo)
dataset.<...>.data_sources[*].dataset_name from Step 2
dataset.<...>.data_sources[*].data_file with the path from Step 1
For deploy-side evaluate: enforce dataset.test_dataset.batch_size: 1 (see references/tao-deploy-foundation-stereo.md).

Shape consistency: the crop_size in dataset.test_dataset.augmentation.crop_size should match export.input_height / input_width so the trained-model evaluator and the deploy-side TensorRT evaluator operate at the same shape. Note that crop_size is decorative on the pyt evaluate path but authoritative on the deploy evaluate side — see references/troubleshooting-foundation-stereo.md and references/tao-deploy-foundation-stereo.md.

Fresh-install smoke runs are validated at crop_size: [128, 128] with dataset.max_disparity: 128 and model.max_disparity: 128. Avoid 112×112 crops and avoid setting max_disparity smaller than the square crop side for smoke tests: those combinations can fail inside FoundationStereo with feature-map or loss-mask shape mismatches before a checkpoint is produced.

Data source overrides are mandatory for every action. Each data_sources entry is a dict with two mandatory fields: data_file and dataset_name. See references/spec-overrides-foundation-stereo.md for the per-action dataset-requirements table, every action's override block, and the quantize known-issue note.

Step 4 — Run

Create writable home/cache directories inside the mounted output path before using --user. Some TAO containers do not have an /etc/passwd entry for the host UID, and PyTorch / matplotlib need writable cache paths when running as that UID.

mkdir -p <output_dir>/home \
         <output_dir>/.cache/matplotlib \
         <output_dir>/.cache/torchinductor \
         <output_dir>/.cache/xdg

docker run --gpus 'device=0' --shm-size 16G --shm-size=8g \
  --user "$(id -u):$(id -g)" \
  -e USER="$(id -un)" \
  -e LOGNAME="$(id -un)" \
  -e HOME=<output_dir>/home \
  -e MPLCONFIGDIR=<output_dir>/.cache/matplotlib \
  -e TORCHINDUCTOR_CACHE_DIR=<output_dir>/.cache/torchinductor \
  -e XDG_CACHE_HOME=<output_dir>/.cache/xdg \
  -v <data_root>:<data_root>:ro \
  -v <output_dir>:<output_dir> \
  <container> \
  depth_net <action> -e <spec.yaml>

Without --user "$(id -u):$(id -g)" the container writes outputs as nobody:nogroup, blocking host-side cleanup / retry.

Step 5 — Verify

Container exit code 0
status.json kpi block populated
For train: inspect per-step train_loss directly (the entrypoint reports Execution status: PASS even when loss is NaN)
For evaluate: rely on epe / bp1 / bp2 / bp3 / d1 / rmse (the evaluator also emits abs_rel / sq_rel / rmse_log which are non-meaningful for stereo — see references/parameters-foundation-stereo.md)
For inference: artifacts under results_dir

For TAO Deploy TensorRT actions (gen_trt_engine, TensorRT evaluate, and TensorRT inference), read references/tao-deploy-foundation-stereo.md first. Deploy spec templates live in this skill's references/ folder with the spec_template_deploy_*.yaml prefix.

Training Requirements

AutoML metric contract: when recommendations are scored with the packaged evaluate action, use val/epe with minimize direction.
Training monitoring metric: val/loss Use this only for live training-loss monitoring.
AutoML/evaluation KPI: val/epe, minimized. The packaged evaluate action writes this key to status.json; using val/loss as the AutoML objective makes trial metric extraction fail. Its Lightning console table labels the same evaluator output test/epe, so compare AutoML's recorded val/epe value with the console's test/epe value when verifying a run.
Recommended AutoML train parameters: train.optim.lr and train.optim.weight_decay. Do not search validation/test/inference augmentation fields, and do not search model.volume_dim because the current implementation does not consume it.
Eval dataset: optional. Val dataset configured via dataset.val_dataset.data_sources (each entry needs data_file and dataset_name).

See references/spec-overrides-foundation-stereo.md for the per-action dataset-requirements table and every action's mandatory data-source override block.

Parameters, Metrics, Multi-GPU, Export/TRT, Hardware

See references/parameters-foundation-stereo.md for the full Important Parameters list (incl. model.encoder vits override, model.max_disparity default 416, model.volume_dim no-op note, dataset.baseline, dataset.focal_x, train.precision, export.batch_size), the Evaluation Metrics table, Multi-GPU / Multi-Node launch keys, Export / TRT Defaults (opset_version/on_cpu pairing, NGC 576×960 settings), and Hardware requirements.

Error Patterns and Troubleshooting

See references/troubleshooting-foundation-stereo.md for disparity overflow, smoke-test shape mismatch, missing pretrained paths, the encoder / dataset_name struct errors, the depth_net_stereo: not found entrypoint note, the pyt-vs-deploy crop_size discussion, and the deploy evaluate scalar-conversion failure.

Spec Param / Parent Model Inference

See references/checkpoint-inference-mappings-foundation-stereo.md for the checkpoint-resolution rules (model_epoch_<epoch>_step_<step>.pth, dn_model_latest.pth policy), the absence of parent PyT gen_trt_engine, and the full per-action inference-mapping table from depth_net_stereo.config.json (including parent_model / parent_job_id resolution).

Deployment

tao-deploy-foundation-stereo

Files

ATTRIBUTION.mdAdded by AutoGineer951 B
BENCHMARK.md7.0 KiB
LICENSELicence file added by AutoGineer11.2 KiB
SKILL.md12.7 KiB
config/skillspector-baseline.yaml1010 B
evals/evals.json819 B
references/checkpoint-inference-mappings-foundation-stereo.md3.9 KiB
references/parameters-foundation-stereo.md5.6 KiB
references/skill_info.yaml5.5 KiB
references/spec-overrides-foundation-stereo.md4.1 KiB
references/spec_template_deploy.yaml2.0 KiB
references/spec_template_evaluate.yaml5.5 KiB
references/spec_template_export.yaml5.6 KiB
references/spec_template_gen_trt_engine.yaml5.6 KiB
references/spec_template_inference.yaml5.6 KiB
references/spec_template_quantize.yaml5.4 KiB
references/spec_template_train.yaml5.4 KiB
references/tao-deploy-foundation-stereo.md11.7 KiB
references/tao-deploy-foundation-stereo.skill_info.yaml3.2 KiB
references/troubleshooting-foundation-stereo.md2.4 KiB
schemas/evaluate.schema.json104.5 KiB
schemas/export.schema.json105.1 KiB
schemas/gen_trt_engine.schema.json107.0 KiB
schemas/inference.schema.json104.9 KiB
schemas/manifest.json29.9 KiB
schemas/quantize.schema.json101.4 KiB
schemas/train.schema.json101.4 KiB
skill-card.md4.4 KiB
skill.oms.sig9.9 KiB

references/tao-deploy-foundation-stereo.md

DepthNet Stereo Deploy

DepthNet Stereo deploy covers the TAO Deploy actions for an exported FoundationStereo model. Use the depth-net-stereo model skill for training, checkpoint evaluation, quantization, export, or non-TensorRT inference where those actions exist. Use this deploy workflow after export when the input artifact is an ONNX model and the desired output is a TensorRT engine or TensorRT-backed predictions.

Supported actions: gen_trt_engine, evaluate, inference. Direct TAO Deploy command name: depth_net.

Quick Start

Set the deploy container once at the top of the session — every command below uses it:

TAO_DEPLOY_IMAGE=nvcr.io/nvidia/tao/tao-toolkit:7.3.0-deploy  # versions-key: images.tao_toolkit.deploy

Generate TensorRT Engine

docker run --gpus all --rm --shm-size=16g \
  -v /path/to/specs:/specs \
  -v /path/to/export:/models \
  -v /path/to/results:/results \
  "$TAO_DEPLOY_IMAGE" \
  depth_net gen_trt_engine -e /specs/gen_trt_engine.yaml

Evaluate TensorRT Engine

docker run --gpus all --rm --shm-size=16g \
  -v /path/to/specs:/specs \
  -v /path/to/eval:/data \
  -v /path/to/results:/results \
  "$TAO_DEPLOY_IMAGE" \
  depth_net evaluate -e /specs/evaluate.yaml

TensorRT Inference

docker run --gpus all --rm --shm-size=16g \
  -v /path/to/specs:/specs \
  -v /path/to/inference:/data \
  -v /path/to/results:/results \
  "$TAO_DEPLOY_IMAGE" \
  depth_net inference -e /specs/inference.yaml

Deploy action metadata is in tao-deploy-foundation-stereo.skill_info.yaml. Deploy spec template lives in this references folder:

spec_template_deploy.yaml

Deploy Workflow

Train and export with the depth-net-stereo skill.
Keep the exported ONNX artifact and any sidecar files together in the mounted model directory.
Build the TensorRT engine with this workflow.
Run TensorRT evaluate or inference from the engine artifact produced by gen_trt_engine.

Direct TAO Launcher spelling is tao deploy depth_net gen_trt_engine, tao deploy depth_net evaluate, tao deploy depth_net inference.

Required Inputs

Action	Required artifact or data	Spec key
`gen_trt_engine`	Exported FoundationStereo ONNX model	`gen_trt_engine.onnx_file`
`evaluate`	TensorRT engine	`evaluate.trt_engine`
`evaluate`	Stereo annotation file (3-col with GT, 4-col adds occlusion mask)	`dataset.test_dataset.data_sources[0].data_file`
`inference`	TensorRT engine	`inference.trt_engine`
`inference`	Stereo annotation file (2-col left+right, no GT)	`dataset.infer_dataset.data_sources[0].data_file`

gen_trt_engine.trt_engine is a generated output path, not an input artifact. For direct Docker runs, mount input folders at the same paths used in the spec. For chained jobs, map exported ONNX artifacts into gen_trt_engine.onnx_file and map the engine artifact into evaluate.trt_engine or inference.trt_engine.

Spec Template

Stereo deploy supports one model (FoundationStereo). Copy spec_template_deploy.yaml as a starting point and override only paths and environment-specific values (data_file, results_dir, trt_engine paths, batch size as needed).

Adjustments by use case:

Inference (no GT) — switch dataset.infer_dataset.data_sources[0].dataset_name to GenericDataset (the default in the template). Use a 2-column annotation file (left + right).
Evaluate / Inference with GT — pick a dataset-specific class (Middlebury, Kitti, Eth3d, FSD, IsaacRealDataset, Crestereo) when GT or occlusion-mask handling matches that class's conventions. Use a 3-column annotation (left + right + GT) or 4-column (with nocc mask).
Variable-aspect datasets (Middlebury) — pick a single (H, W) export shape per dataset (multiple of 32, close to the dataset's median aspect) and rebuild the engine for each (H, W) you serve. The engine is fully static on H/W; per-image variable shape is not supported.
Shape consistency — match dataset.test_dataset.augmentation.crop_size to evaluate.input_height/input_width and to the export-time ONNX shape (see "Shape consistency" below).

Common:

The TAO Deploy command is depth_net for both mono and stereo DepthNet model skills.
Fresh-install TRT precision: fp32. fp16 is supported on the static-shape and batch-only-dynamic deploy paths, but use FP32 for validation smoke tests unless the user requests FP16. Engine input H/W are pinned to the trace shape on every path.
The current TAO Deploy image interprets gen_trt_engine.tensorrt.workspace_size as GiB. Use 4 for a 4 GiB workspace; values such as 1024 request a 1024 GiB workspace and may fail on ordinary systems.

Deploy paths

Three deploy paths are supported. All produce a static-H/W engine; only the batch axis can be marked dynamic.

Path 1 — NGC pretrained static-shape ONNX

Use the NGC release deployable_foundationstereo_small_576x960_v2.0.onnx directly (skip train and export).

gen_trt_engine:
  onnx_file: <NGC ONNX path>
  trt_engine: <out engine path>
  batch_size: 1
  tensorrt:
    data_type: fp32
    workspace_size: 4
evaluate:
  trt_engine: <built engine>
  input_height: 576
  input_width: 960
model:
  model_type: FoundationStereo
dataset:
  test_dataset:
    augmentation:
      crop_size: [576, 960]

Path 2 — User-trained static-shape ONNX (NGC-compatible)

export:
  checkpoint: <user-trained ckpt>
  onnx_file: <out.onnx>
  input_height: 576
  input_width: 960
  opset_version: 17        # 17 OK with on_cpu=True (NGC release uses 17); 16 also works
  batch_size: 1            # static
  on_cpu: True             # required at 576×960 to avoid GPU OOM during trace

Then use the Path 1 spec yamls for gen_trt_engine, evaluate, inference.

Path 3 — User-trained batch-only-dynamic ONNX

Use this when the engine must accept multiple batch sizes from one build, with input H/W fixed by upstream preprocessing.

export:
  batch_size: -1           # batch axis dynamic; H, W are static at the trace shape
  input_height: 320
  input_width: 736
  opset_version: 16        # required when on_cpu=False (opset 17 + on_cpu=False is broken on TRT 10.13 fp16)
  on_cpu: False            # GPU trace fits ≤320×736; use on_cpu: True for ≥480×736

gen_trt_engine:
  onnx_file: <user batch-dynamic ONNX>
  trt_engine: <out engine>
  batch_size: -1
  tensorrt:
    data_type: fp32
    workspace_size: 4
    min_batch_size: 1
    opt_batch_size: 1
    max_batch_size: 4
evaluate:
  trt_engine: <built engine>
  input_height: 320        # same as export
  input_width: 736
inference:
  trt_engine: <built engine>
  input_height: 320
  input_width: 736

Recommended `opset_version` and `on_cpu` for FS small fp16 deploy

opset_version must be paired with on_cpu per the validated combinations below:

`on_cpu`	`opset_version` for fp16	Status
`True` (CPU trace)	16 or 17	Deterministic PASS (validated at 480×736 and 576×960)
`False` (GPU trace)	16 only	Mostly works; occasional non-deterministic build failure on TRT 10.13 — re-run on `costTensor.cpp::indexOfMin::120` or `optimizer.cpp::reduce::1258` assertions
`False` + `17`	—	Deterministically broken on TRT 10.13 fp16 — do not use

on_cpu is driven by export-trace GPU memory:

≤320×736: on_cpu: False is feasible (GPU trace fits in 47 GB VRAM).
≥480×736: on_cpu: True is required (PyTorch GPU trace OOMs on a 47 GB GPU).

Prefer on_cpu: True whenever feasible — at on_cpu=True the fp16 build is empirically deterministic at every tested shape (including the NGC release recipe 576×960+opset 17). fp32 builds are unaffected by these constraints.

Shape consistency: export ↔ evaluate ↔ deploy

The TRT engine is built from an ONNX file that fixes the input height and width at export time (export.input_height, export.input_width). The pyt-side evaluator and the deploy-side TRT evaluator must operate at the same shape to produce comparable disparity values, since disparity is in pixel units and scales with image width.

Knob	Where	Recommended convention
`export.input_height`, `export.input_width`	export action spec	the (height, width) the engine will see at inference time
`dataset.test_dataset.augmentation.crop_size`	pyt evaluate spec	match `[input_height, input_width]` exactly
`dataset.test_dataset.augmentation.crop_size`	deploy `evaluate` spec	match the engine input shape

Mismatched shapes between pyt and deploy paths produce different disparity values because the cropped/resized image presents a different pixel-disparity distribution to the model. Pick one shape (e.g., [320, 736]) and use it across export, pyt eval, and deploy eval. For datasets whose native aspect differs from the chosen shape, build a separate engine per (H, W) target.

Spec filename invariant

The spec yaml's basename (modulo .yaml) must match the action verb passed on the command line. For example, gen_trt_engine requires the spec at a path ending in gen_trt_engine.yaml; evaluate requires evaluate.yaml. Mismatched filenames produce a non-obvious FileNotFoundError from the hydra config loader before any action work begins.

TRT engine build time

gen_trt_engine for FoundationStereo is dominated by cost-volume convolution kernels and takes several minutes on x86 with a single A100/L40 (≈ 5 min for the FP32 engine at [1, 3, 320, 736]). Plan the deploy chain (gen_trt_engine → inference → evaluate) accordingly; the long build is one-time per (shape, precision) tuple.

Job Chain Mapping

Action	Spec field	Parent or output
`gen_trt_engine`	`gen_trt_engine.onnx_file`	export job ONNX
`gen_trt_engine`	`gen_trt_engine.trt_engine`	new engine output path
`evaluate`	`evaluate.trt_engine`	engine job output
`inference`	`inference.trt_engine`	engine job output

Outputs

Action	Output
`gen_trt_engine`	TensorRT engine at `gen_trt_engine.trt_engine`
`evaluate`	Stereo metrics under `results_dir` — primary metrics `epe`, `bp1`/`bp2`/`bp3`, `d1`, `rmse`. The simultaneously-emitted `abs_rel`, `sq_rel`, `rmse_log` are non-meaningful for stereo (formulated for mono metric depth); ignore them
`inference`	Disparity outputs under `results_dir` (PNGs; injective filenames per scene via `<scene>_im0.png`)

Common errors

Engine profile mismatch: Runtime batch size for evaluate or inference must fit within the TensorRT min/opt/max batch profile used during gen_trt_engine. Default profile in the spec template is min=1 / opt=1 / max=4.

Aspect-stretched predictions on variable-aspect datasets: forcing the engine input H/W to a single fixed shape distorts samples whose native aspect differs from that shape, degrading disparity quality. Build a separate engine per dataset (H, W) target close to the dataset's median aspect, multiple of 32. Per-image variable shape is not supported on the engine side.

Stereo inference 2-col GenericDataset: 2-column (left + right, no GT) annotation with dataset_name: GenericDataset is the supported inference path. Dataset-specific classes (Middlebury / Kitti / Eth3d / FSD / IsaacRealDataset / Crestereo) require 3-column input.

Deploy evaluate scalar conversion failure: In the current TAO Deploy image, TRT evaluate can complete prediction generation and then fail in stereo_evaluator.py with TypeError: only 0-dimensional arrays can be converted to Python scalars. Treat this as a deploy evaluator issue; the engine and inference handoff may still be valid.

Mounted paths do not exist: TAO Deploy checks local paths inside the container. Make sure every path in the spec has a matching Docker mount or job artifact mapping.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Declares allowed tools, Needs a command-line tool

The author says: Requires docker + nvidia-container-toolkit.

Security

What our scanner noticed· Medium risk

Large base64-encoded blobs

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

NVIDIA Corporation

Publisher

NVIDIA (Official)

Source

NVIDIA/skills · skills/tao-train-foundation-stereo · commit fce3cca

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE-APACHE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by NVIDIA Corporation or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download tao-train-foundation-stereo.zip