Tao Port Huggingface Model

NVIDIAOfficialCreated by NVIDIA Corporation

Skill
Workflow
Software development
Apache-2.0
Medium risk

From the author

Integrate a HuggingFace Computer Vision model into the NVIDIA TAO Toolkit ecosystem (tao-core config, tao-pytorch trainer, tao-deploy TensorRT pipeline). Use when the user asks to "integrate a HuggingFace model into TAO", "add an HF model to TAO Toolkit", "wire a HuggingFace ViT/DETR/ SegFormer into tao-pytorch", "build a TAO trainer + deploy pipeline for an HF CV model", or pastes a HuggingFace model URL/ID and wants it turned into a TAO model. Covers the full 7-phase loop: prerequisites check, HuggingFace inspection and validation, codebase exploration, tao-core configuration and native trainer implementation, ONNX export plus TensorRT deploy integration, packaging and L0 testing, container-based end-to-end validation, and (conditional) accuracy/latency tuning. Supports classification, object detection, semantic / instance / panoptic segmentation, zero-shot detection, and depth estimation.

— the description by NVIDIA Corporation

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured, seven-phase workflow for porting HuggingFace Computer Vision models into the NVIDIA TAO Toolkit. It guides the user through inspecting the model, configuring the TAO core, implementing the trainer, and setting up the deployment pipeline for TensorRT. The process emphasizes a local-only development loop, requiring the user to manage four independent repository clones and execute all validation, testing, and training tasks within locally prepared TAO Toolkit containers.

What it does

Validates HuggingFace model compatibility for computer vision tasks.
Configures TAO core and implements native trainers for PyTorch.
Integrates ONNX export and TensorRT deployment pipelines.
Automates packaging and L0 unit testing for model components.
Performs end-to-end validation within containerized environments.
Provides optional tuning for accuracy, latency, and model size.

When to use it

When you need to bring a custom HuggingFace computer vision model into the TAO Toolkit.
When you require a full training-to-deployment pipeline for a new model architecture.
When you need to verify model performance and exportability to TensorRT.
When you are working in a local environment with NVIDIA GPU support.

Advantages

Ensures consistency across training, export, and deployment stages.
Provides a rigorous, step-by-step debugging and testing loop.
Maintains environment isolation using containerized workflows.
Supports a wide range of computer vision tasks including detection and segmentation.

Usage examples

“Integrate a HuggingFace model into TAO.”: The agent initiates the Phase 0 prerequisite check to verify your environment and prepare the necessary TAO Toolkit container images.
“Wire a HuggingFace SegFormer into tao-pytorch.”: The agent begins the codebase exploration phase to map the model to the appropriate TAO reference architecture and prepare the configuration.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/nvidia/tao-port-huggingface-model/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip tao-port-huggingface-model.zip -d .agents/skills

Documentation by NVIDIA Corporation

Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).

TAO-HF Integration Skill

Integrate a HuggingFace (HF) Computer Vision model into the NVIDIA TAO Toolkit ecosystem. Work the phases iteratively — not purely linearly — via a build → test → debug → fix → retest loop at every step: when something fails, diagnose and fix before moving on; when it passes, move to the next step.

This SKILL.md is the workflow coordinator. Each phase has a dedicated references/phase-N-*.md with the full step-by-step content, code, docker invocations, and gates. Read the matching reference at the start of each phase — the summaries below are not sufficient.

Local-Only Rule

All work is strictly local. Do NOT push/commit/branch on any remote (GitLab, GitHub, HuggingFace), create merge/pull requests or issues, or upload/publish Docker images to any registry or artifact store. You may only read/clone from remotes — all edits, Docker builds, and test runs stay on the local machine.

Submodule Override Strategy

The user clones the four TAO repos (tao-core, tao-pytorch, tao-deploy, tao-dataservices) independently into one working directory. The tao-core/ submodule nested inside each repo points to the original unmodified commit; modifications only exist in the top-level tao-core/. Always install from the top-level tao-core/, never <repo>/tao-core/ — the nested submodule silently ignores all modifications. Override rules: (1) mount the working directory -v $(pwd):/workspace; (2) pip install /workspace/tao-core FIRST, before tao-pytorch/tao-deploy; (3) PYTHONPATH top-level tao-core first, e.g. -e PYTHONPATH=/workspace/tao-core:/workspace/tao-pytorch. See references/cross-cutting.md for the directory tree.

Execution platform

Every test, smoke run, and end-to-end validation executes inside a locally prepared TAO Toolkit container (tao-pytorch-base:latest, tao-deploy-base:latest, optionally tao-dataservices-base:latest — all from Phase 0). The platform skills own how to run them; this skill specifies what. Default platform: local-docker. Phase 0 delegates the driver / CUDA / NCT preflight to tao-setup-nvidia-gpu-host. See references/cross-cutting.md for the authoritative-skill table, bind-mount rationale, and canonical docker-run flag set.

Phase Map

Phase	Goal	Reference
0	Prerequisites + TAO Toolkit images + local image tags	phase-0-prereqs.md
1	Inputs, HF-inspection container, validate model + dataset	phase-1-inspection.md, hf-inspection.md
2	Closest existing TAO reference model	phase-2-codebase.md, task-type-guide.md
3	tao-core config + tao-pytorch trainer / eval / inference	phase-3-implementation.md, tao-patterns.md, repo-structure.md
4	ONNX export + tao-deploy TRT engine / inference / eval	phase-4-deploy.md
5	Packaging (`console_scripts`) + L0 tests	phase-5-packaging.md
6	Container testing + end-to-end validation	phase-6-container-tests.md, docker-patterns.md
7	(conditional) Accuracy / latency / size tuning	phase-7-optimization.md

Cross-cutting refs: workflow-consistency.md (CLI flow, config field paths, cross-phase dependencies); cross-cutting.md (platform, isolation, module pitfalls, debugging).

IMPORTANT — Continuous Execution Through Phase 6: do NOT stop after Phases 3–5 to wait for the user to run tests. Phase 6 is mandatory — not complete until tests pass inside the containers and the end-to-end pipeline is validated.

Development Loop

At every step: write code → test immediately (import check, unit test, or dry-run) → if it fails, read traceback → diagnose → fix → retest; if it passes, move on. Do NOT accumulate untested code — testing only at the end compounds bugs.

Debugging Playbook

When something fails, consult the symptom → likely-cause → fix table in references/cross-cutting.md before trying random fixes — it covers ModuleNotFoundError, BACKBONE_REGISTRY KeyError, shape mismatch, NaN loss, ONNX/TRT build failures, TRT-vs-PyTorch accuracy gaps, OOM, DDP hangs, checkpoint load failures, and stale-submodule config issues.

Environment Isolation Strategy

All Python work runs inside Docker containers — no host venvs, no pip installs into host Python (the host needs only Docker, from tao-setup-nvidia-gpu-host). Three contexts: A (Phase 1 HF inspection in tao-hf-inspect, python:3.12-slim fallback), B (Phase 3/4/6 smoke/L0/e2e in the prepared container, source via pip install /workspace/tao-core && python setup.py develop), C (host-bind-mount scratch). See references/cross-cutting.md for the contexts in full and the four numbered rules verbatim (--check-only host packages; Phase 1 --user $(id -u):$(id -g) vs. root; HOME=/workspace/PIP_USER=1 fallback; distro package-manager list; root:root trade-off; tao-hf-inspect cleanup).

Phase 0 — Prerequisites Check

Goal: verify Python 3.10+ and git; delegate the driver / CUDA / Docker / NVIDIA Container Toolkit host check to tao-setup-nvidia-gpu-host; verify NGC docker login for nvcr.io. Then ask the user for the TAO Toolkit image references (tao-pytorch, tao-deploy, optionally tao-dataservices), pull, and prepare local tags tao-pytorch-base:latest, tao-deploy-base:latest, tao-dataservices-base:latest for later phases — preparation removes the pre-installed released TAO packages so the user's /workspace/... clones install/load via pip install /workspace/tao-core && python setup.py develop. Hard stop on any failed check. Required user inputs: the image references + credentials (NGC login, HF_TOKEN). Full commands, prompt wording, and per-image Dockerfile snippets: the Phase 0 reference.

Gate: all prerequisite checks pass; the user supplied the required image references; tao-pytorch-base:latest and tao-deploy-base:latest exist locally; tao-dataservices-base:latest exists if dataservices work is anticipated.

Phase 1 — Information Gathering & Validation

Goal: decide whether to proceed at all. Gather credentials, locate/clone the four TAO repos, create a consistent working branch, launch the tao-hf-inspect container (Context A), validate the HF model is CV with a supported pipeline_tag, extract config + state-dict schema, sanity-check ONNX export, clean up. Full steps: Phase 1 references.

Reject if: pipeline_tag is NLP / audio / LLM (non-CV); AutoConfig raises; or ONNX export fundamentally cannot work (no rewrite path).

Gate: all 4 TAO repos located/cloned with a consistent branch; pipeline_tag confirmed CV; model_type, image_size, hidden_size, num_labels extracted; state-dict keys documented + HF→TAO remapping plan drafted; ONNX export sanity check passed (or failure understood); user confirmed model_short_name + task type. (Full checklist: phase-1-inspection.md.) Present findings and get user confirmation first.

Phase 2 — Codebase Exploration

Goal: find the closest existing TAO reference model for the detected pipeline_tag, read its implementation across tao-core / tao-pytorch / tao-deploy, and decide whether the backbone exists in backbone_v2/ or is new.

The HF pipeline_tag → TAO reference model mapping (classification → classification_pyt, detection → dino/rtdetr, segmentation → segformer, instance → mask2former, panoptic → oneformer, zero-shot → grounding_dino, depth → mono_depth) drives everything downstream (config, architecture, loss, ONNX shape, TRT builder, deploy classes, metrics, dataset format). See the Phase 2 references for the full reference list (12 files per model), the backbone_v2/ and tao-dataservices coverage checks, and per-task architecture.

If a new backbone is needed, decide the strategy (timm wrap > re-implement > HF black-box wrap) before Phase 3 — it changes weight loading, ONNX export, deploy. Never dual-inherit from transformers.PreTrainedModel and BackboneBase (metaclass conflict — compose instead).

Gate: reference TAO model identified + all 12 reference locations read; task-type implications understood (architecture, loss, ONNX outputs, deploy classes, metrics, dataset); backbone coverage decided (reuse / wrap timm / new); dataservices coverage checked. (Full checklist: phase-2-codebase.md.)

Phase 3 — TAO Core Configuration & Native Implementation

Goal: write the tao-core config schema + the tao-pytorch trainer / native inference / evaluation, smoke-testing between steps. (<model_name> = snake_case short-name; <ModelName> = PascalCase.)

Steps 1–7 (each builds on the previous, smoke-test between): tao-core config (1), tao-pytorch trainer (2), multi-GPU/multi-node (3), native inference → result.csv (4), native evaluation → results.json (5), MLOps for training and eval/infer → status.json (6–7). The ExperimentConfig(CommonExperimentConfig) must contain model, dataset, train, evaluate, inference, export, gen_trt_engine, quantize. All ??? fields are MISSING (user supplies via YAML/CLI); the augmentation.mean/std, model.head.in_channels, checkpoint-name, and onnx_file matches are in the checklist below.

Full per-step bodies, code, the canonical experiment_spec.yaml, and smoke-test commands: the Phase 3 references.

Gates: Step 1 — ExperimentConfig imports cleanly in-container; Step 2 — build_model(cfg) runs + PLModel instantiates in-container; Phase 3 — all 7 steps complete, smoke tests pass, no missing __init__.py.

Phase 4 — Export, Deployment & TensorRT Integration

Goal: ONNX export from tao-pytorch, then TRT engine builder + inference + evaluation in tao-deploy reusing the tao-core ExperimentConfig.

Steps 8–11: ONNX exporter (8 — task-specific input/output names, batch_size=-1 ⇒ dynamic batch); TRT engine builder (9 — subclass EngineBuilder or reuse ClassificationEngineBuilder; write specs/{gen_trt_engine,inference,evaluate}.yaml, same ExperimentConfig schema, augmentation.mean/std MUST match training); TRT inference → result.csv (10); TRT eval → results.json (11). See the Phase 4 reference for full code and the Phase 3+4 gate (3 in-container checks: imports, model build + forward, ONNX round-trip).

Module pitfalls: tao-pytorch and tao-deploy have separate hydra_runner and monitor_status — use the deploy versions in deploy scripts. ExperimentConfig comes from nvidia_tao_core in both (same schema/field paths).

Phase 3+4 gate: all three in-container checks pass (tao-pytorch imports + model + ONNX export; tao-deploy imports).

Phase 5 — Packaging & L0 Testing

Goal: register the model as a console_script in both repos and add unit tests.

Steps 12–15: register '<model_name>=...:main' in console_scripts of tao-pytorch/setup.py (12) and tao-deploy/setup.py (13, creating the deploy entrypoint/<model_name>.py via entrypoint_hydra); deploy L0 tests (14); trainer L0 tests — Trainer(..., fast_dev_run=True) + @pytest.mark.cv_unit @pytest.mark.<model_name> (15). See the Phase 5 reference for exact entry-point strings, code, and L0 test file lists.

Gate: entrypoints registered; pytest files exist and follow the marker convention. Do NOT stop — go directly to Phase 6.

Cross-Phase Data Flow & Consistency Verification

Before Docker testing, verify the chain train → export → gen_trt_engine → inference / evaluate (the *_model_latest.pth → .onnx → .engine artifact flow + the config fields each stage reads/writes — full diagram in references/cross-cutting.md).

Consistency checklist (verify before proceeding): self.checkpoint_filename → the *_latest.pth name evaluate.checkpoint / export.checkpoint reference; augmentation.mean/std identical across training spec, inference.yaml, evaluate.yaml, engine-builder preprocess_mode; ONNX input_names=['input'] / output_names=['output'] (detection/instance-seg use task-specific names); export.input_width/input_height match dataset.img_size; model.head.in_channels matches model_params_mapping.py; classes.txt at dataset.root_dir readable by both repos; all __init__.py exist (incl. scripts/__init__.py for get_subtasks() via pkgutil). Full paths: workflow-consistency.md.

Phase 6 — Container Testing & End-to-End Validation

Mandatory — start immediately after Phase 5. All TAO models ship as Docker images; code that only works outside a container is incomplete. Testing runs directly inside the TAO Toolkit container — no image build in the loop: mount → install source (setup.py develop) → run pytest / pylint / pydocstyle / flake8 directly. Use vanilla commands, NOT the ci/run_functional_tests.py / ci/run_static_tests.py wrappers (internal-mirror-only; public github.com/NVIDIA-TAO/ mirrors have no ci/ dir).

Steps 16–25: verify local image tags exist (16); unit tests for tao-core / tao-pytorch (-m cv_unit, --shm-size=16G) / tao-deploy (17–19); lint (20); wheels (21); end-to-end — train dry-run + export in one tao-pytorch session, then gen_trt_engine + inference + evaluate in one tao-deploy session (same session critical — --rm discards installs) (22); cross-check native vs TRT (23); debug shells (24); optional release images (25).

Full commands (every docker run, per-container env-vars, exact pytest / lint invocations + full-suite variants, the train/export/gen_trt_engine/inference/evaluate one-liner with all CLI overrides, the ci/ note, the fix-and-retest loop) and build scripts / runner patterns: see the Phase 6 references.

Phase 6 gate (Done criteria): tao-core / tao-pytorch / tao-deploy unit tests pass in their containers; static tests pass (or only legacy lint warnings); wheels build; end-to-end <model_name>_model_latest.pth → model.onnx → model.engine → non-empty result.csv + results.json; native vs TRT agree within tolerance.

Phase 7 — Optimization & Tuning (conditional)

Enter only if Phase 6 passes but accuracy / latency / size needs improvement. Ask the user for target metrics first.

Diagnostic categories: accuracy too low; TRT-vs-native gap; training too slow; inference too slow. Techniques: Step 27 — hyperparameter tuning; Step 28 — INT8 quantization (PTQ via torchao / modelopt, TRT INT8 + calibration); Step 29 — channel pruning + retrain; Step 30 — knowledge distillation; Step 31 — resolution tuning (TAO interpolates ViT positional embeddings automatically). See the Phase 7 reference for each category's checks, config blocks, YAML overrides, decision tree, and rationale.

Argument

$ARGUMENTS

If provided, interpret $ARGUMENTS as the HuggingFace model ID or URL to start Phase 1. If credentials or model short-name are not included, ask the user for them before proceeding.

Files

ATTRIBUTION.mdAdded by AutoGineer948 B
BENCHMARK.md7.0 KiB
LICENSELicence file added by AutoGineer11.2 KiB
SKILL.md17.7 KiB
config/skillspector-baseline.yaml2.6 KiB
eval.config3.2 KiB
evals/evals.json809 B
references/cross-cutting.md10.1 KiB
references/docker-patterns.md15.9 KiB
references/hf-inspection.md6.2 KiB
references/phase-0-prereqs.md9.9 KiB
references/phase-1-inspection.md10.1 KiB
references/phase-2-codebase.md7.3 KiB
references/phase-3-implementation.md34.7 KiB
references/phase-4-deploy.md12.7 KiB
references/phase-5-packaging.md3.3 KiB
references/phase-6-container-tests.md15.4 KiB
references/phase-7-optimization.md7.8 KiB
references/repo-structure.md10.6 KiB
references/tao-patterns.md20.1 KiB
references/task-type-guide.md16.4 KiB
references/workflow-consistency.md27.5 KiB
skill-card.md4.8 KiB
skill.oms.sig8.5 KiB

references/docker-patterns.md

Docker & Container Patterns Reference

Use this reference only when the parent SKILL.md points here for the current task. If this file conflicts with current SKILL.md, skill_info.yaml, schemas, or platform/model skills, the current authoritative source wins.

1. Dockerfile Locations
1. TAO Toolkit Container Images
1. Phase 0 Image Preparation Pattern
1. Build Scripts
1. Requirements Files
1. Wheel Build Patterns
1. Runner Scripts (Container Launchers)
1. CI Pipeline Patterns
- GitLab CI (.gitlab-ci.yml)
- Static lint (vanilla pylint / pydocstyle / flake8)
- Functional tests (vanilla pytest)
- Generating the docker run prefix
1. GPU & CUDA Environment Variables
1. Cross-Platform Detection
1. Container User Setup
1. Security Cleanup (Release Images)
1. TAO-Specific Environment Variables
1. Pytest Configuration
1. xformers / ONNX Runtime Build Memory Management

Concrete patterns extracted from the TAO repos for running tests in containers, building wheels, and (optionally) building release Docker images.

Note: For testing, we run directly inside the prepared TAO Toolkit containers (image tags built in Phase 0) — no Docker build is involved in the test loop. Release Docker images are optional and only for distribution validation. All work must be local only (--load, not --push). Do NOT push images to any registry.

Authority for generic flags: the --gpus, --shm-size=8g / --shm-size, -v host:container, -e VAR passthrough, container-name reuse, and docker inspect / docker logs patterns are owned by tao-skill-bank:tao-run-on-docker. The host GPU runtime (driver 580 / CUDA 13.0 / NVIDIA Container Toolkit 1.19.0) is owned by tao-skill-bank:tao-setup-nvidia-gpu-host. Patterns in this file only layer on the TAO-Toolkit-specific bits — image preparation, pip install /workspace/tao-core, setup.py develop, the per-repo pytest / lint / wheel invocations.

1. Dockerfile Locations

Repo	Development	Release	L4T (Jetson)
tao-pytorch	`docker/Dockerfile`	`release/docker/Dockerfile`	—
tao-deploy	`docker/Dockerfile`	`release/docker/Dockerfile.release`	`docker/Dockerfile.l4t`, `release/docker/Dockerfile.l4t.release`
tao-core	`Dockerfile` (multistage)	same	—
tao-dataservices	`docker/Dockerfile`	`release/docker/Dockerfile.release`	—

2. TAO Toolkit Container Images

The skill runs every test inside a TAO Toolkit container image on nvcr.io. Phase 0 asks the user for each image reference (tags vary per release), pulls them, and prepares them as local image tags that every other reference file uses everywhere:

Repo	Local tag (prepared in Phase 0)	Underlying TAO Toolkit image (user-supplied)
tao-core	`tao-pytorch-base:latest` (or `nvcr.io/nvidia/pytorch:24.03-py3`)	public NGC PyTorch image, or reuses the prepared tao-pytorch image
tao-pytorch	`tao-pytorch-base:latest`	tao-pytorch image (e.g. `nvcr.io/<org>/tao-toolkit:<version>-pyt`)
tao-deploy	`tao-deploy-base:latest`	tao-deploy image (e.g. `nvcr.io/<org>/tao-toolkit:<version>-deploy`)
tao-dataservices	`tao-dataservices-base:latest` (optional)	tao-dataservices image (e.g. `nvcr.io/<org>/tao-toolkit:<version>-data-services`)

The TAO Toolkit images are typically multi-arch manifests, so the same reference works on both x86_64 and aarch64 hosts — Docker auto-selects the matching layer. Detect arch with uname -m if needed (x86_64 → x86, aarch64 → ARM64).

Use these local tags directly as containers for testing — do NOT build Docker images from Dockerfiles for testing. TAO testing runs tests inside a TAO Toolkit container, not inside a Docker image built from a Dockerfile.

Full image-prep snippets: see phase-0-prereqs.md.

3. Phase 0 Image Preparation Pattern

The TAO Toolkit images come with the released TAO Python packages pre-installed. The skill installs the user's local clones of those packages on top at run time, so Phase 0 first removes the pre-installed copies via a tiny Dockerfile per component:

ARG PUB_IMAGE
FROM ${PUB_IMAGE}
# Remove pre-installed TAO packages so the local /workspace clones can be installed at run time.
RUN pip uninstall -y --quiet nvidia_tao_pytorch nvidia_tao_core 2>/dev/null || true
ENV PIP_DISABLE_PIP_VERSION_CHECK=1

Per-component pip uninstall lists (sourced from each repo's release/docker/Dockerfile{,.release}):

Local tag	`pip uninstall -y` list
`tao-pytorch-base:latest`	`nvidia_tao_pytorch nvidia_tao_core`
`tao-deploy-base:latest`	`nvidia_tao_deploy nvidia_tao_core`
`tao-dataservices-base:latest`	`nvidia_tao_ds nvidia_tao_pytorch nvidia_tao_core`

The 2>/dev/null || true keeps the build idempotent — packages absent from a particular image variant don't fail the preparation step.

Release Dockerfile multi-arch pattern (used by the TAO repos themselves to publish the TAO Toolkit images):

ARG TARGETARCH
ARG X86_DIGEST=sha256:...
ARG ARM64_DIGEST=sha256:...

FROM <upstream-image>@${X86_DIGEST} AS base-amd64
FROM <upstream-image>@${ARM64_DIGEST} AS base-arm64
FROM base-${TARGETARCH}

This skill does not invoke that pattern — it is shown only for reference when reading the TAO repos' release Dockerfiles.

4. Build Scripts

All repos share the same build script pattern at docker/build.sh:

# Usage:
./build.sh --build --x86                    # x86_64 only
./build.sh --build --arm                    # ARM64 only
./build.sh --build --multiplatform --push   # Both platforms
./build.sh --build --l4t                    # Jetson (tao-deploy only)
./build.sh --force --build --x86            # Force rebuild (no cache)

Key features:

QEMU setup for cross-platform builds (ARM on x86 host)
Auto-detection of host architecture via uname -m
Uses docker buildx for multi-platform: docker buildx build --platform linux/amd64,linux/arm64 --push
Single platform uses --load (loads into local daemon)
Sets DOCKER_BUILDKIT=1

For testing (recommended): Use the local image tag prepared in Phase 0 (tao-pytorch-base:latest, tao-deploy-base:latest, etc.) and run tests inside it directly with the source mounted — no Docker build needed:

docker run --rm --gpus all \
  -v $(pwd):/workspace \
  -w /workspace/tao-pytorch \
  -e PYTHONPATH=/workspace/tao-core:/workspace/tao-pytorch \
  tao-pytorch-base:latest \
  bash -c "pip install /workspace/tao-core && python setup.py develop && pytest tests/ -v --color=yes -m 'not slow'"

For release Docker images (distribution only):

# Uses the release Dockerfile, not docker/Dockerfile
cd tao-pytorch
docker build --network=host -t tao-pytorch-release:latest -f release/docker/Dockerfile .

cd tao-deploy
docker build --network=host -t tao-deploy-release:latest -f release/docker/Dockerfile.release .

5. Requirements Files

tao-pytorch/docker/requirements/:

File	Purpose
`requirements-apt.txt`	System packages (build-essential, ffmpeg, nginx, etc.)
`requirements-pip.txt`	Core Python packages (~80 deps)
`requirements-pip-pytorch.txt`	PyTorch ecosystem (fairscale, pytorch-lightning, etc.)
`requirements-pip-odise.txt`	ODISE model dependencies
`requirements-pip-quantization.txt`	Quantization tools
`requirements-pip-nvpanoptix3d.txt`	3D panoptic dependencies

tao-deploy/docker/requirements/:

File	Purpose
`requirements-apt.txt`	Minimal APT set (curl, ffmpeg, nginx)
`requirements.txt`	TensorRT/Deploy dependencies
`requirements-dev.txt`	Development environment (CUDA 12.8, cupy)
`requirements-l4t.txt`	Jetson-specific packages

6. Wheel Build Patterns

Build order: tao-core first (both tao-pytorch and tao-deploy depend on it).

CRITICAL — Submodule override: In CI, pip install tao-core/ installs from the submodule inside each repo. Locally, the submodule points to the original (unmodified) commit. Always install from our top-level tao-core/ clone instead: pip install /workspace/tao-core. See SKILL.md "Submodule Override Strategy".

# Standard wheel build
python3 setup.py bdist_wheel
pip3 install dist/*.whl

# Editable install (for development)
pip3 install -e .

Release builds use pyarmor obfuscation (via release/docker/build_wheel.sh):

pyarmor -d reg /release/docker/pyarmor-regfile-1219.zip
pyarmor -d gen --recursive --output /obf_src/ /nvidia_tao_pytorch/
python setup.py bdist_wheel

Makefile targets (tao-deploy):

make build       # python3 setup.py bdist_wheel (+ auditwheel repair for L4T)
make build_l4t   # L4T-specific wheel
make install     # pip3 install dist/*.whl

Wheel install inside container:

COPY dist/*.whl /opt/nvidia/wheels/
RUN cd wheels && ls ./*.whl | xargs -I'{}' python -m pip install '{}' && rm *.whl

7. Runner Scripts (Container Launchers)

Repo	Script
tao-pytorch	`runner/tao_pt.py`
tao-deploy	`runner/tao_deploy.py`
tao-dataservices	`runner/tao_ds.py`

What runners do:

Read docker/manifest.json to get registry/digest
Detect host platform: platform.machine() → x86_64 or aarch64
Configure GPU access based on Docker API version:
- Docker >= 1.40: --gpus all (or --gpus 'device=0,1')
- Docker < 1.40: --runtime=nvidia -e NVIDIA_DRIVER_CAPABILITIES=all
Read user mount config from ~/.tao_mounts.json
Set --shm-size 16G
Inject PYTHONPATH

Generated command pattern:

docker run -it --rm --gpus all \
  -v /path/to/data:/data \
  -e PYTHONPATH=/tao-pt:$PYTHONPATH \
  --shm-size 16G \
  tao-pytorch-base:latest \
  <model_name> train -e /path/to/spec.yaml

(tao-pytorch-base:latest is the local image tag prepared in Phase 0 from the user-supplied tao-pytorch TAO Toolkit image. Substitute tao-deploy-base:latest for deploy-side commands.)

8. CI Pipeline Patterns

GitLab CI (`.gitlab-ci.yml`)

Stages: mr-standards → static-tests → build-docker-image → check-jenkins-status

Static tests: Run pylint, pydocstyle, flake8 on merge requests
Docker build: Triggered on renovate/* and build-base-image/* branches; uses --cache-from with stable tag
Jenkins: Polls Jenkins job status for GPU-heavy functional tests

Static lint (vanilla `pylint` / `pydocstyle` / `flake8`)

The internal NVIDIA TAO mirrors ship a ci/run_static_tests.py helper that wraps pylint + pydocstyle + flake8 with module discovery and --changed-files-only plumbing. The public github mirrors do NOT ship this helper — invoke the tools directly instead:

# Errors only — fast path, suitable for the Phase 6 Step 20 gate
python -m pylint --errors-only nvidia_tao_pytorch/cv/<model_name>/

# Full report (uses repo .pylintrc if present)
pylint nvidia_tao_pytorch/cv/<model_name>/ $([ -f .pylintrc ] && echo --rcfile=.pylintrc)
pydocstyle nvidia_tao_pytorch/cv/<model_name>/
flake8 nvidia_tao_pytorch/cv/<model_name>/

Tools: pylint, pydocstyle, flake8 — install with pip install pylint pydocstyle flake8 if missing in the container.

Functional tests (vanilla `pytest`)

Same story: the internal ci/run_functional_tests.py wrapper adds testmon (incremental retest) and a CI-mode shortcut. The public mirrors do not ship it — invoke pytest directly:

# New model only (fast iteration)
pytest tests/cv_unit_test/<model_name>/ -v --color=yes -m 'cv_unit'

# Full functional suite, skip slow tests (equivalent to internal CI default)
pytest tests/ -v --color=yes -m 'not slow'

# Optional: incremental retest if you've installed testmon yourself
pip install pytest-testmon && pytest tests/ -v --color=yes --testmon

Generating the `docker run` prefix

The internal ci/utils.py exposes get_docker_information() (parses docker/manifest.json) and get_docker_command() (generates a docker run prefix with the right --gpus flag for the host's Docker API version). The public mirrors do not ship ci/utils.py — and this skill no longer parses manifest.json either, since Phase 0 produces the canonical local image tags. Use the patterns shown in §7 above to construct the docker run prefix manually, or invoke tao-pytorch-base:latest / tao-deploy-base:latest directly.

9. GPU & CUDA Environment Variables

# CUDA architecture targets
TORCH_CUDA_ARCH_LIST="7.5 8.0 8.6 9.0 9.0a 10.0 11.0 12.0+PTX"
GPU_ARCHS="75 80 86 90 90a 100 110 120"

# Library paths (platform-specific)
LD_LIBRARY_PATH="/usr/lib/$(uname -m)-linux-gnu:${LD_LIBRARY_PATH}"
# x86: /usr/lib/x86_64-linux-gnu
# ARM: /usr/lib/aarch64-linux-gnu

# CUDA include path (for ONNX Runtime build)
CPATH="/usr/local/cuda/include/cccl:${CPATH}"

# Deterministic CUDA ops
CUBLAS_WORKSPACE_CONFIG=":4096:8"

10. Cross-Platform Detection

In Dockerfiles:

RUN ARCH=$(uname -m) && \
    if [ "$ARCH" = "aarch64" ]; then \
        CUDA_TARGET="sbsa-linux"; \
    else \
        CUDA_TARGET="${ARCH}-linux"; \
    fi

Conditional builds (x86 only):

RUN if [ "$TARGETARCH" = "amd64" ]; then \
    git clone https://github.com/vllm-project/vllm.git && \
    pip install -e .; \
    fi

In runner scripts (Python):

import platform
arch = platform.machine()   # "x86_64" or "aarch64"
digest = manifest["digests"]["x86" if arch == "x86_64" else "arm"]

11. Container User Setup

ARG uid=1000
ARG gid=1000
RUN groupadd -r -f -g ${gid} taotoolkituser && \
    useradd -o -r -l -u ${uid} -g ${gid} -ms /bin/bash taotoolkituser && \
    usermod -aG sudo taotoolkituser

12. Security Cleanup (Release Images)

# Remove git history and CI configs
RUN find / -type d -name ".git" -exec rm -rf {} + && \
    find / -type f -name ".gitlab-ci.yml" -exec rm -f {} +

13. TAO-Specific Environment Variables

ENV NVIDIA_PRODUCT_NAME="TAO Toolkit"
ENV TAO_TOOLKIT_VERSION="6.25.10"
ENV NVIDIA_TAO_TOOLKIT_VERSION="${TAO_TOOLKIT_VERSION}-pytorch"  # or -deploy
ENV TAO_TELEMETRY_SERVER="..."
ENV DEBIAN_FRONTEND=noninteractive
ENV TZ="America/New_York"
ENV PYTHONUNBUFFERED=1

14. Pytest Configuration

tao-pytorch/pytest.ini:

[pytest]
addopts = --verbose --pyargs --durations=0
markers =
    unit: unit tests
    train: training scripts
    finetune: fine-tuning scripts
    evaluate: evaluation scripts
    export: ONNX export
    infer: inference
    tensorrt: TensorRT tests
    cv_unit: CV unit tests

15. xformers / ONNX Runtime Build Memory Management

The tao-pytorch Dockerfile calculates build parallelism based on available memory:

# xformers: max_jobs = (total_mem_gb - 90) / 20 (min 4, max nproc)
MAX_JOBS=$(python3 -c "
import os, psutil
mem = psutil.virtual_memory().total / (1024**3)
jobs = max(4, min(os.cpu_count(), int((mem - 90) / 20)))
print(jobs)
")

If builds OOM, reduce parallelism by limiting MAX_JOBS or ONNXRUNTIME_BUILD_PARALLEL.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Declares allowed tools, Needs Python, Needs Node.js

The author says: Requires Python 3.10+, NVIDIA driver, CUDA 13.0+, docker + nvidia-container-toolkit, an NGC API key (`docker login nvcr.io`), an HF_TOKEN, and access to the TAO Toolkit container images on `nvcr.io` for `tao-pytorch`, `tao-deploy`, and (optionally) `tao-dataservices` — Phase 0 asks the user for the exact image references and prepares them locally as `tao-pytorch-base:latest`, `tao-deploy-base:latest`, `tao-dataservices-base:latest`. Local clones of `tao-core`, `tao-pytorch`, `tao-deploy`,...

Security

What our scanner noticed· Medium risk

Large base64-encoded blobs
Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

NVIDIA Corporation

Publisher

NVIDIA (Official)

Source

NVIDIA/skills · skills/tao-port-huggingface-model · commit fce3cca

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE-APACHE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by NVIDIA Corporation or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download tao-port-huggingface-model.zip