AutoGineer
Generated banner for Tao Finetune Huggingface Model
Generated logo for Tao Finetune Huggingface Model

Tao Finetune Huggingface Model

NVIDIAOfficialCreated by NVIDIA Corporation

  • Skill
  • Workflow
  • Software development
  • Apache-2.0
  • Medium risk
Download ZIP

From the author

Fine-tune any HuggingFace CV / VLM / LLM model on local NVIDIA GPUs inside an NGC PyTorch container when no dedicated TAO model skill matches. Use when the user wants to fine-tune a HuggingFace model (full or LoRA), train a vision / VLM / LLM model end-to-end, generate a reproducible HF training pipeline, smoke-test a HuggingFace model locally before scale-up, push a fine-tuned model to the HF Hub with a model card, or emit a self-contained rerun skill for an existing HuggingFace finetune. Supports image classification, object detection, semantic / instance / panoptic segmentation, depth estimation, image-text-to-text VLM (SFT / LoRA), and LLM SFT / DPO / GRPO. Six-step workflow: inspect and qualify, hardware and NGC image, research, generate and smoke, train + eval + infer, push and emit rerun skill. Do not use for any Hugging Face model ID claimed by a dedicated `skills/models/*` skill; the model skill and its declared execution environment take precedence.
— the description by NVIDIA Corporation

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured, six-step workflow for fine-tuning HuggingFace CV, VLM, and LLM models on local NVIDIA GPUs. It leverages NGC PyTorch containers to ensure environment consistency and reproducibility. The workflow includes automated preflight checks, live research of model-specific recipes, smoke testing on real data, and optional integration with HuggingFace Hub for model sharing. It is designed to be the fallback for models not covered by dedicated TAO skills, ensuring a rigorous, documented path from inspection to deployment.

What it does

  • Inspects and qualifies models and datasets for compatibility.
  • Audits local hardware and selects the appropriate NGC PyTorch container.
  • Performs live research to fetch model-specific training recipes and configurations.
  • Generates a complete, reproducible project structure including training and evaluation scripts.
  • Executes a mandatory smoke test on real data before full-scale training.
  • Pushes fine-tuned models to the HuggingFace Hub and emits a self-contained rerun skill.

When to use it

  • When you need to fine-tune a HuggingFace model and no dedicated TAO skill exists for it.
  • When you require a reproducible, containerized training pipeline on local NVIDIA hardware.
  • When you need to smoke-test a model locally before scaling to larger infrastructure.
  • When you want to automate the generation of model cards and training reports.

Advantages

  • Ensures environment consistency using NVIDIA-optimized NGC containers.
  • Prevents common training failures through mandatory smoke testing and preflight checks.
  • Reduces manual configuration by automating script generation and recipe research.
  • Provides a clear, auditable path from initial inspection to final model deployment.

Usage examples

“Fine-tune google/vit-base-patch16-224 on my local image dataset.”
The agent will inspect the model and dataset, verify hardware, generate the training project, run a smoke test, and initiate the fine-tuning process.
“Train a Llama-3-8B model using LoRA on my local GPU.”
The agent will research the appropriate LoRA configuration, set up the NGC environment, and execute the training pipeline with the specified parameters.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/nvidia/tao-finetune-huggingface-model/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip tao-finetune-huggingface-model.zip -d .agents/skills

Documentation by NVIDIA Corporation

<!-- Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. Licensed under the Apache License, Version 2.0; see http://www.apache.org/licenses/LICENSE-2.0 -->

tao-finetune-huggingface-model

Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).

Local NVIDIA GPU fine-tuning for HuggingFace models, grounded in live-fetched documentation with curated references as a fallback safety net. One NGC container, a few focused scripts, one push to HF Hub. Follow the rules in this file; don't improvise.

Dedicated-model routing gate

Before Step 1 or any probe, image selection, package install, venv creation, or training-code generation, resolve model_id against the packaged model-owner registry. Use the absolute skill-bank root from which this file was loaded:

python <bank-root>/scripts/resolve_tao_model.py \
  --skill-bank <bank-root> \
  --model "$MODEL_ID" \
  --format json

The resolver matches model metadata, including huggingface_model_ids, network_arch, skill names, and legacy aliases. Routing is internal: a model ID and task are enough. Never require prompt boilerplate about skills, containers, or checkpoint formats.

  • Exit 0: stop this workflow and follow the owning model skill's environment, action metadata, preflight, and checkpoint preparation.

  • Exit 3: no packaged model skill owns the ID. This is the only result that permits Step 1 of the generic workflow.

  • Any other nonzero exit: ownership discovery is broken or ambiguous. Stop and resolve that error; do not silently fall back to generic Hugging Face training.

Hugging Face hosting never overrides ownership. Do not use this workflow to bypass a matched skill or ask the user to prescribe its internal preparation. For example, nvidia/Cosmos3-Nano routes to tao-finetune-cosmos-reason.

Do not create a host training venv in this workflow. Its default execution path is the NGC container documented below; any venv-based training path requires an explicit user request.

Order of authority (highest first):

  1. User input — explicit model_id, dataset_id, training_method, config.yaml overrides.

  2. Live research — model card, HF repo example, author finetune script, HF task docs, paper; always fetched (Step 3 + references/research-priorities.md).

  3. Curated references (references/*.md) — fallback when live research is silent/ambiguous.

  4. Your training-data memory — last resort; suspect, cross-check against (2)/(3).

Conflict resolution between (2) and (3) and the source-line discrepancy note are in references/research-priorities.md.

Inputs

Required:

  • model_id — HuggingFace model ID, e.g. google/vit-base-patch16-224

Conditional credentials (read from the session environment — exported before launching or sourced from a user-approved env file):

  • HF_TOKEN — only when the model/dataset is gated (read) or push_to_hub is on (write); public + public + push_to_hub: false needs none. Value never read — presence-only via [ -n "$HF_TOKEN" ].

  • WANDB_API_KEY, WANDB_PROJECT — only when WandB is enabled; WANDB_MODE=disabled opts out.

Dataset — exactly one:

  • dataset_id — HuggingFace dataset ID (source: hf)

  • local_dataset_path — local folder or file (source: local); optional local_dataset_format ∈ {auto, imagefolder, coco, voc, jsonl, arrow, parquet, csv} (default: auto-detect).

  • (omit) — agent recommends popular datasets (source: recommend)

Optional (have defaults):

  • task_type — auto-detected from config + model card

  • n_train=10000, n_eval=1000, n_epochs=3, lora_r=16

  • output_dir=./output/<model_short_name>

  • hf_model_repo — push target; if unset and HF_TOKEN has write access, auto-derived as <whoami>/<model_short_name>-finetuned.

  • push_to_hub=True — set to False to skip

  • skip_baseline=False — skip zero-shot baseline eval

Optional deliverables (off by default):

emit_progress_log: false   # output_dir/PROGRESS.md (per-step journal)
emit_report:       false   # reports/report.{pdf,html} with curves & samples
emit_unit_tests:   false   # tests/ with fake-data heterogeneous-batch tests

All values live in output_dir/config.yaml. Never hardcode in Python.

Execution platform

This skill orchestrates what to run; the platform skills own how to run it on a GPU host — read them first.

ConcernAuthoritative skill
GPU host runtime (driver 580, CUDA Toolkit 13.0, NVIDIA Container Toolkit 1.19.0)tao-skill-bank:tao-setup-nvidia-gpu-host
docker run flags, NGC auth, mounts, env passthrough, local/remote Docker job preflight (daemon, GPU smoke)tao-skill-bank:tao-run-on-docker

Default platform: local-docker — build a one-off image (run-<short>:latest) and run it on the local Docker daemon. Ask only when the user explicitly needs a different backend (Brev remote GPU, SLURM/Kubernetes); then run that platform's Preflight first and route the Steps 4–5 docker run commands through it. The GPU-runtime and presence-only credential preflights (values never read), the canonical docker run flag set, discovery of the execution platforms from the installed platform skills (tao-run-on-docker / -slurm / -kubernetes / -brev, plus any external one; on a runtime that surfaces only the core router skills, read skills/platform/tao-run-on-*/SKILL.md frontmatter), and the workflow-specific flags (--entrypoint /bin/bash -lc, PYTORCH_CUDA_ALLOC_CONF, --name hft_train) are in references/workflow-intake-preflight.md.

References — fallback safety net

Consulted only when live research is silent, ambiguous, or unavailable; live docs always win for the specific model and current API. Each step links the references it needs; full catalog in references/detailed-workflow.md.

Always-on: core-rules.md, error-playbook.md, compat-workarounds.md, model-discovery.md, dataset-recommendations.md, dataset-sources.md, dataset-patterns.md, hardware-container.md, research-priorities.md, cv-scripts.md, vlm-scripts.md, docker-runs.md, hub-push.md, pipeline-skill-template.md, deliverables.md. Opt-in (when their flag/need applies): progress-tracking.md, testing.md, reporting.md, workflow-intake-preflight.md, workflow-generate-train.md, workflow-push-rerun.md.

Rule: before falling back, log the live source you tried and why it was insufficient (config.yaml notes:, and PROGRESS.md if enabled). [FETCH LIVE] markers in cv-scripts.md / vlm-scripts.md are a research checklist, not code to inline — refetch the listed URL if a block has no Step 3 finding.

Core rules

Non-negotiable behaviors. Short version (full enumeration — hallucinated-imports list, never-without-approval list, full error-recovery and hardware-sizing tables — in references/core-rules.md, consult before any training-time decision):

  • Your HF-library knowledge is outdated. Fetch live docs (model card, HF repo example, task doc) before writing any ML code — don't generate trainer args / collator / transforms from memory (Step 3).

  • Smoke-test on real data with --max_steps 1 before any full run; no batch launches without a verified smoke.

  • Never silently substitute model_id, dataset_id, or training_method — if what the user asked for doesn't load, stop and ask.

  • Error recovery is minimal-change. OOM → halve batch, double grad_accum, enable gradient checkpointing (no LoRA switch without approval); NaN → reduce LR 10×; flat loss → inspect collator; same error 3× → stop and ask. Don't loop.

  • Dataset columns verified BEFORE the collator — rename in prepare_data.py; restructuring needed → stop and ask.

  • Hardware-sizing thumb (bf16): ≤3B → 24 GB, 7–13B → 80 GB, 30B+ → multi-GPU or LoRA on 1× 80 GB, 70B+ → 8× 80 GB or LoRA. Full finetune won't fit and no LoRA requested → ask before switching.

Workflow — 6 steps

Single pass, sequential; each step has a clear gate before the next begins.

Step 1 — Inspect & qualify

Goal: decide whether to proceed. Probe model + dataset, apply accept/reject, register applicable compat fixes, write the initial config.yaml.

Prerequisites: MODEL_ID, optional DATASET_ID / local_dataset_path, optional HF_TOKEN, OUTPUT_DIR (default ./output/<model_short_name>). Probes run in a CPU-only python:3.12-slim Docker container (bind-mounted .probe/ scratch) so the host needs no virtualenv — Docker must exist first. Docker-presence guard, container env, full probe invocation, and the model/dataset probe scripts are in references/workflow-intake-preflight.md, references/model-discovery.md, and references/dataset-sources.md.

Probe requirements:

  • Model: load AutoConfig, read model-card tags, detect task from architectures + tags + card examples (fallback logging in model-discovery.md).

  • Dataset: for recommended datasets, first present 3-5 choices from dataset-recommendations.md; for local data, bind-mount read-only and use dataset-sources.md format detection.

  • Reject early if the model config fails, the task is out of scope, no recipe source exists, or the dataset cannot load / match the task schema.

  • Evaluate compat-workarounds.md against the model/task; defer hardware-dependent rules to Step 2.

Write the initial config.yaml (model_id, task, dataset_id or local_dataset_path, research_sources: [] filled in Step 3, applicable_workarounds: from Step 1, notes: [] for reference fallbacks, push_to_hub: true default — annotated template in references/workflow-intake-preflight.md). Optionally rm -rf "$OUTPUT_DIR/.probe" once the gate is met.

Gate: config.yaml exists with model, dataset, task, applicable_workarounds; do not proceed if any field is missing.

Step 2 — Hardware audit & NGC image

Goal: verify Docker + GPU + disk, pick the NGC PyTorch image live, finalize hardware-dependent compat rules.

2a. Audit (hard gate) — three checks (commands in references/workflow-intake-preflight.md):

  1. GPU host runtime — tao-setup-nvidia-gpu-host's setup-nvidia-gpu-host.sh --backend docker --check-only; on fail, ask approval then re-run with --install --yes.

  2. Free-disk soft-warn — override via MIN_DISK_GB (default 100 GB); recommend ≥ 100 GB for NGC base (~20 GB) + HF cache + checkpoints + data.

  3. Conditional credential presence (values never read) — HF_TOKEN only when gated or push_to_hub is on; WANDB_* only when WandB is on.

Do not proceed to Step 4 on a hard-fail — Step 4's docker build pulls a 20+ GB NGC base, and a missing nvidia-container-toolkit only surfaces later as could not select device driver "" with capabilities: [[gpu]]. Record gpu_count, gpu_name, driver_major, vram_gb_per_gpu in config.yaml.

2b. Pick NGC image (live): from the NVIDIA Deep Learning Frameworks support matrix (https://docs.nvidia.com/deeplearning/frameworks/support-matrix/index.html), PyTorch NGC container section, pick the highest-versioned image where Min driver ≤ detected driver_major and container CUDA ≤ host CUDA Toolkit (match closely so cuDNN / TensorRT line up). Do not reject an image for an aN/bN/rcN PyTorch tag — NGC validates the full image; pick the newest CUDA-aligned one and let compat-workarounds.md handle per-version issues. If the matrix is unreachable, use the fallbacks in references/hardware-container.md; default nvcr.io/nvidia/pytorch:24.09-py3 <!-- unpinned: documented fallback --> (driver ≥ 545; SDPA+GQA bug — if num_key_value_heads < num_attention_heads, set attn_implementation: "eager"). Record ngc_image in config.yaml.

2c. Re-evaluate hardware-dependent compat rules: re-run the compat-workarounds.md walk for entries whose detect needs hw; update applicable_workarounds: in place.

2d. Model-fit check: estimate param_bytes ≈ 2×param_count (bf16); if

60% of vram_gb_per_gpu × 1e9, recommend LoRA in the user-facing summary.

Gate: config.yaml has ngc_image, gpu_count, gpu_name, driver_major, vram_gb_per_gpu; hardware-dependent compat fixes recorded.

Step 3 — Research the recipe

Goal: fetch the live recipe — training-data knowledge of transformers/trl/peft is suspect, so Step 3 is non-negotiable. Walk references/research-priorities.md in priority order (Priority 1 → 6); stop once you have, for the detected task:

  • AutoModel / processor class

  • Train + eval transforms

  • Collator

  • compute_metrics

  • Hyperparameter hints (LR, batch size, epochs, scheduler)

Record findings in meta/recipe.md, append source URLs to config.yaml: research_sources:. A slot with no live finding falls back to the matching scaffold (cv-scripts.md / vlm-scripts.md), logged as "fallback to scaffold — no live source for <slot>" under notes:. Conflict-resolution rules are in references/research-priorities.md.

Gate: every required slot filled, with a source URL or scaffold-fallback note.

Step 4 — Generate project & smoke-test

Goal: write all scripts, build the image, prepare data, run a 1-step smoke on real data (one docker build, two docker runs).

4a. Generate project files in output_dir/: config.yaml, Dockerfile, requirements.txt, prepare_data.py, train.py, run_eval.py, infer.py, optional merge_lora.py, optional tests/, .gitignore. Live Step 3 research is authority; cv-scripts.md / vlm-scripts.md give scaffold shape only. Apply every applicable_workarounds entry as a Dockerfile block, requirement pin, config override, or runtime env var. Hard rules: run_eval.py keeps that exact filename (avoids colliding with the HF evaluate package); every generated .py starts with the NVIDIA Apache-2.0 copyright header and any emitter fails when it is missing; emit_unit_tests: true generates and runs tests per references/testing.md. Script bodies, Dockerfile shape, and the emitter contract are in references/workflow-generate-train.md.

4b. Build, prepare, smoke — docker build -t run-<short>:latest ., then prepare_data and the --smoke --max_steps 1 run (references/docker-runs.md §1-3). Smoke pass criteria (in logs/smoke.log):

  • No exception

  • Loss is finite (not 0.0, not NaN)

  • grad_norm > 0 at step 1

If emit_unit_tests: true, also run pytest tests/ in the container. Any failure → STOP.

4c. Preflight summary — before full training, print and verify: reference URL, dataset columns, Hub target, monitoring target, NGC image, hardware, smoke loss/grad norm.

Gate: project files written, image built, smoke PASSED, preflight has no blank fields.

Step 5 — Train, evaluate, infer

Goal: baseline eval, full training, post-train eval, optional LoRA merge, 5 inference samples (all commands: references/docker-runs.md §4-8).

Sub-stepdocker-runs.mdSkip if
5a. Baseline eval (zero-shot)§4skip_baseline: true
5b. Full training (detached)§5—
5c. LoRA merge§6not VLM+LoRA
5d. Post-train eval§7—
5e. Inference (5 samples)§8—

Multi-GPU: prepend torchrun --nproc_per_node=$gpu_count to python train.py.

While training streams, watch docker logs -f hft_train: loss should drop within 10-20 steps; flat loss (collator/label-masking bug), NaN (LR too high), and OOM all stop the run — recovery in references/core-rules.md. If emit_report: true, run report.py after Step 5e per references/reporting.md.

Gate: all of:

  • checkpoints/final/ (or checkpoints/merged/ for LoRA) exists

  • reports/eval_results.json has a numeric primary metric

  • reports/baseline_results.json exists (unless skipped)

  • reports/inference_samples/ has 5 samples

  • wandb URL shows descending loss

Step 6 — Push & emit rerun skill

Goal: publish the run and make it reproducible without re-research.

Push per references/hub-push.md (weights, model card, eval/baseline JSONs, config.yaml, Dockerfile, requirements.txt, inference samples, reports when emitted) unless push_to_hub: false is explicit. Emit <output_dir>/skills/run-<short>/SKILL.md from references/pipeline-skill-template.md — substitute every placeholder, include full YAML metadata + the NVIDIA copyright HTML comment, and make any emitter fail if those are missing.

Gate (Done criteria): all of:

  • Step 5 gate met

  • HF Hub repo exists at the resolved URL with weights + card + results/ (unless push_to_hub: false)

  • <output_dir>/skills/run-<short>/SKILL.md exists, no <placeholder> left, with metadata + copyright HTML comment per pipeline-skill-template.md

Final message: wandb URL, HF Hub URL, baseline -> fine-tuned primary metric, reports/inference_samples/, and the rerun skill path.

Error playbook

On a known runtime error, consult the symptom → minimal-fix table in references/error-playbook.md (NGC entrypoint, PyTorch/Transformers regressions, numpy ABI, Albumentations bbox, PEFT/checkpointing, LoRA target breadth, CV augmentation gaps, OOM at step 0) before redesigning anything. When a row there fires twice across runs, lift it into compat-workarounds.md with a detect rule — auto-applied in Step 1 before the error can fire.

Communication style

  • Terse. No filler, no restating the request; one-word answers when appropriate.

  • Always include direct Hub and wandb URLs when referencing artifacts.

  • On error: state what went wrong, why, what you changed — no menus.

  • Never present "Option A/B/C" for a request with a clear answer. Act.

Files

references/compat-workarounds.md

<!-- Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->

Compatibility Workarounds Registry

Use this reference only when the parent SKILL.md points here for the current task. If this file conflicts with current SKILL.md, skill_info.yaml, schemas, or platform/model skills, the current authoritative source wins.

Contents

  • Entry format

  • Registry

      1. idefics3-llama-generate

      1. pytorch-2.5-sdpa-gqa

      1. hf-hub-xet-hang

      1. vlm-heterogeneous-pixel-values

      1. script-name-evaluate

      1. vlm-image-token-truncation

      1. pip-cache-purge-ngc

      1. wheel-find-packages-empty

  • Phase 0.5 runner (pseudocode)

  • Phase 4.3 Dockerfile injection (pseudocode)

  • Invariants

  • When to add to this registry

Known HuggingFace / PyTorch / NVIDIA ecosystem incompatibilities with auto-detection rules. Consulted in Phase 0.5 to write meta/phase0_compat.yaml. Phase 4.3 reads that file and injects only the applicable fixes into the generated Dockerfile or config.

Adding a new workaround: when you hit a new bug and fix it for one project, add an entry here. Future generated projects auto-inherit the fix.

Entry format

id: <short-kebab-case-id>
title: <one line>
symptom: <exact error the user sees>
root_cause: <one line>
detect:                            # any/all field = Python expression on phase0 vars
  any:                             # match if ANY expression is True
    - "cfg.model_type == 'idefics3'"
    - "getattr(cfg, 'text_config', None) and cfg.text_config.model_type == 'llama'"
fix:
  type: dockerfile_block | config_override | requirements_pin | runtime_env | dataset_template | naming_rule
  content: |
    <text/dict to inject>
references: [<links or PRs>]

Detection vars available in detect expressions:

  • cfg — AutoConfig.from_pretrained(model_id) result

  • hw — meta/phase1_hardware.yaml dict (ngc_image, driver_major, gpu_name, etc.)

  • task — detected task string (image-classification, image-text-to-text, ...)

Registry

1. idefics3-llama-generate
id: idefics3-llama-generate
title: Idefics3/Llama-backed VLM generate() broken on transformers ≥ 4.51
symptom: |
  TypeError: Missing `**kwargs` in the signature of the `@check_model_inputs`-decorated
  function (LlamaModel.forward)
root_cause: |
  transformers 4.51 introduced @check_model_inputs which expects **kwargs in the wrapped
  forward. Idefics3 wraps LlamaModel as text_model; LlamaModel.forward doesn't advertise
  **kwargs in its signature, so the decorator raises at generate() time.
detect:
  any:
    - "cfg.model_type in {'idefics3', 'mllama', 'llava', 'llava_next'}"
    - "getattr(cfg, 'text_config', None) is not None and getattr(cfg.text_config, 'model_type', '') == 'llama'"
fix:
  type: dockerfile_block
  content: |
    # Workaround idefics3-llama-generate — transformers ≥ 4.51 breaks LlamaModel.forward
    RUN pip install --no-cache-dir --force-reinstall --no-deps \
        transformers==4.49.0 tokenizers==0.21.0 "huggingface-hub>=0.26,<1.0"
references:
  - https://github.com/huggingface/transformers/issues/35928
2. pytorch-2.5-sdpa-gqa
id: pytorch-2.5-sdpa-gqa
title: PyTorch 2.5.0 SDPA crashes on grouped-query attention
symptom: |
  TypeError: scaled_dot_product_attention() got an unexpected keyword argument 'enable_gqa'
root_cause: |
  PyTorch 2.5.0 (shipped in NGC 24.09-py3) called SDPA without checking GQA support. Fixed
  in 2.5.1. Affects any model with num_key_value_heads < num_attention_heads (Llama 3,
  Mistral, Qwen2, Gemma, etc.).
detect:
  all:
    - "'24.09-py3' in hw['ngc_image']"
    - "getattr(cfg, 'num_key_value_heads', None) is not None and cfg.num_key_value_heads < cfg.num_attention_heads"
fix:
  type: config_override
  content:
    attn_implementation: "eager"
references:
  - https://github.com/pytorch/pytorch/pull/137524
3. hf-hub-xet-hang
id: hf-hub-xet-hang
title: HF Hub Xet CDN downloads hang at 0 bytes intermittently
symptom: |
  prepare_data.py or model load stalls at 0 bytes on a .incomplete blob for many minutes
root_cause: |
  HuggingFace's Xet CDN (new dedup-aware storage) has intermittent timeouts on some routes.
  The legacy LFS path is reliable.
detect:
  any:
    - "True"                   # always apply (benign)
fix:
  type: runtime_env
  content:
    HF_HUB_DISABLE_XET: "1"
references:
  - https://github.com/huggingface/huggingface_hub/issues/2700
4. vlm-heterogeneous-pixel-values
id: vlm-heterogeneous-pixel-values
title: VLM pixel_values can have variable num_images per sample — collator must pad
symptom: |
  AttributeError: 'list' object has no attribute 'shape'
  — raised inside Idefics3/SmolVLM get_image_features when pixel_values couldn't be stacked
root_cause: |
  Idefics3-family processors split high-res images into tiles; the leading dim `num_images`
  can differ between samples in a batch. A naive torch.stack collator returns a Python list,
  which the model then dereferences as a Tensor.
detect:
  any:
    - "cfg.model_type in {'idefics3', 'idefics2', 'mllama'}"
    - "task == 'image-text-to-text' and getattr(cfg, 'do_image_splitting', False)"
fix:
  type: dataset_template
  content:
    collator: collate_vlm
    sample_padding: none
    rationale: "Use the heterogeneous-safe collator from references/vlm-scripts.md §collate_vlm"
references:
  - This skill's own production bug — caught by Phase 4.5 after the fact
5. script-name-evaluate
id: script-name-evaluate
title: Script named evaluate.py collides with HF evaluate library
symptom: |
  ImportError: cannot import name 'main' from 'evaluate'
  (at wheel install time or first hft-eval call)
root_cause: |
  The HF `evaluate` library is installed as a top-level `evaluate` module in site-packages.
  A wheel whose entry point points to `evaluate:main` gets shadowed at runtime.
detect:
  any:
    - "True"
fix:
  type: naming_rule
  content:
    script_file: run_eval.py
    entry_point: "hft-eval=run_eval:main"
references:
  - Enforced by the skill at generation time
6. vlm-image-token-truncation
id: vlm-image-token-truncation
title: VLM truncation=True at max_length<prompt_length cuts mid-image-token
symptom: |
  ValueError: Mismatch in `image` token count between text and `input_ids`.
root_cause: |
  Idefics3 / Qwen2-VL / LLaVA-Next expand an image into ~800+ text tokens in the prompt.
  A dataset __getitem__ that sets truncation=True with max_length<prompt_length cuts
  inside the image-token span, breaking alignment with pixel_values.
detect:
  any:
    - "task == 'image-text-to-text'"
fix:
  type: dataset_template
  content:
    sample_truncation: false
    sample_padding: none
    rationale: "No truncation/padding in __getitem__ — let collator handle batch padding"
references:
  - See vlm-scripts.md VLMDataset template
7. pip-cache-purge-ngc
id: pip-cache-purge-ngc
title: pip cache purge fails in NGC base images
symptom: |
  ERROR: pip cache directory is not configured
  — during Docker build when `RUN pip cache purge` is executed
root_cause: |
  NGC PyTorch images disable pip's cache by default to keep images small. Any Dockerfile
  line calling `pip cache purge` fails.
detect:
  all:
    - "'nvcr.io/nvidia/pytorch' in hw['ngc_image']"
fix:
  type: dockerfile_lint
  content:
    forbidden_lines: ["pip cache purge"]
references:
  - NGC release notes
8. wheel-find-packages-empty
id: wheel-find-packages-empty
title: setup.py find_packages() returns empty for flat-script projects
symptom: |
  Wheel builds successfully but is < 5 KB; entry points fail with ModuleNotFoundError at install
root_cause: |
  setuptools.find_packages() requires each Python file to live in a directory with __init__.py.
  The skill generates flat top-level scripts (train.py, model.py, ...), so find_packages()
  returns [] and the wheel ships zero code.
detect:
  any:
    - "True"
fix:
  type: naming_rule
  content:
    setup_py_modules: py_modules
    forbid: find_packages()
references:
  - See references/packaging.md

Phase 0.5 runner (pseudocode)

# Inputs
cfg  = AutoConfig.from_pretrained(model_id, trust_remote_code=True)
hw   = yaml.safe_load(open("meta/phase1_hardware.yaml"))
task = yaml.safe_load(open("meta/phase0_model_info.yaml"))["task"]

# Load registry (parse this .md file's YAML blocks, or maintain a parallel .yaml)
registry = load_compat_registry()

def match_rules(rules, **ctx):
    """Evaluate detect expressions. `any`/`all` semantics."""
    if "any" in rules:
        return any(eval(expr, {}, ctx) for expr in rules["any"])
    if "all" in rules:
        return all(eval(expr, {}, ctx) for expr in rules["all"])
    return False

applied = []
skip = set(yaml.safe_load(open("config.yaml")).get("skip_workarounds", []))
for entry in registry:
    if entry["id"] in skip:
        continue
    if match_rules(entry["detect"], cfg=cfg, hw=hw, task=task):
        applied.append(entry)

yaml.safe_dump({"applicable_workarounds": applied},
               open("meta/phase0_compat.yaml", "w"))

for a in applied:
    log_progress(f"Compat: applying '{a['id']}' — {a['title']}", status="⚠️")

Phase 4.3 Dockerfile injection (pseudocode)

compat = yaml.safe_load(open("meta/phase0_compat.yaml"))["applicable_workarounds"]

dockerfile = render_base_dockerfile(ngc_image=ngc_image)

# Dockerfile-level injections
for entry in compat:
    if entry["fix"]["type"] == "dockerfile_block":
        # Insert after the Python-deps layer, before project wheel copy
        dockerfile = inject_after(dockerfile,
                                  marker="RUN pip install --no-cache-dir -r requirements.txt",
                                  block=entry["fix"]["content"])
    elif entry["fix"]["type"] == "runtime_env":
        dockerfile = append_env(dockerfile, entry["fix"]["content"])

write("Dockerfile", dockerfile)

# Config overrides merged into config.yaml
cfg_overrides = {k: v for entry in compat if entry["fix"]["type"] == "config_override"
                 for k, v in entry["fix"]["content"].items()}
if cfg_overrides:
    merge_yaml("config.yaml", cfg_overrides)
    log_progress(f"Compat: applied config overrides {list(cfg_overrides)}", status="⚠️")

# Dataset-template and naming-rule entries are enforced at Phase 4 generation time,
# not post-processed here.

Invariants

  1. Applicable-set is the minimum fix set. Every fix has a matching detection rule. No speculative fixes.

  2. Registry is the single source of truth. When this file grows, every future generated project benefits automatically.

  3. The skill never silently applies a fix. Every applied workaround logged in PROGRESS.md.

  4. Users can override via skip_workarounds: [<id>] in config.yaml.

When to add to this registry

You've hit a new HF / PyTorch / NVIDIA incompatibility and fixed it. Add an entry with:

  • The exact error the user would see

  • The one-line root cause

  • A detection rule that's specific enough to not fire on unrelated models

  • The minimal fix (Dockerfile line, config value, env var)

Avoid adding entries for:

  • Bugs in the skill itself — those should be fixed in the skill, not patched in generated code

  • Model-specific performance tuning — goes in config.yaml, not a fix

  • Workarounds a user could reasonably discover from the error message (e.g. "add --shm-size")

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Declares allowed tools, Needs an API key, Needs Python

The author says: Requires docker + nvidia-container-toolkit, NVIDIA GPU (driver ≥ 545, ≥ 24 GB VRAM for ≤3B models), ~40 GB free disk. Optional credentials (read from the session environment) — HF_TOKEN is read only when the model/dataset is gated or `push_to_hub` is on; WANDB_API_KEY and WANDB_PROJECT only when WandB logging is enabled.

Security

What our scanner noticed· Medium risk

  • Large base64-encoded blobs
  • Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
NVIDIA Corporation
Publisher
NVIDIA (Official)
Source
NVIDIA/skills · skills/tao-finetune-huggingface-model · commit fce3cca
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE-APACHE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by NVIDIA Corporation or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download tao-finetune-huggingface-model.zip