AutoGineer
Generated banner for Nvflare Convert Huggingface
Generated logo for Nvflare Convert Huggingface

Nvflare Convert Huggingface

NVIDIAOfficialCreated by NVIDIA FLARE Team

  • Skill
  • Tool integration
  • Automation and integrations
  • Apache-2.0
  • Medium risk
  • Has scripts
Download ZIP

From the author

Convert existing Hugging Face Transformers Trainer or TRL SFTTrainer training code into an NVFLARE federated job using flare.patch(trainer), local validation, and job export; use when the user names Hugging Face or preliminary source inspection identifies one Hugging Face owner, and not for manual PyTorch loops, Lightning, inference-only pipelines, deployment, or experiment workflows.
— the description by NVIDIA FLARE Team

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured procedure for migrating existing Hugging Face Transformers or TRL SFTTrainer training scripts into an NVFLARE-compatible federated learning job. It leverages the NVFLARE Client API to patch the Trainer, enabling federated training while preserving the original model architecture, dataset handling, and evaluation logic. The process involves static inspection of the source code, recipe selection, job construction using maintained assets, and validation through simulation.

What it does

  • Inspects source code to confirm Hugging Face Trainer ownership.
  • Patches the Trainer using flare.patch(trainer) for federated model exchange.
  • Constructs an NVFLARE job using standard FedAvg recipes.
  • Preserves original Trainer arguments, metrics, and PEFT configurations.
  • Validates the converted job through simulation and evidence reporting.

When to use it

  • When you have existing training code built around transformers.Trainer or TRL SFTTrainer.
  • When you need to transition a centralized Hugging Face training workflow to a federated environment.
  • When you require federated fine-tuning of models using PEFT/LoRA.
  • When you need to maintain existing Trainer callbacks and evaluation metrics in a federated setup.

Advantages

  • Maintains existing Hugging Face training logic and Trainer semantics.
  • Automates the integration of federated learning capabilities via patching.
  • Ensures consistent model exchange using standard NVFLARE recipes.
  • Provides a clear validation ladder to verify the converted job.

Usage examples

“Convert my existing Hugging Face SFTTrainer script to run on NVFLARE.”
The agent will inspect your script, confirm Trainer ownership, apply the required patches, and generate an NVFLARE job structure ready for simulation.
“How do I make my Transformers Trainer-based model work in a federated setting?”
The agent will guide you through the conversion workflow, including patching your Trainer and configuring the server-client job assets.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/nvidia/nvflare-convert-huggingface/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip nvflare-convert-huggingface.zip -d .agents/skills

Documentation by NVIDIA FLARE Team

NVFLARE Convert Hugging Face

Use When

Use when converting training code built around transformers.Trainer, Seq2SeqTrainer, TRL SFTTrainer, or another Trainer subclass. Support full-model and PEFT/LoRA fine-tuning, datasets/tokenizers, Trainer callbacks and metrics, checkpoint continuity, and replicated torch.distributed training.

Do Not Use When

Do not use for an AutoModel driven by a manual PyTorch loop without a Hugging Face Trainer (route to nvflare-convert-pytorch), PyTorch Lightning (route to nvflare-convert-lightning, including Lightning modules that contain Transformers models), inference-only pipelines, model serving, failed jobs (route to nvflare-diagnose-job), or federated statistics without training (route to nvflare-fed-stats). Route a project with active Lightning and Hugging Face Trainer entrypoints to nvflare-orient to select one training-loop owner or separate jobs. Route unresolved Trainer ownership, such as a Trainer factory without a bound owner call, to nvflare-orient; do not patch either Trainer. Out of scope: DeepSpeed, FSDP, production/POC deployment, controller rewrites, experiment search, and privacy-protection requests such as HE, encrypted aggregation, differential privacy, or privacy filters; never substitute an unprotected recipe or present a disclaimer as implementation. If a request combines federated statistics and model-training conversion, treat it as two independent jobs and workflows: do not merge or automatically chain them, do not route the combination to nvflare-orient, and ask which workflow to run first before generating or running either job. Recommend nvflare-fed-stats first only when the user's purpose is to understand data distribution; handle conversion later as a separate request.

Available Scripts

ScriptPurposeArguments
scripts/resolve_model_snapshot.pyResolve a local or Hub model/dataset snapshot and emit JSON evidence.Identifier; required --source; optional source-root, download, revision, cache, and repository-type flags.

Use run_script() only during the validation phase after loading references/huggingface-validation.md. For example:

  • run_script("scripts/resolve_model_snapshot.py", ["--source", "local", "--source-root", "<absolute-source-root>", "<configured-path>"])

  • run_script("scripts/resolve_model_snapshot.py", ["--source", "hub", "<org/model>"])

Workflow

  1. Load ../nvflare-shared/references/conversion-common.md and apply it for the whole conversion; this SKILL.md states only the framework-specific deltas. Load ../nvflare-shared/references/conversion-workflow.md only for a non-standard rerun, authorization, or missing-semantics case; it no longer holds the data-location or partitioning contracts, whose invariants conversion-common.md owns. Load ../nvflare-shared/references/site-data-and-paths.md for generated partitions, relative paths, or per-site data locations.

  2. Inspect before editing with nvflare agent inspect source <path> --format json plus direct source reading. Load references/huggingface-detection.md during this phase. If inspect recommends nvflare-orient for unresolved Trainer ownership or active Lightning/Hugging Face owners, stop before editing. Extract the entrypoint, Trainer subclass, model constructor, tokenizer or processor, datasets and collator, Trainer arguments, compute_metrics, callbacks, checkpoint and PEFT settings, precision, local budget, distributed launcher, site/round counts, data location, and aggregation intent. Do not import or execute user training modules to discover them.

  3. Apply the dependency-install ordering rule in ../nvflare-shared/references/conversion-common.md before any Python command imports user, framework, NVFLARE, or declared dependencies. Keep dependency inventory single-purpose: check NVFLARE separately, inventory non-product packages separately, and do not append Hugging Face cache or filesystem discovery. Defer model and dataset availability checks to the maintained resolver during validation. If an optional path must be inspected, run it separately and report a missing directory with exit code zero.

  4. Select the recipe from FL intent. For explicit FedAvg, run nvflare recipe show fedavg-pt --format json, then immediately load ../nvflare-shared/references/pytorch-family-recipe-construction.md and use the returned module, class, and parameters with the required construction and execution shape in assets/job.py. Import FedAvgRecipe from nvflare.app_opt.pt.recipes.fedavg, never from nvflare.recipe. Treat class_path as the public recipe key and path as its normalized exported representation; do not inspect Recipe source or signatures to reconcile them. Do not guess adjacent symbols or add per-site recipe config unless sites genuinely differ. Load ../nvflare-shared/references/pytorch-family-recipe-selection.md only for ambiguous, evaluation-only, or non-FedAvg requests.

  5. Convert with references/huggingface-conversion.md and adapt assets/client_with_eval.py rather than drafting a new round loop. Preserve model, tokenizer/processor, datasets, collator, Trainer arguments, callbacks, and metrics. Apply the step-1 data-location rules to the client's data argument. Import the Client API as import nvflare.client.hf as flare, so flare.init(), flare.patch(), and flare.is_running() resolve to nvflare.client.hf. Keep flare.patch(trainer) simple with inferred params_scope="auto" and encode one per-round budget in Trainer arguments: requested steps use max_steps, requested epochs use num_train_epochs, and a silent prompt uses the reported default max_steps=10 unless source-budget preservation was requested. Do not duplicate the budget in patch local_steps/local_epochs. When the client uses HfArgumentParser, construct it with allow_abbrev=False.

  6. Adapt assets/server_model.py and assets/job.py instead of inventing server-model, packaging, export, or SimEnv wiring. Keep generated and packaged project-local modules in the same writable source directory. Never use .. in train_script, add_server_file(), or add_client_file(); use an existing resolved absolute path when co-location is impossible. Keep the server and Trainer model factory and exchange keyspace identical. Use the recipe's documented class_path or path key plus complete args for required or overridden values; a direct zero-argument instance must not use from_pretrained(), downloads, or checkpoint loading during job construction. Apply only options confirmed by the construction reference. Preserve the job asset's recipe-before-parser ordering, ArgumentParser(allow_abbrev=False), and strict parse_args(); do not use parse_known_args().

  7. Only after generated files exist, load ../nvflare-shared/references/validation-evidence.md and references/huggingface-validation.md. Follow the shared compile, construction, simulation, and terminal-evidence ladder. Inspect export/package evidence only for an exported final target; inspect a local target's materialized evidence after its run. Apply only the standard HF Trainer checks and stop at the first failed rung. Review and exercise the maintained assets directly; do not inspect NVFLARE implementation source, improvise Recipe API probes, or write one-off AST programs to re-prove them. Use references/huggingface-state-and-distributed.md only when inspection found PEFT, DDP, checkpoint/restore overrides, auxiliary trainable models, or another non-default patch setting.

  8. Report the recipe, source facts, parameter scope, data partition, changed files, validation status, and exact artifact paths. When validation produces metrics, load ../nvflare-shared/references/metrics-and-artifact-reporting.md before the final response and report each observed primary scalar with its metric name, numeric value, and artifact or bounded-log source.

Requirements

  • Must use flare.patch(trainer) as the sole model-exchange owner. receive() inside a patched loop may inspect task metadata only; it must not load a second copy of the global model.

  • Must follow the Client API initialization and conditional rank contract in ../nvflare-shared/references/conversion-common.md. Keep the generated client rankless; nvflare.client.hf.init() owns distributed-rank resolution and rejection. Load references/huggingface-state-and-distributed.md only for a distributed multi-process source path.

  • Must preserve source evaluation. When per-round global-model evaluation is required, call trainer.evaluate() before trainer.train() on every rank. Do not invent compute_metrics, label mappings, averaging denominators, or metric direction.

  • Must follow the Best-Model Metric policy in ../nvflare-shared/references/pytorch-family-recipe-construction.md; the Hugging Face delta is only how the delivered key is named and produced. Must preserve source metric names when practical: if the generated trainer.evaluate() emits accuracy, set key_metric="accuracy"; if Trainer emits a prefixed key such as eval_accuracy, set the server to that exact key and report the source-to-server mapping. When best-model selection is requested, every lower-is-better metric, including Trainer-generated eval_loss, is delivered as an explicitly negated companion and selected by that key — never as raw loss. When selection is not requested, use key_metric=""; do not omit it and accidentally activate the recipe default.

  • Must preserve PEFT configuration exactly and verify adapter key compatibility between the server model and patched Trainer. Do not infer LoRA target modules, silently switch adapter/full-model scope, or solve key mismatches with non-strict loading.

  • Must verify that trainer.model owns all federated trainable state for Trainer subclasses with reference, reward, value-head, or other auxiliary models. Ask or fail closed when params_scope="auto" would omit trainable state required by the algorithm.

  • Must preserve model constructor values needed on both server and clients per ../nvflare-shared/references/pytorch-model-exchange.md (State-Dict Compatibility). Ask one semantic question or fail closed when required values are not statically available.

  • Must patch only one Trainer per Python process. Preserve a single Trainer lifecycle across rounds when restore_state=True.

  • Must use a positive TrainingArguments.max_steps budget for a length-less iterable training dataset and let flare.patch(trainer) infer it.

  • Must reject or report DeepSpeed, FSDP, save_only_model=True with restore_state=True, load_best_model_at_end=True, prebuilt optimizer/scheduler instances with restore_state=False, and checkpoint paths not visible to every distributed rank. Do not rewrite these settings silently. launch_once is a framework-neutral recipe parameter owned by ../nvflare-shared/references/pytorch-family-recipe-construction.md; the Hugging Face delta is only that the product rejects explicit launch_once=False together with restore_state=True.

  • Must initialize torch.distributed before patching when rank environment variables declare multiple ranks. All ranks must call patched Trainer methods in identical order.

  • Must use the maintained HF validation resolver with an explicit local/Hub source. For authorized downloads it obtains or validates a full commit-SHA revision before downloading. Must not copy it into generated job code, set trust_remote_code=True, download model/data artifacts unless requested, or recover from a cache-only miss by going online. Cache misses, remote identifiers, and validation requests do not authorize online retries; see ../nvflare-shared/references/conversion-common.md.

  • Site partitioning, custom aggregation, the Source Of Truth Boundary, and user input/authorization follow ../nvflare-shared/references/conversion-common.md.

Always read this converter SKILL.md together with ../nvflare-shared/references/conversion-common.md. Complete each workflow phase before loading the next phase's reference. Do not preload validation, state/DDP, broad workflow, dependency, or reporting references. The standard FedAvg path loads, in order: ../nvflare-shared/references/conversion-common.md, references/huggingface-detection.md, and ../nvflare-shared/references/site-data-and-paths.md only for its stated triggers; ../nvflare-shared/references/pytorch-family-recipe-construction.md, references/huggingface-conversion.md, and ../nvflare-shared/references/pytorch-model-exchange.md; then ../nvflare-shared/references/validation-evidence.md and references/huggingface-validation.md; load references/huggingface-state-and-distributed.md and other shared references only under the triggers above. Do not depend on repository examples.

Files

assets/job.py

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""Standard Hugging Face Trainer FedAvg job template.

Copy this file beside ``client.py``, ``server_model.py``, and packaged
project-local modules such as ``model.py``. Adapt the client argument names and
capability-gated recipe options to the source project. Use
``resolve_source_local_path`` only for an inspected source argument that names a
local file or directory, and pass the inspected source-project root explicitly;
do not use it for Hub identifiers, URLs, or site-local paths. Keep the required
Recipe, packaging, and execution structure; do not replace local file names
with parent traversal paths.
"""

import argparse
import math
import os
import tempfile
from contextlib import contextmanager
from numbers import Integral, Real
from pathlib import Path

from nvflare.app_opt.pt.recipes.fedavg import FedAvgRecipe
from nvflare.recipe import SimEnv, set_per_site_config

DEFAULT_MAX_STEPS = 10
SOURCE_DIR = Path(__file__).resolve().parent


def resolve_source_local_path(value: str | Path, *, source_root: str | Path) -> Path:
    """Resolve a local path against its explicit inspected source-project root."""
    path = Path(value)
    if path.is_absolute():
        return path.resolve()

    root = Path(source_root).expanduser()
    if not root.is_absolute():
        raise ValueError("source_root must be an absolute path")
    return (root / path).resolve()


@contextmanager
def _source_directory():
    """Resolve Recipe resources beside this job without retaining a cwd change."""
    original_cwd = Path.cwd()
    os.chdir(SOURCE_DIR)
    try:
        yield
    finally:
        os.chdir(original_cwd)


def _simulation_workspace(workspace_root):
    """Return an explicit workspace or a private one that preserves run results."""
    if workspace_root is not None:
        return workspace_root
    return Path(tempfile.mkdtemp(prefix="nvflare-hf-trainer-"))


def _token(value, name: str) -> str:
    text = str(value)
    if not text or any(char.isspace() for char in text):
        raise ValueError(f"{name} must be a non-empty whitespace-free value")
    return text


def _positive_int_arg(value: str) -> int:
    try:
        parsed = int(value)
    except (TypeError, ValueError) as exc:
        raise argparse.ArgumentTypeError("must be a positive integer") from exc
    if parsed <= 0:
        raise argparse.ArgumentTypeError("must be a positive integer")
    return parsed


def _positive_float_arg(value: str) -> float:
    try:
        parsed = float(value)
    except (TypeError, ValueError) as exc:
        raise argparse.ArgumentTypeError("must be a finite positive number") from exc
    if not math.isfinite(parsed) or parsed <= 0:
        raise argparse.ArgumentTypeError("must be a finite positive number")
    return parsed


def build_train_args(
    model_name_or_path: str,
    num_clients: int,
    *,
    max_steps: int | None = None,
    num_train_epochs: float | None = None,
    preserve_source_budget: bool = False,
) -> str:
    """Build arguments consumed by the generated client's strict parser."""
    if preserve_source_budget and (max_steps is not None or num_train_epochs is not None):
        raise ValueError("preserve_source_budget cannot be combined with max_steps or num_train_epochs")
    if max_steps is not None and num_train_epochs is not None:
        raise ValueError("specify only one of max_steps or num_train_epochs")
    if max_steps is not None and (isinstance(max_steps, bool) or not isinstance(max_steps, Integral) or max_steps <= 0):
        raise ValueError("max_steps must be a positive integer")
    if num_train_epochs is not None and (
        isinstance(num_train_epochs, bool)
        or not isinstance(num_train_epochs, Real)
        or not math.isfinite(num_train_epochs)
        or num_train_epochs <= 0
    ):
        raise ValueError("num_train_epochs must be a finite positive number")

    args = [
        "--model_name_or_path",
        _token(model_name_or_path, "model_name_or_path"),
        "--num_clients",
        str(num_clients),
    ]
    if not preserve_source_budget:
        if max_steps is None and num_train_epochs is None:
            max_steps = DEFAULT_MAX_STEPS
        if max_steps is not None:
            args.extend(("--max_steps", str(max_steps)))
        else:
            args.extend(("--num_train_epochs", str(num_train_epochs)))
    return " ".join(args)


def build_recipe(
    *,
    name: str,
    model_name_or_path: str,
    num_clients: int,
    num_rounds: int,
    key_metric: str = "",
    max_steps: int | None = None,
    num_train_epochs: float | None = None,
    preserve_source_budget: bool = False,
    per_site_config: dict[str, dict] | None = None,
    recipe_options: dict | None = None,
):
    """Build FedAvg using only options confirmed by ``recipe show``."""
    train_args = build_train_args(
        model_name_or_path,
        num_clients,
        max_steps=max_steps,
        num_train_epochs=num_train_epochs,
        preserve_source_budget=preserve_source_budget,
    )
    options = dict(recipe_options or {})
    if "per_site_config" in options:
        raise ValueError(
            "pass per_site_config to build_recipe instead of using the deprecated recipe constructor option"
        )
    if isinstance(per_site_config, dict) and any(
        isinstance(site_config, dict) and "train_script" in site_config for site_config in per_site_config.values()
    ):
        raise ValueError(
            "per_site_config must use the shared train_script='client.py'; "
            "site-specific train_script overrides are not supported by this job template"
        )

    with _source_directory():
        recipe = FedAvgRecipe(
            name=name,
            model={
                "class_path": "server_model.ServerModel",
                "args": {"model_name_or_path": model_name_or_path},
            },
            min_clients=num_clients,
            num_rounds=num_rounds,
            train_script="client.py",
            train_args=train_args,
            key_metric=key_metric,
            **options,
        )
        if per_site_config is not None:
            set_per_site_config(recipe, per_site_config)
        recipe.add_server_file(str(SOURCE_DIR / "server_model.py"))
        recipe.add_server_file(str(SOURCE_DIR / "model.py"))
        recipe.add_client_file(str(SOURCE_DIR / "client.py"))
        recipe.add_client_file(str(SOURCE_DIR / "model.py"))
    return recipe


def build_sim_env(
    num_clients: int,
    workspace_root: Path,
    *,
    per_site_config: dict[str, dict] | None = None,
) -> SimEnv:
    """Build a simulator environment with exactly one client-topology owner."""
    if per_site_config is None:
        return SimEnv(
            num_clients=num_clients,
            workspace_root=str(workspace_root),
        )

    clients = list(per_site_config)
    return SimEnv(
        num_clients=num_clients,
        clients=clients,
        workspace_root=str(workspace_root),
    )


def main():
    # Importing the Recipe API before parsing lets it consume --export and
    # --export-dir; this parser owns only the generated job's local options.
    parser = argparse.ArgumentParser(description="Hugging Face Trainer FedAvg job", allow_abbrev=False)
    parser.add_argument("--name", default="hf-trainer-fedavg")
    parser.add_argument("--model_name_or_path", required=True)
    parser.add_argument("--num_clients", type=int, default=2)
    parser.add_argument("--num_rounds", type=int, default=2)
    budget = parser.add_mutually_exclusive_group()
    budget.add_argument("--max_steps", type=_positive_int_arg)
    budget.add_argument("--num_train_epochs", type=_positive_float_arg)
    budget.add_argument("--preserve_source_budget", action="store_true")
    parser.add_argument(
        "--key_metric",
        default="",
        help="exact higher-is-better server metric key; leave empty to disable best-model selection",
    )
    parser.add_argument(
        "--workspace_root",
        type=Path,
        default=None,
        help="simulation workspace; defaults to a new private temporary directory retained for result inspection",
    )
    args = parser.parse_args()

    recipe = build_recipe(
        name=args.name,
        model_name_or_path=args.model_name_or_path,
        num_clients=args.num_clients,
        num_rounds=args.num_rounds,
        max_steps=args.max_steps,
        num_train_epochs=args.num_train_epochs,
        preserve_source_budget=args.preserve_source_budget,
        key_metric=args.key_metric,
    )
    workspace_root = _simulation_workspace(args.workspace_root)
    run = recipe.execute(build_sim_env(args.num_clients, workspace_root))
    print("Job Status is:", run.get_status())
    print("Result can be found in:", run.get_result())


if __name__ == "__main__":
    main()

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Medium risk

  • Large base64-encoded blobs
  • Instructions that use the network
  • Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
NVIDIA FLARE Team
Publisher
NVIDIA (Official)
Source
NVIDIA/skills · skills/nvflare-convert-huggingface · commit fce3cca
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE-APACHE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by NVIDIA FLARE Team or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download nvflare-convert-huggingface.zip