Tao Train Oneformer

NVIDIAOfficialCreated by NVIDIA Corporation

Skill
Tool integration
Data and AI
Apache-2.0
Medium risk

From the author

OneFormer for universal image segmentation. Unifies panoptic, instance, and semantic segmentation with a single architecture using task-conditioned queries. Use when training, evaluating, exporting, quantizing, or running inference for a TAO OneFormer model. Trigger phrases include "train OneFormer", "universal segmentation", "task-conditioned segmentation", "panoptic / instance / semantic in one model".

— the description by NVIDIA Corporation

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The tao-train-oneformer skill provides a comprehensive interface for managing the lifecycle of OneFormer models within the NVIDIA TAO ecosystem. It supports universal image segmentation, enabling users to perform panoptic, instance, and semantic segmentation using a single task-conditioned architecture. The skill handles training, evaluation, model export, quantization, and inference, with built-in support for AutoML to optimize training parameters. It includes detailed guidance on dataset requirements, multi-GPU training configurations, and deployment workflows via TensorRT.

What it does

Trains OneFormer models for universal image segmentation tasks.
Supports AutoML for automated hyperparameter optimization during training.
Evaluates model performance using metrics like mIoU.
Exports trained models to ONNX format for deployment.
Quantizes models to improve inference efficiency.
Integrates with TAO Deploy for TensorRT engine generation and optimized inference.

When to use it

When you need a unified model for panoptic, instance, and semantic segmentation.
When training a segmentation model within the NVIDIA TAO framework.
When you need to optimize model performance using AutoML.
When preparing a model for deployment on NVIDIA hardware using TensorRT.

Advantages

Unifies multiple segmentation tasks into a single architecture.
Streamlined integration with TAO AutoML for efficient training.
Provides clear paths for model export and TensorRT deployment.
Includes robust error handling and configuration guidance for complex training setups.

Usage examples

“Train a OneFormer model for universal segmentation.”: The agent initiates the training process using the specified dataset and default AutoML policy, configuring the model for panoptic, instance, and semantic segmentation.
“Export my trained OneFormer model to ONNX.”: The agent generates an ONNX file from the specified checkpoint, ensuring the model is ready for subsequent TensorRT engine generation.
“Run inference on my OneFormer model.”: The agent executes the inference pipeline using the provided checkpoint and test dataset, outputting segmentation results in the configured mode.
“Disable AutoML for my OneFormer training run.”: The agent proceeds with a manual training configuration, bypassing the AutoML optimization process as requested.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/nvidia/tao-train-oneformer/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip tao-train-oneformer.zip -d .agents/skills

Documentation by NVIDIA Corporation

OneFormer

Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).

OneFormer for universal image segmentation. Unifies panoptic, instance, and semantic segmentation with a single architecture using task-conditioned queries.

Set train.pretrained_backbone and/or train.pretrained_model.

For TAO Deploy TensorRT actions (gen_trt_engine, TensorRT evaluate, and TensorRT inference), read references/tao-deploy-oneformer.md first. Deploy spec templates live in this skill's references/ folder with the spec_template_deploy_*.yaml prefix.

Dataclass Schemas

Generated TAO Core schemas are packaged in schemas/<action>.schema.json, with schemas/manifest.json listing available actions. Each generated schema also emits references/spec_template_<action>.yaml from the schema top-level default field. AutoML enablement is declared at the model layer in references/skill_info.yaml via automl_enabled. Runnable AutoML for an action requires schemas/<action>.schema.json and references/spec_template_<action>.yaml to exist and parse. Use the packaged selected-action schema for automl_default_parameters, automl_disabled_parameters, defaults, min/max bounds, enums, option weights, math conditions, dependencies, and popular parameters. Do not expect ~/tao-core at runtime; maintainers regenerate schemas/templates before packaging the skill bank.

Train Action Policy

This model is AutoML-enabled at the model layer. Before handling any train-stage request, read references/skill_info.yaml and resolve the run override from either an explicit automl_policy value or the user's workflow request. Use automl_policy: on by default and only expose on / off in new launch prompts. Treat phrases like "turn off AutoML", "disable AutoML", "no HPO", or "plain training" as automl_policy: off for this run only. When automl_policy: on, automl_enabled: true, and both schemas/train.schema.json and references/spec_template_train.yaml are packaged, route the train action through tao-skill-bank:tao-run-automl by default with this model's skill_dir. Preserve workflow/application overrides for datasets, specs, output directories, GPU/platform settings, parent checkpoints, and automl_policy. Use direct model training only when automl_policy: off or the packaged train schema/template is missing; in the missing-schema case, report that AutoML is enabled but not runnable for this model until schemas are generated.

Non-train actions such as evaluate, inference, export, and deploy flows stay in this model skill. The per-run automl_policy override does not change model metadata.

Training Requirements

Dataset type: segmentation
Formats: coco_panoptic, coco
AutoML training metric: mIoU, with direction=maximize
Standalone evaluation metric: test_mIoU, with direction=maximize

Per-Action Dataset Requirements

Action	Spec Key	Source	Files	List?
evaluate	dataset.train.images	train_datasets	images.tar.gz	No
evaluate	dataset.label_map	train_datasets	label_map.json	No
evaluate	dataset.train.annotations	train_datasets	annotations.json	No
evaluate	dataset.train.panoptic	train_datasets	images_panoptic.tar.gz	No
evaluate	dataset.val.images	eval_dataset	images.tar.gz	No
evaluate	dataset.val.annotations	eval_dataset	annotations.json	No
evaluate	dataset.val.panoptic	eval_dataset	images_panoptic.tar.gz	No
evaluate	dataset.test.images	eval_dataset	images.tar.gz	No
evaluate	dataset.test.annotations	eval_dataset	annotations.json	No
evaluate	dataset.test.panoptic	eval_dataset	images_panoptic.tar.gz	No
inference	dataset.train.images	train_datasets	images.tar.gz	No
inference	dataset.label_map	train_datasets	label_map.json	No
inference	dataset.train.annotations	train_datasets	annotations.json	No
inference	dataset.train.panoptic	train_datasets	images_panoptic.tar.gz	No
inference	dataset.val.images	eval_dataset	images.tar.gz	No
inference	dataset.val.annotations	eval_dataset	annotations.json	No
inference	dataset.val.panoptic	eval_dataset	images_panoptic.tar.gz	No
inference	dataset.test.images	inference_dataset	images.tar.gz	No
quantize	dataset.train.images	train_datasets	images.tar.gz	No
quantize	dataset.train.annotations	train_datasets	annotations.json	No
quantize	dataset.label_map	train_datasets	label_map.json	No
quantize	dataset.train.panoptic	train_datasets	images_panoptic.tar.gz	No
quantize	dataset.val.images	eval_dataset	images.tar.gz	No
quantize	dataset.val.annotations	eval_dataset	annotations.json	No
quantize	dataset.val.panoptic	eval_dataset	images_panoptic.tar.gz	No
quantize	dataset.test.images	eval_dataset	images.tar.gz	No
quantize	dataset.quant_calibration_dataset.images_dir	calibration_dataset	images.tar.gz	No
train	dataset.train.images	train_datasets	images.tar.gz	No
train	dataset.train.annotations	train_datasets	annotations.json	No
train	dataset.label_map	train_datasets	label_map.json	No
train	dataset.train.panoptic	train_datasets	images_panoptic.tar.gz	No
train	dataset.val.images	eval_dataset	images.tar.gz	No
train	dataset.val.annotations	eval_dataset	annotations.json	No
train	dataset.val.panoptic	eval_dataset	images_panoptic.tar.gz	No
train	dataset.test.images	eval_dataset	images.tar.gz	No

Typical Spec Overrides

Data source overrides are mandatory for every action — the agent MUST construct data source paths from the Per-Action Dataset Requirements table above and include them in spec_overrides.

S3_TRAIN = "s3://bucket/data/train"
S3_EVAL = "s3://bucket/data/eval"
S3_INFERENCE = "s3://bucket/data/inference"
S3_CALIBRATION = "s3://bucket/data/calibration"

train (mandatory data sources):

{
    "train.num_gpus": 1,
    "train.num_epochs": 10,
    "train.checkpoint_interval": 10,
    "train.validation_interval": 10,
    "model.sem_seg_head.num_classes": 133,
    "dataset.contiguous_id": True,
    "train.precision": "32",
    "dataset.train.images": f"{S3_TRAIN}/images.tar.gz",
    "dataset.train.annotations": f"{S3_TRAIN}/annotations.json",
    "dataset.label_map": f"{S3_TRAIN}/label_map.json",
    "dataset.train.panoptic": f"{S3_TRAIN}/images_panoptic.tar.gz",
    "dataset.val.images": f"{S3_EVAL}/images.tar.gz",
    "dataset.val.annotations": f"{S3_EVAL}/annotations.json",
    "dataset.val.panoptic": f"{S3_EVAL}/images_panoptic.tar.gz",
    "dataset.test.images": f"{S3_EVAL}/images.tar.gz",
}

evaluate (mandatory data sources):

{
    "evaluate.checkpoint": "<selected train/AutoML checkpoint>",
    "model.sem_seg_head.num_classes": 133,
    "dataset.contiguous_id": True,
    "dataset.train.images": f"{S3_TRAIN}/images.tar.gz",
    "dataset.label_map": f"{S3_TRAIN}/label_map.json",
    "dataset.train.annotations": f"{S3_TRAIN}/annotations.json",
    "dataset.train.panoptic": f"{S3_TRAIN}/images_panoptic.tar.gz",
    "dataset.val.images": f"{S3_EVAL}/images.tar.gz",
    "dataset.val.annotations": f"{S3_EVAL}/annotations.json",
    "dataset.val.panoptic": f"{S3_EVAL}/images_panoptic.tar.gz",
    "dataset.test.images": f"{S3_EVAL}/images.tar.gz",
    "dataset.test.annotations": f"{S3_EVAL}/annotations.json",
    "dataset.test.panoptic": f"{S3_EVAL}/images_panoptic.tar.gz",
}

export:

{
    "export.checkpoint": "<selected train/AutoML checkpoint>",
    "model.sem_seg_head.num_classes": 133,
    "model.export": True,
    "export.onnx_file": "/results/oneformer_export_640.onnx",
}

inference (mandatory data sources):

{
    "inference.checkpoint": "<selected train/AutoML checkpoint>",
    "dataset.train.images": f"{S3_TRAIN}/images.tar.gz",
    "dataset.label_map": f"{S3_TRAIN}/label_map.json",
    "dataset.train.annotations": f"{S3_TRAIN}/annotations.json",
    "dataset.train.panoptic": f"{S3_TRAIN}/images_panoptic.tar.gz",
    "dataset.val.images": f"{S3_EVAL}/images.tar.gz",
    "dataset.val.annotations": f"{S3_EVAL}/annotations.json",
    "dataset.val.panoptic": f"{S3_EVAL}/images_panoptic.tar.gz",
    "dataset.test.images": f"{S3_INFERENCE}/images.tar.gz",
    "inference.images_dir": f"{S3_INFERENCE}/images.tar.gz",
}

quantize (mandatory data sources):

{
    "quantize.model_path": "<selected train/AutoML checkpoint>",
    "dataset.train.images": f"{S3_TRAIN}/images.tar.gz",
    "dataset.train.annotations": f"{S3_TRAIN}/annotations.json",
    "dataset.label_map": f"{S3_TRAIN}/label_map.json",
    "dataset.train.panoptic": f"{S3_TRAIN}/images_panoptic.tar.gz",
    "dataset.val.images": f"{S3_EVAL}/images.tar.gz",
    "dataset.val.annotations": f"{S3_EVAL}/annotations.json",
    "dataset.val.panoptic": f"{S3_EVAL}/images_panoptic.tar.gz",
    "dataset.test.images": f"{S3_EVAL}/images.tar.gz",
    "dataset.quant_calibration_dataset.images_dir": f"{S3_CALIBRATION}/images.tar.gz",
}

Checkpoint Selection

OneFormer training writes epoch-step checkpoints such as model_epoch_000_step_00017.pth and may also write a oneformer_model_latest.pth symlink. For checkpoint-dependent actions, use the model-skill or SDK parent-model resolver and pass the exact selected checkpoint path into evaluate.checkpoint, inference.checkpoint, export.checkpoint, quantize.model_path, or train.resume_training_checkpoint_path. Do not pick the oneformer_model_latest.pth symlink by name unless the user explicitly asks for latest checkpoint behavior. If the resolver reports a best checkpoint, use that best checkpoint for evaluation/export/inference; if the user asks for a specific epoch or step, use the matching epoch-step checkpoint.

Eval Dataset

Optional. Val data configured alongside train in the dataset config.

Important Parameters

model.sem_seg_head.num_classes: Number of segmentation class indices available to the head. Default 133 for COCO panoptic data when dataset.contiguous_id: True remaps raw category ids through the label map. Do not shrink this to a global workflow class count unless the label map and annotations have actually been reduced to that class set.
model.one_former.hidden_dim: Keep at 256 for local smoke runs unless the text encoder width is changed in lock-step. Reducing hidden_dim alone causes a text feature/context dimension mismatch during training.
model.backbone.name: Default D2SwinTransformer (Swin-based). embed_dim=192, depths=[2,2,18,2] by default.
train.num_epochs: Default 50 — significantly higher than most TAO models. OneFormer needs more epochs for convergence.
train.optim.lr: Learning rate. Default 1e-5. Lower than Mask2Former's 2e-4.
model.task_toggling: Enable/disable specific tasks: semantic_on, instance_on, panoptic_on.
export.task: Export task mode. Options: semantic, instance, panoptic. Default semantic. Export input defaults to 640x640.
inference.mode: Inference mode. Options: semantic, instance, panoptic. Default semantic. image_size defaults to [1024, 1024].
evaluate.iou_per_class: Report per-class IoU in evaluation. Default True.

Multi-GPU / Multi-Node

Launch method: Lightning-managed (single python process, Lightning spawns workers).

Spec Key	Description	Default
`train.num_gpus`	Number of GPUs	1
`train.gpu_ids`	GPU device indices	[0]
`train.num_nodes`	Number of nodes	1

Uses explicit DDPStrategy with find_unused_parameters=True, gradient_as_bucket_view=True, process_group_backend="nccl"
sync_batchnorm is always enabled
No fsdp support — DDP only

Multi-node env vars (set by orchestrator): WORLD_SIZE, NODE_RANK, MASTER_ADDR, MASTER_PORT, NUM_GPU_PER_NODE.

Hardware

Minimum 2 GPU(s), recommended 4 GPU(s). 24GB+ (A100 recommended) VRAM per GPU. OneFormer is memory-intensive like Mask2Former. batch_size=1 is the default. Multi-GPU needed for reasonable training speed, especially with 50 epochs.

Error Patterns

CUDA out of memory: batch_size is already 1. Reduce image resolution or use a smaller Swin configuration.

Extracted S3 tarball points one level too high: For local Docker runs, images.tar.gz and images_panoptic.tar.gz may extract wrapper directories such as images/ and images_panoptic/. Set dataset.*.images, dataset.*.panoptic, inference.images_dir, and quantization calibration paths to the actual folder containing image or panoptic files, not the wrapper directory. A one-level-too-high path fails with FileNotFoundError for the first annotation image even though recursive file counts look correct.

default_specs missing results_dir: The CLI default_specs subtask ignores -e experiment specs for results_dir; pass a Hydra-style override instead: oneformer default_specs results_dir=/path/to/default_specs.

Invalid Lightning precision fp32: Use train.precision: "32" in train/AutoML/evaluate/inference specs. The current Lightning stack rejects the legacy fp32 string.

PyTorch 2.6 checkpoint load failure on downstream actions: Current OneFormer checkpoints include OmegaConf objects. For checkpoints produced by the same trusted TAO train/AutoML workflow, set TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1 in downstream evaluate, inference, export, quantize, or resume job env vars so Lightning can load the full checkpoint. Do not use this env var for untrusted checkpoints.

CUDA device-side assert in matcher/class cost: If training fails in oneformer/utils/matcher.py while indexing out_prob[:, tgt_ids], compare the effective target ids with model.sem_seg_head.num_classes. The packaged COCO panoptic sample has 133 compact classes after dataset.contiguous_id: True remapping, so use model.sem_seg_head.num_classes: 133 even when a broader validation workflow passes a smaller generic num_classes value. Only use a smaller class count when the label map and annotations are reduced to that exact contiguous class set.

Inference returns PASS with no predictions: OneFormer prediction reads inference.images_dir, not dataset.test.images. Declare and populate inference.images_dir with the image folder or tarball for every inference run. dataset.test.images may still be useful for shared dataset context, but it does not drive the PyTorch predict dataloader.

Export output path pre-created as a directory: Do not declare export.onnx_file as a file output. The OneFormer exporter asserts that the ONNX path does not already exist, while the local runner pre-creates declared output paths. Set export.onnx_file explicitly in the spec to a non-existing file path under the mounted results tree. Keep the default 640x640 export shape for smoke validation; very small export shapes can trigger PyTorch ONNX shape-inference failures.

Quantize cannot find the training label map from an AutoML checkpoint: OneFormer Lightning checkpoints retain train-time absolute dataset paths in their saved hparams. When running downstream actions from an AutoML child checkpoint, keep the parent AutoML job directory accessible at its original /results/<job_id> path inside the action container in addition to passing the resolved checkpoint path. Otherwise quantize can fail while loading checkpoint hparams even when the current spec includes a valid dataset.label_map.

Slow training: 50 default epochs with batch_size=1 is slow on single GPU. Use multi-GPU distributed training.

Spec Param / Parent Model Inference

Model-specific inference mappings belong in this MD file, not in config.json. Generated runners should read this section and apply the mappings with SDK helpers before create_job(). This mirrors the old microservices infer_params.py flow.

Inference mappings from TAO Core oneformer.config.json:

Action	Spec Field	Inference Function	Meaning
evaluate	`encryption_key`	`key`	encryption key
evaluate	`evaluate.checkpoint`	`parent_model`	model file inferred from the parent job results folder
evaluate	`evaluate.trt_engine`	`parent_model`	model file inferred from the parent job results folder
evaluate	`results_dir`	`output_dir`	current job results directory
export	`encryption_key`	`key`	encryption key
export	`export.checkpoint`	`parent_model`	model file inferred from the parent job results folder
export	`export.onnx_file`	`create_onnx_file`	output ONNX path
export	`results_dir`	`output_dir`	current job results directory
gen_trt_engine	`encryption_key`	`key`	encryption key
gen_trt_engine	`gen_trt_engine.onnx_file`	`parent_model`	model file inferred from the parent job results folder
gen_trt_engine	`gen_trt_engine.trt_engine`	`create_engine_file`	output TensorRT engine path
gen_trt_engine	`results_dir`	`output_dir`	current job results directory
inference	`encryption_key`	`key`	encryption key
inference	`inference.checkpoint`	`parent_model`	model file inferred from the parent job results folder
inference	`inference.trt_engine`	`parent_model`	model file inferred from the parent job results folder
inference	`results_dir`	`output_dir`	current job results directory
quantize	`encryption_key`	`key`	encryption key
quantize	`quantize.model_path`	`parent_model`	model file inferred from the parent job results folder
quantize	`results_dir`	`output_dir`	current job results directory
train	`encryption_key`	`key`	encryption key
train	`results_dir`	`output_dir`	current job results directory
train	`train.pretrained_backbone`	`{'link': 'https://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_tiny_patch4_window7_224_22k.pth', 'destination_path': '/ptm/mask2former/swin_tiny_patch4_window7_224_22k/swin_tiny_patch4_window7_224_22k.pth'}`	{'link': 'https://github.com/SwinTransformer/storage/releases/download/v1.0.8/swin_tiny_patch4_window7_224_22k.pth', 'destination_path': '/ptm/mask2former/swin_tiny_patch4_window7_224_22k/swin_tiny_patch4_window7_224_22k.pth'}
train	`train.pretrained_model`	`ptm_if_no_resume_model`	PTM when no resume checkpoint exists
train	`train.resume_training_checkpoint_path`	`resume_model`	model file inferred from the current job results folder

For parent_model or parent_model_folder, pass the upstream train/export/AutoML child job id as parent_job_id. The SDK lists the parent result folder, filters checkpoint artifacts, and returns the selected model file or folder. Do not add these mappings back to config.json and do not patch generated runner scripts to guess checkpoint paths.

Deployment

tao-deploy-oneformer

Files

ATTRIBUTION.mdAdded by AutoGineer927 B
BENCHMARK.md7.0 KiB
LICENSELicence file added by AutoGineer11.2 KiB
SKILL.md19.5 KiB
config/skillspector-baseline.yaml834 B
evals/evals.json783 B
references/skill_info.yaml9.5 KiB
references/spec_template_deploy_evaluate.yaml521 B
references/spec_template_deploy_gen_trt_engine.yaml277 B
references/spec_template_deploy_inference.yaml521 B
references/spec_template_evaluate.yaml4.6 KiB
references/spec_template_export.yaml4.7 KiB
references/spec_template_gen_trt_engine.yaml4.8 KiB
references/spec_template_inference.yaml4.7 KiB
references/spec_template_quantize.yaml4.5 KiB
references/spec_template_train.yaml4.5 KiB
references/tao-deploy-oneformer.md6.1 KiB
references/tao-deploy-oneformer.skill_info.yaml3.7 KiB
schemas/evaluate.schema.json72.9 KiB
schemas/export.schema.json73.5 KiB
schemas/gen_trt_engine.schema.json75.4 KiB
schemas/inference.schema.json73.5 KiB
schemas/manifest.json18.2 KiB
schemas/quantize.schema.json70.2 KiB
schemas/train.schema.json70.1 KiB
skill-card.md4.5 KiB
skill.oms.sig9.4 KiB

schemas/gen_trt_engine.schema.json

{
  "automl_default_parameters": [
    "dataset.augmentation.test_min_size",
    "dataset.augmentation.test_max_size",
    "train.optim.weight_decay",
    "train.optim.backbone_multiplier",
    "dataset.augmentation.train_max_size",
    "train.optim.momentum",
    "train.optim.lr"
  ],
  "automl_disabled_parameters": [
    "quantize",
    "train.cudnn",
    "model.backbone.swin.out_features",
    "quantize.backend_kwargs",
    "train.gpu_ids",
    "model.sem_seg_head",
    "wandb.tags",
    "model.backbone.swin.depths",
    "model.backbone.swin.num_heads",
    "model.backbone",
    "model.backbone.swin.out_indices",
    "model.text_encoder",
    "model.test",
    "dataset.pixel_mean",
    "quantize.skip_names",
    "model.backbone.radio.summary_idxs",
    "train.optim.milestones",
    "evaluate",
    "inference",
    "model.one_former",
    "train",
    "train.clip_gradients",
    "dataset.augmentation",
    "dataset.augmentation.train_crop_size",
    "gen_trt_engine",
    "gen_trt_engine.tensorrt.layers_precision",
    "model.sem_seg_head.in_features",
    "dataset",
    "gen_trt_engine.tensorrt",
    "dataset.pixel_std",
    "quantize.layers",
    "dataset.quant_calibration_dataset",
    "model.backbone.radio.out_features",
    "dataset.task_prob_train",
    "model.sem_seg_head.deformable_transformer_encoder_in_features",
    "dataset.train",
    "model",
    "train.freeze",
    "evaluate.gpu_ids",
    "dataset.augmentation.train_min_size",
    "train.optim",
    "dataset.val",
    "dataset.task_prob_val",
    "model.backbone.radio.resolution",
    "train.optim.steps",
    "model.backbone.swin",
    "export",
    "model.backbone.radio",
    "wandb",
    "inference.image_size",
    "inference.gpu_ids",
    "dataset.test"
  ],
  "default": {
    "dataset": {
      "augmentation": {
        "test_max_size": 1333,
        "test_min_size": 800,
        "train_crop_size": [
          1024,
          1024
        ],
        "train_max_size": 1333,
        "train_min_size": [
          800
        ]
      },
      "contiguous_id": true,
      "cutmix_prob": 0.0,
      "image_size": 1024,
      "label_map": "",
      "max_scale": 2.0,
      "max_seq_len": 77,
      "min_scale": 0.1,
      "pin_memory": true,
      "pixel_mean": [
        123.675,
        116.28,
        103.53
      ],
      "pixel_std": [
        58.395,
        57.12,
        57.375
      ],
      "quant_calibration_dataset": {
        "images_dir": ""
      },
      "task_prob_train": {
        "instance": 0.66,
        "panoptic": 0.01,
        "semantic": 0.33
      },
      "task_prob_val": {
        "instance": 0.66,
        "panoptic": 0.01,
        "semantic": 0.33
      },
      "task_seq_len": 77,
      "test": {
        "annotations": "",
        "batch_size": 1,
        "images": "",
        "num_workers": 1,
        "panoptic": ""
      },
      "train": {
        "annotations": "",
        "batch_size": 1,
        "images": "",
        "num_workers": 1,
        "panoptic": ""
      },
      "val": {
        "annotations": "",
        "batch_size": 1,
        "images": "",
        "num_workers": 1,
        "panoptic": ""
      },
      "workers": 8
    },
    "encryption_key": "",
    "gen_trt_engine": {
      "batch_size": 0,
      "gpu_id": 0,
      "onnx_file": "",
      "results_dir": "",
      "tensorrt": {
        "data_type": "fp16",
        "layers_precision": [],
        "max_batch_size": 1,
        "min_batch_size": 1,
        "opt_batch_size": 1,
        "workspace_size": 1024
      },
      "timing_cache": "",
      "trt_engine": "",
      "verbose": false
    },
    "model": {
      "backbone": {
        "freeze_at": 0,
        "name": "D2SwinTransformer",
        "radio": {
          "backbone": "vit_base_patch16_224",
          "cpe_max_size": 2048,
          "num_teacher": 4,
          "out_features": [
            "res2",
            "res3",
            "res4",
            "res5"
          ],
          "register_multiple": 8,
          "resolution": [
            1024,
            1024
          ],
          "summary_idxs": [
            0,
            1,
            2
          ],
          "use_checkpoint": false
        },
        "swin": {
          "ape": false,
          "attn_drop_rate": 0.0,
          "depths": [
            2,
            2,
            18,
            2
          ],
          "drop_path_rate": 0.3,
          "drop_rate": 0.0,
          "embed_dim": 192,
          "mlp_ratio": 4.0,
          "num_heads": [
            6,
            12,
            24,
            48
          ],
          "out_features": [
            "res2",
            "res3",
            "res4",
            "res5"
          ],
          "out_indices": [
            0,
            1,
            2,
            3
          ],
          "patch_norm": true,
          "patch_size": 4,
          "pretrain_img_size": 384,
          "qkv_bias": true,
          "use_checkpoint": false,
          "window_size": 12
        }
      },
      "export": false,
      "one_former": {
        "class_dec_layers": 2,
        "class_weight": 2.0,
        "contrastive_temperature": 0.07,
        "contrastive_weight": 0.5,
        "dec_layers": 10,
        "deep_supervision": true,
        "dice_weight": 5.0,
        "dim_feedforward": 2048,
        "dropout": 0.1,
        "enc_layers": 0,
        "enforce_input_proj": false,
        "hidden_dim": 256,
        "importance_sample_ratio": 0.75,
        "mask_weight": 5.0,
        "nheads": 8,
        "no_object_weight": 0.1,
        "num_feature_levels": 3,
        "num_object_queries": 150,
        "oversample_ratio": 3.0,
        "pre_norm": false,
        "size_divisibility": 32,
        "train_num_points": 12544,
        "transformer_decoder_name": "ContrastiveMultiScaleMaskedTransformerDecoder",
        "transformer_in_feature": "multi_scale_pixel_decoder",
        "use_task_norm": true
      },
      "sem_seg_head": {
        "common_stride": 4,
        "convs_dim": 256,
        "deformable_transformer_encoder_in_features": [
          "res3",
          "res4",
          "res5"
        ],
        "ignore_value": 255,
        "in_features": [
          "res3",
          "res4",
          "res5"
        ],
        "loss_weight": 1.0,
        "mask_dim": 256,
        "name": "OneFormerHead",
        "norm": "GN",
        "num_classes": 133,
        "pixel_decoder_name": "MSDeformAttnPixelDecoder",
        "transformer_enc_layers": 6
      },
      "test": {
        "detection_on": false,
        "instance_on": false,
        "object_mask_threshold": 0.4,
        "overlap_threshold": 0.5,
        "panoptic_on": false,
        "semantic_on": true,
        "test_topk_per_image": 100
      },
      "text_encoder": {
        "context_length": 77,
        "n_ctx": 16,
        "num_layers": 6,
        "proj_num_layers": 2,
        "vocab_size": 49408,
        "width": 256
      }
    },
    "model_name": "",
    "quantize": {
      "algorithm": "minmax",
      "backend": "torchao",
      "backend_kwargs": {},
      "device": "cuda",
      "layers": [],
      "mode": "weight_only_ptq",
      "model_path": "",
      "results_dir": "",
      "skip_names": []
    },
    "results_dir": "",
    "train": {
      "accumulate_grad_batches": 1,
      "checkpoint_interval": 1,
      "checkpoint_interval_unit": "epoch",
      "clip_grad_norm": 0.1,
      "clip_grad_type": "full",
      "clip_gradients": {
        "clip_type": "full_model",
        "clip_value": 1.0,
        "enabled": true,
        "norm_type": 2.0
      },
      "cudnn": {
        "benchmark": false,
        "deterministic": true
      },
      "distributed_strategy": "ddp",
      "freeze": [],
      "gpu_ids": [
        0
      ],
      "is_dry_run": false,
      "num_epochs": 50,
      "num_gpus": 8,
      "num_nodes": 1,
      "optim": {
        "backbone_multiplier": 0.1,
        "gamma": 0.1,
        "lr": 1e-05,
        "lr_scheduler": "Warmuppoly",
        "max_iter": 368750,
        "milestones": [
          88,
          96
        ],
        "momentum": 0.9,
        "monitor_name": "train_loss",
        "steps": [
          327778,
          355092
        ],
        "type": "AdamW",
        "warmup_factor": 0.001,
        "warmup_iters": 1000,
        "weight_decay": 0.05
      },
      "precision": "fp32",
      "pretrained_backbone": "",
      "pretrained_model": "",
      "results_dir": "",
      "resume_training_checkpoint_path": "",
      "seed": 123,
      "validation_interval": 1,
      "verbose": false
    },
    "wandb": {
      "enable": true,
      "entity": "",
      "group": "",
      "name": "TAO Toolkit Training",
      "project": "TAO Toolkit",
      "reinit": false,
      "run_id": "",
      "save_code": false,
      "sync_tensorboard": false,
      "tags": [
        "tao-toolkit"
      ]
    }
  },
  "popular": {
    "evaluate": {
      "gpu_ids": [
        0
      ],
      "num_gpus": 1,
      "num_nodes": 1
    },
    "gen_trt_engine": {
      "gpu_id": 0,
      "tensorrt": {
        "max_batch_size": 1,
        "min_batch_size": 1,
        "opt_batch_size": 1
      }
    },
    "inference": {
      "gpu_ids": [
        0
      ],
      "num_gpus": 1,
      "num_nodes": 1
    },
    "model": {
      "sem_seg_head": {
        "convs_dim": 256,
        "mask_dim": 256,
        "transformer_enc_layers": 6
      }
    },
    "train": {
      "gpu_ids": [
        0
      ],
      "optim": {
        "backbone_multiplier": 0.1,
        "momentum": 0.9,
        "weight_decay": 0.05
      }
    }
  },
  "properties": {
    "automl_disabled_parameters": [
      "wandb",
      "model",
      "dataset",
      "train",
      "evaluate",
      "inference",
      "export",
      "gen_trt_engine",
      "quantize"
    ],
    "dataset": {
      "automl_disabled_parameters": [
        "dataset.train",
        "dataset.val",
        "dataset.test",
        "dataset.pixel_mean",
        "dataset.pixel_std",
        "dataset.augmentation",
        "dataset.task_prob_train",
        "dataset.task_prob_val",
        "dataset.quant_calibration_dataset"
      ],
      "automl_enabled": false,
      "default": {
        "augmentation": {
          "test_max_size": 1333,
          "test_min_size": 800,
          "train_crop_size": [
            1024,
            1024
          ],
          "train_max_size": 1333,
          "train_min_size": [
            800
          ]
        },
        "contiguous_id": true,
        "cutmix_prob": 0.0,
        "image_size": 1024,
        "label_map": "",
        "max_scale": 2.0,
        "max_seq_len": 77,
        "min_scale": 0.1,
        "pin_memory": true,
        "pixel_mean": [
          123.675,
          116.28,
          103.53
        ],
        "pixel_std": [
          58.395,
          57.12,
          57.375
        ],
        "quant_calibration_dataset": {
          "images_dir": ""
        },
        "task_prob_train": {
          "instance": 0.66,
          "panoptic": 0.01,
          "semantic": 0.33
        },
        "task_prob_val": {
          "instance": 0.66,
          "panoptic": 0.01,
          "semantic": 0.33
        },
        "task_seq_len": 77,
        "test": {
          "annotations": "",
          "batch_size": 1,
          "images": "",
          "num_workers": 1,
          "panoptic": ""
        },
        "train": {
          "annotations": "",
          "batch_size": 1,
          "images": "",
          "num_workers": 1,
          "panoptic": ""
        },
        "val": {
          "annotations": "",
          "batch_size": 1,
          "images": "",
          "num_workers": 1,
          "panoptic": ""
        },
        "workers": 8
      },
      "properties": {
        "augmentation": {
          "automl_default_parameters": [
            "dataset.augmentation.train_max_size",
            "dataset.augmentation.test_min_size",
            "dataset.augmentation.test_max_size"
          ],
          "automl_disabled_parameters": [
            "dataset.augmentation.train_min_size",
            "dataset.augmentation.train_crop_size"
          ],
          "automl_enabled": false,
          "default": {
            "test_max_size": 1333,
            "test_min_size": 800,
            "train_crop_size": [
              1024,
              1024
            ],
            "train_max_size": 1333,
            "train_min_size": [
              800
            ]
          },
          "description": "Configuration parameters for data augmentation",
          "properties": {
            "test_max_size": {
              "automl_enabled": true,
              "default": 1333,
              "description": "The maximum resize size for test",
              "maximum": Infinity,
              "minimum": 32,
              "title": "Test max size",
              "type": "int"
            },
            "test_min_size": {
              "automl_enabled": true,
              "default": 800,
              "description": "The minimum resize size for test data",
              "maximum": Infinity,
              "minimum": 32,
              "title": "Test min size",
              "type": "int"
            },
            "train_crop_size": {
              "automl_enabled": false,
              "default": [
                1024,
                1024
              ],
              "description": "The random crop size for training data in [H, W]",
              "title": "Train crop size",
              "type": "list"
            },
            "train_max_size": {
              "automl_enabled": true,
              "default": 1333,
              "description": "The maximum random crop size for training data",
              "maximum": Infinity,
              "minimum": 32,
              "title": "Train max size",
              "type": "int"
            },
            "train_min_size": {
              "automl_enabled": false,
              "default": [
                800
              ],
              "description": "A list of sizes to perform random resize.",
              "title": "Train min size",
              "type": "list"
            }
          },
          "type": "collection"
        },
        "contiguous_id": {
          "default": true,
          "description": "Flag to enable contiguous ids for labels.",
          "title": "contiguous id",
          "type": "bool"
        },
        "cutmix_prob": {
          "default": 0.0,
          "description": "Cutmix probability",
          "title": "cutmix probability",
          "type": "float"
        },
        "image_size": {
          "default": 1024,
          "description": "Image size",
          "title": "image size",
          "type": "int"
        },
        "label_map": {
          "default": "",
          "description": "A path to label map file",
          "title": "label map",
          "type": "string"
        },
        "max_scale": {
          "default": 2.0,
          "description": "Maximum scale",
          "title": "maximum scale",
          "type": "float"
        },
        "max_seq_len": {
          "default": 77,
          "description": "Maximum sequence length",
          "title": "maximum sequence length",
          "type": "int"
        },
        "min_scale": {
          "default": 0.1,
          "description": "Minimum scale",
          "title": "minimum scale",
          "type": "float"
        },
        "pin_memory": {
          "default": true,
          "description": "Flag to enable the dataloader to allocate pagelocked memory",
          "title": "pin_memory",
          "type": "bool"
        },
        "pixel_mean": {
          "automl_enabled": false,
          "default": [
            123.675,
            116.28,
            103.53
          ],
          "description": "The input mean for RGB frames",
          "title": "input mean per pixel",
          "type": "list"
        },
        "pixel_std": {
          "automl_enabled": false,
          "default": [
            58.395,
            57.12,
            57.375
          ],
          "description": "The input standard deviation per pixel for RGB frames",
          "title": "input std per pixel",
          "type": "list"
        },
        "quant_calibration_dataset": {
          "automl_enabled": false,
          "default": {
            "images_dir": ""
          },
          "description": "Configurable parameters for the quantization calibration dataset.",
          "properties": {
            "images_dir": {
              "default": "",
              "description": "Path to images directory for quantization calibration",
              "title": "images directory",
              "type": "string"
            }
          },
          "type": "collection"
        },
        "task_prob_train": {
          "automl_enabled": false,
          "default": {
            "instance": 0.66,
            "panoptic": 0.01,
            "semantic": 0.33
          },
          "description": "Task probabilities",
          "title": "task probabilities",
          "type": "collection"
        },
        "task_prob_val": {
          "automl_enabled": false,
          "default": {
            "instance": 0.66,
            "panoptic": 0.01,
            "semantic": 0.33
          },
          "description": "Task probabilities",
          "title": "task probabilities",
          "type": "collection"
        },
        "task_seq_len": {
          "default": 77,
          "description": "Task sequence length",
          "title": "task sequence length",
          "type": "int"
        },
        "test": {
          "automl_enabled": false,
          "default": {
            "annotations": "",
            "batch_size": 1,
            "images": "",
            "num_workers": 1,
            "panoptic": ""
          },
          "description": "Configurable parameters to construct the test dataset.",
          "properties": {
            "annotations": {
              "default": "",
              "description": "A path to annotation root",
              "title": "annotation root",
              "type": "string"
            },
            "batch_size": {
              "default": 1,
              "description": "Batch size",
              "math_cond": ">0",
              "maximum": Infinity,
              "minimum": 1,
              "title": "batch size",
              "type": "int"
            },
            "images": {
              "default": "",
              "description": "A path to image root",
              "title": "image root",
              "type": "string"
            },
            "num_workers": {
              "default": 1,
              "description": "Number of workers",
              "maximum": Infinity,
              "minimum": 0,
              "title": "Number of workers",
              "type": "int"
            },
            "panoptic": {
              "default": "",
              "description": "A path to panoptic root",
              "title": "panoptic root",
              "type": "string"
            }
          },
          "type": "collection"
        },
        "train": {
          "automl_enabled": false,
          "default": {
            "annotations": "",
            "batch_size": 1,
            "images": "",
            "num_workers": 1,
            "panoptic": ""
          },
          "description": "Configurable parameters to construct the train dataset.",
          "properties": {
            "annotations": {
              "default": "",
              "description": "A path to annotation root",
              "title": "annotation root",
              "type": "string"
            },
            "batch_size": {
              "default": 1,
              "description": "Batch size",
              "math_cond": ">0",
              "maximum": Infinity,
              "minimum": 1,
              "title": "batch size",
              "type": "int"
            },
            "images": {
              "default": "",
              "description": "A path to image root",
              "title": "image root",
              "type": "string"
            },
            "num_workers": {
              "default": 1,
              "description": "Number of workers",
              "maximum": Infinity,
              "minimum": 0,
              "title": "Number of workers",
              "type": "int"
            },
            "panoptic": {
              "default": "",
              "description": "A path to panoptic root",
              "title": "panoptic root",
              "type": "string"
            }
          },
          "type": "collection"
        },
        "val": {
          "automl_enabled": false,
          "default": {
            "annotations": "",
            "batch_size": 1,
            "images": "",
            "num_workers": 1,
            "panoptic": ""
          },
          "description": "Configurable parameters to construct the validation dataset.",
          "properties": {
            "annotations": {
              "default": "",
              "description": "A path to annotation root",
              "title": "annotation root",
              "type": "string"
            },
            "batch_size": {
              "default": 1,
              "description": "Batch size",
              "math_cond": ">0",
              "maximum": Infinity,
              "minimum": 1,
              "title": "batch size",
              "type": "int"
            },
            "images": {
              "default": "",
              "description": "A path to image root",
              "title": "image root",
              "type": "string"
            },
            "num_workers": {
              "default": 1,
              "description": "Number of workers",
              "maximum": Infinity,
              "minimum": 0,
              "title": "Number of workers",
              "type": "int"
            },
            "panoptic": {
              "default": "",
              "description": "A path to panoptic root",
              "title": "panoptic root",
              "type": "string"
            }
          },
          "type": "collection"
        },
        "workers": {
          "default": 8,
          "description": "The number of parallel workers processing data",
          "maximum": Infinity,
          "minimum": 1,
          "title": "workers",
          "type": "int"
        }
      },
      "type": "collection"
    },
    "encryption_key": {
      "default": "",
      "description": "Key for encrypting model checkpoints",
      "title": "Encryption key",
      "type": "string"
    },
    "gen_trt_engine": {
      "automl_disabled_parameters": [
        "gen_trt_engine.tensorrt"
      ],
      "automl_enabled": false,
      "default": {
        "batch_size": 0,
        "gpu_id": 0,
        "onnx_file": "",
        "results_dir": "",
        "tensorrt": {
          "data_type": "fp16",
          "layers_precision": [],
          "max_batch_size": 1,
          "min_batch_size": 1,
          "opt_batch_size": 1,
          "workspace_size": 1024
        },
        "timing_cache": "",
        "trt_engine": "",
        "verbose": false
      },
      "description": "Configurable parameters to construct the deployer for a OneFormer checkpoint.",
      "popular": [
        "gpu_id",
        "tensorrt"
      ],
      "properties": {
        "batch_size": {
          "default": 0,
          "description": "Batch size for the TensorRT engine.",
          "title": "Batch size",
          "type": "int"
        },
        "gpu_id": {
          "default": 0,
          "description": "The index of the GPU to build the TensorRT engine.",
          "minimum": 0,
          "popular": true,
          "title": "GPU ID",
          "type": "int"
        },
        "onnx_file": {
          "default": "",
          "description": "Path to the ONNX model file.",
          "title": "ONNX file",
          "type": "string"
        },
        "results_dir": {
          "default": "",
          "description": "\n        Path to where all the assets generated from a task are stored.\n        ",
          "title": "Results directory",
          "type": "string"
        },
        "tensorrt": {
          "automl_disabled_parameters": [
            "gen_trt_engine.tensorrt.layers_precision"
          ],
          "automl_enabled": false,
          "default": {
            "data_type": "fp16",
            "layers_precision": [],
            "max_batch_size": 1,
            "min_batch_size": 1,
            "opt_batch_size": 1,
            "workspace_size": 1024
          },
          "description": "Hyper parameters to configure the TensorRT Engine builder.",
          "popular": [
            "min_batch_size",
            "max_batch_size",
            "opt_batch_size"
          ],
          "properties": {
            "data_type": {
              "default": "fp16",
              "description": "The precision to be set for building the TensorRT engine.",
              "enum": [
                "fp16",
                "fp32"
              ],
              "title": "data type",
              "type": "categorical"
            },
            "layers_precision": {
              "automl_enabled": false,
              "default": [],
              "description": "The list to specify layer precision.",
              "title": "layers_precision",
              "type": "list"
            },
            "max_batch_size": {
              "default": 1,
              "description": "The maximum batch size in the optimization profile for\n                    the input tensor of the TensorRT engine.",
              "minimum": 1,
              "popular": true,
              "title": "Maximum batch size",
              "type": "int"
            },
            "min_batch_size": {
              "default": 1,
              "description": "The minimum batch size in the optimization profile for\n                    the input tensor of the TensorRT engine.",
              "minimum": 1,
              "popular": true,
              "title": "Min batch size",
              "type": "int"
            },
            "opt_batch_size": {
              "default": 1,
              "description": "The optimum batch size in the optimization profile for\n                    the input tensor of the TensorRT engine.",
              "minimum": 1,
              "popular": true,
              "title": "Optimum batch size",
              "type": "int"
            },
            "workspace_size": {
              "default": 1024,
              "description": "The workspace size to be set for building the TensorRT engine.",
              "minimum": 1,
              "title": "workspace size",
              "type": "int"
            }
          },
          "title": "TensorRT hyper params.",
          "type": "collection"
        },
        "timing_cache": {
          "default": "",
          "description": "Path to a TensorRT timing cache that speeds up engine generation.\n                    This will be created/read/updated.",
          "title": "TensorRT timing cache",
          "type": "string"
        },
        "trt_engine": {
          "default": "",
          "description": "Path to the TensorRT engine file.",
          "title": "TensorRT engine",
          "type": "string"
        },
        "verbose": {
          "default": false,
          "description": "Verbose for the TensorRT engine.",
          "title": "Verbose",
          "type": "bool"
        }
      },
      "type": "collection"
    },
    "model": {
      "automl_disabled_parameters": [
        "model.sem_seg_head",
        "model.one_former",
        "model.text_encoder",
        "model.backbone",
        "model.test"
      ],
      "automl_enabled": false,
      "default": {
        "backbone": {
          "freeze_at": 0,
          "name": "D2SwinTransformer",
          "radio": {
            "backbone": "vit_base_patch16_224",
            "cpe_max_size": 2048,
            "num_teacher": 4,
            "out_features": [
              "res2",
              "res3",
              "res4",
              "res5"
            ],
            "register_multiple": 8,
            "resolution": [
              1024,
              1024
            ],
            "summary_idxs": [
              0,
              1,
              2
            ],
            "use_checkpoint": false
          },
          "swin": {
            "ape": false,
            "attn_drop_rate": 0.0,
            "depths": [
              2,
              2,
              18,
              2
            ],
            "drop_path_rate": 0.3,
            "drop_rate": 0.0,
            "embed_dim": 192,
            "mlp_ratio": 4.0,
            "num_heads": [
              6,
              12,
              24,
              48
            ],
            "out_features": [
              "res2",
              "res3",
              "res4",
              "res5"
            ],
            "out_indices": [
              0,
              1,
              2,
              3
            ],
            "patch_norm": true,
            "patch_size": 4,
            "pretrain_img_size": 384,
            "qkv_bias": true,
            "use_checkpoint": false,
            "window_size": 12
          }
        },
        "export": false,
        "one_former": {
          "class_dec_layers": 2,
          "class_weight": 2.0,
          "contrastive_temperature": 0.07,
          "contrastive_weight": 0.5,
          "dec_layers": 10,
          "deep_supervision": true,
          "dice_weight": 5.0,
          "dim_feedforward": 2048,
          "dropout": 0.1,
          "enc_layers": 0,
          "enforce_input_proj": false,
          "hidden_dim": 256,
          "importance_sample_ratio": 0.75,
          "mask_weight": 5.0,
          "nheads": 8,
          "no_object_weight": 0.1,
          "num_feature_levels": 3,
          "num_object_queries": 150,
          "oversample_ratio": 3.0,
          "pre_norm": false,
          "size_divisibility": 32,
          "train_num_points": 12544,
          "transformer_decoder_name": "ContrastiveMultiScaleMaskedTransformerDecoder",
          "transformer_in_feature": "multi_scale_pixel_decoder",
          "use_task_norm": true
        },
        "sem_seg_head": {
          "common_stride": 4,
          "convs_dim": 256,
          "deformable_transformer_encoder_in_features": [
            "res3",
            "res4",
            "res5"
          ],
          "ignore_value": 255,
          "in_features": [
            "res3",
            "res4",
            "res5"
          ],
          "loss_weight": 1.0,
          "mask_dim": 256,
          "name": "OneFormerHead",
          "norm": "GN",
          "num_classes": 133,
          "pixel_decoder_name": "MSDeformAttnPixelDecoder",
          "transformer_enc_layers": 6
        },
        "test": {
          "detection_on": false,
          "instance_on": false,
          "object_mask_threshold": 0.4,
          "overlap_threshold": 0.5,
          "panoptic_on": false,
          "semantic_on": true,
          "test_topk_per_image": 100
        },
        "text_encoder": {
          "context_length": 77,
          "n_ctx": 16,
          "num_layers": 6,
          "proj_num_layers": 2,
          "vocab_size": 49408,
          "width": 256
        }
      },
      "popular": [
        "sem_seg_head"
      ],
      "properties": {
        "backbone": {
          "automl_disabled_parameters": [
            "model.backbone.swin",
            "model.backbone.radio"
          ],
          "automl_enabled": false,
          "default": {
            "freeze_at": 0,
            "name": "D2SwinTransformer",
            "radio": {
              "backbone": "vit_base_patch16_224",
              "cpe_max_size": 2048,
              "num_teacher": 4,
              "out_features": [
                "res2",
                "res3",
                "res4",
                "res5"
              ],
              "register_multiple": 8,
              "resolution": [
                1024,
                1024
              ],
              "summary_idxs": [
                0,
                1,
                2
              ],
              "use_checkpoint": false
            },
            "swin": {
              "ape": false,
              "attn_drop_rate": 0.0,
              "depths": [
                2,
                2,
                18,
                2
              ],
              "drop_path_rate": 0.3,
              "drop_rate": 0.0,
              "embed_dim": 192,
              "mlp_ratio": 4.0,
              "num_heads": [
                6,
                12,
                24,
                48
              ],
              "out_features": [
                "res2",
                "res3",
                "res4",
                "res5"
              ],
              "out_indices": [
                0,
                1,
                2,
                3
              ],
              "patch_norm": true,
              "patch_size": 4,
              "pretrain_img_size": 384,
              "qkv_bias": true,
              "use_checkpoint": false,
              "window_size": 12
            }
          },
          "description": "Backbone.",
          "properties": {
            "freeze_at": {
              "default": 0,
              "description": "Freeze at.",
              "title": "freeze at",
              "type": "int"
            },
            "name": {
              "default": "D2SwinTransformer",
              "description": "Name of the backbone.",
              "title": "name",
              "type": "string"
            },
            "radio": {
              "automl_disabled_parameters": [
                "model.backbone.radio.resolution",
                "model.backbone.radio.summary_idxs",
                "model.backbone.radio.out_features"
              ],
              "automl_enabled": false,
              "default": {
                "backbone": "vit_base_patch16_224",
                "cpe_max_size": 2048,
                "num_teacher": 4,
                "out_features": [
                  "res2",
                  "res3",
                  "res4",
                  "res5"
                ],
                "register_multiple": 8,
                "resolution": [
                  1024,
                  1024
                ],
                "summary_idxs": [
                  0,
                  1,
                  2
                ],
                "use_checkpoint": false
              },
              "description": "Radio.",
              "properties": {
                "backbone": {
                  "default": "vit_base_patch16_224",
                  "description": "Name of the radio backbone.",
                  "title": "backbone",
                  "type": "string"
                },
                "cpe_max_size": {
                  "default": 2048,
                  "description": "Maximum size of the cropped positional embedding.",
                  "title": "cpe max size",
                  "type": "int"
                },
                "num_teacher": {
                  "default": 4,
                  "description": "Number of teachers.",
                  "title": "num teacher",
                  "type": "int"
                },
                "out_features": {
                  "automl_enabled": false,
                  "default": [
                    "res2",
                    "res3",
                    "res4",
                    "res5"
                  ],
                  "description": "List of output features.",
                  "title": "out features",
                  "type": "list"
                },
                "register_multiple": {
                  "default": 8,
                  "description": "Number of extra tokens.",
                  "title": "register multiple",
                  "type": "int"
                },
                "resolution": {
                  "automl_enabled": false,
                  "default": [
                    1024,
                    1024
                  ],
                  "description": "Resolution of the radio.",
                  "title": "resolution",
                  "type": "list"
                },
                "summary_idxs": {
                  "automl_enabled": false,
                  "default": [
                    0,
                    1,
                    2
                  ],
                  "description": "Summary indices.",
                  "title": "summary idxs",
                  "type": "list"
                },
                "use_checkpoint": {
                  "default": false,
                  "description": "Use checkpoint.",
                  "title": "use checkpoint",
                  "type": "bool"
                },
                "window_size": {
                  "description": "Window size.",
                  "title": "window size",
                  "type": "int"
                }
              },
              "title": "radio",
              "type": "collection"
            },
            "swin": {
              "automl_disabled_parameters": [
                "model.backbone.swin.depths",
                "model.backbone.swin.num_heads",
                "model.backbone.swin.out_features",
                "model.backbone.swin.out_indices"
              ],
              "automl_enabled": false,
              "default": {
                "ape": false,
                "attn_drop_rate": 0.0,
                "depths": [
                  2,
                  2,
                  18,
                  2
                ],
                "drop_path_rate": 0.3,
                "drop_rate": 0.0,
                "embed_dim": 192,
                "mlp_ratio": 4.0,
                "num_heads": [
                  6,
                  12,
                  24,
                  48
                ],
                "out_features": [
                  "res2",
                  "res3",
                  "res4",
                  "res5"
                ],
                "out_indices": [
                  0,
                  1,
                  2,
                  3
                ],
                "patch_norm": true,
                "patch_size": 4,
                "pretrain_img_size": 384,
                "qkv_bias": true,
                "use_checkpoint": false,
                "window_size": 12
              },
              "description": "Swin.",
              "properties": {
                "ape": {
                  "default": false,
                  "description": "APE.",
                  "title": "ape",
                  "type": "bool"
                },
                "attn_drop_rate": {
                  "default": 0.0,
                  "description": "Attention dropout rate.",
                  "title": "attn drop rate",
                  "type": "float"
                },
                "depths": {
                  "automl_enabled": false,
                  "default": [
                    2,
                    2,
                    18,
                    2
                  ],
                  "description": "Depths of each stage.",
                  "title": "depths",
                  "type": "list"
                },
                "drop_path_rate": {
                  "default": 0.3,
                  "description": "Drop path rate.",
                  "title": "drop path rate",
                  "type": "float"
                },
                "drop_rate": {
                  "default": 0.0,
                  "description": "Dropout rate.",
                  "title": "drop rate",
                  "type": "float"
                },
                "embed_dim": {
                  "default": 192,
                  "description": "Embedding dimension.",
                  "title": "embed dim",
                  "type": "int"
                },
                "mlp_ratio": {
                  "default": 4.0,
                  "description": "MLP ratio.",
                  "title": "mlp ratio",
                  "type": "float"
                },
                "num_heads": {
                  "automl_enabled": false,
                  "default": [
                    6,
                    12,
                    24,
                    48
                  ],
                  "description": "Number of heads of each stage.",
                  "title": "num heads",
                  "type": "list"
                },
                "out_features": {
                  "automl_enabled": false,
                  "default": [
                    "res2",
                    "res3",
                    "res4",
                    "res5"
                  ],
                  "description": "List of output features.",
                  "title": "out features",
                  "type": "list"
                },
                "out_indices": {
                  "automl_enabled": false,
                  "default": [
                    0,
                    1,
                    2,
                    3
                  ],
                  "description": "List of output indices.",
                  "title": "out indices",
                  "type": "list"
                },
                "patch_norm": {
                  "default": true,
                  "description": "Patch normalization.",
                  "title": "patch norm",
                  "type": "bool"
                },
                "patch_size": {
                  "default": 4,
                  "description": "Patch size.",
                  "title": "patch size",
                  "type": "int"
                },
                "pretrain_img_size": {
                  "default": 384,
                  "description": "Pretrained image size.",
                  "title": "pretrained image size",
                  "type": "int"
                },
                "qk_scale": {
                  "description": "Override default qk scale of head_dim ** -0.5 if set.",
                  "title": "qk scale",
                  "type": "float"
                },
                "qkv_bias": {
                  "default": true,
                  "description": "QKV bias.",
                  "title": "qkv bias",
                  "type": "bool"
                },
                "use_checkpoint": {
                  "default": false,
                  "description": "Use checkpoint.",
                  "title": "use checkpoint",
                  "type": "bool"
                },
                "window_size": {
                  "default": 12,
                  "description": "Window size.",
                  "title": "window size",
                  "type": "int"
                }
              },
              "title": "swin",
              "type": "collection"
            }
          },
          "title": "backbone",
          "type": "collection"
        },
        "export": {
          "default": false,
          "description": "A flag to enable export mode.",
          "title": "export",
          "type": "bool"
        },
        "one_former": {
          "automl_enabled": false,
          "default": {
            "class_dec_layers": 2,
            "class_weight": 2.0,
            "contrastive_temperature": 0.07,
            "contrastive_weight": 0.5,
            "dec_layers": 10,
            "deep_supervision": true,
            "dice_weight": 5.0,
            "dim_feedforward": 2048,
            "dropout": 0.1,
            "enc_layers": 0,
            "enforce_input_proj": false,
            "hidden_dim": 256,
            "importance_sample_ratio": 0.75,
            "mask_weight": 5.0,
            "nheads": 8,
            "no_object_weight": 0.1,
            "num_feature_levels": 3,
            "num_object_queries": 150,
            "oversample_ratio": 3.0,
            "pre_norm": false,
            "size_divisibility": 32,
            "train_num_points": 12544,
            "transformer_decoder_name": "ContrastiveMultiScaleMaskedTransformerDecoder",
            "transformer_in_feature": "multi_scale_pixel_decoder",
            "use_task_norm": true
          },
          "description": "OneFormer.",
          "properties": {
            "class_dec_layers": {
              "default": 2,
              "description": "Number of class decoder layers.",
              "title": "class dec layers",
              "type": "int"
            },
            "class_weight": {
              "default": 2.0,
              "description": "Class weight.",
              "title": "class weight",
              "type": "float"
            },
            "contrastive_temperature": {
              "default": 0.07,
              "description": "Contrastive temperature.",
              "title": "contrastive temperature",
              "type": "float"
            },
            "contrastive_weight": {
              "default": 0.5,
              "description": "Contrastive weight.",
              "title": "contrastive weight",
              "type": "float"
            },
            "dec_layers": {
              "default": 10,
              "description": "Number of decoder layers.",
              "title": "dec layers",
              "type": "int"
            },
            "deep_supervision": {
              "default": true,
              "description": "Deep supervision.",
              "title": "deep supervision",
              "type": "bool"
            },
            "dice_weight": {
              "default": 5.0,
              "description": "Dice weight.",
              "title": "dice weight",
              "type": "float"
            },
            "dim_feedforward": {
              "default": 2048,
              "description": "Dimension of the feedforward network.",
              "title": "dim feedforward",
              "type": "int"
            },
            "dropout": {
              "default": 0.1,
              "description": "Dropout rate.",
              "title": "dropout",
              "type": "float"
            },
            "enc_layers": {
              "default": 0,
              "description": "Number of encoder layers.",
              "title": "enc layers",
              "type": "int"
            },
            "enforce_input_proj": {
              "default": false,
              "description": "Enforce input projection.",
              "title": "enforce input proj",
              "type": "bool"
            },
            "hidden_dim": {
              "default": 256,
              "description": "Dimension of the hidden units.",
              "title": "hidden dim",
              "type": "int"
            },
            "importance_sample_ratio": {
              "default": 0.75,
              "description": "Importance sample ratio.",
              "title": "importance sample ratio",
              "type": "float"
            },
            "mask_weight": {
              "default": 5.0,
              "description": "Mask weight.",
              "title": "mask weight",
              "type": "float"
            },
            "nheads": {
              "default": 8,
              "description": "Number of heads.",
              "title": "nheads",
              "type": "int"
            },
            "no_object_weight": {
              "default": 0.1,
              "description": "No object weight.",
              "title": "no object weight",
              "type": "float"
            },
            "num_feature_levels": {
              "default": 3,
              "description": "Number of feature levels.",
              "title": "num feature levels",
              "type": "int"
            },
            "num_object_queries": {
              "default": 150,
              "description": "Number of object queries.",
              "title": "num object queries",
              "type": "int"
            },
            "oversample_ratio": {
              "default": 3.0,
              "description": "Oversample ratio.",
              "title": "oversample ratio",
              "type": "float"
            },
            "pre_norm": {
              "default": false,
              "description": "Pre-norm.",
              "title": "pre norm",
              "type": "bool"
            },
            "size_divisibility": {
              "default": 32,
              "description": "Size divisibility.",
              "title": "size divisibility",
              "type": "int"
            },
            "train_num_points": {
              "default": 12544,
              "description": "Number of training points.",
              "title": "train num points",
              "type": "int"
            },
            "transformer_decoder_name": {
              "default": "ContrastiveMultiScaleMaskedTransformerDecoder",
              "description": "Name of the transformer decoder.",
              "title": "transformer decoder name",
              "type": "string"
            },
            "transformer_in_feature": {
              "default": "multi_scale_pixel_decoder",
              "description": "Name of the transformer input feature.",
              "title": "transformer in feature",
              "type": "string"
            },
            "use_task_norm": {
              "default": true,
              "description": "Use task norm.",
              "title": "use task norm",
              "type": "bool"
            }
          },
          "title": "oneformer",
          "type": "collection"
        },
        "sem_seg_head": {
          "automl_disabled_parameters": [
            "model.sem_seg_head.in_features",
            "model.sem_seg_head.deformable_transformer_encoder_in_features"
          ],
          "automl_enabled": false,
          "default": {
            "common_stride": 4,
            "convs_dim": 256,
            "deformable_transformer_encoder_in_features": [
              "res3",
              "res4",
              "res5"
            ],
            "ignore_value": 255,
            "in_features": [
              "res3",
              "res4",
              "res5"
            ],
            "loss_weight": 1.0,
            "mask_dim": 256,
            "name": "OneFormerHead",
            "norm": "GN",
            "num_classes": 133,
            "pixel_decoder_name": "MSDeformAttnPixelDecoder",
            "transformer_enc_layers": 6
          },
          "description": "Semantic segmentation head.",
          "popular": [
            "transformer_enc_layers",
            "mask_dim",
            "convs_dim"
          ],
          "properties": {
            "common_stride": {
              "default": 4,
              "description": "Common stride.",
              "minimum": 2,
              "title": "Common stride",
              "type": "int"
            },
            "convs_dim": {
              "default": 256,
              "description": "Convolutional layer dimension.",
              "minimum": 1,
              "popular": true,
              "title": "conv layer dim.",
              "type": "int"
            },
            "deformable_transformer_encoder_in_features": {
              "automl_enabled": false,
              "default": [
                "res3",
                "res4",
                "res5"
              ],
              "description": "List of feature names for deformable transformer encoder input.",
              "title": "transformer encoder in_features",
              "type": "list"
            },
            "ignore_value": {
              "default": 255,
              "description": "Value to ignore in the semantic segmentation head.",
              "title": "ignore value",
              "type": "int"
            },
            "in_features": {
              "automl_enabled": false,
              "default": [
                "res3",
                "res4",
                "res5"
              ],
              "description": "List of feature names for the semantic segmentation head input.",
              "title": "in features",
              "type": "list"
            },
            "loss_weight": {
              "default": 1.0,
              "description": "Loss weight of the semantic segmentation head.",
              "title": "loss weight",
              "type": "float"
            },
            "mask_dim": {
              "default": 256,
              "description": "Mask head dimension.",
              "minimum": 1,
              "popular": true,
              "title": "mask head dim.",
              "type": "int"
            },
            "name": {
              "default": "OneFormerHead",
              "description": "Name of the semantic segmentation head.",
              "title": "name",
              "type": "string"
            },
            "norm": {
              "default": "GN",
              "description": "Norm layer type.",
              "title": "norm type",
              "type": "string"
            },
            "num_classes": {
              "default": 133,
              "description": "Number of classes.",
              "title": "num classes",
              "type": "int"
            },
            "pixel_decoder_name": {
              "default": "MSDeformAttnPixelDecoder",
              "description": "Name of the pixel decoder.",
              "title": "pixel decoder name",
              "type": "string"
            },
            "transformer_enc_layers": {
              "default": 6,
              "description": "Number of transformer encoder layers.",
              "minimum": 1,
              "popular": true,
              "title": "Number of transformer encoder layers.",
              "type": "int"
            }
          },
          "title": "sem seg head",
          "type": "collection"
        },
        "test": {
          "automl_enabled": false,
          "default": {
            "detection_on": false,
            "instance_on": false,
            "object_mask_threshold": 0.4,
            "overlap_threshold": 0.5,
            "panoptic_on": false,
            "semantic_on": true,
            "test_topk_per_image": 100
          },
          "description": "Test.",
          "properties": {
            "detection_on": {
              "default": false,
              "description": "Enable detection.",
              "title": "detect on",
              "type": "bool"
            },
            "instance_on": {
              "default": false,
              "description": "Enable instance segmentation.",
              "title": "instance on",
              "type": "bool"
            },
            "object_mask_threshold": {
              "default": 0.4,
              "description": "The value of the threshold to be used when\n                    filtering out the object mask.",
              "title": "object mask threshold",
              "type": "float"
            },
            "overlap_threshold": {
              "default": 0.5,
              "description": "The value of the threshold to be used when\n                    evaluating overlap.",
              "title": "overlap threshold",
              "type": "float"
            },
            "panoptic_on": {
              "default": false,
              "description": "Enable panoptic segmentation.",
              "title": "panoptic on",
              "type": "bool"
            },
            "semantic_on": {
              "default": true,
              "description": "Enable semantic segmentation.",
              "title": "semantic on",
              "type": "bool"
            },
            "test_topk_per_image": {
              "default": 100,
              "description": " keep topk instances per image for instance segmentation.",
              "title": "top k per image",
              "type": "int"
            }
          },
          "title": "Test configs",
          "type": "collection"
        },
        "text_encoder": {
          "automl_enabled": false,
          "default": {
            "context_length": 77,
            "n_ctx": 16,
            "num_layers": 6,
            "proj_num_layers": 2,
            "vocab_size": 49408,
            "width": 256
          },
          "description": "Text encoder.",
          "properties": {
            "context_length": {
              "default": 77,
              "description": "Context length.",
              "title": "context length",
              "type": "int"
            },
            "n_ctx": {
              "default": 16,
              "description": "Context length.",
              "title": "context length",
              "type": "int"
            },
            "num_layers": {
              "default": 6,
              "description": "Number of layers.",
              "title": "num layers",
              "type": "int"
            },
            "proj_num_layers": {
              "default": 2,
              "description": "Number of projection layers.",
              "title": "proj num layers",
              "type": "int"
            },
            "vocab_size": {
              "default": 49408,
              "description": "Vocabulary size.",
              "title": "vocab size",
              "type": "int"
            },
            "width": {
              "default": 256,
              "description": "Width.",
              "title": "width",
              "type": "int"
            }
          },
          "title": "text encoder",
          "type": "collection"
        }
      },
      "type": "collection"
    },
    "model_name": {
      "default": "",
      "description": "Name of model if invoking task via :code:`model_agnostic`",
      "title": "Model name",
      "type": "string"
    },
    "quantize": {
      "automl_disabled_parameters": [
        "quantize.layers",
        "quantize.skip_names",
        "quantize.backend_kwargs"
      ],
      "automl_enabled": false,
      "default": {
        "algorithm": "minmax",
        "backend": "torchao",
        "backend_kwargs": {},
        "device": "cuda",
        "layers": [],
        "mode": "weight_only_ptq",
        "model_path": "",
        "results_dir": "",
        "skip_names": []
      },
      "description": "Configurable parameters to run model quantization for a OneFormer experiment.",
      "properties": {
        "algorithm": {
          "default": "minmax",
          "description": "Calibration/optimization algorithm. Used by ModelOpt backends (modelopt.pytorch and modelopt.onnx). Ignored by torchao backend.",
          "enum": [
            "minmax",
            "max",
            "entropy",
            "awq_clip",
            "awq_lite",
            "awq_full",
            "rtn_dq"
          ],
          "title": "Calibration algorithm",
          "type": "categorical"
        },
        "backend": {
          "default": "torchao",
          "description": "The quantization backend to use",
          "enum": [
            "modelopt.pytorch",
            "torchao",
            "modelopt.onnx"
          ],
          "title": "Quantization backend",
          "type": "categorical"
        },
        "backend_kwargs": {
          "automl_enabled": false,
          "description": "Additional keyword arguments to pass to the backend",
          "title": "Backend kwargs",
          "type": "collection"
        },
        "device": {
          "default": "cuda",
          "description": "Device to use for calibration. Accepts 'cuda' (uses default GPU), 'cpu', 'trt' (TensorRT for ONNX backend), or specific GPU device like 'cuda:0', 'cuda:1', etc. If 'cuda' is specified but no GPU is available, the framework will automatically fall back to 'cpu'. Note: 'trt' is only supported by the modelopt.onnx backend for ONNX Runtime with TensorRT execution provider.",
          "pattern": "^(cuda|cpu|trt|cuda:[0-9]+)$",
          "title": "Calibration device",
          "type": "string"
        },
        "layers": {
          "automl_enabled": false,
          "default": [],
          "description": "List of per-module quantization configurations. Each entry specifies which modules to quantize and their data types. This is the primary way to configure quantization.",
          "title": "Layer quantization configs",
          "type": "list"
        },
        "mode": {
          "default": "weight_only_ptq",
          "description": "The quantization mode to use",
          "enum": [
            "static_ptq",
            "weight_only_ptq"
          ],
          "title": "Quantization mode",
          "type": "categorical"
        },
        "model_path": {
          "default": "",
          "description": "Path to the model to be quantized. For ONNX backend, path to ONNX file.",
          "title": "Model path",
          "type": "string"
        },
        "results_dir": {
          "default": "",
          "description": "Path to where all the assets generated from a task are stored.",
          "title": "Results directory",
          "type": "string"
        },
        "skip_names": {
          "automl_enabled": false,
          "default": [],
          "description": "List of module/layer names or patterns to exclude from quantization",
          "title": "Skip names",
          "type": "list"
        }
      },
      "required": [
        "model_path",
        "results_dir"
      ],
      "type": "collection"
    },
    "results_dir": {
      "default": "",
      "description": "\n        Path to where all the assets generated from a task are stored.\n        ",
      "title": "Results directory",
      "type": "string"
    },
    "train": {
      "automl_disabled_parameters": [
        "train.gpu_ids",
        "train.cudnn",
        "train.freeze",
        "train.optim",
        "train.clip_gradients"
      ],
      "automl_enabled": false,
      "default": {
        "accumulate_grad_batches": 1,
        "checkpoint_interval": 1,
        "checkpoint_interval_unit": "epoch",
        "clip_grad_norm": 0.1,
        "clip_grad_type": "full",
        "clip_gradients": {
          "clip_type": "full_model",
          "clip_value": 1.0,
          "enabled": true,
          "norm_type": 2.0
        },
        "cudnn": {
          "benchmark": false,
          "deterministic": true
        },
        "distributed_strategy": "ddp",
        "freeze": [],
        "gpu_ids": [
          0
        ],
        "is_dry_run": false,
        "num_epochs": 50,
        "num_gpus": 8,
        "num_nodes": 1,
        "optim": {
          "backbone_multiplier": 0.1,
          "gamma": 0.1,
          "lr": 1e-05,
          "lr_scheduler": "Warmuppoly",
          "max_iter": 368750,
          "milestones": [
            88,
            96
          ],
          "momentum": 0.9,
          "monitor_name": "train_loss",
          "steps": [
            327778,
            355092
          ],
          "type": "AdamW",
          "warmup_factor": 0.001,
          "warmup_iters": 1000,
          "weight_decay": 0.05
        },
        "precision": "fp32",
        "pretrained_backbone": "",
        "pretrained_model": "",
        "results_dir": "",
        "resume_training_checkpoint_path": "",
        "seed": 123,
        "validation_interval": 1,
        "verbose": false
      },
      "description": "Configurable parameters to construct the trainer for a OneFormer experiment.",
      "popular": [
        "optim",
        "gpu_ids"
      ],
      "properties": {
        "accumulate_grad_batches": {
          "default": 1,
          "description": "Number of batches to accumulate gradients over.",
          "title": "accumulate grad batches",
          "type": "int"
        },
        "checkpoint_interval": {
          "default": 1,
          "description": "Number of epochs to checkpoint.",
          "title": "checkpoint interval",
          "type": "int"
        },
        "checkpoint_interval_unit": {
          "default": "epoch",
          "description": "The unit of the checkpoint interval.",
          "enum": [
            "epoch",
            "step"
          ],
          "title": "Checkpoint interval unit",
          "type": "categorical"
        },
        "clip_grad_norm": {
          "default": 0.1,
          "description": "\n        Amount to clip the gradient by L2 Norm.\n        A value of 0.0 specifies no clipping.",
          "math_cond": "> 0.0",
          "title": "clip gradient norm",
          "type": "float"
        },
        "clip_grad_type": {
          "default": "full",
          "description": "Gradient clip type.",
          "title": "clip gradient type",
          "type": "string"
        },
        "clip_gradients": {
          "automl_enabled": false,
          "default": {
            "clip_type": "full_model",
            "clip_value": 1.0,
            "enabled": true,
            "norm_type": 2.0
          },
          "description": "Hyper parameters to configure the gradient clipping.",
          "properties": {
            "clip_type": {
              "default": "full_model",
              "description": "Gradient clip type.",
              "title": "clip gradient type",
              "type": "string"
            },
            "clip_value": {
              "default": 1.0,
              "description": "Gradient clip value.",
              "title": "clip gradient value",
              "type": "float"
            },
            "enabled": {
              "default": true,
              "description": "Enable gradient clipping.",
              "title": "enable clip gradient",
              "type": "bool"
            },
            "norm_type": {
              "default": 2.0,
              "description": "Gradient clip norm type.",
              "title": "clip gradient norm type",
              "type": "float"
            }
          },
          "title": "clip gradients",
          "type": "collection"
        },
        "cudnn": {
          "automl_enabled": false,
          "default": {
            "benchmark": false,
            "deterministic": true
          },
          "properties": {
            "benchmark": {
              "default": false,
              "type": "bool"
            },
            "deterministic": {
              "default": true,
              "type": "bool"
            }
          },
          "type": "collection"
        },
        "distributed_strategy": {
          "default": "ddp",
          "description": "\n        The multi-GPU training strategy.\n        DDP (Distributed Data Parallel) and Fully Sharded DDP are supported.",
          "enum": [
            "ddp",
            "fsdp"
          ],
          "title": "distributed_strategy",
          "type": "categorical"
        },
        "freeze": {
          "automl_enabled": false,
          "default": [],
          "description": "\n        List of layer names to freeze.\n        Example: [\"backbone\", \"transformer.encoder\", \"input_proj\"].",
          "title": "freeze",
          "type": "list"
        },
        "gpu_ids": {
          "automl_enabled": false,
          "default": [
            0
          ],
          "description": "\n        List of GPU IDs to run the training on. The length of this list\n        must be equal to the number of gpus in train.num_gpus.",
          "popular": true,
          "title": "GPU IDs",
          "type": "list"
        },
        "is_dry_run": {
          "default": false,
          "description": "\n        Whether to run the trainer in Dry Run mode. This serves\n        as a good means to validate the spec file and run a sanity check on the trainer\n        without actually initializing and running the trainer.",
          "title": "Is dry run",
          "type": "bool"
        },
        "iters_per_epoch": {
          "description": "Number of iteration per epoch.",
          "title": "iteration per epoch",
          "type": "int"
        },
        "num_epochs": {
          "default": 50,
          "description": "Number of epochs to train for.",
          "title": "number of epochs",
          "type": "int"
        },
        "num_gpus": {
          "default": 8,
          "description": "Number of GPUs to train on.",
          "title": "number of gpus",
          "type": "int"
        },
        "num_nodes": {
          "default": 1,
          "description": "Number of nodes to train on.",
          "title": "number of nodes",
          "type": "int"
        },
        "optim": {
          "automl_default_parameters": [
            "train.optim.lr",
            "train.optim.backbone_multiplier",
            "train.optim.momentum",
            "train.optim.weight_decay"
          ],
          "automl_disabled_parameters": [
            "train.optim.milestones",
            "train.optim.steps"
          ],
          "automl_enabled": false,
          "default": {
            "backbone_multiplier": 0.1,
            "gamma": 0.1,
            "lr": 1e-05,
            "lr_scheduler": "Warmuppoly",
            "max_iter": 368750,
            "milestones": [
              88,
              96
            ],
            "momentum": 0.9,
            "monitor_name": "train_loss",
            "steps": [
              327778,
              355092
            ],
            "type": "AdamW",
            "warmup_factor": 0.001,
            "warmup_iters": 1000,
            "weight_decay": 0.05
          },
          "description": "Hyper parameters to configure the optimizer.",
          "popular": [
            "momentum",
            "weight_decay",
            "backbone_multiplier"
          ],
          "properties": {
            "backbone_multiplier": {
              "automl_enabled": true,
              "default": 0.1,
              "description": "A multiplier for backbone learning rate.",
              "math_cond": "> 0.0",
              "maximum": 1.0,
              "minimum": 0.0,
              "popular": true,
              "title": "backbone learning rate multiplier",
              "type": "float"
            },
            "gamma": {
              "default": 0.1,
              "description": "Multiplicative factor of learning rate decay.",
              "math_cond": "> 0.0",
              "title": "gamma",
              "type": "float"
            },
            "lr": {
              "automl_enabled": true,
              "default": 1e-05,
              "description": "The initial learning rate for training the model.",
              "math_cond": "> 0.0",
              "maximum": 1.0,
              "minimum": 0.0,
              "title": "learning rate",
              "type": "float"
            },
            "lr_scheduler": {
              "default": "Warmuppoly",
              "description": "The learning scheduler:\n                    * MultiStep : Decrease the lr by lr_decay from lr_steps\n                    * Warmuppoly : Poly learning rate schedule.",
              "enum": [
                "MultiStep",
                "Warmuppoly"
              ],
              "title": "learning rate scheduler",
              "type": "categorical"
            },
            "max_iter": {
              "default": 368750,
              "description": "Number of iterations to train for.",
              "title": "max iter",
              "type": "int"
            },
            "milestones": {
              "automl_enabled": false,
              "default": [
                88,
                96
              ],
              "description": "learning rate decay epochs.",
              "title": "learning rate decay epochs.",
              "type": "list"
            },
            "momentum": {
              "automl_enabled": true,
              "default": 0.9,
              "description": "The momentum for the AdamW optimizer.",
              "math_cond": "> 0.0",
              "maximum": 1.0,
              "minimum": 0.0,
              "popular": true,
              "title": "momentum - AdamW",
              "type": "float"
            },
            "monitor_name": {
              "default": "train_loss",
              "description": "The metric value to be monitored for the :code:`AutoReduce` Scheduler.",
              "enum": [
                "val_loss",
                "train_loss"
              ],
              "title": "monitor_name",
              "type": "categorical"
            },
            "steps": {
              "automl_enabled": false,
              "default": [
                327778,
                355092
              ],
              "description": "learning rate decay epochs.",
              "title": "learning rate decay epochs.",
              "type": "list"
            },
            "type": {
              "default": "AdamW",
              "description": "Type of optimizer used to train the network.",
              "enum": [
                "AdamW"
              ],
              "type": "categorical"
            },
            "warmup_factor": {
              "default": 0.001,
              "description": "Factor to warmup the learning rate.",
              "title": "warmup factor",
              "type": "float"
            },
            "warmup_iters": {
              "default": 1000,
              "description": "Number of iterations to warmup.",
              "title": "warmup iters",
              "type": "int"
            },
            "weight_decay": {
              "automl_enabled": true,
              "default": 0.05,
              "description": "The weight decay coefficient.",
              "math_cond": "> 0.0",
              "maximum": 1.0,
              "minimum": 0.0,
              "popular": true,
              "title": "weight decay",
              "type": "float"
            }
          },
          "title": "optimizer",
          "type": "collection"
        },
        "precision": {
          "default": "fp32",
          "description": "Precision to run the training on.",
          "enum": [
            "fp16",
            "fp32"
          ],
          "title": "precision",
          "type": "categorical"
        },
        "pretrained_backbone": {
          "default": "",
          "description": "Path to a pre-trained backbone to initialize the current training from.",
          "type": "string"
        },
        "pretrained_model": {
          "default": "",
          "description": "Path to a pre-trained OneFormer model to initialize the current training from.",
          "type": "string"
        },
        "results_dir": {
          "default": "",
          "description": "\n        Path to where all the assets generated from a task are stored.\n        ",
          "title": "Results directory",
          "type": "string"
        },
        "resume_training_checkpoint_path": {
          "default": "",
          "description": "Path to a pre-trained OneFormer model to initialize the current training from.",
          "type": "string"
        },
        "seed": {
          "default": 123,
          "description": "Seed for reproducibility.",
          "title": "seed",
          "type": "int"
        },
        "validation_interval": {
          "default": 1,
          "description": "Number of epochs to validate.",
          "title": "validation interval",
          "type": "int"
        },
        "verbose": {
          "default": false,
          "description": "\n        Flag to enable printing of detailed learning rate scaling from the optimizer.\n        ",
          "title": "enable verbose logs",
          "type": "bool"
        }
      },
      "type": "collection"
    },
    "wandb": {
      "automl_disabled_parameters": [
        "wandb.tags"
      ],
      "automl_enabled": false,
      "default": {
        "enable": true,
        "entity": "",
        "group": "",
        "name": "TAO Toolkit Training",
        "project": "TAO Toolkit",
        "reinit": false,
        "run_id": "",
        "save_code": false,
        "sync_tensorboard": false,
        "tags": [
          "tao-toolkit"
        ]
      },
      "properties": {
        "enable": {
          "default": true,
          "type": "bool"
        },
        "entity": {
          "default": "",
          "type": "string"
        },
        "group": {
          "default": "",
          "type": "string"
        },
        "name": {
          "default": "TAO Toolkit Training",
          "type": "string"
        },
        "project": {
          "default": "TAO Toolkit",
          "type": "string"
        },
        "reinit": {
          "default": false,
          "type": "bool"
        },
        "run_id": {
          "default": "",
          "type": "string"
        },
        "save_code": {
          "default": false,
          "type": "bool"
        },
        "sync_tensorboard": {
          "default": false,
          "type": "bool"
        },
        "tags": {
          "automl_enabled": false,
          "default": [
            "tao-toolkit"
          ],
          "type": "list"
        }
      },
      "type": "collection"
    }
  },
  "required": [
    "quantize"
  ],
  "type": "object",
  "x_tao_schema": {
    "action": "gen_trt_engine",
    "core_module": "oneformer",
    "model": "oneformer",
    "network_arch": "oneformer",
    "schema_action": "gen_trt_engine",
    "schema_version": 1,
    "source": "tao-core dataclass config"
  }
}

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Declares allowed tools, Needs Python, Needs Node.js

The author says: Requires docker + nvidia-container-toolkit.

Security

What our scanner noticed· Medium risk

Large base64-encoded blobs

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

NVIDIA Corporation

Publisher

NVIDIA (Official)

Source

NVIDIA/skills · skills/tao-train-oneformer · commit fce3cca

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE-APACHE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by NVIDIA Corporation or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download tao-train-oneformer.zip