Tao Train Sparse4d
NVIDIAOfficialCreated by NVIDIA Corporation
- Skill
- Workflow
- Software development
- Apache-2.0
- Medium risk
- Has scripts
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
The tao-train-sparse4d skill provides a structured workflow for preparing multi-camera data and managing the full lifecycle of Sparse4D 3D detection and tracking models within the NVIDIA TAO ecosystem. It supports data conversion, training, evaluation, inference, model export, and quantization. The skill includes specialized support for 2D-to-3D geometric distillation using released LTT weights, enabling temporal multi-camera perception workflows.
What it does
- Converts raw multi-camera scene data into OVPKL format for model training.
- Executes training, evaluation, and inference for Sparse4D models.
- Supports 2D-to-3D geometric distillation for enhanced perception.
- Exports trained models to ONNX format and performs model quantization.
- Integrates with AutoML policies for optimized training runs.
- Provides guidance on backbone selection and checkpoint management.
When to use it
- When developing 3D detection or tracking systems using multi-camera inputs.
- When adapting Sparse4D models to new datasets or specific camera configurations.
- When performing geometric distillation to leverage 2D annotations for 3D tasks.
- When preparing models for deployment through export and quantization.
Advantages
- Standardized workflow for complex multi-camera perception tasks.
- Automated handling of geometric distillation and temporal data.
- Direct integration with TAO Toolkit runtime and data services.
- Consistent artifact management across training, evaluation, and export.
Usage examples
- “Convert my raw multi-camera dataset for Sparse4D training.”
- The agent will trigger the dataset_convert action using the specified experiment template and data root, producing the necessary OVPKL files.
- “Train a Sparse4D model using ResNet-50 and geometric distillation.”
- The agent will configure the training job with the ResNet-50 backbone, apply the LTT weights, and initiate the training process on the allocated infrastructure.
Install
Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/nvidia/tao-train-sparse4d/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip tao-train-sparse4d.zip -d .agents/skills
Documentation by NVIDIA Corporation
Sparse4D
Standalone install? If the TAO skill bank plugin did not initialize the session, run
tao-setupfor host preflight, credentials, and cross-skill discovery.
Sparse4D predicts 3D boxes and tracking IDs from synchronized, calibrated multi-camera images. Start with the selected model's experiment specification and checkpoint. The canonical TAO documentation covers the model workflow.
Instructions
Identify the requested action, data supervision, selected checkpoint, and platform. Read the action contract and the selected platform skill. Preserve the user's platform, model, data, and training-budget choices.
Match the backbone, ordered classes, image preprocessing, anchors, and temporal dimensions to the selected checkpoint. Follow Backbone and checkpoint selection below before preparing the specification.
Use existing compatible info PKLs when supplied. For raw data, follow local conversion and artifact checks. For calibrated real scenes with 2D targets, read geometric distillation; that route does not require 3D annotations or conversion-generated anchors.
Start from
references/spec_template_<action>.yamland the selected release's experiment specification. Resolve paths against actual staged artifacts. Bindresults_dirand any action-specificresults_dirto the current job output directory; released specifications can contain old output paths. Write nested YAML, not flat dotted keys; do not fabricate train/val/test files or scene suffixes. Check the installed runtime exposes the fields used by the requested workflow.Resolve the model-level image for model actions and the action-level Data Services image for
dataset_convert. For a requested TAO release, use the matching skill-bank release branch and its image pins; do not substitute the default branch's release. Followtao-launch-workflowand the chosen platform'ssubmit/status/logs/cancelcontract, retaining its job record and explicit output directory. Existing user authorization applies; request missing launch approval only for work not already authorized.Verify the produced artifacts and model metrics. A successful process or short training run is a smoke result, not evidence of improved 3D accuracy. Do not report unpublished experimental fixes or another run's results as validation of the selected image.
Backbone and checkpoint selection
Respect an explicit ResNet-50 or ResNet-101 choice. For an existing Sparse4D checkpoint, use the accompanying experiment specification and release metadata to determine
model.backbone.type(resnet_50orresnet_101). If they conflict with the requested backbone, resolve that mismatch before launch; do not change the architecture to make a checkpoint load partially.If no checkpoint/backbone has been selected, ask which compatible released model to use. The schema default
resnet_101is a configuration default, not a recommendation or evidence about the user's checkpoint.Preserve the selected checkpoint's class order, anchors, input dimensions, and temporal/query dimensions. A filename alone is insufficient when its provenance is unclear.
The LTT MLP is a separate geometry adapter. Downloading it from the ResNet-50 bundle does not select the Sparse4D backbone. Its ordered taxonomy must match the training data and model.
Fine-tuning uses
train.pretrained_model_path. Continuing a run usestrain.resume_training_checkpoint_pathand a compatible saved training state. Random initialization with an empty pretrained path is suitable only when explicitly requested, for example a wiring smoke test.
Train action policy
Read references/skill_info.yaml before train requests. This model is AutoML-enabled. Preserve an explicit automl_policy: on / off; requests for plain training, no HPO, a fixed-specification reproduction, a bounded wiring smoke test, or an exact continuation mean off for that run. Otherwise default to on and route through tao-skill-bank:tao-run-automl when both schemas/train.schema.json and references/spec_template_train.yaml are available. Preserve dataset, checkpoint, specification, GPU/platform, and budget overrides. If the schema/template is missing, report that AutoML is unavailable and use direct training within the user's request.
AutoML's monitoring name is val_mAP; Sparse4D emits img_bbox_NuScenes/mAP and mAP as aliases. A promoted resume job may carry the source rung's metric without a new evaluation: report its provenance and verify the resumed checkpoint independently. Follow the AutoML skill's baseline-evaluation requirement. Evaluate, inference, export, and quantize remain in this skill.
Training Requirements
Training needs converted OVPKL data, held-out labeled validation data, matching anchors, and the selected pretrained checkpoint or resume state. Preserve the selected backbone and ordered taxonomy. Geometric distillation additionally needs the released LTT checkpoint and prepared supervision caches. Choose the GPU count and training budget from the user's request and available resources.
Per-Action Dataset Requirements
The model uses OVPKL annotations. The action contract declares staged inputs, commands, outputs, and optional artifacts. Paths embedded in info PKLs, split lists, and lazy indexes must also remain visible inside the selected runtime; staging a split file alone does not stage its referenced data.
| Action | Runtime command | Required data / model artifacts |
|---|---|---|
dataset_convert | annotations convert -e SPEC (Data Services) | Raw scene root, calibration, images; 3D annotations for labeled conversion only |
train | sparse4d train -e SPEC | Training and held-out validation PKLs/splits, image root, matching anchors, selected pretrained or resume checkpoint; optional LTT/cache artifacts for the chosen supervision route |
evaluate | sparse4d evaluate -e SPEC | evaluate.checkpoint, matching anchors, labeled held-out PKLs at dataset.test_dataset.ann_file, image root |
inference | sparse4d inference -e SPEC | inference.checkpoint, matching anchors, input PKLs at dataset.test_dataset.ann_file, image root |
export | sparse4d export -e SPEC | export.checkpoint, matching anchors and architecture, explicit export.onnx_file; no training datasets or external LTT checkpoint |
quantize | sparse4d quantize -e SPEC | quantize.model_path and the selected backend's calibration inputs; check runtime backend support |
The evaluation CLI calls the test dataloader. Setting only dataset.val_dataset.ann_file does not select its evaluation data; that field is used during training validation. Keep the unused dataset sections from the complete specification without inventing or staging unused annotation files.
Resolve model.head.instance_bank.anchor as an explicit model artifact from the selected bundle or a compatible labeled conversion. It is not inferred from a mandatory training dataset for evaluation or export.
Use checkpoint and output resolution for exact artifact handoff. Do not choose the first checkpoint found or rely on a removed SDK job resolver.
Typical Spec Overrides
For AutoML, construct spec_overrides from concrete staged paths in the table above and skill_info.yaml's data_sources; dotted keys are an AutoML override map, not the container YAML format. Set only the inputs needed by the selected action.
| Scope | Overrides to resolve |
|---|---|
| Training data | dataset.data_root, dataset.train_dataset.ann_file, dataset.val_dataset.ann_file |
| Model initialization | model.backbone.type, model.head.instance_bank.anchor, dataset.classes, and train.pretrained_model_path or train.resume_training_checkpoint_path |
| Training resources | train.num_gpus, train.num_epochs, dataset.batch_size, and dataset.num_workers, preserving the requested budget |
| Geometric distillation | The LTT/cache paths and route fields in the training fragment; preserve their validated class order and supervision policy during HPO |
| Evaluation and inference | dataset.data_root, dataset.test_dataset.ann_file, matching anchors, and the action's checkpoint |
| Output | results_dir and the action-specific output paths bound to the current job |
With dataset.lazy_load: true, stage the split's parent directory as a folder, including <split>_lazy_index.pkl beside the split and _pkl_cam_counts.pkl when used. Point ann_file into that staged directory; do not let file-only staging relocate it away from its index. Keep referenced PKLs and images visible, and rebuild the index after relocation. dataset.pkl_cam_counts_path is an optional camera-count override, not the lazy index.
Configuration and resource checks
Generated schemas and templates describe the packaged contract; schemas/manifest.json lists actions. Do not require a TAO Core checkout on the user's machine. Use the runtime's matching release specification for launch: a schema that knows a new field does not make an older image support it.
model.backbone.type: match the selected checkpoint.train.optim.lr,train.optim.lr_scheduler, andtrain.optim.grad_clip: optimizer settings are nested underoptim.dataset.batch_size: per-GPU batch size.dataset.max_camerascan reduce sampled training cameras when supported; validate the resulting supervision coverage.train.precision:bf16,fp16, orfp32, subject to hardware/runtime support. BF16 is not mandatory.train.num_gpus,train.gpu_ids,train.num_nodes: use the selected platform's allocation. One-GPU smoke training is supported; memory needs depend on the backbone, resolution, cameras, batch size, and temporal configuration.dataset.num_framesanddataset.num_bev_groupsare scheduling/data-size settings, not an activation-memory sequence-length knob. Inspect actual dataloader length and iteration counts before comparing training budgets across datasets or GPU counts.dataset.eval_dist_fcnanddataset.eval_hota: select supported detection/tracking evaluation. Keep thresholds, class order, dataset, and raw-versus-EMA checkpoint choice identical for comparisons.
Lightning manages the local workers. Do not add a second distributed launcher. Use the platform's multi-node environment contract; TAO's WORLD_SIZE launcher variable denotes node count, not the global GPU count. Mixed 3D/2D distributed training additionally needs the synchronized route settings in the geometric-distillation reference.
Error patterns
Missing annotation or image: inspect the converter outputs and embedded paths. Preserve the emitted BEV-group suffix, camera IDs, frame IDs, and mount roots. A conversion of
traindoes not also createvalandtestoutputs.Missing anchor: reuse the selected pretrained bundle's matching anchors, or use labeled conversion's generated anchors when compatible. Annotation-free conversion intentionally skips anchor initialization.
Legacy HDF5 depth tuple: inspect the affected labeled PKL and use the optional normalization helper in the conversion reference only when its documented layout matches. Unlabeled scenes have no depth paths to normalize.
Temporal OOM: first reduce per-GPU batch size or supported camera sampling; preserve checkpoint architecture. Do not blindly shrink anchor/temporal dimensions or treat
num_framesas a memory fix.LTT or teacher cache mismatch: verify ordered classes, scene/camera/frame joins, and container paths. The released LTT route supports its original seven classes; changing class names does not adapt the weights.
Export reshape: check the exporter contract in the conversion reference before training a reduced smoke model. Current export uses 20 dummy cameras and 600 cached temporal instances; successful inference alone does not establish export compatibility.
Unsupported quantize backend/image: report the selected image, backend, exact checkpoint, and failure. Do not claim a result from a historical validation image proves the current runtime works, or remove the action to hide the failure.
Files
- .gitleaks.toml252 B
- .gitleaksignore8.1 KiB
- ATTRIBUTION.mdAdded by AutoGineer924 B
- BENCHMARK.md8.2 KiB
- LICENSELicence file added by AutoGineer11.2 KiB
- SKILL.md12.8 KiB
- config/skillspector-baseline.yaml834 B
- evals/evals.json7.2 KiB
- references/geometric_distillation.md9.5 KiB
- references/geometric_distillation_train.yaml926 B
- references/local_docker_conversion.md6.8 KiB
- references/skill_info.yaml7.3 KiB
- references/spec_param_inference.md2.0 KiB
- references/spec_template_dataset_convert.yaml1.7 KiB
- references/spec_template_evaluate.yaml8.6 KiB
- references/spec_template_export.yaml8.7 KiB
- references/spec_template_inference.yaml8.7 KiB
- references/spec_template_quantize.yaml8.4 KiB
- references/spec_template_train.yaml8.4 KiB
- schemas/dataset_convert.schema.json14.4 KiB
- schemas/evaluate.schema.json136.1 KiB
- schemas/export.schema.json135.7 KiB
- schemas/inference.schema.json136.6 KiB
- schemas/manifest.json16.9 KiB
- schemas/quantize.schema.json132.3 KiB
- schemas/train.schema.json132.3 KiB
- scripts/normalize_depth_paths.pyScript3.4 KiB
- skill-card.md4.9 KiB
- skill.oms.sig9.8 KiB
schemas/export.schema.json
{
"automl_default_parameters": [
"train.optim.lr"
],
"automl_disabled_parameters": [
"model.head.deformable_model",
"model.head.anchor_encoder.embed_dims",
"train.optim.lr_scheduler",
"model.head.graph_model",
"evaluate.tracking",
"train.cudnn",
"visualize",
"model.head.instance_bank",
"dataset.quant_calibration_dataset",
"model.head.visibility_net",
"quantize.layers",
"model.neck.in_channels",
"dataset.normalize.std",
"train",
"dataset.normalize",
"model.head.loss",
"model.head.ffn.act_cfg",
"model",
"dataset.classes",
"quantize",
"train.optim.grad_clip",
"model.backbone",
"dataset.train_dataset",
"evaluate.metrics",
"model.head",
"evaluate.gpu_ids",
"dataset.val_dataset",
"dataset.real_scene_keywords",
"model.head.refine_layer",
"export",
"model.head.deformable_model.kps_generator.fix_scale",
"model.head.loss.reg.cls_allow_reverse",
"quantize.backend_kwargs",
"train.optim.paramwise_cfg",
"model.head.bnneck",
"wandb.tags",
"dataset.sequences",
"model.head.ffn.pre_norm",
"inference.gpu_ids",
"wandb",
"inference",
"model.head.loss.reg",
"model.head.ffn",
"dataset.augmentation.image_size",
"model.head.loss.id",
"model.head.temp_graph_model",
"dataset",
"model.head.anchor_encoder",
"dataset.augmentation.resize_lim",
"dataset.augmentation.rot_lim",
"dataset.augmentation",
"dataset.augmentation.final_dim",
"quantize.skip_names",
"dataset.augmentation.bot_pct_lim",
"model.head.sampler.reg_weights",
"evaluate",
"model.head.reg_weights",
"dataset.rtdetr_2d_per_class_score_thr",
"dataset.normalize.mean",
"model.head.deformable_model.kps_generator",
"model.neck",
"model.head.decoder",
"model.input_shape",
"inference.tracking",
"train.gpu_ids",
"model.head.norm_layer",
"train.checkpointer",
"dataset.target_fps_choices",
"model.head.loss.cls",
"model.head.operation_order",
"model.head.sampler.dn_noise_scale",
"model.depth_branch",
"train.optim",
"dataset.augmentation.rot3d_range",
"model.head.sampler",
"dataset.test_dataset",
"model.head.loose_to_tight"
],
"default": {
"dataset": {
"augmentation": {
"bot_pct_lim": [
0.0,
0.0
],
"final_dim": [
512,
1408
],
"image_size": [
1080,
1920
],
"rand_flip": true,
"resize_lim": [
0.7,
0.77
],
"rot3d_range": [
-0.3925,
0.3925
],
"rot_lim": [
-5.4,
5.4
]
},
"batch_size": 2,
"classes": [
"person",
"gr1_t2",
"agility_digit",
"nova_carter"
],
"data_root": "???",
"eval_dist_fcn": "center_distance",
"eval_hota": false,
"fps_drop_prob": 0.0,
"lazy_load": false,
"lazy_load_cache_size": 50,
"ltt_2dgt_cache_size": 8,
"ltt_2dgt_dedup_regex": "^CT[\\w.]+?__",
"ltt_2dgt_frame_regex": "",
"ltt_2dgt_sidecar_dir": "",
"ltt_2dgt_warn_on_miss": true,
"max_cameras": -1,
"normalize": {
"mean": [
123.675,
116.28,
103.53
],
"std": [
58.395,
57.12,
57.375
],
"to_rgb": true
},
"num_bev_groups": 1,
"num_frames": 200,
"num_ids": 70,
"num_workers": 4,
"pkl_cam_counts_path": "",
"pkl_sample_size": 0,
"quant_calibration_dataset": {
"images_dir": ""
},
"real_block_prob": -1.0,
"real_scene_keywords": [],
"rtdetr_2d_cache_dir": "",
"rtdetr_2d_cache_path": "",
"rtdetr_2d_cache_size": 4,
"rtdetr_2d_dedup_regex": "^CT[\\w.]+?__",
"rtdetr_2d_mark_real": true,
"rtdetr_2d_per_class_score_thr": {},
"rtdetr_2d_score_thr": 0.0,
"scene_switch_iters": 0,
"sequences": {
"keep_consistent_aug": true,
"same_scene_in_batch": true,
"split_num": 100
},
"sync_route": false,
"target_fps_choices": [
30,
20,
15,
10,
6,
5,
3,
2,
1
],
"test_dataset": {
"ann_file": "???",
"same_scene_in_batch": true,
"test_mode": true,
"tracking": true,
"tracking_threshold": 0.2,
"use_valid_flag": true
},
"train_dataset": {
"ann_file": "???",
"keep_consistent_seq_aug": true,
"same_scene_in_batch": true,
"sequences_split_num": 100,
"test_mode": false,
"use_valid_flag": true,
"with_seq_flag": true
},
"type": "omniverse_3d_det_track",
"use_h5_file_for_depth": true,
"use_h5_file_for_rgb": false,
"val_dataset": {
"ann_file": "???",
"same_scene_in_batch": true,
"test_mode": false,
"tracking": true,
"tracking_threshold": 0.2,
"use_valid_flag": true
}
},
"encryption_key": "",
"export": {
"batch_size": -1,
"checkpoint": "???",
"format": "onnx",
"gpu_id": 0,
"input_channel": 3,
"input_height": 544,
"input_width": 960,
"on_cpu": false,
"onnx_file": "???",
"opset_version": 17,
"results_dir": "",
"verbose": false
},
"model": {
"backbone": {
"type": "resnet_101"
},
"cotrain_param_touch": false,
"depth_branch": {
"embed_dims": 256,
"loss_weight": 0.2,
"num_depth_layers": 3,
"type": "dense_depth"
},
"embed_dims": 256,
"head": {
"anchor_encoder": {
"embed_dims": [
128,
32,
32,
64
],
"in_loops": 1,
"mode": "cat",
"out_loops": 4,
"output_fc": false,
"pos_embed_only": false,
"type": "SparseBox3DEncoder",
"vel_dims": 3
},
"bnneck": {
"feat_dim": 256,
"num_ids": 70,
"type": "bnneck"
},
"cls_threshold_to_reg": 0.05,
"decoder": {
"score_threshold": 0.05,
"type": "SparseBox3DDecoder"
},
"decouple_attn": true,
"deformable_model": {
"attn_drop": 0.15,
"embed_dims": 256,
"kps_generator": {
"embed_dims": 256,
"fix_scale": [
[
0,
0,
0
],
[
0.45,
0,
0
],
[
-0.45,
0,
0
],
[
0,
0.45,
0
],
[
0,
-0.45,
0
],
[
0,
0,
0.45
],
[
0,
0,
-0.45
]
],
"num_learnable_pts": 6
},
"max_num_cams": 20,
"num_cams": 6,
"num_groups": 8,
"num_levels": 4,
"proj_drop": 0.0,
"residual_mode": "cat",
"use_camera_embed": false,
"use_deformable_func": true
},
"drop_out": 0.1,
"embed_dims": 256,
"ffn": {
"act_cfg": {
"inplace": true,
"type": "ReLU"
},
"embed_dims": 256,
"feedforward_channels": 1024,
"ffn_drop": 0.1,
"in_channels": 512,
"num_fcs": 2,
"pre_norm": {
"normalized_shape": 256,
"type": "LN"
},
"type": "AsymmetricFFN"
},
"graph_model": {
"batch_first": true,
"dropout": 0.1,
"embed_dims": 512,
"num_heads": 8,
"type": "MultiheadAttention"
},
"instance_bank": {
"anchor": "",
"confidence_decay": 0.8,
"default_time_interval": 0.033333,
"embed_dims": 256,
"feat_grad": false,
"num_anchor": 900,
"num_temp_instances": 600,
"reset_on_time_gap": false,
"use_temporal_align": false
},
"loose_to_tight": {
"box2d_key": "gt_boxes_2d_visible",
"class_gate": true,
"containment_weight": 1.0,
"cost_cls": 1.0,
"cost_giou": 2.0,
"cost_l1": 1.0,
"dedup_dist": 0.0,
"det_box_key": "det_boxes_2d",
"det_cls_key": "det_classes_2d",
"det_score_key": "det_scores_2d",
"det_score_thr": 0.0,
"ego2cam_key": "cam2world_transform",
"enable": false,
"eps": 0.1,
"giou_thr": 0.3,
"has_3d_gt_key": "has_3d_gt",
"instance_id_key": "instance_id",
"loss_weight": 0.1,
"min_cams": 1,
"min_gt_area": 1.0,
"mlp_ckpt": "",
"num_classes": 0,
"occ_key": "gt_occ_weight",
"pseudo_box_weight": 0.1,
"pseudo_cls_weight": 1.0,
"pseudo_enable": false,
"tight_l1_weight": 1.0
},
"loss": {
"cls": {
"alpha": 0.25,
"gamma": 2.0,
"loss_weight": 2.0,
"type": "focal",
"use_sigmoid": true
},
"id": {
"num_ids": 70,
"type": "cross_entropy_label_smooth"
},
"reg": {
"box_weight": 0.25,
"cls_allow_reverse": [],
"type": "sparse_box_3d"
}
},
"norm_layer": {
"normalized_shape": 256,
"type": "LN"
},
"num_decoder": 6,
"num_groups": 8,
"num_output": 300,
"num_single_frame_decoder": 1,
"operation_order": [
"deformable",
"ffn",
"norm",
"refine",
"temp_gnn",
"gnn",
"norm",
"deformable",
"ffn",
"norm",
"refine",
"temp_gnn",
"gnn",
"norm",
"deformable",
"ffn",
"norm",
"refine",
"temp_gnn",
"gnn",
"norm",
"deformable",
"ffn",
"norm",
"refine",
"temp_gnn",
"gnn",
"norm",
"deformable",
"ffn",
"norm",
"refine",
"temp_gnn",
"gnn",
"norm",
"deformable",
"ffn",
"norm",
"refine"
],
"refine_layer": {
"embed_dims": 256,
"refine_yaw": true,
"type": "sparse_box_3d_refinement_module",
"with_quality_estimation": true
},
"reg_weights": [
2.0,
2.0,
2.0,
1.0,
1.0,
1.0,
1.0,
1.0,
1.0,
1.0,
1.0
],
"reid_dims": 0,
"return_feature": true,
"sampler": {
"add_neg_dn": true,
"box_weight": 0.25,
"cls_weight": 2.0,
"dn_noise_scale": [
2.0,
2.0,
2.0,
0.5,
0.5,
0.5,
0.5,
0.5,
0.5,
0.5,
0.5
],
"gt_assign_threshold": 0.5,
"max_dn_gt": 128,
"num_dn_groups": 5,
"num_temp_dn_groups": 3,
"reg_weights": [
2.0,
2.0,
2.0,
0.5,
0.5,
0.5,
0.0,
0.0,
0.0,
0.0,
0.0
],
"use_temporal_align": false
},
"temp_graph_model": {
"batch_first": true,
"dropout": 0.1,
"embed_dims": 512,
"num_heads": 8,
"type": "MultiheadAttention"
},
"temporal": true,
"type": "sparse4d",
"use_reid_sampling": false,
"valid_vel_weight": -1.0,
"visibility_net": {
"embedding_dim": 256,
"hidden_channels": 32,
"type": "visibility_net"
},
"with_quality_estimation": true
},
"input_shape": [
1408,
512
],
"neck": {
"add_extra_convs": "on_output",
"in_channels": [
256,
512,
1024,
2048
],
"num_outs": 4,
"out_channels": 256,
"relu_before_extra_convs": true,
"start_level": 0,
"type": "FPN"
},
"type": "sparse4d",
"use_deformable_func": true,
"use_grid_mask": true,
"use_temporal_align": false
},
"model_name": "",
"quantize": {
"algorithm": "minmax",
"backend": "torchao",
"backend_kwargs": {},
"device": "cuda",
"layers": [],
"mode": "weight_only_ptq",
"model_path": "",
"results_dir": "",
"skip_names": []
},
"results_dir": "",
"train": {
"checkpoint_interval": 1,
"checkpoint_interval_unit": "epoch",
"checkpointer": {
"auto_insert_metric_name": false,
"enable_topk": false,
"filename": "model_best_{epoch:03d}",
"replace_periodic": false,
"save_top_k": 1
},
"cudnn": {
"benchmark": false,
"deterministic": true
},
"gpu_ids": [
0
],
"num_epochs": 10,
"num_gpus": 1,
"num_nodes": 1,
"optim": {
"grad_clip": {
"max_norm": 25,
"norm_type": "L2"
},
"lr": 5e-05,
"lr_scheduler": {
"min_lr_ratio": 0.001,
"policy": "cosine",
"warmup": "linear",
"warmup_iters": 500,
"warmup_ratio": 0.333333
},
"momentum": 0.9,
"paramwise_cfg": {
"custom_keys": {
"img_backbone": {
"lr_mult": 0.2
}
}
},
"type": "adamw",
"weight_decay": 0.001
},
"precision": "bf16",
"pretrained_model_path": "",
"results_dir": "",
"resume_training_checkpoint_path": "",
"scrub_nan_gradients": false,
"seed": 1234,
"validation_interval": 1
},
"visualize": {
"n_images_col": 6,
"show": false,
"vis_dir": "./vis",
"vis_score_threshold": 0.25,
"viz_down_sample": 3
},
"wandb": {
"enable": true,
"entity": "",
"group": "",
"name": "TAO Toolkit Training",
"project": "TAO Toolkit",
"reinit": false,
"run_id": "",
"save_code": false,
"sync_tensorboard": false,
"tags": [
"tao-toolkit"
]
}
},
"popular": {
"evaluate": {
"gpu_ids": [
0
],
"num_gpus": 1,
"num_nodes": 1
},
"inference": {
"gpu_ids": [
0
],
"num_gpus": 1,
"num_nodes": 1
},
"train": {
"gpu_ids": [
0
],
"num_epochs": 10,
"num_gpus": 1,
"num_nodes": 1
}
},
"properties": {
"automl_disabled_parameters": [
"wandb",
"train",
"model",
"dataset",
"inference",
"evaluate",
"export",
"visualize",
"quantize"
],
"dataset": {
"automl_disabled_parameters": [
"dataset.target_fps_choices",
"dataset.rtdetr_2d_per_class_score_thr",
"dataset.real_scene_keywords",
"dataset.classes",
"dataset.augmentation",
"dataset.normalize",
"dataset.sequences",
"dataset.train_dataset",
"dataset.val_dataset",
"dataset.test_dataset",
"dataset.quant_calibration_dataset"
],
"automl_enabled": false,
"default": {
"augmentation": {
"bot_pct_lim": [
0.0,
0.0
],
"final_dim": [
512,
1408
],
"image_size": [
1080,
1920
],
"rand_flip": true,
"resize_lim": [
0.7,
0.77
],
"rot3d_range": [
-0.3925,
0.3925
],
"rot_lim": [
-5.4,
5.4
]
},
"batch_size": 2,
"classes": [
"person",
"gr1_t2",
"agility_digit",
"nova_carter"
],
"data_root": "???",
"eval_dist_fcn": "center_distance",
"eval_hota": false,
"fps_drop_prob": 0.0,
"lazy_load": false,
"lazy_load_cache_size": 50,
"ltt_2dgt_cache_size": 8,
"ltt_2dgt_dedup_regex": "^CT[\\w.]+?__",
"ltt_2dgt_frame_regex": "",
"ltt_2dgt_sidecar_dir": "",
"ltt_2dgt_warn_on_miss": true,
"max_cameras": -1,
"normalize": {
"mean": [
123.675,
116.28,
103.53
],
"std": [
58.395,
57.12,
57.375
],
"to_rgb": true
},
"num_bev_groups": 1,
"num_frames": 200,
"num_ids": 70,
"num_workers": 4,
"pkl_cam_counts_path": "",
"pkl_sample_size": 0,
"quant_calibration_dataset": {
"images_dir": ""
},
"real_block_prob": -1.0,
"real_scene_keywords": [],
"rtdetr_2d_cache_dir": "",
"rtdetr_2d_cache_path": "",
"rtdetr_2d_cache_size": 4,
"rtdetr_2d_dedup_regex": "^CT[\\w.]+?__",
"rtdetr_2d_mark_real": true,
"rtdetr_2d_per_class_score_thr": {},
"rtdetr_2d_score_thr": 0.0,
"scene_switch_iters": 0,
"sequences": {
"keep_consistent_aug": true,
"same_scene_in_batch": true,
"split_num": 100
},
"sync_route": false,
"target_fps_choices": [
30,
20,
15,
10,
6,
5,
3,
2,
1
],
"test_dataset": {
"ann_file": "???",
"same_scene_in_batch": true,
"test_mode": true,
"tracking": true,
"tracking_threshold": 0.2,
"use_valid_flag": true
},
"train_dataset": {
"ann_file": "???",
"keep_consistent_seq_aug": true,
"same_scene_in_batch": true,
"sequences_split_num": 100,
"test_mode": false,
"use_valid_flag": true,
"with_seq_flag": true
},
"type": "omniverse_3d_det_track",
"use_h5_file_for_depth": true,
"use_h5_file_for_rgb": false,
"val_dataset": {
"ann_file": "???",
"same_scene_in_batch": true,
"test_mode": false,
"tracking": true,
"tracking_threshold": 0.2,
"use_valid_flag": true
}
},
"description": "Dataset config",
"properties": {
"augmentation": {
"automl_disabled_parameters": [
"dataset.augmentation.resize_lim",
"dataset.augmentation.final_dim",
"dataset.augmentation.bot_pct_lim",
"dataset.augmentation.rot_lim",
"dataset.augmentation.image_size",
"dataset.augmentation.rot3d_range"
],
"automl_enabled": false,
"default": {
"bot_pct_lim": [
0.0,
0.0
],
"final_dim": [
512,
1408
],
"image_size": [
1080,
1920
],
"rand_flip": true,
"resize_lim": [
0.7,
0.77
],
"rot3d_range": [
-0.3925,
0.3925
],
"rot_lim": [
-5.4,
5.4
]
},
"description": "Augmentation config",
"properties": {
"bot_pct_lim": {
"automl_enabled": false,
"default": [
0.0,
0.0
],
"description": "Bottom percentage limits",
"title": "Bottom percentage limits",
"type": "list"
},
"final_dim": {
"automl_enabled": false,
"default": [
512,
1408
],
"description": "Final dimensions",
"title": "Final dimensions",
"type": "list"
},
"image_size": {
"automl_enabled": false,
"default": [
1080,
1920
],
"description": "Original image size",
"title": "Original image size",
"type": "list"
},
"rand_flip": {
"default": true,
"description": "Random flip",
"title": "Random flip",
"type": "bool"
},
"resize_lim": {
"automl_enabled": false,
"default": [
0.7,
0.77
],
"description": "Resize limits",
"title": "Resize limits",
"type": "list"
},
"rot3d_range": {
"automl_enabled": false,
"default": [
-0.3925,
0.3925
],
"description": "3D rotation range in radians",
"title": "3D rotation range in radians",
"type": "list"
},
"rot_lim": {
"automl_enabled": false,
"default": [
-5.4,
5.4
],
"description": "Rotation limits in degrees",
"title": "Rotation limits in degrees",
"type": "list"
}
},
"title": "Augmentation config",
"type": "collection"
},
"batch_size": {
"default": 2,
"description": "Batch size",
"maximum": Infinity,
"minimum": 1,
"title": "Batch size",
"type": "int"
},
"classes": {
"automl_enabled": false,
"default": [
"person",
"gr1_t2",
"agility_digit",
"nova_carter"
],
"description": "Classes to detect",
"title": "Classes to detect",
"type": "list"
},
"data_root": {
"default": "???",
"description": "Path to data root",
"title": "Path to data root",
"type": "string"
},
"eval_dist_fcn": {
"default": "center_distance",
"description": "Evaluation distance function",
"enum": [
"center_distance",
"iou_3d",
"both"
],
"title": "Evaluation distance function",
"type": "categorical"
},
"eval_hota": {
"default": false,
"description": "Run HOTA tracking evaluation in addition to AMOTA",
"title": "Enable HOTA evaluation",
"type": "bool"
},
"fps_drop_prob": {
"default": 0.0,
"description": "Probability of downsampling a training scene to a lower FPS",
"maximum": 1.0,
"minimum": 0.0,
"title": "FPS drop probability",
"type": "float"
},
"lazy_load": {
"default": false,
"description": "Defer pkl loading to __getitem__ using a pre-built index",
"title": "Lazy load annotations",
"type": "bool"
},
"lazy_load_cache_size": {
"default": 50,
"description": "Maximum pkl files held in the lazy-loading LRU cache",
"maximum": Infinity,
"minimum": 1,
"title": "Lazy load cache size",
"type": "int"
},
"ltt_2dgt_cache_size": {
"default": 8,
"description": "Number of per-scene LTT sidecar indices cached per worker",
"maximum": Infinity,
"minimum": 1,
"title": "LTT sidecar cache size",
"type": "int"
},
"ltt_2dgt_dedup_regex": {
"default": "^CT[\\w.]+?__",
"description": "Regex removed from scene names before resolving LTT sidecars",
"title": "LTT scene deduplication regex",
"type": "string"
},
"ltt_2dgt_frame_regex": {
"default": "",
"description": "Optional regex used to extract frame IDs for LTT sidecar joins",
"title": "LTT frame regex",
"type": "string"
},
"ltt_2dgt_sidecar_dir": {
"default": "",
"description": "Directory of per-scene LooseToTight 2D ground-truth NPZ sidecars",
"title": "LTT 2D ground-truth sidecar directory",
"type": "string"
},
"ltt_2dgt_warn_on_miss": {
"default": true,
"description": "Warn once per scene when an LTT sidecar join has no matches",
"title": "Warn on LTT sidecar miss",
"type": "bool"
},
"max_cameras": {
"default": -1,
"description": "Limit training cameras per frame to reduce GPU memory; a non-positive value disables sampling",
"maximum": Infinity,
"minimum": -1,
"title": "Maximum cameras per frame",
"type": "int"
},
"normalize": {
"automl_disabled_parameters": [
"dataset.normalize.mean",
"dataset.normalize.std"
],
"automl_enabled": false,
"default": {
"mean": [
123.675,
116.28,
103.53
],
"std": [
58.395,
57.12,
57.375
],
"to_rgb": true
},
"description": "Normalize config",
"properties": {
"mean": {
"automl_enabled": false,
"default": [
123.675,
116.28,
103.53
],
"description": "Mean values for normalization",
"title": "Mean values for normalization",
"type": "list"
},
"std": {
"automl_enabled": false,
"default": [
58.395,
57.12,
57.375
],
"description": "Standard deviation values for normalization",
"title": "Standard deviation values for normalization",
"type": "list"
},
"to_rgb": {
"default": true,
"description": "Convert to RGB",
"title": "Convert to RGB",
"type": "bool"
}
},
"title": "Normalize config",
"type": "collection"
},
"num_bev_groups": {
"default": 1,
"description": "Number of BEV groups",
"maximum": Infinity,
"minimum": 1,
"title": "Number of BEV groups",
"type": "int"
},
"num_frames": {
"default": 200,
"description": "Number of frames",
"maximum": Infinity,
"minimum": 1,
"title": "Number of frames",
"type": "int"
},
"num_ids": {
"default": 70,
"description": "Number of IDs",
"maximum": Infinity,
"minimum": 1,
"title": "Number of IDs",
"type": "int"
},
"num_workers": {
"default": 4,
"description": "Number of workers",
"maximum": Infinity,
"minimum": 0,
"title": "Number of workers",
"type": "int"
},
"pkl_cam_counts_path": {
"default": "",
"description": "Path to the pkl-to-camera-count mapping used for balanced sampling",
"title": "PKL camera counts path",
"type": "string"
},
"pkl_sample_size": {
"default": 0,
"description": "Number of PKLs sampled per epoch; zero disables sampling. Requires lazy_load=True",
"maximum": Infinity,
"minimum": 0,
"title": "PKL sample size per epoch",
"type": "int"
},
"quant_calibration_dataset": {
"automl_enabled": false,
"default": {
"images_dir": ""
},
"description": "Configurable parameters for quantization calibration dataset.",
"properties": {
"images_dir": {
"default": "",
"description": "Path to the directory containing calibration images.",
"title": "calibration images directory",
"type": "string"
}
},
"type": "collection"
},
"real_block_prob": {
"default": -1.0,
"description": "Probability in [0, 1], or exactly -1 for automatic scene-count weighting; (-1, 0) is invalid",
"maximum": 1.0,
"minimum": -1.0,
"title": "Real route block probability",
"type": "float"
},
"real_scene_keywords": {
"automl_enabled": false,
"default": [],
"description": "Scene-name substrings identifying real 2D-supervised data",
"title": "Real scene keywords",
"type": "list"
},
"rtdetr_2d_cache_dir": {
"default": "",
"description": "Directory of per-scene RT-DETR 2D pseudo-label NPZ caches",
"title": "RT-DETR 2D cache directory",
"type": "string"
},
"rtdetr_2d_cache_path": {
"default": "",
"description": "Optional single RT-DETR 2D pseudo-label NPZ cache",
"title": "RT-DETR 2D cache path",
"type": "string"
},
"rtdetr_2d_cache_size": {
"default": 4,
"description": "Number of RT-DETR scene indices cached per worker",
"maximum": Infinity,
"minimum": 1,
"title": "RT-DETR cache size",
"type": "int"
},
"rtdetr_2d_dedup_regex": {
"default": "^CT[\\w.]+?__",
"description": "Regex removed from scene names before resolving RT-DETR caches",
"title": "RT-DETR scene deduplication regex",
"type": "string"
},
"rtdetr_2d_mark_real": {
"default": true,
"description": "Mark samples with an RT-DETR scene cache as lacking 3D ground truth",
"title": "Treat RT-DETR-cached samples as 2D-only",
"type": "bool"
},
"rtdetr_2d_per_class_score_thr": {
"automl_enabled": false,
"description": "Optional class-name to score-threshold overrides",
"title": "RT-DETR per-class score thresholds",
"type": "collection"
},
"rtdetr_2d_score_thr": {
"default": 0.0,
"description": "Global score threshold for RT-DETR pseudo-labels",
"maximum": 1.0,
"minimum": 0.0,
"title": "RT-DETR score threshold",
"type": "float"
},
"scene_switch_iters": {
"default": 0,
"description": "Iterations per scene before a synchronized route switch; zero uses legacy cadence",
"maximum": Infinity,
"minimum": 0,
"title": "Scene switch iterations",
"type": "int"
},
"sequences": {
"automl_enabled": false,
"default": {
"keep_consistent_aug": true,
"same_scene_in_batch": true,
"split_num": 100
},
"description": "Sequences config",
"properties": {
"keep_consistent_aug": {
"default": true,
"description": "Keep consistent augmentation",
"title": "Keep consistent augmentation",
"type": "bool"
},
"same_scene_in_batch": {
"default": true,
"description": "Keep same scene in batch",
"title": "Keep same scene in batch",
"type": "bool"
},
"split_num": {
"default": 100,
"description": "Number of sequence splits",
"maximum": Infinity,
"minimum": 1,
"title": "Number of sequence splits",
"type": "int"
}
},
"title": "Sequences config",
"type": "collection"
},
"sync_route": {
"default": false,
"description": "Keep distributed ranks on the same 3D or 2D supervision route",
"title": "Synchronize co-training route",
"type": "bool"
},
"target_fps_choices": {
"automl_enabled": false,
"default": [
30,
20,
15,
10,
6,
5,
3,
2,
1
],
"description": "Candidate target FPS values for FPS-drop augmentation",
"title": "Target FPS choices",
"type": "list"
},
"test_dataset": {
"automl_enabled": false,
"default": {
"ann_file": "???",
"same_scene_in_batch": true,
"test_mode": true,
"tracking": true,
"tracking_threshold": 0.2,
"use_valid_flag": true
},
"description": "Test dataset config",
"properties": {
"ann_file": {
"default": "???",
"description": "Path to annotation file",
"title": "Path to annotation file",
"type": "string"
},
"same_scene_in_batch": {
"default": true,
"description": "Same scene in batch",
"title": "Same scene in batch",
"type": "bool"
},
"test_mode": {
"default": true,
"description": "Test mode",
"title": "Test mode",
"type": "bool"
},
"tracking": {
"default": true,
"description": "Tracking",
"title": "Tracking",
"type": "bool"
},
"tracking_threshold": {
"default": 0.2,
"description": "Tracking threshold",
"maximum": 1.0,
"minimum": 0.0,
"title": "Tracking threshold",
"type": "float"
},
"use_valid_flag": {
"default": true,
"description": "Use valid flag",
"title": "Use valid flag",
"type": "bool"
}
},
"title": "Test dataset config",
"type": "collection"
},
"train_dataset": {
"automl_enabled": false,
"default": {
"ann_file": "???",
"keep_consistent_seq_aug": true,
"same_scene_in_batch": true,
"sequences_split_num": 100,
"test_mode": false,
"use_valid_flag": true,
"with_seq_flag": true
},
"description": "Train dataset config",
"properties": {
"ann_file": {
"default": "???",
"description": "Path to annotation file",
"title": "Path to annotation file",
"type": "string"
},
"keep_consistent_seq_aug": {
"default": true,
"description": "Keep consistent sequence augmentation",
"title": "Keep consistent sequence augmentation",
"type": "bool"
},
"same_scene_in_batch": {
"default": true,
"description": "Same scene in batch",
"title": "Same scene in batch",
"type": "bool"
},
"sequences_split_num": {
"default": 100,
"description": "Number of sequences",
"maximum": Infinity,
"minimum": 1,
"title": "Number of sequences",
"type": "int"
},
"test_mode": {
"default": false,
"description": "Test mode",
"title": "Test mode",
"type": "bool"
},
"use_valid_flag": {
"default": true,
"description": "Use valid flag",
"title": "Use valid flag",
"type": "bool"
},
"with_seq_flag": {
"default": true,
"description": "With sequence flag",
"title": "With sequence flag",
"type": "bool"
}
},
"title": "Train dataset config",
"type": "collection"
},
"type": {
"default": "omniverse_3d_det_track",
"description": "Dataset type",
"title": "Dataset type",
"type": "string"
},
"use_h5_file_for_depth": {
"default": true,
"description": "Use H5 file for depth maps",
"title": "Use H5 file for depth",
"type": "bool"
},
"use_h5_file_for_rgb": {
"default": false,
"description": "Read RGB camera images from HDF5 files instead of image directories",
"title": "Use HDF5 for RGB images",
"type": "bool"
},
"val_dataset": {
"automl_enabled": false,
"default": {
"ann_file": "???",
"same_scene_in_batch": true,
"test_mode": false,
"tracking": true,
"tracking_threshold": 0.2,
"use_valid_flag": true
},
"description": "Val dataset config",
"properties": {
"ann_file": {
"default": "???",
"description": "Path to annotation file",
"title": "Path to annotation file",
"type": "string"
},
"same_scene_in_batch": {
"default": true,
"description": "Same scene in batch",
"title": "Same scene in batch",
"type": "bool"
},
"test_mode": {
"default": false,
"description": "Test mode",
"title": "Test mode",
"type": "bool"
},
"tracking": {
"default": true,
"description": "Tracking",
"title": "Tracking",
"type": "bool"
},
"tracking_threshold": {
"default": 0.2,
"description": "Tracking threshold",
"maximum": 1.0,
"minimum": 0.0,
"title": "Tracking threshold",
"type": "float"
},
"use_valid_flag": {
"default": true,
"description": "Use valid flag",
"title": "Use valid flag",
"type": "bool"
}
},
"title": "Val dataset config",
"type": "collection"
}
},
"title": "Dataset config",
"type": "collection"
},
"encryption_key": {
"default": "",
"description": "Key for encrypting model checkpoints",
"title": "Encryption key",
"type": "string"
},
"export": {
"automl_enabled": false,
"default": {
"batch_size": -1,
"checkpoint": "???",
"format": "onnx",
"gpu_id": 0,
"input_channel": 3,
"input_height": 544,
"input_width": 960,
"on_cpu": false,
"onnx_file": "???",
"opset_version": 17,
"results_dir": "",
"verbose": false
},
"description": "Export config",
"properties": {
"batch_size": {
"default": -1,
"description": "The batch size of the input Tensor for the engine.\n A value of :code:`-1` implies dynamic tensor shapes.",
"minimum": -1,
"title": "batch size",
"type": "int"
},
"checkpoint": {
"default": "???",
"description": "Path to the checkpoint file to run export.",
"title": "checkpoint",
"type": "string"
},
"format": {
"default": "onnx",
"description": "File format to export to.",
"enum": [
"onnx",
"xdl"
],
"title": "export format",
"type": "categorical"
},
"gpu_id": {
"default": 0,
"description": "The index of the GPU to build the TensorRT engine.",
"title": "GPU ID",
"type": "int"
},
"input_channel": {
"default": 3,
"description": "Number of channels in the input Tensor.",
"enum": [
1,
3
],
"minimum": 1,
"title": "input channel",
"type": "ordered_int"
},
"input_height": {
"default": 544,
"description": "Height of the input image tensor.",
"minimum": 32,
"title": "input height",
"type": "int"
},
"input_width": {
"default": 960,
"description": "Width of the input image tensor.",
"minimum": 32,
"title": "input width",
"type": "int"
},
"on_cpu": {
"default": false,
"description": "Flag to export CPU compatible model.",
"title": "on cpu",
"type": "bool"
},
"onnx_file": {
"default": "???",
"description": "\n Path to the onnx model file.\n ",
"title": "onnx file",
"type": "string"
},
"opset_version": {
"default": 17,
"description": "Operator set version of the ONNX model used to generate\n the TensorRT engine.",
"minimum": 1,
"title": "opset version",
"type": "int"
},
"results_dir": {
"default": "",
"description": "\n Path to where all the assets generated from a task are stored.\n ",
"title": "Results directory",
"type": "string"
},
"verbose": {
"default": false,
"description": "Flag to enable verbose TensorRT logging.",
"title": "verbose",
"type": "bool"
}
},
"title": "Export config",
"type": "collection"
},
"model": {
"automl_disabled_parameters": [
"model.input_shape",
"model.backbone",
"model.neck",
"model.depth_branch",
"model.head"
],
"automl_enabled": false,
"default": {
"backbone": {
"type": "resnet_101"
},
"cotrain_param_touch": false,
"depth_branch": {
"embed_dims": 256,
"loss_weight": 0.2,
"num_depth_layers": 3,
"type": "dense_depth"
},
"embed_dims": 256,
"head": {
"anchor_encoder": {
"embed_dims": [
128,
32,
32,
64
],
"in_loops": 1,
"mode": "cat",
"out_loops": 4,
"output_fc": false,
"pos_embed_only": false,
"type": "SparseBox3DEncoder",
"vel_dims": 3
},
"bnneck": {
"feat_dim": 256,
"num_ids": 70,
"type": "bnneck"
},
"cls_threshold_to_reg": 0.05,
"decoder": {
"score_threshold": 0.05,
"type": "SparseBox3DDecoder"
},
"decouple_attn": true,
"deformable_model": {
"attn_drop": 0.15,
"embed_dims": 256,
"kps_generator": {
"embed_dims": 256,
"fix_scale": [
[
0,
0,
0
],
[
0.45,
0,
0
],
[
-0.45,
0,
0
],
[
0,
0.45,
0
],
[
0,
-0.45,
0
],
[
0,
0,
0.45
],
[
0,
0,
-0.45
]
],
"num_learnable_pts": 6
},
"max_num_cams": 20,
"num_cams": 6,
"num_groups": 8,
"num_levels": 4,
"proj_drop": 0.0,
"residual_mode": "cat",
"use_camera_embed": false,
"use_deformable_func": true
},
"drop_out": 0.1,
"embed_dims": 256,
"ffn": {
"act_cfg": {
"inplace": true,
"type": "ReLU"
},
"embed_dims": 256,
"feedforward_channels": 1024,
"ffn_drop": 0.1,
"in_channels": 512,
"num_fcs": 2,
"pre_norm": {
"normalized_shape": 256,
"type": "LN"
},
"type": "AsymmetricFFN"
},
"graph_model": {
"batch_first": true,
"dropout": 0.1,
"embed_dims": 512,
"num_heads": 8,
"type": "MultiheadAttention"
},
"instance_bank": {
"anchor": "",
"confidence_decay": 0.8,
"default_time_interval": 0.033333,
"embed_dims": 256,
"feat_grad": false,
"num_anchor": 900,
"num_temp_instances": 600,
"reset_on_time_gap": false,
"use_temporal_align": false
},
"loose_to_tight": {
"box2d_key": "gt_boxes_2d_visible",
"class_gate": true,
"containment_weight": 1.0,
"cost_cls": 1.0,
"cost_giou": 2.0,
"cost_l1": 1.0,
"dedup_dist": 0.0,
"det_box_key": "det_boxes_2d",
"det_cls_key": "det_classes_2d",
"det_score_key": "det_scores_2d",
"det_score_thr": 0.0,
"ego2cam_key": "cam2world_transform",
"enable": false,
"eps": 0.1,
"giou_thr": 0.3,
"has_3d_gt_key": "has_3d_gt",
"instance_id_key": "instance_id",
"loss_weight": 0.1,
"min_cams": 1,
"min_gt_area": 1.0,
"mlp_ckpt": "",
"num_classes": 0,
"occ_key": "gt_occ_weight",
"pseudo_box_weight": 0.1,
"pseudo_cls_weight": 1.0,
"pseudo_enable": false,
"tight_l1_weight": 1.0
},
"loss": {
"cls": {
"alpha": 0.25,
"gamma": 2.0,
"loss_weight": 2.0,
"type": "focal",
"use_sigmoid": true
},
"id": {
"num_ids": 70,
"type": "cross_entropy_label_smooth"
},
"reg": {
"box_weight": 0.25,
"cls_allow_reverse": [],
"type": "sparse_box_3d"
}
},
"norm_layer": {
"normalized_shape": 256,
"type": "LN"
},
"num_decoder": 6,
"num_groups": 8,
"num_output": 300,
"num_single_frame_decoder": 1,
"operation_order": [
"deformable",
"ffn",
"norm",
"refine",
"temp_gnn",
"gnn",
"norm",
"deformable",
"ffn",
"norm",
"refine",
"temp_gnn",
"gnn",
"norm",
"deformable",
"ffn",
"norm",
"refine",
"temp_gnn",
"gnn",
"norm",
"deformable",
"ffn",
"norm",
"refine",
"temp_gnn",
"gnn",
"norm",
"deformable",
"ffn",
"norm",
"refine",
"temp_gnn",
"gnn",
"norm",
"deformable",
"ffn",
"norm",
"refine"
],
"refine_layer": {
"embed_dims": 256,
"refine_yaw": true,
"type": "sparse_box_3d_refinement_module",
"with_quality_estimation": true
},
"reg_weights": [
2.0,
2.0,
2.0,
1.0,
1.0,
1.0,
1.0,
1.0,
1.0,
1.0,
1.0
],
"reid_dims": 0,
"return_feature": true,
"sampler": {
"add_neg_dn": true,
"box_weight": 0.25,
"cls_weight": 2.0,
"dn_noise_scale": [
2.0,
2.0,
2.0,
0.5,
0.5,
0.5,
0.5,
0.5,
0.5,
0.5,
0.5
],
"gt_assign_threshold": 0.5,
"max_dn_gt": 128,
"num_dn_groups": 5,
"num_temp_dn_groups": 3,
"reg_weights": [
2.0,
2.0,
2.0,
0.5,
0.5,
0.5,
0.0,
0.0,
0.0,
0.0,
0.0
],
"use_temporal_align": false
},
"temp_graph_model": {
"batch_first": true,
"dropout": 0.1,
"embed_dims": 512,
"num_heads": 8,
"type": "MultiheadAttention"
},
"temporal": true,
"type": "sparse4d",
"use_reid_sampling": false,
"valid_vel_weight": -1.0,
"visibility_net": {
"embedding_dim": 256,
"hidden_channels": 32,
"type": "visibility_net"
},
"with_quality_estimation": true
},
"input_shape": [
1408,
512
],
"neck": {
"add_extra_convs": "on_output",
"in_channels": [
256,
512,
1024,
2048
],
"num_outs": 4,
"out_channels": 256,
"relu_before_extra_convs": true,
"start_level": 0,
"type": "FPN"
},
"type": "sparse4d",
"use_deformable_func": true,
"use_grid_mask": true,
"use_temporal_align": false
},
"description": "Model config",
"properties": {
"backbone": {
"automl_enabled": false,
"default": {
"type": "resnet_101"
},
"description": "Backbone config",
"properties": {
"type": {
"default": "resnet_101",
"description": "Backbone type",
"title": "Backbone type",
"type": "string"
}
},
"title": "Backbone config",
"type": "collection"
},
"cotrain_param_touch": {
"default": false,
"description": "Keep route-specific parameters in the distributed autograd graph",
"title": "Co-training parameter touch",
"type": "bool"
},
"depth_branch": {
"automl_enabled": false,
"default": {
"embed_dims": 256,
"loss_weight": 0.2,
"num_depth_layers": 3,
"type": "dense_depth"
},
"description": "Depth branch config",
"properties": {
"embed_dims": {
"default": 256,
"description": "Embedding dimensions",
"maximum": Infinity,
"minimum": 1,
"title": "Embedding dimensions",
"type": "int"
},
"loss_weight": {
"default": 0.2,
"description": "Weight for depth loss",
"maximum": Infinity,
"minimum": 0.0,
"title": "Weight for depth loss",
"type": "float"
},
"num_depth_layers": {
"default": 3,
"description": "Number of depth layers",
"maximum": Infinity,
"minimum": 1,
"title": "Number of depth layers",
"type": "int"
},
"type": {
"default": "dense_depth",
"description": "Depth branch type",
"title": "Depth branch type",
"type": "string"
}
},
"title": "Depth branch config",
"type": "collection"
},
"embed_dims": {
"default": 256,
"description": "Embedding dimensions",
"maximum": Infinity,
"minimum": 1,
"title": "Embedding dimensions",
"type": "int"
},
"head": {
"automl_disabled_parameters": [
"model.head.operation_order",
"model.head.visibility_net",
"model.head.instance_bank",
"model.head.anchor_encoder",
"model.head.sampler",
"model.head.reg_weights",
"model.head.loss",
"model.head.bnneck",
"model.head.deformable_model",
"model.head.refine_layer",
"model.head.graph_model",
"model.head.temp_graph_model",
"model.head.decoder",
"model.head.norm_layer",
"model.head.ffn",
"model.head.loose_to_tight"
],
"automl_enabled": false,
"default": {
"anchor_encoder": {
"embed_dims": [
128,
32,
32,
64
],
"in_loops": 1,
"mode": "cat",
"out_loops": 4,
"output_fc": false,
"pos_embed_only": false,
"type": "SparseBox3DEncoder",
"vel_dims": 3
},
"bnneck": {
"feat_dim": 256,
"num_ids": 70,
"type": "bnneck"
},
"cls_threshold_to_reg": 0.05,
"decoder": {
"score_threshold": 0.05,
"type": "SparseBox3DDecoder"
},
"decouple_attn": true,
"deformable_model": {
"attn_drop": 0.15,
"embed_dims": 256,
"kps_generator": {
"embed_dims": 256,
"fix_scale": [
[
0,
0,
0
],
[
0.45,
0,
0
],
[
-0.45,
0,
0
],
[
0,
0.45,
0
],
[
0,
-0.45,
0
],
[
0,
0,
0.45
],
[
0,
0,
-0.45
]
],
"num_learnable_pts": 6
},
"max_num_cams": 20,
"num_cams": 6,
"num_groups": 8,
"num_levels": 4,
"proj_drop": 0.0,
"residual_mode": "cat",
"use_camera_embed": false,
"use_deformable_func": true
},
"drop_out": 0.1,
"embed_dims": 256,
"ffn": {
"act_cfg": {
"inplace": true,
"type": "ReLU"
},
"embed_dims": 256,
"feedforward_channels": 1024,
"ffn_drop": 0.1,
"in_channels": 512,
"num_fcs": 2,
"pre_norm": {
"normalized_shape": 256,
"type": "LN"
},
"type": "AsymmetricFFN"
},
"graph_model": {
"batch_first": true,
"dropout": 0.1,
"embed_dims": 512,
"num_heads": 8,
"type": "MultiheadAttention"
},
"instance_bank": {
"anchor": "",
"confidence_decay": 0.8,
"default_time_interval": 0.033333,
"embed_dims": 256,
"feat_grad": false,
"num_anchor": 900,
"num_temp_instances": 600,
"reset_on_time_gap": false,
"use_temporal_align": false
},
"loose_to_tight": {
"box2d_key": "gt_boxes_2d_visible",
"class_gate": true,
"containment_weight": 1.0,
"cost_cls": 1.0,
"cost_giou": 2.0,
"cost_l1": 1.0,
"dedup_dist": 0.0,
"det_box_key": "det_boxes_2d",
"det_cls_key": "det_classes_2d",
"det_score_key": "det_scores_2d",
"det_score_thr": 0.0,
"ego2cam_key": "cam2world_transform",
"enable": false,
"eps": 0.1,
"giou_thr": 0.3,
"has_3d_gt_key": "has_3d_gt",
"instance_id_key": "instance_id",
"loss_weight": 0.1,
"min_cams": 1,
"min_gt_area": 1.0,
"mlp_ckpt": "",
"num_classes": 0,
"occ_key": "gt_occ_weight",
"pseudo_box_weight": 0.1,
"pseudo_cls_weight": 1.0,
"pseudo_enable": false,
"tight_l1_weight": 1.0
},
"loss": {
"cls": {
"alpha": 0.25,
"gamma": 2.0,
"loss_weight": 2.0,
"type": "focal",
"use_sigmoid": true
},
"id": {
"num_ids": 70,
"type": "cross_entropy_label_smooth"
},
"reg": {
"box_weight": 0.25,
"cls_allow_reverse": [],
"type": "sparse_box_3d"
}
},
"norm_layer": {
"normalized_shape": 256,
"type": "LN"
},
"num_decoder": 6,
"num_groups": 8,
"num_output": 300,
"num_single_frame_decoder": 1,
"operation_order": [
"deformable",
"ffn",
"norm",
"refine",
"temp_gnn",
"gnn",
"norm",
"deformable",
"ffn",
"norm",
"refine",
"temp_gnn",
"gnn",
"norm",
"deformable",
"ffn",
"norm",
"refine",
"temp_gnn",
"gnn",
"norm",
"deformable",
"ffn",
"norm",
"refine",
"temp_gnn",
"gnn",
"norm",
"deformable",
"ffn",
"norm",
"refine",
"temp_gnn",
"gnn",
"norm",
"deformable",
"ffn",
"norm",
"refine"
],
"refine_layer": {
"embed_dims": 256,
"refine_yaw": true,
"type": "sparse_box_3d_refinement_module",
"with_quality_estimation": true
},
"reg_weights": [
2.0,
2.0,
2.0,
1.0,
1.0,
1.0,
1.0,
1.0,
1.0,
1.0,
1.0
],
"reid_dims": 0,
"return_feature": true,
"sampler": {
"add_neg_dn": true,
"box_weight": 0.25,
"cls_weight": 2.0,
"dn_noise_scale": [
2.0,
2.0,
2.0,
0.5,
0.5,
0.5,
0.5,
0.5,
0.5,
0.5,
0.5
],
"gt_assign_threshold": 0.5,
"max_dn_gt": 128,
"num_dn_groups": 5,
"num_temp_dn_groups": 3,
"reg_weights": [
2.0,
2.0,
2.0,
0.5,
0.5,
0.5,
0.0,
0.0,
0.0,
0.0,
0.0
],
"use_temporal_align": false
},
"temp_graph_model": {
"batch_first": true,
"dropout": 0.1,
"embed_dims": 512,
"num_heads": 8,
"type": "MultiheadAttention"
},
"temporal": true,
"type": "sparse4d",
"use_reid_sampling": false,
"valid_vel_weight": -1.0,
"visibility_net": {
"embedding_dim": 256,
"hidden_channels": 32,
"type": "visibility_net"
},
"with_quality_estimation": true
},
"description": "Head config",
"properties": {
"anchor_encoder": {
"automl_disabled_parameters": [
"model.head.anchor_encoder.embed_dims"
],
"automl_enabled": false,
"default": {
"embed_dims": [
128,
32,
32,
64
],
"in_loops": 1,
"mode": "cat",
"out_loops": 4,
"output_fc": false,
"pos_embed_only": false,
"type": "SparseBox3DEncoder",
"vel_dims": 3
},
"description": "Anchor encoder config",
"properties": {
"embed_dims": {
"automl_enabled": false,
"default": [
128,
32,
32,
64
],
"description": "Embedding dimensions",
"title": "Embedding dimensions",
"type": "list"
},
"in_loops": {
"default": 1,
"description": "In loops",
"maximum": Infinity,
"minimum": 1,
"title": "In loops",
"type": "int"
},
"mode": {
"default": "cat",
"description": "Mode",
"enum": [
"cat",
"add"
],
"title": "Mode",
"type": "categorical"
},
"out_loops": {
"default": 4,
"description": "Out loops",
"maximum": Infinity,
"minimum": 1,
"title": "Out loops",
"type": "int"
},
"output_fc": {
"default": false,
"description": "Output FC",
"title": "Output FC",
"type": "bool"
},
"pos_embed_only": {
"default": false,
"description": "Pos embed only",
"title": "Pos embed only",
"type": "bool"
},
"type": {
"default": "SparseBox3DEncoder",
"description": "Anchor encoder type",
"title": "Anchor encoder type",
"type": "string"
},
"vel_dims": {
"default": 3,
"description": "Velocity dimensions",
"maximum": Infinity,
"minimum": 1,
"title": "Velocity dimensions",
"type": "int"
}
},
"title": "Anchor encoder config",
"type": "collection"
},
"bnneck": {
"automl_enabled": false,
"default": {
"feat_dim": 256,
"num_ids": 70,
"type": "bnneck"
},
"description": "BN neck config",
"properties": {
"feat_dim": {
"default": 256,
"description": "Feature dimension",
"maximum": Infinity,
"minimum": 1,
"title": "Feature dimension",
"type": "int"
},
"num_ids": {
"default": 70,
"description": "Number of IDs",
"maximum": Infinity,
"minimum": 1,
"title": "Number of IDs",
"type": "int"
},
"type": {
"default": "bnneck",
"description": "BNNeck type",
"title": "BNNeck type",
"type": "string"
}
},
"title": "BN neck config",
"type": "collection"
},
"cls_threshold_to_reg": {
"default": 0.05,
"description": "Classification threshold for regression",
"maximum": 1.0,
"minimum": 0.0,
"title": "Classification threshold for regression",
"type": "float"
},
"decoder": {
"automl_enabled": false,
"default": {
"score_threshold": 0.05,
"type": "SparseBox3DDecoder"
},
"description": "Decoder config",
"properties": {
"score_threshold": {
"default": 0.05,
"description": "Score threshold",
"maximum": 1.0,
"minimum": 0.0,
"title": "Score threshold",
"type": "float"
},
"type": {
"default": "SparseBox3DDecoder",
"description": "Decoder type",
"title": "Decoder type",
"type": "string"
}
},
"title": "Decoder config",
"type": "collection"
},
"decouple_attn": {
"default": true,
"description": "Decouple attention",
"title": "Decouple attention",
"type": "bool"
},
"deformable_model": {
"automl_disabled_parameters": [
"model.head.deformable_model.kps_generator"
],
"automl_enabled": false,
"default": {
"attn_drop": 0.15,
"embed_dims": 256,
"kps_generator": {
"embed_dims": 256,
"fix_scale": [
[
0,
0,
0
],
[
0.45,
0,
0
],
[
-0.45,
0,
0
],
[
0,
0.45,
0
],
[
0,
-0.45,
0
],
[
0,
0,
0.45
],
[
0,
0,
-0.45
]
],
"num_learnable_pts": 6
},
"max_num_cams": 20,
"num_cams": 6,
"num_groups": 8,
"num_levels": 4,
"proj_drop": 0.0,
"residual_mode": "cat",
"use_camera_embed": false,
"use_deformable_func": true
},
"description": "Deformable model config",
"properties": {
"attn_drop": {
"default": 0.15,
"description": "Attention dropout",
"maximum": 1.0,
"minimum": 0.0,
"title": "Attention dropout",
"type": "float"
},
"embed_dims": {
"default": 256,
"description": "Embedding dimensions",
"maximum": Infinity,
"minimum": 1,
"title": "Embedding dimensions",
"type": "int"
},
"kps_generator": {
"automl_disabled_parameters": [
"model.head.deformable_model.kps_generator.fix_scale"
],
"automl_enabled": false,
"default": {
"embed_dims": 256,
"fix_scale": [
[
0,
0,
0
],
[
0.45,
0,
0
],
[
-0.45,
0,
0
],
[
0,
0.45,
0
],
[
0,
-0.45,
0
],
[
0,
0,
0.45
],
[
0,
0,
-0.45
]
],
"num_learnable_pts": 6
},
"description": "KPS generator config",
"properties": {
"embed_dims": {
"default": 256,
"description": "Embedding dimensions",
"maximum": Infinity,
"minimum": 1,
"title": "Embedding dimensions",
"type": "int"
},
"fix_scale": {
"automl_enabled": false,
"default": [
[
0,
0,
0
],
[
0.45,
0,
0
],
[
-0.45,
0,
0
],
[
0,
0.45,
0
],
[
0,
-0.45,
0
],
[
0,
0,
0.45
],
[
0,
0,
-0.45
]
],
"description": "Fixed scale",
"title": "Fixed scale",
"type": "list"
},
"num_learnable_pts": {
"default": 6,
"description": "Number of learnable points",
"maximum": Infinity,
"minimum": 1,
"title": "Number of learnable points",
"type": "int"
}
},
"title": "KPS generator config",
"type": "collection"
},
"max_num_cams": {
"default": 20,
"description": "Maximum number of cameras",
"maximum": Infinity,
"minimum": 1,
"title": "Maximum number of cameras",
"type": "int"
},
"num_cams": {
"default": 6,
"description": "Number of cameras",
"maximum": Infinity,
"minimum": 1,
"title": "Number of cameras",
"type": "int"
},
"num_groups": {
"default": 8,
"description": "Number of groups",
"maximum": Infinity,
"minimum": 1,
"title": "Number of groups",
"type": "int"
},
"num_levels": {
"default": 4,
"description": "Number of levels",
"maximum": Infinity,
"minimum": 1,
"title": "Number of levels",
"type": "int"
},
"proj_drop": {
"default": 0.0,
"description": "Projection dropout",
"maximum": 1.0,
"minimum": 0.0,
"title": "Projection dropout",
"type": "float"
},
"residual_mode": {
"default": "cat",
"description": "Residual mode",
"enum": [
"cat",
"add"
],
"title": "Residual mode",
"type": "categorical"
},
"use_camera_embed": {
"default": false,
"description": "Use camera embedding",
"title": "Use camera embedding",
"type": "bool"
},
"use_deformable_func": {
"default": true,
"description": "Use deformable function",
"title": "Use deformable function",
"type": "bool"
}
},
"title": "Deformable model config",
"type": "collection"
},
"drop_out": {
"default": 0.1,
"description": "Dropout rate",
"maximum": 1.0,
"minimum": 0.0,
"title": "Dropout rate",
"type": "float"
},
"embed_dims": {
"default": 256,
"description": "Embedding dimensions",
"maximum": Infinity,
"minimum": 1,
"title": "Embedding dimensions",
"type": "int"
},
"ffn": {
"automl_disabled_parameters": [
"model.head.ffn.pre_norm",
"model.head.ffn.act_cfg"
],
"automl_enabled": false,
"default": {
"act_cfg": {
"inplace": true,
"type": "ReLU"
},
"embed_dims": 256,
"feedforward_channels": 1024,
"ffn_drop": 0.1,
"in_channels": 512,
"num_fcs": 2,
"pre_norm": {
"normalized_shape": 256,
"type": "LN"
},
"type": "AsymmetricFFN"
},
"description": "FFN config",
"properties": {
"act_cfg": {
"automl_enabled": false,
"default": {
"inplace": true,
"type": "ReLU"
},
"description": "Activation config",
"properties": {
"inplace": {
"default": true,
"description": "Inplace",
"title": "Inplace",
"type": "bool"
},
"type": {
"default": "ReLU",
"description": "Activation type",
"title": "Activation type",
"type": "string"
}
},
"title": "Activation config",
"type": "collection"
},
"embed_dims": {
"default": 256,
"description": "Embedding dimensions",
"maximum": Infinity,
"minimum": 1,
"title": "Embedding dimensions",
"type": "int"
},
"feedforward_channels": {
"default": 1024,
"description": "Feedforward channels",
"maximum": Infinity,
"minimum": 1,
"title": "Feedforward channels",
"type": "int"
},
"ffn_drop": {
"default": 0.1,
"description": "FFN dropout",
"maximum": 1.0,
"minimum": 0.0,
"title": "FFN dropout",
"type": "float"
},
"in_channels": {
"default": 512,
"description": "In channels",
"maximum": Infinity,
"minimum": 1,
"title": "In channels",
"type": "int"
},
"num_fcs": {
"default": 2,
"description": "Number of feedforward channels",
"maximum": Infinity,
"minimum": 1,
"title": "Number of feedforward channels",
"type": "int"
},
"pre_norm": {
"automl_enabled": false,
"default": {
"normalized_shape": 256,
"type": "LN"
},
"description": "Pre-norm config",
"properties": {
"normalized_shape": {
"default": 256,
"description": "Normalized shape",
"maximum": Infinity,
"minimum": 1,
"title": "Normalized shape",
"type": "int"
},
"type": {
"default": "LN",
"description": "Norm layer type",
"title": "Norm layer type",
"type": "string"
}
},
"title": "Pre-norm config",
"type": "collection"
},
"type": {
"default": "AsymmetricFFN",
"description": "FFN type",
"title": "FFN type",
"type": "string"
}
},
"title": "FFN config",
"type": "collection"
},
"graph_model": {
"automl_enabled": false,
"default": {
"batch_first": true,
"dropout": 0.1,
"embed_dims": 512,
"num_heads": 8,
"type": "MultiheadAttention"
},
"description": "Graph model config",
"properties": {
"batch_first": {
"default": true,
"description": "Batch first",
"title": "Batch first",
"type": "bool"
},
"dropout": {
"default": 0.1,
"description": "Dropout rate",
"maximum": 1.0,
"minimum": 0.0,
"title": "Dropout rate",
"type": "float"
},
"embed_dims": {
"default": 512,
"description": "Embedding dimensions",
"maximum": Infinity,
"minimum": 1,
"title": "Embedding dimensions",
"type": "int"
},
"num_heads": {
"default": 8,
"description": "Number of heads",
"maximum": Infinity,
"minimum": 1,
"title": "Number of heads",
"type": "int"
},
"type": {
"default": "MultiheadAttention",
"description": "Graph model type",
"title": "Graph model type",
"type": "string"
}
},
"title": "Graph model config",
"type": "collection"
},
"instance_bank": {
"automl_enabled": false,
"default": {
"anchor": "",
"confidence_decay": 0.8,
"default_time_interval": 0.033333,
"embed_dims": 256,
"feat_grad": false,
"num_anchor": 900,
"num_temp_instances": 600,
"reset_on_time_gap": false,
"use_temporal_align": false
},
"description": "Instance bank config",
"properties": {
"anchor": {
"default": "",
"description": "Path to anchor file",
"title": "Path to anchor file",
"type": "string"
},
"confidence_decay": {
"default": 0.8,
"description": "Confidence decay factor",
"maximum": 1.0,
"minimum": 0.0,
"title": "Confidence decay factor",
"type": "float"
},
"default_time_interval": {
"default": 0.033333,
"description": "Default time interval",
"maximum": Infinity,
"minimum": 0.0,
"title": "Default time interval",
"type": "float"
},
"embed_dims": {
"default": 256,
"description": "Embedding dimensions",
"maximum": Infinity,
"minimum": 1,
"title": "Embedding dimensions",
"type": "int"
},
"feat_grad": {
"default": false,
"description": "Enable gradients for features",
"title": "Enable gradients for features",
"type": "bool"
},
"grid_size": {
"description": "Grid size",
"title": "Grid size",
"type": "float"
},
"num_anchor": {
"default": 900,
"description": "Number of anchors",
"maximum": Infinity,
"minimum": 1,
"title": "Number of anchors",
"type": "int"
},
"num_temp_instances": {
"default": 600,
"description": "Number of temporal instances",
"maximum": Infinity,
"minimum": 0,
"title": "Number of temporal instances",
"type": "int"
},
"reset_on_time_gap": {
"default": false,
"description": "Reset recurrent state when timestamps cross a clip boundary",
"title": "Reset on time gap",
"type": "bool"
},
"use_temporal_align": {
"default": false,
"description": "Use temporal alignment",
"title": "Use temporal alignment",
"type": "bool"
}
},
"title": "Instance bank config",
"type": "collection"
},
"loose_to_tight": {
"automl_enabled": false,
"default": {
"box2d_key": "gt_boxes_2d_visible",
"class_gate": true,
"containment_weight": 1.0,
"cost_cls": 1.0,
"cost_giou": 2.0,
"cost_l1": 1.0,
"dedup_dist": 0.0,
"det_box_key": "det_boxes_2d",
"det_cls_key": "det_classes_2d",
"det_score_key": "det_scores_2d",
"det_score_thr": 0.0,
"ego2cam_key": "cam2world_transform",
"enable": false,
"eps": 0.1,
"giou_thr": 0.3,
"has_3d_gt_key": "has_3d_gt",
"instance_id_key": "instance_id",
"loss_weight": 0.1,
"min_cams": 1,
"min_gt_area": 1.0,
"mlp_ckpt": "",
"num_classes": 0,
"occ_key": "gt_occ_weight",
"pseudo_box_weight": 0.1,
"pseudo_cls_weight": 1.0,
"pseudo_enable": false,
"tight_l1_weight": 1.0
},
"description": "Loose-to-tight geometric 2D distillation config",
"properties": {
"box2d_key": {
"default": "gt_boxes_2d_visible",
"description": "Batch key for visible 2D ground-truth boxes",
"title": "Visible 2D box key",
"type": "string"
},
"class_gate": {
"default": true,
"description": "Require class agreement during pseudo-label matching",
"title": "Pseudo-match class gate",
"type": "bool"
},
"containment_weight": {
"default": 1.0,
"description": "Weight for the visible-box containment term",
"maximum": Infinity,
"minimum": 0.0,
"title": "Containment weight",
"type": "float"
},
"cost_cls": {
"default": 1.0,
"description": "Classification component weight in pseudo-label matching cost",
"maximum": Infinity,
"minimum": 0.0,
"title": "Pseudo-match class cost",
"type": "float"
},
"cost_giou": {
"default": 2.0,
"description": "GIoU component weight in pseudo-label matching cost",
"maximum": Infinity,
"minimum": 0.0,
"title": "Pseudo-match GIoU cost",
"type": "float"
},
"cost_l1": {
"default": 1.0,
"description": "L1 component weight in pseudo-label matching cost",
"maximum": Infinity,
"minimum": 0.0,
"title": "Pseudo-match L1 cost",
"type": "float"
},
"dedup_dist": {
"default": 0.0,
"description": "BEV distance used to deduplicate pseudo matches; zero disables it",
"maximum": Infinity,
"minimum": 0.0,
"title": "Pseudo-match deduplication distance",
"type": "float"
},
"det_box_key": {
"default": "det_boxes_2d",
"description": "Batch key for per-camera 2D detection boxes",
"title": "2D detection box key",
"type": "string"
},
"det_cls_key": {
"default": "det_classes_2d",
"description": "Batch key for per-camera 2D detection classes",
"title": "2D detection class key",
"type": "string"
},
"det_score_key": {
"default": "det_scores_2d",
"description": "Batch key for per-camera 2D detection scores",
"title": "2D detection score key",
"type": "string"
},
"det_score_thr": {
"default": 0.0,
"description": "Minimum 2D detection confidence used for matching",
"maximum": 1.0,
"minimum": 0.0,
"title": "2D detection score threshold",
"type": "float"
},
"ego2cam_key": {
"default": "cam2world_transform",
"description": "Batch key for ego-to-camera transforms",
"title": "Ego-to-camera key",
"type": "string"
},
"enable": {
"default": false,
"description": "Enable loose-to-tight 2D distillation",
"title": "Enable loose-to-tight distillation",
"type": "bool"
},
"eps": {
"default": 0.1,
"description": "Minimum projection depth used for numerical stability",
"maximum": Infinity,
"minimum": 0.0,
"title": "Projection epsilon",
"type": "float"
},
"giou_thr": {
"default": 0.3,
"description": "Minimum generalized IoU for pseudo-label matching",
"maximum": 1.0,
"minimum": -1.0,
"title": "Pseudo-match GIoU threshold",
"type": "float"
},
"has_3d_gt_key": {
"default": "has_3d_gt",
"description": "Batch key indicating whether a sample has 3D ground truth",
"title": "Has 3D ground truth key",
"type": "string"
},
"instance_id_key": {
"default": "instance_id",
"description": "Batch key for instance identifiers",
"title": "Instance ID key",
"type": "string"
},
"loss_weight": {
"default": 0.1,
"description": "Weight for supervised loose-to-tight 2D box loss",
"maximum": Infinity,
"minimum": 0.0,
"title": "Loose-to-tight loss weight",
"type": "float"
},
"min_cams": {
"default": 1,
"description": "Minimum number of cameras supporting a pseudo match",
"maximum": Infinity,
"minimum": 1,
"title": "Minimum matching cameras",
"type": "int"
},
"min_gt_area": {
"default": 1.0,
"description": "Minimum visible 2D ground-truth area in pixels",
"maximum": Infinity,
"minimum": 0.0,
"title": "Minimum 2D GT area",
"type": "float"
},
"mlp_ckpt": {
"default": "",
"description": "Path to the trained frozen loose-to-tight MLP checkpoint",
"title": "Loose-to-tight MLP checkpoint",
"type": "string"
},
"num_classes": {
"default": 0,
"description": "Number of classes used by the loose-to-tight MLP; zero derives the taxonomy from dataset.classes",
"maximum": Infinity,
"minimum": 0,
"title": "Loose-to-tight classes",
"type": "int"
},
"occ_key": {
"default": "gt_occ_weight",
"description": "Batch key for 2D occlusion weights",
"title": "Occlusion weight key",
"type": "string"
},
"pseudo_box_weight": {
"default": 0.1,
"description": "Weight for pseudo-label 2D box loss",
"maximum": Infinity,
"minimum": 0.0,
"title": "Pseudo 2D box loss weight",
"type": "float"
},
"pseudo_cls_weight": {
"default": 1.0,
"description": "Weight for pseudo-label classification loss",
"maximum": Infinity,
"minimum": 0.0,
"title": "Pseudo classification loss weight",
"type": "float"
},
"pseudo_enable": {
"default": false,
"description": "Enable 2D pseudo-label loss for batches without 3D ground truth",
"title": "Enable 2D pseudo labels",
"type": "bool"
},
"tight_l1_weight": {
"default": 1.0,
"description": "Weight for the tight-box L1 term",
"maximum": Infinity,
"minimum": 0.0,
"title": "Tight L1 weight",
"type": "float"
}
},
"title": "Loose-to-tight config",
"type": "collection"
},
"loss": {
"automl_disabled_parameters": [
"model.head.loss.cls",
"model.head.loss.reg",
"model.head.loss.id"
],
"automl_enabled": false,
"default": {
"cls": {
"alpha": 0.25,
"gamma": 2.0,
"loss_weight": 2.0,
"type": "focal",
"use_sigmoid": true
},
"id": {
"num_ids": 70,
"type": "cross_entropy_label_smooth"
},
"reg": {
"box_weight": 0.25,
"cls_allow_reverse": [],
"type": "sparse_box_3d"
}
},
"description": "Loss config",
"properties": {
"cls": {
"automl_enabled": false,
"default": {
"alpha": 0.25,
"gamma": 2.0,
"loss_weight": 2.0,
"type": "focal",
"use_sigmoid": true
},
"description": "Classification loss config",
"properties": {
"alpha": {
"default": 0.25,
"description": "Focal loss alpha",
"maximum": 1.0,
"minimum": 0.0,
"title": "Focal loss alpha",
"type": "float"
},
"gamma": {
"default": 2.0,
"description": "Focal loss gamma",
"maximum": Infinity,
"minimum": 0.0,
"title": "Focal loss gamma",
"type": "float"
},
"loss_weight": {
"default": 2.0,
"description": "Loss weight",
"maximum": Infinity,
"minimum": 0.0,
"title": "Loss weight",
"type": "float"
},
"type": {
"default": "focal",
"description": "Classification loss type",
"title": "Classification loss type",
"type": "string"
},
"use_sigmoid": {
"default": true,
"description": "Use sigmoid",
"title": "Use sigmoid",
"type": "bool"
}
},
"title": "Classification loss config",
"type": "collection"
},
"id": {
"automl_enabled": false,
"default": {
"num_ids": 70,
"type": "cross_entropy_label_smooth"
},
"description": "ID loss config",
"properties": {
"num_ids": {
"default": 70,
"description": "Number of IDs",
"maximum": Infinity,
"minimum": 1,
"title": "Number of IDs",
"type": "int"
},
"type": {
"default": "cross_entropy_label_smooth",
"description": "ID loss type",
"title": "ID loss type",
"type": "string"
}
},
"title": "ID loss config",
"type": "collection"
},
"reg": {
"automl_disabled_parameters": [
"model.head.loss.reg.cls_allow_reverse"
],
"automl_enabled": false,
"default": {
"box_weight": 0.25,
"cls_allow_reverse": [],
"type": "sparse_box_3d"
},
"description": "Regression loss config",
"properties": {
"box_weight": {
"default": 0.25,
"description": "Box loss weight",
"maximum": Infinity,
"minimum": 0.0,
"title": "Box loss weight",
"type": "float"
},
"cls_allow_reverse": {
"automl_enabled": false,
"default": [],
"description": "Class allow reverse",
"title": "Class allow reverse",
"type": "list"
},
"type": {
"default": "sparse_box_3d",
"description": "Regression loss type",
"title": "Regression loss type",
"type": "string"
}
},
"title": "Regression loss config",
"type": "collection"
}
},
"title": "Loss config",
"type": "collection"
},
"norm_layer": {
"automl_enabled": false,
"default": {
"normalized_shape": 256,
"type": "LN"
},
"description": "Norm layer config",
"properties": {
"normalized_shape": {
"default": 256,
"description": "Normalized shape",
"maximum": Infinity,
"minimum": 1,
"title": "Normalized shape",
"type": "int"
},
"type": {
"default": "LN",
"description": "Norm layer type",
"title": "Norm layer type",
"type": "string"
}
},
"title": "Norm layer config",
"type": "collection"
},
"num_decoder": {
"default": 6,
"description": "Number of decoder layers",
"maximum": Infinity,
"minimum": 1,
"title": "Number of decoder layers",
"type": "int"
},
"num_groups": {
"default": 8,
"description": "Number of groups",
"maximum": Infinity,
"minimum": 1,
"title": "Number of groups",
"type": "int"
},
"num_output": {
"default": 300,
"description": "Number of output instances",
"maximum": Infinity,
"minimum": 1,
"title": "Number of output instances",
"type": "int"
},
"num_single_frame_decoder": {
"default": 1,
"description": "Number of single-frame decoder layers",
"maximum": Infinity,
"minimum": 1,
"title": "Number of single-frame decoder layers",
"type": "int"
},
"operation_order": {
"automl_enabled": false,
"default": [
"deformable",
"ffn",
"norm",
"refine",
"temp_gnn",
"gnn",
"norm",
"deformable",
"ffn",
"norm",
"refine",
"temp_gnn",
"gnn",
"norm",
"deformable",
"ffn",
"norm",
"refine",
"temp_gnn",
"gnn",
"norm",
"deformable",
"ffn",
"norm",
"refine",
"temp_gnn",
"gnn",
"norm",
"deformable",
"ffn",
"norm",
"refine",
"temp_gnn",
"gnn",
"norm",
"deformable",
"ffn",
"norm",
"refine"
],
"description": "Operation order",
"title": "Operation order",
"type": "list"
},
"refine_layer": {
"automl_enabled": false,
"default": {
"embed_dims": 256,
"refine_yaw": true,
"type": "sparse_box_3d_refinement_module",
"with_quality_estimation": true
},
"description": "Refine layer config",
"properties": {
"embed_dims": {
"default": 256,
"description": "Embedding dimensions",
"maximum": Infinity,
"minimum": 1,
"title": "Embedding dimensions",
"type": "int"
},
"refine_yaw": {
"default": true,
"description": "Refine yaw",
"title": "Refine yaw",
"type": "bool"
},
"type": {
"default": "sparse_box_3d_refinement_module",
"description": "Refine layer type",
"title": "Refine layer type",
"type": "string"
},
"with_quality_estimation": {
"default": true,
"description": "With quality estimation",
"title": "With quality estimation",
"type": "bool"
}
},
"title": "Refine layer config",
"type": "collection"
},
"reg_weights": {
"automl_enabled": false,
"default": [
2.0,
2.0,
2.0,
1.0,
1.0,
1.0,
1.0,
1.0,
1.0,
1.0,
1.0
],
"description": "Regression weights",
"title": "Regression weights",
"type": "list"
},
"reid_dims": {
"default": 0,
"description": "Re-ID dimensions",
"maximum": Infinity,
"minimum": 0,
"title": "Re-ID dimensions",
"type": "int"
},
"return_feature": {
"default": true,
"description": "Return instance features",
"title": "Return instance features",
"type": "bool"
},
"sampler": {
"automl_disabled_parameters": [
"model.head.sampler.dn_noise_scale",
"model.head.sampler.reg_weights"
],
"automl_enabled": false,
"default": {
"add_neg_dn": true,
"box_weight": 0.25,
"cls_weight": 2.0,
"dn_noise_scale": [
2.0,
2.0,
2.0,
0.5,
0.5,
0.5,
0.5,
0.5,
0.5,
0.5,
0.5
],
"gt_assign_threshold": 0.5,
"max_dn_gt": 128,
"num_dn_groups": 5,
"num_temp_dn_groups": 3,
"reg_weights": [
2.0,
2.0,
2.0,
0.5,
0.5,
0.5,
0.0,
0.0,
0.0,
0.0,
0.0
],
"use_temporal_align": false
},
"description": "Sampler config",
"properties": {
"add_neg_dn": {
"default": true,
"description": "Add negative DN",
"title": "Add negative DN",
"type": "bool"
},
"box_weight": {
"default": 0.25,
"description": "Box weight",
"maximum": Infinity,
"minimum": 0.0,
"title": "Box weight",
"type": "float"
},
"cls_weight": {
"default": 2.0,
"description": "Classification weight",
"maximum": Infinity,
"minimum": 0.0,
"title": "Classification weight",
"type": "float"
},
"dn_noise_scale": {
"automl_enabled": false,
"default": [
2.0,
2.0,
2.0,
0.5,
0.5,
0.5,
0.5,
0.5,
0.5,
0.5,
0.5
],
"description": "DN noise scale",
"title": "DN noise scale",
"type": "list"
},
"gt_assign_threshold": {
"default": 0.5,
"description": "GT assign threshold",
"maximum": 1.0,
"minimum": 0.0,
"title": "GT assign threshold",
"type": "float"
},
"max_dn_gt": {
"default": 128,
"description": "Maximum DN ground truth",
"maximum": Infinity,
"minimum": 1,
"title": "Maximum DN ground truth",
"type": "int"
},
"num_dn_groups": {
"default": 5,
"description": "Number of DN groups",
"maximum": Infinity,
"minimum": 1,
"title": "Number of DN groups",
"type": "int"
},
"num_temp_dn_groups": {
"default": 3,
"description": "Number of temporal DN groups",
"maximum": Infinity,
"minimum": 0,
"title": "Number of temporal DN groups",
"type": "int"
},
"reg_weights": {
"automl_enabled": false,
"default": [
2.0,
2.0,
2.0,
0.5,
0.5,
0.5,
0.0,
0.0,
0.0,
0.0,
0.0
],
"description": "Regression weights",
"title": "Regression weights",
"type": "list"
},
"use_temporal_align": {
"default": false,
"description": "Use temporal alignment",
"title": "Use temporal alignment",
"type": "bool"
}
},
"title": "Sampler config",
"type": "collection"
},
"temp_graph_model": {
"automl_enabled": false,
"default": {
"batch_first": true,
"dropout": 0.1,
"embed_dims": 512,
"num_heads": 8,
"type": "MultiheadAttention"
},
"description": "Temp graph model config",
"properties": {
"batch_first": {
"default": true,
"description": "Batch first",
"title": "Batch first",
"type": "bool"
},
"dropout": {
"default": 0.1,
"description": "Dropout rate",
"maximum": 1.0,
"minimum": 0.0,
"title": "Dropout rate",
"type": "float"
},
"embed_dims": {
"default": 512,
"description": "Embedding dimensions",
"maximum": Infinity,
"minimum": 1,
"title": "Embedding dimensions",
"type": "int"
},
"num_heads": {
"default": 8,
"description": "Number of heads",
"maximum": Infinity,
"minimum": 1,
"title": "Number of heads",
"type": "int"
},
"type": {
"default": "MultiheadAttention",
"description": "Graph model type",
"title": "Graph model type",
"type": "string"
}
},
"title": "Temp graph model config",
"type": "collection"
},
"temporal": {
"default": true,
"description": "Enable temporal modeling",
"title": "Enable temporal modeling",
"type": "bool"
},
"type": {
"default": "sparse4d",
"description": "Head type",
"title": "Head type",
"type": "string"
},
"use_reid_sampling": {
"default": false,
"description": "Use Re-ID sampling",
"title": "Use Re-ID sampling",
"type": "bool"
},
"valid_vel_weight": {
"default": -1.0,
"description": "Valid velocity weight",
"maximum": Infinity,
"minimum": -1.0,
"title": "Valid velocity weight",
"type": "float"
},
"visibility_net": {
"automl_enabled": false,
"default": {
"embedding_dim": 256,
"hidden_channels": 32,
"type": "visibility_net"
},
"description": "Visibility net config",
"properties": {
"embedding_dim": {
"default": 256,
"description": "Embedding dimension",
"maximum": Infinity,
"minimum": 1,
"title": "Embedding dimension",
"type": "int"
},
"hidden_channels": {
"default": 32,
"description": "Hidden channels",
"maximum": Infinity,
"minimum": 1,
"title": "Hidden channels",
"type": "int"
},
"type": {
"default": "visibility_net",
"description": "VisibilityNet type",
"title": "VisibilityNet type",
"type": "string"
}
},
"title": "Visibility net config",
"type": "collection"
},
"with_quality_estimation": {
"default": true,
"description": "Enable quality estimation",
"title": "Enable quality estimation",
"type": "bool"
}
},
"title": "Head config",
"type": "collection"
},
"input_shape": {
"automl_enabled": false,
"default": [
1408,
512
],
"description": "Input image shape",
"title": "Input image shape",
"type": "list"
},
"neck": {
"automl_disabled_parameters": [
"model.neck.in_channels"
],
"automl_enabled": false,
"default": {
"add_extra_convs": "on_output",
"in_channels": [
256,
512,
1024,
2048
],
"num_outs": 4,
"out_channels": 256,
"relu_before_extra_convs": true,
"start_level": 0,
"type": "FPN"
},
"description": "Neck config",
"properties": {
"add_extra_convs": {
"default": "on_output",
"description": "Type of extra conv",
"enum": [
"on_input",
"on_lateral",
"on_output",
"False"
],
"title": "Type of extra conv",
"type": "categorical"
},
"in_channels": {
"automl_enabled": false,
"default": [
256,
512,
1024,
2048
],
"description": "Input channels",
"title": "Input channels",
"type": "list"
},
"num_outs": {
"default": 4,
"description": "Number of output levels",
"maximum": Infinity,
"minimum": 1,
"title": "Number of output levels",
"type": "int"
},
"out_channels": {
"default": 256,
"description": "Output channels",
"maximum": Infinity,
"minimum": 1,
"title": "Output channels",
"type": "int"
},
"relu_before_extra_convs": {
"default": true,
"description": "Apply ReLU before extra convs",
"title": "Apply ReLU before extra convs",
"type": "bool"
},
"start_level": {
"default": 0,
"description": "Start level for FPN",
"maximum": Infinity,
"minimum": 0,
"title": "Start level for FPN",
"type": "int"
},
"type": {
"default": "FPN",
"description": "Neck type",
"enum": [
"FPN"
],
"title": "Neck type",
"type": "categorical"
}
},
"title": "Neck config",
"type": "collection"
},
"type": {
"default": "sparse4d",
"description": "Model type",
"title": "Model type",
"type": "string"
},
"use_deformable_func": {
"default": true,
"description": "Use deformable function",
"title": "Use deformable function",
"type": "bool"
},
"use_grid_mask": {
"default": true,
"description": "Use grid mask",
"title": "Use grid mask",
"type": "bool"
},
"use_temporal_align": {
"default": false,
"description": "Use temporal alignment",
"title": "Use temporal alignment",
"type": "bool"
}
},
"title": "Model config",
"type": "collection"
},
"model_name": {
"default": "",
"description": "Name of model if invoking task via :code:`model_agnostic`",
"title": "Model name",
"type": "string"
},
"quantize": {
"automl_disabled_parameters": [
"quantize.layers",
"quantize.skip_names",
"quantize.backend_kwargs"
],
"automl_enabled": false,
"default": {
"algorithm": "minmax",
"backend": "torchao",
"backend_kwargs": {},
"device": "cuda",
"layers": [],
"mode": "weight_only_ptq",
"model_path": "",
"results_dir": "",
"skip_names": []
},
"description": "Configurable parameters for model quantization.",
"properties": {
"algorithm": {
"default": "minmax",
"description": "Calibration/optimization algorithm. Used by ModelOpt backends (modelopt.pytorch and modelopt.onnx). Ignored by torchao backend.",
"enum": [
"minmax",
"max",
"entropy",
"awq_clip",
"awq_lite",
"awq_full",
"rtn_dq"
],
"title": "Calibration algorithm",
"type": "categorical"
},
"backend": {
"default": "torchao",
"description": "The quantization backend to use",
"enum": [
"modelopt.pytorch",
"torchao",
"modelopt.onnx"
],
"title": "Quantization backend",
"type": "categorical"
},
"backend_kwargs": {
"automl_enabled": false,
"description": "Additional keyword arguments to pass to the backend",
"title": "Backend kwargs",
"type": "collection"
},
"device": {
"default": "cuda",
"description": "Device to use for calibration. Accepts 'cuda' (uses default GPU), 'cpu', 'trt' (TensorRT for ONNX backend), or specific GPU device like 'cuda:0', 'cuda:1', etc. If 'cuda' is specified but no GPU is available, the framework will automatically fall back to 'cpu'. Note: 'trt' is only supported by the modelopt.onnx backend for ONNX Runtime with TensorRT execution provider.",
"pattern": "^(cuda|cpu|trt|cuda:[0-9]+)$",
"title": "Calibration device",
"type": "string"
},
"layers": {
"automl_enabled": false,
"default": [],
"description": "List of per-module quantization configurations. Each entry specifies which modules to quantize and their data types. This is the primary way to configure quantization.",
"title": "Layer quantization configs",
"type": "list"
},
"mode": {
"default": "weight_only_ptq",
"description": "The quantization mode to use",
"enum": [
"static_ptq",
"weight_only_ptq"
],
"title": "Quantization mode",
"type": "categorical"
},
"model_path": {
"default": "",
"description": "Path to the model to be quantized. For ONNX backend, path to ONNX file.",
"title": "Model path",
"type": "string"
},
"results_dir": {
"default": "",
"description": "Path to where all the assets generated from a task are stored.",
"title": "Results directory",
"type": "string"
},
"skip_names": {
"automl_enabled": false,
"default": [],
"description": "List of module/layer names or patterns to exclude from quantization",
"title": "Skip names",
"type": "list"
}
},
"required": [
"results_dir",
"model_path"
],
"type": "collection"
},
"results_dir": {
"default": "",
"description": "\n Path to where all the assets generated from a task are stored.\n ",
"title": "Results directory",
"type": "string"
},
"train": {
"automl_disabled_parameters": [
"train.gpu_ids",
"train.cudnn",
"train.checkpointer",
"train.optim"
],
"automl_enabled": false,
"default": {
"checkpoint_interval": 1,
"checkpoint_interval_unit": "epoch",
"checkpointer": {
"auto_insert_metric_name": false,
"enable_topk": false,
"filename": "model_best_{epoch:03d}",
"replace_periodic": false,
"save_top_k": 1
},
"cudnn": {
"benchmark": false,
"deterministic": true
},
"gpu_ids": [
0
],
"num_epochs": 10,
"num_gpus": 1,
"num_nodes": 1,
"optim": {
"grad_clip": {
"max_norm": 25,
"norm_type": "L2"
},
"lr": 5e-05,
"lr_scheduler": {
"min_lr_ratio": 0.001,
"policy": "cosine",
"warmup": "linear",
"warmup_iters": 500,
"warmup_ratio": 0.333333
},
"momentum": 0.9,
"paramwise_cfg": {
"custom_keys": {
"img_backbone": {
"lr_mult": 0.2
}
}
},
"type": "adamw",
"weight_decay": 0.001
},
"precision": "bf16",
"pretrained_model_path": "",
"results_dir": "",
"resume_training_checkpoint_path": "",
"scrub_nan_gradients": false,
"seed": 1234,
"validation_interval": 1
},
"description": "Train config",
"popular": [
"gpu_ids",
"num_gpus",
"num_nodes",
"num_epochs"
],
"properties": {
"checkpoint_interval": {
"default": 1,
"description": "Checkpoint interval in epochs",
"maximum": Infinity,
"minimum": 1,
"title": "Checkpoint interval in epochs",
"type": "int"
},
"checkpoint_interval_unit": {
"default": "epoch",
"description": "The unit of the checkpoint interval.",
"enum": [
"epoch",
"step"
],
"title": "Checkpoint interval unit",
"type": "categorical"
},
"checkpointer": {
"automl_enabled": false,
"default": {
"auto_insert_metric_name": false,
"enable_topk": false,
"filename": "model_best_{epoch:03d}",
"replace_periodic": false,
"save_top_k": 1
},
"properties": {
"auto_insert_metric_name": {
"default": false,
"type": "bool"
},
"dirpath": {
"description": "Directory for best checkpoints. Defaults to results_dir.",
"type": "string"
},
"enable_topk": {
"default": false,
"description": "Save checkpoint(s) ranked by a monitored metric. This is additive to periodic checkpoints unless replace_periodic is enabled.",
"title": "Enable best-checkpoint saving",
"type": "bool"
},
"filename": {
"default": "model_best_{epoch:03d}",
"title": "Best-checkpoint filename pattern",
"type": "string"
},
"mode": {
"description": "Leave unset to use the network's default direction.",
"enum": [
"min",
"max"
],
"title": "Monitor mode",
"type": "categorical"
},
"monitor": {
"description": "Leave unset to use the network's default metric.",
"title": "Monitored metric",
"type": "string"
},
"replace_periodic": {
"default": false,
"description": "When best-checkpoint saving is enabled, replace periodic history with one metric-best checkpoint and its latest symlink.",
"title": "Replace periodic checkpoints",
"type": "bool"
},
"save_top_k": {
"default": 1,
"minimum": 1,
"title": "Number of best checkpoints",
"type": "int"
}
},
"type": "collection"
},
"cudnn": {
"automl_enabled": false,
"default": {
"benchmark": false,
"deterministic": true
},
"properties": {
"benchmark": {
"default": false,
"type": "bool"
},
"deterministic": {
"default": true,
"type": "bool"
}
},
"type": "collection"
},
"gpu_ids": {
"automl_enabled": false,
"default": [
0
],
"description": "\n List of GPU IDs to run the training on. The length of this list\n must be equal to the number of gpus in train.num_gpus.",
"popular": true,
"title": "GPU IDs",
"type": "list"
},
"num_epochs": {
"default": 10,
"description": "Number of epochs to run the training.",
"maximum": Infinity,
"minimum": 1,
"popular": true,
"title": "Number of epochs",
"type": "int"
},
"num_gpus": {
"default": 1,
"description": "The number of GPUs to run the train job.",
"minimum": 1,
"popular": true,
"title": "Number of GPUs",
"type": "int"
},
"num_nodes": {
"default": 1,
"description": "Number of nodes to run the training on. If > 1, then multi-node is enabled.",
"minimum": 1,
"popular": true,
"title": "Number of nodes",
"type": "int"
},
"optim": {
"automl_default_parameters": [
"train.optim.lr"
],
"automl_disabled_parameters": [
"train.optim.paramwise_cfg",
"train.optim.grad_clip",
"train.optim.lr_scheduler"
],
"automl_enabled": false,
"default": {
"grad_clip": {
"max_norm": 25,
"norm_type": "L2"
},
"lr": 5e-05,
"lr_scheduler": {
"min_lr_ratio": 0.001,
"policy": "cosine",
"warmup": "linear",
"warmup_iters": 500,
"warmup_ratio": 0.333333
},
"momentum": 0.9,
"paramwise_cfg": {
"custom_keys": {
"img_backbone": {
"lr_mult": 0.2
}
}
},
"type": "adamw",
"weight_decay": 0.001
},
"description": "Optimizer configuration",
"properties": {
"grad_clip": {
"automl_enabled": false,
"default": {
"max_norm": 25,
"norm_type": "L2"
},
"description": "Gradient clipping configuration",
"title": "Gradient clipping configuration",
"type": "collection"
},
"lr": {
"automl_enabled": true,
"default": 5e-05,
"description": "Learning rate",
"maximum": Infinity,
"minimum": 0.0,
"title": "Learning rate",
"type": "float"
},
"lr_scheduler": {
"automl_enabled": false,
"default": {
"min_lr_ratio": 0.001,
"policy": "cosine",
"warmup": "linear",
"warmup_iters": 500,
"warmup_ratio": 0.333333
},
"description": "Learning rate scheduler configuration",
"title": "Learning rate scheduler configuration",
"type": "collection"
},
"momentum": {
"default": 0.9,
"description": "Momentum for SGD",
"title": "Momentum for SGD",
"type": "float"
},
"paramwise_cfg": {
"automl_enabled": false,
"default": {
"custom_keys": {
"img_backbone": {
"lr_mult": 0.2
}
}
},
"description": "Parameters-wise configuration",
"title": "Parameters-wise configuration",
"type": "collection"
},
"type": {
"default": "adamw",
"description": "Optimizer type",
"enum": [
"adamw",
"adam",
"sgd"
],
"title": "Optimizer type",
"type": "categorical"
},
"weight_decay": {
"default": 0.001,
"description": "Weight decay coefficient",
"title": "Weight decay coefficient",
"type": "float"
}
},
"title": "Optimizer configuration",
"type": "collection"
},
"precision": {
"default": "bf16",
"description": "Precision",
"enum": [
"bf16",
"fp16",
"fp32"
],
"title": "Precision",
"type": "categorical"
},
"pretrained_model_path": {
"default": "",
"description": "Path to pretrained model",
"title": "Path to pretrained model",
"type": "string"
},
"results_dir": {
"default": "",
"description": "\n Path to where all the assets generated from a task are stored.\n ",
"title": "Results directory",
"type": "string"
},
"resume_training_checkpoint_path": {
"default": "",
"description": "Path to the checkpoint to resume training from.",
"title": "Resume checkpoint path",
"type": "string"
},
"scrub_nan_gradients": {
"default": false,
"description": "Opt in to classification-logit clipping, non-finite loss recovery, and NaN-gradient scrubbing after backward; gradient infinities remain visible to mixed-precision overflow detection",
"title": "Scrub NaN gradients",
"type": "bool"
},
"seed": {
"default": 1234,
"description": "The seed for the initializer in PyTorch. If < 0, disable fixed seed.",
"maximum": Infinity,
"minimum": -1,
"title": "Seed for randomization",
"type": "int"
},
"validation_interval": {
"default": 1,
"description": "Validation interval in epochs",
"maximum": Infinity,
"minimum": 1,
"title": "Validation interval in epochs",
"type": "int"
}
},
"title": "Train config",
"type": "collection"
},
"visualize": {
"automl_enabled": false,
"default": {
"n_images_col": 6,
"show": false,
"vis_dir": "./vis",
"vis_score_threshold": 0.25,
"viz_down_sample": 3
},
"description": "Visualize config",
"properties": {
"n_images_col": {
"default": 6,
"description": "Number of images per column",
"maximum": Infinity,
"minimum": 1,
"title": "Number of images per column",
"type": "int"
},
"show": {
"default": false,
"description": "Show visualization",
"title": "Show visualization",
"type": "bool"
},
"vis_dir": {
"default": "./vis",
"description": "Visualization directory",
"title": "Visualization directory",
"type": "string"
},
"vis_score_threshold": {
"default": 0.25,
"description": "Visualization score threshold",
"maximum": 1.0,
"minimum": 0.0,
"title": "Visualization score threshold",
"type": "float"
},
"viz_down_sample": {
"default": 3,
"description": "Visualization down sample",
"maximum": Infinity,
"minimum": 1,
"title": "Visualization down sample",
"type": "int"
}
},
"title": "Visualize config",
"type": "collection"
},
"wandb": {
"automl_disabled_parameters": [
"wandb.tags"
],
"automl_enabled": false,
"default": {
"enable": true,
"entity": "",
"group": "",
"name": "TAO Toolkit Training",
"project": "TAO Toolkit",
"reinit": false,
"run_id": "",
"save_code": false,
"sync_tensorboard": false,
"tags": [
"tao-toolkit"
]
},
"properties": {
"enable": {
"default": true,
"type": "bool"
},
"entity": {
"default": "",
"type": "string"
},
"group": {
"default": "",
"type": "string"
},
"name": {
"default": "TAO Toolkit Training",
"type": "string"
},
"project": {
"default": "TAO Toolkit",
"type": "string"
},
"reinit": {
"default": false,
"type": "bool"
},
"run_id": {
"default": "",
"type": "string"
},
"save_code": {
"default": false,
"type": "bool"
},
"sync_tensorboard": {
"default": false,
"type": "bool"
},
"tags": {
"automl_enabled": false,
"default": [
"tao-toolkit"
],
"type": "list"
}
},
"type": "collection"
}
},
"required": [
"quantize"
],
"type": "object",
"x_tao_schema": {
"action": "export",
"core_module": "sparse4d",
"model": "sparse4d",
"network_arch": "sparse4d",
"schema_action": "export",
"schema_version": 1,
"source": "tao-core dataclass config"
}
}
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Declares allowed tools, Needs Python, Needs Node.js
The author says: Requires a compatible TAO runtime; Docker execution requires docker and nvidia-container-toolkit.
Security
What our scanner noticed· Medium risk
- Large base64-encoded blobs
- Scripts that your agent may run
This skill includes scripts that your agent may run. Read them before you use it.
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Created by
- NVIDIA Corporation
- Publisher
- NVIDIA (Official)
- Source
- NVIDIA/skills · skills/tao-train-sparse4d · commit fce3cca
- Licence
- Apache License 2.0 (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE-APACHE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
AutoGineer is not affiliated with or endorsed by NVIDIA Corporation or NVIDIA.
Related skills
Migrate To Shoehorn
Matt Pocock· Community
Automate the migration of TypeScript test assertions from unsafe as-casting to the type-safe shoehorn library.
Software developmentClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository317.5k
Tdd
Matt Pocock· Community
A reference guide for implementing the red-green-refactor loop in test-driven development.
Software developmentClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository317.5k
Test Driven Development
Jesse Vincent (obra)· Community
A rigorous framework for Test-Driven Development, mandating that failing tests precede all production code.
Software developmentClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository314k
Webapp Testing
AnthropicOfficial
A toolkit for automating local web application testing using Playwright and server management helpers.
Software developmentClaude CodeClaude appsOpenAI Codex+14 moreApache-2.0Low riskHas scriptsGitHub stars of the source repository188.6k
Test Driven Development
Addy Osmani· Community
A systematic approach to software development using the red-green-refactor cycle to ensure code reliability and maintainability.
Software developmentClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository113.9k
Browser Testing With Devtools
Addy Osmani· Community
Integrate Chrome DevTools to enable live browser inspection, debugging, and automated UI verification for your AI agent.
Software developmentClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository113.9k
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download tao-train-sparse4d.zip