Tao Run Automl

TAO AutoML Intent And Algorithms

User intent parsing, train-vs-AutoML policy, algorithm selection, and the algorithm decision tree.

Load this file only when the compact SKILL.md points here for the current task. If this reference conflicts with SKILL.md, skill_info.yaml, schemas, or platform/model skills, the compact/current source wins.

Contents

• network_arch is NOT a runner.run() arg anymore; resolve the user's model

• request to a packaged skill_dir, then read network_arch from that skill's

• metadata.

• Step 2: Select Algorithm

• Classical Algorithms

• LLM/Agentic Algorithms (NEW)

• Quick Reference: Algorithm Decision Tree

Step 1: Parse User Intent

Default to a quick-start run unless the user explicitly asks to customize AutoML or agrees to a customization offer. Do not present algorithm, budget, or search-space choices as required inputs for a normal "run AutoML" request.

Any workflow/application that reaches an AutoML-capable model action must consult the selected model's automl_enabled metadata. If it is true, use this AutoML workflow for train-like optimization unless the run/workflow setting has automl_policy: off or the user explicitly asks for a plain single action run. Treat automl_policy: on as the default enabled state for train and distill. For single-shot compression actions such as prune and quantize, use AutoML when the user asks to optimize/tune the action or when the workflow profile explicitly sets automl_policy: on.

Extract these fields for a default run:

Use these quick-start AutoML defaults without asking:

If any required field is missing, ask the user. Do NOT guess dataset paths, skill bank paths, credentials, or hardware that the model skill marks as required.

When asking for missing AutoML launch inputs, use a first-time-user friendly prompt. Do not say only "train dataset root" / "eval dataset root", and do not say "attached monitoring every 5 minutes" without explaining it. Include:

• platform choices;

• root-mode dataset examples for the selected platform;

• direct spec-parameter mode as an equal option;

• model-required spec keys from the model skill's Per-Action Dataset Requirements table;

• resolved train container image and the option to override it with image=<override>;

• monitoring meaning and cadence choices.

Before generating an AutoML script, verify platform access and dataset visibility using the shared launch preflight. For SLURM, that means passwordless SSH to at least one login host and remote test -e checks for each required annotation/media path. If preflight fails, stop with remediation steps instead of creating a runner that will immediately fail.

If the selected model skill's Per-Action Dataset Requirements or Typical Spec Overrides show train/evaluate/inference inputs that come from a prior dataset_convert action, run that conversion before calling AutoMLRunner.run. Use the dataset_convert action's own container_image when it overrides the model default, persist the conversion output under the current run's results root, verify every required converted artifact named by the model skill exists, and pass those current-run converted paths in spec_overrides. Do not reuse stale conversion paths from another AutoML algorithm/run folder.

Also verify container image confirmation using the shared launch preflight. AutoML launches real action jobs for each recommendation, so the confirmed action image must be passed into AutoMLRunner.run(..., image=chosen_image, ...) or into the SDK adapter's create_job(..., image=chosen_image, ...). Do not rely on an implicit default after the user has chosen a platform and dataset.

Also run any model-specific annotation content checks documented by the model skill. Missing required annotation fields are a preflight failure, not an AutoML recommendation failure.

Before submitting any recommendation job, generate the initial recommendation batch in a review-only step and show the exact configs to the user together with metric, direction, expected runtime, and effective-batch checks. Do not replace concrete configs with only search bounds.

After all platform, image, credential, data, and model preflight checks pass, submit the model's evaluate action once against the selected validation/eval data before submitting AutoML recommendations. This is the automatic baseline eval job for the run. Show its job id, result path, and metric value in the launch review, then ask for confirmation before spending training budget on recommendations. If the eval job cannot run because the model has no packaged evaluate action, validation data is missing, or the eval job fails, stop and surface the blocker. Proceed without this baseline only when the user explicitly accepts a proxy-metric run with no impact baseline.

If the workflow starts from scratch and has no compatible checkpoint to evaluate, run a minimal default-configuration training job outside the AutoML recommendation budget, resolve its exact epoch/step checkpoint, and evaluate that checkpoint for the baseline. Then use the same evaluator task metric via eval_fn for every recommendation and final_eval_fn for the winner. An empty checkpoint value or container directory is not a valid baseline model.

Customization gate: After the required quick-start fields are resolved, you may briefly offer customization. If the user declines or does not ask for it, proceed with the defaults above. If the user chooses customization, then present the additional options below.

Customization-only fields:

MANDATORY: Read the generated dataclass schema before configuring AutoML.

For the selected model/action, read:

• ${TAO_SKILL_BANK_PATH:-~/tao-skill-bank}/skills/models/<model_skill>/schemas/<action>.schema.json

• ${TAO_SKILL_BANK_PATH:-~/tao-skill-bank}/skills/models/<model_skill>/schemas/manifest.json

AutoML is enabled by the model skill, but it can run only when schemas/<action>.schema.json is packaged with the plugin and valid for the selected model/action. Do not fall back to hand-written model notes, old runner scripts, or a local ~/tao-core checkout for AutoML parameter metadata. If the action schema is missing, stop and report that AutoML is enabled for that model but not runnable for that action until the schema is generated and shipped in the skill bank.

Use the schema JSON as the source of truth for automl_default_parameters, automl_disabled_parameters, per-parameter defaults, ranges, enums, option_weights, math_cond, depends_on, parent_param, and popular.

When automl_hyperparameters=None, the runner automatically discovers all params marked automl_enabled=True in the network's generated schema. Each network has its own set; never hardcode them in this workflow skill.

Quick-start runner shape:

# network_arch is NOT a runner.run() arg anymore. It is read from the model
# metadata at skill_dir, which was passed to AutoMLRunner(...) at construction.
result = runner.run(
    train_dataset_uri=TRAIN_DATASET_URI,
    automl_settings={
        "algorithm": "bayesian",
        "metric": metric,
        "automl_max_recommendations": 10,
        "session_id": session_id,
    },
    automl_hyperparameters=None,  # use schema params marked automl_enabled=true
    custom_param_ranges=None,     # use schema ranges/options/defaults
    spec_overrides={...},         # from model skill + dataset requirements
    workspace_path=f"./automl/{TIMESTAMP}",
)

Customization runner additions:

result = runner.run(
    ...,
    automl_hyperparameters=selected_param_names,
    custom_param_ranges={
        "<param_name>": {"valid_min": min_value, "valid_max": max_value},
        "<categorical_param>": {
            "valid_options": ["option_a", "option_b"],
            "option_weights": [0.7, 0.3],
        },
    },
)

MANDATORY LLM configuration for LLM-based algorithms (llm, hybrid, autoresearch):

When the user requests or customizes into an LLM-powered algorithm, resolve ALL THREE of the following before generating the script. Do not ask for these on default bayesian quick-start runs.

1. llm_endpoint — user input -> AUTOML_LLM_ENDPOINT -> https://inference-api.nvidia.com

2. llm_model — user input -> AUTOML_LLM_MODEL -> gcp/google/gemini-3.1-pro-preview

3. llm_api_key — AUTOML_LLM_API_KEY -> NVIDIA_API_KEY -> declared local secret file when allowed. Do not print the value.

If the runner does not receive valid LLM settings, the LLM brain may silently fall back to random sampling — wasting GPU budget on random configs instead of intelligent ones. There is no error message; the only clue is "LLM call failed... Falling back to random" in the logs.

MANDATORY: Read the model skill before generating the script.

AutoML runs the selected TAO action. Before generating any AutoML script, read <bank-root>/skills/models/<model_skill>/SKILL.md (where <bank-root> is wherever the agent loaded this SKILL.md from). The model skill contains all model-specific knowledge:

• Training Requirements — dataset type, formats, monitoring metric, required dataset URIs to prompt for, required user prompts (data format, num_classes, etc.), and mandatory spec_overrides. Prompt the user for every required field. Apply mandatory spec_overrides exactly.

• Per-Action Dataset Requirements — table mapping each action to its spec keys, data source, expected files, and whether the field is a list. Use this table to construct the correct data source spec_overrides for the requested action. If the model's Typical Spec Overrides mark data sources as "mandatory", construct them from this table and the user's dataset URIs.

• Typical Spec Overrides — per-action override suggestions (train, evaluate, export, inference, etc.) extracted from SDK notebooks. Use these as the starting point for spec_overrides and suggest them to the user. When overrides are marked "mandatory data sources", they MUST be included — the runner cannot auto-resolve them. Merge with any other mandatory overrides from Training Requirements.

• AutoML / HPO Notes — metric, direction, model-specific constraints, and any guidance that narrows or overrides the generated schema. Hyperparameter names/ranges/defaults come first from schemas/<action>.schema.json.

• Error Patterns — common training failure modes that apply to AutoML recs too.

Do NOT hardcode model-specific knowledge in the AutoML script without reading the model skill first. Each network has different requirements.

MANDATORY: No model-specific constants in this AutoML skill.

The AutoML skill must not define model-specific hyperparameter names, ranges, defaults, metric names, dataset layouts, archive names, class-count rules, spec override keys, container images, checkpoint quirks, or custom metric regexes. Hyperparameter metadata belongs in <bank-root>/skills/models/<model_skill>/schemas/<action>.schema.json; model-specific runtime guidance belongs in the model skill's Training Requirements, Typical Spec Overrides, AutoML / HPO Notes, and Error Patterns sections. This skill may describe how to read and apply those sources, but not the concrete per-model values.

MANDATORY: Timestamped workspace folders.

ALWAYS generate workspace_path with a timestamp suffix. Running the same script twice without a timestamp overwrites the previous experiment. Pattern:

from datetime import datetime
TIMESTAMP = datetime.now().strftime("%Y%m%d_%H%M%S")
workspace_path = f"./experiment_name/{TIMESTAMP}"

Do NOT use a flat path like workspace_path="./my_experiment". The user should never have to manually delete old workspace folders.

MANDATORY: Fresh runner per new AutoML request, after preflight passes.

Every new user request to run AutoML MUST create a new runner script and launch a new AutoML job, even if an older runner script for the same network/algorithm already exists. This freshness rule starts only after platform and dataset preflight passes. Existing runner files and logs may be read only as references for dataset URIs, credentials patterns, and proven fixes; do not reuse them as the execution target for a new request.

Use a unique timestamp in the new runner filename, log filename, PID filename, SDK state_file, and workspace_path. Derive path components from the requested network_arch and algorithm; do not hardcode any model or algorithm name unless it is the actual requested value.

import re

def slug(value):
    return re.sub(r"[^A-Za-z0-9_.-]+", "_", str(value)).strip("_").lower()

TIMESTAMP = datetime.now().strftime("%Y%m%d_%H%M%S")
RUN_NAME = f"{slug(network_arch)}_{slug(algorithm)}"
runner_path = f"automl_runs/run_{RUN_NAME}_{TIMESTAMP}.py"
log_path = f"automl_runs/{RUN_NAME}_{TIMESTAMP}.log"
pid_path = f"automl_runs/{RUN_NAME}_{TIMESTAMP}.pid"
state_file = f"tao_session_state_{RUN_NAME}_{TIMESTAMP}.json"
workspace_path = f"./automl_runs/{RUN_NAME}/{TIMESTAMP}"

Only resume an existing runner/workspace when the user explicitly asks to resume, continue, recover, or inspect an existing experiment. If the user says "run automl" or asks for a new AutoML run, treat it as a fresh job.

Best-practice on metric choice:

• Training loss is cheap, but can overfit on small fine-tuning datasets. Prefer the model skill's recommended validation or task metric when available.

• If the model skill recommends a validation proxy, also apply the model skill's required validation-related spec_overrides so the metric is actually emitted.

• A real task metric via eval_fn is often the most honest but adds per-rec cost. Use it when the model skill says log-based metrics are insufficient or the user explicitly wants downstream evaluation.

• For AutoML runs with a runnable evaluate action and validation/eval data, run the automatic baseline eval job after preflight and before recommendations, then include that baseline metric in the final comparison.

Checkpoint / resume behavior:

• Resume-based algorithms (hyperband, asha, bohb, dehb, pbt, hyperband_es) must resume from the checkpoint for the stopped rung/generation epoch or step, not a generic *_latest.* file.

• The runner records the intended resume_from_epoch / resume_from_step on promoted recommendations and resolves the model-specific checkpoint path through the SDK checkpoint resolver. Do not patch runner scripts to guess names like model_latest.pth.

• If the intended checkpoint is epoch 1, prefer artifacts such as epoch_1, epoch_001, model_epoch_001.pth, model_epoch_000_step_*.pth when the trainer writes zero-indexed epoch files, or a checkpoint directory like checkpoints/epoch_1.

• Use a latest checkpoint only when the requested action explicitly has no epoch/step target. If no epoch/step-specific resume artifact exists, report the model as blocked and fix the model/skill checkpoint metadata rather than silently resuming from latest.

Step 2: Select Algorithm

Classical Algorithms

These require no external services — they use statistical/mathematical methods to pick hyperparameters.

LLM/Agentic Algorithms (NEW)

These use a large language model to reason about hyperparameter choices. They require an LLM endpoint (NVIDIA NIM, OpenAI, vLLM, Ollama, etc.) and the openai Python package.

Default to bayesian unless the user specifically asks for something else, has a large GPU budget, or needs early-stopping on cheap intermediate metrics (ASHA / hyperband).

Use llm / hybrid / autoresearch when the user wants LLM-guided search, has an API key for NVIDIA NIM or OpenAI, and wants richer reasoning about why certain hyperparameters are chosen.

Compression action policy (distill, prune, quantize):

• distill is train-like when it runs for epochs and emits checkpoints. Use bayesian by default; use hyperband, asha, bohb, dehb, hyperband_es, or pbt only when the action schema contains epoch/checkpoint budget fields and the model skill says short-rung metrics are meaningful.

• prune is often a single-shot compression transform followed by optional retrain/evaluate. Default to bayesian, bfbo, llm, hybrid, or autoresearch over prune ratios, thresholds, skip/layer choices, and retrain knobs exposed by the action schema. Use eval_fn to run evaluate/retrain when the prune action itself does not emit the final task metric.

• quantize is usually a single-shot calibration/export transform. Default to bayesian, bfbo, llm, hybrid, or autoresearch over calibration batch count, backend/layer/skip choices, precision policies, and calibration data knobs exposed by the schema. Treat latency/size/accuracy as a multi-objective metric policy in eval_fn until a dedicated Pareto optimizer is added.

Caveat on ASHA with expensive checkpoints: ASHA's whole point is running many configs cheaply for early rungs, then promoting survivors. If the model skill warns that checkpoints, validation, or startup cost dominate short trials, prefer the model skill's recommended algorithm instead of assuming ASHA will be cheaper.

Quick Reference: Algorithm Decision Tree

Is your budget tiny (≤10 recs)?
  YES → bayesian
  NO  ↓

Do you have an LLM API key and want AI-guided search?
  YES → Do you want full autonomy? → autoresearch
        Just LLM proposals?        → llm
        LLM orchestrating phases?  → hybrid
  NO  ↓

Do you need parallel execution?
  YES → asha (or bohb for smarter sampling)
  NO  ↓

Is your search space large (10+ parameters)?
  YES → hyperband or dehb
  NO  ↓

Do hyperparameters need to change during training (schedules)?
  YES → pbt
  NO  → bayesian (safe default)