Finetuning

MicrosoftOfficialCreated by Microsoft

Skill
Tool integration
Data and AI
MIT
Low risk
Has scripts

From the author

Fine-tune models on Microsoft Foundry using SFT (supervised), DPO (preference), or RFT (reinforcement with graders). Covers dataset preparation, training job submission, deployment, and evaluation. USE FOR: fine-tune, SFT, DPO, RFT, training data, grader, distillation, fine-tuned model, training job, large file upload, calibrate grader, deploy fine-tuned model, evaluate fine-tuned model. DO NOT USE FOR: general model deployment without fine-tuning (use deploy-model), agent creation (use agents), prompt optimization without training (use prompt-optimizer).

— the description by Microsoft

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides the necessary procedures and tools for fine-tuning AI models on Microsoft Foundry. It supports three primary training methodologies: Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Reinforcement Fine-Tuning (RFT). The skill encompasses the entire lifecycle, including dataset preparation, training job submission, monitoring, model evaluation, and deployment. It also includes specific guidance for agentic RFT, where models are trained to utilize external tools, and provides scripts for data validation, grader calibration, and resource management.

What it does

Supports SFT, DPO, and RFT training methodologies.
Provides data validation scripts for SFT, DPO, and RFT formats.
Facilitates training job submission and monitoring on Microsoft Foundry.
Enables calibration of RFT graders and pass thresholds.
Offers tools for model evaluation and deployment via ARM REST API.
Includes utilities for synthetic data generation and dataset quality scoring.

When to use it

When you need to adapt a base model to specific tasks or domains.
When preparing or validating training datasets for SFT, DPO, or RFT.
When training models for agentic scenarios involving tool calling.
When you need to evaluate and deploy a fine-tuned model.
When troubleshooting training jobs or diagnosing poor model performance.

Advantages

Standardized workflows for different training types.
Includes built-in validation to prevent common data formatting errors.
Supports complex agentic RFT training with tool integration.
Provides robust scripts for monitoring and evaluating training progress.
Offers clear guidance on grader design and hyperparameter tuning.

Usage examples

“How do I validate my SFT dataset before training?”: The agent will instruct you to use the python scripts/validate/validate_sft.py script on your data.jsonl file to ensure it meets the required format.
“Submit an SFT training job using gpt-4.1-mini.”: The agent will provide the command using scripts/submit_training.py with the appropriate flags for model, training file, validation file, and type.
“How can I calibrate my RFT grader?”: The agent will guide you to use scripts/calibrate_grader.py to find the optimal pass_threshold, aiming for a 25-50% failure rate on the base model.
“Deploy my fine-tuned model after training.”: The agent will provide the command to use scripts/deploy_model.py, specifying the model-id and a deployment name.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/microsoft/finetuning/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip finetuning.zip -d .agents/skills

Documentation by Microsoft

Fine-Tuning on Microsoft Foundry

Fine-tune models using SFT (supervised), DPO (preference), or RFT (reinforcement with graders). Covers dataset prep, training, deployment, and evaluation.

When to Use

Use this sub-skill when the user asks about:

Fine-tuning a model (SFT, DPO, or RFT)
Preparing, validating, or formatting training data
Submitting, monitoring, or diagnosing training jobs
Calibrating graders or pass thresholds for RFT
Deploying or evaluating a fine-tuned model
Choosing between training types (SFT vs DPO vs RFT)
Distillation, synthetic data generation, or dataset quality scoring
Large file uploads for training data
Cleaning up fine-tuning resources (files, deployments)

Do NOT use for: General model deployment without fine-tuning (use deploy-model), agent creation (use agents), prompt optimization without training (use prompt-optimizer).

Workflows

Stage	Guide
Quick start	workflows/quickstart.md
Full pipeline	workflows/full-pipeline.md
Create data	workflows/dataset-creation.md
Iterate	workflows/iterative-training.md
Diagnose	workflows/diagnose-poor-results.md

References

Topic	File
SFT vs DPO vs RFT	references/training-types.md
Hyperparameters	references/hyperparameters.md
Data formats	references/dataset-formats.md
Grader design (RFT)	references/grader-design.md
Reward hacking	references/reward-hacking.md
Agentic RFT (tools)	references/agentic-rft.md
Deployment	references/deployment.md
Training curves	references/training-curves.md
Evaluation	references/evaluation.md
Vision fine-tuning	references/vision-fine-tuning.md
Large file uploads	references/large-file-uploads.md
Platform gotchas	references/platform-gotchas.md

Scripts

Script	Purpose
`scripts/submit_training.py`	Submit SFT/DPO/RFT jobs
`scripts/monitor_training.py`	Poll job until completion
`scripts/calibrate_grader.py`	Find optimal RFT pass_threshold
`scripts/check_training.py`	Analyze curves, list checkpoints
`scripts/deploy_model.py`	Deploy via ARM REST API
`scripts/evaluate_model.py`	LLM judge evaluation
`scripts/convert_dataset.py`	Convert between SFT/DPO/RFT formats
`scripts/generate_distillation_data.py`	Generate synthetic training data
`scripts/score_dataset.py`	Quality scoring on training data
`scripts/cleanup.py`	Delete old files and deployments
`scripts/validate/`	Data validators (SFT, DPO, RFT) + stats

Rules

Always baseline first — evaluate the base model before fine-tuning
Validate data before submitting — run scripts/validate/validate_sft.py
Calibrate RFT graders — target 25-50% failure rate on the base model
Evaluate checkpoints — don't blindly deploy the final one
Measure token cost alongside accuracy when comparing models

Quick Reference

Task	Command
Validate SFT data	`python scripts/validate/validate_sft.py data.jsonl`
Submit SFT job	`python scripts/submit_training.py --model gpt-4.1-mini --training-file train.jsonl --validation-file val.jsonl --type sft`
Monitor job	`python scripts/monitor_training.py --job-id ftjob-xxx`
Analyze curves	`python scripts/check_training.py --job-id ftjob-xxx`
Deploy model	`python scripts/deploy_model.py --model-id ft:gpt-4.1-mini:... --name my-eval`
Evaluate model	`python scripts/evaluate_model.py --deployment-name my-eval --test-file test.jsonl`

Error Handling

Error	Cause	Fix
"API version not supported"	Older `openai` SDK on `/v1/` endpoint	Upgrade to `openai>=1.0`
"does not support fine-tuning with Standard TrainingType"	OSS model needs `globalStandard`	Use `--use-rest` flag or script auto-falls back
Job stuck in post-training eval	Under-provisioned tool endpoint (RFT)	Scale to S2+, enable Always On
"DeploymentNotReady" after ARM succeeds	ARM/data-plane race condition	Delete and recreate deployment, wait 5 min
Content safety block at deployment	PII-dense training data	Remove problematic document types

Files

ATTRIBUTION.mdAdded by AutoGineer998 B
LICENSELicence file added by AutoGineer1.1 KiB
SKILL.md5.4 KiB
references/agentic-rft.md3.2 KiB
references/dataset-formats.md3.7 KiB
references/deployment.md3.4 KiB
references/evaluation.md5.8 KiB
references/grader-design.md3.4 KiB
references/hyperparameters.md3.2 KiB
references/large-file-uploads.md927 B
references/platform-gotchas.md2.0 KiB
references/reward-hacking.md2.9 KiB
references/training-curves.md4.1 KiB
references/training-types.md3.1 KiB
references/vision-fine-tuning.md4.1 KiB
scripts/calibrate_grader.pyScript9.3 KiB
scripts/check_training.pyScript7.8 KiB
scripts/cleanup.pyScript9.5 KiB
scripts/common.pyScript7.6 KiB
scripts/convert_dataset.pyScript10.7 KiB
scripts/deploy_model.pyScript9.4 KiB
scripts/evaluate_model.pyScript11.4 KiB
scripts/generate_distillation_data.pyScript10.1 KiB
scripts/monitor_training.pyScript5.7 KiB
scripts/score_dataset.pyScript7.7 KiB
scripts/submit_training.pyScript10.8 KiB
scripts/validate/__init__.pyScript305 B
scripts/validate/data_stats.pyScript6.2 KiB
scripts/validate/validate_dpo.pyScript3.7 KiB
scripts/validate/validate_rft.pyScript8.6 KiB
scripts/validate/validate_sft.pyScript4.5 KiB
workflows/dataset-creation.md3.1 KiB
workflows/diagnose-poor-results.md2.3 KiB
workflows/full-pipeline.md3.1 KiB
workflows/iterative-training.md3.1 KiB
workflows/quickstart.md4.5 KiB

references/reward-hacking.md

Reward Hacking Prevention in RFT

What Is Reward Hacking?

The model optimizes for the grader's scoring function rather than the actual task. The training grader becomes a proxy reward that diverges from true quality — the model games the proxy instead of improving.

Core rule: Your training grader MUST produce the same ranking as your evaluation methodology.

If you evaluate with…	Then train with…	NOT with…
LLM judge (semantic)	LLM judge	AST / regex / structural matching
Exact match	Exact match	Fuzzy or partial matching
Unit tests	Unit tests	Static analysis alone

Misaligned graders are the #1 cause of reward hacking.

Train-Val Gap Thresholds

Train-Val Gap	Status	Action
≤ 0.05	✅ Healthy	Continue training
0.05–0.10	⚠️ Warning	Monitor closely, check outputs qualitatively
> 0.10	🛑 Stop	Stop training — reward hacking is likely

Pre-Training Checklist

Baseline the grader: Run training grader on base model outputs. Record scores as your floor.
Cross-validate graders: If training grader ≠ eval grader, generate 50 outputs, score with both, compute Spearman ρ. Proceed only if ρ ≥ 0.8. If ρ < 0.6, fix alignment first.
Test hackability: Generate 5 intentionally bad outputs that might score well. If grader scores any > 5/10, redesign it.
Set gap threshold: Monitor train-val gap every eval_interval. Stop if > 0.10.

Grader Iteration Loop

When reward hacking is detected:

1. STOP the training run
        ↓
2. COLLECT "hacked" outputs (high train score, low eval score)
        ↓
3. ANALYZE what pattern the model exploited
   (structural mimicry? verbosity? keyword stuffing?)
        ↓
4. UPDATE the grader to penalize that pattern
        ↓
5. RE-BASELINE the updated grader on base model outputs
        ↓
6. RESTART training with the improved grader

Red Flags Checklist

Investigate immediately if any are true:

[ ] Train-val gap > 0.10
[ ] Training reward increasing but eval quality stable or declining
[ ] Model outputs are longer/more verbose than base model
[ ] Outputs structurally match references but are semantically wrong
[ ] Different LLM judges disagree on quality
[ ] Conciseness/style scores dropping while correctness climbs
[ ] Model produces "template" responses

Key Principles

Principle	Action
Align graders	Training grader must rank outputs same as eval
Cross-validate first	Spearman ρ ≥ 0.8 between training and eval graders
Monitor train-val gap	≤ 0.05 healthy, > 0.10 stop
Test hackability	Bad outputs should score < 5/10
Prefer SFT when possible	Use RFT only for verifiable-answer tasks
Iterate graders, not models	Fix grader before restarting training

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Low risk

Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

Microsoft

Publisher

Microsoft (Official)

Source

microsoft/skills · .github/plugins/azure-skills/skills/microsoft-foundry/finetuning · commit 0696600

Licence

MIT License (redistribution allowed)

LICENSE (added by AutoGineer from .github/plugins/azure-skills/LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

Also distributed in

microsoft/azure-skills

AutoGineer is not affiliated with or endorsed by Microsoft.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download finetuning.zip