Finetuning

Dataset Creation Workflow

Three paths to training data (these combine well: curate seeds → augment → generate at scale):

If you already have data, skip to validation: python scripts/validate/validate_sft.py your_data.jsonl

Approach 1: Manual Curation

Write examples by hand, collect from production logs, or adapt existing datasets.

When to use:

• You have real-world examples (production logs, support tickets, labeled data)

• Your task requires domain expertise an LLM can't reliably generate

• You need a gold-standard evaluation set (always curate manually)

Tips:

• Start with 10-20 examples to establish quality standards and format consistency

• These seed examples also serve as the foundation of your evaluation test set

• For RFT, you only need prompts + expected answers — no model responses needed

Approach 2: LLM Augmentation

Expand a small curated dataset through rephrasing — generating diverse variations while keeping the same expected answer. Especially useful for RFT.

When to use:

• Well-defined task with clear correct answers

• You can write quality examples but need more volume

• Diversity of phrasing matters more than diversity of scenarios

Workflow:

1. Write base examples with correct expected answers

2. For each, use an LLM to generate rephrasings varying tone, detail, and wording

3. Each rephrasing gets the same expected answer — only the phrasing changes

4. Validate the augmented dataset

Rephrasing prompt:

Generate N different phrasings of this request. Each should:
- Use different wording, tone, or level of detail
- Include the same key identifiers (order IDs, item names)
- Vary between formal, casual, frustrated, brief, and detailed styles
Return a JSON array of N strings.

Original: [your example]

A cheap model (gpt-4.1-mini) works well — no new ground truth needed, just phrasing diversity.

Approach 3: Synthetic Generation

Generate training data from scratch using LLM prompts.

1. Define topic/scenario categories for diversity

2. Generate prompts from an LLM

3. Generate responses (or preferred/non-preferred pairs for DPO)

4. Grade quality with an LLM judge

5. Filter to a quality threshold

6. Split into train/validation/test sets

7. Write JSONL in the correct format (see references/dataset-formats.md)

Quality Checklist

Before training, verify:

• [ ] No duplicates: Exact or near-duplicate examples waste budget

• [ ] Balanced distribution: Topics, difficulty, output lengths well-distributed

• [ ] Consistent formatting: All examples follow the same structure

• [ ] Correct outputs: Spot-check 20 random examples manually

• [ ] Reasonable lengths: No extremely short or extremely long outputs

• [ ] Clean text: No encoding errors, garbled text, or template artifacts

Dataset Size vs. Quality

From experiments:

• 335 high-quality examples (carefully curated) → best combined eval score (9.15)

• 1,576 examples (broader but noisier) → higher correctness but lower conciseness (8.53)

Takeaway: A small, pristine dataset usually beats a large, noisy one. Quality filter aggressively.