Experiment Design

MicrosoftOfficialPublished by Microsoft

Skill
Guidelines
Agent tooling
CC-BY-4.0
No issues found

From the author

Experiment design reference for problem-class framing, Minimum Viable Experiment coaching, hypothesis formation, vetting and red flags, and experiment readiness. Use when translating a stated business outcome into candidate data-science problem classes, or when framing, vetting, scoping, or evaluating an experiment of any kind, including data feasibility, architecture, LLM, performance, use-case, UX, prototyping, and hardware experiments.

— the description by Microsoft

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The experiment-design skill provides a structured framework for end-to-end experiment management. It helps teams translate business outcomes into specific problem classes, identify and prioritize risks, and design testable, falsifiable hypotheses. By distinguishing between problem-class framing and experiment execution, it ensures that teams focus on resolving critical unknowns rather than building premature solutions. The pack covers various experiment types, including data feasibility, architecture, LLM, and UX, while emphasizing reproducibility and decision-ready outcomes.

What it does

Translates business outcomes into candidate data-science problem classes.
Provides criteria for vetting hypotheses and identifying red flags.
Facilitates risk-based prioritization of experiments.
Guides the design of Minimum Viable Experiments (MVEs).
Ensures experiment results are reproducible and comparable.
Separates problem-class framing from experiment design.

When to use it

When you need to translate a business goal into a technical problem class.
When you have competing unknowns and need to prioritize which to test first.
When you need to validate assumptions before committing to full-scale production.
When you need to design a test that is falsifiable and decision-ready.
When you need to distinguish between a genuine experiment and a demo.

Advantages

Reduces project risk by validating assumptions early.
Prevents wasted effort on demos disguised as experiments.
Provides a clear, repeatable process for experiment design.
Ensures experiments are tied to actionable business decisions.
Promotes knowledge transfer and independent ownership in collaborative teams.

Usage examples

“I have a business goal to reduce customer churn. How do I frame this as a data science problem?”: The agent will use the problem-framing reference to identify candidate problem classes, such as classification or regression, while documenting the reasoning and any gaps without selecting a specific model.
“We are unsure if our current architecture can handle the expected load. How should we test this?”: The agent will guide you through forming a falsifiable hypothesis, vetting it against MVE criteria, and designing an architectural feasibility experiment to resolve the uncertainty.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/microsoft/experiment-design/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip experiment-design.zip -d .agents/skills

Documentation by Microsoft

Experiment Design Reference Pack

Goal

Support experiment work end to end: turning unknowns into testable hypotheses, screening out work that is not a real experiment, and scoping it so the result is comparable and decision-ready.

Support the step that precedes it as well: translating a stated business outcome into candidate data-science problem classes with the reasoning that produced them, so a practitioner knows what kind of problem is on the table before deciding what to test.

The two concerns stay distinct. Problem-class framing exposes candidates and never selects one. Experiment work assumes a candidate direction already exists and concludes by selecting an experiment with the team.

This pack is general purpose. It applies to data feasibility, architecture, LLM, performance, use-case, UX, prototyping, and hardware experiments, not to data science alone.

Inputs

The problem statement, customer context, and business driver
The stated business outcome, when the active concern is problem-class framing
Known unknowns, assumptions, and risks
The decision the experiment is meant to unblock
Prior experiment results, when a sequence of experiments is in flight

Reference index

Read only the reference that matches the active concern.

Reference	Read this when
problem-framing.md	Translating a stated business outcome into candidate data-science problem classes, applying per-paradigm entry tests, ordering discriminating questions, or recording assignable gaps
mve-coaching.md	Framing an MVE, forming or sharpening hypotheses, applying vetting criteria and red flags, designing the experiment, evaluating results, or producing session and backlog-bridge artifacts
experiment-readiness.md	Deciding which experiment to run at all: turning a risk landscape into candidates, prioritizing among competing unknowns, comparing options with evidence, or re-prioritizing mid-flight
rpi-research-preparation.md	Preparing an experiment with bounded prior-art or current-constraint Research while keeping validation and experiment verdicts with their current owners
provenance.md	Confirming what is upstream guidance, what is HVE Core derivation or repository convention, and where upstream is silent

A confirmed problem-class framing request reads problem-framing.md only. Experiment requests read mve-coaching.md or experiment-readiness.md by concern.

Success criteria

Problem-class framing returns the surviving candidate problem classes with the entry-test reasoning that kept or discarded each one, and never returns a selection, a ranking, or a confidence score.
Every framing claim is visibly either a cited entry test or repository-original routing, and no claim implies that a universal external standard governs cross-paradigm routing.
An unanswered framing question becomes a recorded gap with an answer holder and a coordinator, and the remaining analysis continues.
Each hypothesis is testable, specific, falsifiable, and tied to a stated rationale.
The experiment selected is the one whose result changes the most downstream work, chosen from a candidate list rather than from the first idea proposed.
Work that is a demo, a mini-MVP, or an already-answered question is named as such rather than run as an experiment.
Every input that could change the result is recorded precisely enough that another person could repeat the experiment and get a comparable answer. An identifier without a version is a label, not a reproducible reference.
Experiment scope stays minimum-but-sufficient, and experiment code is treated as disposable.

Constraints

Reproduce only the minimum text necessary for a specific technical point, and paraphrase everything else. Attribute every reference and describe accurately what each reference reproduces.
Label repository conventions as substitutions rather than upstream recommendations.
Keep experiment framing separate from experiment infrastructure. This pack decides whether and how an experiment is worth running; it does not stand up tooling.
Keep problem-class framing separate from experiment framing. Problem-class framing asks what kind of problem this is; experiment framing asks what to test about a direction already chosen. Do not apply experiment vetting criteria, red flags, or readiness prioritization to a confirmed problem-class framing request.
Never rank, score, threshold, or recommend a candidate problem class. The practitioner is the subject-matter expert who chooses.

Ownership boundaries

Concern	Owner
Candidate problem classes for a stated business outcome, entry-test reasoning, discriminating questions, and assignable framing gaps	This pack, through problem-framing.md
Whether a proposed outcome is achievable with available data and evidence	`feasibility`, the evidence-led feasibility study reference
MVE session directory, artifact filenames, placement, and tracking-file hygiene	`experiment-designer.instructions.md`, applied automatically to MVE tracking paths
Phase order, gates, session writes, and coaching flow	The consuming experiment agent
ML environments, reproducibility, tracking, model evaluation, abstractions, and readiness	`ml-experimentation`, the ML-specific experimentation reference
Pipeline mechanics, data tiering, replay, validation, and DS/MLOps test technique	`dataops`, the DataOps and testing reference
Metric names, instruments, units, cardinality, and PII-safe telemetry	`telemetry-foundations`, the OpenTelemetry-aligned instrumentation skill
Data sensitivity classification and DPIA thresholds	`privacy-standards`, the privacy classification reference

Stop rules

For a confirmed problem-class framing request, an unstated business outcome is the sole stop before candidate analysis. problem-framing.md owns that stop and its job-specific exceptions. The experiment stop rules below govern experiment requests only.
Stop and name the red flag when the request is a demo, a scaled-down product build, or a question already answered elsewhere. Then offer either the falsifiable hypothesis hiding underneath it or an explicit non-experiment path, rather than halting on the refusal.
Stop and separate concerns when the request is production implementation rather than experiment design.
Stop and state the gap when the request depends on guidance this pack does not provide, such as a universal framework, tool, or metric choice made without project context.

Attribution

This pack declares CC-BY-4.0.

mve-coaching.md is repository-original content under CC BY 4.0. It is not derived from any upstream source and cites no upstream URL.

problem-framing.md layers three source treatments. Its machine-learning entry tests are informed by Google for Developers machine-learning guidance, licensed CC BY 4.0; that material is paraphrased and reorganized, both upstream URLs are cited, and changes are stated. It points readers to the NEOS Guide for optimization problem types as a citation only, and states no NEOS classification of its own. Its cross-paradigm routing, the broader analytical-fit judgement, and the gap and output contract are repository-original.

experiment-readiness.md is HVE Core guidance informed by two Microsoft CSE Code With Engineering Playbook documentation pages, which are licensed CC BY 4.0. It paraphrases rather than reproduces, generalizes the upstream practices beyond engagement-shaped engineering work, cites both upstream URLs, and states that changes were made.

See provenance.md for the consolidated source map and derivation labels.

Files

ATTRIBUTION.mdAdded by AutoGineer977 B
LICENSEAdded by AutoGineer16.6 KiB
SKILL.md10.8 KiB
references/experiment-readiness.md6.2 KiB
references/mve-coaching.md19.4 KiB
references/problem-framing.md9.8 KiB
references/provenance.md8.5 KiB
references/rpi-research-preparation.md7.7 KiB

references/provenance.md

title: experiment-design provenance and attribution description: Source map, reproduction scope, HVE Core derivations, and boundaries for the experiment-design reference pack

Purpose

This file records where each part of experiment-design comes from and what is repository convention rather than upstream guidance.

Licensing posture

This pack holds repository-original content and HVE Core guidance informed by upstream sources, both licensed CC BY 4.0. It reproduces no upstream reference text.

The pack previously carried two Microsoft CSE Code With Engineering Playbook references. Those moved to ml-experimentation when the ML-only material was separated from general experiment coaching. Their source map, reproduction scope, and licensing statements travelled with them and are recorded in that pack's provenance.

Microsoft CSE Code With Engineering Playbook documentation is licensed CC BY 4.0, which requires attribution and an indication that changes were made; THIRD-PARTY-NOTICES carries both. This repository's licensing posture is stricter than the license and limits reproduction to the minimum text necessary for a specific technical point. experiment-readiness.md satisfies that limit: it paraphrases throughout and reproduces no upstream passage.

Google for Developers machine-learning documentation is licensed CC BY 4.0 under the same requirements, and THIRD-PARTY-NOTICES carries its attribution and change statement. The NEOS Guide reserves rights and is therefore cited only: no NEOS text, taxonomy, diagram, or derivative of them appears in this pack, and no content claims derivation from it.

Source map

Content area	Source	Reproduction scope
MVE methodology, hypothesis format, vetting criteria, red flags, experiment types, design practices, results evaluation, and the backlog bridge	HVE Core original material	Not derived from any upstream source
Machine-learning versus non-machine-learning comparison, and the supervised entry conditions of labelled outcomes and prediction-time features	Understand the problem	Paraphrased and reorganized into per-paradigm entry tests; no passage reproduced
Beginning with a heuristic and moving to machine learning once the heuristic becomes hard to maintain	Rules of Machine Learning	Paraphrased; no passage reproduced
Optimization problem types, as further reading only	NEOS Guide	Cited only; nothing reproduced, paraphrased, classified, or derived
Cross-paradigm routing, the broader analytical-fit judgement, and the gap and output contracts	HVE Core original material	Not derived from any upstream source
Bounded `rpi-research` preparation triggers, assumption reconciliation statuses, and the preparation-versus-validation boundary	HVE Core original material	Not derived from any upstream source
Candidate generation from a failure exercise, and the mid-flight share that re-prioritizes the next experiment	Engineering Feasibility Spikes	Paraphrased and generalized; upstream terms named for findability; no passage reproduced
Evidence-based comparison of competing options, and when not to run one	Trade Studies	Paraphrased and generalized; upstream terms named for findability; no passage reproduced

MVE coaching content in this pack is HVE Core original material. It was consolidated here from a repository instruction file so that on-demand methodology has a single authoritative home.

Precision worth preserving

An identifier recorded without its version is a label, not a reproducible reference. Whether it names a dataset, a firmware build, a model endpoint, a container image, or a hardware revision, dropping the version turns a reproducibility requirement into a labelling suggestion.
Consistency across inputs and evaluation is what makes experiments comparable. A tracking tool alone does not deliver comparability.

HVE Core derivations

Item	Position
MVE methodology, vetting criteria, red flags, and backlog bridge	HVE Core original coaching material, consolidated into this pack
Cross-paradigm routing among candidate problem classes	HVE Core original. No external standard adjudicates across paradigms; the cited source is machine-learning scoped.
The broader not-a-modelling-problem judgement	HVE Core original. The cited source decides machine learning versus not machine learning only.
Optimization problem-type classification	Not stated by this pack. The framing procedure records the structural question and cites NEOS as further reading.
The gap contract with an answer holder and a separate coordinator	HVE Core original, established by prototype testing rather than upstream guidance
Prioritization signals for choosing among candidate experiments	HVE Core guidance. Upstream describes re-prioritizing after each share but does not enumerate selection signals.
Generalizing spikes and trade studies beyond engagement work	HVE Core framing. Upstream scopes both practices to customer engagements with named ceremonies and meeting lengths.

Where this pack is silent

Whether a proposed outcome is achievable with the available data and evidence belongs to feasibility. This pack exposes candidate problem classes; it does not assess them.
ML experimentation setup, experiment tracking frameworks, dataset and model abstractions, model evaluation flow, and ML checklists belong to ml-experimentation.
Pipeline mechanics, data tiering, replay semantics, and DS/MLOps test technique belong to dataops.
This pack does not select a universal framework, tool, metric, or threshold. It requires that a team decide, document, and apply one consistently, and that decision depends on project context.

Requirements and compatibility

Written for: GitHub Copilot, Claude Code

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Claude Code frontmatter

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

Microsoft

Publisher

Microsoft (Official)

Source

microsoft/hve-core · .github/skills/project-planning/experiment-design · commit f425e94

Licence

Creative Commons Attribution 4.0 International (redistribution allowed)

LICENSE (written by AutoGineer)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by Microsoft.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download experiment-design.zip