Content Experimentation Best Practices

From the author

Content experimentation and A/B testing guidance covering experiment design, hypotheses, metrics, sample size, statistical foundations, CMS-managed variants, and common analysis pitfalls. Use this skill when planning experiments, setting up variants, choosing success metrics, interpreting statistical results, or building experimentation workflows in a CMS or frontend stack.

— the description by Sanity

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured framework for content experimentation, covering the entire lifecycle from hypothesis generation to statistical analysis. It offers best practices for designing tests, avoiding common analytical pitfalls, and integrating experimentation workflows directly into CMS architectures. The guidance emphasizes data-driven decision-making, helping teams move beyond intuition to validate content changes through rigorous testing methodologies.

What it does

Defines frameworks for A/B and multivariate testing.
Provides patterns for integrating experiment variants into CMS schemas.
Outlines statistical foundations including significance and power analysis.
Identifies common pitfalls in experiment design, execution, and interpretation.
Offers guidance on hypothesis formulation and metric selection.
Details architectural approaches for managing variants within content platforms.

When to use it

When planning A/B or multivariate tests for website content.
When designing the schema for experiment-managed content in a CMS.
When troubleshooting inconclusive or suspicious test results.
When establishing an experimentation culture to replace opinion-based decisions.
When determining the required sample size for a new experiment.
When evaluating whether to use CMS-native variants or external platforms.

Advantages

Reduces the risk of false positives and invalid test conclusions.
Provides clear architectural patterns for CMS-based experimentation.
Helps teams avoid common statistical and design mistakes.
Promotes a systematic approach to learning from content changes.
Supports better collaboration between content and development teams.

Usage examples

“How should I structure my CMS schema to support A/B testing on landing page headlines?”: The skill provides specific schema examples for field-level variants or dedicated experiment documents, including fields for variant weights and status tracking.
“Why are my A/B test results fluctuating so much?”: The skill explains the 'peeking problem' and provides guidance on pre-calculating sample sizes and committing to run times to ensure statistical validity.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/sanity-io/content-experimentation-best-practices/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip content-experimentation-best-practices.zip -d .agents/skills

Documentation by Sanity

Principles and patterns for running effective content experiments to improve conversion rates, engagement, and user experience.

When to Apply

Reference these guidelines when:

Setting up A/B or multivariate testing infrastructure
Designing experiments for content changes
Analyzing and interpreting test results
Building CMS integrations for experimentation
Deciding what to test and how

Core Concepts

A/B Testing

Comparing two variants (A vs B) to determine which performs better.

Multivariate Testing

Testing multiple variables simultaneously to find optimal combinations.

Statistical Significance

The confidence level that results aren't due to random chance.

Experimentation Culture

Making decisions based on data rather than opinions (HiPPO avoidance).

References

Start with the reference that matches the current problem, such as design, statistics, CMS integration, or pitfalls. See references/ for detailed guidance:

references/experiment-design.md — Hypothesis framework, metrics, sample size, and what to test
references/statistical-foundations.md — p-values, confidence intervals, power analysis, Bayesian methods
references/cms-integration.md — CMS-managed variants, field-level variants, external platforms
references/common-pitfalls.md — 17 common mistakes across statistics, design, execution, and interpretation

Files

ATTRIBUTION.mdAdded by AutoGineer907 B
LICENSELicence file added by AutoGineer1.0 KiB
SKILL.md1.8 KiB
references/cms-integration.md6.8 KiB
references/common-pitfalls.md6.7 KiB
references/experiment-design.md3.1 KiB
references/statistical-foundations.md4.9 KiB

references/experiment-design.md

Experiment Design Principles

Well-designed experiments produce actionable insights. Poorly designed ones waste time and can mislead.

The Experiment Framework

1. Hypothesis

State what you believe and why.

Bad: "Let's test a new headline" Good: "We believe a benefit-focused headline will increase signup rate by 10% because users are currently confused about our value proposition"

Structure: "We believe [change] will [impact metric] because [reasoning]"

2. Success Metric

Define primary and guardrail metrics.

Primary metric: The main thing you're trying to improve (conversion rate, engagement time) Guardrail metrics: Things that shouldn't get worse (bounce rate, page load time)

3. Sample Size

Calculate required sample size before starting.

Factors:

Baseline conversion rate
Minimum detectable effect (MDE)
Statistical significance level (usually 95%)
Statistical power (usually 80%)

Use calculators like Evan Miller's.

4. Duration

Run tests for full business cycles.

Minimum: 1-2 weeks (capture weekly patterns)
Include weekends
Avoid holidays and major events
Don't stop early when you see "winning" results

What to Test

High-Impact Areas

Headlines and value propositions
Call-to-action text and placement
Form length and fields
Pricing presentation
Social proof placement

Lower-Impact (Usually)

Button colors
Minor copy tweaks
Image variations (unless hero)
Footer changes

Test Priority Matrix

Impact	Effort	Priority
High	Low	Do first
High	High	Plan carefully
Low	Low	Quick wins
Low	High	Avoid

Sanity Integration Pattern

// Experiment variant schema
defineType({
  name: 'experimentVariant',
  type: 'object',
  fields: [
    defineField({ name: 'name', type: 'string' }),
    defineField({ name: 'weight', type: 'number', description: 'Traffic allocation (0-100)' }),
    defineField({ name: 'content', type: 'reference', to: [{ type: 'page' }] }),
  ]
})

// Experiment document
defineType({
  name: 'experiment',
  type: 'document',
  fields: [
    defineField({ name: 'name', type: 'string' }),
    defineField({ name: 'hypothesis', type: 'text' }),
    defineField({ name: 'status', type: 'string', options: { 
      list: ['draft', 'running', 'concluded'] 
    }}),
    defineField({ name: 'variants', type: 'array', of: [{ type: 'experimentVariant' }] }),
    defineField({ name: 'startDate', type: 'datetime' }),
    defineField({ name: 'endDate', type: 'datetime' }),
    defineField({ name: 'winner', type: 'string' }),
    defineField({ name: 'learnings', type: 'text' }),
  ]
})

Avoiding Common Mistakes

Don't peek and stop early

Statistical significance can fluctuate. Commit to your sample size.

Don't test too many things at once

Each variable multiplies required sample size.

Don't ignore segmentation

Winners may differ by device, traffic source, or user type.

Document everything

Future you (and your team) will thank you.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Marketing Plan
Corey Haines· Community
Generates a comprehensive, executable 12-month marketing plan structured by the AARRR framework for fractional CMOs and founders.
Business and productivityClaude CodeClaude appsOpenAI Codex+14 more
MITLow riskGitHub stars of the source repository59k
Revops
Corey Haines· Community
Expert guidance for designing and optimizing revenue operations, lead lifecycle management, and marketing-to-sales handoff processes.
Business and productivityClaude CodeClaude appsOpenAI Codex+14 more
MITGitHub stars of the source repository59k
Ab Testing
Corey Haines· Community
A comprehensive framework for planning, executing, and analyzing statistically rigorous A/B tests and growth experimentation programs.
Business and productivityClaude CodeClaude appsOpenAI Codex+14 more
MITGitHub stars of the source repository59k
Pricing
Corey Haines· Community
Expert guidance on SaaS pricing strategy, packaging, monetization, and auditing pricing pages for human and AI-agent readability.
Business and productivityClaude CodeClaude appsOpenAI Codex+14 more
MITGitHub stars of the source repository59k
Community Marketing
Corey Haines· Community
A strategic framework for building and managing online communities to drive product growth, brand loyalty, and user engagement.
Business and productivityClaude CodeClaude appsOpenAI Codex+14 more
MITGitHub stars of the source repository59k
Paywalls
Corey Haines· Community
Expert guidance for designing and optimizing in-app upgrade flows, paywalls, and feature gates to maximize conversion.
Business and productivityClaude CodeClaude appsOpenAI Codex+14 more
MITGitHub stars of the source repository59k

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download content-experimentation-best-practices.zip