AutoGineer
Generated banner for Content Experimentation Best Practices
Generated logo for Content Experimentation Best Practices

Content Experimentation Best Practices

SanityOfficialPublished by Sanity

  • Skill
  • Guidelines
  • Business and productivity
  • MIT
  • No issues found
Download ZIP

From the author

Content experimentation and A/B testing guidance covering experiment design, hypotheses, metrics, sample size, statistical foundations, CMS-managed variants, and common analysis pitfalls. Use this skill when planning experiments, setting up variants, choosing success metrics, interpreting statistical results, or building experimentation workflows in a CMS or frontend stack.
— the description by Sanity

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured framework for content experimentation, covering the entire lifecycle from hypothesis generation to statistical analysis. It offers best practices for designing tests, avoiding common analytical pitfalls, and integrating experimentation workflows directly into CMS architectures. The guidance emphasizes data-driven decision-making, helping teams move beyond intuition to validate content changes through rigorous testing methodologies.

What it does

  • Defines frameworks for A/B and multivariate testing.
  • Provides patterns for integrating experiment variants into CMS schemas.
  • Outlines statistical foundations including significance and power analysis.
  • Identifies common pitfalls in experiment design, execution, and interpretation.
  • Offers guidance on hypothesis formulation and metric selection.
  • Details architectural approaches for managing variants within content platforms.

When to use it

  • When planning A/B or multivariate tests for website content.
  • When designing the schema for experiment-managed content in a CMS.
  • When troubleshooting inconclusive or suspicious test results.
  • When establishing an experimentation culture to replace opinion-based decisions.
  • When determining the required sample size for a new experiment.
  • When evaluating whether to use CMS-native variants or external platforms.

Advantages

  • Reduces the risk of false positives and invalid test conclusions.
  • Provides clear architectural patterns for CMS-based experimentation.
  • Helps teams avoid common statistical and design mistakes.
  • Promotes a systematic approach to learning from content changes.
  • Supports better collaboration between content and development teams.

Usage examples

“How should I structure my CMS schema to support A/B testing on landing page headlines?”
The skill provides specific schema examples for field-level variants or dedicated experiment documents, including fields for variant weights and status tracking.
“Why are my A/B test results fluctuating so much?”
The skill explains the 'peeking problem' and provides guidance on pre-calculating sample sizes and committing to run times to ensure statistical validity.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/sanity-io/content-experimentation-best-practices/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip content-experimentation-best-practices.zip -d .agents/skills

Documentation by Sanity

Content Experimentation Best Practices

Principles and patterns for running effective content experiments to improve conversion rates, engagement, and user experience.

When to Apply

Reference these guidelines when:

  • Setting up A/B or multivariate testing infrastructure

  • Designing experiments for content changes

  • Analyzing and interpreting test results

  • Building CMS integrations for experimentation

  • Deciding what to test and how

Core Concepts

A/B Testing

Comparing two variants (A vs B) to determine which performs better.

Multivariate Testing

Testing multiple variables simultaneously to find optimal combinations.

Statistical Significance

The confidence level that results aren't due to random chance.

Experimentation Culture

Making decisions based on data rather than opinions (HiPPO avoidance).

References

Start with the reference that matches the current problem, such as design, statistics, CMS integration, or pitfalls. See references/ for detailed guidance:

  • references/experiment-design.md — Hypothesis framework, metrics, sample size, and what to test

  • references/statistical-foundations.md — p-values, confidence intervals, power analysis, Bayesian methods

  • references/cms-integration.md — CMS-managed variants, field-level variants, external platforms

  • references/common-pitfalls.md — 17 common mistakes across statistics, design, execution, and interpretation

Files

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
Sanity
Publisher
Sanity (Official)
Source
sanity-io/agent-toolkit · skills/content-experimentation-best-practices · commit 3ed3a14
Licence
MIT License (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by Sanity.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download content-experimentation-best-practices.zip