AutoGineer
Generated banner for Auditing Experiments Flags
Generated logo for Auditing Experiments Flags

Auditing Experiments Flags

PostHogOfficialPublished by PostHog

  • Skill
  • Review and audit
  • Data and AI
  • MIT
  • No issues found
Download ZIP

From the author

Audit PostHog experiments and feature flags for configuration issues, staleness, and best-practice violations. Read when the user asks to audit, health-check, or review experiments or feature flags, check flag hygiene, or verify experiment setup.
— the description by PostHog

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured approach to auditing PostHog experiments and feature flags. It enables users to perform quick checks on individual entities, scoped audits of specific domains, or comprehensive full-project audits. The skill uses defined check taxonomies to identify configuration errors, waste, and process violations, providing clear remediation actions for each finding.

What it does

  • Performs configuration audits on PostHog experiments and feature flags.
  • Identifies critical correctness issues, potential waste, and process violations.
  • Supports single-entity checks, scoped domain audits, and full-project reviews.
  • Categorizes findings by severity (CRITICAL, WARNING, INFO) and domain.
  • Provides actionable remediation steps for every identified issue.
  • Handles data gaps gracefully by noting skipped checks when activity logs are unavailable.

When to use it

  • When you need to verify the health and setup of a specific experiment or feature flag.
  • When performing routine hygiene checks to identify stale or misconfigured flags.
  • When preparing for a launch to ensure experiment metrics and flag rollouts are correctly configured.
  • When cleaning up project resources to reduce technical debt and waste.

Advantages

  • Standardizes the auditing process across different experiments and flags.
  • Prioritizes findings by severity to help focus on the most impactful issues first.
  • Provides clear, actionable guidance for resolving configuration problems.
  • Maintains consistency in reporting through a structured output format.

Usage examples

“Audit the experiment with ID 42.”
The agent fetches the experiment, runs all relevant configuration checks, and reports findings grouped by severity with actionable remediation steps.
“Perform a health check on all active feature flags.”
The agent retrieves all flags, evaluates them against best-practice criteria, and provides a summary report of any issues found.
“Run a comprehensive audit of all experiments and flags in this project.”
The agent performs a full audit, synthesizes recurring patterns across the project, and presents the findings in a structured report or notebook.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/posthog/auditing-experiments-flags/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip auditing-experiments-flags.zip -d .agents/skills

Documentation by PostHog

Auditing experiments and feature flags

This skill teaches you how to run configuration audits on experiments and feature flags. All checks use the experiment and feature flag read tools (experiment-get, experiment-list, feature-flag-get-definition, feature-flag-get-all) — no SQL queries are needed for Phase 1 checks.

Usage modes

Quick check (single entity)

When the user asks about a specific experiment or flag:

  1. Fetch the entity via experiment-get (experiment ID) or feature-flag-get-definition (numeric flag ID).

  2. Apply the relevant checks from experiment checks or flag checks.

  3. Report findings inline as markdown, grouped by severity (CRITICAL first, then WARNING, then INFO).

  4. Include entity links as [Experiment: name](/experiments/id) or [Flag: key](/feature_flags/id).

Scoped audit (one domain)

When the user asks to audit all experiments or all flags:

  1. Bulk-fetch via experiment-list or feature-flag-get-all.

  2. Run all checks for that domain against each entity.

  3. Group findings by severity, then by entity.

  4. Report as inline markdown.

Full audit (comprehensive)

When the user asks for a comprehensive audit of both experiments and flags:

  1. Fetch all experiments via experiment-list and all flags via feature-flag-get-all.

  2. Run all experiment checks and all flag checks.

  3. Apply recurring patterns to identify patterns across multiple findings.

  4. If there are more than 5 entities with findings, write them to a notebook for easier navigation. Otherwise report inline. Create the notebook from the project's own notebook tools. Run search notebooks?- to load them and read the titles.

Output format

For each finding, include:

  • Severity badge: 🔴 CRITICAL, 🟡 WARNING, or 🔵 INFO

  • Check name: Which check produced this finding

  • Entity link: Markdown link to the entity

  • What's wrong: One-sentence description

  • Action: What to do about it (see remediation actions)

Example:

🟡 WARNING — Flag integration · Experiment: checkout-redesign The linked feature flag is inactive (paused). Traffic is not being split. Action: Re-enable the flag or end the experiment.

Handling unavailable data

Some checks require activity logs (feature-flags-activity-retrieve for flags), which may not be available in every session. If activity log data is unavailable:

  • Skip checkActivityHistory (experiment check) entirely.

  • Skip the "toggle instability" and "never activated" sub-checks in flag lifecycle checks.

  • In your report, note which checks were skipped and why:

    Skipped: Activity history checks (activity logs not available via current tools)

Partial failures

If a fetch call fails for some entities:

  • Continue with the entities you could fetch.

  • Report which entities could not be assessed and why.

  • Do not silently omit entities from the audit.

Reference files

Files

references/flag-checks.md

Feature flag checks

Run these checks against each flag fetched via feature-flag-get-definition or feature-flag-get-all.

1. Staleness: fully rolled out

Detects active boolean flags that are effectively permanent and can be removed from code.

Look at: active, filters.multivariate (should be absent or null for boolean flags), filters.groups, last_called_at

Findings:

  • Fully rolled out boolean flag: Flag is active: true, has no multivariate config, and at least one release condition (filters.groups entry) with rollout_percentage: 100 and no properties (empty array or missing). This flag always evaluates to true.

    • Severity: INFO · Category: Cleanup

    • Report: "This boolean flag is rolled out to 100% with no targeting conditions. It always evaluates to true."

    • Action: Remove the flag from code and hardcode the value.

  • Possibly unused: Flag has last_called_at that is more than 30 days ago, regardless of rollout configuration.

    • Severity: INFO · Category: Cleanup

    • Report: "This flag hasn't been evaluated in N days. It may no longer be referenced in code."

    • Action: Check if the flag is still referenced in your codebase. If not, delete it.

2. Staleness: stale draft

Detects flags that were created but never activated.

Look at: active, created_at, activity logs (if available)

Findings:

  • Stale draft flag: active is false, flag is more than 30 days old (based on created_at), and activity logs confirm it was never activated.

    • Severity: INFO · Category: Cleanup

    • Report: "This flag has been inactive for N days and was never activated."

    • Action: Delete the flag if it's no longer needed, or activate it.

    Note: The "never activated" check requires activity logs. If activity logs are unavailable, skip this sub-check and only report based on the flag being inactive and old:

    • Report: "This flag has been inactive for N days. Could not verify whether it was ever activated (activity logs unavailable)."

3. Staleness: orphaned experiment flag

Detects flags whose linked experiments are all done.

Look at: experiment_set (list of linked experiment IDs), and for each experiment, check its end_date and archived status via experiment-get.

Findings:

  • Orphaned experiment flag: Flag is active: true, has entries in experiment_set, and ALL linked experiments have end_date set (completed) or archived: true.

    • Severity: INFO · Category: Cleanup

    • Report: "This flag's linked experiments are all completed or archived. The flag is no longer serving an active experiment."

    • Action: Roll out the winning variant at 100% and remove the flag from code, or disable the flag.

4. Rollout integrity: variant sum

Checks multivariate flag rollout percentages for correctness.

Look at: filters.multivariate.variants (array of {key, rollout_percentage, ...}), experiment_set

Findings:

  • Variant sum != 100%: The sum of all rollout_percentage values across filters.multivariate.variants does not equal 100.

    • Severity: WARNING · Category: Correctness

    • Report: "Multivariate rollout percentages sum to N%, not 100%. Traffic distribution is incorrect."

    • Action: Adjust variant rollout percentages to sum to 100%.

  • Dead variant (0% rollout): A variant has rollout_percentage: 0 on a flag that is NOT linked to an experiment (empty experiment_set).

    • Severity: INFO · Category: Cleanup

    • Report: "Variant 'X' has 0% rollout and receives no traffic."

    • Action: Either give the variant traffic or remove it.

  • Dead condition (0% rollout): A release condition in filters.groups has rollout_percentage: 0.

    • Severity: INFO · Category: Cleanup

    • Report: "A release condition has 0% rollout and is not serving any traffic."

    • Action: Either increase the rollout or remove the condition.

  • Manual rollout on experiment flag: A flag with entries in experiment_set has release conditions where rollout_percentage is not the expected even split. This suggests someone manually adjusted the rollout outside the experiment.

    • Severity: INFO · Category: Process

    • Report: "This experiment flag has manual rollout overrides that differ from the experiment's expected split."

    • Action: Remove manual overrides and let the experiment control the variant split.

5. Lifecycle

Checks for flags with unstable or high-churn configurations. These checks require activity logs. If unavailable, skip and note it.

Look at: Activity log entries for the flag, created_at

Findings:

  • Toggle instability: The flag has been toggled on/off (active → inactive or vice versa) more than 3 times based on activity logs.

    • Severity: WARNING · Category: Complexity

    • Report: "This flag has been toggled on/off N times. Frequent toggling suggests it may be used as a kill switch or there's uncertainty about its state."

    • Action: Consider whether the flag is being used as intended. If it's a kill switch, document that purpose.

  • High config churn: The flag has more than 20 activity log entries AND the average rate exceeds 0.5 changes per day (calculated from first to last activity log entry).

    • Severity: WARNING · Category: Complexity

    • Report: "This flag has been modified N times at a rate of X changes/day. High churn can indicate instability."

    • Action: Consider stabilizing the configuration or splitting into multiple simpler flags.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
PostHog
Publisher
PostHog (Official)
Source
PostHog/skills · skills/omnibus/auditing-experiments-flags · commit a5d30e7
Licence
MIT License (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by PostHog.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download auditing-experiments-flags.zip