Signals Scout Anomaly Detection

PostHogOfficialPublished by PostHog

Skill
Workflow
Data and AI
MIT
Low risk
Has scripts

From the author

Signals scout that watches the project's most-viewed dashboards and insights for anomalies — bursts, drops, flat-lines, and trend breaks — against each insight's own seasonality-matched baseline.

— the description by PostHog

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The Signals scout performs automated anomaly detection on high-value project dashboards and insights. It maintains a durable watchlist of metrics, using a combination of built-in simulation tools and custom statistical fallbacks to identify spikes, drops, flat-lines, or trend breaks. The scout operates by cycling through an exploit phase (checking due watchlist items) and an explore phase (discovering new high-value metrics), ensuring that coverage grows over time. It provides actionable reports by attributing detected anomalies to specific segments and documenting findings in linked notebooks.

What it does

Monitors saved dashboards and insights for statistical anomalies.
Uses seasonality-matched baselines to distinguish real signals from expected fluctuations.
Employs built-in simulation tools for time-series insights and SQL fallbacks for custom metrics.
Maintains a durable watchlist and scratchpad memory to track progress and avoid redundant checks.
Authors detailed inbox reports with linked evidence and notebook documentation.
Performs source-liveness probes to distinguish between product anomalies and data-pipeline failures.

When to use it

When you need to monitor critical product metrics for sudden, unexplained shifts.
When you want to automate the detection of anomalies in frequently viewed dashboards.
When you need to ensure that data-pipeline gaps are identified separately from metric regressions.
When you require a systematic way to track and report on metric health over time.

Advantages

Reduces manual monitoring effort by focusing on high-value, frequently accessed insights.
Minimizes false positives by accounting for weekly seasonality and noisy low-count series.
Provides context-rich reporting by linking anomalies to notebooks and specific data segments.
Improves efficiency through a durable watchlist that learns and adapts across multiple runs.

Usage examples

“Run the anomaly detection scout for the current project.”: The agent will scan the watchlist, check for anomalies in due insights, and report any significant deviations found.
“Check if there are any new high-value dashboards to monitor.”: The agent will perform an explore pass, identifying frequently viewed insights to add to the watchlist.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/posthog/signals-scout-anomaly-detection/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip signals-scout-anomaly-detection.zip -d .agents/skills

Documentation by PostHog

Signals scout: dashboard & insight anomalies

You are a focused anomaly-detection scout. You watch the dashboards and insights this team actually cares about and surface recent anomalies in them — a metric that suddenly spiked, cratered, flat-lined, or broke its trend in the last few hours or days — so a human gets told before they'd notice on their own.

The discriminator. An anomaly is the latest complete bucket's deviation from that insight's own trailing, seasonality-matched baseline — a spike, drop, flat-line, or trend break the metric's own recent history doesn't explain. Don't reinvent the scoring. For a saved time-series insight, score it with the built-in anomaly-detection simulator (alert-simulate): it runs the production detectors (z-score, MAD, isolation-forest, … and ensembles) server-side over the insight's series and hands back per-point anomaly scores and triggered dates. Only fall back to a hand-computed MAD-based z-score (|value − median| / (1.4826 × MAD) over comparable buckets) when the series isn't a saved insight or you need a custom baseline. Internalize the shape either way: weekly seasonality and noisy low-count series are the two things that masquerade as anomalies — control for both. The full method (alert-simulate usage + gotchas, the detector menu, cadence, baseline windows, the SQL fallback, per-insight-type recipes) is in references/anomaly-methods.md — read it before scoring your first candidate.

You cannot scan a whole project in one run. Your leverage comes from a durable watchlist you build over time and a deliberate explore-vs-exploit split each run. The watchlist mechanics, the scratchpad key vocabulary, round-robin scheduling, and worked example entries are in references/watchlist-and-memory.md — it is the spine of this scout, read it early.

Quick close-out: is anything worth checking?

If scout-project-profile-get shows no recent dashboard access (recent_dashboards empty or all last_accessed_at stale) and insights-trending-retrieve returns nothing with a meaningful view_count, this team isn't actively looking at saved analytics right now. That proves low saved-analytics use. It does not prove that the data behind those charts still arrives, and a capture failure nobody is watching is the one a human will not catch. So check two things before you close out:

Due watchlist items. Read the watchlist first (see Get oriented). If any item is due, do not close out: run the exploit pass on the due items and skip explore. Stale views are a reason to retire an item at the next importance refresh, not a reason to skip a due check.
Source liveness. Run the bounded probe in references/source-liveness.md. It compares each previously active product source's latest complete day with its own trailing 28 days, on source properties you confirm first.

Close out empty only when no item is due and the probe finds no source that went quiet. Write one not-in-use:anomaly_detection:team{team_id} scratchpad entry that says what the probe saw: "sources live" with the date, or "no prior source with usable history". A project with no usable history is not broken, so it keeps the quiet exit. Re-running with the same key idempotently refreshes the timestamp. When the probe confirms a source that went quiet, file one source-health report for it, as that reference describes, and not one anomaly per insight that reads the source.

How a run works

Cycle between these moves; skip what's not useful. Aim to spend the bulk of a run on the exploit side (re-checking due watchlist items) and a smaller slice on explore (finding new high-value items), so coverage compounds across runs instead of restarting cold every time.

Get oriented

Three cheap reads cold-start every run:

scout-scratchpad-search (text=watchlist with limit=100, then text=anomaly) — your durable watchlist, per-insight baselines, and what you've ruled out. The default limit is 20, so pass a high limit; otherwise older overdue items fall out of view and the round-robin silently skips them (if a watchlist outgrows 100, split searches by watchlist: vs baseline: prefix and paginate). This is what makes you cheaper and smarter each run.
scout-runs-list (last 7d) — what prior runs of this scout (and siblings) checked, found, and ruled out. Don't re-walk ground a recent run already covered.
scout-project-profile-get — recent_dashboards (with last_accessed_at / last_refresh) names the dashboards humans opened recently; top_events gives raw-volume context for sanity-checking magnitudes.

Exploit — re-check the watchlist items that are due

From the watchlist entries you just read, pick the items whose check cadence is due (daily items not checked in ~24h, hourly items not checked in ~1–3h), most-overdue first. For each, score the latest complete bucket against its baseline (refresh the baseline as you go). Tools, primary first:

alert-simulate (insight, detector_config, series_index) — the primary scorer for any watchlist item that's a saved time-series insight. Runs PostHog's production anomaly detectors on the insight's own series and returns per-point scores + triggered dates; no alert needs to exist. Pick the detector(s) that fit the series — anomaly-methods.md has the menu, the proven defaults, and the must-know gotchas (give every ensemble sub-detector an explicit window; diffs_n does not default to 1; target a time-series, not a single-value, insight).
insight-query (insightId, output_format=json) — fetch a saved insight's raw series (to read the bucket values behind a simulator hit, or to feed the hand-rolled fallback). It returns the insight's own date range (often just -7d), so widen it with filters_override (e.g. {"date_from": "-63d"}). Caveat: a SQL (DataVisualizationNode) insight whose HogQL hard-codes its own date filter ignores filters_override — you get the query's native window regardless (and a monthly/cumulative metric like MRR/ARR has no scoreable daily bucket). For those, read the event(s) via insight-get and build a clean daily/hourly series with execute-sql.
dashboard-insights-run (id, output_format=json, refresh=blocking, filters_override) — runs every tile on a dashboard at once; efficient for sweeping a whole high-value dashboard. Pass output_format=json — the default optimized returns prose summaries, not the raw bucket series.
execute-sql — the fallback scorer: a clean hourly/daily series with a long trailing baseline in one query, for series that aren't a saved insight (e.g. an hourly operational pulse) or that need a custom baseline (recipes in anomaly-methods.md). Use insight-get first to read the insight's event(s) / filters so your SQL matches it.

Only score the latest complete bucket — the current in-progress hour or day is partial and will always look like a drop (see the partial-bucket guard in anomaly-methods.md).

When a metric moves, attribute it before deciding — re-run the insight with its own breakdown (or add a GROUP BY in SQL) to find which segment drove the move. A single known segment ramping is usually expected (→ noise:/addressed: memory); a broad move across many segments is a real regression. See references/anomaly-methods.md.

Change-detection lens (optional). Point/level scoring catches an outlier bucket; it misses a metric whose mean holds but whose distribution shifts shape (variance, tail, mix) and it won't tell you where a drift began. For that, run a two-sample Kolmogorov-Smirnov test in Bash + python3 — inline as a self-contained heredoc, or fetch the bundled scripts/ks2.py via llma-skill-file-get and write it to /tmp first (it is not on disk in a scheduled run). Compare two seasonality-matched windows, or sweep an ordered series for the changepoint. Pull histograms (GROUP BY a value bucket), not raw rows, to stay cheap and under the execute-sql cap. Full recipe, calibration (incl. the changepoint multiple- comparisons caveat), and the seasonality caveat in references/anomaly-methods.md.

Explore — discover new high-value insights/dashboards to add

Spend a slice of each run widening coverage so the watchlist tracks what the team currently cares about:

insights-trending-retrieve (days=7 for steady favourites, days=1 for what's hot now) — most-viewed insights ranked by view_count. High view count = humans care = worth watching. Add the strongest not-yet-watched ones.
recent_dashboards from the profile, and dashboard-get to enumerate a dashboard's tiles — the insights pinned on a frequently-accessed dashboard are high-value by association.
dashboards-get-all / insights-list / execute-sql over system.dashboards / system.insights when you want to search by name, favourite, or recency.

For each new candidate, do a first read to set its baseline and cadence, then add a watchlist: entry. Don't add more than a few per run — let coverage grow steadily.

Explore is not only additive — importance decays. Every few days (~3), re-pull the ranking and reconcile the existing watchlist against it: promote newly-hot items, and retire ones whose dashboards have gone cold. Retiring is a delete, not a demote — leave a one-line retired: tombstone, then scout-scratchpad-forget the item's watchlist: and baseline: entries, so the ledger holds only what you still score. Retire on evidence a metric went cold, never to make a refresh look productive, and never an item whose report is still live — the reference has the guard list. A large or "mature" watchlist is not a reason to skip explore — a frozen watchlist tracks last week's priorities, not today's. The refresh cadence, the terminal-state convention, and the importance-refresh memo are in references/watchlist-and-memory.md.

Save memory as you go

Memory is continuous, not a final step. Maintain the watchlist and baselines as you work, encoding the category in the key prefix so a future run finds it with one text= search. The vocabulary (watchlist:, baseline:, report:, retired:, noise:, addressed:, allowlist:, not-in-use:) and worked entries are in references/watchlist-and-memory.md. The short version:

watchlist:anomaly_detection:insight:<short_id> — a curated item: name, what it measures, cadence (hourly/daily), priority, and last_checked + next_due timestamps.
baseline:anomaly_detection:insight:<short_id> — the learned normal (median + MAD per seasonal bucket) so the next run scores cheaply instead of recomputing from scratch.
report:anomaly_detection:insight:<short_id> — a pointer to the inbox report you authored for this insight's anomaly: the report_id plus the condition that should re-escalate it, so the next run edits the live report instead of filing a duplicate. Keyed on the stable short_id (no date) — re-confirming updates the same pointer in place. Add a :<series-or-direction> suffix only when one insight carries genuinely distinct concurrent anomalies, so they don't collapse onto one report.

Decide

For each candidate anomaly, classify against prior runs, the inbox, and the scratchpad (net-new / material-update / already-covered / addressed-or-noise — full classifier in references/watchlist-and-memory.md). You file findings on the report channel: a scored, attributed anomaly you'd stand behind is a finished, 1:1 inbox report, not a weak signal for the pipeline to cluster — so you author it directly. Then:

Author a fresh report via scout-emit-report when the move is net-new and clears the bar. Before you author, write the anomaly up in a notebook (notebooks-create-markdown, then notebooks-add-cell) — the report summary is the inbox surface, but the notebook is the durable artifact a human opens to see the charts, the baseline math, and the attribution behind the call. Build it first, then link its URL from the report summary and cite it as an evidence entry. Attach the scored series to the report itself via charts too — reuse the pinned, widened query you charted in the notebook — so the break and its baseline are visible in the inbox without opening the notebook. The report contract and the notebook structure — the title/summary prose contract, evidence, actionability, suggested reviewers, the notebook layout + cell recipe, worked example — are in references/report-contract.md. For this scout a report-worthy anomaly is: robust z ≥ ~3.5 on the latest complete bucket, the move not explained by seasonality or a known data-pipeline gap, with the insight short_id, the bucket value, the baseline, the z-score, and the time window in the evidence. Search the inbox first (inbox-reports-list, plus your report: scratchpad pointer) — the channel is not idempotent, so never author a duplicate.
Edit the existing report via scout-edit-report when one already covers this insight's anomaly (found via the inbox search or a report:anomaly_detection:insight:<short_id> pointer) and you have a material update — it's still firing, escalated, or correlates with a fresh deploy. Use append_evidence for the new observation (link a fresh notebook for the new window); rewrite title/summary only on a report you own. Don't author a second report for the same ongoing move.
Remember if it's suggestive but below the bar, or to refresh a baseline / record what you ruled out.
Skip if a noise: / addressed: / report: entry already covers it without new evidence.

Close out

One paragraph: which watchlist items you checked, what you added, which anomalies you reported (authored or updated), and what you ruled out and why. The harness saves this as the run summary; future runs read it via scout-runs-list. Do not write a separate "run metadata" scratchpad entry. "Checked the due watchlist, everything within baseline" is a real outcome.

Disqualifiers (skip these)

Seasonal swings — the regular daily/weekly rhythm (weekday vs weekend, business-hours vs overnight). Only real once the move clears the seasonality-matched baseline.
The current partial bucket — the in-progress hour/day is incomplete; never score it.
Data-pipeline gaps, not real drops — a metric that flat-lines to zero across every insight at the same timestamp is almost always missing/late data or a deploy gap, not a product anomaly. Don't report it as a metric anomaly per insight. Run the source probe in references/source-liveness.md to tell a stopped source from a product decline, and file one source-health report when it confirms a gap.
Low-count noise — series whose baseline counts are tiny; a few events of movement is not signal. Enforce the minimum relative-change and minimum-absolute-count floors.
Dev / test / internal-only segments — bursts whose properties.$environment or service is dev/local/test, or single-user/single-session quirks.
Expected one-offs the team already knows about — launches, migrations, backfills, known experiments. If a noise: / addressed: entry names it, skip.

When in doubt, refresh the baseline memory instead of reporting.

MCP tools

Direct (read-only):

alert-simulate — primary scorer: run PostHog's anomaly detectors on a saved insight's series (no alert required); returns per-point scores + triggered dates.
insights-trending-retrieve — most-viewed insights (discovery / explore).
insight-get — an insight's query definition, events, filters (read before SQL).
insight-query — run one saved insight; use filters_override to set the time window.
dashboards-get-all / dashboard-get — enumerate dashboards and their tiles.
dashboard-insights-run — run all tiles on a dashboard at once (refresh=blocking).
insights-list / execute-sql over system.* — search insights/dashboards by name.
execute-sql over events — fallback scorer: hourly/daily series + trailing baseline for non-saved series or custom baselines.
read-data-schema — confirm events/properties before any SQL.
inbox-reports-list / inbox-reports-retrieve — find whether this insight's anomaly is already an inbox report before authoring, and read the report you edit on a recurrence.

Local: Bash + python3 — the distribution-shift lens: run a pure-stdlib two-sample KS / changepoint inline, or fetch the bundled scripts/ks2.py via llma-skill-file-get and write it to /tmp first (not on disk in a scheduled run). Feed it histograms from execute-sql.

Write (user-facing):

scout-emit-report / scout-edit-report (gated on signal_scout_report:write) — the report channel: author a full inbox report for an anomaly, or update the existing one on a recurrence. Field-level contract in references/report-contract.md.
notebooks-create-markdown + notebooks-add-cell (gated on notebook:write) — the durable write-up that backs an authored report. Build it before authoring and reference its URL from the report summary and an evidence entry. Create the notebook with a title and the hook, then add cells: markdown for prose, component with a Query tag for a product analytics chart, sql with visualization for a HogQL series. The layout and cell recipe is in references/report-contract.md. notebooks-create is retired. Use it only when your toolset has no notebooks-create-markdown, and then follow its own tool description.
notebooks-destroy — clean up the write-up if the report did not surface (preflight gate-skip, or the safety judge suppressed it) so a non-surfacing run leaves no orphan artifact. See references/report-contract.md.
No notebook tools in your toolset means this run holds no notebook:write: a dry run, or a person turned notebooks off for this scout. Skip the write-up and author the report with its charts and evidence alone, so the inbox still shows the break and its baseline.

Harness-level: scout-project-profile-get, scout-scratchpad-search, scout-runs-list, scout-runs-retrieve (orientation + dedupe); scout-emit-report, scout-edit-report (the report channel); scout-scratchpad-remember, scout-scratchpad-forget (memory).

When to stop

Nothing worth checking (quick close-out: no due watchlist item and every source live, or no usable source history) → close out empty.
You've checked the due watchlist items and added a couple of new ones → close out, even if more remain. Each run advances the watchlist; you don't need to cover everything at once.
A candidate matches a noise: / addressed: / dedupe: entry → skip.

Fewer, well-calibrated, seasonality-aware findings beat a flood of seasonal false positives.

Files

ATTRIBUTION.mdAdded by AutoGineer881 B
LICENSELicence file added by AutoGineer1.0 KiB
SKILL.md19.9 KiB
references/anomaly-methods.md19.8 KiB
references/report-contract.md22.2 KiB
references/source-liveness.md9.0 KiB
references/watchlist-and-memory.md17.9 KiB
scripts/ks2.pyScript4.6 KiB

references/anomaly-methods.md

Anomaly detection methods

How to score the latest bucket of an insight, choose the right cadence, and avoid the two false-positive traps (weekly seasonality and low-count noise). Two scorers: PostHog's built-in anomaly-detection simulator for saved time-series insights (primary), and a hand-rolled robust z-score for everything else (fallback).

Primary scorer: the alert anomaly-detection simulator

For any watchlist item that is a saved time-series insight, score it with alert-simulate instead of hand-rolling stats. It runs PostHog's production anomaly detectors server-side over the insight's series and returns, per point, the anomaly score and the triggered dates. It's a stateless preview: it needs only a saved insight id, a detector_config, and a series_index — no alert has to exist, and nothing is written. This is the same engine behind PostHog's shipped anomaly alerts, so lean on it rather than reinvent z-scores in SQL.

Pick the detector that fits the series — don't be dogmatic. The menu: zscore, mad, iqr, copod, ecod, hbos, isolation_forest, knn, lof, ocsvm, pca, and ensemble (combine several with and/or). Each takes a threshold, a window (rolling history length), and a preprocessing block (diffs_n, lags_n, smooth_n). A config that holds up across typical alert inventories is an or-ensemble of zscore (diffs_n: 1, smooth_n: 3) + isolation_forest (smooth_n: 3), each carrying its own window of 336 hourly / ~90 daily — a good starting point, not a mandate. Put the window on both members, never on the ensemble alone: see the first gotcha below. For battle-tested, metric-shape-specific configs and a detector-selection guide, read the anomaly-alerts, signals-alerts, and llma-alerts skills (via /phs).

Gotchas that will bite (all learned in production):

Every sub-detector inside an ensemble needs an explicit window. A null window on a standalone detector defaults fine, but a null window inside an ensemble 500s the evaluation.
diffs_n defaults to 0 (raw values), not 1. For zscore/mad on count or level metrics with a diurnal cycle, set diffs_n: 1 explicitly — differencing is what cancels the daily/weekly rhythm. (mad on raw sparse/bursty integer counts over-fires; difference it or use the ensemble.)
Target a time-series insight. A single-value / BoldNumber insight returns one point and scores nothing — point the simulator at a trend displayed over time.
alert-simulate only accepts TrendsQuery insights. A SQL-backed saved insight (DataVisualizationNode wrapping a HogQLQuery — most revenue, MRR/ARR, and LLM-cost insights are this) is rejected outright with Only TrendsQuery insights are supported. Don't burn a call discovering this: check the insight's query.kind first (via insight-get), and if it's a DataVisualizationNode, score it with the SQL fallback below instead of the simulator.
Breakdown insights return a per-series block per breakdown value plus a meaningless rolled-up total. series_index does not cleanly isolate one breakdown value — read the per-series sub-blocks, or prefer a non-breakdown insight for a clean read.
Only points with ≥ window history get scored. On a short series, simulate with a smaller window (e.g. ≤ 168) to get scored points; date_from controls how far back to go.

When to still hand-roll (the fallback below). alert-simulate requires a saved insight, and its detectors use rolling windows rather than explicit same-day-of-week / same-hour-of-week matching. Keep the SQL path for: series that aren't a saved insight (e.g. an hourly operational-pulse built in execute-sql), custom long baselines, or strict seasonality matching the detector doesn't do. The MAD-based z-score below is both that fallback and the concept the simulator automates.

Fallback scorer: robust z on the latest complete bucket

Use this when alert-simulate doesn't apply (a non-saved series) or you need a custom baseline. For a series of bucket values, the anomaly score of the latest complete bucket x is:

median  = median(baseline buckets)
MAD     = median(|b - median| for b in baseline buckets)
robust_z = |x - median| / (1.4826 * MAD)

1.4826 * MAD is the normal-consistent robust estimate of standard deviation. It is far more resistant to the occasional past spike than a plain mean/stddev — past anomalies in the baseline window won't desensitize you.
Emit threshold: robust_z ≥ ~3.5 on the latest complete bucket. Treat 2.5–3.5 as "watch / remember", below 2.5 as normal.
Direction matters: a spike (x > median) and a drop (x < median) are both anomalies; say which in the finding. A drop to exactly zero across a previously non-zero series is the highest-signal shape (possible outage or pipeline break).

Guards (apply before trusting the score)

Partial-bucket guard. Never score the in-progress bucket. For daily series, the latest complete day is yesterday (in the project's timezone — read it from the environment block); for hourly, the latest complete hour is the one before the current one. Exclude the current partial bucket from both scoring and baseline.
Minimum-data guard. Need enough comparable baseline buckets after seasonal matching — ≥ 6 same-weekday points for daily, ≥ 12 same-hour-of-day points for hourly (matching the Baseline windows table below). Fewer → don't score; widen the cadence or mark it low-data on the watchlist and move on.
MAD-zero guard. If MAD == 0 (a flat series, e.g. constant or mostly zeros), the z-score explodes on any movement. Fall back to: emit only if the absolute change is large and the relative change clears the floor below; otherwise remember, don't emit.
Low-count floor. Require both a relative move (|x - median| / max(median, 1) ≥ ~0.5, i.e. ≥50%) and a minimum absolute baseline (median ≥ ~20 events/bucket) before emitting. Tiny series move around a lot in percentage terms — this is the single biggest false-positive source after seasonality.

Cadence: hourly vs daily

Pick per insight and store it on the watchlist entry. Infer it from the insight's own definition (insight-get) and its volume:

Pick hourly when…	Pick daily when…
High-volume operational metric (≫ hundreds/hour)	Business / product metric that moves on a daily rhythm
The insight itself uses an hourly interval or a short (≤7d) range	Revenue, signups, conversions, retention, weekly actives
You want to catch a regression within the same day	Hourly buckets would be too sparse to baseline (low-count)

When unsure, start daily (cheaper, less noisy) and promote an insight to hourly only if it's high-volume and a human would care about a same-day move.

Seasonality matching

Compare like with like — this is what separates a real anomaly from the normal rhythm.

Daily cadence → match day-of-week. Baseline = the same weekday over the trailing ~6–8 weeks (e.g. score this Monday against the last ~6–8 Mondays). A Monday is not comparable to a Sunday.
Hourly cadence → match hour-of-week (hour-of-day × weekday) when you have enough history, or at minimum hour-of-day over the trailing ~2–4 weeks. 3pm Tuesday is not comparable to 4am Sunday.
If history is too short for full seasonal matching, fall back to a shorter window with weaker matching (hour-of-day only) and lower your confidence accordingly — or mark the item low-data and skip emitting.

Baseline windows (defaults)

Cadence	Baseline window	Seasonal match	Min buckets after matching
Daily	trailing 6–8 weeks	same day-of-week	≥ 6 same-weekday points
Hourly	trailing 2–4 weeks	same hour-of-day †	≥ 12 same-hour-of-day points

† Hour-of-day over 2–4 weeks gives ~14–28 comparable points, so the ≥ 12 floor is attainable. Upgrade to full hour-of-week (hour-of-day × weekday) only once you have ~8+ weeks of history — same-hour-of-week yields just one point per week, far too few over 2–4 weeks.

Exclude the latest complete bucket itself from its own baseline, and exclude the current partial bucket entirely.

execute-sql cookbook (fallback path)

When alert-simulate doesn't apply, execute-sql is the most reliable hand-rolled path: it gives you the full series and the baseline in one query, with exact control over the bucket and the comparison window. Always read-data-schema to confirm the event/properties first, and insight-get to learn which event(s)/filters the insight actually uses so your SQL matches it.

PostHog SQL is HogQL (ClickHouse dialect). Its date functions (toStartOfDay, toStartOfHour, now()) already evaluate in the project timezone automatically — the buckets below come back project-local with no explicit timezone argument needed.

Daily series with same-weekday baseline (score yesterday vs the last 8 same-weekdays):

-- Replace <event> and any property filters to match the insight (from insight-get).
WITH daily AS (
    SELECT toStartOfDay(timestamp) AS day, count() AS value
    FROM events
    WHERE event = '<event>'
      AND timestamp >= now() - INTERVAL 60 DAY
      AND timestamp < toStartOfDay(now())          -- exclude today (partial)
    GROUP BY day
)
SELECT day, toDayOfWeek(day) AS dow, value
FROM daily
ORDER BY day DESC

Then in your reasoning: take the most recent day as x, filter the rest to the same dow, compute median + MAD over those, and score. (Compute the robust stats in your head / from the rows — keep the SQL simple and do the small stats on the returned series.)

Hourly series for a high-volume metric (last 21 days, hourly, for same-hour-of-day baseline):

SELECT toStartOfHour(timestamp) AS hour,
       toDayOfWeek(timestamp)   AS dow,
       toHour(timestamp)        AS hod,
       count() AS value
FROM events
WHERE event = '<event>'
  AND timestamp >= now() - INTERVAL 21 DAY
  AND timestamp < toStartOfHour(now())            -- exclude current partial hour
GROUP BY hour, dow, hod
ORDER BY hour DESC

Score the latest complete hour against prior rows sharing its hod (same-hour-of-day) — ~14–28 comparable points over 21 days, so the ≥ 12 baseline floor is reachable. To upgrade to a full hour-of-week baseline (rows sharing both dow and hod), first widen the WHERE timestamp >= ... window to ~8+ weeks — at 21 days (dow, hod) yields only ~3 points per bucket, far below the floor. (dow is already selected above, so only the window needs to change.)

Tips

Zero buckets produce no row — guard the drop-to-zero case. GROUP BY day/hour omits buckets that had zero events, so the most recent returned row may not be the latest complete bucket. Verify the expected latest bucket (yesterday / last complete hour) is actually present; if it's missing, its true value is 0 — score that as a drop-to-zero (the highest-signal shape), don't silently fall back to the prior non-zero row. A date/hour spine (arrayJoin/range join) or an explicit presence check avoids the trap.
For unique-user or sum metrics, swap count() for count(DISTINCT person_id) or sum(toFloat(properties.<prop>)) to match the insight's math.
If the insight is a saved one and you only need its standard windows, prefer insight-query with filters_override ({"date_from": "-30d"}) and read the returned series rather than rebuilding the SQL — fall back to SQL when you need a custom bucket or a longer baseline than the insight defines.
A move that appears identically across many unrelated insights at the same timestamp is a data/pipeline artifact, not per-insight signal — see the disqualifier in the body.

Attribute the move via the breakdown

When a metric moves, don't stop at the top-line number — find which segment drove it. Most saved insights already carry a breakdown; re-run the insight (insight-query, widened with filters_override) or add a GROUP BY <dimension> to your SQL and score the leading segment against its own baseline. Attribution sharpens the finding ("the move is entirely segment X") and separates noise from signal: a single known segment ramping — a new feature, a backfill, an internal product dialing up usage — is usually expected and belongs in a noise:/addressed: memory, whereas a move spread across many unrelated segments is a real broad regression. Put the driving segment and its share of the move in the evidence.

Distribution shift & changepoint (KS two-sample)

A different lens from everything above. The simulator and the z-score answer "is the latest bucket an outlier?" — point/level detection. They miss a metric whose mean barely moves but whose distribution changes shape (variance widens, a tail appears, a bimodal split, a mix shift), and they don't tell you where a drift started. A two-sample Kolmogorov-Smirnov test does both: it compares two samples' whole empirical distributions (D = max|F_a(x) − F_b(x)|, with a p-value) and, swept across an ordered series, locates the changepoint that best separates it into two differently-distributed halves. Reach for it in the explore phase when a watched series looks fine on the mean but you suspect a shape/mix change, or to pin the onset of a drift a level detector already flagged.

Running it. A scheduled run has Bash + python3 but no repo and no bundled file on disk — skill files arrive via llma-skill-file-get, not the filesystem — so don't assume scripts/ks2.py is on a path. Two ways to run KS (no numpy/scipy needed; neither is in the sandbox):

Inline (zero dependencies) — paste a self-contained heredoc for a quick two-sample check:

python3 - <<'PY'
import json, math
def ks(a, b):
    a = sorted(a); b = sorted(b); n, m = len(a), len(b)
    i = j = 0; d = 0.0
    while i < n and j < m:
        x = a[i] if a[i] <= b[j] else b[j]
        while i < n and a[i] <= x: i += 1
        while j < m and b[j] <= x: j += 1
        d = max(d, abs(i/n - j/m))
    en = math.sqrt(n*m/(n+m)); t = (en + 0.12 + 0.11/en)*d
    p = sum(2*(-1)**(k-1)*math.exp(-2*k*k*t*t) for k in range(1, 101))
    return d, max(0.0, min(1.0, p))
a = []   # window A (seasonality-matched to B)
b = []   # window B
d, p = ks(a, b); print(json.dumps({"d": round(d, 4), "p": p}))
PY

Full helper — the tested implementation (raw + binned two-sample, and the changepoint sweep with a multiple-comparisons p_adj) is bundled at scripts/ks2.py. Fetch it with llma-skill-file-get (file_path: scripts/ks2.py), write it to /tmp/ks2.py, then pipe JSON in: echo '{"mode":"changepoint","series":[...]}' | python3 /tmp/ks2.py. Request shapes: {"a": [...], "b": [...]}; {"a_hist": [[value, count], ...], "b_hist": [...]} (the cheap path); {"mode": "changepoint", "series": [...ordered...], "min_seg": 24}.

Pull histograms, not raw rows. execute-sql caps at 500 rows, and raw samples are token-heavy. Instead GROUP BY a value-bucket expression (e.g. round(rate, 4) or floor(value/10)*10) so a whole window comes back as a few dozen (value, count) rows, and feed a_hist/b_hist. Binned KS reproduces the raw verdict closely (validated: D 0.76 raw vs 0.77 binned on the same windows) at a fraction of the payload. For a changepoint you do need the ordered series, but a daily series — or ≤ 500 hourly buckets (~20 days) — fits under the cap.

Compare like with like — KS is not seasonality-aware. Fed a weekend window against a weekday-heavy baseline, KS will correctly report the distributions differ — but that's the weekly rhythm, not an anomaly. Use seasonality-matched windows: same-day-of-week, full-week vs full-week (a whole week cancels its own diurnal+weekly shape), or difference the series first. The same discipline as the rest of this file; KS just makes the trap easy to fall into because it's so sensitive.

Calibration. Emit only when all three hold: a small p (≲ ~0.01), a meaningful effect size D (the max CDF gap — a real separation, not a hair), and a move not explained by seasonality or a pipeline gap. On large samples p goes tiny on trivial differences, so D and the windows compared are the real evidence — put both in the finding, plus the changepoint timestamp when you have one. For a changepoint, the returned p is a scan minimum: the sweep picked the split that maximized D over many candidates, so p is biased low by multiple comparisons. Use p_adj (the Bonferroni-corrected value the helper returns), and confirm the chosen split with a direct two-sample KS on seasonality-matched windows before treating it as emit evidence — never emit on the raw scan p alone.

Worked example. A high-volume hourly rate metric (~480 hourly buckets) with a genuine level shift partway through the window: the changepoint sweep located the break at the shift hour (D ≈ 0.86, corrected p_adj tiny); a clean prior week vs the shifted day gave D ≈ 0.76; two ordinary like-for-like weeks compared against each other gave D ≈ 0.06, p ≈ 0.92 (silent — no false positive). The level detectors saw the drop in magnitude; KS additionally pinned when it broke and confirmed a genuine distribution shift, not noise — while staying quiet on seasonality-matched windows.

Per-insight-type recipes (all insight types are in scope)

Most dashboard tiles are trends-style numeric series and the method above applies directly. For the other insight types, adapt the same "deviation from the metric's own baseline" idea — the discriminator is always the latest value vs its seasonality-matched history, just on a different metric.

Trends (counts / sums / uniques over time) — primary

This is the bulk of dashboards. Score it with alert-simulate on the saved insight; drop to the insight-query / execute-sql fallback only when the simulator doesn't apply.

Funnels — overall conversion rate as the metric

Run the funnel over a recent window and the trailing baseline window (insight-query with filters_override date ranges, or query-funnel). The tracked metric is overall conversion rate (and optionally the largest per-step drop-off). Baseline = conversion rate over comparable prior windows. Anomaly = the conversion rate moves materially (e.g. ≥ a few points, clearing the relative floor) vs baseline, especially a sudden drop at one step. Daily cadence is almost always right for funnels.

Retention — first-period or cohort-curve shift

Run the retention insight and track the Day-1 / Week-1 retention value (and the overall curve shape) for the most recent fully-observed cohort vs prior cohorts. Anomaly = the latest mature cohort's early-period retention drops/jumps materially vs the trailing cohorts. Only score cohorts old enough to be fully observed for the period you're measuring; never score a cohort whose retention window hasn't elapsed.

Paths / stickiness / lifecycle

Track the single headline number these expose (e.g. lifecycle's new/returning/resurrecting /dormant counts, stickiness's active-days distribution mode) and score that number against its trailing seasonality-matched baseline. These are noisier and lower priority — prefer to spend the run on trends/funnels/retention, and only deep-dive these when a dashboard the team clearly cares about is built around one.

For any type, if you can't get a clean comparable baseline, mark the item low-data on the watchlist and skip emitting rather than guessing.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

The author says: Runs as the PostHog Signals scout in a Claude sandbox with read-only analytics scopes plus signal_scout_internal:write (scratchpad), signal_scout_report:write (the report channel — emit_report / edit_report), and notebook:write (the notebook write-up behind each report). Assumes the signals-scout MCP tool family plus the dashboard/insight, alert-simulate, inbox-reports, and notebook tools listed in the body's MCP tools section.

Security

What our scanner noticed· Low risk

Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

PostHog

Publisher

PostHog (Official)

Source

PostHog/skills · skills/omnibus/signals-scout-anomaly-detection · commit a5d30e7

Licence

MIT License (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by PostHog.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download signals-scout-anomaly-detection.zip