Suggesting Path Cleaning Rules

Suggesting path-cleaning rules

Many teams never configure path cleaning, so their Web analytics breakdowns fragment across thousands of near-identical URLs. This feature proactively suggests cleaning rules for the web-analytics precompute cohort: weekly, for each team, it samples real paths, asks the LLM for {regex, alias} rules, validates them against the team's own paths, and stores them for review.

It only suggests — it never auto-applies. Applying rewrites historical numbers in every cleaned chart, so that stays a human decision (the existing settings UI, or the --apply flag below after review). To hand-author or directly apply rules, use the managing-path-cleaning-rules skill.

Architecture

• Core: products/web_analytics/backend/path_cleaning_suggestions/service.py

  • sample_pathnames / count_distinct_pathnames — top $pathname by views via HogQL.

  • call_llm_for_rules — one-shot call through the LLM gateway (get_llm_client(product="web_analytics", team_id=...), model WEB_ANALYTICS_PATH_CLEANING_SUGGESTIONS_MODEL, default claude-haiku-4-5).

  • validate_and_annotate_rules — compiles each regex with re2 (the engine ClickHouse replaceRegexpAll uses) and test-applies it to the sampled paths. Rules that don't compile or match nothing are dropped; survivors get a dense order, a match_count, and in-memory before/after examples (printed by the management command, never stored — health-issue payloads are readable with just health_issue:read and must not leak real paths). This is the skill's "test before saving" step, automated.

  • generate_suggestions_for_team — orchestrates the above with gating (see below); pure generation, no storage.

  • apply_suggestions_to_team — merges rules into path_cleaning_filters, never overwrites (dedupes by regex, continues order).

• Storage: a path_cleaning_suggestions health issue (HealthIssue, severity info) — no dedicated model. One active issue per team (hash_keys=[]); payload carries rules, model, sampled_path_count, distinct_path_count. Applying (or hand-configuring rules) resolves the issue on the next check run; dismissal is the health-issue dismissed flag.

• Schedule: PathCleaningSuggestionsCheck (products/web_analytics/backend/temporal/health_checks/path_cleaning_suggestions.py), a health check on the shared health-check framework, weekly (Mon 06:23 UTC), small sequential batches because each eligible team costs an LLM call. Teams with an existing active suggestion are re-emitted without a fresh LLM round trip.

• Cohort: WEB_ANALYTICS_PATH_CLEANING_SUGGESTIONS_TEAM_IDS, defaulting to the precompute enrollment list WEB_ANALYTICS_LAZY_PRECOMPUTE_TEAM_IDS.

Gating (why a team is skipped)

generate_suggestions_for_team returns a status:

• skipped_inactive — team sent no $pageview within visited_within_days (default 30); we only suggest for teams actively using web analytics. Bypass with --ignore-visit-gate.

• skipped_configured — team already has path cleaning rules (override with include_configured).

• skipped_low_cardinality — fewer distinct paths than min_distinct_paths (default 50); cleaning adds no value, so we don't spend tokens.

• skipped_no_paths — no pageviews in the window.

• generated — rules produced (may be an empty list if paths are already clean; empty generations are never stored, so they can't shadow an actionable suggestion).

• error — sampling/LLM failed; captured per-team, never aborts the cohort sweep.

How users see and apply suggestions

• Settings banner: PathCleaningSuggestionsBanner on /settings/project#path_cleaning shows the latest suggested row as regex → alias previews with match counts; "Apply all" (project admins only) merges the rules, the close button dismisses. Driven by pathCleaningSuggestionsLogic.

• Onboarding step: OnboardingWebAnalyticsPathCleaningStep (stepKey path_cleaning) surfaces the same banner during Web analytics onboarding.

• API (products/web_analytics/backend/api/web_analytics_path_cleaning_suggestions.py): POST /api/projects/:id/web_analytics_path_cleaning_suggestions/generate/ produces and stores a fresh suggestion on demand; GET .../{issue_id}/preview/ applies the rules to a fresh sample of the team's top paths and returns before/after pairs (read scope, computed on demand, never stored — this backs the banner's "Preview on your paths" modal); POST .../{issue_id}/apply/ merges the rules and resolves the issue (project admin only — the same gate the team API puts on path_cleaning_filters). Listing and dismissing go through the generic health-issues API (GET /api/projects/:id/health_issues/?kind=path_cleaning_suggestions&status=active&dismissed=false, PATCH .../health_issues/{id}/ with {"dismissed": true}).

• Health page: the check renders on /web/health alongside the other web-analytics checks, with remediation guidance for humans and agents.

• PostHog AI (Max): generate/apply are exposed as MCP tools in products/web_analytics/mcp/tools.yaml (web-analytics-path-cleaning-suggestions-{generate,apply}), so a user can ask Max to suggest path-cleaning rules and apply them conversationally. Apply is destructive (it changes historical chart numbers), so the MCP confirmation gate applies.

Running it

# Default cohort, print suggestions, store health issues:
python manage.py suggest_path_cleaning_rules

# Specific teams, dry run (nothing stored):
python manage.py suggest_path_cleaning_rules --teams 2,19279 --no-store

# Generate AND apply for one reviewed team (merges, never overwrites):
python manage.py suggest_path_cleaning_rules --teams 2 --apply

Useful flags: --days (lookback), --limit (top-N paths sampled), --min-distinct-paths, --include-configured, --no-store, --apply.

The health check can also be triggered per team from the health-issues refresh endpoint or the admin UI, like any other health check.

Reviewing suggestions

Read a team's active suggestion:

HealthIssue.objects.filter(team_id=team_id, kind="path_cleaning_suggestions", status="active").first()

Each rule in payload["rules"] carries regex, alias, order, reason, and match_count — that's what to show a human deciding whether to apply. Before/after examples on real paths are only printed by the management command at generation time; they are deliberately kept out of the stored payload.

Extending

• Adding a surfacing channel (in-app notification, settings banner, onboarding wizard step): read the team's active path_cleaning_suggestions health issue and render its payload["rules"]. Keep apply manual.

• Changing the model: it must be allowlisted for the web_analytics product in services/llm-gateway/src/llm_gateway/products/config.py.

• The agentic alternative — a signals-scout-web-analytics-path-cleaning scout — is sketched in the design notes; prefer the dedicated job for the precompute cohort because it targets that exact cohort and surfaces structured, validated rows rather than Signals-inbox findings.