Prometheus Cardinality Troubleshooter

Grafana LabsOfficialPublished by Grafana Labs

Skill
Review and audit
Cloud and DevOps
Apache-2.0
Low risk

From the author

Diagnostic guide for active Prometheus cardinality problems — slow queries, OOMing Prometheus, high Grafana Cloud Active Series or DPM bills, "too many samples" ingest errors, series churn, or rapid memory growth. Walks through tsdb status endpoints, per-metric and per-label drill-downs, common-culprit galleries, and remediation paths. Use when the user is *currently experiencing* a cardinality fire. For preventing cardinality issues at the source, route to prometheus-label-strategy. For post-ingest aggregation, route to adaptive-metrics. For DPM-specific analysis, route to dpm-finder.

— the description by Grafana Labs

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a systematic, operational framework for triaging Prometheus cardinality fires, such as OOM events, high memory growth, or unexpected cost spikes. It emphasizes safe remediation paths, prioritizing the preservation of data integrity by avoiding common pitfalls like improper label dropping. The guide walks users through TSDB status analysis, metric drill-downs, and identifying churn, while providing clear decision trees for when to drop metrics, fix source code, or utilize post-ingest aggregation.

What it does

Diagnoses active Prometheus cardinality issues causing OOMs or performance degradation.
Provides PromQL and API commands to identify high-cardinality metrics and labels.
Explains the risks of improper label dropping and how it corrupts time-series data.
Offers a decision tree for remediation, from emergency metric dropping to long-term source code fixes.
Includes specific patterns for handling histogram blowups and kube-state-metrics misconfigurations.
Distinguishes between immediate emergency actions and structural architectural improvements.

When to use it

When Prometheus is experiencing OOMKilled errors or linear memory growth.
When Grafana Cloud or Mimir reports 'too many samples' or unexpected DPM/Active Series spikes.
When a specific PromQL query is causing performance degradation or OOMs.
When series churn is causing rapid growth in active series counts.
When you need to identify which specific metric or label is causing a cardinality fire.

Advantages

Prevents data corruption by warning against unsafe label-dropping practices.
Provides actionable, copy-paste ready diagnostic commands for TSDB endpoints.
Helps distinguish between transient churn and permanent cardinality bloat.
Offers clear guidance on when to hand off to specialized skills like adaptive-metrics.
Focuses on rapid triage to stop production fires while maintaining data accuracy.

Usage examples

“My Prometheus is OOMing, how do I find the high cardinality metrics?”: The agent will guide you to use the TSDB status endpoint and provide PromQL queries to identify top metrics by series count and label value distribution.
“I have a cardinality fire, can I just drop the pod label to fix it?”: The agent will explain why dropping a distinguishing label like 'pod' breaks data integrity and will instead suggest dropping the entire metric or using Adaptive Metrics.
“How do I identify if my cardinality issue is caused by churn?”: The agent will provide PromQL queries to compare series creation versus removal rates and help you correlate spikes with deployment events.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/grafana/prometheus-cardinality-troubleshooter/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip prometheus-cardinality-troubleshooter.zip -d .agents/skills

Documentation by Grafana Labs

Prometheus Cardinality Troubleshooter

You are an expert in diagnosing live Prometheus cardinality problems. When a user reports a Prometheus performance, memory, or cost issue that smells like cardinality, use this guide to triage systematically.

This skill is diagnostic and operational. For schema design and prevention, route to prometheus-label-strategy.

Before You Remediate: The One Rule

Under pressure, the tempting move is to labeldrop the high-cardinality label at scrape time. Do not. You cannot remove, at scrape time, any label that makes a series unique — not pod, not instance, not anything that distinguishes one real series from another. It looks like it stops the bleeding; it actually breaks the data:

Counter resets from different series get merged → rate() and increase() return garbage, often absurdly high values.
Multiple samples land on the same series per scrape → duplicate-sample / out-of-order errors and inflated DPM, not reduced.
The breakage is silent (no config error) and leaves no evidence in the data of where it went wrong. Weeks later someone asks "why is my DPM so high / why is rate() absurd?" and there's nothing to point to.

The only safe remediations are:

Drop an entire unwanted metric (action: drop on __name__) — you're discarding the whole metric, not merging distinct series.
Fix the source — stop the application emitting the bad label (the real fix for unbounded path, user_id, etc.).
Adaptive Metrics — for structural cardinality on series you can't fix at the source. It aggregates correctly (counter-reset-aware, audited, reversible). This is the right way to reduce the cost of a label like pod. Route to adaptive-metrics.

Everywhere below that says "drop a label," read it through this rule: drop whole metrics, fix the source, or use Adaptive Metrics — never labeldrop a distinguishing label.

Symptom → Likely Cause

Symptom	Likely Cause	First Action
Prometheus OOMKilled or memory growing linearly	Active series growth (often from a new bad metric or label)	Active Series triage
Single PromQL query slow or OOMs the querier	One or more metrics in the query have high cardinality	Per-query drill-down
Remote write lagging, WAL growing	Sample throughput spike — series count OR scrape interval changed	Active Series triage + check scrape intervals
`429 Too Many Samples` / `out of bounds` errors	Hitting Mimir/Cortex ingester per-tenant series limit	Per-metric drill-down, find the new offender
Grafana Cloud Active Series bill spiked	New metric, new label, or rollout creating churn	Per-metric drill-down + churn check
Grafana Cloud DPM bill spiked but Active Series flat	Scrape interval shortened, OR remote_write sending duplicates	DPM-side issue — route to `dpm-finder`
`series_limit_per_user` errors after a deploy	Application change introduced a new bad label	Recent change diff
Series count grows then resets every restart	Series churn from ephemeral label values	Churn diagnosis

Step 1: Active Series Triage

Get the headline number

# Total active series in the local Prometheus
prometheus_tsdb_head_series

# Or for Mimir / Grafana Cloud Metrics (per tenant)
cortex_ingester_memory_series{user="<tenant>"}

Compare to recent history:

# Growth over the last 7 days
deriv(prometheus_tsdb_head_series[7d]) * 86400

A growth rate > a few % per day on a stable application set is a red flag.

Use the TSDB status endpoint

Prometheus exposes a built-in cardinality breakdown:

curl -s http://prometheus:9090/api/v1/status/tsdb | jq

Returns:

seriesCountByMetricName — top metrics by series count
labelValueCountByLabelName — top labels by unique value count
memoryInBytesByLabelName — top labels by memory footprint
seriesCountByLabelValuePair — top label-value pairs by series count

This is usually the fastest path to "which metric / which label is the problem."

For Grafana Cloud:

# Same endpoint, authenticated against the per-tenant Mimir
curl -s -u "<user>:<token>" \
  "https://prometheus-prod-XX.grafana.net/api/prom/api/v1/status/tsdb" | jq

Step 2: Read the Output

Top metrics by series count

"seriesCountByMetricName": [
  { "name": "http_request_duration_seconds_bucket", "value": 184320 },
  { "name": "go_gc_duration_seconds",               "value": 80 },
  ...
]

Heuristics:

A histogram (_bucket) at the top is almost always the answer — those have a 14× multiplier (bucket count + 3). The fix is usually reducing the labels on the underlying histogram at the source (in instrumentation code), not stripping them at scrape and not touching the buckets themselves.
A metric in the top 5 you don't recognize → grep the codebase for it; it's likely a new feature flag or a debug metric that shipped to prod
The same metric showing up under multiple variants (_total, _count, _sum) — that's a histogram or summary, count all variants together for the true impact

Top labels by unique value count

"labelValueCountByLabelName": [
  { "name": "url",       "value": 84210 },
  { "name": "trace_id",  "value": 41000 },
  { "name": "pod",       "value": 1820 }
]

Red flags:

Any label with >10K unique values is almost certainly a bug. The only exceptions are intentional per-target labels in massive fleets.
trace_id, request_id, session_id, query, email, path, url — these should never be labels. They belong in exemplars, logs, or traces.
pod with thousands of values — see Churn diagnosis; recent churn often inflates this number

Step 3: Per-Metric Drill-Down

Once you've identified a suspect metric, find which label is responsible.

Count distinct label values per label, for one metric

# How many unique values does each label have on this metric?
count by (__name__) (
  count by (__name__, label_name_here) (
    http_request_duration_seconds_bucket
  )
)

Repeat per label, or use the helper:

# Via the Prometheus HTTP API
curl -s "http://prometheus:9090/api/v1/labels?match[]=http_request_duration_seconds_bucket" | jq -r '.data[]' | \
  while read label; do
    count=$(curl -s "http://prometheus:9090/api/v1/label/${label}/values?match[]=http_request_duration_seconds_bucket" | jq '.data | length')
    echo "${count}  ${label}"
  done | sort -rn | head -20

Find the top label values for one label

# Top 20 path values for http_requests_total
topk(20,
  count by (path) (http_requests_total)
)

If you see UUIDs, hashes, timestamps, or numeric IDs in the top values → that label has unbounded values from the source.

Per-metric series count, grouped

# Series-per-instance breakdown — if uneven, one instance is misbehaving
sum by (job, instance) ({__name__=~"my_metric.*"})

Step 4: Recent Change Diff

If the cardinality fire started recently, the cause is almost always a recent change. Diff what's there now against what was there before.

List of metrics, current vs. yesterday

Via Grafana Cloud cardinality dashboard, or:

# Current metrics
group by (__name__) ({__name__!=""})

# Compare to last week (offset)
group by (__name__) ({__name__!=""} offset 7d)

Diff externally. A new metric near the top of seriesCountByMetricName that wasn't there a week ago → that's your offender.

Correlate with deploys

# Active series correlated with build_info
prometheus_tsdb_head_series
# Overlay with:
changes(app_build_info[1d])

A vertical step in series count aligned with a deploy is conclusive.

Step 5: Churn Diagnosis

High churn means series are being created and abandoned faster than they age out. Symptoms: series count keeps climbing, then drops sharply on Prometheus restart.

Churn signal

# Series created vs. removed per second
rate(prometheus_tsdb_head_series_created_total[5m])
rate(prometheus_tsdb_head_series_removed_total[5m])

# Ratio of churned to live
prometheus_tsdb_head_series_created_total / prometheus_tsdb_head_series

A creation rate that materially exceeds the removal rate, sustained, means cardinality is on a one-way trip up. Common causes:

Cause	Tell
Pod rollouts emitting `pod` label	Churn spike aligns with deploy timing; affects pod-discovered scrapes
`version` / `git_sha` / `image_tag` label on every metric	Churn spike on every deploy across many metrics
Ephemeral hostnames in `instance`	Cloud autoscaling event timing
Bug: dynamic label names	Churn climbs forever, never plateaus
Application bug emitting fresh UUIDs as labels	Linear unbounded growth, no deploy correlation

Memory impact of churn

# A churn-driven head block carries old series until tsdb compaction
prometheus_tsdb_head_chunks
go_memstats_heap_inuse_bytes{job="prometheus"}

Restarting Prometheus drops churned series but is not a fix. The fix is at the source.

Common-Culprit Gallery

Histogram blowup

Tell: *_bucket metric at the top of seriesCountByMetricName. Multiplier ≈ 14×.

Fix:

First, reduce labels on the histogram at the source — every label removed saves 14× series. Trim path, method, or status_code in the instrumentation code (don't labeldrop them at scrape — that merges distinct histograms and corrupts the buckets). For series already in Grafana Cloud you can't change, aggregate them with Adaptive Metrics.
Then, reduce bucket count if appropriate (custom buckets vs. defaults).
For high-resolution latency tracking, consider native histograms (Prometheus 2.40+) — single sparse series replaces the bucket family.

kube-state-metrics label explosion

Tell: kube_pod_labels or kube_pod_annotations at the top, with label_* or annotation_* labels driving cardinality.

Fix: configure kube-state-metrics with --metric-labels-allowlist and --metric-annotations-allowlist. By default it emits all labels and annotations as series.

# kube-state-metrics flags
--metric-labels-allowlist=pods=[app,team,version]
--metric-annotations-allowlist=pods=[checksum/config]

Path / route blowup from a new endpoint

Tell: http_requests_total (or framework equivalent) grew 10×+ overnight. topk(20, count by (path) (http_requests_total)) shows hundreds of /users/123456-style values.

Fix: the real fix is to template the path in application code (/users/:id) — route the user to prometheus-label-strategy. For series already in Grafana Cloud, Adaptive Metrics can aggregate path away correctly — route to adaptive-metrics.

Do not "normalize" path with a relabel replacement rule — collapsing /users/123, /users/456, … into one /users/:id value at scrape merges distinct series and produces duplicate-sample errors and broken rate(). The merge has to happen at the source (templating) or post-ingest (Adaptive Metrics), never at scrape.

If you must stop a production fire right now and templating isn't deployable yet, the only safe scrape-time action is to drop the entire offending metric (you lose it completely until the code fix lands — a deliberate trade, not a silent corruption):

# Emergency: drop the whole metric until the source is templated
metric_relabel_configs:
  - source_labels: [__name__]
    regex: http_requests_total
    action: drop

Application emitting a debug metric in prod

Tell: A metric you don't recognize in the top 10. Grep the source — often a _details or _per_request debug metric the developer forgot to gate.

Fix: drop entirely at scrape:

metric_relabel_configs:
  - source_labels: [__name__]
    regex: my_app_request_details
    action: drop

Open a ticket against the team to remove it from the code.

App-emitted labels colliding with target labels

Tell: Series count for one job is several × what it should be. Looking at one series, you see both an app-emitted instance=... AND the target instance=... collided into something weird (Prometheus renames the conflicting one to exported_instance).

Fix: the right fix is in the application — stop emitting instance/node/host from code; they belong to the scrape target. Confirm honor_labels is false (the default) so the target labels win.

If you need a scrape-time stopgap, you may remove a label only where it exactly duplicates a target label — that's the one safe labeldrop, because the target label still provides uniqueness. Scope it tightly to the duplicated names and never include pod (or any other label that is the source of uniqueness):

# Stopgap ONLY for app-emitted duplicates of target labels.
# Drops the `exported_*` collisions — NOT pod, which makes K8s series unique.
metric_relabel_configs:
  - regex: exported_(instance|node|host)
    action: labeldrop

Then the target labels from relabel_configs apply cleanly. Prefer fixing the app.

Federation amplifying cardinality

Tell: A federated Prometheus or Mimir global view has way more series than expected. Each source has its own cluster / region label, multiplying.

Fix: this is usually expected — federation by design preserves source labels. If the series count is too high, federate only aggregated recording rules, not raw metrics:

- job_name: federate
  honor_labels: true
  metrics_path: /federate
  params:
    'match[]':
      - '{__name__=~".*:.*"}'  # Recording-rule naming convention only

Remediation Decision Tree

Cardinality fire confirmed
│
├── Need to stop the bleeding NOW (production OOM, ingest 429s)
│   └── Drop the ENTIRE offending metric via metric_relabel_configs (action: drop on __name__)
│       (also applies to Alloy/Agent — same syntax)
│       Do NOT labeldrop a distinguishing label — it breaks the data, see "The One Rule".
│       Then schedule the proper fix.
│
├── It's a Grafana Cloud Active Series bill issue, not a perf issue
│   ├── Cardinality is structural and you can't fix the app
│   │   └── Route to `adaptive-metrics` skill (post-ingest aggregation rules — the safe way)
│   └── You want metric-by-metric DPM breakdown
│       └── Route to `dpm-finder` skill
│
├── It's a fixable application bug (unbounded label, debug metric in prod)
│   ├── Short-term: drop the whole metric at scrape, OR aggregate via Adaptive Metrics
│   └── Long-term: fix in code; route to `prometheus-label-strategy` for design guidance
│
├── It's histogram cardinality
│   ├── Reduce labels on the underlying histogram AT THE SOURCE (14× win per label)
│   ├── Reduce bucket count if appropriate
│   └── Consider native histograms for high-resolution latency
│
└── It's churn (deploy-driven)
    ├── Stop EMITTING `version`/`git_sha`/`instance` from app code (use info-metric for version)
    ├── Keep `pod` — never drop it; if pod-level series are too costly, use Adaptive Metrics
    └── Verify K8s SD relabel rules aren't mapping in `uid` or other ephemeral fields

Emergency Drop Patterns (copy-paste ready)

These are the safe scrape-time emergency actions: dropping an entire unwanted metric. They do not merge distinct series, so they don't corrupt the data.

⚠️ There is intentionally no labeldrop of a distinguishing label and no value-normalizing relabel here. Both merge distinct series and break rate()/DPM (see The One Rule). To reduce cardinality without dropping the whole metric, fix the source or use Adaptive Metrics (route to adaptive-metrics). The only safe labeldrop is removing a label that exactly duplicates a target label (e.g. exported_instance) — see App-emitted labels colliding with target labels.

For Prometheus scrape_configs:

metric_relabel_configs:
  # Drop a specific bad metric entirely
  - source_labels: [__name__]
    regex: bad_metric_name
    action: drop

  # Drop a set of debug/temporary metrics by name prefix
  - source_labels: [__name__]
    regex: debug_.*
    action: drop

For Grafana Alloy (prometheus.relabel component):

prometheus.relabel "drop_bad_metric" {
  forward_to = [prometheus.remote_write.default.receiver]

  rule {
    source_labels = ["__name__"]
    regex = "bad_metric_name"
    action = "drop"
  }
}

Always test in staging first, and prefer fixing the source or using Adaptive Metrics over any scrape-time drop.

When to Hand Off

"Now design a label strategy so this doesn't happen again" → prometheus-label-strategy
"We need to keep these metrics but reduce cost" → adaptive-metrics
"Which metric is the most expensive in DPM terms?" → dpm-finder
"Write the PromQL to find this" → promql
"Configure this in Alloy" → alloy
"Why is my Loki slow?" → loki-label-analyzer (different system, same family of problems)

This skill's lane is diagnosis under pressure. Prevention, design, and post-ingest cost optimization live elsewhere.

Files

ATTRIBUTION.mdAdded by AutoGineer895 B
LICENSELicence file added by AutoGineer11.1 KiB
SKILL.md18.1 KiB

LICENSE

                                 Apache License
                           Version 2.0, January 2004
                        http://www.apache.org/licenses/

   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

   1. Definitions.

      "License" shall mean the terms and conditions for use, reproduction,
      and distribution as defined by Sections 1 through 9 of this document.

      "Licensor" shall mean the copyright owner or entity authorized by
      the copyright owner that is granting the License.

      "Legal Entity" shall mean the union of the acting entity and all
      other entities that control, are controlled by, or are under common
      control with that entity. For the purposes of this definition,
      "control" means (i) the power, direct or indirect, to cause the
      direction or management of such entity, whether by contract or
      otherwise, or (ii) ownership of fifty percent (50%) or more of the
      outstanding shares, or (iii) beneficial ownership of such entity.

      "You" (or "Your") shall mean an individual or Legal Entity
      exercising permissions granted by this License.

      "Source" form shall mean the preferred form for making modifications,
      including but not limited to software source code, documentation
      source, and configuration files.

      "Object" form shall mean any form resulting from mechanical
      transformation or translation of a Source form, including but
      not limited to compiled object code, generated documentation,
      and conversions to other media types.

      "Work" shall mean the work of authorship, whether in Source or
      Object form, made available under the License, as indicated by a
      copyright notice that is included in or attached to the work
      (an example is provided in the Appendix below).

      "Derivative Works" shall mean any work, whether in Source or Object
      form, that is based on (or derived from) the Work and for which the
      editorial revisions, annotations, elaborations, or other modifications
      represent, as a whole, an original work of authorship. For the purposes
      of this License, Derivative Works shall not include works that remain
      separable from, or merely link (or bind by name) to the interfaces of,
      the Work and Derivative Works thereof.

      "Contribution" shall mean any work of authorship, including
      the original version of the Work and any modifications or additions
      to that Work or Derivative Works thereof, that is intentionally
      submitted to Licensor for inclusion in the Work by the copyright owner
      or by an individual or Legal Entity authorized to submit on behalf of
      the copyright owner. For the purposes of this definition, "submitted"
      means any form of electronic, verbal, or written communication sent
      to the Licensor or its representatives, including but not limited to
      communication on electronic mailing lists, source code control systems,
      and issue tracking systems that are managed by, or on behalf of, the
      Licensor for the purpose of discussing and improving the Work, but
      excluding communication that is conspicuously marked or otherwise
      designated in writing by the copyright owner as "Not a Contribution."

      "Contributor" shall mean Licensor and any individual or Legal Entity
      on behalf of whom a Contribution has been received by Licensor and
      subsequently incorporated within the Work.

   2. Grant of Copyright License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      copyright license to reproduce, prepare Derivative Works of,
      publicly display, publicly perform, sublicense, and distribute the
      Work and such Derivative Works in Source or Object form.

   3. Grant of Patent License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      (except as stated in this section) patent license to make, have made,
      use, offer to sell, sell, import, and otherwise transfer the Work,
      where such license applies only to those patent claims licensable
      by such Contributor that are necessarily infringed by their
      Contribution(s) alone or by combination of their Contribution(s)
      with the Work to which such Contribution(s) was submitted. If You
      institute patent litigation against any entity (including a
      cross-claim or counterclaim in a lawsuit) alleging that the Work
      or a Contribution incorporated within the Work constitutes direct
      or contributory patent infringement, then any patent licenses
      granted to You under this License for that Work shall terminate
      as of the date such litigation is filed.

   4. Redistribution. You may reproduce and distribute copies of the
      Work or Derivative Works thereof in any medium, with or without
      modifications, and in Source or Object form, provided that You
      meet the following conditions:

      (a) You must give any other recipients of the Work or
          Derivative Works a copy of this License; and

      (b) You must cause any modified files to carry prominent notices
          stating that You changed the files; and

      (c) You must retain, in the Source form of any Derivative Works
          that You distribute, all copyright, patent, trademark, and
          attribution notices from the Source form of the Work,
          excluding those notices that do not pertain to any part of
          the Derivative Works; and

      (d) If the Work includes a "NOTICE" text file as part of its
          distribution, then any Derivative Works that You distribute must
          include a readable copy of the attribution notices contained
          within such NOTICE file, excluding those notices that do not
          pertain to any part of the Derivative Works, in at least one
          of the following places: within a NOTICE text file distributed
          as part of the Derivative Works; within the Source form or
          documentation, if provided along with the Derivative Works; or,
          within a display generated by the Derivative Works, if and
          wherever such third-party notices normally appear. The contents
          of the NOTICE file are for informational purposes only and
          do not modify the License. You may add Your own attribution
          notices within Derivative Works that You distribute, alongside
          or as an addendum to the NOTICE text from the Work, provided
          that such additional attribution notices cannot be construed
          as modifying the License.

      You may add Your own copyright statement to Your modifications and
      may provide additional or different license terms and conditions
      for use, reproduction, or distribution of Your modifications, or
      for any such Derivative Works as a whole, provided Your use,
      reproduction, and distribution of the Work otherwise complies with
      the conditions stated in this License.

   5. Submission of Contributions. Unless You explicitly state otherwise,
      any Contribution intentionally submitted for inclusion in the Work
      by You to the Licensor shall be under the terms and conditions of
      this License, without any additional terms or conditions.
      Notwithstanding the above, nothing herein shall supersede or modify
      the terms of any separate license agreement you may have executed
      with Licensor regarding such Contributions.

   6. Trademarks. This License does not grant permission to use the trade
      names, trademarks, service marks, or product names of the Licensor,
      except as required for reasonable and customary use in describing the
      origin of the Work and reproducing the content of the NOTICE file.

   7. Disclaimer of Warranty. Unless required by applicable law or
      agreed to in writing, Licensor provides the Work (and each
      Contributor provides its Contributions) on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
      implied, including, without limitation, any warranties or conditions
      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
      PARTICULAR PURPOSE. You are solely responsible for determining the
      appropriateness of using or redistributing the Work and assume any
      risks associated with Your exercise of permissions under this License.

   8. Limitation of Liability. In no event and under no legal theory,
      whether in tort (including negligence), contract, or otherwise,
      unless required by applicable law (such as deliberate and grossly
      negligent acts) or agreed to in writing, shall any Contributor be
      liable to You for damages, including any direct, indirect, special,
      incidental, or consequential damages of any character arising as a
      result of this License or out of the use or inability to use the
      Work (including but not limited to damages for loss of goodwill,
      work stoppage, computer failure or malfunction, or any and all
      other commercial damages or losses), even if such Contributor
      has been advised of the possibility of such damages.

   9. Accepting Warranty or Additional Liability. While redistributing
      the Work or Derivative Works thereof, You may choose to offer,
      and charge a fee for, acceptance of support, warranty, indemnity,
      or other liability obligations and/or rights consistent with this
      License. However, in accepting such obligations, You may act only
      on Your own behalf and on Your sole responsibility, not on behalf
      of any other Contributor, and only if You agree to indemnify,
      defend, and hold each Contributor harmless for any liability
      incurred by, or claims asserted against, such Contributor by reason
      of your accepting any such warranty or additional liability.

   END OF TERMS AND CONDITIONS

   APPENDIX: How to apply the Apache License to your work.

      To apply the Apache License to your work, attach the following
      boilerplate notice, with the fields enclosed by brackets "[]"
      replaced with your own identifying information. (Don't include
      the brackets!)  The text should be enclosed in the appropriate
      comment syntax for the file format. We also recommend that a
      file or class name and description of purpose be included on the
      same "printed page" as the copyright notice for easier
      identification within third-party archives.

   Copyright 2026 Grafana Labs

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs network access, Needs Node.js

Security

What our scanner noticed· Low risk

Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

Grafana Labs

Publisher

Grafana Labs (Official)

Source

grafana/skills · skills/grafana-cloud/prometheus-cardinality-troubleshooter · commit 8fb72a5

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by Grafana Labs.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download prometheus-cardinality-troubleshooter.zip

From the author

AutoGineer summary

What it does

When to use it

Advantages

Usage examples

Install

Documentation by Grafana Labs

Prometheus Cardinality Troubleshooter

Before You Remediate: The One Rule

Symptom → Likely Cause

Step 1: Active Series Triage

Get the headline number

Use the TSDB status endpoint

Step 2: Read the Output

Top metrics by series count

Top labels by unique value count

Step 3: Per-Metric Drill-Down

Count distinct label values per label, for one metric

Find the top label values for one label

Per-metric series count, grouped

Step 4: Recent Change Diff

List of metrics, current vs. yesterday

Correlate with deploys

Step 5: Churn Diagnosis

Churn signal

Memory impact of churn

Common-Culprit Gallery

Histogram blowup

kube-state-metrics label explosion

Path / route blowup from a new endpoint

Application emitting a debug metric in prod

App-emitted labels colliding with target labels

Federation amplifying cardinality

Remediation Decision Tree

Emergency Drop Patterns (copy-paste ready)

When to Hand Off

Files

LICENSE

Requirements and compatibility

Security

Attribution and licence

Related skills

Spark Memory Thermal Ops

Observability And Instrumentation

Gke Alert Configuration

Application Performance Observability Engineer

Distributed Debugging Devops Troubleshooter

Distributed Tracing

Download ZIP