Cloud Monitoring Alert Configuration

Select GKE Metrics and Kubernetes Alerting Policies

This catalog compiles public Google Kubernetes Engine (GKE) telemetry metrics from Google Cloud Kubernetes Metrics and critical alerting rules from the open-source Awesome Prometheus Alerts for Kubernetes catalog.

It serves as the reference catalog for the cloud-monitoring-alert-configuration skill when generating evaluation suites and production alerting policies in Google Cloud Managed Service for Prometheus (GMP) and Cloud Monitoring.

Table of Contents

• Enforce Kubernetes and GKE Telemetry Guardrails: Lines 37-138

  • Classify Telemetry Tiers and Optimize Ingestion Costs: Lines 102-138

  • Filter Tier 2 Metrics with a PodMonitoring Allowlist: Lines 112-138

• Query Native GKE System Metrics: Lines 139-189

  • Monitor Container and Pod Core Metrics: Lines 144-163

  • Monitor Node and Infrastructure Core Metrics: Lines 164-177

  • Monitor Autoscaling and Hardware Accelerator Metrics: Lines 178-189

• Map Open Source Kubernetes Alert Rules: Lines 190-241

  • Configure Node Health and Capacity Alerts: Lines 194-204

  • Configure Workload and Container Reliability Alerts: Lines 205-219

  • Configure Job CronJob and Storage Alerts: Lines 220-230

  • Configure API Server and Control Plane Alerts: Lines 231-241

• Implement Non KSM Native Alternative Alerts: Lines 242-350

  • Alert on Container Memory Saturation and OOM Risk: Lines 248-277

  • Alert on Pod CrashLoopBackOff and Container Restarts: Lines 278-297

  • Alert on Node Readiness and Health: Lines 298-307

  • Alert on Multi Window Multi Burn Rate SLO Error Rates: Lines 308-340

  • Detect Traffic Outages: Lines 341-350

Enforce Kubernetes and GKE Telemetry Guardrails

For all Kubernetes and GKE requests, read the references/kubernetes_*.md files and respond directly. Do not search the workspace for additional files before answering. Do NOT search the codebase for external Terraform modules or extra reference files, and do NOT write .tf files to disk unless the user explicitly specifies a target file path.

1. Mandatory KSM Permission Guardrail (Tier 2 kube_* metrics): In Google Cloud Managed Service for Prometheus (GMP), billing is based on samples ingested. Deploying open-source kube-state-metrics (KSM) unmodified scrapes state for every object across the cluster, generating high time-series volume and significant ingestion cost. If the requested alert uses KSM metrics (kube_deployment_*, kube_statefulset_*, kube_daemonset_*, kube_cronjob_*, kube_job_*, kube_persistentvolume_*, kube_pod_status_phase, kube_pod_container_resource_limits), you MUST NOT assume KSM is enabled or write or apply any Terraform files. At the very top of your response, you MUST:

   • (a) Alert the user that the policy requires kube-state-metrics (KSM).

   • (b) Explain the GMP cost impact: Explain that KSM incurs billable Managed Service for Prometheus (GMP) sample ingestion costs (see Cloud Monitoring pricing).

   • (c) Ask a direct question in second-person for explicit permission (literally include this exact question in bold): "Do you grant explicit permission to enable and ingest kube-state-metrics (KSM) in your cluster for this alert?" (Critical: Never phrase this as a passive statement such as "ensure you have approved KSM" or "before enabling KSM, review below" — always ask the user directly with a question mark!)

   • (d) Recommend a filtered PodMonitoring allowlist (metricRelabeling with action: keep) that keeps only the required metrics (along with any conditional draft HCL clearly marked as pending approval), and offer Non-KSM Native Alternatives where possible.

2. Tier 1 Native Metrics (Zero KSM Surcharge): For native cAdvisor, Kubelet, or Control Plane metrics (container_memory_working_set_bytes, container_spec_memory_limit_bytes, kube_pod_container_status_restarts_total, kubernetes.io/container/restart_count, kube_node_status_condition, kubelet_volume_stats_available_bytes, apiserver_*, rest_client_*), explicitly state that the metric is a Tier 1 native/standard metric with zero KSM cost surcharge (a Tier 1 Kubelet/cAdvisor metric that does not require kube-state-metrics or a Tier 1 Control Plane metric with zero KSM cost surcharge), explain that container resource limits (resources.limits) must be defined in pod specs for saturation alerts to resolve, and provide a clear plain-English explanation of the alert and thresholds.

3. Mandatory Terraform Variables (project_id, cluster_name, namespace) and user_labels: In EVERY Kubernetes Terraform HCL configuration (including multi-service latency and SLO burn-rate alerts), you MUST define variable "project_id", variable "cluster_name", and variable "namespace" blocks, interpolate all three (var.project_id, ${var.cluster_name}, ${var.namespace}), and include user_labels = { "created-with-google-skill" = "cloud-monitoring-alert-configuration" } in every google_monitoring_alert_policy resource.

flowchart TD
    A["User Request for Alert Policy"] --> B{"Can alert use Native GKE or cAdvisor?"}
    B -->|Yes - Tier 1| C["Use Native GKE or cAdvisor Metric (Zero KSM Surcharge)"]
    B -->|No - Requires KSM| D["Alert User on KSM Cost and Ask Explicit Permission Question"]
    D --> E{"User Approved KSM?"}
    E -->|No or Pending| F["Do NOT Write Files to Disk; Show Filtered PodMonitoring Allowlist and Conditional Preview Only"]
    E -->|Yes - Tier 2| G["Generate KSM Alert Policy and Recommend Filtered PodMonitoring Allowlist"]

Classify Telemetry Tiers and Optimize Ingestion Costs

• Tier 1: Native GKE Built-ins and cAdvisor (Zero KSM Cost): Metrics collected natively by GKE, Kubelet/cAdvisor, or Control Plane without kube-state-metrics (zero KSM cost surcharge).

• Tier 2: Filtered KSM (Requires Explicit Consent Question and Allowlist): Rules requiring Kubernetes control-plane state (such as Deployment, StatefulSet, DaemonSet, CronJob, Job, PersistentVolume, or Pod phase metrics) that require kube-state-metrics.

Filter Tier 2 Metrics with a PodMonitoring Allowlist

When a Tier 2 (KSM) alert is requested, ALWAYS ask the user for explicit permission first and provide a filtered PodMonitoring resource (to be used if KSM is approved) that scrapes only the required metrics and drops all others:

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: kube-state-metrics-filtered
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
  endpoints:
  - port: http-metrics
    interval: 30s
    metricRelabeling:
    - action: keep
      sourceLabels: [__name__]
      regex: "^(kube_deployment_spec_replicas|kube_deployment_status_replicas_available|kube_statefulset_status_replicas_ready|kube_statefulset_status_replicas)$"

Query Native GKE System Metrics

Below are the public native GKE telemetry metrics (kubernetes.io/) available in Cloud Monitoring.

Monitor Container and Pod Core Metrics

Monitor Node and Infrastructure Core Metrics

Monitor Autoscaling and Hardware Accelerator Metrics

Map Open Source Kubernetes Alert Rules

Compiled from the open-source awesome-prometheus-alerts community catalog.

Configure Node Health and Capacity Alerts

Configure Workload and Container Reliability Alerts

Configure Job CronJob and Storage Alerts

Configure API Server and Control Plane Alerts

Implement Non KSM Native Alternative Alerts

When a user requests open-source catalog alerts that normally rely on kube-state-metrics, first offer these Tier 1 (Non-KSM) equivalents to avoid customer ingestion costs:

Alert on Container Memory Saturation and OOM Risk

Instead of container_memory_working_set_bytes / kube_pod_container_resource_limits, use native cAdvisor limits:

sum(
  container_memory_working_set_bytes{
    cluster="${var.cluster_name}",
    namespace="${var.namespace}",
    container!=""
  }
) by (pod, container)
/
sum(
  container_spec_memory_limit_bytes{
    cluster="${var.cluster_name}",
    namespace="${var.namespace}",
    container!=""
  }
) > 0.90

[!NOTE] Memory Compared to CPU Saturation: In Kubernetes, CPU is compressible and throttled by the CFS scheduler without killing containers. Memory is uncompressible and triggers fatal OOMKilled events. Therefore, cluster saturation alerting focuses on Memory Saturation, and CPU saturation alerts (and container_cpu_usage_seconds_total) are excluded from the recommended cluster alert suite.

Alert on Pod CrashLoopBackOff and Container Restarts

Use native GKE Cloud Monitoring metric kubernetes.io/container/restart_count or the cAdvisor restart counter:

sum(
  increase(
    kube_pod_container_status_restarts_total{
      cluster="${var.cluster_name}",
      namespace="${var.namespace}"
    }[15m]
  )
) by (pod, container) > 3

• Duration Rule: Set Terraform duration = "0s" (or "60s"). The [15m] lookback window already smooths transient spikes; adding duration = "300s" delays critical crashloop alerts by 20 minutes total.

Alert on Node Readiness and Health

kube_node_status_condition{
  cluster="${var.cluster_name}",
  condition="Ready",
  status="true"
} == 0

Alert on Multi Window Multi Burn Rate SLO Error Rates

For service error rate monitoring, use multi-window burn rate alerts instead of simple static ratios:

[!IMPORTANT] Service availability and latency SLOs are owned by the google-cloud-slo-alert-configuration skill. Delegate to it whenever the user asks for an SLO, an error budget, or burn-rate alerting on a GKE service. The full handoff contract, the SLI source table, and the fallback variants — including the Medium (6.0, 6h/30m) and Slow (1.0, 3d/6h) burn rates — are in kubernetes_promql_queries.md. If you emit the fallback queries, your response MUST still name google-cloud-slo-alert-configuration as the owner of SLO configuration and state that what you emitted is a local fallback.

(
  (
    sum(rate(http_requests_total{cluster="${var.cluster_name}", namespace="${var.namespace}", status=~"5.."}[5m])) by (service)
    /
    sum(rate(http_requests_total{cluster="${var.cluster_name}", namespace="${var.namespace}"}[5m])) by (service)
  ) > (1 - ${var.slo_target}) * 14.4
)
and
(
  (
    sum(rate(http_requests_total{cluster="${var.cluster_name}", namespace="${var.namespace}", status=~"5.."}[1h])) by (service)
    /
    sum(rate(http_requests_total{cluster="${var.cluster_name}", namespace="${var.namespace}"}[1h])) by (service)
  ) > (1 - ${var.slo_target}) * 14.4
)

Detect Traffic Outages

Avoid brittle == 0 comparisons. Use default 0 or absent() to detect total traffic loss:

(sum(rate(http_requests_total{cluster="${var.cluster_name}", namespace="${var.namespace}"}[5m])) by (service) default 0) == 0
and
(sum(rate(http_requests_total{cluster="${var.cluster_name}", namespace="${var.namespace}"}[5m] offset 1w)) by (service)) > 1