Generated banner for Cloud Monitoring Alert Configuration
Generated logo for Cloud Monitoring Alert Configuration

Cloud Monitoring Alert Configuration

GoogleOfficialPublished by Google

  • Skill
  • Workflow
  • Cloud and DevOps
  • Apache-2.0
  • Low risk
  • Has scripts

From the author

Configures Cloud Monitoring alerting policies using Terraform and PromQL across Cloud Run, Google Kubernetes Engine (GKE), Compute Engine, AI Hypercomputer reservations, Vertex AI Agent Engine, and generic Google Cloud resources. Use when creating, reviewing, migrating, or deduplicating Terraform alerting policies for latency, traffic, error rates, saturation, host or reservation maintenance, GPU XID insights, or AI agent telemetry. Don't use for standalone PromQL query authoring without alerting, log querying, or billing cost audits.
— the description by Google

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured workflow for defining, reviewing, and managing Cloud Monitoring alerting policies. It leverages Terraform for infrastructure-as-code and PromQL for metric conditions, ensuring consistent observability across Google Cloud resources including Cloud Run, GKE, Compute Engine, and AI Agent platforms. The skill enforces best practices like the Four Golden Signals, dynamic grouping, and proper labeling, while providing clear paths for delegating SLO-specific configurations and notification channel management.

What it does

  • Generates parameterized Terraform configurations for Cloud Monitoring alert policies.
  • Implements PromQL-based conditions for high-signal monitoring.
  • Provides resource-specific guidance for GKE, Cloud Run, Compute Engine, and AI agents.
  • Enforces standardized labeling for all generated alerting resources.
  • Integrates with specialized scripts to analyze traffic patterns for AI agent telemetry.
  • Delegates SLO and notification channel configuration to dedicated skills.

When to use it

  • When creating or migrating Terraform-based alerting policies for Google Cloud infrastructure.
  • When setting up monitoring for AI agent reliability, quality, and cost telemetry.
  • When deduplicating existing alert policies to improve signal-to-noise ratios.
  • When configuring alerts for latency, traffic, error rates, or saturation across cloud services.

Advantages

  • Ensures consistent observability standards across diverse cloud environments.
  • Reduces manual configuration errors through structured Terraform templates.
  • Promotes best practices like dynamic grouping and the Four Golden Signals.
  • Maintains clear separation of concerns by delegating SLO and notification tasks.
  • Provides data-driven traffic analysis for accurate AI agent alert thresholds.

Usage examples

“Create a Terraform alerting policy for high latency on my GKE service.”
The agent will guide you through selecting the appropriate GKE PromQL queries, generate the google_monitoring_alert_policy Terraform resource, and prompt you to attach notification channels.
“Configure a cost alert for my Vertex AI Agent based on token usage.”
The agent will run the traffic analysis script to determine your workload pattern and generate a PromQL-based alert policy tailored to your agent's historical token consumption.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/google/cloud-monitoring-alert-configuration/download
  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip cloud-monitoring-alert-configuration.zip -d .agents/skills

Documentation by Google

Configure Cloud Monitoring Alerts

Configure high-signal alerting policies in Cloud Monitoring using Terraform (google_monitoring_alert_policy) and PromQL (condition_prometheus_query_language).

Usage Attribution

For all gcloud CLI, REST API, and MCP attribution requirements when executing commands with this skill, follow the instructions in Usage Attribution.

Enforce Core Alerting Rules

  1. Apply the Four Golden Signals: Cover latency, traffic, errors, and saturation for the target workload.

  2. Use Dynamic Grouping: Aggregate metrics by resource labels and parameterize thresholds with Terraform variables. Never hardcode ephemeral pod or instance identifiers.

  3. Default to PromQL: Use Prometheus Query Language (PromQL) for metric conditions, and use log-based alerts (condition_matched_log or google_logging_metric) for lifecycle and audit events.

  4. Attach Skill Attribution Labels: Always include user_labels with created-with-google-skill = "cloud-monitoring-alert-configuration" on every generated google_monitoring_alert_policy resource.

  5. Require User Confirmation Before Mutating Resources: Never execute commands that modify infrastructure, enable telemetry, create online monitors, or apply Terraform without waiting for explicit user confirmation.

  6. Prompt for Notification Channels: Alerting policies without notification channels incur charges without notifying anyone during an incident. Whenever notification channels are not specified in the user prompt across any workflow, explicitly inform the user that no notification channels are attached to the alerting policies, advise them to configure and attach notification channels before deploying, and ask at the end of your response if they want to configure notification channels.

Route by Target Resource Type

Identify the target resource type and read only the matching reference file directly. Never run directory listings or workspace file searches to look for queries or .tf files unless the prompt explicitly provides a file path. If a user asks to fix or validate a PromQL query without providing the query text or a file path, construct a representative broken-and-corrected query example directly.

Configure Cloud Run Alerts
Configure Google Kubernetes Engine Alerts
Configure AI Agent Alerts
Configure Compute Engine and AI Hypercomputer Alerts
  • Compute Engine VMs, MIGs, GPU XID Insights, Host or Reservation Maintenance (type=scheduled, unscheduled, or multiple), and Machine or Domain Health: compute_engine_alerts.md

Configure Generic Resource Alerts
Delegate Service Level Objective Alerts

Delegate to the google-cloud-slo-alert-configuration skill. That skill owns the SLO setup wizard, the burn-rate PromQL templates (references/promql_templates.md), the SLI metric mapping (references/service_metrics.md), and the App Hub scoping labels (references/app_hub_labels.md). Do NOT duplicate those templates here or claim that this skill owns its own SLO PromQL template library.

Delegate whenever the user asks for an availability or latency SLO, an error budget or burn-rate alert, or a Service Level Objective on any resource. When SLOs are only one part of a broader request (for example, a full production alerting suite for a Cloud Run service), delegate the SLO policies and continue generating the remaining golden-signal policies from the resource-specific reference in the same response.

Every resource reference carries a Context to hand off list naming the SLI metric, its good or bad filter, and the grouping labels — under a ## Delegate Service Level Objective Alerts heading for Cloud Run and generic resources, and inside the SLO section itself for GKE and Agent Platform. Read it and pass that context on, so the SLO skill does not repeat discovery you have already done. When delegating, you MUST also:

  1. State the SLO target and window explicitly if the user gave them, or let the SLO skill apply its SRE defaults (99.9% over a rolling 28 days) if they did not.

  2. Require the returned policies to carry user_labels with created-with-google-skill = "cloud-monitoring-alert-configuration" so attribution stays consistent with the rest of the generated suite. This literal value does NOT change when you delegate: the label records the entry-point skill, not whichever skill supplied the template. Do NOT set created-with-google-skill = "google-cloud-slo-alert-configuration" on policies generated through this delegation.

  3. Preserve the burn-rate duration contract, which google-cloud-slo-alert-configuration does not specify: Fast Burn (1h and 5m windows, factor 14.4) carries duration = "300s", and Slow Burn ([3d] and [6h] lookback, factor 1.0) omits the duration field entirely — do not emit duration = "0s" in its place. Cloud Monitoring rejects a non-zero retest buffer when the lookback window exceeds 25 hours.

  4. When no notification channels are provided, inform the user that no notification channels are attached to the alerting policies, advise them to configure and attach notification channels, and ask if they want to configure notification channels.

[!IMPORTANT] Always tell the user about the referral, even on the fallback path. Some reference files in this skill include standalone burn-rate templates to use when the google-cloud-slo-alert-configuration skill is not installed. If you emit policies from those fallback templates, your response MUST still tell the user that google-cloud-slo-alert-configuration owns SLO configuration, state that the emitted policies use the local fallback templates, and list the handoff context. Never silently present SLO policies as if this skill owned them.

[!NOTE] Exception: Agent Platform (OpenTelemetry-instrumented AI agents on any runtime, including Vertex AI Agent Engine and Cloud Run) is the one resource whose Fast and Slow burn policies live in this skill and remain authoritative. Delegate only for SLO customization — see Reliability.

  • Notification Channels: Delegate to cloud-monitoring-notification-channels-configuration to configure email, PagerDuty, Pub/Sub, or Slack channels. When an alerting policy has no notification channels attached, inform the user that no notification channels are attached and that they should configure notification channels and attach them to the alerting policy.

  • PromQL Authoring: Delegate to cloud-monitoring-promql-query for standalone PromQL query authoring or metric exploration.

Follow the Alert Configuration Workflow

  1. Discover: Identify target resources, metric schemas, and scoping projects using the lookup commands in the matching reference file (such as discover_metric_schema.py for Compute Engine metrics and other Google Cloud resource metrics, or gather_agent_info.py, check_telemetry.py, analyze_traffic.py, create_online_monitor.py, list_trace_scope_table_names.py, and list_log_scope_table_names.py for AI agents).

  2. Route: Load the reference file for the target resource type or delegate SLOs to google-cloud-slo-alert-configuration.

  3. Generate: Produce parameterized Terraform HCL configurations with required user_labels.

  4. Validate: When the loaded reference file requires a validation script (such as lint_promql.py, validate_config.py, lint_syntax.py, or scan_duplicates.py), run or print that exact script command. Do not list or search the scripts/ directory.

  5. Explain and Prompt: Explain each alert policy in plain English. When no notification channels are provided, tell the user that no notification channels are attached to the alerting policy, advise them to configure and attach notification channels, and ask if they want to configure notification channels.

Files

references/kubernetes_gke_configuration_prerequisites.md

Configure GKE Telemetry Prerequisites for Alerting Policies

This document details the Google Kubernetes Engine (GKE) and Google Cloud configurations required to support the 16 critical alerting policies defined in the reference catalog. It provides cluster operators with the setup commands, configuration parameters, and verification queries to ensure telemetry flows correctly for each alert.

Table of Contents

  • Review the Telemetry Ingestion Architecture: Lines 35-60

  • Verify Prerequisites by Alert: Lines 61-86

  • Configure GKE Control Plane Metrics: Lines 87-119

    • Enable Control Plane Metrics: Lines 94-109

    • Verify Control Plane Metrics: Lines 110-119

  • Configure Kube State Metrics: Lines 120-176

    • Configure KSM Scraping: Lines 136-166

    • Apply the PodMonitoring Allowlist Target: Lines 142-166

    • Verify KSM Deployment: Lines 167-176

  • Configure Kubelet and cAdvisor Metrics: Lines 177-218

    • Define Container Resource Limits for Saturation Alerts: Lines 183-218

    • Identify Containers Missing Limits: Lines 193-218

  • Configure Volume and Storage Telemetry: Lines 219-238

    • Verify CSI Driver Requirements: Lines 224-231

    • Verify Volume Telemetry: Lines 232-238

[!IMPORTANT] Mandatory Kubernetes and GKE Alerting Guardrails:

  • Direct Response: Read references/kubernetes_*.md and respond directly. Do not search the workspace for additional files before answering. Do NOT search the codebase for external Terraform modules or extra reference files, and do NOT write .tf files to disk unless the user explicitly specifies a target file path.

  • Tier 2 KSM Permission Guardrail (kube_* state metrics): For any Tier 2 kube-state-metrics (KSM) alert (kube_deployment_*, kube_statefulset_*, kube_daemonset_*, kube_cronjob_*, kube_job_*, kube_persistentvolume_*, kube_pod_status_phase, kube_pod_container_resource_limits), do NOT assume KSM is enabled or write or apply any Terraform files. At the very top of your response, you MUST: (a) alert the user that the policy requires kube-state-metrics (KSM), (b) explain that KSM incurs billable Managed Service for Prometheus (GMP) sample ingestion costs (see Cloud Monitoring pricing), (c) ask the user a direct question in second-person asking for explicit permission (literally include: "Do you grant explicit permission to enable and ingest kube-state-metrics (KSM) in your cluster for this alert?"), and (d) recommend a filtered PodMonitoring allowlist (metricRelabeling with action: keep) that keeps only the required metrics.

  • Tier 1 Native Metrics (Zero KSM Surcharge): For native cAdvisor, Kubelet, or Control Plane metrics (container_memory_working_set_bytes, container_spec_memory_limit_bytes, kube_pod_container_status_restarts_total, kubernetes.io/container/restart_count, kube_node_status_condition, kubelet_volume_stats_available_bytes, apiserver_*, rest_client_*), explicitly state that the metric is a Tier 1 native/standard metric with zero KSM cost surcharge (a Tier 1 Kubelet/cAdvisor metric that does not require kube-state-metrics or a Tier 1 Control Plane metric with zero KSM cost surcharge), explain that container resource limits (resources.limits) must be defined in pod specs for saturation alerts to resolve, and provide a clear plain-English explanation of the alert and thresholds.

  • Mandatory Terraform Variables and user_labels: Every Kubernetes Terraform HCL configuration MUST define variable "project_id", variable "cluster_name", and variable "namespace" (var.project_id, ${var.cluster_name}, ${var.namespace}) and include user_labels = { "created-with-google-skill" = "cloud-monitoring-alert-configuration" } in every google_monitoring_alert_policy resource.


Review the Telemetry Ingestion Architecture

The following diagram illustrates how metrics for each alert category are scraped from the cluster and ingested into Cloud Monitoring and Google Cloud Managed Service for Prometheus (GMP).

graph TD
    subgraph GKE_Cluster["GKE Cluster"]
        subgraph Control_Plane["Control Plane (Google Managed)"]
            APIServer["Kubernetes API Server"] -->|Exposes apiserver_* and rest_*| GMP_CP["GKE Control Plane Metrics Collector"]
        end
        subgraph Worker_Nodes["Worker Nodes"]
            subgraph Kubelet["Kubelet"]
                cAdvisor["cAdvisor"] -->|Exposes container_*| GMP_Operator["GMP Operator / PodMonitoring"]
                KubeletVol["Volume Stats"] -->|Exposes kubelet_volume_*| GMP_Operator
            end
            KSM["kube-state-metrics Pod"] -->|Exposes kube_*| GMP_Operator
        end
    end
    GMP_CP -->|Writes to| GCP_Monitoring["Cloud Monitoring / GMP"]
    GMP_Operator -->|Writes to| GCP_Monitoring

Verify Prerequisites by Alert

This table lists the GKE and Google Cloud prerequisites and configuration requirements for each of the 16 critical alerts.

Alert NameCost TierRequired MetricsGKE Cluster RequirementWorkload or Resource Requirement
Kubernetes Node not readyTier 1 (Native)kube_node_status_conditionSystem Monitoring EnabledGKE Nodes running standard Kubelet
Kubernetes Node memory pressureTier 1 (Native)kube_node_status_conditionSystem Monitoring EnabledGKE Nodes running standard Kubelet
Kubernetes Node disk pressureTier 1 (Native)kube_node_status_conditionSystem Monitoring EnabledGKE Nodes running standard Kubelet
Kubernetes Node network unavailableTier 1 (Native)kube_node_status_conditionSystem Monitoring EnabledGKE Nodes running standard Kubelet
Kubernetes Volume full in four daysTier 1 (Native)kubelet_volume_stats_available_bytesSystem Monitoring EnabledPods with attached volumes (CSI Driver supported)
Kubernetes API server errorsTier 1 (Native)apiserver_request_totalControl Plane Metrics EnabledGKE control plane active
Kubernetes API client errorsTier 1 (Native)rest_client_requests_totalControl Plane Metrics EnabledAPI Clients communicating with API Server
Kubernetes client certificate expires soonTier 1 (Native)apiserver_client_certificate_expiration_seconds_countControl Plane Metrics EnabledClient certificate authentication enabled
Kubernetes CronJob failingTier 2 (KSM)kube_cronjob_status_*, kube_cronjob_spec_*kube-state-metrics ScrapedActive CronJobs in cluster
Kubernetes PersistentVolume errorTier 2 (KSM)kube_persistentvolume_status_phasekube-state-metrics ScrapedPersistentVolume resources configured
Kubernetes StatefulSet downTier 2 (KSM)kube_statefulset_*kube-state-metrics ScrapedStatefulSets deployed
Kubernetes Pod not healthyTier 2 (KSM)kube_pod_status_phasekube-state-metrics ScrapedPods running in target namespaces
Kubernetes Deployment generation mismatchTier 2 (KSM)kube_deployment_*kube-state-metrics ScrapedDeployments deployed
Kubernetes StatefulSet generation mismatchTier 2 (KSM)kube_statefulset_*kube-state-metrics ScrapedStatefulSets deployed
Kubernetes DaemonSet misscheduledTier 2 (KSM)kube_daemonset_*kube-state-metrics ScrapedDaemonSets deployed
Kubernetes Job slow completionTier 2 (KSM)kube_job_*kube-state-metrics ScrapedJobs deployed

Configure GKE Control Plane Metrics

The control plane alerts (KubernetesAPIServerErrors, KubernetesAPIClientErrors, and KubernetesClientCertificateExpiresSoon) use a Tier 1 Control Plane metric with zero KSM cost surcharge and require API server and API client telemetry.

Enable Control Plane Metrics

By default, GKE control plane metrics are not scraped. Enable them in the GKE cluster configuration using gcloud:

gcloud container clusters update CLUSTER_NAME \
    --zone=COMPUTE_ZONE \
    --monitoring=SYSTEM,API_SERVER,CONTROLLER_MANAGER,SCHEDULER
  • API_SERVER: Exposes API server request volumes, error rates, and client request metrics.

  • CONTROLLER_MANAGER and SCHEDULER: Exposes scheduler queue metrics and controller manager execution states.

Verify Control Plane Metrics

Verify that API server telemetry flows to Cloud Monitoring:

sum(rate(apiserver_request_total[5m])) by (code)

Configure Kube State Metrics

All workload, replica mismatch, DaemonSet, StatefulSet, Job, CronJob, Pod phase, and PersistentVolume alerts require cluster state metrics (kube_* metrics) from kube-state-metrics (KSM).

[!IMPORTANT] Mandatory Explicit Permission Guardrail: Before assuming or enabling KSM ingestion, you MUST alert the user that the alert requires kube-state-metrics (KSM), explain that KSM incurs billable Managed Service for Prometheus (GMP) sample ingestion costs (see Cloud Monitoring pricing), and ask the user a direct question for explicit permission: "Do you grant explicit permission to enable and ingest kube-state-metrics (KSM) in your cluster for this alert?" Do NOT write .tf files to disk prior to receiving user permission; present the PodMonitoring allowlist and any Terraform HCL strictly as a conditional configuration to use if KSM is approved.

Configure KSM Scraping

If the user approves KSM ingestion, deploy the kube-state-metrics service in the cluster and configure a filtered PodMonitoring resource (GMP) to scrape only the required metrics.

Apply the PodMonitoring Allowlist Target

To control costs, apply a filtered PodMonitoring specification that keeps only the metrics required for these 16 critical alerts. Place this manifest in kube-system:

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: kube-state-metrics-filtered
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
  endpoints:
  - port: http-metrics
    interval: 60s
    metricRelabeling:
    - action: keep
      sourceLabels: [__name__]
      regex: "^(kube_node_status_condition|kube_pod_status_phase|kube_pod_container_status_restarts_total|kube_deployment_spec_replicas|kube_deployment_status_replicas_available|kube_statefulset_replicas|kube_statefulset_status_replicas_ready|kube_statefulset_status_replicas|kube_deployment_status_observed_generation|kube_deployment_metadata_generation|kube_statefulset_status_observed_generation|kube_statefulset_metadata_generation|kube_daemonset_status_number_misscheduled|kube_cronjob_status_last_schedule_time|kube_cronjob_status_last_successful_time|kube_cronjob_status_active|kube_cronjob_spec_suspend|kube_persistentvolume_status_phase|kube_job_spec_completions|kube_job_status_succeeded|kube_job_status_failed)$"
Verify KSM Deployment

Confirm that KSM is deployed and active:

up{job="kube-state-metrics"} == 1

Configure Kubelet and cAdvisor Metrics

System metrics such as container memory usage and CPU duty cycles are exposed natively by the Kubelet on each node as a Tier 1 Kubelet/cAdvisor metric that does not require kube-state-metrics.

Define Container Resource Limits for Saturation Alerts

Saturation alerts (such as non-KSM memory alerts) compare active usage against container resource limits (container_spec_memory_limit_bytes and container_spec_cpu_quota).

[!WARNING] The Resource Limit Gotcha: If a pod or container does not have explicit limits defined in its deployment specification, these metrics return NaN or default values, causing the alert policy to fail to resolve.

Identify Containers Missing Limits

Run the following verification query to find pods that fail to resolve in saturation alerting policies:

# Find containers missing memory limits
container_memory_working_set_bytes{container!=""}
  unless on (pod, container)
container_spec_memory_limit_bytes > 0

Ensure all workloads define a limit in their manifest:

spec:
  containers:
  - name: my-app
    resources:
      limits:
        memory: "512Mi"
        cpu: "1000m"

Configure Volume and Storage Telemetry

The volume alert (KubernetesVolumeFullInFourDays) relies on kubelet_volume_stats_available_bytes.

Verify CSI Driver Requirements
  1. Configure the storage class for the PersistentVolume (PV) to use a CSI driver that supports Volume Stats (such as the GKE Persistent Disk CSI Driver pd.csi.storage.gke.io).

  2. Local SSDs or HostPath volumes do not report volume capacity and availability statistics consistently.

Verify Volume Telemetry

Confirm volume statistics are reporting:

kubelet_volume_stats_available_bytes > 0

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Low risk

  • Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
Google
Publisher
Google (Official)
Source
google/skills · skills/cloud/cloud-monitoring-alert-configuration · commit 2194d3b
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by Google.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download cloud-monitoring-alert-configuration.zip