AutoGineer
Generated banner for Oncall Irm
Generated logo for Oncall Irm

Oncall Irm

Grafana LabsOfficialPublished by Grafana Labs

  • Skill
  • Tool integration
  • Cloud and DevOps
  • Apache-2.0
  • Low risk
Download ZIP

From the author

Route alerts, run on-call rotations, and drive incidents in Grafana IRM / OnCall — integrations (Alertmanager / Grafana Alerting / generic webhook / PagerDuty), Jinja2 routing + grouping templates, escalation chains (wait → notify schedule → notify team → webhook → auto-resolve), schedules (web + iCal + Terraform `grafana_oncall_schedule`), Slack chatops with Acknowledge/Resolve/Silence, and the P1-P4 incident lifecycle. Use when wiring Alertmanager to OnCall, deciding which team gets paged, building rotations from a Google Calendar / iCal, hooking up Slack notifications, or declaring an incident from an alert — even when the user says "page the platform team on critical alerts", "send Prometheus alerts to Slack", "set up our on-call rota", "escalation policy", or "auto-resolve when the alert clears" without naming OnCall / IRM. Heads up — OnCall OSS is in maintenance mode (archived March 2026); Grafana Cloud users should use IRM.
— the description by Grafana Labs

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides guidance for configuring Grafana IRM and OnCall to manage incident response workflows. It covers the setup of alert integrations, Jinja2-based routing templates, escalation chains, and on-call scheduling via web, iCal, or Terraform. It also details how to manage the incident lifecycle from initial alert grouping to resolution, including Slack-based ChatOps and API interactions.

What it does

  • Configures integrations for Alertmanager, Grafana Alerting, and generic webhooks.
  • Defines alert routing and grouping logic using Jinja2 templates.
  • Constructs escalation chains with wait steps, notifications, and automated actions.
  • Manages on-call schedules through calendar imports or infrastructure-as-code.
  • Enables incident management workflows including declaration, severity assignment, and status tracking.
  • Facilitates ChatOps via Slack for acknowledging, resolving, and silencing alerts.

When to use it

  • When connecting monitoring sources like Alertmanager to Grafana IRM.
  • When defining which teams receive alerts based on specific criteria.
  • When setting up automated on-call rotations from external calendars.
  • When configuring Slack notifications for incident response.
  • When automating the incident lifecycle from alert detection to resolution.

Advantages

  • Centralizes alert management and incident response in one platform.
  • Supports flexible routing logic using standard Jinja2 templates.
  • Allows for programmatic schedule management via API and Terraform.
  • Integrates incident tracking directly into communication channels like Slack.
  • Provides structured escalation paths to ensure alerts reach the correct responders.

Usage examples

“How do I route critical alerts to the platform team?”
Use a Jinja2 routing template in the integration settings to match the severity label and assign the corresponding escalation chain.
“Set up an on-call rotation from an iCal URL.”
Use the schedules API to create a new schedule, providing the iCal URL for primary and override shifts.
“How can I auto-resolve alerts when they clear in Alertmanager?”
Ensure the Alertmanager webhook configuration includes send_resolved: true and the integration is mapped to an escalation chain with an auto-resolve step.
“Declare an incident from an alert group.”
Use the IRM interface to select the alert group and trigger the Declare Incident action, or use the Slack command /incident declare.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/grafana/oncall-irm/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip oncall-irm.zip -d .agents/skills

Documentation by Grafana Labs

Grafana OnCall & IRM

OnCall docs: https://grafana.com/docs/oncall/latest/ IRM docs: https://grafana.com/docs/grafana-cloud/alerting-and-irm/

Grafana OnCall OSS is in maintenance mode (archived March 2026). Cloud users → IRM. Concepts (chains, schedules, integrations) are identical.

Prerequisites

  • Grafana Cloud stack with IRM/OnCall enabled

  • API token (Authorization: <token>)

  • Slack workspace + admin to install the OnCall app (for ChatOps)

Core concepts

ConceptDescription
IntegrationWebhook URL accepting alerts; one per source
RouteJinja2 condition that maps to an escalation chain (first True wins)
Escalation ChainWait / notify schedule / notify team / webhook / auto-resolve steps
ScheduleCalendar-based rotation (web / iCal / Terraform)
Alert GroupRelated alerts collapsed by a Grouping ID template
Notification PolicyPer-user channels — Slack, mobile push, SMS, phone, email

Flow: alert → integration → routing template → escalation chain → notifications → ack / resolve.

Common Workflows

1. Wire Alertmanager → IRM and verify routing
# 1. In IRM: New integration → Alertmanager. Copy the webhook URL.
# 2. alertmanager.yml (see references/integrations.md for full block):
receivers:
  - name: grafana-oncall
    webhook_configs:
      - url: https://<stack>.grafana.net/integrations/v1/alertmanager/<id>/
        send_resolved: true
        max_alerts: 100
# 3. Test the routing template BEFORE going live (UI: Integration → Route → "Preview")
#    Paste a sample payload (use a real Alertmanager test webhook). Expect:
#      Routing result: True
#      Escalation chain selected: <expected chain>
#    If False — your Jinja expression is wrong; fix and re-preview.

# 4. End-to-end test — fire amtool (or any test webhook) at the URL
amtool alert add foo severity=critical team=platform --alertmanager.url http://localhost:9093

# 5. Verify in IRM → Alert Groups (should appear within ~5s) — confirm:
#    - Correct route fired
#    - Correct schedule/user notified
#    - Slack message appeared in the configured channel

Routing template syntax + Jinja helpers: references/templates-schedules.md. Other integrations (Grafana Alerting, generic webhook, Slack): references/integrations.md.

2. Build an escalation chain + verify
1. Notify "Primary On-Call" (Important Notifications)
2. Wait 5 min
3. Notify "Primary On-Call" (Default Notifications)
4. Wait 10 min
5. Notify team "Platform"
6. Trigger outgoing webhook (PagerDuty / ticket)
# Verify: create a test alert (UI → Integration → "Send demo alert"),
# then watch the alert-group timeline tick through steps 1 → 6.
# Use IRM → Escalation Chains → "Test" if available, otherwise the demo alert is the canonical check.
3. Create a schedule from iCal + verify "who is on-call right now"
# 1. Create the schedule
curl -X POST https://<stack>.grafana.net/api/v1/schedules/ \
  -H "Authorization: <token>" -H "Content-Type: application/json" \
  -d '{"name":"Platform On-Call",
       "ical_url_primary":"https://calendar.example.com/platform.ics",
       "slack":{"channel_id":"C123456ABC","user_group_id":"S123456ABC"}}'

# 2. Verify the schedule was created
curl -s https://<stack>.grafana.net/api/v1/schedules/ \
  -H "Authorization: <token>" | jq '.results[] | select(.name=="Platform On-Call")'

# 3. Verify who is on-call right now
curl -s https://<stack>.grafana.net/api/v1/schedules/<schedule_id>/next_shifts/ \
  -H "Authorization: <token>" | jq '.results[0]'
# Expect a shift starting now (or recently) with the right user_id.

Terraform variant + shift block: references/templates-schedules.md.

Best practices

  • Keep chains ≤4 levels with a definitive final step (webhook to PagerDuty or auto-resolve)

  • Always set send_resolved: true in Alertmanager so OnCall auto-resolves

  • Use max_alerts: 100 in Alertmanager webhook config

  • Combine Slack + mobile push for delivery reliability

  • Assign integrations/schedules to teams for RBAC

Resources

Files

references/integrations.md

Integrations

Alertmanager / Prometheus

# alertmanager.yml
receivers:
  - name: grafana-oncall
    webhook_configs:
      - url: https://<your-stack>.grafana.net/integrations/v1/alertmanager/<id>/
        send_resolved: true
        max_alerts: 100        # prevent oversized payloads

route:
  receiver:        grafana-oncall
  group_by:        [alertname, cluster]
  group_wait:      30s
  group_interval:  5m
  repeat_interval: 4h

Grafana Alerting (same instance)

  1. OnCall / IRM → Integrations → New Integration → Grafana Alerting

  2. Click Quick Connect — auto-creates a contact point

  3. Link the contact point to a notification policy in Grafana Alerting

Webhook (custom / generic)

# Send
curl -X POST https://<stack>.grafana.net/integrations/v1/formatted_webhook/<id>/ \
  -H "Content-Type: application/json" \
  -d '{
    "alert_uid": "incident-123",
    "title": "Database CPU High",
    "state": "alerting",
    "message": "db-prod-01 CPU at 95% for 10 minutes",
    "link_to_upstream_details": "https://grafana.example.com/d/abc123"
  }'

# Resolve
curl -X POST https://<stack>.grafana.net/integrations/v1/formatted_webhook/<id>/ \
  -H "Content-Type: application/json" \
  -d '{"alert_uid": "incident-123", "state": "ok"}'

Recognized fields: alert_uid, title, state (alerting/ok), message, image_url, link_to_upstream_details.

Slack

  1. Install: OnCall Settings → Chat Ops → Slack → Install Slack Integration

  2. Connect users: each user → Profile → Connect to Slack

  3. Set default channel for alert routing

  4. Add to escalation: "Notify by Slack mentions" step

Slack message actions: Acknowledge, Resolve, Silence, Add responders, Add note. Slash commands: /escalate, /oncall.

RBAC roles

RoleAccess
oncall-adminFull access
oncall-editorCreate/edit integrations, schedules, escalation chains
oncall-viewerRead-only
oncall-notifications-receiverReceive alerts only

Rate limits

  • 300 alerts / integration / 5 min

  • 500 alerts / org / 5 min

  • 300 API requests / key / 5 min

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs network access

Security

What our scanner noticed· Low risk

  • Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
Grafana Labs
Publisher
Grafana Labs (Official)
Source
grafana/skills · skills/grafana-cloud/oncall-irm · commit 8fb72a5
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by Grafana Labs.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download oncall-irm.zip