Alloy

Grafana LabsOfficialPublished by Grafana Labs

Skill
Tool integration
Cloud and DevOps
Apache-2.0
Low risk

From the author

Build a unified telemetry pipeline with Grafana Alloy — one OpenTelemetry-compatible binary that collects metrics, logs, traces, and profiles and ships to Grafana Cloud / Prometheus / Loki / Tempo / Pyroscope. Covers the Alloy config language (blocks, `sys.env`, component refs), `prometheus.scrape` → `remote_write`, `loki.source.file` + `loki.process` → `loki.write`, `otelcol.receiver.otlp` → `otelcol.exporter.otlp`, `pyroscope.scrape`, K8s / Docker / EC2 discovery, relabeling, modules (`import.file/git/http`), clustering, Fleet Management `remotecfg`, the Alloy UI at `:12345`, and `alloy fmt` / `alloy validate`. Use when writing a `config.alloy`, replacing Grafana Agent / OTel Collector, scraping K8s pods, parsing logs, ingesting OTLP, or debugging "Alloy isn't sending anything" — even when the user says "set up the agent", "write me a scrape config", "drop these logs before sending", or "OTel collector config" without naming Alloy.

— the description by Grafana Labs

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

Grafana Alloy is an OpenTelemetry-compatible collector that uses a unified configuration language to build telemetry pipelines. This skill provides guidance on using the Alloy CLI, configuring components for data collection and shipping, and managing pipelines for metrics, logs, traces, and profiles. It covers the Alloy configuration syntax, component discovery, relabeling, and integration with backends like Grafana Cloud, Prometheus, Loki, Tempo, and Pyroscope.

What it does

Provides syntax and structure for config.alloy files using blocks and component references.
Explains how to use discovery components for Kubernetes, Docker, and cloud infrastructure.
Details the configuration of exporters and receivers for metrics, logs, and traces.
Offers commands for validating and formatting configuration files.
Describes how to use the Alloy UI to monitor component health and data flow.
Covers advanced features like modules, clustering, and remote configuration management.

When to use it

When setting up a telemetry pipeline to collect and ship observability data.
When migrating from Grafana Agent or OpenTelemetry Collector to Alloy.
When configuring scrape targets for Kubernetes pods or other infrastructure.
When debugging issues with data ingestion or pipeline connectivity.
When needing to transform or drop specific logs or metrics before they are shipped.

Advantages

Unified binary for handling multiple telemetry signals.
Native support for OpenTelemetry and Prometheus protocols.
Built-in validation and formatting tools to ensure configuration correctness.
Extensive component library for service discovery and data routing.
Integrated web UI for inspecting component status and troubleshooting.

Usage examples

“How do I set up a basic Prometheus scrape job in Alloy?”: The agent provides a configuration block using prometheus.scrape to target an address and forward metrics to a remote_write component.
“Help me debug why my Alloy logs aren't appearing in Loki.”: The agent suggests checking the component health via the local UI at port 12345 and verifying the loki.source.file configuration.
“How do I validate my config.alloy file?”: The agent instructs to run the alloy validate command to perform a syntax check on the configuration file.
“Write a configuration to receive OTLP traces and send them to Tempo.”: The agent provides an otelcol.receiver.otlp block connected to an otelcol.exporter.otlp block with appropriate authentication.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/grafana/alloy/download
Unzip it into the cross-client skills folder of your project:
```
mkdir -p .agents/skills && unzip alloy.zip -d .agents/skills
```

Documentation by Grafana Labs

Grafana Alloy

Docs: https://grafana.com/docs/alloy/latest/

OpenTelemetry-compatible collector — one binary for metrics + logs + traces + profiles.

Prerequisites

alloy CLI installed (brew install grafana/grafana/alloy, apt install alloy, or grafana/alloy Docker image)
An endpoint to ship to (Grafana Cloud, Prometheus, Loki, Tempo, Pyroscope)
API key + username for that endpoint, exported as GRAFANA_API_KEY etc.

Common Workflows

1. Write + validate a config locally

// config.alloy — metrics → Grafana Cloud
prometheus.scrape "app" {
  targets = [{"__address__" = "localhost:9090"}]
  forward_to = [prometheus.remote_write.cloud.receiver]
  scrape_interval = "30s"
}

prometheus.remote_write "cloud" {
  endpoint {
    url = sys.env("PROMETHEUS_URL")
    basic_auth {
      username = sys.env("PROM_USER")
      password = sys.env("GRAFANA_API_KEY")
    }
  }
}

# 1. Format + syntax-check (catches typos before run)
alloy fmt config.alloy
alloy validate config.alloy

# 2. Run it
alloy run config.alloy
# or as a service: systemctl restart alloy

# 3. Verify all components are healthy via the UI on port 12345
curl -s http://localhost:12345/api/v0/web/components \
  | jq '.[] | select(.health.state != "healthy") | {id, state:.health.state, msg:.health.message}'
# Expect: empty (nothing unhealthy). Otherwise the row shows the failing component + reason.

# 4. Verify samples are flowing
curl -s http://localhost:12345/metrics \
  | grep -E '^prometheus_remote_storage_(samples_total|enqueue_retries_total)' | head
# samples_total should be > 0 and rising; retries should be 0.

2. Add log shipping (file → Loki)

loki.source.file "app_logs" {
  targets    = [{ __path__ = "/var/log/app/*.log", job = "app" }]
  forward_to = [loki.write.cloud.receiver]
}

loki.write "cloud" {
  endpoint {
    url = sys.env("LOKI_URL")
    basic_auth {
      username = sys.env("LOKI_USER")
      password = sys.env("GRAFANA_API_KEY")
    }
  }
}

# Verify in Grafana → Explore → Loki:
#   {job="app"}
# Expect lines streaming. If empty:
#   - check /var/log/app/*.log actually exists + readable by the alloy user
#   - http://localhost:12345 → loki.source.file.app_logs → "Targets" tab

3. Receive OTLP traces and ship to Tempo

otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  http { endpoint = "0.0.0.0:4318" }
  output { traces = [otelcol.exporter.otlp.tempo.input] }
}

otelcol.exporter.otlp "tempo" {
  client {
    endpoint = "tempo-xxx.grafana.net/tempo:443"
    auth     = otelcol.auth.basic.grafana_cloud.handler
  }
}

otelcol.auth.basic "grafana_cloud" {
  username = sys.env("TEMPO_USER")
  password = sys.env("GRAFANA_API_KEY")
}

# Verify in Grafana → Explore → Tempo → service.name = your-service
# At the Alloy level, watch the receiver:
curl -s http://localhost:12345/metrics | grep otelcol_receiver_accepted_spans

Full pattern set (Kubernetes discovery + relabel, complete Cloud pipeline for all 4 signals): references/collection-patterns.md.

Reference

references/config-syntax.md — block/attribute/expression grammar, import.*, remotecfg, clustering
references/components.md — full component catalog with purpose + typical args
references/collection-patterns.md — end-to-end pipelines (K8s pods, OTLP, profiles, logs)

Troubleshooting

alloy validate → "component not found" → version too old; alloy --version and upgrade
UI shows component unhealthy → click into it on http://localhost:12345 for the live error
prometheus_remote_storage_samples_dropped_total rising → check enqueue_retries_total and the remote_write endpoint URL + creds
No spans landing in Tempo → check otelcol_receiver_refused_spans and the exporter's otelcol_exporter_sent_spans / otelcol_exporter_send_failed_spans

Resources

Files

ATTRIBUTION.mdAdded by AutoGineer830 B
LICENSELicence file added by AutoGineer11.1 KiB
SKILL.md5.2 KiB
references/collection-patterns.md10.7 KiB
references/components.md16.2 KiB
references/config-syntax.md9.2 KiB

references/collection-patterns.md

Alloy Collection Patterns

Canonical recipes for the common things Alloy does. Configs are taken from the official Alloy "Collect" docs (https://grafana.com/docs/alloy/latest/collect/) and adjusted to be runnable as-is with placeholder values.

Pipeline shape, at a glance

discovery / receive  →  process / relabel  →  write / export

Every pattern below fits this shape. The label of each component (in quotes after the block type) is your choice; references use block.type.label.export.

1. Prometheus metrics — scrape and remote-write

Pull /metrics from a static list of targets and ship to a Prometheus-compatible endpoint.

prometheus.scrape "custom_targets" {
  targets = [
    {__address__ = "prometheus:9090"},
    {__address__ = "custom-app:80", __metrics_path__ = "/custom-metrics-path", __scheme__ = "https"},
    {__address__ = "app:12345", environment = "production"},
  ]
  forward_to      = [prometheus.remote_write.default.receiver]
  scrape_interval = "30s"
}

prometheus.remote_write "default" {
  endpoint {
    url = sys.env("PROMETHEUS_URL")
    basic_auth {
      username = sys.env("PROMETHEUS_USER")
      password = sys.env("GRAFANA_API_KEY")
    }
  }
  external_labels = {
    cluster = "prod-us-east",
    env     = "production",
  }
}

Notes:

Labels beginning with __ are stripped after relabeling; they control the scrape, not the output.
__scheme__ defaults to http, __metrics_path__ defaults to /metrics.
Add multiple endpoint blocks to fan-out to several backends.

2. Kubernetes service discovery → scrape

Scrape every pod the API server lists, filtered by relabeling.

discovery.kubernetes "pods" {
  role = "pod"
}

discovery.relabel "pods" {
  targets = discovery.kubernetes.pods.targets

  rule {
    source_labels = ["__meta_kubernetes_pod_label_app"]
    target_label  = "app"
  }
  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    target_label  = "namespace"
  }
  // Drop pods that didn't set the "app" label
  rule {
    source_labels = ["__meta_kubernetes_pod_label_app"]
    regex         = ""
    action        = "drop"
  }
}

prometheus.scrape "k8s_pods" {
  targets    = discovery.relabel.pods.output
  forward_to = [prometheus.remote_write.default.receiver]
}

Other role values for discovery.kubernetes: service, endpoints, endpointslice, node, ingress.

3. Kubernetes pod logs → Loki

Tail container logs from the node Alloy is running on. Uses the standard meta-label relabeling recommended by the docs.

loki.write "default" {
  endpoint {
    url = sys.env("LOKI_URL")
    basic_auth {
      username = sys.env("LOKI_USER")
      password = sys.env("GRAFANA_API_KEY")
    }
  }
}

discovery.kubernetes "pod" {
  role = "pod"
  selectors {
    role  = "pod"
    field = "spec.nodeName=" + coalesce(sys.env("HOSTNAME"), constants.hostname)
  }
}

discovery.relabel "pod_logs" {
  targets = discovery.kubernetes.pod.targets

  rule { source_labels = ["__meta_kubernetes_namespace"]                  target_label = "namespace"  action = "replace" }
  rule { source_labels = ["__meta_kubernetes_pod_name"]                   target_label = "pod"        action = "replace" }
  rule { source_labels = ["__meta_kubernetes_pod_container_name"]         target_label = "container"  action = "replace" }
  rule { source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"] target_label = "app"  action = "replace" }

  rule {
    source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"]
    target_label  = "job"
    separator     = "/"
    replacement   = "$1"
    action        = "replace"
  }

  rule {
    source_labels = ["__meta_kubernetes_pod_uid", "__meta_kubernetes_pod_container_name"]
    target_label  = "__path__"
    separator     = "/"
    replacement   = "/var/log/pods/*$1/*.log"
    action        = "replace"
  }
}

loki.source.kubernetes "pod_logs" {
  targets    = discovery.relabel.pod_logs.output
  forward_to = [loki.process.pod_logs.receiver]
}

loki.process "pod_logs" {
  forward_to = [loki.write.default.receiver]
  stage.static_labels {
    values = { cluster = "prod-us-east" }
  }
}

4. Kubernetes cluster events → Loki

loki.source.kubernetes_events "cluster_events" {
  job_name   = "integrations/kubernetes/eventhandler"
  log_format = "logfmt"
  forward_to = [loki.process.cluster_events.receiver]
}

loki.process "cluster_events" {
  forward_to = [loki.write.default.receiver]
  stage.static_labels {
    values = { cluster = "prod-us-east" }
  }
  stage.labels {
    values = { kubernetes_cluster_events = "job" }
  }
}

5. Node/system logs → Loki

local.file_match "node_logs" {
  path_targets = [{
    __path__  = "/var/log/syslog",
    job       = "node/syslog",
    node_name = sys.env("HOSTNAME"),
    cluster   = "prod-us-east",
  }]
}

loki.source.file "node_logs" {
  targets    = local.file_match.node_logs.targets
  forward_to = [loki.write.default.receiver]
}

6. systemd journal → Loki

loki.source.journal "system" {
  forward_to = [loki.write.default.receiver]
  labels     = { job = "systemd-journal" }
  // optional: relabel_rules, matches, format_as_json
}

7. OpenTelemetry → Grafana LGTM stack

OTLP in, split to metrics/logs/traces with a batch processor, ship to Grafana Cloud.

otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  http { endpoint = "0.0.0.0:4318" }

  output {
    metrics = [otelcol.processor.batch.default.input]
    logs    = [otelcol.processor.batch.default.input]
    traces  = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  output {
    metrics = [otelcol.exporter.prometheus.grafana_cloud_metrics.input]
    logs    = [otelcol.exporter.loki.grafana_cloud_logs.input]
    traces  = [otelcol.exporter.otlphttp.grafana_cloud_traces.input]
  }
}

otelcol.exporter.otlphttp "grafana_cloud_traces" {
  client {
    endpoint = "https://tempo-us-central1.grafana.net:443"
    auth     = otelcol.auth.basic.grafana_cloud.handler
  }
}

otelcol.exporter.prometheus "grafana_cloud_metrics" {
  forward_to = [prometheus.remote_write.grafana_cloud.receiver]
}

otelcol.exporter.loki "grafana_cloud_logs" {
  forward_to = [loki.write.grafana_cloud.receiver]
}

otelcol.auth.basic "grafana_cloud" {
  username = sys.env("GRAFANA_CLOUD_INSTANCE_ID")
  password = sys.env("GRAFANA_CLOUD_API_KEY")
}

prometheus.remote_write "grafana_cloud" {
  endpoint {
    url = "https://prometheus-us-central1.grafana.net/api/prom/push"
    basic_auth {
      username = sys.env("GRAFANA_CLOUD_METRICS_USER")
      password = sys.env("GRAFANA_CLOUD_API_KEY")
    }
  }
}

loki.write "grafana_cloud" {
  endpoint {
    url = "https://logs-prod-us-central1.grafana.net/loki/api/v1/push"
    basic_auth {
      username = sys.env("GRAFANA_CLOUD_LOGS_USER")
      password = sys.env("GRAFANA_CLOUD_API_KEY")
    }
  }
}

Add a memory_limiter in front of batch in production:

otelcol.processor.memory_limiter "default" {
  check_interval   = "1s"
  limit_percentage = 80
  output {
    metrics = [otelcol.processor.batch.default.input]
    logs    = [otelcol.processor.batch.default.input]
    traces  = [otelcol.processor.batch.default.input]
  }
}

…and point the OTLP receiver's output.* at otelcol.processor.memory_limiter.default.input.

8. Profiles — pprof scraping → Pyroscope

pyroscope.scrape "default" {
  targets = [
    {__address__ = "localhost:6060", service_name = "myapp"},
  ]
  forward_to = [pyroscope.write.default.receiver]
}

pyroscope.write "default" {
  endpoint {
    url = sys.env("PYROSCOPE_URL")
    basic_auth {
      username = sys.env("PYROSCOPE_USER")
      password = sys.env("GRAFANA_API_KEY")
    }
  }
}

9. Zero-code traces and metrics with Beyla

beyla.ebpf "default" {
  open_port = "8080"   // or attach_pid, executable_name, k8s selectors

  output {
    metrics = [otelcol.processor.batch.default.input]
    traces  = [otelcol.processor.batch.default.input]
  }
}

10. Frontend telemetry — Faro

faro.receiver "default" {
  server {
    listen_address = "0.0.0.0"
    listen_port    = 12347
    // Replace with your actual web origins before deploying — wildcard "*" lets any site send
    // telemetry to this endpoint and is not suitable for production.
    cors_allowed_origins = ["https://app.example.com"]
  }

  output {
    logs   = [loki.write.grafana_cloud.receiver]
    traces = [otelcol.processor.batch.default.input]
  }
}

11. Clustering — distribute scrape load

Run multiple Alloy replicas and let them shard targets among themselves.

clustering {
  enabled = true
}

prometheus.scrape "cluster_aware" {
  targets    = discovery.kubernetes.pods.targets
  forward_to = [prometheus.remote_write.default.receiver]
  clustering { enabled = true }
}

Cluster discovery uses memberlist; configure peer addresses via the --cluster.* CLI flags on alloy run (e.g. --cluster.join-addresses=alloy-headless:12345).

12. Fleet Management — remote config

remotecfg {
  url = "https://fleet-management-prod-XXX.grafana.net"
  basic_auth {
    username = sys.env("FM_USERNAME")
    password = sys.env("FM_TOKEN")
  }
  poll_interval = "1m"
}

With remotecfg, the local file becomes the bootstrap. The fleet server delivers the rest as modules Alloy imports automatically.

13. Modules and imports

Pull components from a Git module library:

import.git "k8s_monitoring" {
  repository = "https://github.com/grafana/alloy-modules"
  revision   = "main"
  path       = "modules/kubernetes/"
}

k8s_monitoring.pods "default" {
  forward_to = [prometheus.remote_write.default.receiver]
}

14. Pipeline processing — relabel and parse

Drop noisy Go metrics, add an environment label:

prometheus.relabel "filter" {
  forward_to = [prometheus.remote_write.default.receiver]

  rule {
    source_labels = ["__name__"]
    regex         = "go_.*"
    action        = "drop"
  }
  rule {
    target_label = "environment"
    replacement  = "production"
  }
}

Parse JSON logs and lift one field to a label:

loki.process "json_parse" {
  forward_to = [loki.write.default.receiver]

  stage.json {
    expressions = { level = "level", msg = "message" }
  }
  stage.labels {
    values = { level = "" }
  }
  stage.drop {
    expression = ".*health check.*"
  }
}

Running Alloy

alloy run /etc/alloy/config.alloy \
  --server.http.listen-addr=0.0.0.0:12345 \
  --storage.path=/var/lib/alloy/data \
  --cluster.enabled=true \
  --cluster.join-addresses=alloy-headless.alloy.svc:12345

Reload after editing: kill -HUP <pid> or curl -XPOST http://localhost:12345/-/reload.

UI and component graph: http://localhost:12345/.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs network access, Needs an API key

Security

What our scanner noticed· Low risk

Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

Grafana Labs

Publisher

Grafana Labs (Official)

Source

grafana/skills · skills/grafana-core/alloy · commit 8fb72a5

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by Grafana Labs.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download alloy.zip