Opentelemetry

Grafana LabsOfficialPublished by Grafana Labs

Skill
Tool integration
Cloud and DevOps
Apache-2.0
Low risk

From the author

Instrument any app with OpenTelemetry and ship metrics / logs / traces to Grafana Cloud or self-hosted Mimir / Loki / Tempo / Pyroscope. Covers SDK auto-instrumentation for Go, Java (Grafana JVM agent), Python (`opentelemetry-instrument`), Node.js, .NET (`Grafana.OpenTelemetry`), Beyla eBPF for zero-code; Grafana Cloud OTLP gateway + Basic-auth (instanceID + API key, base64); env-var config (`OTEL_EXPORTER_OTLP_*`, `OTEL_RESOURCE_ATTRIBUTES`); Alloy / OTel-Collector pipelines; Kubernetes Operator inject-annotations; and head + tail sampling. Use when instrumenting a service, pointing OTLP at Grafana Cloud, switching from Jaeger / Datadog / New Relic, choosing head- vs tail-sampling, or debugging "spans aren't showing in Explore" — even when the user says "auto-instrument my Java app", "send traces to Grafana", "what env vars do I set", "OTLP endpoint", or "Operator inject" without naming OpenTelemetry.

— the description by Grafana Labs

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a comprehensive guide for instrumenting applications using OpenTelemetry (OTel) and configuring them to export metrics, logs, traces, and profiles to Grafana Cloud or self-hosted Grafana backends (Mimir, Loki, Tempo, Pyroscope). It covers various implementation methods, including SDK auto-instrumentation for multiple languages, zero-code instrumentation via Beyla eBPF, and the use of Grafana Alloy or the OTel Collector for pipeline management. The documentation includes authentication procedures, environment variable configurations, Kubernetes operator integration, and troubleshooting common connectivity issues.

What it does

Configures OTLP exporters to send telemetry to Grafana Cloud or self-hosted backends.
Provides auto-instrumentation patterns for Java, Python, Go, Node.js, and .NET.
Demonstrates zero-code instrumentation using Beyla eBPF.
Explains how to set up Grafana Alloy as a collector for buffering and tail-sampling.
Details Kubernetes auto-instrumentation using the OTel Operator.
Offers troubleshooting steps for common OTLP gateway and connectivity errors.

When to use it

When you need to instrument a service to send telemetry data to Grafana Cloud.
When migrating from other monitoring tools like Jaeger, Datadog, or New Relic.
When you need to configure head or tail sampling for trace data.
When debugging missing spans or connectivity issues between your app and Grafana.
When setting up automated instrumentation in a Kubernetes environment.

Advantages

Vendor-neutral instrumentation approach using standard OTLP protocols.
Supports both manual SDK integration and zero-code instrumentation.
Provides clear guidance on authentication and environment variable setup.
Includes robust troubleshooting paths for common configuration pitfalls.

Usage examples

“How do I auto-instrument my Java application to send traces to Grafana?”: The agent provides the command to download the Grafana JVM agent and the necessary environment variables to authenticate with the Grafana Cloud OTLP gateway.
“What environment variables are required to connect to the Grafana Cloud OTLP endpoint?”: The agent lists the specific variables including OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL, and the base64-encoded authorization header.
“How can I use the Kubernetes Operator to instrument my pods?”: The agent provides the YAML configuration for an Instrumentation resource and the specific annotations required to inject auto-instrumentation into your pods.
“My spans are not showing up in Grafana Explore, how do I debug this?”: The agent guides you through verifying your OTLP endpoint URL, checking that the protocol matches the transport, and ensuring your API key has the correct publisher roles.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/grafana/opentelemetry/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip opentelemetry.zip -d .agents/skills

Documentation by Grafana Labs

OpenTelemetry with Grafana

Docs: https://grafana.com/docs/opentelemetry/

Vendor-neutral instrumentation pipeline. Apps speak OTLP → Alloy (or direct) → Grafana Cloud (Mimir / Loki / Tempo / Pyroscope).

Backends

Signal	Backend
Metrics	Grafana Mimir
Logs	Grafana Loki
Traces	Grafana Tempo
Profiles	Grafana Pyroscope

Prerequisites

Grafana Cloud stack OR self-hosted Mimir / Loki / Tempo
Cloud OTLP endpoint: https://otlp-gateway-<region>.grafana.net/otlp
Basic-auth credentials: numeric instance ID + API token with MetricsPublisher + LogsPublisher + TracesPublisher
An app to instrument

Common Workflows

1. Authenticate to the Grafana Cloud OTLP endpoint

# 1. Build the auth header
INSTANCE_ID=123456
API_KEY="glc_eyJ..."
export OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(echo -n "${INSTANCE_ID}:${API_KEY}" | base64)"
export OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=prod"

# 2. Smoke-test creds with a curl POST against the OTLP traces endpoint (empty body)
curl -s -o /dev/null -w "%{http_code}\n" \
  -X POST -H "Content-Type: application/x-protobuf" \
  -H "Authorization: Basic $(echo -n "${INSTANCE_ID}:${API_KEY}" | base64)" \
  "$OTEL_EXPORTER_OTLP_ENDPOINT/v1/traces" --data-binary '\n'
# Expect 400 (malformed payload) — NOT 401 (auth) or 404 (wrong endpoint).

2. Auto-instrument a Java app + verify

# 1. Download the Grafana JVM agent (single jar)
curl -sLO https://github.com/grafana/grafana-opentelemetry-java/releases/latest/download/grafana-opentelemetry-java.jar

# 2. Run with the agent + env from step 1
java -javaagent:./grafana-opentelemetry-java.jar -jar myapp.jar

# 3. Generate traffic, then verify in Grafana → Explore → Tempo:
#    TraceQL: { resource.service.name = "myapp" }
#    Expect spans within ~30s. Also verify metrics:
#    PromQL: count by (service_name)({service_name="myapp"})

3. Auto-instrument a Python app

pip install "opentelemetry-distro[otlp]"
opentelemetry-bootstrap -a install

# Same env vars as step 1, then:
opentelemetry-instrument python app.py

# Verify the same way — Explore → Traces filter service.name=myapp.

4. Add Alloy as a buffering / sampling collector

# Application points at local Alloy (gRPC fastest)
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc

# Alloy environment for forwarding to Cloud
export GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp
export GRAFANA_CLOUD_INSTANCE_ID=$INSTANCE_ID
export GRAFANA_CLOUD_API_KEY=$API_KEY
alloy run /etc/alloy/config.alloy

# Verify Alloy received and forwarded
curl -s http://localhost:12345/metrics | grep otelcol_exporter_sent_spans

Full Alloy config + tail-sampling block + OTel Collector YAML + K8s Operator install: references/collector-config.md.

SDK-by-language details (Go full code, Node manual setup, .NET ASP.NET Core, all the env-var quirks): references/instrumentation.md.

5. Kubernetes — auto-inject via the Operator

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata: { name: my-instrumentation }
spec:
  exporter: { endpoint: http://otelcol:4317 }
  propagators: [tracecontext, baggage]
  java:
    image: us-docker.pkg.dev/grafanalabs-global/docker-grafana-opentelemetry-java-prod/grafana-opentelemetry-java:2.3.0-beta.1
  nodejs: {}
  python: {}

Then annotate pods:

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"
    # or: inject-nodejs, inject-python, inject-dotnet

# Verify the operator injected the agent
kubectl describe pod <pod> | grep -A2 'opentelemetry-auto-instrumentation'
# Then run the same Grafana Explore checks.

Sampling — when to pick which

# Head sampling (cheap, decided at start; may lose rare errors)
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1   # 10%

Tail sampling (decides after seeing the whole trace — keep errors + sample the rest) requires an Alloy / OTel-Collector tail_sampling processor; full block in references/collector-config.md.

Key environment variables

Variable	Example
`OTEL_EXPORTER_OTLP_ENDPOINT`	`https://otlp-gateway-prod-us-east-0.grafana.net/otlp`
`OTEL_EXPORTER_OTLP_PROTOCOL`	`grpc` or `http/protobuf`
`OTEL_EXPORTER_OTLP_HEADERS`	`Authorization=Basic <base64>`
`OTEL_RESOURCE_ATTRIBUTES`	`service.name=app,service.namespace=team,deployment.environment=prod`
`OTEL_SERVICE_NAME`	shorthand for `service.name`
`OTEL_TRACES_SAMPLER` / `_ARG`	`parentbased_traceidratio` / `0.1`

Troubleshooting

401 from OTLP gateway → instance ID is not numeric, or API key missing publisher roles
404 → endpoint URL wrong (must end with /otlp)
Spans missing → check OTEL_EXPORTER_OTLP_PROTOCOL matches transport (Cloud OTLP gateway = http/protobuf, Alloy local = grpc)
Node.js auto-instrumentation broken after bundling → bundlers like @vercel/ncc defeat the require hooks
Python under Gunicorn / uWSGI shows no spans → reinit OTel providers in a post-fork hook

Resources

Files

ATTRIBUTION.mdAdded by AutoGineer846 B
LICENSELicence file added by AutoGineer11.1 KiB
SKILL.md6.7 KiB
references/collector-config.md18.1 KiB
references/instrumentation.md20.6 KiB

references/collector-config.md

OTel Collector Configuration Examples

Grafana Alloy

Grafana Alloy uses a River-based configuration language (.alloy files). It is the recommended collector distribution from Grafana Labs.

Installation

# macOS
brew install grafana/grafana/alloy

# Linux (Debian/Ubuntu)
sudo apt-get install alloy

# Docker
docker run -v /path/to/config.alloy:/etc/alloy/config.alloy \
  -p 4317:4317 -p 4318:4318 \
  grafana/alloy run /etc/alloy/config.alloy

Complete Application Observability Config

Save as config.alloy:

// =========================================
// OTLP Receiver - accepts telemetry from apps
// =========================================
otelcol.receiver.otlp "default" {
  grpc {
    endpoint = "0.0.0.0:4317"
  }
  http {
    endpoint = "0.0.0.0:4318"
  }

  output {
    metrics = [otelcol.processor.resourcedetection.default.input]
    logs    = [otelcol.processor.resourcedetection.default.input]
    traces  = [otelcol.processor.resourcedetection.default.input]
  }
}

// =========================================
// Resource Detection - enrich with host/cloud metadata
// =========================================
otelcol.processor.resourcedetection "default" {
  detectors = ["env", "system", "docker"]

  system {
    hostname_sources = ["os"]
  }

  output {
    metrics = [otelcol.processor.transform.drop_unneeded_resource_attributes.input]
    logs    = [otelcol.processor.transform.drop_unneeded_resource_attributes.input]
    traces  = [otelcol.processor.transform.drop_unneeded_resource_attributes.input]
  }
}

// =========================================
// Transform - remove noisy/unnecessary attributes
// =========================================
otelcol.processor.transform "drop_unneeded_resource_attributes" {
  error_mode = "ignore"

  trace_statements {
    context = "resource"
    statements = [
      "delete_key(attributes, \"process.pid\")",
      "delete_key(attributes, \"process.runtime.description\")",
      "delete_key(attributes, \"process.runtime.name\")",
      "delete_key(attributes, \"process.runtime.version\")",
    ]
  }

  metric_statements {
    context = "resource"
    statements = [
      "delete_key(attributes, \"process.pid\")",
    ]
  }

  log_statements {
    context = "resource"
    statements = [
      "delete_key(attributes, \"process.pid\")",
    ]
  }

  output {
    metrics = [otelcol.connector.host_info.default.input]
    logs    = [otelcol.processor.batch.default.input]
    traces  = [otelcol.connector.host_info.default.input]
  }
}

// =========================================
// Host Info Connector - generate host metrics from traces
// =========================================
otelcol.connector.host_info "default" {
  host_identifiers = ["host.name"]

  output {
    metrics = [otelcol.processor.batch.default.input]
  }
}

// =========================================
// Batch Processor - group data for efficiency
// =========================================
otelcol.processor.batch "default" {
  output {
    metrics = [otelcol.exporter.otlphttp.grafana_cloud.input]
    logs    = [otelcol.exporter.otlphttp.grafana_cloud.input]
    traces  = [otelcol.exporter.otlphttp.grafana_cloud.input]
  }
}

// =========================================
// OTLP HTTP Exporter - send to Grafana Cloud
// =========================================
otelcol.exporter.otlphttp "grafana_cloud" {
  client {
    endpoint = env("GRAFANA_CLOUD_OTLP_ENDPOINT")

    auth = otelcol.auth.basic.grafana_cloud.handler
  }
}

otelcol.auth.basic "grafana_cloud" {
  username = env("GRAFANA_CLOUD_INSTANCE_ID")
  password = env("GRAFANA_CLOUD_API_KEY")
}

Run Alloy

# Set required environment variables
export GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp
export GRAFANA_CLOUD_INSTANCE_ID=123456
export GRAFANA_CLOUD_API_KEY=glc_eyJ...

# Run
alloy run config.alloy

Minimal Config (no processors)

otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  http { endpoint = "0.0.0.0:4318" }
  output {
    metrics = [otelcol.exporter.otlphttp.cloud.input]
    logs    = [otelcol.exporter.otlphttp.cloud.input]
    traces  = [otelcol.exporter.otlphttp.cloud.input]
  }
}

otelcol.exporter.otlphttp "cloud" {
  client {
    endpoint = env("GRAFANA_CLOUD_OTLP_ENDPOINT")
    auth     = otelcol.auth.basic.cloud.handler
  }
}

otelcol.auth.basic "cloud" {
  username = env("GRAFANA_CLOUD_INSTANCE_ID")
  password = env("GRAFANA_CLOUD_API_KEY")
}

Grafana Alloy - Head-Based Sampling

otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  http { endpoint = "0.0.0.0:4318" }
  output {
    traces = [otelcol.processor.probabilistic_sampler.default.input]
    metrics = [otelcol.exporter.otlphttp.cloud.input]
    logs    = [otelcol.exporter.otlphttp.cloud.input]
  }
}

otelcol.processor.probabilistic_sampler "default" {
  sampling_percentage = 10   // keep 10% of traces

  output {
    traces = [otelcol.exporter.otlphttp.cloud.input]
  }
}

otelcol.exporter.otlphttp "cloud" {
  client {
    endpoint = env("GRAFANA_CLOUD_OTLP_ENDPOINT")
    auth     = otelcol.auth.basic.cloud.handler
  }
}

otelcol.auth.basic "cloud" {
  username = env("GRAFANA_CLOUD_INSTANCE_ID")
  password = env("GRAFANA_CLOUD_API_KEY")
}

Grafana Alloy - Tail-Based Sampling

Tail sampling makes decisions after all spans of a trace are collected (requires buffering). Useful for: always keeping errors, slow requests, or specific services.

otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  http { endpoint = "0.0.0.0:4318" }
  output {
    traces  = [otelcol.processor.tail_sampling.default.input]
    metrics = [otelcol.exporter.otlphttp.cloud.input]
    logs    = [otelcol.exporter.otlphttp.cloud.input]
  }
}

otelcol.processor.tail_sampling "default" {
  // Wait this long after seeing first span before deciding
  decision_wait               = "10s"
  num_traces                  = 100000
  expected_new_traces_per_sec = 100

  // Keep ALL error traces
  policy {
    name = "keep-errors"
    type = "status_code"
    status_code {
      status_codes = ["ERROR"]
    }
  }

  // Keep slow traces (> 500ms)
  policy {
    name = "keep-slow-traces"
    type = "latency"
    latency {
      threshold_ms = 500
    }
  }

  // Sample 10% of remaining
  policy {
    name = "probabilistic-sample"
    type = "probabilistic"
    probabilistic {
      sampling_percentage = 10
    }
  }

  output {
    traces = [otelcol.exporter.otlphttp.cloud.input]
  }
}

otelcol.exporter.otlphttp "cloud" {
  client {
    endpoint = env("GRAFANA_CLOUD_OTLP_ENDPOINT")
    auth     = otelcol.auth.basic.cloud.handler
  }
}

otelcol.auth.basic "cloud" {
  username = env("GRAFANA_CLOUD_INSTANCE_ID")
  password = env("GRAFANA_CLOUD_API_KEY")
}

Grafana Alloy - Kubernetes (Helm)

Helm Values (k8s-monitoring chart)

# values.yaml
cluster:
  name: my-cluster

externalServices:
  prometheus:
    host: https://prometheus-us-central1.grafana.net
    basicAuth:
      username: "123456"
      password: "${GRAFANA_API_KEY}"
  loki:
    host: https://logs-us-central1.grafana.net
    basicAuth:
      username: "123456"
      password: "${GRAFANA_API_KEY}"
  tempo:
    host: https://tempo-us-central1.grafana.net
    basicAuth:
      username: "123456"
      password: "${GRAFANA_API_KEY}"

receivers:
  otlp:
    enabled: true
    grpc:
      enabled: true
    http:
      enabled: true

opencost:
  enabled: false

helm repo add grafana https://grafana.github.io/helm-charts
helm upgrade --install k8s-monitoring grafana/k8s-monitoring \
  --namespace monitoring --create-namespace \
  -f values.yaml

Upstream OpenTelemetry Collector

For environments where Grafana Alloy cannot be used.

Installation

# Download otelcol-contrib (includes all components)
curl -L https://github.com/open-telemetry/opentelemetry-collector-releases/releases/latest/download/otelcol-contrib_linux_amd64.tar.gz | tar xz

# Docker
docker run -v /path/to/config.yaml:/etc/otelcol-contrib/config.yaml \
  -p 4317:4317 -p 4318:4318 \
  otel/opentelemetry-collector-contrib:latest

Complete config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  # Detect host/cloud resource attributes
  resourcedetection:
    detectors: [env, system, docker]
    system:
      hostname_sources: [os]
    timeout: 2s

  # Remove noisy attributes
  transform:
    error_mode: ignore
    trace_statements:
      - context: resource
        statements:
          - delete_key(attributes, "process.pid")
          - delete_key(attributes, "process.runtime.description")

  # Batch for efficiency
  batch:
    send_batch_size: 1000
    timeout: 10s

  # Memory limiter to prevent OOM
  memory_limiter:
    check_interval: 5s
    limit_mib: 512
    spike_limit_mib: 128

exporters:
  # Send to Grafana Cloud
  otlphttp/grafana:
    endpoint: ${env:GRAFANA_CLOUD_OTLP_ENDPOINT}
    auth:
      authenticator: basicauth/grafana

  # Debug - log to console
  debug:
    verbosity: detailed

extensions:
  basicauth/grafana:
    client_auth:
      username: ${env:GRAFANA_CLOUD_INSTANCE_ID}
      password: ${env:GRAFANA_CLOUD_API_KEY}

  health_check:
    endpoint: 0.0.0.0:13133

service:
  extensions: [basicauth/grafana, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, transform, batch]
      exporters: [otlphttp/grafana]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [otlphttp/grafana]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [otlphttp/grafana]

Run

export GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp
export GRAFANA_CLOUD_INSTANCE_ID=123456
export GRAFANA_CLOUD_API_KEY=glc_eyJ...

./otelcol-contrib --config config.yaml

Upstream Collector - Tail Sampling

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    expected_new_traces_per_sec: 100
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]

      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 500

      - name: probabilistic
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

      - name: keep-specific-service
        type: string_attribute
        string_attribute:
          key: service.name
          values: [payment-service]   # always keep

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlphttp/grafana]

Kubernetes - OTel Collector as DaemonSet

apiVersion: v1
kind: ConfigMap
metadata:
  name: otelcol-config
  namespace: monitoring
data:
  config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
      # Collect kubelet metrics
      kubeletstats:
        collection_interval: 30s
        auth_type: serviceAccount
        endpoint: "https://${env:K8S_NODE_NAME}:10250"
        insecure_skip_verify: true

    processors:
      k8sattributes:
        auth_type: serviceAccount
        passthrough: false
        extract:
          metadata:
            - k8s.pod.name
            - k8s.pod.uid
            - k8s.deployment.name
            - k8s.namespace.name
            - k8s.node.name
        pod_association:
          - sources:
            - from: resource_attribute
              name: k8s.pod.ip
          - sources:
            - from: connection

      batch:
        send_batch_size: 1000
        timeout: 10s

    exporters:
      otlphttp/grafana:
        endpoint: ${env:GRAFANA_CLOUD_OTLP_ENDPOINT}
        auth:
          authenticator: basicauth/grafana

    extensions:
      basicauth/grafana:
        client_auth:
          username: ${env:GRAFANA_CLOUD_INSTANCE_ID}
          password: ${env:GRAFANA_CLOUD_API_KEY}

    service:
      extensions: [basicauth/grafana]
      pipelines:
        traces:
          receivers: [otlp]
          processors: [k8sattributes, batch]
          exporters: [otlphttp/grafana]
        metrics:
          receivers: [otlp, kubeletstats]
          processors: [k8sattributes, batch]
          exporters: [otlphttp/grafana]
        logs:
          receivers: [otlp]
          processors: [k8sattributes, batch]
          exporters: [otlphttp/grafana]
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otelcol
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: otelcol
  template:
    metadata:
      labels:
        app: otelcol
    spec:
      serviceAccountName: otelcol
      containers:
      - name: otelcol
        image: otel/opentelemetry-collector-contrib:latest
        args: ["--config=/etc/otelcol/config.yaml"]
        env:
        - name: GRAFANA_CLOUD_OTLP_ENDPOINT
          valueFrom:
            secretKeyRef:
              name: grafana-cloud-creds
              key: otlp-endpoint
        - name: GRAFANA_CLOUD_INSTANCE_ID
          valueFrom:
            secretKeyRef:
              name: grafana-cloud-creds
              key: instance-id
        - name: GRAFANA_CLOUD_API_KEY
          valueFrom:
            secretKeyRef:
              name: grafana-cloud-creds
              key: api-key
        - name: K8S_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        ports:
        - name: otlp-grpc
          containerPort: 4317
        - name: otlp-http
          containerPort: 4318
        volumeMounts:
        - name: config
          mountPath: /etc/otelcol
      volumes:
      - name: config
        configMap:
          name: otelcol-config
---
apiVersion: v1
kind: Service
metadata:
  name: otelcol
  namespace: monitoring
spec:
  selector:
    app: otelcol
  ports:
  - name: otlp-grpc
    port: 4317
    targetPort: 4317
  - name: otlp-http
    port: 4318
    targetPort: 4318

Kubernetes - OTel Operator with Instrumentation CR

Install Operator

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

OpenTelemetryCollector CR

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otelcol
  namespace: monitoring
spec:
  mode: daemonset   # or deployment, statefulset, sidecar
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318

    processors:
      batch:
        send_batch_size: 1000
        timeout: 10s

    exporters:
      otlphttp/grafana:
        endpoint: ${env:GRAFANA_CLOUD_OTLP_ENDPOINT}
        headers:
          authorization: "Basic ${env:GRAFANA_CLOUD_AUTH}"

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlphttp/grafana]
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlphttp/grafana]
        logs:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlphttp/grafana]
  env:
  - name: GRAFANA_CLOUD_OTLP_ENDPOINT
    valueFrom:
      secretKeyRef:
        name: grafana-cloud-creds
        key: otlp-endpoint
  - name: GRAFANA_CLOUD_AUTH
    valueFrom:
      secretKeyRef:
        name: grafana-cloud-creds
        key: basic-auth-base64

Instrumentation CR (auto-injection)

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: grafana-instrumentation
  namespace: default
spec:
  exporter:
    endpoint: http://otelcol-collector.monitoring:4317

  propagators:
    - tracecontext
    - baggage

  sampler:
    type: parentbased_traceidratio
    argument: "0.25"   # 25% sampling

  java:
    # Use Grafana distribution
    image: us-docker.pkg.dev/grafanalabs-global/docker-grafana-opentelemetry-java-prod/grafana-opentelemetry-java:2.3.0-beta.1
    env:
    - name: OTEL_EXPORTER_OTLP_PROTOCOL
      value: grpc

  nodejs:
    env:
    - name: OTEL_EXPORTER_OTLP_PROTOCOL
      value: grpc

  python:
    env:
    - name: OTEL_EXPORTER_OTLP_PROTOCOL
      value: grpc

  dotnet:
    env:
    - name: OTEL_EXPORTER_OTLP_PROTOCOL
      value: grpc

Inject into Pods

Add ONE annotation to your pod template:

# Java
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"

# Node.js
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-nodejs: "true"

# Python
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-python: "true"

# .NET
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-dotnet: "true"

# Use specific Instrumentation from another namespace
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "monitoring/grafana-instrumentation"

Secret Management for Grafana Cloud Credentials

# Create Kubernetes secret
kubectl create secret generic grafana-cloud-creds \
  --namespace monitoring \
  --from-literal=otlp-endpoint=https://otlp-gateway-prod-us-east-0.grafana.net/otlp \
  --from-literal=instance-id=123456 \
  --from-literal=api-key=glc_eyJ... \
  --from-literal=basic-auth-base64=$(echo -n '123456:glc_eyJ...' | base64)

Alloy Docker Compose

# docker-compose.yaml
services:
  alloy:
    image: grafana/alloy:latest
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
      - "12345:12345" # Alloy UI
    volumes:
      - ./config.alloy:/etc/alloy/config.alloy
    command: run --server.http.listen-addr=0.0.0.0:12345 /etc/alloy/config.alloy
    environment:
      - GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp
      - GRAFANA_CLOUD_INSTANCE_ID=123456
      - GRAFANA_CLOUD_API_KEY=glc_eyJ...

  myapp:
    build: .
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://alloy:4317
      - OTEL_EXPORTER_OTLP_PROTOCOL=grpc
      - OTEL_RESOURCE_ATTRIBUTES=service.name=myapp,service.namespace=myteam,deployment.environment=dev
    depends_on:
      - alloy

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs network access, Needs an API key, Needs a command-line tool, Needs Python

Security

What our scanner noticed· Low risk

Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

Grafana Labs

Publisher

Grafana Labs (Official)

Source

grafana/skills · skills/grafana-core/opentelemetry · commit 8fb72a5

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by Grafana Labs.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download opentelemetry.zip

From the author

AutoGineer summary

What it does

When to use it

Advantages

Usage examples

Install

Documentation by Grafana Labs

OpenTelemetry with Grafana

Backends

Prerequisites

Common Workflows

1. Authenticate to the Grafana Cloud OTLP endpoint

2. Auto-instrument a Java app + verify

3. Auto-instrument a Python app

4. Add Alloy as a buffering / sampling collector

5. Kubernetes — auto-inject via the Operator

Sampling — when to pick which

Key environment variables

Troubleshooting

Resources

Files

references/collector-config.md

OTel Collector Configuration Examples

Grafana Alloy

Installation

Complete Application Observability Config

Run Alloy

Minimal Config (no processors)

Grafana Alloy - Head-Based Sampling

Grafana Alloy - Tail-Based Sampling

Grafana Alloy - Kubernetes (Helm)

Helm Values (k8s-monitoring chart)

Upstream OpenTelemetry Collector

Installation

Complete config.yaml

Run

Upstream Collector - Tail Sampling

Kubernetes - OTel Collector as DaemonSet

Kubernetes - OTel Operator with Instrumentation CR

Install Operator

OpenTelemetryCollector CR

Instrumentation CR (auto-injection)

Inject into Pods

Secret Management for Grafana Cloud Credentials

Alloy Docker Compose

Requirements and compatibility

Security

Attribution and licence

Related skills

Spark Memory Thermal Ops

Observability And Instrumentation

Gke Alert Configuration

Application Performance Observability Engineer

Distributed Debugging Devops Troubleshooter

Distributed Tracing

Download ZIP