Instrument LLM Analytics

PostHogOfficialCreated by PostHog

Skill
Tool integration
Data and AI
MIT
Low risk

From the author

Add PostHog AI Observability to trace AI model usage. Use after implementing LLM features or reviewing PRs to ensure all generations are captured with token counts, latency, and costs. Also handles initial PostHog SDK setup if not yet installed.

— the description by PostHog

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured approach to instrumenting LLM-powered applications with PostHog AI Observability. It guides the agent through detecting the existing LLM stack, installing necessary SDKs, and configuring event capture for token counts, latency, and costs. The skill supports a wide range of LLM providers and frameworks, ensuring that observability is added with minimal disruption to existing codebases.

What it does

Detects LLM providers and frameworks in the codebase.
Installs the PostHog SDK and required integration packages.
Instruments LLM calls to capture tokens, latency, and costs.
Configures environment variables for secure credential management.
Associates LLM generations with specific user IDs.
Provides fallback manual capture methods for unsupported providers.

When to use it

After implementing new LLM features in an application.
During code reviews to ensure proper observability coverage.
When setting up AI monitoring for the first time in a project.
When needing to track costs and performance metrics for AI model usage.

Advantages

Standardizes observability across diverse LLM stacks.
Ensures minimal impact on existing application architecture.
Provides granular visibility into token usage and latency.
Follows best practices for environment variable management.
Supports a broad ecosystem of AI frameworks and providers.

Usage examples

“Add PostHog AI observability to my LangChain project.”: The agent will scan the project for LangChain dependencies, install the necessary PostHog packages, and instrument the LLM chains to report usage metrics to PostHog.
“I need to track costs and latency for my OpenAI API calls.”: The agent will identify the OpenAI implementation, configure the appropriate middleware or callback, and ensure all generations are captured with the required metadata.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/posthog/instrument-llm-analytics/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip instrument-llm-analytics.zip -d .agents/skills

Documentation by PostHog

Add PostHog AI Observability

Use this skill to add PostHog AI Observability that traces AI model usage in new or changed code. Use it after implementing LLM features or reviewing PRs to ensure all generations are captured with token counts, latency, and costs. If PostHog is not yet installed, this skill also covers initial SDK setup. Supports any provider or framework.

Supported providers: OpenAI, Azure OpenAI, Anthropic, Google, Cohere, Mistral, Perplexity, DeepSeek, Groq, Together AI, Fireworks AI, xAI, Cerebras, Hugging Face, Ollama, OpenRouter.

Supported frameworks: LangChain, LlamaIndex, CrewAI, AutoGen, DSPy, LangGraph, Pydantic AI, Vercel AI, LiteLLM, Instructor, Semantic Kernel, Mirascope, Mastra, SmolAgents, OpenAI Agents.

Proxy/gateway: Portkey, Helicone.

Instructions

Follow these steps IN ORDER:

STEP 1: Analyze the codebase and detect the LLM stack.

Look for LLM provider SDKs (openai, anthropic, google-generativeai, etc.) and AI frameworks (langchain, llamaindex, crewai, etc.) in dependency files and imports.
Look for lockfiles to determine the package manager.
Check for existing PostHog or observability setup. If PostHog is already installed and LLM tracing is configured, skip to STEP 4 to add tracing for any new LLM calls.

STEP 2: Research instrumentation. (Skip if PostHog LLM tracing is already set up.) 2.1. Find the reference file below that matches the detected provider or framework — it is the source of truth for callback setup, middleware configuration, and event capture. Read it now. 2.2. If no reference matches, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.

STEP 3: Install the PostHog SDK. (Skip if PostHog is already set up.)

Add the PostHog SDK and any required callback/integration packages.
Do not manually edit dependency files — use the package manager's install command.

STEP 4: Add LLM tracing.

Instrument LLM calls to capture input tokens, output tokens, model name, latency, and costs for every generation.
Follow the provider-specific reference for the exact callback/middleware setup.
Do not alter the fundamental architecture of existing files. Make additions minimal and targeted.
You must read a file immediately before attempting to write it.

STEP 5: Link to users.

Associate LLM generations with identified users via distinct IDs when possible.

STEP 6: Set up environment variables.

Check if the project already has PostHog environment variables configured (e.g. in .env, .env.local, or framework-specific env files). If valid values already exist, skip this step.
If the PostHog project token is missing, use the PostHog MCP server's projects-get tool to retrieve the project's api_token. If multiple projects are returned, ask the user which project to use. If the MCP server is not connected or not authenticated, ask the user for their PostHog project token instead.
For the PostHog host URL: check the projects-get MCP response for a region field — US maps to https://us.i.posthog.com, EU maps to https://eu.i.posthog.com. If the region is not available from the MCP response or from existing project configuration, ask the user: "Are you on PostHog US Cloud or EU Cloud?" Do not assume US Cloud.
Write these values to the appropriate env file using the framework's naming convention.
Reference these environment variables in code instead of hardcoding them.

Reference files

references/openai.md - Openai observability installation
references/azure-openai.md - Azure openai observability installation
references/README.md - PostHog.ai for .net
references/anthropic.md - Anthropic ai observability installation
references/google.md - Google ai observability installation
references/cohere.md - Cohere ai observability installation
references/mistral.md - Mistral ai observability installation
references/perplexity.md - Perplexity ai observability installation
references/deepseek.md - Deepseek ai observability installation
references/groq.md - Groq ai observability installation
references/together-ai.md - Together ai observability installation
references/fireworks-ai.md - Fireworks ai observability installation
references/xai.md - Xai observability installation
references/cerebras.md - Cerebras ai observability installation
references/hugging-face.md - Hugging face ai observability installation
references/ollama.md - Ollama ai observability installation
references/openrouter.md - Openrouter ai observability installation
references/langchain.md - Langchain ai observability installation
references/llamaindex.md - Llamaindex ai observability installation
references/crewai.md - Crewai observability installation
references/autogen.md - Autogen ai observability installation
references/dspy.md - Dspy ai observability installation
references/langgraph.md - Langgraph ai observability installation
references/pydantic-ai.md - Pydantic ai observability installation
references/vercel-ai.md - Vercel ai SDK observability installation
references/litellm.md - Litellm ai observability installation
references/instructor.md - Instructor ai observability installation
references/semantic-kernel.md - Semantic kernel ai observability installation
references/mirascope.md - Mirascope ai observability installation
references/mastra.md - Mastra ai observability installation
references/smolagents.md - Smolagents ai observability installation
references/openai-agents.md - Openai agents SDK observability installation
references/portkey.md - Portkey ai observability installation
references/helicone.md - Helicone ai observability installation
references/manual-capture.md - Manual capture ai observability installation
references/basics.md - Ai observability basics
references/traces.md - Traces
references/calculating-costs.md - Calculating llm costs
references/COMMANDMENTS.md - Framework-specific rules the integration must follow

Each provider reference contains installation instructions, SDK setup, and code examples specific to that provider or framework. Find the reference that matches the user's stack.

If the user's provider isn't listed, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.

Key principles

Environment variables: Always use environment variables for PostHog and LLM provider keys. Never hardcode them.
Minimal changes: Add LLM analytics alongside existing LLM calls. Don't replace or restructure existing code.
Trace all generations: Capture input tokens, output tokens, model name, latency, and costs for every LLM call.
Link to users: Associate LLM generations with identified users via distinct IDs when possible.
One provider at a time: Only instrument the provider(s) the user is actually using. Don't add instrumentation for providers not present in the codebase.

Files

ATTRIBUTION.mdAdded by AutoGineer958 B
LICENSELicence file added by AutoGineer1.0 KiB
SKILL.md7.3 KiB
references/COMMANDMENTS.md1.5 KiB
references/README.md543 B
references/anthropic.md27.1 KiB
references/autogen.md7.7 KiB
references/azure-openai.md10.7 KiB
references/basics.md5.6 KiB
references/calculating-costs.md8.8 KiB
references/cerebras.md8.3 KiB
references/cohere.md8.3 KiB
references/crewai.md7.2 KiB
references/deepseek.md8.2 KiB
references/dspy.md7.4 KiB
references/fireworks-ai.md8.4 KiB
references/google.md10.0 KiB
references/groq.md8.2 KiB
references/helicone.md8.3 KiB
references/hugging-face.md8.4 KiB
references/instructor.md10.5 KiB
references/langchain.md8.5 KiB
references/langgraph.md8.3 KiB
references/litellm.md9.5 KiB
references/llamaindex.md8.1 KiB
references/manual-capture.md23.7 KiB
references/mastra.md6.7 KiB
references/mirascope.md8.0 KiB
references/mistral.md8.3 KiB
references/ollama.md8.2 KiB
references/openai-agents.md8.2 KiB
references/openai.md10.2 KiB
references/openrouter.md8.3 KiB
references/perplexity.md8.3 KiB
references/portkey.md8.2 KiB
references/pydantic-ai.md7.4 KiB
references/semantic-kernel.md7.7 KiB
references/smolagents.md7.5 KiB
references/together-ai.md8.3 KiB
references/traces.md8.5 KiB
references/vercel-ai.md10.0 KiB
references/xai.md8.2 KiB

references/helicone.md

AI agents: this is one page from PostHog's docs. Full index of Markdown docs for LLMs: https://posthog.com/llms.txt

Helicone AI Observability installation

Canonical URL: https://posthog.com/docs/ai-observability/installation/helicone

Cloud region: This page uses US Cloud hosts. For an EU Cloud project, use https://eu.i.posthog.com instead of https://us.i.posthog.com (ingestion host), and use https://eu.posthog.com instead of https://app.posthog.com (app host for links).

Install dependencies

Required

Full working examples

See the complete Node.js and Python examples on GitHub.

Install the PostHog SDK and the OpenAI SDK.

Python

pip install posthog openai

Node

npm install @posthog/ai posthog-node openai

Go

PostHog doesn't have a Go wrapper for this SDK yet. Use the [OpenTelemetry integration](https://posthog.com/docs/ai-observability/installation/opentelemetry) instead.

Configure PostHog

Required

Create a PostHog client, then swap in PostHog's OpenAI wrapper, pointed at Helicone.

Python

from posthog import Posthog
from posthog.ai.openai import OpenAI
import time, uuid, json

posthog = Posthog("<ph_project_token>", host="https://us.i.posthog.com")

client = OpenAI(
    base_url="https://ai-gateway.helicone.ai/",
    api_key="<helicone_api_key>",
    posthog_client=posthog,
)

Node

import { OpenAI } from '@posthog/ai/openai'
import { PostHog } from 'posthog-node'

const posthog = new PostHog('<ph_project_token>', { host: 'https://us.i.posthog.com' })

const client = new OpenAI({
  baseURL: 'https://ai-gateway.helicone.ai/',
  apiKey: '<helicone_api_key>',
  posthog,
})

Call Helicone

Required

When you use the wrapped client to call Helicone, PostHog automatically captures an $ai_generation event.

Python

trace_id = str(uuid.uuid4())

response = client.chat.completions.create(
    model="gpt-5-mini",
    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
    tools=tools,
    posthog_distinct_id="user_123",
    posthog_trace_id=trace_id,
    posthog_properties={
        "$ai_session_id": "conversation-abc",
        "$ai_provider": "helicone",
    },
)

Node

const traceId = crypto.randomUUID()

const response = await client.chat.completions.create({
  model: 'gpt-5-mini',
  messages: [{ role: 'user', content: "What's the weather in Paris?" }],
  tools,
  posthogDistinctId: 'user_123',
  posthogTraceId: traceId,
  posthogProperties: {
    $ai_session_id: 'conversation-abc',
    $ai_provider: 'helicone',
  },
})

Note: If you want to capture LLM events anonymously, omit posthog_distinct_id from the call. See our docs on anonymous vs identified events to learn more.

You can expect captured $ai_generation events to have the following properties:

Property	Description
`$ai_model`	The specific model, like `gpt-5-mini` or `claude-4-sonnet`
`$ai_latency`	The latency of the LLM call in seconds
`$ai_time_to_first_token`	Time to first token in seconds (streaming only)
`$ai_tools`	Tools and functions available to the LLM
`$ai_input`	List of messages sent to the LLM
`$ai_input_tokens`	The number of tokens in the input (often found in response.usage)
`$ai_output_choices`	List of response choices from the LLM
`$ai_output_tokens`	The number of tokens in the output (often found in `response.usage`)
`$ai_total_cost_usd`	The total cost in USD (input + output)
[...]	See full list of properties

Capture tool calls as spans

Optional

For standard responses, the posthog client captures it as a generation. For all tool calls, you must manually capture them as $ai_span events.

Python

for call in response.choices[0].message.tool_calls or []:
    start = time.time()
    result = run_tool(call.function.name, json.loads(call.function.arguments))

    posthog.capture(
        distinct_id="user_123",
        event="$ai_span",
        properties={
            "$ai_trace_id": trace_id,
            "$ai_session_id": "conversation-abc",
            "$ai_span_id": str(uuid.uuid4()),
            "$ai_span_name": call.function.name,
            "$ai_input_state": call.function.arguments,
            "$ai_output_state": result,
            "$ai_latency": time.time() - start,
        },
    )

Node

for (const call of response.choices[0].message.tool_calls ?? []) {
  const start = Date.now()
  const result = await runTool(call.function.name, JSON.parse(call.function.arguments))

  posthog.capture({
    distinctId: 'user_123',
    event: '$ai_span',
    properties: {
      $ai_trace_id: traceId,
      $ai_session_id: 'conversation-abc',
      $ai_span_id: crypto.randomUUID(),
      $ai_span_name: call.function.name,
      $ai_input_state: call.function.arguments,
      $ai_output_state: result,
      $ai_latency: (Date.now() - start) / 1000,
    },
  })
}

See spans for the full list of span properties.

Verify traces and generations
Recommended
Confirm LLM events are being sent to PostHog
Let's make sure LLM events are being captured and sent to PostHog. Under AI Observability, you should see rows of data appear in the Traces and Generations tabs.
Image: LLM generations in PostHog
Check for LLM events in PostHog

Next steps

Recommended

Now that you're capturing AI conversations, continue with the resources below to learn what else AI Observability enables within the PostHog platform.

Resource	Description
Basics	Learn the basics of how LLM calls become events in PostHog.
Generations	Read about the `$ai_generation` event and its properties.
Traces	Explore the trace hierarchy and how to use it to debug LLM calls.
Spans	Review spans and their role in representing individual operations.
Anaylze LLM performance	Learn how to create dashboards to analyze LLM performance.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an MCP server

Security

What our scanner noticed· Low risk

Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

PostHog

Publisher

PostHog (Official)

Source

PostHog/skills · skills/omnibus/instrument-llm-analytics · commit a5d30e7

Licence

MIT License (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by PostHog.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download instrument-llm-analytics.zip

From the author

AutoGineer summary

What it does

When to use it

Advantages

Usage examples

Install

Documentation by PostHog

Add PostHog AI Observability

Instructions

Reference files

Key principles

Files

references/helicone.md

Helicone AI Observability installation

On this page

Install dependencies

Python

Node

Go

Configure PostHog

Python

Node

Call Helicone

Python

Node

Capture tool calls as spans

Python

Node

Verify traces and generations

Next steps

Requirements and compatibility

Security

Attribution and licence

Related skills

Claude API

Transformers

Open Notebook

Vision Sft

LLM Evaluation

Lora Qlora Recipes

Download ZIP