Instrument LLM Analytics
PostHogOfficialCreated by PostHog
- Skill
- Tool integration
- Data and AI
- MIT
- Low risk
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
This skill provides a structured approach to instrumenting LLM-powered applications with PostHog AI Observability. It guides the agent through detecting the existing LLM stack, installing necessary SDKs, and configuring event capture for token counts, latency, and costs. The skill supports a wide range of LLM providers and frameworks, ensuring that observability is added with minimal disruption to existing codebases.
What it does
- Detects LLM providers and frameworks in the codebase.
- Installs the PostHog SDK and required integration packages.
- Instruments LLM calls to capture tokens, latency, and costs.
- Configures environment variables for secure credential management.
- Associates LLM generations with specific user IDs.
- Provides fallback manual capture methods for unsupported providers.
When to use it
- After implementing new LLM features in an application.
- During code reviews to ensure proper observability coverage.
- When setting up AI monitoring for the first time in a project.
- When needing to track costs and performance metrics for AI model usage.
Advantages
- Standardizes observability across diverse LLM stacks.
- Ensures minimal impact on existing application architecture.
- Provides granular visibility into token usage and latency.
- Follows best practices for environment variable management.
- Supports a broad ecosystem of AI frameworks and providers.
Usage examples
- “Add PostHog AI observability to my LangChain project.”
- The agent will scan the project for LangChain dependencies, install the necessary PostHog packages, and instrument the LLM chains to report usage metrics to PostHog.
- “I need to track costs and latency for my OpenAI API calls.”
- The agent will identify the OpenAI implementation, configure the appropriate middleware or callback, and ensure all generations are captured with the required metadata.
Install
Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/posthog/instrument-llm-analytics/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip instrument-llm-analytics.zip -d .agents/skills
Documentation by PostHog
Add PostHog AI Observability
Use this skill to add PostHog AI Observability that traces AI model usage in new or changed code. Use it after implementing LLM features or reviewing PRs to ensure all generations are captured with token counts, latency, and costs. If PostHog is not yet installed, this skill also covers initial SDK setup. Supports any provider or framework.
Supported providers: OpenAI, Azure OpenAI, Anthropic, Google, Cohere, Mistral, Perplexity, DeepSeek, Groq, Together AI, Fireworks AI, xAI, Cerebras, Hugging Face, Ollama, OpenRouter.
Supported frameworks: LangChain, LlamaIndex, CrewAI, AutoGen, DSPy, LangGraph, Pydantic AI, Vercel AI, LiteLLM, Instructor, Semantic Kernel, Mirascope, Mastra, SmolAgents, OpenAI Agents.
Proxy/gateway: Portkey, Helicone.
Instructions
Follow these steps IN ORDER:
STEP 1: Analyze the codebase and detect the LLM stack.
Look for LLM provider SDKs (openai, anthropic, google-generativeai, etc.) and AI frameworks (langchain, llamaindex, crewai, etc.) in dependency files and imports.
Look for lockfiles to determine the package manager.
Check for existing PostHog or observability setup. If PostHog is already installed and LLM tracing is configured, skip to STEP 4 to add tracing for any new LLM calls.
STEP 2: Research instrumentation. (Skip if PostHog LLM tracing is already set up.) 2.1. Find the reference file below that matches the detected provider or framework — it is the source of truth for callback setup, middleware configuration, and event capture. Read it now. 2.2. If no reference matches, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.
STEP 3: Install the PostHog SDK. (Skip if PostHog is already set up.)
Add the PostHog SDK and any required callback/integration packages.
Do not manually edit dependency files — use the package manager's install command.
STEP 4: Add LLM tracing.
Instrument LLM calls to capture input tokens, output tokens, model name, latency, and costs for every generation.
Follow the provider-specific reference for the exact callback/middleware setup.
Do not alter the fundamental architecture of existing files. Make additions minimal and targeted.
You must read a file immediately before attempting to write it.
STEP 5: Link to users.
Associate LLM generations with identified users via distinct IDs when possible.
STEP 6: Set up environment variables.
Check if the project already has PostHog environment variables configured (e.g. in
.env,.env.local, or framework-specific env files). If valid values already exist, skip this step.If the PostHog project token is missing, use the PostHog MCP server's
projects-gettool to retrieve the project'sapi_token. If multiple projects are returned, ask the user which project to use. If the MCP server is not connected or not authenticated, ask the user for their PostHog project token instead.For the PostHog host URL: check the
projects-getMCP response for aregionfield —USmaps tohttps://us.i.posthog.com,EUmaps tohttps://eu.i.posthog.com. If the region is not available from the MCP response or from existing project configuration, ask the user: "Are you on PostHog US Cloud or EU Cloud?" Do not assume US Cloud.Write these values to the appropriate env file using the framework's naming convention.
Reference these environment variables in code instead of hardcoding them.
Reference files
references/openai.md- Openai observability installationreferences/azure-openai.md- Azure openai observability installationreferences/README.md- PostHog.ai for .netreferences/anthropic.md- Anthropic ai observability installationreferences/google.md- Google ai observability installationreferences/cohere.md- Cohere ai observability installationreferences/mistral.md- Mistral ai observability installationreferences/perplexity.md- Perplexity ai observability installationreferences/deepseek.md- Deepseek ai observability installationreferences/groq.md- Groq ai observability installationreferences/together-ai.md- Together ai observability installationreferences/fireworks-ai.md- Fireworks ai observability installationreferences/xai.md- Xai observability installationreferences/cerebras.md- Cerebras ai observability installationreferences/hugging-face.md- Hugging face ai observability installationreferences/ollama.md- Ollama ai observability installationreferences/openrouter.md- Openrouter ai observability installationreferences/langchain.md- Langchain ai observability installationreferences/llamaindex.md- Llamaindex ai observability installationreferences/crewai.md- Crewai observability installationreferences/autogen.md- Autogen ai observability installationreferences/dspy.md- Dspy ai observability installationreferences/langgraph.md- Langgraph ai observability installationreferences/pydantic-ai.md- Pydantic ai observability installationreferences/vercel-ai.md- Vercel ai SDK observability installationreferences/litellm.md- Litellm ai observability installationreferences/instructor.md- Instructor ai observability installationreferences/semantic-kernel.md- Semantic kernel ai observability installationreferences/mirascope.md- Mirascope ai observability installationreferences/mastra.md- Mastra ai observability installationreferences/smolagents.md- Smolagents ai observability installationreferences/openai-agents.md- Openai agents SDK observability installationreferences/portkey.md- Portkey ai observability installationreferences/helicone.md- Helicone ai observability installationreferences/manual-capture.md- Manual capture ai observability installationreferences/basics.md- Ai observability basicsreferences/traces.md- Tracesreferences/calculating-costs.md- Calculating llm costsreferences/COMMANDMENTS.md- Framework-specific rules the integration must follow
Each provider reference contains installation instructions, SDK setup, and code examples specific to that provider or framework. Find the reference that matches the user's stack.
If the user's provider isn't listed, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.
Key principles
Environment variables: Always use environment variables for PostHog and LLM provider keys. Never hardcode them.
Minimal changes: Add LLM analytics alongside existing LLM calls. Don't replace or restructure existing code.
Trace all generations: Capture input tokens, output tokens, model name, latency, and costs for every LLM call.
Link to users: Associate LLM generations with identified users via distinct IDs when possible.
One provider at a time: Only instrument the provider(s) the user is actually using. Don't add instrumentation for providers not present in the codebase.
Files
- ATTRIBUTION.mdAdded by AutoGineer958 B
- LICENSELicence file added by AutoGineer1.0 KiB
- SKILL.md7.3 KiB
- references/COMMANDMENTS.md1.5 KiB
- references/README.md543 B
- references/anthropic.md27.1 KiB
- references/autogen.md7.7 KiB
- references/azure-openai.md10.7 KiB
- references/basics.md5.6 KiB
- references/calculating-costs.md8.8 KiB
- references/cerebras.md8.3 KiB
- references/cohere.md8.3 KiB
- references/crewai.md7.2 KiB
- references/deepseek.md8.2 KiB
- references/dspy.md7.4 KiB
- references/fireworks-ai.md8.4 KiB
- references/google.md10.0 KiB
- references/groq.md8.2 KiB
- references/helicone.md8.3 KiB
- references/hugging-face.md8.4 KiB
- references/instructor.md10.5 KiB
- references/langchain.md8.5 KiB
- references/langgraph.md8.3 KiB
- references/litellm.md9.5 KiB
- references/llamaindex.md8.1 KiB
- references/manual-capture.md23.7 KiB
- references/mastra.md6.7 KiB
- references/mirascope.md8.0 KiB
- references/mistral.md8.3 KiB
- references/ollama.md8.2 KiB
- references/openai-agents.md8.2 KiB
- references/openai.md10.2 KiB
- references/openrouter.md8.3 KiB
- references/perplexity.md8.3 KiB
- references/portkey.md8.2 KiB
- references/pydantic-ai.md7.4 KiB
- references/semantic-kernel.md7.7 KiB
- references/smolagents.md7.5 KiB
- references/together-ai.md8.3 KiB
- references/traces.md8.5 KiB
- references/vercel-ai.md10.0 KiB
- references/xai.md8.2 KiB
references/traces.md
AI agents: this is one page from PostHog's docs. Full index of Markdown docs for LLMs: https://posthog.com/llms.txt
Traces
Canonical URL: https://posthog.com/docs/ai-observability/traces
Cloud region: This page uses US Cloud hosts. For an EU Cloud project, use
https://eu.posthog.cominstead ofhttps://app.posthog.com(app host for links).
On this page
Sessions vs Traces
Trace timeline
Conversation display options
Tool calls
Sentiment classification
Search traces with PostHog AI
AI event hierarchy
Event properties
Traces are a collection of generations and spans that capture a full interaction between a user and an LLM. The traces tab lists them along with the properties autocaptured by PostHog like the person, total cost, total latency, and more.
Sessions vs Traces
Trace (
$ai_trace_id): Groups related generations and spans together. Required for all AI Observability events.Session (
$ai_session_id): Optional property that groups multiple traces together based on your chosen grouping strategy.
See the Sessions documentation for more details on how to use $ai_session_id.
If you also send OpenTelemetry spans to Distributed tracing, you can set $ai_trace_id to the trace ID of the surrounding span and correlate the two products. See Link traces to AI Observability.
Trace timeline
Clicking on a trace opens a timeline of the interaction with all the generation and span events. The trace timeline enables you to see the entire conversation, profiling details, and the individual generations and spans.
A trace presents LLM event data in a timeline, tree-structured view
Waterfall timeline
Above the trace tree, a waterfall timeline shows when each operation started and how long it ran. Each bar represents a generation, span, or embedding event, color-coded to match the tree view.
The waterfall helps you:
See latency breakdown — Identify which operations took the longest and where time was spent waiting.
Spot concurrency — Operations running in parallel appear on separate lanes. Parent-child connectors distinguish nested operations from concurrent siblings.
Find errors at a glance — Errored operations display a red border on their bar.
Navigate the trace — Click any bar to select the corresponding event in the tree view.
The time axis uses clock-friendly intervals (e.g. 15s, 30s, 1m, 2m). For deeply nested traces, the timeline caps at 6 visible lanes and scrolls vertically. You can collapse the timeline or expand it to full screen.
The waterfall also appears in the session view's steps drawer when inspecting a specific trace.
Conversation display options
When viewing a trace, you can control how conversation messages are displayed using the display options dropdown. The available options are:
Expand all - Shows the full content of all messages in the conversation
Expand user only - Expands only user messages, keeping system and assistant messages collapsed for easier scanning of user inputs
Collapse except output and last input - The default view that shows the model's output and the most recent user input, keeping earlier messages collapsed
Tool calls
Traces display any tools called by the generations within them, shown as tags in the traces list. This makes it easy to see which conversations involved tool use at a glance.
Sentiment classification
PostHog can classify the sentiment of user messages in a trace as negative, neutral, or positive using a local model — no data is sent to third-party services. Sentiment runs as a sentiment analysis evaluation that you configure. Each trace gets an overall sentiment label and score, with a per-generation and per-message breakdown. See Sentiment classification for more details.
Search traces with PostHog AI
PostHog AI can search and analyze your LLM traces using natural language. When you're on an AI Observability page, PostHog AI automatically switches to its AI Observability mode, giving it access to tools for searching traces by date range, model, cost, error status, and other properties.
Example prompts you can try:
"Show me recent LLM traces from the past week"
"What are the most expensive LLM calls from today?"
"Find traces with errors in the last 30 days"
"What's happening in my most expensive trace?"
PostHog AI returns trace details including name, latency, cost, token counts, and error count. It can also read individual traces to provide a detailed summary of what happened.
AI event hierarchy
flowchart TD S["<strong>$ai_session_id</strong><br/>(optional)"] A[<strong>$ai_trace</strong>] A2[<strong>$ai_trace</strong>] B[<strong>$ai_generation</strong>] C@{ shape: processes, label: "<strong>$ai_spans</strong>" } D[<strong>$ai_generation</strong>] E@{ shape: processes, label: "<strong>$ai_spans</strong>" } F[<strong>$ai_generation</strong>] S -.-> A S -.-> A2 A --> B A --> C C --> D C --> E E --> F
Traces consist of the following event hierarchy:
(Optional) A session (
$ai_session_id) can group multiple traces together.A trace (
$ai_trace_id) is the top-level required grouping for LLM events.A trace can contain multiple spans and generations.
A span can be the parent of other spans.
A generation can be the child of a span or trace.
Event properties
A trace is a group that contains multiple spans, generations, and embeddings. Traces can be manually sent as events or appear as pseudo-events automatically created from child events.
Event name: $ai_trace
Core properties
| Property | Description |
|---|---|
$ai_trace_id | The trace ID (a UUID to group related AI events together) Must contain only letters, numbers, and special characters: -, _, ~, ., @, (, ), !, ', :, | Example: d9222e05-8708-41b8-98ea-d4a21849e761 |
$ai_session_id | (Optional) Groups related traces into a session, which is what the Sessions tab reads. Set it if your product has multi-turn conversations. A workload that finishes in a single trace does not need it. Send it as null to say so explicitly, which tells the instrumentation checklist the workload is complete rather than missing a session id. Must contain only letters, numbers, and special characters: -, _, ~, ., @, (, ), !, ', :, | Example: session-abc-123, conv-user-456 |
$ai_input_state | The input of the whole trace Example: json [ { "role": "user", "content": "What's the weather in SF?" } ] or any JSON-serializable state |
$ai_output_state | The output of the whole trace Example: json [ { "role": "assistant", "content": "The weather in San Francisco is..." } ] or any JSON-serializable state |
$ai_latency | (Optional) The latency of the trace in seconds |
$ai_span_name | (Optional) The name of the trace Example: chat_completion, rag_pipeline |
$ai_is_error | (Optional) Boolean to indicate if the trace encountered an error |
$ai_error | (Optional) The error message or object if the trace failed |
Pseudo-trace Events
When you send generation ($ai_generation), span ($ai_span), or embedding ($ai_embedding) events with a $ai_trace_id, PostHog automatically creates a pseudo-trace event that appears in the dashboard as a parent grouping. These pseudo-traces:
Are not actual events in your data
Automatically aggregate metrics from child events (latency, tokens, costs)
Provide a hierarchical view of your AI operations
Do not require sending an explicit
$ai_traceevent
This means you can either:
Send explicit
$ai_traceevents to control the trace metadataLet PostHog automatically create pseudo-traces from your generation/span events
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Needs an MCP server
Security
What our scanner noticed· Low risk
- Instructions that use the network
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Created by
- PostHog
- Publisher
- PostHog (Official)
- Source
- PostHog/skills · skills/omnibus/instrument-llm-analytics · commit a5d30e7
- Licence
- MIT License (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
AutoGineer is not affiliated with or endorsed by PostHog.
Related skills
Claude API
AnthropicOfficial
Comprehensive reference for integrating Anthropic's Claude API, SDKs, and Managed Agents into your applications.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreApache-2.0Medium riskHas scriptsGitHub stars of the source repository188.6k
Transformers
K-Dense· Community
Integrate Hugging Face Transformers for model loading, inference, and fine-tuning across NLP, vision, and audio tasks.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreMITLow riskGitHub stars of the source repository56.4k
Open Notebook
K-Dense· Community
Integrate with the self-hosted Open Notebook platform to automate research ingestion, context-aware chat, and AI-driven analysis.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreMITLow riskHas scriptsGitHub stars of the source repository56.4k
Vision Sft
Seth Hobson· Community
A structured workflow for supervised fine-tuning of vision-language models, focusing on adapter configuration and silent failure prevention.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository42.3k
LLM Evaluation
Seth Hobson· Community
A comprehensive framework for evaluating LLM application performance using automated metrics, human feedback, and LLM-as-a-judge techniques.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository42.3k
Lora Qlora Recipes
Seth Hobson· Community
Provides configuration recipes and best-practice hyperparameters for LoRA and QLoRA supervised fine-tuning.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository42.3k
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download instrument-llm-analytics.zip