AutoGineer
Generated banner for Omnibus Instrument LLM Analytics
Generated logo for Omnibus Instrument LLM Analytics

Omnibus Instrument LLM Analytics

PostHogOfficialCreated by PostHog

  • Skill
  • Tool integration
  • Data and AI
  • MIT
  • Low risk
Download ZIP

From the author

Add PostHog LLM analytics to trace AI model usage. Use after implementing LLM features or reviewing PRs to ensure all generations are captured with token counts, latency, and costs. Also handles initial PostHog SDK setup if not yet installed.
— the description by PostHog

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured approach to instrumenting AI applications with PostHog analytics. It guides developers through detecting their LLM stack, installing the necessary PostHog SDKs, and configuring wrappers or middleware to capture detailed telemetry for every LLM generation. By following provider-specific references, users can ensure consistent tracking of token counts, latency, and costs across a wide range of LLM providers and frameworks without altering their core application architecture.

What it does

  • Detects existing LLM providers and frameworks in the codebase.
  • Installs and configures the PostHog SDK for LLM observability.
  • Instruments LLM calls to capture input/output tokens, latency, and costs.
  • Associates LLM generations with specific user IDs for better tracking.
  • Provides fallback manual capture methods for unsupported providers.
  • Configures environment variables for secure API key management.

When to use it

  • After implementing new LLM features in an application.
  • When reviewing pull requests that introduce or modify AI model interactions.
  • When you need to monitor token usage and costs for AI features.
  • When setting up observability for a new AI-powered project.

Advantages

  • Centralized visibility into AI model performance and costs.
  • Minimal code changes required through wrapper-based instrumentation.
  • Supports a broad ecosystem of LLM providers and frameworks.
  • Enables user-level attribution for AI interactions.
  • Maintains security by enforcing environment variable usage.

Usage examples

“Instrument my existing Anthropic Claude implementation with PostHog analytics.”
The agent will identify the Anthropic SDK usage, install the required PostHog packages, and wrap the client to automatically capture generation events.
“Set up PostHog tracking for my AutoGen agents.”
The agent will configure the PostHog OpenAI wrapper and integrate it into the AutoGen model client to start logging agentic LLM calls.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/posthog/omnibus-instrument-llm-analytics/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip omnibus-instrument-llm-analytics.zip -d .agents/skills

Documentation by PostHog

Add PostHog LLM analytics

Use this skill to add PostHog LLM analytics that trace AI model usage in new or changed code. Use it after implementing LLM features or reviewing PRs to ensure all generations are captured with token counts, latency, and costs. If PostHog is not yet installed, this skill also covers initial SDK setup. Supports any provider or framework.

Supported providers: OpenAI, Azure OpenAI, Anthropic, Google, Cohere, Mistral, Perplexity, DeepSeek, Groq, Together AI, Fireworks AI, xAI, Cerebras, Hugging Face, Ollama, OpenRouter.

Supported frameworks: LangChain, LlamaIndex, CrewAI, AutoGen, DSPy, LangGraph, Pydantic AI, Vercel AI, LiteLLM, Instructor, Semantic Kernel, Mirascope, Mastra, SmolAgents, OpenAI Agents.

Proxy/gateway: Portkey, Helicone.

Instructions

Follow these steps IN ORDER:

STEP 1: Analyze the codebase and detect the LLM stack.

  • Look for LLM provider SDKs (openai, anthropic, google-generativeai, etc.) and AI frameworks (langchain, llamaindex, crewai, etc.) in dependency files and imports.

  • Look for lockfiles to determine the package manager.

  • Check for existing PostHog or observability setup. If PostHog is already installed and LLM tracing is configured, skip to STEP 4 to add tracing for any new LLM calls.

STEP 2: Research instrumentation. (Skip if PostHog LLM tracing is already set up.) 2.1. Find the reference file below that matches the detected provider or framework — it is the source of truth for callback setup, middleware configuration, and event capture. Read it now. 2.2. If no reference matches, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.

STEP 3: Install the PostHog SDK. (Skip if PostHog is already set up.)

  • Add the PostHog SDK and any required callback/integration packages.

  • Do not manually edit dependency files — use the package manager's install command.

  • Always install packages as a background task. Don't await completion; proceed with other work immediately.

STEP 4: Add LLM tracing.

  • Instrument LLM calls to capture input tokens, output tokens, model name, latency, and costs for every generation.

  • Follow the provider-specific reference for the exact callback/middleware setup.

  • Do not alter the fundamental architecture of existing files. Make additions minimal and targeted.

  • You must read a file immediately before attempting to write it.

STEP 5: Link to users.

  • Associate LLM generations with identified users via distinct IDs when possible.

STEP 6: Set up environment variables.

  • If an env-file-tools MCP server is connected, use check_env_keys then set_env_values to configure the PostHog API key and host.

  • Reference these environment variables in code instead of hardcoding them.

Reference files

  • references/openai.md - Openai llm analytics installation - docs

  • references/azure-openai.md - Azure openai llm analytics installation - docs

  • references/anthropic.md - Anthropic llm analytics installation - docs

  • references/google.md - Google llm analytics installation - docs

  • references/cohere.md - Cohere llm analytics installation - docs

  • references/mistral.md - Mistral llm analytics installation - docs

  • references/perplexity.md - Perplexity llm analytics installation - docs

  • references/deepseek.md - Deepseek llm analytics installation - docs

  • references/groq.md - Groq llm analytics installation - docs

  • references/together-ai.md - Together ai llm analytics installation - docs

  • references/fireworks-ai.md - Fireworks ai llm analytics installation - docs

  • references/xai.md - Xai llm analytics installation - docs

  • references/cerebras.md - Cerebras llm analytics installation - docs

  • references/hugging-face.md - Hugging face llm analytics installation - docs

  • references/ollama.md - Ollama llm analytics installation - docs

  • references/openrouter.md - Openrouter llm analytics installation - docs

  • references/langchain.md - Langchain llm analytics installation - docs

  • references/llamaindex.md - Llamaindex llm analytics installation - docs

  • references/crewai.md - Crewai llm analytics installation - docs

  • references/autogen.md - Autogen llm analytics installation - docs

  • references/dspy.md - Dspy llm analytics installation - docs

  • references/langgraph.md - Langgraph llm analytics installation - docs

  • references/pydantic-ai.md - Pydantic ai llm analytics installation - docs

  • references/vercel-ai.md - Vercel ai SDK llm analytics installation - docs

  • references/litellm.md - Litellm llm analytics installation - docs

  • references/instructor.md - Instructor llm analytics installation - docs

  • references/semantic-kernel.md - Semantic kernel llm analytics installation - docs

  • references/mirascope.md - Mirascope llm analytics installation - docs

  • references/mastra.md - Mastra llm analytics installation - docs

  • references/smolagents.md - Smolagents llm analytics installation - docs

  • references/openai-agents.md - Openai agents SDK llm analytics installation - docs

  • references/portkey.md - Portkey llm analytics installation - docs

  • references/helicone.md - Helicone llm analytics installation - docs

  • references/manual-capture.md - Manual capture llm analytics installation - docs

  • references/basics.md - Llm analytics basics - docs

  • references/traces.md - Traces - docs

  • references/calculating-costs.md - Calculating llm costs - docs

Each provider reference contains installation instructions, SDK setup, and code examples specific to that provider or framework. Find the reference that matches the user's stack.

If the user's provider isn't listed, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.

Key principles

  • Environment variables: Always use environment variables for PostHog and LLM provider keys. Never hardcode them.

  • Minimal changes: Add LLM analytics alongside existing LLM calls. Don't replace or restructure existing code.

  • Trace all generations: Capture input tokens, output tokens, model name, latency, and costs for every LLM call.

  • Link to users: Associate LLM generations with identified users via distinct IDs when possible.

  • One provider at a time: Only instrument the provider(s) the user is actually using. Don't add instrumentation for providers not present in the codebase.

Files

references/traces.md

Traces - Docs

Traces are a collection of generations and spans that capture a full interaction between a user and an LLM. The traces tab lists them along with the properties autocaptured by PostHog like the person, total cost, total latency, and more.

Sessions vs Traces

  • Trace ($ai_trace_id): Groups related generations and spans together. Required for all LLM analytics events.

  • Session ($ai_session_id): Optional property that groups multiple traces together based on your chosen grouping strategy.

See the Sessions documentation for more details on how to use $ai_session_id.

Trace timeline

Clicking on a trace opens a timeline of the interaction with all the generation and span events. The trace timeline enables you to see the entire conversation, profiling details, and the individual generations and spans.

Image: LLM tracesImage: LLM traces

A trace presents LLM event data in a timeline, tree-structured view

Conversation display options

When viewing a trace, you can control how conversation messages are displayed using the display options dropdown. The available options are:

  • Expand all - Shows the full content of all messages in the conversation

  • Expand user only - Expands only user messages, keeping system and assistant messages collapsed for easier scanning of user inputs

  • Collapse except output and last input - The default view that shows the model's output and the most recent user input, keeping earlier messages collapsed

Tool calls

Traces display any tools called by the generations within them, shown as tags in the traces list. This makes it easy to see which conversations involved tool use at a glance.

Sentiment classification

PostHog can classify the sentiment of user messages in a trace as negative, neutral, or positive. Sentiment is computed on-demand using a local model when you view a trace — no data is sent to third-party services. Each trace gets an overall sentiment label and score, with a per-generation and per-message breakdown. See Sentiment classification for more details.

Search traces with PostHog AI

PostHog AI can search and analyze your LLM traces using natural language. When you're on an LLM Analytics page, PostHog AI automatically switches to its LLM analytics mode, giving it access to tools for searching traces by date range, model, cost, error status, and other properties.

Example prompts you can try:

  • "Show me recent LLM traces from the past week"

  • "What are the most expensive LLM calls from today?"

  • "Find traces with errors in the last 30 days"

  • "What's happening in my most expensive trace?"

PostHog AI returns trace details including name, latency, cost, token counts, and error count. It can also read individual traces to provide a detailed summary of what happened.

AI event hierarchy

flowchart TD S["<strong>$ai_session_id</strong><br/>(optional)"] A[<strong>$ai_trace</strong>] A2[<strong>$ai_trace</strong>] B[<strong>$ai_generation</strong>] C@{ shape: processes, label: "<strong>$ai_spans</strong>" } D[<strong>$ai_generation</strong>] E@{ shape: processes, label: "<strong>$ai_spans</strong>" } F[<strong>$ai_generation</strong>] S -.-> A S -.-> A2 A --> B A --> C C --> D C --> E E --> F

Traces consist of the following event hierarchy:

  1. (Optional) A session ($ai_session_id) can group multiple traces together.

  2. A trace ($ai_trace_id) is the top-level required grouping for LLM events.

  3. A trace can contain multiple spans and generations.

  4. A span can be the parent of other spans.

  5. A generation can be the child of a span or trace.

Event properties

A trace is a group that contains multiple spans, generations, and embeddings. Traces can be manually sent as events or appear as pseudo-events automatically created from child events.

Event name: $ai_trace

Core properties
PropertyDescription
$ai_trace_idThe trace ID (a UUID to group related AI events together)Must contain only letters, numbers, and special characters: -, _, ~, ., @, (, ), !, ', :, |Example: d9222e05-8708-41b8-98ea-d4a21849e761
$ai_session_id(Optional) Groups related traces together. Use this to organize traces by whatever grouping makes sense for your application (user sessions, workflows, conversations, or other logical boundaries).Example: session-abc-123, conv-user-456
$ai_latency(Optional) The latency of the trace in seconds
$ai_span_name(Optional) The name of the traceExample: chat_completion, rag_pipeline
$ai_is_error(Optional) Boolean to indicate if the trace encountered an error
$ai_error(Optional) The error message or object if the trace failed
Pseudo-trace Events

When you send generation ($ai_generation), span ($ai_span), or embedding ($ai_embedding) events with a $ai_trace_id, PostHog automatically creates a pseudo-trace event that appears in the dashboard as a parent grouping. These pseudo-traces:

  • Are not actual events in your data

  • Automatically aggregate metrics from child events (latency, tokens, costs)

  • Provide a hierarchical view of your AI operations

  • Do not require sending an explicit $ai_trace event

This means you can either:

  1. Send explicit $ai_trace events to control the trace metadata

  2. Let PostHog automatically create pseudo-traces from your generation/span events

Community questions

Ask a question

Was this page useful?

HelpfulCould be better

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an MCP server, Needs an API key

Security

What our scanner noticed· Low risk

  • Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
PostHog
Publisher
PostHog (Official)
Source
PostHog/skills · skills/posthog/llm-analytics/skills/all · commit a5d30e7
Licence
MIT License (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced
Also distributed in
PostHog/skills

AutoGineer is not affiliated with or endorsed by PostHog.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download omnibus-instrument-llm-analytics.zip