Instrument LLM Analytics

PostHogOfficialCreated by PostHog

Skill
Tool integration
Data and AI
MIT
Low risk

From the author

Add PostHog AI Observability to trace AI model usage. Use after implementing LLM features or reviewing PRs to ensure all generations are captured with token counts, latency, and costs. Also handles initial PostHog SDK setup if not yet installed.

— the description by PostHog

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured approach to instrumenting LLM-powered applications with PostHog AI Observability. It guides the agent through detecting the existing LLM stack, installing necessary SDKs, and configuring event capture for token counts, latency, and costs. The skill supports a wide range of LLM providers and frameworks, ensuring that observability is added with minimal disruption to existing codebases.

What it does

Detects LLM providers and frameworks in the codebase.
Installs the PostHog SDK and required integration packages.
Instruments LLM calls to capture tokens, latency, and costs.
Configures environment variables for secure credential management.
Associates LLM generations with specific user IDs.
Provides fallback manual capture methods for unsupported providers.

When to use it

After implementing new LLM features in an application.
During code reviews to ensure proper observability coverage.
When setting up AI monitoring for the first time in a project.
When needing to track costs and performance metrics for AI model usage.

Advantages

Standardizes observability across diverse LLM stacks.
Ensures minimal impact on existing application architecture.
Provides granular visibility into token usage and latency.
Follows best practices for environment variable management.
Supports a broad ecosystem of AI frameworks and providers.

Usage examples

“Add PostHog AI observability to my LangChain project.”: The agent will scan the project for LangChain dependencies, install the necessary PostHog packages, and instrument the LLM chains to report usage metrics to PostHog.
“I need to track costs and latency for my OpenAI API calls.”: The agent will identify the OpenAI implementation, configure the appropriate middleware or callback, and ensure all generations are captured with the required metadata.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/posthog/instrument-llm-analytics/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip instrument-llm-analytics.zip -d .agents/skills

Documentation by PostHog

Add PostHog AI Observability

Use this skill to add PostHog AI Observability that traces AI model usage in new or changed code. Use it after implementing LLM features or reviewing PRs to ensure all generations are captured with token counts, latency, and costs. If PostHog is not yet installed, this skill also covers initial SDK setup. Supports any provider or framework.

Supported providers: OpenAI, Azure OpenAI, Anthropic, Google, Cohere, Mistral, Perplexity, DeepSeek, Groq, Together AI, Fireworks AI, xAI, Cerebras, Hugging Face, Ollama, OpenRouter.

Supported frameworks: LangChain, LlamaIndex, CrewAI, AutoGen, DSPy, LangGraph, Pydantic AI, Vercel AI, LiteLLM, Instructor, Semantic Kernel, Mirascope, Mastra, SmolAgents, OpenAI Agents.

Proxy/gateway: Portkey, Helicone.

Instructions

Follow these steps IN ORDER:

STEP 1: Analyze the codebase and detect the LLM stack.

Look for LLM provider SDKs (openai, anthropic, google-generativeai, etc.) and AI frameworks (langchain, llamaindex, crewai, etc.) in dependency files and imports.
Look for lockfiles to determine the package manager.
Check for existing PostHog or observability setup. If PostHog is already installed and LLM tracing is configured, skip to STEP 4 to add tracing for any new LLM calls.

STEP 2: Research instrumentation. (Skip if PostHog LLM tracing is already set up.) 2.1. Find the reference file below that matches the detected provider or framework — it is the source of truth for callback setup, middleware configuration, and event capture. Read it now. 2.2. If no reference matches, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.

STEP 3: Install the PostHog SDK. (Skip if PostHog is already set up.)

Add the PostHog SDK and any required callback/integration packages.
Do not manually edit dependency files — use the package manager's install command.

STEP 4: Add LLM tracing.

Instrument LLM calls to capture input tokens, output tokens, model name, latency, and costs for every generation.
Follow the provider-specific reference for the exact callback/middleware setup.
Do not alter the fundamental architecture of existing files. Make additions minimal and targeted.
You must read a file immediately before attempting to write it.

STEP 5: Link to users.

Associate LLM generations with identified users via distinct IDs when possible.

STEP 6: Set up environment variables.

Check if the project already has PostHog environment variables configured (e.g. in .env, .env.local, or framework-specific env files). If valid values already exist, skip this step.
If the PostHog project token is missing, use the PostHog MCP server's projects-get tool to retrieve the project's api_token. If multiple projects are returned, ask the user which project to use. If the MCP server is not connected or not authenticated, ask the user for their PostHog project token instead.
For the PostHog host URL: check the projects-get MCP response for a region field — US maps to https://us.i.posthog.com, EU maps to https://eu.i.posthog.com. If the region is not available from the MCP response or from existing project configuration, ask the user: "Are you on PostHog US Cloud or EU Cloud?" Do not assume US Cloud.
Write these values to the appropriate env file using the framework's naming convention.
Reference these environment variables in code instead of hardcoding them.

Reference files

references/openai.md - Openai observability installation
references/azure-openai.md - Azure openai observability installation
references/README.md - PostHog.ai for .net
references/anthropic.md - Anthropic ai observability installation
references/google.md - Google ai observability installation
references/cohere.md - Cohere ai observability installation
references/mistral.md - Mistral ai observability installation
references/perplexity.md - Perplexity ai observability installation
references/deepseek.md - Deepseek ai observability installation
references/groq.md - Groq ai observability installation
references/together-ai.md - Together ai observability installation
references/fireworks-ai.md - Fireworks ai observability installation
references/xai.md - Xai observability installation
references/cerebras.md - Cerebras ai observability installation
references/hugging-face.md - Hugging face ai observability installation
references/ollama.md - Ollama ai observability installation
references/openrouter.md - Openrouter ai observability installation
references/langchain.md - Langchain ai observability installation
references/llamaindex.md - Llamaindex ai observability installation
references/crewai.md - Crewai observability installation
references/autogen.md - Autogen ai observability installation
references/dspy.md - Dspy ai observability installation
references/langgraph.md - Langgraph ai observability installation
references/pydantic-ai.md - Pydantic ai observability installation
references/vercel-ai.md - Vercel ai SDK observability installation
references/litellm.md - Litellm ai observability installation
references/instructor.md - Instructor ai observability installation
references/semantic-kernel.md - Semantic kernel ai observability installation
references/mirascope.md - Mirascope ai observability installation
references/mastra.md - Mastra ai observability installation
references/smolagents.md - Smolagents ai observability installation
references/openai-agents.md - Openai agents SDK observability installation
references/portkey.md - Portkey ai observability installation
references/helicone.md - Helicone ai observability installation
references/manual-capture.md - Manual capture ai observability installation
references/basics.md - Ai observability basics
references/traces.md - Traces
references/calculating-costs.md - Calculating llm costs
references/COMMANDMENTS.md - Framework-specific rules the integration must follow

Each provider reference contains installation instructions, SDK setup, and code examples specific to that provider or framework. Find the reference that matches the user's stack.

If the user's provider isn't listed, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.

Key principles

Environment variables: Always use environment variables for PostHog and LLM provider keys. Never hardcode them.
Minimal changes: Add LLM analytics alongside existing LLM calls. Don't replace or restructure existing code.
Trace all generations: Capture input tokens, output tokens, model name, latency, and costs for every LLM call.
Link to users: Associate LLM generations with identified users via distinct IDs when possible.
One provider at a time: Only instrument the provider(s) the user is actually using. Don't add instrumentation for providers not present in the codebase.

Files

ATTRIBUTION.mdAdded by AutoGineer958 B
LICENSELicence file added by AutoGineer1.0 KiB
SKILL.md7.3 KiB
references/COMMANDMENTS.md1.5 KiB
references/README.md543 B
references/anthropic.md27.1 KiB
references/autogen.md7.7 KiB
references/azure-openai.md10.7 KiB
references/basics.md5.6 KiB
references/calculating-costs.md8.8 KiB
references/cerebras.md8.3 KiB
references/cohere.md8.3 KiB
references/crewai.md7.2 KiB
references/deepseek.md8.2 KiB
references/dspy.md7.4 KiB
references/fireworks-ai.md8.4 KiB
references/google.md10.0 KiB
references/groq.md8.2 KiB
references/helicone.md8.3 KiB
references/hugging-face.md8.4 KiB
references/instructor.md10.5 KiB
references/langchain.md8.5 KiB
references/langgraph.md8.3 KiB
references/litellm.md9.5 KiB
references/llamaindex.md8.1 KiB
references/manual-capture.md23.7 KiB
references/mastra.md6.7 KiB
references/mirascope.md8.0 KiB
references/mistral.md8.3 KiB
references/ollama.md8.2 KiB
references/openai-agents.md8.2 KiB
references/openai.md10.2 KiB
references/openrouter.md8.3 KiB
references/perplexity.md8.3 KiB
references/portkey.md8.2 KiB
references/pydantic-ai.md7.4 KiB
references/semantic-kernel.md7.7 KiB
references/smolagents.md7.5 KiB
references/together-ai.md8.3 KiB
references/traces.md8.5 KiB
references/vercel-ai.md10.0 KiB
references/xai.md8.2 KiB

references/dspy.md

AI agents: this is one page from PostHog's docs. Full index of Markdown docs for LLMs: https://posthog.com/llms.txt

DSPy AI Observability installation

Canonical URL: https://posthog.com/docs/ai-observability/installation/dspy

Cloud region: This page uses US Cloud hosts. For an EU Cloud project, use https://eu.i.posthog.com instead of https://us.i.posthog.com (ingestion host), and use https://eu.posthog.com instead of https://app.posthog.com (app host for links).

1
Install the PostHog SDK
Required
Setting up analytics starts with installing the PostHog SDK. The DSPy integration uses PostHog's LiteLLM callback.
```
pip install posthog
```
2
Install DSPy and LiteLLM
Required
Install DSPy and LiteLLM. DSPy uses LiteLLM natively for provider access, and PostHog integrates with LiteLLM's callback system.
```
pip install dspy litellm
```

Configure PostHog with LiteLLM

Required

Set your PostHog project token and host as environment variables, then configure LiteLLM to use PostHog as a callback handler. You can find your project token in your project settings.

import os
import dspy
import litellm

# Set PostHog environment variables
os.environ["POSTHOG_API_KEY"] = "<ph_project_token>"
os.environ["POSTHOG_API_URL"] = "https://us.i.posthog.com"

# Enable PostHog callbacks in LiteLLM
litellm.success_callback = ["posthog"]
litellm.failure_callback = ["posthog"]

# Configure DSPy to use an LLM
lm = dspy.LM(
    "openai/gpt-5-mini",
    api_key="your_openai_api_key",
    metadata={
        "user_id": "user_123",  # Maps to PostHog distinct_id
        "$ai_session_id": "conversation-abc",  # Groups calls into one session
    },
)
dspy.configure(lm=lm)

How this works

DSPy uses LiteLLM under the hood for LLM provider access. By configuring PostHog as a LiteLLM callback, all LLM calls made through DSPy are automatically captured as $ai_generation events.

Run DSPy modules

Required

Use DSPy as normal. PostHog automatically captures an $ai_generation event for each LLM call made through LiteLLM.

from posthog import Posthog
import time, uuid

posthog = Posthog("<ph_project_token>", host="https://us.i.posthog.com")

trace_id = str(uuid.uuid4())

lm = dspy.LM(
    "openai/gpt-5-mini",
    api_key="your_openai_api_key",
    metadata={
        "user_id": "user_123",
        "$ai_session_id": "conversation-abc",
        "$ai_trace_id": trace_id,
    },
)
dspy.configure(lm=lm)


# Define a simple signature
class QA(dspy.Signature):
    """Answer the question."""
    question: str = dspy.InputField()
    answer: str = dspy.OutputField()

predictor = dspy.Predict(QA)
result = predictor(question="What's a fun fact about hedgehogs?")
print(result.answer)

You can expect captured $ai_generation events to have the following properties:

Property	Description
`$ai_model`	The specific model, like `gpt-5-mini` or `claude-4-sonnet`
`$ai_latency`	The latency of the LLM call in seconds
`$ai_time_to_first_token`	Time to first token in seconds (streaming only)
`$ai_tools`	Tools and functions available to the LLM
`$ai_input`	List of messages sent to the LLM
`$ai_input_tokens`	The number of tokens in the input (often found in response.usage)
`$ai_output_choices`	List of response choices from the LLM
`$ai_output_tokens`	The number of tokens in the output (often found in `response.usage`)
`$ai_total_cost_usd`	The total cost in USD (input + output)
[...]	See full list of properties

Capture tool calls as spans

Optional

Capture tool calls as a span yourself, as the example below does before calling predictor.

# retrieve() is your existing retrieval setup
start = time.time()
context = retrieve("hedgehog facts")

posthog.capture(
    distinct_id="user_123",
    event="$ai_span",
    properties={
        "$ai_trace_id": trace_id,
        "$ai_session_id": "conversation-abc",
        "$ai_span_id": str(uuid.uuid4()),
        "$ai_span_name": "retrieve",
        "$ai_input_state": "hedgehog facts",
        "$ai_output_state": context,
        "$ai_latency": time.time() - start,
    },
)

question = f"Using this context, answer what a fun fact about hedgehogs is: {context}"
result = predictor(question=question)

See spans for the full list of span properties.

Verify traces and generations
Recommended
Confirm LLM events are being sent to PostHog
Let's make sure LLM events are being captured and sent to PostHog. Under AI Observability, you should see rows of data appear in the Traces and Generations tabs.
Image: LLM generations in PostHog
Check for LLM events in PostHog

Next steps

Recommended

Now that you're capturing AI conversations, continue with the resources below to learn what else AI Observability enables within the PostHog platform.

Resource	Description
Basics	Learn the basics of how LLM calls become events in PostHog.
Generations	Read about the `$ai_generation` event and its properties.
Traces	Explore the trace hierarchy and how to use it to debug LLM calls.
Spans	Review spans and their role in representing individual operations.
Anaylze LLM performance	Learn how to create dashboards to analyze LLM performance.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an MCP server

Security

What our scanner noticed· Low risk

Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

PostHog

Publisher

PostHog (Official)

Source

PostHog/skills · skills/omnibus/instrument-llm-analytics · commit a5d30e7

Licence

MIT License (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by PostHog.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download instrument-llm-analytics.zip

From the author

AutoGineer summary

What it does

When to use it

Advantages

Usage examples

Install

Documentation by PostHog

Add PostHog AI Observability

Instructions

Reference files

Key principles

Files

references/dspy.md

DSPy AI Observability installation

On this page

Install the PostHog SDK

Install DSPy and LiteLLM

Configure PostHog with LiteLLM

Run DSPy modules

Capture tool calls as spans

Verify traces and generations

Next steps

Requirements and compatibility

Security

Attribution and licence

Related skills

Claude API

Transformers

Open Notebook

Vision Sft

LLM Evaluation

Lora Qlora Recipes

Download ZIP