Omnibus Instrument LLM Analytics

PostHogOfficialCreated by PostHog

Skill
Tool integration
Data and AI
MIT
Low risk

From the author

Add PostHog LLM analytics to trace AI model usage. Use after implementing LLM features or reviewing PRs to ensure all generations are captured with token counts, latency, and costs. Also handles initial PostHog SDK setup if not yet installed.

— the description by PostHog

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured approach to instrumenting AI applications with PostHog analytics. It guides developers through detecting their LLM stack, installing the necessary PostHog SDKs, and configuring wrappers or middleware to capture detailed telemetry for every LLM generation. By following provider-specific references, users can ensure consistent tracking of token counts, latency, and costs across a wide range of LLM providers and frameworks without altering their core application architecture.

What it does

Detects existing LLM providers and frameworks in the codebase.
Installs and configures the PostHog SDK for LLM observability.
Instruments LLM calls to capture input/output tokens, latency, and costs.
Associates LLM generations with specific user IDs for better tracking.
Provides fallback manual capture methods for unsupported providers.
Configures environment variables for secure API key management.

When to use it

After implementing new LLM features in an application.
When reviewing pull requests that introduce or modify AI model interactions.
When you need to monitor token usage and costs for AI features.
When setting up observability for a new AI-powered project.

Advantages

Centralized visibility into AI model performance and costs.
Minimal code changes required through wrapper-based instrumentation.
Supports a broad ecosystem of LLM providers and frameworks.
Enables user-level attribution for AI interactions.
Maintains security by enforcing environment variable usage.

Usage examples

“Instrument my existing Anthropic Claude implementation with PostHog analytics.”: The agent will identify the Anthropic SDK usage, install the required PostHog packages, and wrap the client to automatically capture generation events.
“Set up PostHog tracking for my AutoGen agents.”: The agent will configure the PostHog OpenAI wrapper and integrate it into the AutoGen model client to start logging agentic LLM calls.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/posthog/omnibus-instrument-llm-analytics/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip omnibus-instrument-llm-analytics.zip -d .agents/skills

Documentation by PostHog

Add PostHog LLM analytics

Use this skill to add PostHog LLM analytics that trace AI model usage in new or changed code. Use it after implementing LLM features or reviewing PRs to ensure all generations are captured with token counts, latency, and costs. If PostHog is not yet installed, this skill also covers initial SDK setup. Supports any provider or framework.

Supported providers: OpenAI, Azure OpenAI, Anthropic, Google, Cohere, Mistral, Perplexity, DeepSeek, Groq, Together AI, Fireworks AI, xAI, Cerebras, Hugging Face, Ollama, OpenRouter.

Supported frameworks: LangChain, LlamaIndex, CrewAI, AutoGen, DSPy, LangGraph, Pydantic AI, Vercel AI, LiteLLM, Instructor, Semantic Kernel, Mirascope, Mastra, SmolAgents, OpenAI Agents.

Proxy/gateway: Portkey, Helicone.

Instructions

Follow these steps IN ORDER:

STEP 1: Analyze the codebase and detect the LLM stack.

Look for LLM provider SDKs (openai, anthropic, google-generativeai, etc.) and AI frameworks (langchain, llamaindex, crewai, etc.) in dependency files and imports.
Look for lockfiles to determine the package manager.
Check for existing PostHog or observability setup. If PostHog is already installed and LLM tracing is configured, skip to STEP 4 to add tracing for any new LLM calls.

STEP 2: Research instrumentation. (Skip if PostHog LLM tracing is already set up.) 2.1. Find the reference file below that matches the detected provider or framework — it is the source of truth for callback setup, middleware configuration, and event capture. Read it now. 2.2. If no reference matches, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.

STEP 3: Install the PostHog SDK. (Skip if PostHog is already set up.)

Add the PostHog SDK and any required callback/integration packages.
Do not manually edit dependency files — use the package manager's install command.
Always install packages as a background task. Don't await completion; proceed with other work immediately.

STEP 4: Add LLM tracing.

Instrument LLM calls to capture input tokens, output tokens, model name, latency, and costs for every generation.
Follow the provider-specific reference for the exact callback/middleware setup.
Do not alter the fundamental architecture of existing files. Make additions minimal and targeted.
You must read a file immediately before attempting to write it.

STEP 5: Link to users.

Associate LLM generations with identified users via distinct IDs when possible.

STEP 6: Set up environment variables.

If an env-file-tools MCP server is connected, use check_env_keys then set_env_values to configure the PostHog API key and host.
Reference these environment variables in code instead of hardcoding them.

Reference files

references/openai.md - Openai llm analytics installation - docs
references/azure-openai.md - Azure openai llm analytics installation - docs
references/anthropic.md - Anthropic llm analytics installation - docs
references/google.md - Google llm analytics installation - docs
references/cohere.md - Cohere llm analytics installation - docs
references/mistral.md - Mistral llm analytics installation - docs
references/perplexity.md - Perplexity llm analytics installation - docs
references/deepseek.md - Deepseek llm analytics installation - docs
references/groq.md - Groq llm analytics installation - docs
references/together-ai.md - Together ai llm analytics installation - docs
references/fireworks-ai.md - Fireworks ai llm analytics installation - docs
references/xai.md - Xai llm analytics installation - docs
references/cerebras.md - Cerebras llm analytics installation - docs
references/hugging-face.md - Hugging face llm analytics installation - docs
references/ollama.md - Ollama llm analytics installation - docs
references/openrouter.md - Openrouter llm analytics installation - docs
references/langchain.md - Langchain llm analytics installation - docs
references/llamaindex.md - Llamaindex llm analytics installation - docs
references/crewai.md - Crewai llm analytics installation - docs
references/autogen.md - Autogen llm analytics installation - docs
references/dspy.md - Dspy llm analytics installation - docs
references/langgraph.md - Langgraph llm analytics installation - docs
references/pydantic-ai.md - Pydantic ai llm analytics installation - docs
references/vercel-ai.md - Vercel ai SDK llm analytics installation - docs
references/litellm.md - Litellm llm analytics installation - docs
references/instructor.md - Instructor llm analytics installation - docs
references/semantic-kernel.md - Semantic kernel llm analytics installation - docs
references/mirascope.md - Mirascope llm analytics installation - docs
references/mastra.md - Mastra llm analytics installation - docs
references/smolagents.md - Smolagents llm analytics installation - docs
references/openai-agents.md - Openai agents SDK llm analytics installation - docs
references/portkey.md - Portkey llm analytics installation - docs
references/helicone.md - Helicone llm analytics installation - docs
references/manual-capture.md - Manual capture llm analytics installation - docs
references/basics.md - Llm analytics basics - docs
references/traces.md - Traces - docs
references/calculating-costs.md - Calculating llm costs - docs

Each provider reference contains installation instructions, SDK setup, and code examples specific to that provider or framework. Find the reference that matches the user's stack.

If the user's provider isn't listed, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.

Key principles

Environment variables: Always use environment variables for PostHog and LLM provider keys. Never hardcode them.
Minimal changes: Add LLM analytics alongside existing LLM calls. Don't replace or restructure existing code.
Trace all generations: Capture input tokens, output tokens, model name, latency, and costs for every LLM call.
Link to users: Associate LLM generations with identified users via distinct IDs when possible.
One provider at a time: Only instrument the provider(s) the user is actually using. Don't add instrumentation for providers not present in the codebase.

Files

ATTRIBUTION.mdAdded by AutoGineer966 B
LICENSELicence file added by AutoGineer1.0 KiB
SKILL.md6.6 KiB
references/anthropic.md6.6 KiB
references/autogen.md4.5 KiB
references/azure-openai.md6.4 KiB
references/basics.md3.2 KiB
references/calculating-costs.md8.3 KiB
references/cerebras.md6.7 KiB
references/cohere.md6.7 KiB
references/crewai.md4.4 KiB
references/deepseek.md6.7 KiB
references/dspy.md4.4 KiB
references/fireworks-ai.md6.8 KiB
references/google.md7.6 KiB
references/groq.md6.7 KiB
references/helicone.md6.9 KiB
references/hugging-face.md6.8 KiB
references/instructor.md5.8 KiB
references/langchain.md6.9 KiB
references/langgraph.md6.6 KiB
references/litellm.md6.6 KiB
references/llamaindex.md4.7 KiB
references/manual-capture.md15.6 KiB
references/mastra.md5.1 KiB
references/mirascope.md4.5 KiB
references/mistral.md6.7 KiB
references/ollama.md6.7 KiB
references/openai-agents.md5.7 KiB
references/openai.md6.2 KiB
references/openrouter.md7.1 KiB
references/perplexity.md6.7 KiB
references/portkey.md7.0 KiB
references/pydantic-ai.md4.5 KiB
references/semantic-kernel.md4.5 KiB
references/smolagents.md4.3 KiB
references/together-ai.md6.7 KiB
references/traces.md5.8 KiB
references/vercel-ai.md4.5 KiB
references/xai.md6.6 KiB

references/smolagents.md

smolagents LLM analytics installation - Docs

1
Install the PostHog SDK
Required
Setting up analytics starts with installing the PostHog SDK. The smolagents integration uses PostHog's OpenAI wrapper.
```
pip install posthog
```
2
Install smolagents and OpenAI
Required
Install smolagents and the OpenAI SDK. PostHog instruments your LLM calls by wrapping the OpenAI client, which you can pass to smolagents' OpenAIServerModel.
```
pip install smolagents openai
```
3
Initialize PostHog and smolagents
Required
Initialize PostHog with your project token and host from your project settings, then create a PostHog OpenAI wrapper and pass it to smolagents' OpenAIServerModel.
```
from smolagents import CodeAgent, OpenAIServerModel
from posthog.ai.openai import OpenAI
from posthog import Posthog
posthog = Posthog(
    "<ph_project_token>",
    host="https://us.i.posthog.com"
)
openai_client = OpenAI(
    api_key="your_openai_api_key",
    posthog_client=posthog
)
model = OpenAIServerModel(
    model_id="gpt-5-mini",
    client=openai_client,
)
```
How this works
PostHog's OpenAI wrapper is a drop-in replacement for openai.OpenAI. By passing it as the client to OpenAIServerModel, all LLM calls made by smolagents are automatically captured as $ai_generation events.

Run your agent

Required

Use smolagents as normal. PostHog automatically captures an $ai_generation event for each LLM call made through the wrapped OpenAI client.

agent = CodeAgent(
    tools=[],
    model=model,
)
result = agent.run(
    "What is a fun fact about hedgehogs?"
)
print(result)

You can expect captured $ai_generation events to have the following properties:

Property	Description
$ai_model	The specific model, like gpt-5-mini or claude-4-sonnet
$ai_latency	The latency of the LLM call in seconds
$ai_time_to_first_token	Time to first token in seconds (streaming only)
$ai_tools	Tools and functions available to the LLM
$ai_input	List of messages sent to the LLM
$ai_input_tokens	The number of tokens in the input (often found in response.usage)
$ai_output_choices	List of response choices from the LLM
$ai_output_tokens	The number of tokens in the output (often found in response.usage)
$ai_total_cost_usd	The total cost in USD (input + output)
[...]	See full list of properties

Verify traces and generations
Recommended
Confirm LLM events are being sent to PostHog
Let's make sure LLM events are being captured and sent to PostHog. Under LLM analytics, you should see rows of data appear in the Traces and Generations tabs.
Image: LLM generations in PostHogImage: LLM generations in PostHog
Check for LLM events in PostHog

Next steps

Recommended

Now that you're capturing AI conversations, continue with the resources below to learn what else LLM Analytics enables within the PostHog platform.

Resource	Description
Basics	Learn the basics of how LLM calls become events in PostHog.
Generations	Read about the $ai_generation event and its properties.
Traces	Explore the trace hierarchy and how to use it to debug LLM calls.
Spans	Review spans and their role in representing individual operations.
Anaylze LLM performance	Learn how to create dashboards to analyze LLM performance.

Community questions

Ask a question

Was this page useful?

HelpfulCould be better

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an MCP server, Needs an API key

Security

What our scanner noticed· Low risk

Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

PostHog

Publisher

PostHog (Official)

Source

PostHog/skills · skills/posthog/llm-analytics/skills/all · commit a5d30e7

Licence

MIT License (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

Also distributed in

PostHog/skills

AutoGineer is not affiliated with or endorsed by PostHog.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download omnibus-instrument-llm-analytics.zip

From the author

AutoGineer summary

What it does

When to use it

Advantages

Usage examples

Install

Documentation by PostHog

Add PostHog LLM analytics

Instructions

Reference files

Key principles

Files

references/smolagents.md

smolagents LLM analytics installation - Docs

Install the PostHog SDK

Install smolagents and OpenAI

Initialize PostHog and smolagents

Run your agent

Verify traces and generations

Next steps

Community questions

Was this page useful?

Requirements and compatibility

Security

Attribution and licence

Related skills

Claude API

Transformers

Open Notebook

Vision Sft

LLM Evaluation

Lora Qlora Recipes

Download ZIP