Omnibus Instrument LLM Analytics

PostHogOfficialCreated by PostHog

Skill
Tool integration
Data and AI
MIT
Low risk

From the author

Add PostHog LLM analytics to trace AI model usage. Use after implementing LLM features or reviewing PRs to ensure all generations are captured with token counts, latency, and costs. Also handles initial PostHog SDK setup if not yet installed.

— the description by PostHog

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured approach to instrumenting AI applications with PostHog analytics. It guides developers through detecting their LLM stack, installing the necessary PostHog SDKs, and configuring wrappers or middleware to capture detailed telemetry for every LLM generation. By following provider-specific references, users can ensure consistent tracking of token counts, latency, and costs across a wide range of LLM providers and frameworks without altering their core application architecture.

What it does

Detects existing LLM providers and frameworks in the codebase.
Installs and configures the PostHog SDK for LLM observability.
Instruments LLM calls to capture input/output tokens, latency, and costs.
Associates LLM generations with specific user IDs for better tracking.
Provides fallback manual capture methods for unsupported providers.
Configures environment variables for secure API key management.

When to use it

After implementing new LLM features in an application.
When reviewing pull requests that introduce or modify AI model interactions.
When you need to monitor token usage and costs for AI features.
When setting up observability for a new AI-powered project.

Advantages

Centralized visibility into AI model performance and costs.
Minimal code changes required through wrapper-based instrumentation.
Supports a broad ecosystem of LLM providers and frameworks.
Enables user-level attribution for AI interactions.
Maintains security by enforcing environment variable usage.

Usage examples

“Instrument my existing Anthropic Claude implementation with PostHog analytics.”: The agent will identify the Anthropic SDK usage, install the required PostHog packages, and wrap the client to automatically capture generation events.
“Set up PostHog tracking for my AutoGen agents.”: The agent will configure the PostHog OpenAI wrapper and integrate it into the AutoGen model client to start logging agentic LLM calls.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/posthog/omnibus-instrument-llm-analytics/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip omnibus-instrument-llm-analytics.zip -d .agents/skills

Documentation by PostHog

Add PostHog LLM analytics

Use this skill to add PostHog LLM analytics that trace AI model usage in new or changed code. Use it after implementing LLM features or reviewing PRs to ensure all generations are captured with token counts, latency, and costs. If PostHog is not yet installed, this skill also covers initial SDK setup. Supports any provider or framework.

Supported providers: OpenAI, Azure OpenAI, Anthropic, Google, Cohere, Mistral, Perplexity, DeepSeek, Groq, Together AI, Fireworks AI, xAI, Cerebras, Hugging Face, Ollama, OpenRouter.

Supported frameworks: LangChain, LlamaIndex, CrewAI, AutoGen, DSPy, LangGraph, Pydantic AI, Vercel AI, LiteLLM, Instructor, Semantic Kernel, Mirascope, Mastra, SmolAgents, OpenAI Agents.

Proxy/gateway: Portkey, Helicone.

Instructions

Follow these steps IN ORDER:

STEP 1: Analyze the codebase and detect the LLM stack.

Look for LLM provider SDKs (openai, anthropic, google-generativeai, etc.) and AI frameworks (langchain, llamaindex, crewai, etc.) in dependency files and imports.
Look for lockfiles to determine the package manager.
Check for existing PostHog or observability setup. If PostHog is already installed and LLM tracing is configured, skip to STEP 4 to add tracing for any new LLM calls.

STEP 2: Research instrumentation. (Skip if PostHog LLM tracing is already set up.) 2.1. Find the reference file below that matches the detected provider or framework — it is the source of truth for callback setup, middleware configuration, and event capture. Read it now. 2.2. If no reference matches, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.

STEP 3: Install the PostHog SDK. (Skip if PostHog is already set up.)

Add the PostHog SDK and any required callback/integration packages.
Do not manually edit dependency files — use the package manager's install command.
Always install packages as a background task. Don't await completion; proceed with other work immediately.

STEP 4: Add LLM tracing.

Instrument LLM calls to capture input tokens, output tokens, model name, latency, and costs for every generation.
Follow the provider-specific reference for the exact callback/middleware setup.
Do not alter the fundamental architecture of existing files. Make additions minimal and targeted.
You must read a file immediately before attempting to write it.

STEP 5: Link to users.

Associate LLM generations with identified users via distinct IDs when possible.

STEP 6: Set up environment variables.

If an env-file-tools MCP server is connected, use check_env_keys then set_env_values to configure the PostHog API key and host.
Reference these environment variables in code instead of hardcoding them.

Reference files

references/openai.md - Openai llm analytics installation - docs
references/azure-openai.md - Azure openai llm analytics installation - docs
references/anthropic.md - Anthropic llm analytics installation - docs
references/google.md - Google llm analytics installation - docs
references/cohere.md - Cohere llm analytics installation - docs
references/mistral.md - Mistral llm analytics installation - docs
references/perplexity.md - Perplexity llm analytics installation - docs
references/deepseek.md - Deepseek llm analytics installation - docs
references/groq.md - Groq llm analytics installation - docs
references/together-ai.md - Together ai llm analytics installation - docs
references/fireworks-ai.md - Fireworks ai llm analytics installation - docs
references/xai.md - Xai llm analytics installation - docs
references/cerebras.md - Cerebras llm analytics installation - docs
references/hugging-face.md - Hugging face llm analytics installation - docs
references/ollama.md - Ollama llm analytics installation - docs
references/openrouter.md - Openrouter llm analytics installation - docs
references/langchain.md - Langchain llm analytics installation - docs
references/llamaindex.md - Llamaindex llm analytics installation - docs
references/crewai.md - Crewai llm analytics installation - docs
references/autogen.md - Autogen llm analytics installation - docs
references/dspy.md - Dspy llm analytics installation - docs
references/langgraph.md - Langgraph llm analytics installation - docs
references/pydantic-ai.md - Pydantic ai llm analytics installation - docs
references/vercel-ai.md - Vercel ai SDK llm analytics installation - docs
references/litellm.md - Litellm llm analytics installation - docs
references/instructor.md - Instructor llm analytics installation - docs
references/semantic-kernel.md - Semantic kernel llm analytics installation - docs
references/mirascope.md - Mirascope llm analytics installation - docs
references/mastra.md - Mastra llm analytics installation - docs
references/smolagents.md - Smolagents llm analytics installation - docs
references/openai-agents.md - Openai agents SDK llm analytics installation - docs
references/portkey.md - Portkey llm analytics installation - docs
references/helicone.md - Helicone llm analytics installation - docs
references/manual-capture.md - Manual capture llm analytics installation - docs
references/basics.md - Llm analytics basics - docs
references/traces.md - Traces - docs
references/calculating-costs.md - Calculating llm costs - docs

Each provider reference contains installation instructions, SDK setup, and code examples specific to that provider or framework. Find the reference that matches the user's stack.

If the user's provider isn't listed, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.

Key principles

Environment variables: Always use environment variables for PostHog and LLM provider keys. Never hardcode them.
Minimal changes: Add LLM analytics alongside existing LLM calls. Don't replace or restructure existing code.
Trace all generations: Capture input tokens, output tokens, model name, latency, and costs for every LLM call.
Link to users: Associate LLM generations with identified users via distinct IDs when possible.
One provider at a time: Only instrument the provider(s) the user is actually using. Don't add instrumentation for providers not present in the codebase.

Files

ATTRIBUTION.mdAdded by AutoGineer966 B
LICENSELicence file added by AutoGineer1.0 KiB
SKILL.md6.6 KiB
references/anthropic.md6.6 KiB
references/autogen.md4.5 KiB
references/azure-openai.md6.4 KiB
references/basics.md3.2 KiB
references/calculating-costs.md8.3 KiB
references/cerebras.md6.7 KiB
references/cohere.md6.7 KiB
references/crewai.md4.4 KiB
references/deepseek.md6.7 KiB
references/dspy.md4.4 KiB
references/fireworks-ai.md6.8 KiB
references/google.md7.6 KiB
references/groq.md6.7 KiB
references/helicone.md6.9 KiB
references/hugging-face.md6.8 KiB
references/instructor.md5.8 KiB
references/langchain.md6.9 KiB
references/langgraph.md6.6 KiB
references/litellm.md6.6 KiB
references/llamaindex.md4.7 KiB
references/manual-capture.md15.6 KiB
references/mastra.md5.1 KiB
references/mirascope.md4.5 KiB
references/mistral.md6.7 KiB
references/ollama.md6.7 KiB
references/openai-agents.md5.7 KiB
references/openai.md6.2 KiB
references/openrouter.md7.1 KiB
references/perplexity.md6.7 KiB
references/portkey.md7.0 KiB
references/pydantic-ai.md4.5 KiB
references/semantic-kernel.md4.5 KiB
references/smolagents.md4.3 KiB
references/together-ai.md6.7 KiB
references/traces.md5.8 KiB
references/vercel-ai.md4.5 KiB
references/xai.md6.6 KiB

references/calculating-costs.md

Calculating LLM costs - Docs

How are LLM costs calculated?

PostHog calculates cost based on the number of input (prompt) and output (completion) tokens generated by specific AI models, plus additional cost components like per-request and per-web-search pricing when applicable.

Model matching

We use OpenRouter's pricing data as our primary source to match your model and calculate costs. OpenRouter provides comprehensive pricing information for models across different providers, and we use both the $ai_provider and $ai_model properties from your events to find the exact pricing.

If OpenRouter doesn't have pricing data for a specific model, we fall back to our manually maintained pricing database for additional model support.

Cost components

The total cost ($ai_total_cost_usd) is calculated from multiple components:

Token-based costs: Input (prompt) and output (completion) tokens
Cache costs: Cached token reads and writes (when applicable)
Request-based costs: Some models charge per request in addition to token costs
Web search costs: Some models charge per web search performed

For cached LLM responses, our pricing models include cached token pricing which we automatically apply.

We also take into account the reasoning / thinking tokens for models that support it.

Cache token accounting

Different LLM providers report cache tokens differently:

Exclusive counting (Anthropic/Claude) - Cache tokens are separate from $ai_input_tokens. For example, if you have 100 input tokens and 50 cached tokens, $ai_input_tokens is 100.
Inclusive counting (OpenAI and most others) - Cache tokens are included in $ai_input_tokens. Using the same example, $ai_input_tokens is 150.

PostHog auto-detects which counting style to use based on the $ai_provider and $ai_model properties. For Anthropic and Claude models, PostHog assumes exclusive counting. For all other providers, it assumes inclusive counting.

Overriding auto-detection

If you're manually capturing events or using a framework wrapper that normalizes token counts differently, override the auto-detection by setting the $ai_cache_reporting_exclusive property:

$ai_cache_reporting_exclusive: true - Cache tokens are separate from input tokens (Anthropic-style)
$ai_cache_reporting_exclusive: false - Input tokens already include cache tokens

When not set, PostHog resolves this automatically and writes the resolved value back to the event for downstream consumers.

Setting custom pricing

You can override PostHog's automatic cost calculation by providing custom pricing for your LLM models. This is useful when:

You have negotiated custom pricing with your LLM provider
You're using a model that PostHog doesn't support yet
PostHog's automatic pricing doesn't match your specific use case

Option 1: Custom price per token

If you know your pricing per token, you can set the following custom properties when calling your LLM:

$ai_input_token_price (required): Price per input/prompt token
$ai_output_token_price (required): Price per output/completion token
$ai_cache_read_token_price (optional): Price per cached token read
$ai_cache_write_token_price (optional): Price per cached token write
$ai_request_price (optional): Price per request (for models that charge per request)
$ai_web_search_price (optional): Price per web search (for models that charge per search)

Important: Token prices should be per individual token, not per million tokens. For example, if your provider charges $0.03 per 1M tokens, you would set $ai_input_token_price: 0.00000003 (0.03 / 1,000,000).

PostHog AI

JavaScript

import { OpenAI } from '@posthog/ai'
import { PostHog } from 'posthog-node'
const phClient = new PostHog(
  '<ph_project_token>',
  { host: 'https://us.i.posthog.com' }
)
const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  posthog: phClient
})
const response = await openai.responses.create({
  model: 'my-custom-model',
  messages: [{ role: 'user', content: 'Hello' }],
  posthogProperties: {
    $ai_input_token_price: 0.00000003,   // $0.03 per 1M tokens = $0.00000003 per token
    $ai_output_token_price: 0.00000006,  // $0.06 per 1M tokens = $0.00000006 per token
    // Optional: cache pricing
    $ai_cache_read_token_price: 0.000000015,
    $ai_cache_write_token_price: 0.0000000375
  }
})

Python

from posthog.ai.openai import OpenAI
from posthog import Posthog
posthog = Posthog(
    "<ph_project_token>",
    host="https://us.i.posthog.com"
)
client = OpenAI(
    api_key="sk-...",
    posthog_client=posthog
)
response = client.responses.create(
    model="my-custom-model",
    messages=[{"role": "user", "content": "Hello"}],
    posthog_properties={
        "$ai_input_token_price": 0.00000003,   # $0.03 per 1M tokens = $0.00000003 per token
        "$ai_output_token_price": 0.00000006,  # $0.06 per 1M tokens = $0.00000006 per token
        # Optional: cache pricing
        "$ai_cache_read_token_price": 0.000000015,
        "$ai_cache_write_token_price": 0.0000000375
    }
)

Both $ai_input_token_price and $ai_output_token_price must be provided for custom pricing to take effect. PostHog will then calculate the total cost based on the token counts and your custom prices.

Option 2: Pre-calculated costs

If you're manually capturing LLM events and have already calculated the total costs yourself, you can send them directly:

$ai_input_cost_usd: Total cost for input/prompt tokens in USD
$ai_output_cost_usd: Total cost for output/completion tokens in USD
$ai_request_cost_usd: Total cost for requests in USD
$ai_web_search_cost_usd: Total cost for web searches in USD

PostHog AI

JavaScript

// After making your LLM call and calculating costs
posthog.capture('$ai_generation', {
  $ai_trace_id: traceId,
  $ai_model: 'my-custom-model',
  $ai_provider: 'my-provider',
  $ai_input_tokens: inputTokens,
  $ai_output_tokens: outputTokens,
  $ai_input_cost_usd: 0.0042,
  $ai_output_cost_usd: 0.0028
  // ... other required properties
})

Python

# After making your LLM call and calculating costs
posthog.capture(
    event='$ai_generation',
    properties={
        '$ai_trace_id': trace_id,
        '$ai_model': 'my-custom-model',
        '$ai_provider': 'my-provider',
        '$ai_input_tokens': input_tokens,
        '$ai_output_tokens': output_tokens,
        '$ai_input_cost_usd': 0.0042,
        '$ai_output_cost_usd': 0.0028
        # ... other required properties
    }
)

PostHog will use these values directly without any additional calculation.

Precedence

Cost calculation follows this precedence order:

Pre-calculated costs ($ai_input_cost_usd, $ai_output_cost_usd, $ai_request_cost_usd, $ai_web_search_cost_usd): These values are used directly without any additional calculation
Custom pricing ($ai_input_token_price, $ai_output_token_price, $ai_request_price, $ai_web_search_price): PostHog calculates costs from token counts and usage metrics using your custom prices
Automatic model matching: PostHog matches your model using OpenRouter's pricing data first, then falls back to manually maintained pricing if needed

Cost calculation metadata

When PostHog calculates costs automatically, it sets the following metadata properties on the event to help you understand how costs were determined:

Property	Description	Examples
$ai_model_cost_used	The model identifier used for cost calculation. May differ from the requested model when a variant or alias is resolved.	openai/gpt-4o-mini
$ai_cost_model_source	Where the cost data for this model was sourced from.	openrouter, manual, custom, passthrough
$ai_cost_model_provider	The provider used to look up the cost for this model.	openai, anthropic, custom

These properties are useful for debugging cost discrepancies or understanding which pricing was applied when using model aliases or custom configurations.

You can find the code for this on GitHub.

Community questions

Ask a question

Was this page useful?

HelpfulCould be better

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an MCP server, Needs an API key

Security

What our scanner noticed· Low risk

Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

PostHog

Publisher

PostHog (Official)

Source

PostHog/skills · skills/posthog/llm-analytics/skills/all · commit a5d30e7

Licence

MIT License (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

Also distributed in

PostHog/skills

AutoGineer is not affiliated with or endorsed by PostHog.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download omnibus-instrument-llm-analytics.zip

From the author

AutoGineer summary

What it does

When to use it

Advantages

Usage examples

Install

Documentation by PostHog

Add PostHog LLM analytics

Instructions

Reference files

Key principles

Files

references/calculating-costs.md

Calculating LLM costs - Docs

How are LLM costs calculated?

Model matching

Cost components

Cache token accounting

Overriding auto-detection

Setting custom pricing

Option 1: Custom price per token

JavaScript

Python

Option 2: Pre-calculated costs

JavaScript

Python

Precedence

Cost calculation metadata

Community questions

Was this page useful?

Requirements and compatibility

Security

Attribution and licence

Related skills

Claude API

Transformers

Open Notebook

Vision Sft

LLM Evaluation

Lora Qlora Recipes

Download ZIP