Instrument LLM Analytics

PostHogOfficialCreated by PostHog

Skill
Tool integration
Data and AI
MIT
Low risk

From the author

Add PostHog AI Observability to trace AI model usage. Use after implementing LLM features or reviewing PRs to ensure all generations are captured with token counts, latency, and costs. Also handles initial PostHog SDK setup if not yet installed.

— the description by PostHog

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured approach to instrumenting LLM-powered applications with PostHog AI Observability. It guides the agent through detecting the existing LLM stack, installing necessary SDKs, and configuring event capture for token counts, latency, and costs. The skill supports a wide range of LLM providers and frameworks, ensuring that observability is added with minimal disruption to existing codebases.

What it does

Detects LLM providers and frameworks in the codebase.
Installs the PostHog SDK and required integration packages.
Instruments LLM calls to capture tokens, latency, and costs.
Configures environment variables for secure credential management.
Associates LLM generations with specific user IDs.
Provides fallback manual capture methods for unsupported providers.

When to use it

After implementing new LLM features in an application.
During code reviews to ensure proper observability coverage.
When setting up AI monitoring for the first time in a project.
When needing to track costs and performance metrics for AI model usage.

Advantages

Standardizes observability across diverse LLM stacks.
Ensures minimal impact on existing application architecture.
Provides granular visibility into token usage and latency.
Follows best practices for environment variable management.
Supports a broad ecosystem of AI frameworks and providers.

Usage examples

“Add PostHog AI observability to my LangChain project.”: The agent will scan the project for LangChain dependencies, install the necessary PostHog packages, and instrument the LLM chains to report usage metrics to PostHog.
“I need to track costs and latency for my OpenAI API calls.”: The agent will identify the OpenAI implementation, configure the appropriate middleware or callback, and ensure all generations are captured with the required metadata.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/posthog/instrument-llm-analytics/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip instrument-llm-analytics.zip -d .agents/skills

Documentation by PostHog

Add PostHog AI Observability

Use this skill to add PostHog AI Observability that traces AI model usage in new or changed code. Use it after implementing LLM features or reviewing PRs to ensure all generations are captured with token counts, latency, and costs. If PostHog is not yet installed, this skill also covers initial SDK setup. Supports any provider or framework.

Supported providers: OpenAI, Azure OpenAI, Anthropic, Google, Cohere, Mistral, Perplexity, DeepSeek, Groq, Together AI, Fireworks AI, xAI, Cerebras, Hugging Face, Ollama, OpenRouter.

Supported frameworks: LangChain, LlamaIndex, CrewAI, AutoGen, DSPy, LangGraph, Pydantic AI, Vercel AI, LiteLLM, Instructor, Semantic Kernel, Mirascope, Mastra, SmolAgents, OpenAI Agents.

Proxy/gateway: Portkey, Helicone.

Instructions

Follow these steps IN ORDER:

STEP 1: Analyze the codebase and detect the LLM stack.

Look for LLM provider SDKs (openai, anthropic, google-generativeai, etc.) and AI frameworks (langchain, llamaindex, crewai, etc.) in dependency files and imports.
Look for lockfiles to determine the package manager.
Check for existing PostHog or observability setup. If PostHog is already installed and LLM tracing is configured, skip to STEP 4 to add tracing for any new LLM calls.

STEP 2: Research instrumentation. (Skip if PostHog LLM tracing is already set up.) 2.1. Find the reference file below that matches the detected provider or framework — it is the source of truth for callback setup, middleware configuration, and event capture. Read it now. 2.2. If no reference matches, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.

STEP 3: Install the PostHog SDK. (Skip if PostHog is already set up.)

Add the PostHog SDK and any required callback/integration packages.
Do not manually edit dependency files — use the package manager's install command.

STEP 4: Add LLM tracing.

Instrument LLM calls to capture input tokens, output tokens, model name, latency, and costs for every generation.
Follow the provider-specific reference for the exact callback/middleware setup.
Do not alter the fundamental architecture of existing files. Make additions minimal and targeted.
You must read a file immediately before attempting to write it.

STEP 5: Link to users.

Associate LLM generations with identified users via distinct IDs when possible.

STEP 6: Set up environment variables.

Check if the project already has PostHog environment variables configured (e.g. in .env, .env.local, or framework-specific env files). If valid values already exist, skip this step.
If the PostHog project token is missing, use the PostHog MCP server's projects-get tool to retrieve the project's api_token. If multiple projects are returned, ask the user which project to use. If the MCP server is not connected or not authenticated, ask the user for their PostHog project token instead.
For the PostHog host URL: check the projects-get MCP response for a region field — US maps to https://us.i.posthog.com, EU maps to https://eu.i.posthog.com. If the region is not available from the MCP response or from existing project configuration, ask the user: "Are you on PostHog US Cloud or EU Cloud?" Do not assume US Cloud.
Write these values to the appropriate env file using the framework's naming convention.
Reference these environment variables in code instead of hardcoding them.

Reference files

references/openai.md - Openai observability installation
references/azure-openai.md - Azure openai observability installation
references/README.md - PostHog.ai for .net
references/anthropic.md - Anthropic ai observability installation
references/google.md - Google ai observability installation
references/cohere.md - Cohere ai observability installation
references/mistral.md - Mistral ai observability installation
references/perplexity.md - Perplexity ai observability installation
references/deepseek.md - Deepseek ai observability installation
references/groq.md - Groq ai observability installation
references/together-ai.md - Together ai observability installation
references/fireworks-ai.md - Fireworks ai observability installation
references/xai.md - Xai observability installation
references/cerebras.md - Cerebras ai observability installation
references/hugging-face.md - Hugging face ai observability installation
references/ollama.md - Ollama ai observability installation
references/openrouter.md - Openrouter ai observability installation
references/langchain.md - Langchain ai observability installation
references/llamaindex.md - Llamaindex ai observability installation
references/crewai.md - Crewai observability installation
references/autogen.md - Autogen ai observability installation
references/dspy.md - Dspy ai observability installation
references/langgraph.md - Langgraph ai observability installation
references/pydantic-ai.md - Pydantic ai observability installation
references/vercel-ai.md - Vercel ai SDK observability installation
references/litellm.md - Litellm ai observability installation
references/instructor.md - Instructor ai observability installation
references/semantic-kernel.md - Semantic kernel ai observability installation
references/mirascope.md - Mirascope ai observability installation
references/mastra.md - Mastra ai observability installation
references/smolagents.md - Smolagents ai observability installation
references/openai-agents.md - Openai agents SDK observability installation
references/portkey.md - Portkey ai observability installation
references/helicone.md - Helicone ai observability installation
references/manual-capture.md - Manual capture ai observability installation
references/basics.md - Ai observability basics
references/traces.md - Traces
references/calculating-costs.md - Calculating llm costs
references/COMMANDMENTS.md - Framework-specific rules the integration must follow

Each provider reference contains installation instructions, SDK setup, and code examples specific to that provider or framework. Find the reference that matches the user's stack.

If the user's provider isn't listed, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.

Key principles

Environment variables: Always use environment variables for PostHog and LLM provider keys. Never hardcode them.
Minimal changes: Add LLM analytics alongside existing LLM calls. Don't replace or restructure existing code.
Trace all generations: Capture input tokens, output tokens, model name, latency, and costs for every LLM call.
Link to users: Associate LLM generations with identified users via distinct IDs when possible.
One provider at a time: Only instrument the provider(s) the user is actually using. Don't add instrumentation for providers not present in the codebase.

Files

ATTRIBUTION.mdAdded by AutoGineer958 B
LICENSELicence file added by AutoGineer1.0 KiB
SKILL.md7.3 KiB
references/COMMANDMENTS.md1.5 KiB
references/README.md543 B
references/anthropic.md27.1 KiB
references/autogen.md7.7 KiB
references/azure-openai.md10.7 KiB
references/basics.md5.6 KiB
references/calculating-costs.md8.8 KiB
references/cerebras.md8.3 KiB
references/cohere.md8.3 KiB
references/crewai.md7.2 KiB
references/deepseek.md8.2 KiB
references/dspy.md7.4 KiB
references/fireworks-ai.md8.4 KiB
references/google.md10.0 KiB
references/groq.md8.2 KiB
references/helicone.md8.3 KiB
references/hugging-face.md8.4 KiB
references/instructor.md10.5 KiB
references/langchain.md8.5 KiB
references/langgraph.md8.3 KiB
references/litellm.md9.5 KiB
references/llamaindex.md8.1 KiB
references/manual-capture.md23.7 KiB
references/mastra.md6.7 KiB
references/mirascope.md8.0 KiB
references/mistral.md8.3 KiB
references/ollama.md8.2 KiB
references/openai-agents.md8.2 KiB
references/openai.md10.2 KiB
references/openrouter.md8.3 KiB
references/perplexity.md8.3 KiB
references/portkey.md8.2 KiB
references/pydantic-ai.md7.4 KiB
references/semantic-kernel.md7.7 KiB
references/smolagents.md7.5 KiB
references/together-ai.md8.3 KiB
references/traces.md8.5 KiB
references/vercel-ai.md10.0 KiB
references/xai.md8.2 KiB

references/calculating-costs.md

AI agents: this is one page from PostHog's docs. Full index of Markdown docs for LLMs: https://posthog.com/llms.txt

Calculating LLM costs

Canonical URL: https://posthog.com/docs/ai-observability/calculating-costs

Cloud region: This page uses US Cloud hosts. For an EU Cloud project, use https://eu.i.posthog.com instead of https://us.i.posthog.com (ingestion host).

How are LLM costs calculated?

PostHog calculates cost based on the number of input (prompt) and output (completion) tokens generated by specific AI models, plus additional cost components like per-request and per-web-search pricing when applicable.

Model matching

We use OpenRouter's pricing data as our primary source to match your model and calculate costs. OpenRouter provides comprehensive pricing information for models across different providers, and we use both the $ai_provider and $ai_model properties from your events to find the exact pricing.

If OpenRouter doesn't have pricing data for a specific model, we fall back to our manually maintained pricing database for additional model support.

Cost components

The total cost ($ai_total_cost_usd) is calculated from multiple components:

Token-based costs: Input (prompt) and output (completion) tokens
Cache costs: Cached token reads and writes (when applicable)
Request-based costs: Some models charge per request in addition to token costs
Web search costs: Some models charge per web search performed

For cached LLM responses, our pricing models include cached token pricing which we automatically apply.

We also take into account the reasoning / thinking tokens for models that support it.

Cache token accounting

Different LLM providers report cache tokens differently:

Exclusive counting (Anthropic/Claude) - Cache tokens are separate from $ai_input_tokens. For example, if you have 100 input tokens and 50 cached tokens, $ai_input_tokens is 100.
Inclusive counting (OpenAI and most others) - Cache tokens are included in $ai_input_tokens. Using the same example, $ai_input_tokens is 150.

PostHog auto-detects which counting style to use based on the $ai_provider and $ai_model properties. For Anthropic and Claude models, PostHog assumes exclusive counting. For all other providers, it assumes inclusive counting.

Overriding auto-detection

If you're manually capturing events or using a framework wrapper that normalizes token counts differently, override the auto-detection by setting the $ai_cache_reporting_exclusive property:

$ai_cache_reporting_exclusive: true - Cache tokens are separate from input tokens (Anthropic-style)
$ai_cache_reporting_exclusive: false - Input tokens already include cache tokens

When not set, PostHog resolves this automatically and writes the resolved value back to the event for downstream consumers.

Setting custom pricing

You can override PostHog's automatic cost calculation by providing custom pricing for your LLM models. This is useful when:

You have negotiated custom pricing with your LLM provider
You're using a model that PostHog doesn't support yet
PostHog's automatic pricing doesn't match your specific use case

Option 1: Custom price per token

If you know your pricing per token, you can set the following custom properties when calling your LLM:

$ai_input_token_price (required): Price per input/prompt token
$ai_output_token_price (required): Price per output/completion token
$ai_cache_read_token_price (optional): Price per cached token read
$ai_cache_write_token_price (optional): Price per cached token write
$ai_request_price (optional): Price per request (for models that charge per request)
$ai_web_search_price (optional): Price per web search (for models that charge per search)

Important: Token prices should be per individual token, not per million tokens. For example, if your provider charges $0.03 per 1M tokens, you would set $ai_input_token_price: 0.00000003 (0.03 / 1,000,000).

JavaScript

import { OpenAI } from '@posthog/ai'
import { PostHog } from 'posthog-node'

const phClient = new PostHog(
  '<ph_project_token>',
  { host: 'https://us.i.posthog.com' }
)

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  posthog: phClient
})

const response = await openai.responses.create({
  model: 'my-custom-model',
  messages: [{ role: 'user', content: 'Hello' }],
  posthogProperties: {
    $ai_input_token_price: 0.00000003,   // $0.03 per 1M tokens = $0.00000003 per token
    $ai_output_token_price: 0.00000006,  // $0.06 per 1M tokens = $0.00000006 per token
    // Optional: cache pricing
    $ai_cache_read_token_price: 0.000000015,
    $ai_cache_write_token_price: 0.0000000375
  }
})

Python

from posthog.ai.openai import OpenAI
from posthog import Posthog

posthog = Posthog(
    "<ph_project_token>",
    host="https://us.i.posthog.com"
)

client = OpenAI(
    api_key="sk-...",
    posthog_client=posthog
)

response = client.responses.create(
    model="my-custom-model",
    messages=[{"role": "user", "content": "Hello"}],
    posthog_properties={
        "$ai_input_token_price": 0.00000003,   # $0.03 per 1M tokens = $0.00000003 per token
        "$ai_output_token_price": 0.00000006,  # $0.06 per 1M tokens = $0.00000006 per token
        # Optional: cache pricing
        "$ai_cache_read_token_price": 0.000000015,
        "$ai_cache_write_token_price": 0.0000000375
    }
)

Both $ai_input_token_price and $ai_output_token_price must be provided for custom pricing to take effect. PostHog will then calculate the total cost based on the token counts and your custom prices.

Option 2: Pre-calculated costs

If you're manually capturing LLM events and have already calculated the total costs yourself, you can send them directly:

$ai_input_cost_usd: Total cost for input/prompt tokens in USD
$ai_output_cost_usd: Total cost for output/completion tokens in USD
$ai_request_cost_usd: Total cost for requests in USD
$ai_web_search_cost_usd: Total cost for web searches in USD

JavaScript

// After making your LLM call and calculating costs
posthog.capture('$ai_generation', {
  $ai_trace_id: traceId,
  $ai_model: 'my-custom-model',
  $ai_provider: 'my-provider',
  $ai_input_tokens: inputTokens,
  $ai_output_tokens: outputTokens,
  $ai_input_cost_usd: 0.0042,
  $ai_output_cost_usd: 0.0028
  // ... other required properties
})

Python

# After making your LLM call and calculating costs
posthog.capture(
    event='$ai_generation',
    properties={
        '$ai_trace_id': trace_id,
        '$ai_model': 'my-custom-model',
        '$ai_provider': 'my-provider',
        '$ai_input_tokens': input_tokens,
        '$ai_output_tokens': output_tokens,
        '$ai_input_cost_usd': 0.0042,
        '$ai_output_cost_usd': 0.0028
        # ... other required properties
    }
)

PostHog will use these values directly without any additional calculation.

Precedence

Cost calculation follows this precedence order:

Pre-calculated costs ($ai_input_cost_usd, $ai_output_cost_usd, $ai_request_cost_usd, $ai_web_search_cost_usd): These values are used directly without any additional calculation
Custom pricing ($ai_input_token_price, $ai_output_token_price, $ai_request_price, $ai_web_search_price): PostHog calculates costs from token counts and usage metrics using your custom prices
Automatic model matching: PostHog matches your model using OpenRouter's pricing data first, then falls back to manually maintained pricing if needed

Cost calculation metadata

When PostHog calculates costs automatically, it sets the following metadata properties on the event to help you understand how costs were determined:

Property	Description	Examples
`$ai_model_cost_used`	The model identifier used for cost calculation. May differ from the requested model when a variant or alias is resolved.	`openai/gpt-4o-mini`
`$ai_cost_model_source`	Where the cost data for this model was sourced from.	`openrouter`, `manual`, `custom`, `passthrough`
`$ai_cost_model_provider`	The provider used to look up the cost for this model.	`openai`, `anthropic`, `custom`

These properties are useful for debugging cost discrepancies or understanding which pricing was applied when using model aliases or custom configurations.

You can find the code for this on GitHub.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an MCP server

Security

What our scanner noticed· Low risk

Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

PostHog

Publisher

PostHog (Official)

Source

PostHog/skills · skills/omnibus/instrument-llm-analytics · commit a5d30e7

Licence

MIT License (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

AutoGineer is not affiliated with or endorsed by PostHog.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download instrument-llm-analytics.zip

From the author

AutoGineer summary

What it does

When to use it

Advantages

Usage examples

Install

Documentation by PostHog

Add PostHog AI Observability

Instructions

Reference files

Key principles

Files

references/calculating-costs.md

Calculating LLM costs

On this page

How are LLM costs calculated?

Model matching

Cost components

Cache token accounting

Overriding auto-detection

Setting custom pricing

Option 1: Custom price per token

JavaScript

Python

Option 2: Pre-calculated costs

JavaScript

Python

Precedence

Cost calculation metadata

Requirements and compatibility

Security

Attribution and licence

Related skills

Claude API

Transformers

Open Notebook

Vision Sft

LLM Evaluation

Lora Qlora Recipes

Download ZIP