LLM Analytics Setup

PostHogOfficialCreated by PostHog

Skill
Tool integration
Data and AI
MIT
Low risk

Download ZIP

From the author

PostHog LLM analytics for all supported providers

— the description by PostHog

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a comprehensive guide for integrating PostHog's LLM analytics into applications using various AI providers and frameworks. By wrapping existing LLM clients with PostHog's instrumentation, developers can automatically capture detailed telemetry—including input/output tokens, latency, model names, and costs—without proxying their API calls. The skill covers a wide range of providers and frameworks, offering specific implementation patterns for each, and includes a fallback manual capture method for custom setups.

What it does

Instruments LLM calls to automatically capture $ai_generation events.
Tracks key metrics like latency, token usage, and total cost per generation.
Supports a broad ecosystem of LLM providers and AI frameworks.
Enables linking LLM generations to specific users for better observability.
Provides a manual capture interface for unsupported or custom LLM providers.
Ensures data privacy by distinguishing between event properties and PII.

When to use it

When you need to monitor the performance and cost of LLM calls in production.
When you want to debug AI agent behavior by tracing input and output sequences.
When you need to associate AI interactions with specific user sessions.
When you are building applications using frameworks like LangChain, AutoGen, or LlamaIndex.

Advantages

Non-intrusive integration that does not proxy your LLM API calls.
Centralized visibility into LLM costs and usage across different providers.
Automatic capture of critical performance metrics like time-to-first-token.
Flexible configuration allows for custom properties and trace IDs.

Usage examples

“How do I add PostHog analytics to my Anthropic Claude implementation in Python?”: The agent will provide the code to initialize the PostHog client and wrap the Anthropic client, enabling automatic event capture for all messages.create calls.
“I am using AutoGen for my agents, how can I track their LLM usage in PostHog?”: The agent will guide you to install the necessary extensions and show how to pass a wrapped OpenAI client into the AutoGen OpenAIChatCompletionClient.

Install

Download the skill as a ZIP:
https://mp-staging.autogineer.com/agent-skills/posthog/llm-analytics-setup/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip llm-analytics-setup.zip -d .agents/skills

Documentation by PostHog

PostHog LLM analytics

This skill helps you add PostHog LLM analytics to any application using AI/LLM providers.

Reference files

references/openai.md - Openai llm analytics installation - docs
references/azure-openai.md - Azure openai llm analytics installation - docs
references/anthropic.md - Anthropic llm analytics installation - docs
references/google.md - Google llm analytics installation - docs
references/cohere.md - Cohere llm analytics installation - docs
references/mistral.md - Mistral llm analytics installation - docs
references/perplexity.md - Perplexity llm analytics installation - docs
references/deepseek.md - Deepseek llm analytics installation - docs
references/groq.md - Groq llm analytics installation - docs
references/together-ai.md - Together ai llm analytics installation - docs
references/fireworks-ai.md - Fireworks ai llm analytics installation - docs
references/xai.md - Xai llm analytics installation - docs
references/cerebras.md - Cerebras llm analytics installation - docs
references/hugging-face.md - Hugging face llm analytics installation - docs
references/ollama.md - Ollama llm analytics installation - docs
references/openrouter.md - Openrouter llm analytics installation - docs
references/langchain.md - Langchain llm analytics installation - docs
references/llamaindex.md - Llamaindex llm analytics installation - docs
references/crewai.md - Crewai llm analytics installation - docs
references/autogen.md - Autogen llm analytics installation - docs
references/dspy.md - Dspy llm analytics installation - docs
references/langgraph.md - Langgraph llm analytics installation - docs
references/pydantic-ai.md - Pydantic ai llm analytics installation - docs
references/vercel-ai.md - Vercel ai SDK llm analytics installation - docs
references/litellm.md - Litellm llm analytics installation - docs
references/instructor.md - Instructor llm analytics installation - docs
references/semantic-kernel.md - Semantic kernel llm analytics installation - docs
references/mirascope.md - Mirascope llm analytics installation - docs
references/mastra.md - Mastra llm analytics installation - docs
references/smolagents.md - Smolagents llm analytics installation - docs
references/openai-agents.md - Openai agents SDK llm analytics installation - docs
references/portkey.md - Portkey llm analytics installation - docs
references/helicone.md - Helicone llm analytics installation - docs
references/manual-capture.md - Manual capture llm analytics installation - docs
references/basics.md - Llm analytics basics - docs
references/traces.md - Traces - docs
references/calculating-costs.md - Calculating llm costs - docs

Each provider reference contains installation instructions, SDK setup, and code examples specific to that provider or framework. Find the reference that matches the user's stack and follow its instructions.

If the user's provider isn't listed, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.

Key principles

Environment variables: Always use environment variables for PostHog and LLM provider keys. Never hardcode them.
Minimal changes: Add LLM analytics alongside existing LLM calls. Don't replace or restructure existing code.
Trace all generations: Capture input tokens, output tokens, model name, latency, and costs for every LLM call.
Link to users: Associate LLM generations with identified users via distinct IDs when possible.
One provider at a time: Only instrument the provider(s) the user is actually using. Don't add instrumentation for providers not present in the codebase.

Framework guidelines

Remember that source code is available in the venv/site-packages directory
posthog is the Python SDK package name
Install dependencies with pip install posthog or pip install -r requirements.txt and do NOT use unquoted version specifiers like >= directly in shell commands
In CLIs and scripts: MUST call posthog.shutdown() before exit or all events are lost
Always use the Posthog() class constructor (instance-based API) instead of module-level posthog.api_key config
Always include enable_exception_autocapture=True in the Posthog() constructor to automatically track exceptions
NEVER send PII in capture() event properties — no emails, full names, phone numbers, physical addresses, IP addresses, or user-generated content
PII belongs in identify() person properties, NOT in capture() event properties. Safe event properties are metadata like message_length, form_type, boolean flags.
Register posthog_client.shutdown with atexit.register() to ensure all events are flushed on exit
The Python SDK has NO identify() method — use posthog_client.set(distinct_id=user_id, properties={...}) to set person properties, or use identify_context(user_id) within a context

Files

ATTRIBUTION.mdAdded by AutoGineer957 B
LICENSELicence file added by AutoGineer1.0 KiB
SKILL.md5.0 KiB
references/anthropic.md6.6 KiB
references/autogen.md4.5 KiB
references/azure-openai.md6.4 KiB
references/basics.md3.2 KiB
references/calculating-costs.md8.3 KiB
references/cerebras.md6.7 KiB
references/cohere.md6.7 KiB
references/crewai.md4.4 KiB
references/deepseek.md6.7 KiB
references/dspy.md4.4 KiB
references/fireworks-ai.md6.8 KiB
references/google.md7.6 KiB
references/groq.md6.7 KiB
references/helicone.md6.9 KiB
references/hugging-face.md6.8 KiB
references/instructor.md5.8 KiB
references/langchain.md6.9 KiB
references/langgraph.md6.6 KiB
references/litellm.md6.6 KiB
references/llamaindex.md4.7 KiB
references/manual-capture.md15.6 KiB
references/mastra.md5.1 KiB
references/mirascope.md4.5 KiB
references/mistral.md6.7 KiB
references/ollama.md6.7 KiB
references/openai-agents.md5.7 KiB
references/openai.md6.2 KiB
references/openrouter.md7.1 KiB
references/perplexity.md6.7 KiB
references/portkey.md7.0 KiB
references/pydantic-ai.md4.5 KiB
references/semantic-kernel.md4.5 KiB
references/smolagents.md4.3 KiB
references/together-ai.md6.7 KiB
references/traces.md5.8 KiB
references/vercel-ai.md4.5 KiB
references/xai.md6.6 KiB

references/mastra.md

Mastra LLM analytics installation - Docs

1
Install the PostHog SDK
Required
Setting up analytics starts with installing the PostHog SDK.
```
npm install @posthog/ai posthog-node
```
2
Install Mastra
Required
Install Mastra and a model provider SDK. Mastra uses the Vercel AI SDK under the hood, so you can use any Vercel AI-compatible model provider.
```
npm install @mastra/core @ai-sdk/openai
```
Proxy note
These SDKs do not proxy your calls. They only fire off an async call to PostHog in the background to send the data. You can also use LLM analytics with other SDKs or our API, but you will need to capture the data in the right format. See the schema in the manual capture section for more details.

Initialize PostHog and wrap your model

Required

Initialize PostHog with your project token and host from your project settings, then use withTracing from @posthog/ai to wrap the model you pass to your Mastra agent.

import { Agent } from "@mastra/core/agent";
import { PostHog } from "posthog-node";
import { withTracing } from "@posthog/ai";
import { createOpenAI } from "@ai-sdk/openai";
const phClient = new PostHog(
  '<ph_project_token>',
  { host: 'https://us.i.posthog.com' }
);
const openaiClient = createOpenAI({
  apiKey: 'your_openai_api_key',
  compatibility: 'strict'
});
const agent = new Agent({
  name: "my-agent",
  instructions: "You are a helpful assistant.",
  model: withTracing(openaiClient("gpt-4o"), phClient, {
    posthogDistinctId: "user_123", // optional
    posthogTraceId: "trace_123", // optional
    posthogProperties: { conversationId: "abc123" }, // optional
    posthogPrivacyMode: false, // optional
    posthogGroups: { company: "companyIdInYourDb" }, // optional
  }),
});

You can enrich LLM events with additional data by passing parameters such as the trace ID, distinct ID, custom properties, groups, and privacy mode options.

Use your Mastra agent

Required

Now, when your Mastra agent makes LLM calls, PostHog automatically captures an $ai_generation event for each one.

const result = await agent.generate("What is the capital of France?");
console.log(result.text);
phClient.shutdown();

Note: If you want to capture LLM events anonymously, don't pass a distinct ID to the request. See our docs on anonymous vs identified events to learn more.

You can expect captured $ai_generation events to have the following properties:

Property	Description
$ai_model	The specific model, like gpt-5-mini or claude-4-sonnet
$ai_latency	The latency of the LLM call in seconds
$ai_time_to_first_token	Time to first token in seconds (streaming only)
$ai_tools	Tools and functions available to the LLM
$ai_input	List of messages sent to the LLM
$ai_input_tokens	The number of tokens in the input (often found in response.usage)
$ai_output_choices	List of response choices from the LLM
$ai_output_tokens	The number of tokens in the output (often found in response.usage)
$ai_total_cost_usd	The total cost in USD (input + output)
[...]	See full list of properties

Verify traces and generations
Recommended
Confirm LLM events are being sent to PostHog
Let's make sure LLM events are being captured and sent to PostHog. Under LLM analytics, you should see rows of data appear in the Traces and Generations tabs.
Image: LLM generations in PostHogImage: LLM generations in PostHog
Check for LLM events in PostHog

Next steps

Recommended

Now that you're capturing AI conversations, continue with the resources below to learn what else LLM Analytics enables within the PostHog platform.

Resource	Description
Basics	Learn the basics of how LLM calls become events in PostHog.
Generations	Read about the $ai_generation event and its properties.
Traces	Explore the trace hierarchy and how to use it to debug LLM calls.
Spans	Review spans and their role in representing individual operations.
Anaylze LLM performance	Learn how to create dashboards to analyze LLM performance.

Community questions

Ask a question

Was this page useful?

HelpfulCould be better

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an API key, Needs Python

Security

What our scanner noticed· Low risk

Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

PostHog

Publisher

PostHog (Official)

Source

PostHog/skills · skills/posthog/llm-analytics/skills/setup · commit a5d30e7

Licence

MIT License (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-08

Also distributed in

PostHog/skills

AutoGineer is not affiliated with or endorsed by PostHog.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download llm-analytics-setup.zip

From the author

AutoGineer summary

What it does

When to use it

Advantages

Usage examples

Install

Documentation by PostHog

PostHog LLM analytics

Reference files

Key principles

Framework guidelines

Files

references/mastra.md

Mastra LLM analytics installation - Docs

Install the PostHog SDK

Install Mastra

Initialize PostHog and wrap your model

Use your Mastra agent

Verify traces and generations

Next steps

Community questions

Was this page useful?

Requirements and compatibility

Security

Attribution and licence

Related skills

Claude API

Transformers

Open Notebook

Vision Sft

LLM Evaluation

Lora Qlora Recipes

Download ZIP