LLM Observability

LLM observability is the comprehensive practice of collecting, correlating, and acting on telemetry data from every layer of an LLM-powered application. It draws from three foundational pillars of traditional observability — metrics, traces, and logs — but extends each with AI-specific dimensions that have no equivalent in conventional software monitoring.

Metrics in LLM observability go beyond request counts and error rates. The core metrics include:

• Token throughput: Input and output tokens per second, minute, and hour — broken down by model, endpoint, and user segment.
• Cost rate: Dollars per hour, day, or month — tracked in real time so teams can detect budget overruns as they happen, not after the invoice arrives.
• Latency distribution: Time-to-first-token (TTFT), end-to-end latency, and tokens-per-second generation speed — segmented by model and request complexity.
• Error rates: Rate limit hits (HTTP 429), server errors (HTTP 500/503), content filtering blocks, and malformed responses.
• Quality scores: Application-specific metrics like classification accuracy, hallucination rate, format compliance, and user satisfaction ratings.

Traces in LLM observability capture the full lifecycle of an AI request. A single user interaction might involve multiple LLM calls: a router call to classify intent, a retrieval step to fetch context from a vector database, a generation call to produce the response, and a guard call to check the output for safety. An LLM trace links all of these steps together, showing the total token consumption, latency, and cost for the entire chain — not just individual API calls.

Logs in LLM observability record the actual prompts and completions flowing through your system. This is essential for debugging quality issues ("why did the model say that?"), auditing for compliance, and building evaluation datasets for prompt optimization. However, logging LLM inputs and outputs introduces privacy and storage considerations that do not exist with traditional application logs — a single day of production LLM logs can easily exceed 100 GB for high-traffic applications.

The synthesis of these three pillars creates a unified observability layer that answers questions no single data source can address alone: Which model is giving us the best quality-per-dollar? Is our cost spike caused by more requests or longer prompts? Did last week's prompt change actually improve accuracy, or just make responses longer?

LLM observability is not optional for production AI applications. It is the foundational capability that makes every other optimization — model routing, prompt engineering, caching, budget enforcement — possible. Without it, you are making decisions based on intuition and monthly invoices. With it, you are making decisions based on real-time, per-request data.

Teams that have invested heavily in traditional APM tools like Datadog, New Relic, or Grafana often assume these tools are sufficient for monitoring LLM-powered applications. They are not. While traditional APM provides valuable infrastructure-level visibility, it is blind to the dimensions that matter most for AI applications. Here is a detailed comparison:

The fundamental difference is that traditional APM treats the AI API as a black box: it sees that a request went out and a response came back, with some latency and a status code. LLM observability opens that black box and examines what happened inside: how many tokens were consumed, what model processed them, how much it cost, and whether the output was actually good.

Consider a concrete scenario. Your AI-powered search feature starts receiving user complaints about slow responses. Traditional APM shows you that the /api/search endpoint P95 latency increased from 2 seconds to 8 seconds. That is useful, but it does not tell you why. LLM observability shows you that the average input token count increased from 1,200 to 4,800 because a recent feature change started including full document content instead of summaries in the search context. The latency increase is a direct consequence of the token increase — the model takes longer to process 4x more input. The fix is clear: revert to document summaries or implement chunked retrieval. Without LLM-specific observability, the team might have wasted days investigating infrastructure causes (Is the API provider slow? Is our network degraded?) when the root cause was purely application-level.

The most effective approach is to layer LLM observability on top of traditional APM. Use Datadog or Grafana for infrastructure health (CPU, memory, network, container metrics) and a purpose-built LLM observability tool like CostHawk for AI-specific telemetry (tokens, costs, quality, model performance). This gives you complete visibility across both dimensions.

Effective LLM observability requires tracking a specific set of metrics that collectively describe the health, cost, and quality of your AI features. Below is a comprehensive reference of the metrics every team should track, organized by category:

Not every team needs every metric from day one. The recommended adoption sequence is:

1. Week 1: Cost per request, daily spend, input/output token counts, error rate. These give you basic cost visibility and reliability monitoring.
2. Month 1: Add latency metrics (TTFT, end-to-end, tokens/sec), retry rate, and cost-per-user-action. These enable performance optimization and product economics analysis.
3. Month 3: Add quality metrics (format compliance, hallucination sampling, user feedback). These require more instrumentation but are essential for long-term quality assurance.

CostHawk captures cost, token, latency, and reliability metrics automatically for every request routed through its proxy or synced via the MCP server. Quality metrics can be added through custom tags and the evaluation API.

The LLM observability market has matured rapidly since 2024, with several specialized tools competing alongside extensions from traditional APM vendors. Here is a practical overview of the major players and how they compare:

Helicone is an open-source LLM observability platform that works as a proxy layer in front of your LLM API calls. You swap your API base URL to route through Helicone, and it logs every request with token counts, latency, costs, and the full prompt/completion text. Helicone excels at per-request logging and provides a clean UI for browsing individual requests. Its main strengths are the open-source codebase, easy integration (one-line base URL change), and a generous free tier. Its main limitations are less sophisticated cost analytics (no automated anomaly detection or budget enforcement) and a focus on individual request debugging rather than aggregate cost trends.

Langfuse is an open-source observability and analytics platform designed around the concept of "traces" — multi-step LLM workflows that chain multiple API calls together. Langfuse provides SDKs for Python and JavaScript that let you instrument complex agent workflows with nested spans, scoring, and metadata. Its strongest differentiator is trace-level analytics: you can see the total cost and latency of an entire multi-step agent execution, not just individual API calls. Langfuse integrates well with LangChain and LlamaIndex. Its main limitations are a more complex setup compared to proxy-based tools, and its self-hosted deployment requires significant infrastructure management.

Portkey is a commercial AI gateway that combines observability with operational features like load balancing, fallback routing, and caching. Portkey sits as a proxy between your application and LLM providers, providing real-time dashboards for cost, latency, and reliability. Its key differentiator is the operational layer: automatic failover when a provider goes down, intelligent load balancing across models, and built-in prompt caching. Portkey is strongest for teams running multi-provider, multi-model architectures that need both observability and resilience. Its main limitation is that it is a commercial product with per-request pricing that adds to your AI spend.

CostHawk is purpose-built for the cost and financial dimension of LLM observability. While other tools focus primarily on debugging and tracing, CostHawk centers on the question that matters most to engineering leaders and finance teams: how much are we spending, where is it going, and how do we reduce it? CostHawk provides real-time cost dashboards with per-model, per-key, per-project, and per-tag breakdowns. Its anomaly detection system flags unusual spending patterns within minutes, not days. Budget alerts notify you before you exceed thresholds. The wrapped-key proxy captures every request with zero code changes, and the MCP server syncs local development tool usage (Claude Code, Codex) that other observability tools miss entirely. CostHawk is strongest for teams that need to treat AI spend as a first-class financial metric — with budget enforcement, cost attribution, and savings recommendations that directly reduce your bill.

How these tools compare in practice:

Many teams use a combination: CostHawk for cost management and Langfuse or Helicone for detailed request-level debugging. The tools are complementary, not mutually exclusive.

Implementing LLM observability does not require a massive upfront investment. The most effective approach is incremental: start with basic cost and token tracking, then add layers of sophistication as your needs grow. Here is a practical implementation roadmap:

Phase 1: Basic telemetry (1-2 hours to set up)

The fastest path to observability is intercepting every LLM API call and recording the response metadata. Every major provider returns a usage object in the API response that includes prompt_tokens and completion_tokens. At minimum, log these fields for every request:

{
  "timestamp": "2026-03-16T14:23:01Z",
  "model": "gpt-4o",
  "provider": "openai",
  "input_tokens": 1247,
  "output_tokens": 382,
  "latency_ms": 2340,
  "status": 200,
  "cost_usd": 0.006937,
  "endpoint": "/api/chat",
  "project": "customer-support"
}

If you are using CostHawk, this step is automatic: route your API calls through a CostHawk wrapped key and every field above is captured without any code changes. For custom implementations, wrap your API client with a logging middleware that extracts the usage data from each response.

Phase 2: Cost attribution and dashboards (1-2 days)

Raw telemetry data becomes useful when it is aggregated and visualized. Build or configure dashboards that show:

• Total daily spend by model and provider
• Average cost per request by endpoint or feature
• Token consumption trends over the past 7 and 30 days
• Top 10 most expensive endpoints or features
• Cost breakdown by environment (dev, staging, production)

CostHawk provides these dashboards out of the box. If you are building custom dashboards, tools like Grafana with a time-series database (InfluxDB, TimescaleDB) work well, though you will need to compute costs from raw token counts and model pricing — which requires maintaining an up-to-date pricing table for every model.

Phase 3: Alerting and anomaly detection (1 week)

Static budget alerts ("notify me when daily spend exceeds $500") are a good starting point, but they miss gradual cost creep and can be too slow to catch sudden spikes. Implement anomaly detection that compares current spending patterns to historical baselines:

• Alert when hourly spend exceeds 2x the trailing 7-day average for that hour
• Alert when average input token count per request increases by more than 30% (prompt bloat indicator)
• Alert when error rate exceeds 5% (potential retry storm indicator)
• Alert when a specific model's spend exceeds its monthly budget allocation

CostHawk's anomaly detection uses statistical methods (z-score analysis with configurable sensitivity) and integrates with Slack, PagerDuty, and email for notifications. Custom implementations can use a simple rolling-window z-score calculation over your telemetry data.

Phase 4: Quality monitoring (2-4 weeks)

Quality monitoring requires more instrumentation than cost monitoring because there is no universal quality metric — quality is application-specific. Common approaches include:

• Format compliance checking: Parse every response against your expected schema and log success/failure rates. For JSON outputs, validate against a JSON schema. For structured text, check for required sections or fields.
• LLM-as-judge evaluation: Use a cheaper model to evaluate the quality of a more expensive model's output. For example, send GPT-4o mini a rubric and the original response, and ask it to score accuracy on a 1-5 scale. This costs pennies per evaluation and can run on a sample of production traffic.
• User feedback loops: Add thumbs up/down or 1-5 star ratings to your AI-generated outputs and log them alongside the request telemetry. This is the gold standard for quality measurement but requires product integration.

Phase 5: Optimization loop (ongoing)

Observability without action is just monitoring. The final phase is closing the optimization loop: using your observability data to make changes that reduce cost and improve quality, then measuring the impact of those changes. CostHawk's savings recommendations surface the highest-impact optimizations based on your actual usage patterns — such as "switching your /api/summarize endpoint from GPT-4o to GPT-4o mini would save $2,400/month based on last 30 days of traffic."

The irony of LLM observability is that the teams who need it most — those spending the most on AI APIs — are often the last to implement it. They are moving fast, shipping features, and treating AI costs as a problem for later. Here is what "later" typically looks like, drawn from real scenarios across hundreds of engineering teams:

Scenario 1: The invisible prompt bloat. A product team adds a "provide detailed explanations" instruction to the system prompt for a customer support chatbot. Average output length increases from 150 tokens to 450 tokens. Output tokens cost 4x more than input tokens, so the cost per request triples from $0.003 to $0.009. At 50,000 requests per day, this adds $300/day — $9,000/month — to the bill. Without observability, the team does not connect the prompt change to the cost increase for six weeks. Total wasted spend: $13,500.

Scenario 2: The forgotten dev environment. Developers testing AI features locally route all requests through the production API key. Each developer makes 200-500 API calls per day during active development. A team of 8 developers burns through 2,000-4,000 requests per day in development, at an average cost of $0.01 per request — $20-$40/day, or $600-$1,200/month — with zero production value. Without per-environment cost attribution, this spending is invisible. Multiply across a year and the dev environment has consumed $7,200-$14,400 in pure waste.

Scenario 3: The rate limit retry storm. An application hits a rate limit (HTTP 429) during a traffic spike. The retry logic uses exponential backoff, but a bug in the implementation resets the backoff counter on each new user request, causing hundreds of parallel retry loops. Each retry consumes the same tokens as the original request. Over 45 minutes, the application sends 12x the normal request volume before an engineer manually kills the retry loop. The incident costs $3,800 in excess API spend — more than the team's entire typical daily budget.

Scenario 4: The wrong model for the job. A team builds a document classification pipeline using Claude 3.5 Sonnet ($3.00/$15.00 per MTok) because "we want the best accuracy." The pipeline processes 200,000 documents per month, each requiring ~500 input tokens and ~50 output tokens. Monthly cost: $375 in input + $150 in output = $525. After finally implementing observability and benchmarking, the team discovers that Claude 3.5 Haiku ($0.80/$4.00 per MTok) achieves 97% of Sonnet's accuracy on this specific task. Switching saves $385/month — a 73% reduction. Without observability data to justify the experiment, the team ran Sonnet for 9 months, overspending by $3,465.

Scenario 5: The compliance audit nightmare. A healthcare application uses LLMs to summarize patient notes. During an audit, regulators ask for evidence of what data was sent to the LLM API, what model processed it, and what outputs were generated. Without LLM-specific logging, the team has no record of individual requests — only aggregate API billing data. The audit remediation effort costs 3 engineering-weeks and results in a compliance finding. Proper LLM observability with prompt/completion logging would have provided the audit trail automatically.

The common thread across all these scenarios is that the cost of not having observability always exceeds the cost of implementing it. CostHawk takes less than 30 minutes to set up (swap your API base URL or install the MCP server), and most teams identify optimization opportunities worth 3-10x the subscription cost within the first week. The question is not whether you can afford LLM observability — it is whether you can afford not to have it.