P95 / P99 Latency

Percentile latency metrics describe the distribution of response times across a population of requests by answering the question: "What is the maximum latency experienced by X% of requests?" They are computed by sorting all observed latency values and selecting the value at a specific position in the sorted array.

Formally, the Pth percentile of a dataset is the value below which P% of observations fall. For a dataset of N latency measurements sorted in ascending order:

• P50 (median): The value at position N × 0.50. Half of all requests are faster, half are slower. This is the "typical" experience.
• P75: The value at position N × 0.75. Three-quarters of requests complete within this time.
• P90: 90% of requests are at or below this threshold. Often used as a first-tier alert boundary.
• P95: The industry-standard SLA metric. Only 5% of requests exceed this value. At 100,000 requests per day, 5,000 requests are slower than your P95.
• P99: Captures the tail of the distribution. Only 1 in 100 requests is slower. At scale, this still represents thousands of real users per day.
• P99.9: The extreme tail. 1 in 1,000 requests. Relevant for high-traffic services where even 0.1% represents significant user impact.

In database systems, percentiles are typically computed using the PERCENTILE_CONT function, which performs linear interpolation between adjacent data points when the exact percentile position falls between two observations. For real-time monitoring, many observability platforms use approximate algorithms like t-digest or HDR histograms to compute percentiles over streaming data without storing every individual measurement.

For LLM API monitoring, the most operationally relevant percentiles are P50 (baseline performance), P95 (SLA target), and P99 (tail-latency investigation). CostHawk computes all three using PERCENTILE_CONT over raw request logs, providing exact values rather than approximations. This precision matters when you are holding a provider accountable to an SLA or diagnosing whether a latency regression is real or a statistical artifact.

A critical nuance: percentiles are not additive. If Service A has a P99 of 200 ms and Service B has a P99 of 300 ms, the P99 of a request that hits both services sequentially is not 500 ms. The combined P99 depends on the correlation structure between the two distributions. This makes end-to-end percentile monitoring (which CostHawk provides at the request level) more valuable than aggregating percentiles from individual components.

The arithmetic mean (average) is the most commonly reported latency statistic and also the most misleading. Averages are distorted by outliers and completely mask the shape of the latency distribution. To understand why, consider two hypothetical LLM API endpoints with identical average latencies:

Endpoint A has a tight, well-behaved distribution: most requests cluster near the average, and even the worst requests are only about 2x the median. Endpoint B has a bimodal distribution: most requests are fast (400 ms median), but a significant minority take 10x-100x longer, dragging the average up to 900 ms. If you only look at the average, these endpoints appear identical. In reality, Endpoint B delivers a much worse user experience because 5% of users wait over 3.5 seconds and 1% wait over 12 seconds.

LLM APIs are particularly prone to bimodal and heavy-tailed latency distributions because of several factors:

• Variable output length: A request that generates 50 tokens finishes in 200 ms. A request to the same model that generates 2,000 tokens takes 8,000 ms. Both contribute to the same latency distribution.
• Provider-side batching: AI providers batch multiple requests onto the same GPU for throughput efficiency. When a batch includes one request with a very long output, all other requests in the batch may experience increased latency while waiting for the batch to complete.
• Queue wait time: During peak usage periods, requests may sit in a provider's queue for 500 ms to several seconds before GPU resources become available. This creates a bimodal pattern: requests that get immediate GPU access are fast; requests that queue are slow.
• Cold starts and model loading: Some providers dynamically scale their GPU fleet. The first request to a newly spun-up instance may incur a multi-second cold start penalty while model weights are loaded into GPU memory.
• Network variability: Cross-region API calls add 50-200 ms of network latency, and network jitter can add unpredictable variation.

The practical consequence is clear: never use averages as your primary latency metric for LLM APIs. Report P50 for the "typical" experience, P95 for your SLA target, and P99 to track tail behavior. When someone says "our average latency is 500 ms," the correct response is: "What is your P99?"

Latency characteristics vary enormously across models and providers. Smaller models on optimized infrastructure deliver sub-second P99 latencies, while frontier models processing large contexts can have P99 latencies exceeding 30 seconds. The following table shows representative P95 and P99 latencies observed in production workloads during Q1 2026, based on requests with 1,000 input tokens and 500 output tokens (a typical conversational query):

Key observations from this data:

• The P99/P50 ratio ranges from 4.8x to 7.7x. This means the slowest 1% of requests take 5-8x longer than the median. This ratio is significantly higher than traditional web APIs, where P99/P50 is typically 2-4x.
• Smaller models have tighter distributions. GPT-4o mini, Claude 3.5 Haiku, Gemini Flash, and Mistral Small all have P99/P50 ratios around 5.0x, while their larger counterparts range from 6.2x to 7.7x. Larger models have more variable decode times and are more susceptible to batching delays.
• Provider infrastructure matters. Google's Gemini Flash has the lowest absolute P99 (1,200 ms) thanks to custom TPU hardware optimized for inference. Mistral Small on optimized inference endpoints achieves a 900 ms P99.
• These numbers scale with output length. For a request generating 2,000 tokens instead of 500, multiply all values by roughly 3-4x. A GPT-4o request generating 2,000 tokens can easily hit a P99 of 15,000-18,000 ms.

Input context size also affects latency, primarily through the prefill (time-to-first-token) phase. A request with 50,000 input tokens will have a TTFT 5-10x longer than a request with 1,000 input tokens, adding 500-2,000 ms to the total latency for frontier models. This is why CostHawk tracks both TTFT and total latency separately — a high P95 TTFT with a normal decode rate indicates a context-size problem, while a normal TTFT with a high P95 total latency indicates a long-output problem. Different root causes require different optimization strategies.

A Service Level Agreement (SLA) defines the performance guarantees your system commits to — both internally (engineering team to product team) and externally (your product to your customers). For LLM-powered features, latency SLAs must be expressed as percentile thresholds with clear measurement methodology, scope, and consequences for violation.

Choosing the right percentile for your SLA:

• P95 is the most common SLA metric for LLM API features. It balances operational achievability with user experience coverage — 95% of users get the guaranteed experience, and the 5% tail is monitored but not contractually bound.
• P99 SLAs are appropriate for critical paths — payment processing, safety-critical decisions, or premium-tier customers. They are harder to meet and require more infrastructure investment (redundancy, failover, request hedging).
• P50 SLAs are rarely used because they only guarantee the median experience. A system could have a 300 ms P50 and a 30-second P99, which meets a P50 SLA while delivering an abysmal experience to half a percent of users.

How to define an LLM latency SLA:

1. Segment by request profile. A single SLA for all requests is meaningless when a 100-token classification takes 200 ms and a 4,000-token generation takes 8 seconds. Define SLA tiers based on expected output length or task type. Example: "Tier 1 (classification/extraction, <200 output tokens): P95 < 800 ms. Tier 2 (generation, 200-1,000 output tokens): P95 < 3,000 ms. Tier 3 (long-form, >1,000 output tokens): P95 < 8,000 ms."
2. Specify the measurement window. Is the SLA measured over 1 hour, 1 day, or 1 calendar month? Shorter windows detect regressions faster but are more susceptible to transient spikes. A common approach: measure over rolling 24-hour windows, alert on 1-hour windows, and report monthly for business reviews.
3. Define what counts as a request. Does the SLA include retried requests? Requests that timeout? Requests during provider outages? Typically, provider-side outages are excluded (covered by the provider's own SLA), but application-level retries are included because they affect the user's perceived latency.
4. Set a compliance target. "P95 < 2,000 ms, 99.5% of measurement windows compliant" means you can have 3.6 hours per month where P95 exceeds the target without violating the SLA. This provides headroom for maintenance windows and transient provider degradation.

Example internal SLA for an AI-powered customer support chatbot:

CostHawk supports SLA monitoring by computing percentile latencies in real time across all requests flowing through wrapped keys. You can configure alerts that fire when P95 or P99 exceeds a threshold for a given model, project, or API key — turning your SLA from a document into an automated enforcement mechanism.

Tail latency — the P95, P99, and beyond — is where user experience degrades and where the most impactful optimizations often live. Reducing tail latency for LLM APIs requires a different playbook than reducing median latency, because tail events are caused by specific, identifiable factors rather than baseline model speed. Here are seven proven strategies, ordered by typical impact:

1. Reduce output token count. The single largest driver of LLM latency is the number of tokens generated. Each output token requires a sequential decode step, so a request generating 2,000 tokens takes roughly 4x longer than one generating 500 tokens. The highest-latency requests in your distribution are almost always the ones generating the longest outputs. Set max_tokens to enforce a ceiling, and instruct the model to be concise. Cutting your max output from 2,000 to 800 tokens can reduce P99 by 50-60%.

2. Reduce input context size. Large input contexts increase time-to-first-token (TTFT) because the model must process all input tokens during the prefill phase. A request with 100,000 input tokens may have a TTFT of 3-5 seconds on frontier models, compared to 200-400 ms for a 1,000-token input. If your P95 TTFT is high, audit your context assembly pipeline: are you including full documents when summaries would suffice? Are you sending entire conversation histories when only the last 5 turns are relevant?

3. Implement request hedging. Send the same request to two provider endpoints simultaneously and use whichever responds first. This is the most effective technique for reducing P99 because it converts tail-latency events into a race where the fastest response wins. The cost is 2x token spend for hedged requests, but if you only hedge the slowest 5% (detected by a timeout threshold), the cost increase is manageable. For a request that times out after 3 seconds, fire a parallel request to an alternative model or region.

4. Choose the right model tier. If your P99 on GPT-4o is 4,200 ms and your product can tolerate the quality of GPT-4o mini (P99 of 1,600 ms), the fastest path to lower tail latency is model routing. Many teams implement a tiered system: send latency-sensitive requests to a fast model and quality-sensitive requests to a frontier model. This reduces P99 for the latency-sensitive tier by 60-70%.

5. Use streaming responses. Streaming does not reduce total latency (the last token arrives at the same time), but it dramatically improves perceived latency because the user sees the first token within the TTFT window (typically 200-800 ms) rather than waiting for the entire response. For interactive applications, streaming is effectively mandatory — a 5-second wait followed by a complete response feels much slower than seeing tokens appear after 300 ms.

6. Implement regional routing. If your users are globally distributed but your API calls all route to a single provider region, users far from that region add 100-300 ms of network round-trip time. Route requests to the nearest provider region (OpenAI and Anthropic both have multi-region deployments, and Google Cloud has Gemini endpoints in multiple regions). Network latency is additive and constant, so reducing it from 200 ms to 30 ms improves every percentile equally.

7. Add client-side timeouts with graceful degradation. Set a timeout at the percentile level where user experience becomes unacceptable (e.g., 5 seconds). When a request exceeds the timeout, cancel it and fall back to a cached response, a simpler model, or a non-AI code path. This does not reduce the actual tail latency but caps the user-perceived latency, turning P99 outliers into fast fallback responses. Track timeout rates in CostHawk to understand how often fallbacks trigger and whether the underlying latency problem is worsening.

Monitoring percentile latencies in production requires purpose-built infrastructure because percentiles cannot be computed from pre-aggregated data — you need access to individual request-level measurements or statistically sound approximations. Here is how to build a robust percentile monitoring pipeline for LLM API calls.

Data collection: Every API request must record its total latency (and ideally TTFT, decode time, and queue wait time as separate fields). CostHawk captures this automatically for all requests routed through wrapped keys: each request log includes latency_ms, ttft_ms (when streaming), input_tokens, output_tokens, model, api_key_id, and timestamp. This per-request granularity is essential because percentiles must be computed over the raw data, not over averages or pre-bucketed histograms.

Computation methods:

• Exact computation (SQL): For datasets that fit in a single query (up to millions of rows with proper indexing), use PERCENTILE_CONT in PostgreSQL: SELECT PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY latency_ms) AS p95, PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY latency_ms) AS p99 FROM request_logs WHERE timestamp > NOW() - INTERVAL '1 hour'. This is the approach CostHawk uses for dashboard queries and SLA reporting.
• Approximate computation (streaming): For real-time alerting over high-throughput streams, use t-digest or DDSketch algorithms that maintain a compact data structure approximating the full distribution. These algorithms can compute percentiles with <1% error while using only a few kilobytes of memory, making them suitable for per-second percentile computation across thousands of API keys.
• Histogram buckets (Prometheus-style): Define latency buckets (e.g., 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s, 30s) and increment a counter for each request. Percentiles are estimated by interpolating between bucket boundaries. This approach is less accurate than t-digest but integrates naturally with Prometheus/Grafana monitoring stacks.

Alerting on percentiles:

Configure alerts at two levels:

1. Warning alert: P95 exceeds target over a 15-minute window. This catches gradual degradation (e.g., increasing context sizes, provider performance regression) early enough to investigate before users are heavily impacted.
2. Critical alert: P99 exceeds 2x the target over a 5-minute window. This fires on sudden, severe latency spikes — provider outages, thundering-herd effects after a retry storm, or a code deployment that accidentally increased prompt sizes.

Dashboards:

An effective latency dashboard shows:

• P50, P95, and P99 as three overlaid time-series lines, so you can see whether a latency increase affects all requests (P50 rises) or only the tail (P99 rises while P50 is stable).
• Latency broken down by model, so you can see if one model is disproportionately contributing to tail latency.
• A latency heatmap (time on x-axis, latency buckets on y-axis, color intensity for request count) that reveals the full distribution shape over time, exposing bimodal patterns that percentile lines alone might miss.
• Correlation with request volume, so you can distinguish demand-driven latency increases (more requests = more queuing) from infrastructure-driven ones (same volume, slower responses).

CostHawk's dashboard provides all of these views out of the box for any request flowing through wrapped keys or synced via the MCP server. Each chart is filterable by project, API key, model, and time range, so you can drill into exactly the slice of traffic that matters for your investigation.