Throughput

Throughput quantifies the productive capacity of your LLM-powered system. While the concept is simple — how much work gets done per unit of time — measuring throughput for LLM applications requires understanding which unit of measurement is appropriate for which context.

Requests per second (RPS) is the most intuitive throughput metric. It counts complete API calls regardless of their size. RPS is useful for capacity planning ("can our system handle 100 concurrent users?"), load balancer configuration, and application-level monitoring. However, RPS can be misleading for cost analysis because requests vary enormously in size: a classification request consuming 200 tokens and a document analysis request consuming 50,000 tokens are both "one request" but differ 250x in cost.

Tokens per second (TPS) measures the raw token processing rate of the system. This is the most granular throughput metric and the most useful for understanding model utilization and cost rate. TPS can be broken into input TPS (tokens sent to the model per second) and output TPS (tokens generated by the model per second). Because output tokens cost 4-5x more than input tokens, a system processing 1,000 output TPS costs significantly more than one processing 1,000 input TPS.

Tokens per minute (TPM) is the unit used by most LLM API providers for rate limits. OpenAI specifies rate limits in both RPM (requests per minute) and TPM. Anthropic uses tokens per minute for its rate tier definitions. Understanding your TPM consumption relative to your TPM allowance is essential for avoiding rate limit errors that degrade performance and waste money through retries.

Effective throughput vs raw throughput: An important distinction is between the total requests or tokens your system processes (raw throughput) and the requests or tokens that produce useful output (effective throughput). Retried requests, rate-limited requests, and requests that produce errors or unusable output consume throughput capacity and cost money but do not deliver value. If 8% of your requests are retries and 3% produce errors, your effective throughput is only 89% of your raw throughput — meaning 11% of your token spend is waste. CostHawk tracks both raw and effective throughput, highlighting the gap as an optimization opportunity.

Throughput as a cost rate: The most actionable way to think about throughput is as a cost rate — dollars per minute or dollars per hour. If your system processes 100,000 tokens per minute on GPT-4o, your cost rate is approximately $0.25/minute input + $1.00/minute output = $1.25/minute, or $75/hour. Viewing throughput in dollar terms makes it immediately clear when something changes: if your cost rate jumps from $75/hour to $120/hour, you know throughput increased by 60% — and you need to understand whether that increase is driven by legitimate demand growth or by a technical problem (retry storm, prompt bloat, model routing error).

Accurate throughput measurement requires instrumentation at multiple levels of your stack. Here is a practical guide to measuring each throughput dimension:

Application-level measurement (RPS):

Measure RPS at your API gateway or load balancer by counting the number of outbound LLM API calls per second. If you use CostHawk's proxy, this is automatic — every request is counted and timestamped. For custom measurement, instrument your LLM client wrapper to emit a counter metric on every call. Key considerations:

• Count requests at the point of API call, not at the point of user request. A single user action might trigger multiple LLM calls (router + generator + guardrail = 3 RPS from 1 user request).
• Separate successful requests from failed requests. A 429 rate limit error still consumed an attempt but did not produce useful output.
• Track RPS by model, endpoint, and API key for granular attribution.

Token-level measurement (TPS/TPM):

Extract token counts from every API response's usage object. Aggregate input tokens and output tokens separately over your desired time window. The formula is straightforward:

Input TPS = sum(input_tokens for all requests in 1-second window) / 1
Output TPS = sum(output_tokens for all requests in 1-second window) / 1
Total TPM = sum(all_tokens for all requests in 1-minute window)

Be precise about what you are counting: some providers include system prompt tokens in the input count, some count them separately. Reasoning model thinking tokens are typically included in the output token count even though they do not appear in the visible response. Always use the provider-reported token counts rather than client-side estimates for billing accuracy.

Cost-rate measurement ($/min):

The most actionable throughput metric multiplies token throughput by pricing:

Cost rate ($/min) = (Input TPM × input_price_per_token) + (Output TPM × output_price_per_token)

For a system processing 50,000 input TPM and 15,000 output TPM on GPT-4o:

Cost rate = (50,000 × $0.0000025) + (15,000 × $0.000010) = $0.125 + $0.150 = $0.275/min = $16.50/hour

CostHawk computes and displays cost rate in real time, allowing you to see your AI spend velocity at any moment — not just as a daily or monthly aggregate.

Throughput benchmarking:

Establish throughput baselines during normal operation so you can detect anomalies. Track your peak throughput (highest TPM in any 1-minute window over the past 30 days), average throughput (mean TPM during business hours), and ratio of peak to average. A peak-to-average ratio above 3:1 suggests bursty traffic patterns that may benefit from request queuing, batch processing, or provider-side burst capacity provisions.

The relationship between throughput and cost is deceptively simple at first glance — more throughput means more tokens, which means more money — but the details of how costs scale with throughput contain critical nuances that affect budgeting, architecture, and optimization strategy.

Linear scaling (the baseline):

At the most basic level, LLM API costs scale linearly with token throughput. Double your TPM and you double your cost. Process 10 million tokens per day and you pay 10x what you paid at 1 million tokens per day. There are no volume discounts in standard on-demand API pricing — your ten-millionth token costs exactly the same as your first. This linear relationship means that throughput growth must be carefully monitored and budgeted. A feature that works fine at 10,000 requests per day ($150/day) might become untenable at 100,000 requests per day ($1,500/day) without any code changes.

Sub-linear scaling opportunities:

Several techniques can make your cost scale sub-linearly with throughput — meaning costs grow more slowly than token volume:

• Prompt caching: As throughput increases, cache hit rates tend to improve because more requests share common prefixes. A system prompt cache that serves 60% of requests at 1,000 RPM might serve 85% at 10,000 RPM (because the cache stays warm with higher traffic). This means input token costs grow slower than throughput — more requests, but a higher fraction use cached (90% cheaper) tokens.
• Semantic caching: Caching complete responses for identical or semantically similar queries can eliminate LLM calls entirely. At low throughput, few queries repeat. At high throughput, the probability of cache hits increases because the query distribution follows a power law — a small number of popular queries account for a large share of traffic. A semantic cache with a 15% hit rate at 1,000 RPM might achieve 25% at 10,000 RPM, effectively making 25% of your throughput free.
• Batch API pricing: OpenAI's Batch API and Anthropic's Message Batches API offer 50% discounts for workloads that can tolerate up to 24 hours of latency. At high throughput, more workloads become batch-eligible because the pipeline processes enough volume to fill batches efficiently. A team processing 100,000 documents per day might batch 60% of them at half price, while a team processing 1,000 documents per day cannot fill batches efficiently.

Super-linear scaling risks:

Conversely, some patterns cause costs to scale faster than throughput — a dangerous dynamic that leads to budget blowouts:

• Retry amplification: At low throughput, you are well below rate limits and retries are rare. As throughput approaches rate limit thresholds, rate limit errors increase, triggering retries that further increase throughput, which triggers more rate limits — a positive feedback loop. If 5% of requests at 80% rate limit utilization trigger retries, that might jump to 15% at 95% utilization, adding 10% more effective token cost.
• Context accumulation: In conversational applications, higher throughput often means more active conversations, which means longer conversation histories being sent as context. If average context length grows from 2,000 tokens at 1,000 active conversations to 5,000 tokens at 5,000 active conversations, your input token cost per request has more than doubled even if request volume only increased 5x.
• Infrastructure overhead: At high throughput, you may need to add proxy layers, load balancers, logging pipelines, and monitoring infrastructure that add fixed costs on top of per-token API costs. These infrastructure costs create a per-request overhead that does not exist at lower volumes.

CostHawk's throughput-cost correlation dashboards show you whether your costs are scaling linearly, sub-linearly, or super-linearly with throughput — giving you early warning if your scaling economics are deteriorating.

Every LLM API provider imposes throughput limits (rate limits) that constrain how fast you can process requests. These limits vary by provider, model, pricing tier, and account type. Hitting these limits is one of the most common causes of production incidents in AI applications. Here are the current rate limit structures for major providers as of March 2026:

Understanding rate limit mechanics:

Rate limits are enforced per minute (rolling window) and typically apply per model and per API key. If your rate limit is 5,000 RPM for GPT-4o, you can send at most 5,000 requests to GPT-4o in any 60-second window. Requests to GPT-4o mini have their own separate limit. Some providers use a sliding window (track requests over a rolling 60-second period), while others use fixed 1-minute windows.

When you exceed a rate limit, the provider returns an HTTP 429 (Too Many Requests) response with a Retry-After header indicating how long to wait. This response still consumes a network round-trip but does not consume tokens or incur billing — you are only charged for successfully processed requests.

The real-world impact of rate limits:

Rate limits interact with throughput in ways that create production problems:

• Headroom erosion: If your normal throughput is 3,500 RPM on a 5,000 RPM limit, you have 30% headroom for traffic spikes. A product launch, marketing event, or viral moment that pushes traffic up 40% will breach the limit, causing request failures and degraded user experience.
• Burst penalties: Even if your average throughput is well below the limit, bursts can exceed it momentarily. A batch job that sends 2,000 requests in 10 seconds uses 12,000 RPM instantaneous throughput, which will trigger rate limits even if the average over the full minute is under 5,000.
• Multi-key strategies: Some teams distribute requests across multiple API keys to multiply their effective rate limit. This works but adds complexity and may violate provider terms of service if used to circumvent intentional limits. CostHawk tracks throughput per key so you can monitor the load distribution across keys.

Throughput optimization is about getting more useful work done per unit of time and cost. The strategies fall into two categories: increasing raw capacity (more requests per minute) and increasing efficiency (more value per request).

1. Request batching (maximize tokens per request)

Instead of sending one document per API call, batch multiple items into a single request. If you need to classify 100 emails, you can send them in batches of 10 with a prompt like "Classify each of the following 10 emails." This reduces request overhead (fewer HTTP round-trips, fewer per-message token overheads for system prompts) and stays further from RPM rate limits. The tradeoff is that a single failed batch request loses all 10 results, and latency for the first result increases because you wait for the entire batch. Practical batch sizes range from 5-20 items depending on the total token count — stay well within the model's context window and output token limits.

2. Asynchronous processing (decouple throughput from latency)

For workloads that do not require real-time responses, decouple the user-facing request from the LLM call using a message queue (Redis, SQS, RabbitMQ). The user submits a request, receives an immediate acknowledgment, and the LLM call is processed asynchronously. This allows you to smooth out traffic spikes (queue requests during bursts, process them at a steady rate), process requests during off-peak hours when rate limits have more headroom, and use batch APIs (50% discount) for latency-tolerant workloads. CostHawk's time-of-day throughput analysis helps you identify optimal processing windows.

3. Provider diversification (multiply capacity)

Using multiple LLM providers effectively multiplies your throughput capacity. If you have 5,000 RPM on OpenAI and 4,000 RPM on Anthropic, your combined capacity is 9,000 RPM. Route requests to the provider with the most available capacity, failing over to the secondary when the primary is congested. This also provides resilience — if one provider has an outage, traffic shifts to the other. The complexity cost is maintaining compatible prompts and handling different response formats across providers. Tools like Portkey and LiteLLM provide unified APIs that abstract provider differences.

4. Caching (eliminate redundant throughput)

Every cached response is a request you did not need to make, effectively increasing your throughput capacity by the cache hit rate. Implement caching at two levels: exact-match caching for identical queries (simple hash-based lookup, suitable for FAQ bots and search auto-suggest), and semantic caching for similar queries (embedding-based similarity search, suitable for customer support and knowledge bases where many users ask the same question in different words). A 20% cache hit rate on a 5,000 RPM workload eliminates 1,000 RPM of LLM calls — equivalent to adding 1,000 RPM of rate limit capacity at zero marginal cost.

5. Prompt optimization (reduce tokens per request)

Shorter prompts consume fewer tokens per request, which means your TPM rate limit accommodates more requests. If you reduce average input tokens from 2,000 to 1,200, the same 2,000,000 TPM limit now supports 1,667 RPM instead of 1,000 RPM — a 67% throughput increase with no infrastructure changes. Prompt optimization compounds with all other throughput strategies: every request that passes through your system benefits from shorter prompts, whether it is a real-time user request, a batched background job, or a retry.

6. Model routing for throughput (use different limits)

Different models have different rate limits. If you are hitting the GPT-4o rate limit at 5,000 RPM but GPT-4o mini allows 10,000 RPM, routing simple tasks to the cheaper model not only saves money but also frees up GPT-4o capacity for complex tasks. This is throughput optimization through demand shaping — moving workloads to where capacity is available rather than demanding more capacity where you are constrained.

Monitoring throughput is essential for maintaining service quality, controlling costs, and planning capacity. Here is a comprehensive monitoring framework for LLM throughput:

Real-time metrics to track:

• Current RPS/RPM: The instantaneous request rate, smoothed over a 1-minute window. This is your primary capacity utilization metric. Display it alongside your rate limit to show utilization percentage (e.g., 3,200/5,000 RPM = 64% utilization).
• Current TPS/TPM: Token throughput, split by input and output. Since output tokens cost more, a spike in output TPS has a bigger cost impact than a spike in input TPS of the same magnitude.
• Rate limit utilization: Current throughput as a percentage of your rate limit. Alert when utilization exceeds 80% to give yourself headroom for traffic spikes. CostHawk calculates this automatically based on your provider tier and current throughput.
• Request queue depth: If you use asynchronous processing, the number of pending requests in the queue. A growing queue indicates throughput demand exceeding processing capacity — you need to increase concurrency, add provider capacity, or optimize token efficiency.
• Error rate by type: Break down errors into rate limit errors (429), server errors (500/503), timeout errors, and content filter errors. Each type has a different throughput implication: rate limit errors directly indicate throughput saturation, server errors indicate provider issues, and timeouts indicate latency problems that reduce effective throughput.

Trend analysis metrics:

• Peak throughput trend: Track your daily peak RPM over the past 30, 60, and 90 days. If peak throughput is growing 15% month-over-month, you will hit your rate limit in N months — calculate when and plan capacity increases before you get there.
• Throughput efficiency: Track the ratio of effective throughput (requests that produced useful output) to raw throughput (all requests including retries and errors). If this ratio is declining, an increasing share of your throughput — and cost — is waste.
• Tokens per request trend: Average input and output tokens per request over time. An upward trend means each request consumes more throughput capacity, effectively reducing the number of requests you can process per minute even if your rate limit has not changed.
• Cost per token trend: If you use multiple models, your blended cost per token changes as traffic shifts between models. Track this to detect unintentional model routing changes that increase effective cost.

Alerting rules:

• Rate limit proximity: Alert when RPM or TPM exceeds 80% of the rate limit for any model. This gives you a 20% buffer to respond before users are affected.
• Rate limit errors: Alert immediately when any rate limit error (HTTP 429) is returned. Even a single 429 in production indicates you are at the boundary and more will follow.
• Throughput anomaly: Alert when current RPM exceeds 2x the trailing 7-day average for the current hour. This catches both legitimate traffic spikes (that need capacity planning) and technical issues (retry storms, runaway batch jobs).
• Cost rate spike: Alert when the cost rate ($/hour) exceeds the budgeted rate by more than 30%. This is the ultimate throughput alert — it catches any issue (more requests, bigger requests, more expensive models, more retries) that increases spend velocity.

CostHawk provides all of these metrics and alerts out of the box. Its real-time dashboard shows current throughput, rate limit utilization, cost rate, and error breakdown in a single view. Anomaly detection analyzes throughput patterns using statistical models trained on your historical data, so alerts are calibrated to your specific traffic patterns rather than generic thresholds.