Input vs. Output Tokens

When you send a prompt to an LLM, the model processes your input tokens in parallel using a single forward pass through the neural network. This is computationally efficient — the model reads all input tokens at once and builds an internal representation. Output generation is fundamentally different. The model produces tokens one at a time in a sequential, autoregressive process: each new token depends on every token that came before it. This means generating 1,000 output tokens requires roughly 1,000 sequential forward passes, each one slightly larger than the last.

This computational asymmetry is why providers charge more for output. The GPU time required to generate a token is significantly higher than the GPU time required to process an input token. On most transformer architectures, the ratio of compute per output token to compute per input token ranges from 3x to 6x, depending on model size, batch efficiency, and hardware utilization. Providers pass this cost differential directly to customers through split pricing.

The asymmetry also has implications for capacity planning. Output-heavy workloads consume more GPU-seconds per request, which means providers can serve fewer concurrent requests. This reduced throughput is another factor in the higher output price.

The following table shows current input and output token pricing for major models as of March 2026. All prices are per 1 million tokens.

Notice that every provider maintains a consistent multiplier across their model lineup: OpenAI and Google use 4x, Anthropic uses 5x. This consistency means the optimization math is the same regardless of which model tier you use — reducing output tokens is always 4-5x more valuable than reducing input tokens by the same amount.

The cost difference comes down to three factors rooted in how transformer models work:

• Autoregressive generation: Each output token requires a separate forward pass through the model. A 500-token response requires 500 sequential inference steps, while a 500-token input is processed in a single parallel pass. The sequential nature of output generation prevents the kind of batching optimizations that make input processing efficient.
• KV-cache memory: During output generation, the model maintains a key-value cache that grows with each token produced. This cache consumes GPU memory proportional to the sequence length, reducing the number of concurrent requests a GPU can handle. More memory per request means fewer requests per GPU, which increases the cost per token.
• Speculative decoding overhead: Many providers use speculative decoding to accelerate output generation, where a smaller draft model proposes tokens that the main model verifies. While this improves throughput, it adds computational overhead per output token that does not exist for input processing.

These three factors combine to make output generation 3-6x more expensive in raw compute cost per token. Providers set their pricing multipliers at 4-5x to reflect this reality while maintaining some margin.

Since output tokens cost 4-5x more than input tokens, reducing output length has an outsized impact on your bill. Here are proven strategies:

• Set explicit max_tokens: Always set a max_tokens limit appropriate to your use case. A classification task needs 10-50 tokens, not the default 4,096. Setting max_tokens: 50 on a classification endpoint prevents the model from generating verbose explanations you do not need.
• Request structured output: Use JSON mode or structured output schemas to force concise, machine-readable responses. A JSON object with three fields is typically 50-100 tokens. An equivalent natural-language paragraph is 200-400 tokens. That is a 2-4x reduction in output tokens.
• Use response formatting instructions: Explicitly tell the model how to format its response. "Respond with only the category name" produces 2-5 tokens. "Explain your reasoning and then provide the category" produces 200+ tokens. Be precise about what you want.
• Avoid chain-of-thought when unnecessary: Chain-of-thought prompting ("think step by step") dramatically increases output length — often by 5-10x. Use it for complex reasoning tasks where accuracy matters, but skip it for straightforward extraction and classification tasks.
• Post-process with smaller models: If you need a long response for internal processing but a short summary for the user, generate the full response with a capable model and then summarize it with a cheaper model like GPT-4o-mini.

Applying these strategies in combination can reduce output token spend by 40-70% without degrading the quality of results your application delivers to users.

Your input-to-output ratio is a critical metric for cost forecasting and optimization prioritization. Calculate it as: total_input_tokens / total_output_tokens. A ratio of 10:1 means you send 10 input tokens for every 1 output token — common for classification and extraction tasks. A ratio of 1:3 means the model generates 3 tokens for every 1 you send — typical for creative writing and code generation.

Typical ratios by use case:

Use cases with a low input-to-output ratio (meaning output-heavy) benefit most from output optimization strategies. Use cases with a high ratio (input-heavy) benefit more from prompt compression and caching strategies. CostHawk calculates this ratio automatically on the usage dashboard so you can prioritize optimizations by impact.

CostHawk provides granular visibility into input and output token consumption across every request, model, and project. The dashboard breaks down your spend into input cost and output cost columns, letting you instantly see which direction is driving your bill.

Key CostHawk features for token direction analysis:

• Per-request breakdown: Every logged API call shows input token count, output token count, input cost, and output cost as separate fields. Sort by output cost to find your most expensive responses.
• Aggregate ratio tracking: The usage analytics page displays your organization-wide input-to-output ratio over time. A trending shift toward more output-heavy usage signals growing costs that may need attention.
• Model-level comparison: Compare input/output splits across models to identify routing opportunities. If your GPT-4o requests are 80% output cost, routing some to GPT-4o-mini could save 90% on those specific calls.
• Anomaly detection on output spikes: CostHawk flags requests where output token counts are unusually high — a sign of verbose responses, infinite-loop agent behavior, or missing max_tokens constraints.

By treating input and output tokens as separate cost centers, CostHawk helps teams apply the right optimization strategy to the right side of their token spend.