Large Language Model (LLM)

Understanding how LLMs generate text explains why inference costs money and why output tokens cost more than input tokens.

The process has two phases:

1. Prefill (processing input): The model reads your entire input — system prompt, user message, conversation history, tool definitions — and computes internal representations (called "activations") for each token. This happens in parallel on the GPU, making it relatively fast and computationally efficient. The model builds a key-value (KV) cache that stores contextual information about each input token for use during generation.

2. Autoregressive decoding (generating output): The model generates output one token at a time. For each new token, it:

1. Reads the KV cache from all previous tokens (both input and previously generated output)
2. Computes a probability distribution over the entire vocabulary
3. Samples or selects the next token based on a temperature setting
4. Appends the new token to the KV cache
5. Repeats until a stop condition is met (end-of-sequence token, max_tokens limit, or stop sequence)

This sequential generation process is why output tokens cost 3–5x more than input tokens: each output token requires its own forward pass through the model, and the KV cache grows linearly with sequence length, consuming increasingly more GPU memory.

From a cost perspective, this means:

• Longer inputs increase cost linearly (more tokens to process in the prefill phase)
• Longer outputs increase cost linearly and at a higher per-token rate
• The total cost of a request is: (input_tokens × input_rate) + (output_tokens × output_rate)

Streaming responses (receiving tokens as they are generated) does not change the total cost — it only changes the delivery pattern. You pay the same whether you receive all tokens at once or one at a time.

The LLM landscape in 2026 organizes into distinct tiers based on capability and cost. Understanding these tiers is essential for model routing and cost optimization:

Key observations for cost management:

• Economy models handle 60–80% of typical production workloads. If you are routing everything to a mid-tier model, you are likely overspending on simple tasks.
• Reasoning models have hidden costs. Models like o1 and o3-mini consume "thinking tokens" that count toward your bill but don't appear in the output. A request that generates 500 visible output tokens might consume 5,000+ thinking tokens internally.
• Pricing changes frequently. OpenAI has cut GPT-4o pricing twice since launch. Anthropic adjusts rates with new model releases. CostHawk tracks the latest pricing across all providers so your cost reports are always accurate.
• Batch APIs offer 50% discounts. Both OpenAI and Anthropic offer batch processing at half the real-time rate, suitable for workloads that can tolerate hours of latency.

The cost of an LLM has two fundamentally different components, and which one matters to you depends on your role:

Training cost is the one-time expense of training the model on massive datasets. GPT-4 is estimated to have cost $100+ million to train. Llama 3 405B reportedly cost Meta tens of millions in compute alone. Training costs are borne by the model provider (OpenAI, Anthropic, Google, Meta) and amortized across all customers.

Inference cost is the ongoing expense of running the trained model to process your requests. This is what you pay for as an API customer — every token in, every token out, metered and billed. For API consumers, inference cost is the only cost that matters.

The economics work like this: a provider spends $100 million training a model, then serves billions of requests at $2.50–$15.00 per million tokens. The training cost is fixed and sunk; the inference cost scales linearly with usage. Providers aim to recoup training costs and earn margins through inference pricing.

For your team, this means:

• You cannot control training costs. These are embedded in the provider's per-token pricing.
• You can control inference costs. Every optimization — shorter prompts, output capping, model routing, caching — directly reduces your inference bill.
• Open-source models shift the equation. With models like Llama 3 and Mistral, you can self-host and pay only for the GPU compute, eliminating the provider's margin. But self-hosting introduces operational costs (GPU provisioning, model serving infrastructure, scaling, monitoring) that can exceed API costs for small-to-medium workloads.

The crossover point where self-hosting becomes cheaper than API access typically occurs around $10,000–$30,000/month in API spend, depending on the model and your team's infrastructure capabilities. CostHawk helps you track your spend trajectory so you can evaluate this decision with real data.

The choice between open-source (self-hosted) and API-hosted models is one of the most consequential infrastructure decisions for AI-powered applications. Each approach has distinct cost characteristics:

API-hosted models (OpenAI, Anthropic, Google):

• Zero infrastructure management — the provider handles GPUs, scaling, uptime
• Pay-per-token pricing with no minimum commitment (for most tiers)
• Access to the most capable frontier models that are not available as open-source
• Costs scale linearly with usage — great at low volume, expensive at high volume
• Provider lock-in risk if you build against model-specific features

Open-source models (Llama 3, Mistral, DeepSeek, Qwen):

• No per-token fees — you pay for GPU compute (typically $1–$4/hour per GPU)
• Models lag frontier capabilities by 6–12 months but are improving rapidly
• Full control over model weights, fine-tuning, and deployment configuration
• Fixed infrastructure costs regardless of token volume — great at high volume
• Requires MLOps expertise for deployment, scaling, and monitoring

Cost comparison example: Running Llama 3 70B on 4x A100 GPUs costs approximately $6/hour ($4,320/month). At the throughput these GPUs provide (~500 tokens/second), you can process roughly 1.3 billion tokens per month. The equivalent volume on GPT-4o would cost: 1.3B tokens × $2.50/MTok input = $3,250/month for input alone, plus output costs. For this specific scenario, self-hosting is competitive. But add in engineering time for deployment, monitoring, and maintenance, and the picture changes.

Hybrid approach: Many teams use a hybrid strategy — self-host an economy model for high-volume, simple tasks, and use API-hosted frontier models for complex tasks. CostHawk can monitor both self-hosted and API costs in a unified dashboard, helping you optimize the split between the two approaches.

Model selection is the single highest-leverage decision for AI cost optimization. Here is a practical framework for choosing the right model for each use case:

Step 1: Categorize your requests. Most AI workloads fall into a few task categories, each with different capability requirements:

• Classification and extraction (sentiment, entity recognition, structured data extraction): Economy models handle these with 95%+ accuracy.
• Simple generation (FAQ responses, template filling, translation): Economy models perform well; mid-tier if quality requirements are high.
• Complex reasoning (multi-step analysis, code generation, long-form writing): Mid-tier models are the sweet spot; frontier for critical tasks.
• Novel problem-solving (research, mathematical proofs, adversarial scenarios): Frontier or reasoning models required.

Step 2: Benchmark quality at each tier. For each task category, test representative prompts against models at each pricing tier. Measure output quality using your application's specific criteria (accuracy, completeness, formatting compliance, etc.). You will often find that the quality difference between tiers is smaller than expected for structured tasks.

Step 3: Calculate the cost at your volume. For each task category, estimate daily request volume, average input tokens, and average output tokens. Multiply by the per-token rate for each candidate model to get daily and monthly costs. This often reveals that moving a high-volume, simple task from a mid-tier to an economy model saves more money than optimizing prompts for a low-volume, complex task.

Step 4: Implement model routing. Configure your application to route each request to the appropriate model based on task category. Start with static routing (hardcoded model per endpoint) and evolve toward dynamic routing (a lightweight classifier that selects the model per request) as you gather more data.

Step 5: Monitor and iterate. Use CostHawk to track cost-per-query and quality metrics by model over time. As providers release new models and adjust pricing, re-evaluate your routing decisions. A model that was the best value six months ago may no longer be optimal.

Monitoring LLM costs requires a different approach than traditional infrastructure monitoring. Unlike CPU or memory, where utilization is the primary metric, LLM costs are driven by token volume, model selection, and request patterns. Here is what effective LLM cost monitoring looks like:

Per-model cost tracking: Break down your total AI spend by model to see which models are consuming the most budget. This reveals model routing inefficiencies — for example, if 70% of your spend goes to GPT-4o but analysis shows that 50% of those requests could be handled by GPT-4o mini, you have identified a clear optimization opportunity worth 50% of that 70% slice.

Cost-per-query analysis: Track the average cost per request by endpoint, feature, or use case. A search endpoint averaging $0.002/query is very different from an analysis endpoint averaging $0.15/query. Understanding these unit economics is essential for pricing your own product, forecasting costs as usage grows, and identifying optimization targets.

Token efficiency metrics: Monitor the ratio of useful output tokens to total tokens consumed. If your application generates 500 tokens of output but the user only reads or uses 100 tokens (for example, a chatbot where the user asks for a yes/no answer and gets a paragraph), you are paying for wasted output. Track output utilization to identify opportunities for more concise generation.

Provider-level aggregation: If you use multiple providers (many teams use OpenAI for some tasks and Anthropic for others), aggregate costs across providers for a unified view. CostHawk normalizes pricing across all major providers so you can compare apples-to-apples.

Anomaly detection: LLM costs can spike suddenly due to retry storms, leaked keys, context window bloat, or new feature launches. Set up anomaly detection that alerts when daily or hourly spend deviates significantly from the baseline. CostHawk's anomaly detection uses statistical methods to flag unusual patterns before they become expensive surprises.

Forecasting: Use historical token consumption trends to forecast future costs. If your token usage is growing 15% month-over-month, project out 3–6 months to understand your future budget requirements. This enables proactive conversations with finance about AI infrastructure budgets rather than reactive explanations of unexpected bills.