Pay-Per-Token

Pay-per-token pricing is the standard billing mechanism for large language model APIs. Every API call you make is metered by counting the tokens in your input (the prompt, system message, conversation history, and any tool definitions) and the tokens in the output (the model's generated response). The provider multiplies each count by the respective per-token rate for the model you called, sums the two, and adds the result to your running bill.

This model has three defining characteristics:

• Zero fixed cost: If you make zero API calls in a month, you pay nothing. There is no base fee, platform fee, or minimum spend (though some enterprise contracts add these).
• Linear scaling: Your cost scales linearly with usage. Twice the tokens means twice the cost. There are no volume discounts in the standard pay-per-token model (volume discounts require separate committed-use agreements).
• Separate input/output rates: Output tokens are more expensive because autoregressive generation is more compute-intensive than encoding input. The ratio varies by provider and model — GPT-4o charges $2.50/$10.00 (4x), Claude 3.5 Sonnet charges $3.00/$15.00 (5x), and Gemini 1.5 Pro charges $1.25/$5.00 (4x).

This pricing model dominates because it aligns provider revenue with compute consumption. Providers pay for GPU-seconds, and token counts are a reasonable proxy for GPU time used. For customers, the appeal is low barriers to entry — you can experiment with a few dollars and scale up gradually.

Understanding the mechanics of pay-per-token billing is essential for accurate forecasting. Here is the formula:

Request Cost = (Input Tokens × Input Price per Token) + (Output Tokens × Output Price per Token)

Providers quote prices per million tokens (MTok) for readability, but the actual billing unit is the individual token. Let's walk through a concrete calculation for a customer support chatbot using GPT-4o:

At 50,000 conversations per day with an average of 6 turns each, that is 300,000 API calls per day. Daily cost: 300,000 × $0.01188 = $3,564. Monthly cost: $106,920. Notice that the conversation history grows with each turn — turn 6 sends all previous turns as input, making later turns significantly more expensive than earlier ones. A 10-turn conversation costs roughly 3x more in total than a 3-turn conversation because of this compounding input cost.

Key nuances that affect your bill:

• Cached tokens: Anthropic offers a 90% discount on prompt caching hits. OpenAI offers 50% off cached input tokens. If your system prompt is cached, the 800-token system prompt above drops from $0.002 to $0.0002 (Anthropic) or $0.001 (OpenAI).
• Minimum billing: Some providers round up to minimum token counts. Always check your actual billed tokens against your expected tokens.
• Tool/function definitions: Tool schemas sent to the model count as input tokens. A complex tool schema can add 500–2,000 tokens to every request.

The AI industry now offers two distinct pricing paradigms. Pay-per-token (API billing) charges for exact usage, while flat-rate subscriptions (Claude Max at $100–$200/mo, ChatGPT Pro at $200/mo, ChatGPT Plus at $20/mo) charge a fixed monthly fee for a usage allowance. Each model has clear winners and losers depending on your situation.

The breakeven point varies by model. For Claude 3.5 Sonnet, a developer making roughly 2,500 complex requests per day (averaging 4,000 input + 1,000 output tokens each) would spend approximately $200/month on the API — making Claude Max's $200/month plan break even. Below that volume, pay-per-token is cheaper. Above it, the flat rate wins on cost but loses on flexibility. For production applications serving end users, pay-per-token is the only viable option because flat-rate subscriptions are designed for individual human use and their terms of service prohibit programmatic access at scale.

The biggest drawback of pay-per-token pricing is cost unpredictability. Unlike a $200/month subscription where the bill is known before the month starts, pay-per-token costs are determined by factors that change daily — sometimes hourly. Here are the primary sources of variance:

• Traffic spikes: A viral feature, marketing campaign, or seasonal event can 5–10x your daily API call volume overnight. If your baseline is $500/day and traffic triples, you are suddenly spending $1,500/day with no automatic circuit breaker.
• Prompt drift: Development teams frequently add context, examples, and instructions to prompts without realizing the cost impact. A system prompt that starts at 500 tokens in v1 often grows to 3,000 tokens by v10. Across 100,000 daily requests at $3/MTok input, that 2,500-token increase costs an extra $750/day ($22,500/month).
• Agent loops: Agentic workflows (tool-use, chain-of-thought, multi-step reasoning) make multiple API calls to complete a single user request. A ReAct agent might make 5–15 API calls per task. If each call averages 3,000 tokens in and 800 tokens out using Claude 3.5 Sonnet, a single agent run costs $0.045–$0.135. At 10,000 agent runs per day, that is $450–$1,350 daily — and the variance between 5-call and 15-call runs makes forecasting extremely difficult.
• Retry storms: When a downstream service is degraded, retry logic can amplify costs. A 30-minute outage with aggressive retries (no exponential backoff) can generate 10x normal traffic and 10x normal cost during that window.
• Model upgrades: Switching from GPT-4o-mini ($0.15/$0.60 per MTok) to GPT-4o ($2.50/$10.00 per MTok) increases costs by 16x on the same traffic. Teams sometimes upgrade models in development and forget to benchmark the cost impact before deploying to production.

CostHawk addresses these challenges through real-time cost tracking, anomaly detection alerts, and per-key budget controls that can automatically disable keys when spending exceeds thresholds.

Pay-per-token is not always the most cost-effective option. Here is a framework for evaluating when to switch to alternatives:

Switch to Provisioned Throughput when: Your monthly API spend exceeds $10,000 and your traffic is predictable (less than 2x variance between peak and trough). Anthropic, OpenAI, and AWS Bedrock offer reserved capacity at 30–50% discounts over pay-per-token rates. The breakeven for Anthropic's reserved capacity is roughly $8,000/month — below that, pay-per-token is cheaper because you would be paying for unused capacity.

Switch to Batch API when: Your workload is latency-tolerant (results needed in hours, not seconds). OpenAI's Batch API offers a 50% discount on GPT-4o ($1.25/$5.00 per MTok instead of $2.50/$10.00). Anthropic's Message Batches API offers similar discounts. If even 30% of your traffic can tolerate batch processing (e.g., nightly content generation, bulk classification, data extraction), you save 15% on your total bill by routing those requests to batch endpoints.

Switch to self-hosted models when: Your monthly spend exceeds $50,000 and your use case is well-served by open-source models (Llama 3, Mixtral, Qwen). Running Llama 3 70B on 4x A100 GPUs costs approximately $8,000–$12,000/month in cloud compute, which serves roughly the same throughput as $40,000–$60,000/month in API costs for a comparable model. The breakeven is typically 3–6 months including setup costs.

Stay on pay-per-token when: Your spend is under $5,000/month, your traffic is spiky and unpredictable, you need access to frontier models (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro) that cannot be self-hosted, or you are in an early experimentation phase where usage patterns have not stabilized.

Effective cost management under pay-per-token pricing requires a layered approach combining monitoring, optimization, and guardrails:

1. Real-time visibility: You cannot optimize what you cannot see. Instrument every API call with cost tracking that captures model, token counts, latency, and request metadata (user ID, feature, environment). CostHawk provides this out of the box with a single SDK integration or proxy configuration.

2. Budget controls: Set hard and soft limits at every level — organization, project, API key, and individual user. A hard limit stops requests when the budget is exhausted. A soft limit sends alerts but allows traffic to continue. CostHawk supports both, with configurable alert thresholds at 50%, 75%, 90%, and 100% of budget.

3. Model routing: Not every request needs your most expensive model. Route simple tasks (classification, extraction, reformatting) to cheaper models like GPT-4o-mini ($0.15/$0.60 per MTok) or Claude 3.5 Haiku ($0.80/$4.00 per MTok) and reserve premium models for complex reasoning tasks. Teams that implement intelligent model routing typically reduce costs by 40–60% with minimal quality impact.

4. Prompt optimization: Audit your prompts quarterly. Remove redundant instructions, compress few-shot examples, and use structured output (JSON mode) to reduce output verbosity. A prompt optimization pass typically yields 20–40% token reduction.

5. Caching: Cache responses for identical or near-identical requests. Semantic caching (matching on intent rather than exact string) can achieve 15–30% cache hit rates for customer support and FAQ use cases, eliminating those API calls entirely. Additionally, use provider-level prompt caching (Anthropic's prompt caching, OpenAI's automatic caching) to reduce input token costs by 50–90% on repeated system prompts and context.

6. Output length control: Set max_tokens appropriate to each use case. A classification task needs 10–50 output tokens, not the default 4,096. Reducing max_tokens does not guarantee shorter responses, but it prevents runaway generation that wastes tokens on content you will truncate anyway.