LLM Gateway

An LLM gateway is a reverse proxy purpose-built for AI API traffic. It accepts incoming LLM requests from your application, applies AI-specific policies (model routing, cost tracking, caching, rate limiting), forwards the request to the selected AI provider, and returns the response to your application. The gateway maintains connections to multiple AI providers simultaneously and can switch between them transparently based on configurable rules.

The defining characteristic of an LLM gateway, versus a traditional API gateway with AI plugins bolted on, is that AI is its primary design concern rather than an afterthought. Every feature is optimized for the specific patterns of LLM API consumption:

• Token-level awareness: The gateway parses every request to count (or estimate) input tokens and parses every response to extract actual input and output token counts from the usage object. This enables token-based rate limiting, per-request cost calculation, and usage analytics that are impossible with a token-unaware gateway.
• Model routing intelligence: The gateway maintains a registry of available models across all connected providers, including their capabilities (context window size, function calling support, vision support), pricing (input and output rates), and current availability. Routing decisions can factor in all of these dimensions — sending a request to the cheapest model that meets the capability requirements.
• Provider format translation: Each AI provider has its own request and response format. OpenAI uses {"messages": [{"role": "user", "content": "..."}]}. Anthropic uses {"messages": [{"role": "user", "content": "..."}], "system": "..."} with the system message at the top level. Google uses a completely different structure. The gateway translates between these formats automatically, so your application can use a single format (typically OpenAI-compatible) regardless of which provider serves the request.
• Streaming support: LLM responses are often streamed via Server-Sent Events (SSE) to provide a responsive user experience. The gateway must handle streaming correctly — forwarding chunks as they arrive without buffering, while still extracting token counts from the final chunk for cost tracking. This is a non-trivial engineering challenge that traditional gateways handle poorly or not at all.
• AI-specific retry logic: When an LLM API call fails, the appropriate retry behavior depends on the error type. A 429 (rate limit) should be retried with exponential backoff. A 500 (server error) should be retried, potentially at an alternative provider. A 400 (bad request) should not be retried — the request is malformed. An overloaded error from Anthropic should be retried with backoff or routed to an alternative model. The gateway implements these AI-specific retry semantics rather than applying generic HTTP retry logic.

The result is an infrastructure component that makes multi-provider AI simple, reliable, and cost-efficient — handling the complexity that would otherwise be scattered across every service in your architecture.

While both LLM gateways and traditional API gateways serve as reverse proxies, their feature sets diverge significantly when handling AI workloads. This comparison highlights why teams building serious AI applications eventually adopt a dedicated LLM gateway rather than stretching a general-purpose gateway to fit.

The practical impact of these differences compounds as AI usage grows. A traditional gateway logging an LLM request records: POST /v1/chat/completions 200 1247ms. An LLM gateway records: POST /v1/chat/completions → claude-3.5-sonnet via anthropic | 2,847 input + 612 output tokens | $0.0177 | cache miss | 1,247ms TTFT | streaming 73 chunks. The LLM gateway's log entry enables cost attribution, usage analytics, and optimization analysis. The traditional gateway's log entry tells you almost nothing actionable about AI economics.

Many teams attempt to bridge this gap by adding custom middleware to their existing API gateway. This works for basic cost logging but quickly becomes unsustainable as requirements grow. Token counting, format translation, semantic caching, model routing, and AI-specific retry logic each add hundreds of lines of complex, provider-specific code that must be maintained as providers update their APIs. At that point, adopting a purpose-built LLM gateway is less engineering effort than maintaining the growing pile of custom middleware.

Five features define a production-grade LLM gateway and distinguish it from both traditional API gateways and simple API proxies:

1. Model Routing

Model routing is the ability to direct requests to different models and providers based on configurable rules. Routing strategies include:

• Cost-based: Route to the cheapest model that meets the request's requirements. A classification task does not need GPT-4o ($2.50/$10.00 per MTok) — GPT-4o mini ($0.15/$0.60) handles it at 16x lower cost. The gateway maintains a model capability registry and matches requests to the most cost-efficient option.
• Latency-based: Route to the fastest responding provider, based on real-time latency measurements. If Anthropic's P95 latency is currently 800ms while OpenAI's is 2,400ms (due to load), route to Anthropic.
• Capability-based: Route based on model capabilities. Requests requiring 128K context go to models that support it. Requests with images go to multimodal models. Requests requiring function calling go to models with tool support.
• A/B testing: Split traffic between models to compare quality and cost. Send 90% to production model, 10% to a candidate model, and compare results.

2. Response Caching

LLM gateways cache responses at two levels. Exact caching stores responses keyed on the full prompt content hash — if the same prompt arrives again, the cached response is returned without making an API call. Semantic caching uses embedding similarity to match semantically equivalent prompts — "What is the capital of France?" and "Tell me France's capital city" might return the same cached response. Exact caching is simple and safe (identical input always produces the same output). Semantic caching is more aggressive and achieves higher hit rates but requires tuning the similarity threshold to avoid returning irrelevant cached responses. Cache hit rates vary by workload: FAQ chatbots achieve 25–40%, classification tasks 15–30%, creative generation 2–5%.

3. Fallback and Failover

When the primary provider returns errors (500s, 503s, timeouts), the gateway automatically routes to a configured fallback. Fallback chains can be ordered by preference: try GPT-4o first, fall back to Claude 3.5 Sonnet, then fall back to Gemini 1.5 Pro. The gateway handles format translation automatically — the application receives a response in the same format regardless of which provider served it. Advanced fallbacks can also trigger on quality signals: if the primary model's response is truncated (max_tokens reached) or empty, retry with a model that has a larger output limit.

4. Cost Tracking and Analytics

The gateway computes per-request cost by multiplying token counts from the response's usage object by the model's current pricing rates. These costs are attributed to the requesting API key, project, team, or user and aggregated into time-series data that powers dashboards, alerts, and reports. The gateway maintains a pricing database that is updated when providers change their rates. This is the data that feeds into CostHawk's analytics — when traffic flows through an LLM gateway integrated with CostHawk, every dollar of AI spend is tracked, attributed, and available for analysis.

5. Intelligent Retries

LLM API calls fail for various reasons, and each requires different retry behavior. Rate limit errors (429) should be retried with exponential backoff, respecting the retry-after header. Server errors (500, 503) should be retried immediately at an alternative provider rather than waiting for the same failing server. Overloaded errors (Anthropic's 529) should be retried with longer backoff. Content filter errors should not be retried at all. Timeout errors might be retried with a shorter max_tokens to get a response within time limits. The gateway implements all of these strategies, reducing error rates from the application's perspective while avoiding wasteful retries that burn tokens without producing results.

The LLM gateway landscape has matured rapidly since 2024, with several production-ready options available:

Portkey

Portkey is a managed LLM gateway that emphasizes reliability and observability. It supports 200+ models across all major providers with a unified API. Key features include: automatic fallbacks with configurable retry logic, request-level caching, semantic caching via embedding similarity, load balancing across providers, and a built-in analytics dashboard showing cost, latency, and token usage. Portkey operates as a cloud service — you change your API base URL to Portkey's endpoint and requests are proxied through their infrastructure. Strengths: zero infrastructure to manage, extensive model support, strong reliability features. Weaknesses: adds a third-party dependency in your critical path, all prompts transit through Portkey's servers (potential compliance concern), managed pricing scales with volume. Best for: teams that want managed multi-provider routing and are comfortable with cloud-hosted proxies. Pricing: free tier (10K requests/month), paid plans from $49/month.

LiteLLM

LiteLLM is an open-source Python library and proxy server that provides an OpenAI-compatible API for 100+ LLM providers. You deploy LiteLLM as a self-hosted proxy and configure provider credentials, model aliases, and routing rules. Key features include: OpenAI-compatible API for all providers, model fallbacks, load balancing, spend tracking per key/team, rate limiting, and a basic admin dashboard. Strengths: open-source (MIT license), self-hosted (data stays in your infrastructure), extensive provider support, active development community. Weaknesses: requires self-hosting infrastructure and maintenance, Python-based (may not match your team's stack), less polished than managed alternatives, caching is basic (exact match only). Best for: teams that want full control over their gateway infrastructure and are comfortable managing a Python service. Pricing: free (open-source); hosted version available from $200/month.

Helicone

Helicone is primarily an LLM observability platform that functions as a lightweight proxy gateway. Route traffic through Helicone's endpoint to get detailed logging, cost tracking, caching, rate limiting, and analytics. Key features: request-level logging with prompt/response content, cost tracking across providers, response caching, user-level rate limiting, and a polished analytics dashboard. Strengths: excellent observability and debugging tools, easy setup (just change base URL), strong analytics for prompt engineering. Weaknesses: less sophisticated routing and failover compared to dedicated gateways, observability-first rather than infrastructure-first design, limited model routing capabilities. Best for: teams that prioritize observability and debugging over advanced routing and failover. Pricing: free tier (100K requests/month), paid from $80/month.

Custom Gateway (DIY)

Building a custom LLM gateway gives you complete control but requires significant investment. A minimal implementation needs: an HTTP server that accepts OpenAI-compatible requests, provider client libraries for each supported provider, format translation logic for non-OpenAI providers, token counting for rate limiting and cost tracking, a pricing database with current model rates, retry logic with provider-specific error handling, streaming support with SSE chunk forwarding, and a persistence layer for usage logging. In Node.js or Go, this is 2,000–5,000 lines of production code plus ongoing maintenance. Strengths: complete control, no third-party dependencies, can implement exact business logic needed. Weaknesses: 3–8 weeks to build initially, 8–16 hours/month ongoing maintenance, must handle every provider API change yourself, streaming and error handling edge cases are numerous. Best for: teams spending $50K+/month with unique requirements not served by off-the-shelf options and dedicated platform engineering capacity.

An LLM gateway is the ideal enforcement point for cost optimization strategies because it sees every request before it reaches the provider. Here are six gateway-level optimizations that can reduce AI spend by 30–70%:

1. Intelligent model routing (saves 40–60%). The single highest-impact optimization. Most applications send every request to their most capable (and expensive) model, even when 60–80% of requests could be handled by a model that costs 10–20x less. The gateway can classify incoming requests by complexity and route accordingly: simple classification and extraction tasks go to GPT-4o mini ($0.15/$0.60 per MTok), complex reasoning goes to GPT-4o ($2.50/$10.00), and long-context tasks go to Gemini 2.0 Flash ($0.10/$0.40). A team spending $20,000/month on GPT-4o that routes 65% of traffic to GPT-4o mini reduces its bill to approximately $8,000/month — a 60% reduction.

2. Response caching (saves 10–35%). The gateway caches responses for identical or semantically similar prompts. FAQ chatbots and classification workloads see the highest hit rates (25–40%) because many users ask the same questions. Even workloads with low cache hit rates benefit because each cache hit eliminates a full API call — a $0.02 request served from cache costs $0.00. At scale, even a 10% cache hit rate on 500,000 monthly requests saves 50,000 API calls and their associated cost.

3. Prompt caching coordination (saves 15–25% on input costs). Anthropic offers 90% off cached input tokens, and OpenAI offers 50% off. The gateway can optimize for cache hits by ensuring system prompts are identical across requests (no dynamic content that invalidates the cache), ordering few-shot examples consistently, and batching requests to maximize cache residency. A gateway that achieves 80% prompt cache hit rate on Anthropic reduces effective input cost from $3.00/MTok to $0.60/MTok for Claude 3.5 Sonnet.

4. Request deduplication (saves 2–8%). When multiple services or users submit identical requests within a short window, the gateway can coalesce them into a single API call and fan out the response to all waiting clients. This is especially effective for agentic workloads where multiple agents might independently request the same information. The gateway holds the second (and third, fourth...) identical request in a queue, forwards only the first to the provider, and returns the same response to all queued requests.

5. Output length enforcement (saves 5–15%). The gateway can set or override max_tokens based on the request type. A classification request that needs a single label should have max_tokens set to 50, not the default 4,096. A summarization request might need 500 tokens, not 2,000. By enforcing appropriate output limits at the gateway, you prevent the model from generating verbose responses that waste expensive output tokens. Output tokens cost 4–5x more than input tokens, so even modest output reduction has outsized cost impact.

6. Off-peak batch routing (saves 50% on eligible traffic). For latency-tolerant requests (content generation, data extraction, bulk classification), the gateway can queue them and submit via batch APIs (OpenAI's Batch API, Anthropic's Message Batches) at a 50% discount. The gateway handles the complexity of batch submission, status polling, and result retrieval, presenting a synchronous-looking API to the application while using asynchronous batch processing behind the scenes. Even routing 20% of traffic to batch APIs saves 10% on total spend.

CostHawk's analytics help you identify which of these optimizations will deliver the most savings for your specific workload. By analyzing your traffic patterns — model distribution, cache eligibility, output length distribution, and latency requirements — CostHawk recommends the optimizations with the highest ROI and quantifies the expected savings before you invest in implementation.

Not every AI application needs a dedicated LLM gateway. Here is a decision framework for determining when the investment is justified:

You DO need an LLM gateway if:

• You use multiple AI providers. If your application calls both OpenAI and Anthropic (or any combination of providers), a gateway dramatically simplifies credential management, format translation, and failover. Without a gateway, every service that calls AI APIs must maintain provider-specific client code for each provider.
• Your monthly AI spend exceeds $5,000. At this level, the cost optimization features of an LLM gateway (model routing, caching, batch routing) typically save $1,500–$3,000/month — far exceeding the cost of running the gateway.
• You need high availability for AI features. If AI powers a customer-facing feature (chatbot, search, content generation), a single provider outage should not take it down. Gateway-level failover ensures continuity by routing to backup providers automatically. Without a gateway, implementing failover requires application-level code changes that increase complexity and maintenance burden.
• You have multiple teams or services calling AI APIs. Centralized governance — rate limiting, cost tracking, access control, and audit logging — is much easier at the gateway level than in each service. Without a gateway, each team manages its own keys, tracks its own costs, and competes for rate limits without coordination.
• You need cost attribution and chargeback. If you need to know how much each team, feature, or customer costs in AI spend, the gateway is the natural instrumentation point. It tracks every request with attribution metadata, feeding the data into CostHawk for analytics and reporting.

You do NOT need an LLM gateway if:

• You use a single provider with a single model. If all your AI traffic goes to one endpoint (e.g., only GPT-4o via OpenAI), the gateway's routing and translation features provide no value. Use CostHawk's wrapped keys for cost tracking and a simple rate limiter in your application for throttling.
• Your monthly AI spend is under $1,000. The engineering investment in setting up and maintaining a gateway exceeds the savings it would generate. Use provider billing dashboards for cost visibility and CostHawk's usage sync for basic analytics.
• You are prototyping or experimenting. During early development when you are testing models and iterating on prompts, adding a gateway introduces unnecessary complexity. Call providers directly, collect data on which models and patterns work best, and add a gateway when you move to production.
• Your workload is simple and low-volume. A single-purpose application making 100 requests per day to one model does not benefit from a gateway. The overhead of deploying, configuring, and maintaining the gateway exceeds the value it provides.

For teams in the "maybe" zone ($1,000–$5,000/month, 2–3 services, one primary provider with occasional use of a second), CostHawk's wrapped key proxy provides a lightweight alternative that delivers cost tracking and basic routing without the full complexity of a dedicated LLM gateway. You can always upgrade to a full gateway later as your AI infrastructure matures — the data in CostHawk will help you make that decision by showing exactly where a gateway would save money.