Load Balancing

LLM load balancing is the practice of distributing API inference requests across multiple backends — where a "backend" can be a provider account, a specific model endpoint, a self-hosted deployment, or an entirely different provider — to improve throughput, reduce cost, lower latency, or increase resilience. The load balancer sits between your application and the LLM providers, making routing decisions for each incoming request based on a configurable strategy.

This concept is borrowed from traditional infrastructure load balancing (distributing web traffic across multiple servers), but LLM load balancing introduces unique challenges:

• Heterogeneous backends. In web load balancing, all servers return identical responses. In LLM load balancing, different models produce different quality outputs. Routing a request from GPT-4o to Claude 3.5 Sonnet might produce a superior response for creative writing but a worse one for code generation. The load balancer must respect model compatibility constraints.
• Variable pricing. Each backend has a different cost per token. The load balancer must track pricing to make cost-optimal routing decisions.
• Per-account rate limits. Unlike web servers (where throughput scales linearly with instance count), LLM API rate limits are per-account and often per-model. Adding a second OpenAI account doubles your GPT-4o RPM limit, but only if the load balancer correctly distributes requests across both accounts.
• Stateful conversations. Multi-turn chat applications may need session affinity — all turns in a conversation should go to the same model to maintain response consistency. The load balancer must track session state or accept that model switches mid-conversation may produce inconsistent tone or behavior.
• Streaming support. Most LLM responses are streamed via SSE. The load balancer must maintain the streaming connection for the full duration of the response, which can be several seconds for long generations.

Despite these challenges, load balancing is essential for any production LLM application processing more than a few hundred requests per minute. It is the foundational layer upon which higher-level optimizations — cost-aware routing, failover, and capacity planning — are built.

There are four primary load balancing strategies used for LLM traffic, each optimizing for a different objective. Most production deployments use a combination of two or more strategies.

1. Round-Robin. The simplest strategy: requests are distributed evenly across backends in a rotating sequence. Backend A gets request 1, backend B gets request 2, backend A gets request 3, and so on. Round-robin is easy to implement and ensures even distribution, but it does not account for backend-specific constraints like rate limits, pricing, or latency. It is best suited for distributing traffic across multiple accounts with the same provider and model, where all backends are functionally identical.

// Simple round-robin implementation
let currentIndex = 0
const backends = [accountA, accountB, accountC]

function getNextBackend() {
  const backend = backends[currentIndex]
  currentIndex = (currentIndex + 1) % backends.length
  return backend
}

2. Weighted distribution. Each backend is assigned a weight that determines its share of traffic. If backend A has weight 3 and backend B has weight 1, backend A receives 75% of requests. Weights can be set statically (based on account tier and rate limits) or adjusted dynamically based on real-time metrics. This strategy is ideal when backends have different capacities — for example, a paid OpenAI account (higher rate limits) and a free tier account (lower limits).

3. Cost-based routing. The load balancer routes each request to the cheapest available backend that can serve it. This requires the load balancer to maintain a pricing table for each model on each provider and to estimate the cost of each request (based on input token count and expected output length). Cost-based routing maximizes cost efficiency but may sacrifice latency if the cheapest provider is also the slowest, or quality if the cheapest model is less capable. Best practice is to combine cost-based routing with a minimum quality threshold: only route to cheaper models that have been validated for the specific task type.

4. Latency-based routing. The load balancer tracks the response time of each backend and routes requests to the fastest one. This is measured using a rolling average or P50/P95 of recent request latencies. Latency-based routing optimizes user experience but may increase costs if the fastest provider is also the most expensive. It is particularly valuable for real-time applications (chatbots, code completion) where response time directly impacts user satisfaction. Some implementations use a "two-choice" algorithm: for each request, pick two random backends and route to the one with lower recent latency. This provides near-optimal latency distribution with O(1) decision time.

In practice, the most effective strategy is a composite approach: use cost-based routing as the primary strategy with latency as a tiebreaker, weighted by remaining rate limit headroom. This ensures you always use the cheapest available backend, prefer faster backends when costs are equal, and avoid backends that are approaching their rate limits.

Multi-provider load balancing distributes LLM requests across two or more providers — for example, OpenAI, Anthropic, and Google — to achieve cost savings, higher aggregate throughput, and resilience against single-provider outages. This is the most powerful form of LLM load balancing, but also the most complex because different providers have different API formats, capabilities, and response characteristics.

Provider normalization. The first challenge is protocol translation. OpenAI uses the /v1/chat/completions endpoint with messages arrays. Anthropic uses the /v1/messages endpoint with a different message format and a separate system parameter. Google uses the Gemini API with yet another format. A multi-provider load balancer must translate between these formats seamlessly. Tools like LiteLLM handle this translation, presenting a unified OpenAI-compatible interface to your application while translating to each provider's native format on the backend.

Model equivalence mapping. Not all models are interchangeable. You need to define equivalence classes — groups of models that produce acceptable quality for a given task. For example:

The load balancer routes within an equivalence class: if your application requests a frontier-tier model, the load balancer chooses between GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro based on cost, latency, and availability. It does not downgrade to a mid-tier model unless explicitly configured to do so as a fallback.

Cost arbitrage. Multi-provider load balancing enables provider arbitrage — exploiting price differences across providers for comparable models. If Gemini 1.5 Pro costs $1.25/MTok input and Claude 3.5 Sonnet costs $3.00/MTok input, routing 80% of frontier-tier traffic to Google and 20% to Anthropic yields a blended rate of $1.60/MTok — 47% cheaper than Anthropic-only. The 20% Anthropic allocation ensures you maintain familiarity with the provider and can quickly ramp up if Google experiences issues.

Practical considerations. Multi-provider load balancing introduces output variability. Different models have different response styles, formatting preferences, and edge-case behaviors. For user-facing applications, this variability may be noticeable and undesirable. Mitigations include: constraining output format with structured output modes (JSON mode, function calling), post-processing responses to normalize formatting, and using session affinity to keep all turns in a conversation on the same provider. For backend/batch workloads where output style does not matter (classification, extraction, scoring), multi-provider load balancing is a pure win with minimal downsides.

Cost-aware load balancing is a strategy where the routing decision for each request is driven primarily by minimizing the dollar cost of that request, subject to quality and latency constraints. This goes beyond simple model routing (always use the cheapest model) by incorporating real-time factors like remaining budget, rate limit headroom, cached pricing, and dynamic provider promotions.

How cost-aware routing works. For each incoming request, the load balancer:

1. Estimates the request cost for each available backend by multiplying the estimated input tokens by the backend's input price and adding an estimated output cost (based on historical average output length for similar requests).
2. Filters backends that violate hard constraints: insufficient rate limit headroom, exceeded budget cap, or known outage status.
3. Ranks remaining backends by estimated cost, with latency as a tiebreaker.
4. Selects the cheapest viable backend and forwards the request.

This process happens in under 1 millisecond for a typical setup with 3–5 backends.

Real-world savings example. Consider a production workload processing 500,000 requests per day with an average of 1,500 input tokens and 500 output tokens per request. Without cost-aware routing, all traffic goes to Claude 3.5 Sonnet:

Daily cost = 500K × ((1,500/1M × $3.00) + (500/1M × $15.00))
          = 500K × ($0.0045 + $0.0075)
          = 500K × $0.012
          = $6,000/day → $180,000/month

With cost-aware routing that sends 60% of traffic to Gemini 1.5 Pro (frontier tier at lower cost) and keeps 40% on Claude:

Gemini cost  = 300K × ((1,500/1M × $1.25) + (500/1M × $5.00))
             = 300K × $0.004375 = $1,312.50/day
Claude cost  = 200K × ((1,500/1M × $3.00) + (500/1M × $15.00))
             = 200K × $0.012 = $2,400/day
Total        = $3,712.50/day → $111,375/month

That is a $68,625/month savings (38%) from routing alone, with no change to application code and equivalent output quality for the routed traffic. CostHawk's analytics dashboard shows the actual cost-per-request distribution across backends, making it easy to validate that your routing configuration is delivering the expected savings and to identify opportunities for further optimization.

Dynamic pricing adjustments. Provider pricing changes periodically — Google has historically been aggressive about cutting Gemini pricing, and Anthropic has introduced promotional rates for new models. A cost-aware load balancer should use a pricing table that is updated regularly (daily or weekly) rather than hardcoded. CostHawk maintains a live pricing database that reflects current provider rates, ensuring your routing decisions are based on accurate cost data.

There are three practical approaches to implementing LLM load balancing, ranging from lightweight application-level solutions to dedicated infrastructure.

Approach 1: Application-level load balancing. The simplest approach — implement routing logic directly in your application code. This works well for small teams with a single application and 2–3 backends.

interface LLMBackend {
  name: string
  client: OpenAI
  weight: number
  costPerInputMTok: number
  costPerOutputMTok: number
  currentRPM: number
  maxRPM: number
}

const backends: LLMBackend[] = [
  {
    name: "openai-primary",
    client: new OpenAI({ apiKey: process.env.OPENAI_KEY_1 }),
    weight: 3,
    costPerInputMTok: 2.50,
    costPerOutputMTok: 10.00,
    currentRPM: 0,
    maxRPM: 10000
  },
  {
    name: "openai-secondary",
    client: new OpenAI({ apiKey: process.env.OPENAI_KEY_2 }),
    weight: 2,
    costPerInputMTok: 2.50,
    costPerOutputMTok: 10.00,
    currentRPM: 0,
    maxRPM: 5000
  },
  {
    name: "anthropic-overflow",
    client: new OpenAI({
      apiKey: process.env.ANTHROPIC_KEY,
      baseURL: "https://litellm-proxy:4000/v1"
    }),
    weight: 1,
    costPerInputMTok: 3.00,
    costPerOutputMTok: 15.00,
    currentRPM: 0,
    maxRPM: 4000
  }
]

function selectBackend(): LLMBackend {
  // Filter out backends at rate limit capacity
  const available = backends.filter(
    b => b.currentRPM < b.maxRPM * 0.9
  )
  if (available.length === 0) throw new Error("All backends at capacity")

  // Cost-based selection with weighted randomization
  available.sort((a, b) => a.costPerInputMTok - b.costPerInputMTok)
  return available[0] // Cheapest available
}

Approach 2: Proxy-based load balancing. Use an LLM proxy like LiteLLM that has built-in load balancing across multiple backends. This separates routing logic from application code and provides a single endpoint for all your services.

# litellm-config.yaml
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: sk-key-1
    model_info:
      id: openai-account-1
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: sk-key-2
    model_info:
      id: openai-account-2

router_settings:
  routing_strategy: "cost-based"
  num_retries: 2
  retry_after: 5

Approach 3: Gateway-level load balancing. For enterprise deployments, use a dedicated API gateway (Kong, AWS API Gateway, or a custom solution) with LLM-aware plugins that handle routing, rate limiting, and cost tracking. This approach provides the most control and scalability but requires the most infrastructure investment.

Regardless of approach, monitor your load balancing effectiveness. CostHawk's per-provider and per-model analytics show traffic distribution, cost distribution, error rates, and latency percentiles across all backends, making it easy to identify imbalances and tune your configuration.

Rate limits are the primary reason organizations implement LLM load balancing, and managing them effectively requires understanding how providers enforce limits, how to detect approaching limits, and how to respond when limits are hit.

How providers enforce rate limits. All major LLM providers enforce two types of limits: requests per minute (RPM) and tokens per minute (TPM). Limits vary by model and account tier. When a limit is exceeded, the provider returns a 429 Too Many Requests response with a Retry-After header indicating how many seconds to wait. Some providers (notably Anthropic) also enforce daily token limits and concurrent request limits.

Rate limit headers returned with every response provide real-time visibility into your remaining capacity:

x-ratelimit-limit-requests: 10000
x-ratelimit-remaining-requests: 7423
x-ratelimit-limit-tokens: 2000000
x-ratelimit-remaining-tokens: 1456000
x-ratelimit-reset-requests: 42s
x-ratelimit-reset-tokens: 18s

Proactive rate limit management. Rather than waiting for a 429 error and then failing over, a well-designed load balancer tracks rate limit headroom in real time and routes away from backends that are approaching their limits. The algorithm:

1. After each response, parse the x-ratelimit-remaining-requests and x-ratelimit-remaining-tokens headers.
2. Calculate the utilization percentage for each backend: utilization = 1 - (remaining / limit).
3. When utilization exceeds 80%, begin shifting new requests to other backends.
4. When utilization exceeds 95%, stop sending any new requests to that backend until the rate limit window resets.

This proactive approach prevents 429 errors from ever reaching your application, providing a smoother user experience than reactive retry-based approaches.

Multi-account scaling. The most common load balancing pattern is distributing traffic across multiple accounts with the same provider to multiply effective rate limits. Two OpenAI organization accounts with 10,000 RPM each give you an effective limit of 20,000 RPM. This is straightforward to implement but requires managing multiple billing relationships and ensuring your use case complies with the provider's terms of service regarding multiple accounts.

Retry and backoff strategy. Even with proactive rate limit management, 429 errors can occur during sudden traffic spikes. Implement exponential backoff with jitter for retries:

async function retryWithBackoff<T>(
  fn: () => Promise<T>,
  maxRetries = 3
): Promise<T> {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await fn()
    } catch (err: any) {
      if (err.status !== 429 || i === maxRetries - 1) throw err
      const baseDelay = Math.pow(2, i) * 1000 // 1s, 2s, 4s
      const jitter = Math.random() * 1000
      await new Promise(r => setTimeout(r, baseDelay + jitter))
    }
  }
  throw new Error("Unreachable")
}

CostHawk tracks 429 error rates per provider and per key, alerting you when error rates spike so you can add capacity (more accounts) or adjust routing weights before the problem impacts users.