Time to First Token (TTFT)

Time to First Token is the wall-clock duration from the moment your application sends an HTTP request to an LLM API endpoint to the moment it receives the first byte of the streamed response containing a generated token. The measurement point is precise:

• Start: The instant the HTTP request leaves your application (specifically, when the request body has been fully sent and the TCP socket is waiting for a response)
• End: The instant the first SSE data: event containing a delta (content token) is received by your application

TTFT captures several phases of processing that occur before the model begins generating output:

1. Network transit (outbound): Your request travels from your server to the provider's API gateway. Typical time: 10–50 ms depending on geographic distance and network conditions.
2. API gateway processing: The provider validates your API key, checks rate limits, and routes the request to an inference server. Typical time: 5–20 ms.
3. Queue wait: If all inference GPUs are busy, your request waits in a queue. This is the most variable component — it can be 0 ms under light load or 2,000+ ms during peak traffic or provider incidents.
4. Prefill computation: The model processes your entire input prompt, computing attention across all input tokens to build the key-value cache. This is the core computational step. Typical time: 50–500 ms, scaling roughly linearly with input token count.
5. First token generation: The model samples the first output token from its probability distribution. Typical time: 5–20 ms (one decode step).
6. Network transit (return): The first SSE event travels back to your application. Typical time: 10–50 ms.

In practice, TTFT is dominated by prefill computation and queue wait. A well-optimized request with a short prompt and no queuing might achieve 100–200 ms TTFT. A long prompt during peak traffic might see 3,000+ ms TTFT, with most of that time split between queuing and prefill.

The measurement must be taken at the client side to capture the full end-to-end experience. Server-side TTFT measurements miss network transit time and do not reflect what the user actually experiences.

TTFT and total latency (time to last token, TTLT) measure different aspects of the user experience and are affected by different factors. Understanding the distinction is essential for choosing the right optimization strategy.

Example scenario: A user sends a 2,000-token prompt and the model generates a 400-token response.

• TTFT: 380 ms (prefill dominates)
• Inter-token latency: 25 ms/token average
• Decode time: 400 tokens × 25 ms = 10,000 ms
• Total latency (TTLT): 380 + 10,000 = 10,380 ms

In a streaming scenario, the user sees the first word at 380 ms and watches text flow in over 10 seconds. This feels responsive because there is continuous visual feedback. In a non-streaming scenario, the user waits 10.4 seconds with no feedback, which feels painfully slow.

This is why streaming fundamentally changes which metric matters. For streaming applications, optimizing TTFT has the most impact on perceived performance. For non-streaming applications (batch processing, API-to-API calls, background jobs), total latency matters more than TTFT because there is no user watching a cursor.

When TTFT and TTLT diverge dramatically: A request with 200 ms TTFT but 30,000 ms TTLT indicates a short prompt (fast prefill) but very long output. Conversely, a request with 2,000 ms TTFT but 3,000 ms TTLT indicates a long prompt (slow prefill) with short output. These profiles require different optimization strategies:

• High TTFT, low TTLT → optimize input (shorter prompts, prompt caching, smaller context)
• Low TTFT, high TTLT → optimize output (set max_tokens, request concise responses, use a faster model for generation)

CostHawk tracks both TTFT and TTLT for every traced request, making it easy to diagnose which dimension of latency is dominating for each endpoint or use case.

TTFT varies significantly across models and providers. The table below shows approximate P50 TTFT benchmarks for common models under typical production conditions (1,000-token input prompt, no queuing delay, US-based client). These benchmarks are gathered from public performance reports and community benchmarks as of early 2026:

Important caveats:

• These benchmarks assume a 1,000-token input. TTFT scales roughly linearly with input size for the prefill-dominated component. A 10,000-token input will have roughly 3–5x higher TTFT than a 1,000-token input.
• P95 TTFT is often 2–3x the P50 due to queuing variability. During provider capacity constraints, P95 can spike to 5–10x the P50.
• Reasoning models (o1, o3-mini) have uniquely high TTFT because they generate internal "thinking" tokens before producing visible output. The thinking phase can take seconds or even minutes, and the first visible token only appears after thinking completes.
• Prompt caching dramatically reduces TTFT when the cache is hit. Anthropic reports 80–85% reduction in TTFT for cached prompts. OpenAI's caching provides similar benefits.
• Geographic distance adds 50–150 ms of network latency. An application in Singapore calling a US-based API endpoint will see consistently higher TTFT than one in Virginia.

Use these benchmarks as starting points, then measure your actual TTFT in production. CostHawk records TTFT for every streaming request and computes P50, P95, and P99 percentiles by model, endpoint, and time period so you can establish and track your own baseline.

TTFT is influenced by a chain of factors from your application through the network to the provider's GPU fleet. Understanding each factor helps you identify which ones are within your control and which require provider-level changes.

1. Input prompt size (high impact, within your control). The prefill phase — where the model processes your input to build the key-value cache — scales roughly linearly with the number of input tokens. A 500-token prompt might require 80 ms of prefill on GPT-4o, while a 10,000-token prompt might require 600 ms. This is the factor you have the most control over. Reducing input size through prompt optimization, summarization of conversation history, or selective context injection directly reduces TTFT. Every 1,000 tokens you trim from the input saves approximately 30–80 ms of TTFT depending on the model.

2. Prompt caching (high impact, within your control). When the provider's prompt cache contains the prefill computation for your prompt prefix (typically the system prompt and any static context), the model can skip the prefill for the cached portion and begin generating output much faster. Anthropic's prompt caching reduces TTFT by 80–85% on cache hits. OpenAI's caching provides comparable acceleration. Structuring your prompts so that the static portion comes first (system prompt, few-shot examples) and the variable portion comes last (user query) maximizes cache hit rates and minimizes TTFT.

3. Queue wait time (high impact, outside your control). When a provider's inference fleet is at capacity, requests queue. Queue wait time is added directly to TTFT and is the primary cause of TTFT spikes. During peak hours or provider incidents, queue times can exceed 5,000 ms. You can mitigate this by: using provisioned throughput (reserved capacity) for critical workloads, implementing multi-provider failover (if Anthropic is slow, route to OpenAI), or using batch endpoints for non-latency-sensitive workloads to avoid competing for real-time capacity.

4. Model size (medium impact, within your control via model selection). Larger models have more parameters, which means more computation during both prefill and decode. Economy models (GPT-4o mini, Gemini Flash, Claude Haiku) consistently deliver lower TTFT than their larger counterparts. Choosing a smaller model for latency-sensitive operations is one of the most effective TTFT optimizations.

5. Network latency (medium impact, partially within your control). The round-trip time between your server and the provider's API endpoint adds directly to TTFT. Deploying your application in the same cloud region as the provider's primary inference infrastructure (typically US East for OpenAI, US East or Europe for Anthropic) minimizes network latency. Using the provider's edge endpoints or CDN-accelerated endpoints (where available) can further reduce network transit time by 20–60 ms.

6. Request overhead (low impact, within your control). Large tool/function definitions, extensive JSON schemas, and verbose system prompts all increase the request payload size and processing overhead. While smaller than the prefill impact, minimizing unnecessary request metadata contributes to lower TTFT at the margins. Strip unused tool definitions, minimize JSON schema verbosity, and avoid sending tools that the model is unlikely to need for a given request.

Reducing TTFT requires a systematic approach that addresses each contributing factor. Here are seven optimization strategies, ordered by typical impact:

1. Enable prompt caching. This is the single highest-impact TTFT optimization for most applications. Both Anthropic and OpenAI support automatic prompt caching where repeated prompt prefixes are cached on the provider's infrastructure. Anthropic caches prompt prefixes of 1,024+ tokens automatically and reduces TTFT by 80–85% on cache hits. To maximize cache hit rate: place your system prompt and static context at the beginning of the message array, keep the static prefix consistent across requests, and put the variable user query at the end. A typical RAG application with a 3,000-token system prompt and 2,000-token static few-shot examples can achieve 85%+ cache hit rates, reducing TTFT from ~500 ms to ~100 ms for the cached portion.

2. Reduce input prompt length. Every token in your input adds prefill time. Audit your prompts for redundancy, verbosity, and unnecessary context. Common culprits include: overly detailed system prompts that repeat instructions the model already follows, full conversation history when a summary would suffice, retrieved context chunks that are marginally relevant, and verbose JSON schemas for tool definitions. Teams that audit their prompts typically find 30–50% of input tokens can be eliminated without quality loss, reducing TTFT proportionally.

3. Select faster models for latency-sensitive operations. If your application has a chatbot interface where TTFT is critical, route those requests to a fast model (GPT-4o mini, Gemini Flash, Claude Haiku). Reserve slower, more capable models for background analysis or complex reasoning tasks where latency is less important. A well-implemented model routing strategy can achieve sub-200 ms TTFT for interactive use cases while still using frontier models for tasks that need them.

4. Implement multi-provider failover. When one provider experiences high latency, route to another. A simple implementation sends the request to your primary provider, and if TTFT exceeds a threshold (e.g., 1,500 ms without receiving the first token), cancels the request and retries with a secondary provider. More sophisticated approaches use a load balancer that tracks real-time TTFT per provider and routes new requests to the provider with the lowest current TTFT.

async function llmCallWithFailover(prompt: string) {
  const controller = new AbortController()
  const timeout = setTimeout(() => controller.abort(), 1500)

  try {
    const response = await anthropic.messages.create({
      model: "claude-3-5-sonnet-20241022",
      messages: [{ role: "user", content: prompt }],
      stream: true,
    }, { signal: controller.signal })
    clearTimeout(timeout)
    return response
  } catch (e) {
    // Failover to OpenAI
    return openai.chat.completions.create({
      model: "gpt-4o",
      messages: [{ role: "user", content: prompt }],
      stream: true,
    })
  }
}

5. Use provisioned throughput. For production workloads with consistent volume, provisioned throughput eliminates queue wait time by reserving dedicated GPU capacity. OpenAI, Anthropic, and Google all offer provisioned throughput tiers. The cost is higher per token (you pay for reserved capacity whether you use it or not), but the latency is consistently low because your requests skip the shared queue.

6. Warm the inference path. Some providers cache model weights in GPU memory per endpoint. Infrequently used model-region combinations may require loading model weights from disk, adding seconds to the first request. If your application uses a less common model, consider sending periodic keepalive requests to keep the inference path warm.

7. Optimize client-side measurement. Ensure your TTFT measurement is accurate by capturing the timestamp before the HTTP request is sent (not before prompt construction) and capturing the first token timestamp when the first SSE data: event with content arrives (not the HTTP response headers). Inaccurate measurement leads to misguided optimization efforts.

Effective TTFT monitoring requires tracking percentile distributions, not just averages. A P50 TTFT of 300 ms can mask a P99 of 5,000 ms — and it is the P99 experience that drives user complaints and abandonment.

Key metrics to track:

• P50 TTFT: The median experience. This is what most users see most of the time. Target: under 500 ms for interactive applications.
• P95 TTFT: The tail experience affecting 1 in 20 requests. Users who consistently land in the P95 will perceive your application as slow. Target: under 1,500 ms.
• P99 TTFT: The worst-case experience for 1 in 100 requests. Important for SLA compliance. Target: under 3,000 ms.
• TTFT by model: Break down TTFT percentiles by model to identify which models are contributing to latency. This data directly informs model routing decisions.
• TTFT by input token count: Correlate TTFT with input size to quantify the prefill cost per token and identify requests with unnecessarily large inputs.
• TTFT by time of day: Provider capacity varies throughout the day. US business hours typically have higher TTFT than overnight periods due to demand patterns.

Alerting strategy:

Set tiered alerts that escalate with severity:

Dashboard design:

A TTFT monitoring dashboard should include: (1) a time-series chart showing P50, P95, and P99 TTFT over the last 24 hours, with the baseline overlaid for comparison; (2) a heatmap showing TTFT distribution by hour of day to reveal temporal patterns; (3) a breakdown by model showing which models contribute the most to tail latency; (4) a scatter plot of TTFT vs input token count to visualize the prefill cost curve.

CostHawk captures TTFT for every streaming request, computes all standard percentiles, and provides pre-built dashboards for TTFT monitoring. Anomaly detection uses a rolling-window statistical model that accounts for time-of-day and day-of-week patterns, reducing false alerts while catching genuine regressions. When TTFT spikes, CostHawk's trace drill-down shows whether the spike was caused by increased input size, provider queuing, or a specific model, enabling rapid root-cause identification.