Max Tokens

If max_tokens is set lower than what the model needs to complete its response, the output will be truncated mid-sentence or mid-thought. The API response will include a finish_reason of "length" (OpenAI) or stop_reason of "max_tokens" (Anthropic) instead of the normal "stop" / "end_turn" value. Truncated responses can cause problems downstream — a JSON response cut off mid-object will fail to parse, a code snippet missing its closing brackets will not compile, and a summary missing its conclusion will confuse users. To find the right balance, monitor your truncation rate (percentage of responses with finish_reason="length"). A healthy truncation rate is under 1%. If it exceeds 5%, your max_tokens is too aggressive and you should increase it. CostHawk tracks finish_reason distributions per endpoint so you can spot truncation issues before they affect users.

Yes, but only if the model would have generated more tokens than your new limit. The model generates tokens sequentially — each token takes roughly the same amount of time (typically 10–40 milliseconds per token depending on the model and provider). If you set max_tokens=500 and the model would have generated 2,000 tokens, you save the generation time for 1,500 tokens, which is approximately 15–60 seconds. However, if the model naturally stops at 300 tokens, setting max_tokens=500 versus max_tokens=4096 makes no difference in latency because the model stopped before hitting either limit. The latency benefit is most noticeable for open-ended generation tasks (creative writing, long explanations) where the model tends to use all available output space. For structured tasks (classification, JSON extraction), the model typically stops well short of any reasonable limit, so the latency benefit is minimal.

Yes — dynamic max_tokens is a best practice for applications that handle diverse request types. A customer support bot might need 50 tokens for a "yes/no" answer but 1,500 tokens for a detailed troubleshooting guide. Setting a static max_tokens of 1,500 for all requests wastes money on the simple ones and potentially truncates the complex ones if you set it too low. Implement a routing layer that estimates the required output length based on request characteristics: question type, expected response format, and historical output lengths for similar queries. A simple approach is to maintain a lookup table of max_tokens values keyed by intent or endpoint. A more sophisticated approach uses a lightweight classifier to predict output length category (short/medium/long) and sets max_tokens accordingly. CostHawk's per-request token analytics provide the historical data you need to calibrate these dynamic settings.

They serve the same purpose but reflect naming differences across API versions and providers. OpenAI's newer chat completion API uses max_completion_tokens, while the legacy completion API used max_tokens. Anthropic's Messages API uses max_tokens. Google's Gemini API uses maxOutputTokens. Functionally, they all do the same thing: cap the maximum number of tokens in the model's generated response. The naming inconsistency is a source of confusion when switching between providers. If you use an abstraction layer (LangChain, LiteLLM, Vercel AI SDK), the library typically normalizes the parameter name for you. When working directly with provider SDKs, always check the current API documentation — using the wrong parameter name will either cause an error or be silently ignored, defaulting to the model's maximum output length and potentially inflating your costs.

When using streaming (Server-Sent Events), max_tokens works identically to non-streaming requests — it caps the total number of output tokens. The difference is in how truncation manifests. In a non-streaming request, you receive the full response at once and can check finish_reason before displaying it. In a streaming request, tokens arrive incrementally, and the user sees the response building in real time. If the response hits the max_tokens limit, the stream ends abruptly with a final event containing finish_reason="length". The user sees the response stop mid-sentence, which is a poor experience. For streaming use cases, set max_tokens 20–30% higher than your expected output to avoid visible truncation, and handle the "length" finish_reason gracefully in your UI (e.g., showing a "response truncated" indicator). The cost implication is the same as non-streaming — you pay for exactly the tokens generated regardless of delivery method.

Absolutely — max_tokens is the most direct per-request cost control available. Unlike budget limits (which operate at the account or key level over time), max_tokens caps the cost of each individual API call. The maximum output cost for any request is: max_tokens × output_price_per_token. For Claude 3.5 Sonnet at $15/MTok output, setting max_tokens=500 means the maximum output cost per request is $0.0075, regardless of what the model tries to generate. Combined with input cost (which you can estimate from your prompt size), this gives you a tight cost ceiling per request. For budgeting purposes, your maximum daily output cost is: max_tokens × output_price × daily_request_count. With max_tokens=500, $15/MTok output rate, and 50,000 daily requests, your worst-case daily output cost is $375. Without max_tokens limits, the same 50,000 requests with max output (8,192 tokens each) could theoretically cost $6,144/day.

For structured JSON output, calculate the expected token count based on your schema's field count and value sizes, then add a 50–100% buffer. A simple schema like {"sentiment": "positive", "confidence": 0.95} requires roughly 15–20 tokens. A more complex schema with 10 fields and array values might need 100–300 tokens. The key insight is that JSON formatting tokens (braces, colons, quotes, commas) add 30–50% overhead versus plain text. Here is a rough formula: estimated_tokens = (number_of_fields × 8) + (total_characters_in_values / 4) + 20 (for structure). If you use OpenAI's structured output mode or Anthropic's tool_use for JSON generation, the model is more token-efficient because it follows the schema precisely without verbose explanations. Always test with real data to calibrate — generate 100 sample outputs, measure the 99th percentile token count, and set max_tokens to that value plus 30%. CostHawk's output token histograms per endpoint make this calibration easy to maintain over time.

No. The max_tokens parameter only affects the output side of billing. You are always charged for the full input token count regardless of your max_tokens setting. Even if you set max_tokens=1 (requesting a single output token), you still pay for every token in your prompt, system message, conversation history, and tool definitions. This means max_tokens is only effective at controlling costs when output tokens represent a significant portion of your total token cost. For applications with very long prompts and short outputs (e.g., RAG with large context retrieval), optimizing input tokens through better retrieval, prompt compression, and context management will have a larger cost impact than tuning max_tokens. For applications with short prompts and long outputs (e.g., content generation, code writing), max_tokens is your primary cost lever. CostHawk's input/output cost breakdown per endpoint shows you which lever matters more for each part of your application.

Implement a three-layer defense: (1) Set max_tokens appropriately for each endpoint based on measured output distributions. (2) Check the finish_reason / stop_reason field in every API response. If it indicates truncation ("length" or "max_tokens"), decide whether to retry with a higher limit, return a partial result with a truncation indicator, or request a continuation. (3) Monitor your truncation rate and alert when it exceeds your threshold (we recommend alerting at 2% and investigating at 5%). For retry logic, increase max_tokens by 50–100% on the retry, but set a hard maximum to prevent infinite retries on pathologically long responses. For continuation logic, send the truncated response back as context and ask the model to continue from where it left off — but be aware this doubles the input tokens on the retry. CostHawk tracks finish_reason distributions and can alert you when truncation rates spike, which often indicates a prompt change that increased typical output length.