Evals

LLM evals are systematic, repeatable tests that measure the quality of a language model's outputs against predefined criteria. They serve the same function in AI systems that unit tests and integration tests serve in traditional software engineering — they provide automated confidence that the system behaves correctly, catch regressions before they reach production, and document expected behavior in executable form.

An eval consists of four components:

1. Test cases (the dataset). A collection of inputs (prompts, conversation histories, documents) paired with expected outputs or quality criteria. A summarization eval might include 200 articles with human-written reference summaries. A classification eval might include 1,000 customer messages with ground-truth labels. A code generation eval might include 150 function specifications with test suites that validate the generated code.
2. The system under test. The specific configuration you are evaluating — the model, system prompt, temperature, tools, and any preprocessing or postprocessing logic. Changing any one of these parameters can affect output quality, so evals must capture the full configuration, not just the model name.
3. The evaluation method. How you determine whether an output is good. This ranges from deterministic checks ("does the JSON parse?", "does the output contain required fields?") to subjective assessments ("is this summary accurate and coherent?"). The three primary evaluation methods — automated metrics, human review, and LLM-as-judge — are detailed in the next section.
4. Scoring and aggregation. How individual test case results are combined into overall quality scores. Common approaches include pass/fail rates, mean scores on a 1–5 scale, percentile distributions ("95% of outputs score 4 or higher"), and dimensional breakdowns (accuracy: 92%, tone: 88%, formatting: 97%).

A well-designed eval suite typically covers multiple quality dimensions relevant to the specific application:

For cost optimization specifically, conciseness is a dimension that directly impacts the bill — a model that produces equally accurate output in 150 tokens instead of 400 tokens saves you 62% on output costs per request. Evals that track output token counts alongside quality scores reveal which optimizations save money without losing quality and which ones save money because they lose quality (producing shorter but less complete outputs).

The three primary evaluation methodologies each have different strengths, costs, and appropriate use cases. Most production teams use a combination of all three, applying each method where it delivers the best quality signal per dollar spent.

1. Automated Metrics

Automated evals use deterministic or statistical measures to score outputs without any human or LLM involvement. They are the cheapest and fastest evaluation method, running in milliseconds and costing nothing beyond compute time. Common automated metrics include:

• Exact match / contains: Does the output contain a required string, match an expected value, or follow a regex pattern? Ideal for classification, extraction, and structured output tasks.
• BLEU / ROUGE / METEOR: N-gram overlap metrics that compare generated text to reference text. ROUGE-L (longest common subsequence) is widely used for summarization. These metrics correlate moderately with human judgment — a ROUGE-L score above 0.45 typically indicates acceptable summarization quality.
• Semantic similarity: Embed the output and the reference with an embedding model, then compute cosine similarity. Scores above 0.85 generally indicate semantically equivalent content. Costs ~$0.0001 per comparison using text-embedding-3-small.
• Code execution: For code generation tasks, run the generated code against a test suite. Pass/fail is the most rigorous eval possible for code — if the tests pass, the code works.
• JSON schema validation: For structured output, validate against a JSON schema. This catches formatting regressions instantly and costs nothing to run.

2. Human Review

Human evaluation remains the gold standard for subjective quality dimensions like tone, helpfulness, creativity, and nuanced accuracy. Human reviewers can catch subtle errors that automated metrics miss — a factually accurate summary that buries the most important point, a technically correct response that uses an inappropriate tone for the audience, or a code snippet that works but follows anti-patterns.

Human review is expensive and slow. A trained annotator reviewing LLM outputs typically costs $25–$60 per hour and can evaluate 20–40 outputs per hour, depending on complexity. That translates to $0.60–$3.00 per evaluation. At 500 test cases, a single human eval run costs $300–$1,500 and takes 12–25 hours. For this reason, human review is typically reserved for high-stakes evaluations: validating a major model switch, calibrating LLM-as-judge prompts, or auditing safety-critical outputs.

3. LLM-as-Judge

LLM-as-judge uses a secondary language model to evaluate the outputs of the primary model. A strong frontier model (Claude 3.5 Sonnet, GPT-4o) is given the original prompt, the model's output, and a detailed rubric, then asked to score the output on each quality dimension. This approach sits between automated metrics and human review in both cost and quality:

The cost comparison is striking. Evaluating 500 test cases across 5 quality dimensions:

• Automated metrics: $0 + a few seconds of compute = essentially free
• LLM-as-judge (GPT-4o): 500 cases × 5 dimensions × ~800 tokens per judgment × $10/MTok output = ~$20
• LLM-as-judge (GPT-4o mini): Same volume × $0.60/MTok output = ~$1.50
• Human review: 2,500 judgments at $1.50 each = ~$3,750

LLM-as-judge has become the workhorse method for production evals because it provides subjective quality assessment at 100–1,000x lower cost than human review. Research from Anthropic, OpenAI, and academic labs shows that frontier LLM judges agree with human annotators 80–90% of the time on most quality dimensions — comparable to inter-annotator agreement among humans themselves (typically 75–90%). The primary risks are position bias (LLMs tend to favor the first option in pairwise comparisons), self-preference bias (a model tends to rate its own outputs higher), and rubric sensitivity (small changes in the judge prompt can shift scores significantly). Mitigate these by randomizing option order, using a different model family as judge than the model being evaluated, and calibrating judge prompts against human annotations.

Evals are not free. They consume tokens, compute time, and — for human evaluation — significant labor costs. Understanding and budgeting for eval costs is essential for building a sustainable quality assurance practice that does not itself become a runaway expense line.

Eval spend as a percentage of production spend. Industry benchmarks suggest that eval costs should run between 5% and 20% of production AI spend, depending on the risk profile of your application. A consumer chatbot with low stakes per interaction might allocate 5%. A medical information system or financial advisory tool should allocate 15–20% because the cost of a quality regression is much higher. Here is how this breaks down in practice:

Hidden costs of evaluation. The direct token cost of running LLM-as-judge evaluations is only part of the picture. Factor in:

• Dataset creation and maintenance: Building a high-quality eval dataset takes 20–80 hours of expert time, depending on domain complexity. The dataset needs regular updates as your product evolves — new features mean new test cases, changed requirements mean updated rubrics. Budget 2–5 hours per month for dataset maintenance.
• Judge prompt engineering: Writing evaluation rubrics that produce consistent, calibrated scores is an art. Expect 10–20 hours to develop and calibrate a judge prompt for each quality dimension. Poorly calibrated judges produce noisy scores that do not correlate with real quality differences.
• Infrastructure: Running evals at scale requires orchestration — parallel API calls, result storage, score aggregation, regression detection, and alerting. Tools like Braintrust, Promptfoo, and Evalkit provide this infrastructure, typically at $100–$500/month. Open-source alternatives (OpenAI Evals, Ragas, DeepEval) are free but require self-hosting.
• Regeneration costs: Each eval run requires re-generating outputs from your production model configuration. If you are evaluating 500 test cases with a model that averages 1,200 tokens per response at $10/MTok output, that is $6 per regeneration run just for the outputs — before the judge even sees them. For daily evals, that is $180/month in regeneration costs alone.

Strategies to reduce eval costs without reducing quality:

1. Tiered evaluation. Run cheap automated metrics on every deployment (free). Run LLM-as-judge on daily builds (~$5–$20/run). Run human review monthly or on major changes ($500–$2,000/run). This gives you continuous coverage at low cost with periodic deep validation.
2. Sampling. You do not need to evaluate every test case every time. A random sample of 100–200 cases provides statistically meaningful quality estimates with 95% confidence intervals of ±3–5%. Running 200 cases instead of 2,000 cuts eval cost by 90%.
3. Use cheaper judge models. GPT-4o mini and Claude 3.5 Haiku produce LLM-as-judge scores that correlate 85–92% with frontier model judges at 5–15x lower cost. For routine daily evals, a cheaper judge is sufficient. Reserve frontier judges for high-stakes evaluations.
4. Cache eval inputs. If your eval dataset is stable, cache the model outputs from previous runs. When only the system prompt changes, you only need to regenerate outputs — you can reuse the same judge prompts and scoring infrastructure.
5. Invest in automated metrics first. Every quality dimension that can be captured by an automated metric (JSON validity, required field presence, output length, regex patterns) should be. These cost nothing to run and catch the most common regressions. LLM-as-judge should focus on dimensions that genuinely require subjective judgment.

CostHawk tracks eval-related API costs alongside production costs, so you can monitor your eval budget as a percentage of total spend and ensure it stays within your target range.

The relationship between evals and cost optimization is symbiotic — evals enable safe cost reduction, and cost constraints motivate smarter eval design. Here are the five most common cost optimization scenarios where evals play a critical role:

1. Model downgrade validation. Switching from a frontier model to a cheaper alternative is the highest-leverage cost optimization available — potential savings of 50–95%. But it carries the highest quality risk. The eval workflow:

1. Run your full eval suite against the current model to establish a quality baseline.
2. Run the same eval suite against the candidate cheaper model with the same prompt and parameters.
3. Compare scores dimension by dimension. If the cheaper model scores within 3–5% of the baseline on all critical dimensions, it is likely a safe swap.
4. Run a shadow deployment where both models process production traffic but only the original model's output is served. Compare quality scores on real traffic for 3–7 days.
5. If shadow results confirm eval suite findings, promote the cheaper model to production.

Example: A document summarization service migrating from Claude 3.5 Sonnet ($3/$15 per MTok) to Claude 3.5 Haiku ($0.80/$4 per MTok). Eval results show Haiku scores 93% vs Sonnet's 96% on accuracy, 91% vs 95% on completeness, and 97% vs 97% on formatting. The 3-point accuracy gap is acceptable for internal document summaries but might not be for customer-facing legal summaries. The eval data enables a nuanced decision: use Haiku for internal documents (saving 73%) and keep Sonnet for legal documents (maintaining quality). This targeted routing saves 58% overall while preserving quality where it matters most.

2. Prompt compression validation. Shortening system prompts reduces input token costs on every request. A 1,500-token system prompt sent with 50,000 daily requests at $2.50/MTok input costs $187.50/day. Compressing it to 800 tokens saves $87.50/day ($2,625/month). But shorter prompts may lose critical instructions. Evals quantify the impact: run the eval suite with both the original and compressed prompt, compare quality scores, and promote the shorter prompt only if scores are maintained.

3. Prompt caching ROI measurement. Anthropic's prompt caching charges a 25% premium on the first request (to write to cache) but gives a 90% discount on subsequent cache hits. OpenAI's caching gives a 50% discount with no write premium. Whether caching saves money depends on your cache hit rate, which depends on how many requests share the same prompt prefix. Evals confirm that cached responses maintain quality — theoretically they should be identical, but implementation bugs in caching logic can introduce subtle issues.

4. Temperature and parameter tuning. Reducing temperature from 1.0 to 0.2 produces more deterministic outputs that are often shorter (fewer tokens = lower cost) but may sacrifice creativity. Evals measure the tradeoff: at temperature 0.2, does the model still produce sufficiently varied and useful outputs for your use case? For classification and extraction tasks, low temperature almost always maintains quality while reducing output tokens by 10–25%. For creative writing or brainstorming, the quality loss is measurable and significant.

5. Batch API migration. OpenAI's Batch API offers a 50% discount but returns results within 24 hours instead of seconds. Anthropic's Message Batches API offers similar savings. Evals verify that batch-processed outputs are identical in quality to real-time outputs — they should be, since the same model processes them, but operational differences (timeout handling, retry logic, error recovery) can introduce discrepancies. Run your eval suite on a sample of batch outputs before migrating latency-tolerant workloads.

The common thread across all five scenarios is that evals transform cost optimization from guesswork into engineering. Instead of hoping a change does not break quality, you measure it. Instead of debating whether the cheaper model is "good enough," you have data showing exactly where it falls short and by how much. This data-driven approach is what separates teams that successfully reduce AI costs by 40–60% from teams that bounce between models, reverting changes after quality complaints, never confident in their optimization decisions.

A production eval pipeline is infrastructure — it requires the same engineering rigor as your CI/CD pipeline, monitoring stack, or data pipeline. Here is a reference architecture for building an eval pipeline that supports continuous quality monitoring and cost optimization.

Step 1: Define your eval dataset.

Start with 100–500 test cases that represent the distribution of real production traffic. Each test case should include:

• Input: The full prompt (system message + user message + any context) that will be sent to the model.
• Expected output (optional): A reference answer for cases where one exists. Not all evals require reference answers — LLM-as-judge can evaluate quality without one.
• Metadata: Tags for difficulty level, category, edge-case type, and any other dimensions useful for slicing results.
• Quality criteria: What "good" looks like for this specific test case. This might be a rubric, a set of required elements, or a pass/fail condition.

Source test cases from three places: (1) curate examples from production logs that represent common request types, (2) hand-write edge cases that test known failure modes, and (3) synthetically generate diverse inputs using an LLM. A good dataset overrepresents edge cases and failure modes — production traffic is 80% easy cases that any model handles well, so your eval dataset should be 50%+ hard cases that differentiate model quality.

Step 2: Implement evaluation methods.

Layer three evaluation methods in order of cost and depth:

// Layer 1: Automated metrics (run on every evaluation)
function automatedEval(output: string, testCase: TestCase): AutomatedScore {
  return {
    jsonValid: isValidJSON(output),
    containsRequiredFields: checkRequiredFields(output, testCase.schema),
    outputTokenCount: countTokens(output),
    withinLengthLimit: countTokens(output) <= testCase.maxTokens,
    regexMatch: testCase.pattern ? testCase.pattern.test(output) : null,
  }
}

// Layer 2: LLM-as-judge (run daily or on deployments)
async function llmJudge(input: string, output: string, rubric: string): Promise<JudgeScore> {
  const response = await anthropic.messages.create({
    model: "claude-3-5-sonnet-20241022",
    max_tokens: 1024,
    messages: [{
      role: "user",
      content: `Evaluate the following AI response.\n\n` +
        `USER QUERY: ${input}\n\nAI RESPONSE: ${output}\n\n` +
        `RUBRIC: ${rubric}\n\n` +
        `Score each dimension 1-5 and explain your reasoning. ` +
        `Return valid JSON: {"accuracy": {"score": N, "reason": "..."}, ...}`
    }]
  })
  return JSON.parse(response.content[0].text)
}

// Layer 3: Human review (run monthly or on major changes)
function queueForHumanReview(testCase: TestCase, output: string): void {
  reviewQueue.push({ testCase, output, assignedTo: getNextReviewer() })
}

Step 3: Build the orchestration layer.

The orchestration layer manages the lifecycle of an eval run: generate outputs, apply evaluation methods, aggregate scores, detect regressions, and report results. Key requirements:

• Parallel execution: Generate outputs and run LLM-as-judge evaluations in parallel across test cases, respecting API rate limits. A 500-case eval that runs serially takes 30+ minutes; with 50 parallel requests, it finishes in 3–5 minutes.
• Result storage: Store every eval run's inputs, outputs, scores, and metadata in a database. Historical data enables trend analysis ("is quality improving or degrading over time?") and regression detection ("this deployment dropped accuracy by 4 points").
• Version tracking: Record the exact model, system prompt, temperature, and tools configuration for each run. You must be able to reproduce any eval run and trace quality changes to specific configuration changes.
• Cost tracking: Record the token cost of each eval run. CostHawk can tag eval-related API calls separately from production traffic, making it easy to track your eval budget.

Step 4: Integrate with CI/CD.

The highest-value integration is running evals automatically on pull requests that change prompts, model configurations, or LLM-related code. This catches quality regressions before they merge, the same way unit tests catch code bugs:

# .github/workflows/eval.yml
name: LLM Eval Suite
on:
  pull_request:
    paths:
      - 'src/prompts/**'
      - 'src/config/models.ts'
      - 'src/lib/llm/**'
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm run eval:automated  # Layer 1: free, fast
      - run: npm run eval:llm-judge  # Layer 2: ~$5-20, 5 min
      - run: npm run eval:compare-baseline  # Fails if regression detected

Step 5: Set quality gates.

Define minimum acceptable scores for each quality dimension and block deployments that fall below them. For example:

• Accuracy: minimum 90% (block deployment if below)
• Formatting: minimum 98% (block deployment if below)
• Completeness: minimum 85% (warn but allow deployment)
• Conciseness: advisory only (track but do not block)

Quality gates should be strict enough to catch real regressions but lenient enough to accommodate normal score variance (±2–3 points between runs due to LLM non-determinism). Use a rolling 3-run average rather than a single run's score to reduce false alarms from statistical noise.

Running evals once before launch is necessary but insufficient. Production AI systems drift — models receive silent updates, user behavior shifts, data distributions change, and accumulated prompt tweaks compound in unexpected ways. Continuous evaluation catches these slow regressions that one-time assessments miss.

Online evaluation vs offline evaluation. Offline evals (the pipeline described above) test the model against a fixed dataset in a controlled environment. Online evals measure quality on live production traffic. Both are essential:

Implementing online evaluation. The most practical approach is to sample 1–5% of production requests and run them through LLM-as-judge asynchronously. This does not add latency to the user-facing request and keeps costs proportional to traffic. For a system handling 100,000 requests/day, sampling 2% means evaluating 2,000 requests daily. At $0.01 per LLM-as-judge evaluation, that is $20/day ($600/month) for continuous quality monitoring.

Online eval architecture:

// In your request handler — sample and evaluate asynchronously
async function handleLLMRequest(userInput: string): Promise<string> {
  const response = await callLLM(userInput)
  
  // Sample 2% of requests for online evaluation
  if (Math.random() < 0.02) {
    // Fire-and-forget: do not block the response
    evaluateAsync({
      input: userInput,
      output: response,
      model: currentModelConfig.model,
      timestamp: Date.now(),
      requestId: generateRequestId()
    }).catch(err => logger.warn('Online eval failed', err))
  }
  
  return response
}

Detecting model drift. LLM providers update their models without notice. OpenAI has acknowledged updating GPT-4 and GPT-3.5-turbo between major version releases. These silent updates can change output quality, length, formatting, and behavior in subtle ways. Continuous evaluation is the only reliable way to detect these changes. Track your online eval scores on a 7-day rolling average. If the average drops by more than 2 standard deviations from its 30-day baseline, trigger an alert. This catches both sudden regressions (a model update that breaks a specific behavior) and gradual drift (slowly declining quality over weeks).

User feedback as an eval signal. Implicit and explicit user feedback provides the most direct quality signal available — it measures what actually matters to your users, not what your rubric says should matter. Integrate feedback mechanisms into your eval pipeline:

• Explicit feedback: Thumbs up/down, 1–5 star ratings, "Was this helpful?" buttons. Even a 5–10% response rate provides valuable signal at scale.
• Implicit feedback: Regeneration rate (user clicked "try again"), edit rate (user modified the AI output), copy rate (user copied the output — likely indicates satisfaction), session abandonment (user left after receiving the response — possible dissatisfaction).
• Escalation signals: User contacted support after an AI interaction, user switched from AI-assisted to manual workflow, user explicitly reported an incorrect AI response.

Feed these signals back into your eval pipeline. If users consistently rate responses from a particular model or prompt configuration lower than expected, your offline evals may have a blind spot. Use negative user feedback to generate new test cases that cover the failure modes your existing eval dataset misses.

Eval-driven cost optimization loop. The most mature AI cost management practices combine continuous evaluation with continuous optimization in a closed-loop system:

1. Monitor: CostHawk tracks per-request costs, model utilization, and spend trends in real time.
2. Hypothesize: Cost analytics reveal optimization opportunities — "60% of requests go to GPT-4o but 45% of those are simple classification tasks that a cheaper model could handle."
3. Test: Route a shadow sample of those classification requests to GPT-4o mini. Run evals comparing quality.
4. Validate: Evals show GPT-4o mini scores 96% vs GPT-4o's 97% on classification accuracy. The 1-point gap is within noise.
5. Deploy: Route classification requests to GPT-4o mini. Monitor online eval scores for regression.
6. Measure: CostHawk confirms the optimization saved $4,200/month with no quality degradation in online evals.
7. Repeat: Move to the next optimization opportunity identified by CostHawk's analytics.

This loop turns cost optimization from a periodic exercise ("let's review our AI costs this quarter") into a continuous engineering practice with measurable, data-backed improvements week over week.