Prompt Engineering

Prompt engineering is the systematic practice of designing and optimizing the text inputs sent to LLMs to achieve specific, consistent, and cost-efficient outputs. It is not about tricking the model or finding magic phrases — it is about clear communication that reduces ambiguity, provides appropriate context, and sets explicit expectations for the response format.

A production prompt typically has three components:

System prompt: A set of persistent instructions that define the model's role, behavior constraints, output format, and domain knowledge. System prompts are sent with every request and represent a fixed per-request input token cost. Example:

You are a customer support classifier for an e-commerce company.
Classify each message into exactly one category: billing, shipping, returns, product, or other.
Respond with only the category name, no explanation.

This 40-token system prompt is dramatically more cost-efficient than a 500-token version that includes detailed explanations of each category, multiple examples, and verbose formatting instructions.

User prompt: The per-request input containing the specific task. For the classifier above, this would be the customer message to classify. The user prompt length varies per request and is largely determined by the application (you cannot control how much text the user writes), but you can control how you present it — adding unnecessary framing or context around the user input adds tokens.

Few-shot examples: Input-output pairs included in the prompt to demonstrate the desired behavior. Few-shot examples are powerful for quality but expensive in tokens — each example adds both an input and an output to the prompt. Three examples of 100 tokens each add 300 tokens to every request. At 100,000 requests per day on GPT-4o, those three examples cost $75/day ($2,250/month). The cost-quality tradeoff of few-shot examples must be evaluated empirically.

The goal of prompt engineering is to find the minimal prompt that produces acceptable output quality. Every unnecessary token is waste. Every missing instruction that causes poor output (requiring retries or human correction) is also waste, just less visible. The discipline lies in finding the sweet spot between these two failure modes.

Prompt design creates cost impact at every stage of the request-response cycle. Understanding the specific mechanisms helps you target optimizations with the highest return.

System prompt overhead: The system prompt is the tax you pay on every request. It is the single most leveraged target for cost optimization because its cost is multiplied by your total request volume. A team that reduces their system prompt from 2,000 tokens to 800 tokens and processes 200,000 requests per day on Claude 3.5 Sonnet ($3.00/1M input) saves:

Savings per request: (2,000 - 800) × ($3.00 / 1,000,000) = $0.0036
Daily savings: $0.0036 × 200,000 = $720
Monthly savings: $720 × 30 = $21,600
Annual savings: $259,200

Quarter of a million dollars per year from editing a text file.

Verbose vs concise instructions: Compare these two prompt approaches for the same task:

// Verbose (487 tokens):
"I would like you to carefully analyze the following customer support 
message and determine which category it falls into. The categories are 
as follows: 'billing' for any questions about charges, invoices, 
payment methods, or subscription costs; 'shipping' for questions about 
delivery status, tracking numbers, shipping costs, or delivery 
timeframes; 'returns' for requests to return items..."

// Concise (43 tokens):
"Classify into: billing, shipping, returns, product, other. 
Output the category name only."

The concise version is 91% shorter — 444 fewer input tokens per request — with identical classification accuracy on most models. At scale, this difference is tens of thousands of dollars per month.

Output length control: Prompt phrasing directly influences output verbosity. Phrases like "explain your reasoning," "be thorough," and "provide a detailed analysis" encourage the model to generate long outputs (500–2,000 tokens). Phrases like "respond in under 50 words," "return JSON only," and "one sentence" produce short outputs (20–100 tokens). Since output tokens cost 4–5x more than input tokens, a prompt that elicits 100 output tokens instead of 500 saves significant money:

// On GPT-4o ($10.00/1M output):
// 500 tokens output: $0.005 per request
// 100 tokens output: $0.001 per request
// Savings: $0.004 per request × 100,000 requests/day = $400/day = $12,000/month

Adding "be concise" to your system prompt is perhaps the cheapest optimization in all of AI cost management.

Certain prompt patterns are known to produce better quality-to-cost ratios. Here are the most effective patterns, ranked by typical cost impact.

1. Structured output specification. Telling the model exactly what format to produce eliminates verbose preambles, explanations, and disclaimers that inflate output tokens. Instead of asking for a "response," ask for a specific structure:

// High-cost pattern:
"Analyze this support ticket and tell me what to do."
// Typical output: 200-500 tokens of narrative explanation

// Low-cost pattern:
"Analyze this support ticket. Return JSON: {category, priority: 1-5, suggested_action, confidence: 0-1}"
// Typical output: 30-50 tokens of structured data

Structured output can reduce output tokens by 80–90% for extraction and classification tasks.

2. Selective few-shot examples. Few-shot examples improve quality but cost tokens. The cost-efficient approach is to include the minimum number of examples that achieve your quality target, and select examples that cover distinct edge cases rather than repeating similar patterns. Two well-chosen examples that demonstrate different categories often outperform five examples that show variations of the same pattern — at 60% lower cost. For maximum efficiency, select few-shot examples dynamically based on the input (retrieve the most relevant examples via embedding similarity), rather than including a static set with every request.

3. Chain-of-thought with budget. Chain-of-thought (CoT) prompting improves reasoning quality but generates more output tokens (the model writes its reasoning process). For cost efficiency, use CoT selectively — only for tasks that require multi-step reasoning — and constrain it:

// Expensive CoT:
"Think step by step and explain your reasoning."
// Output: 500-1,000 tokens of reasoning + answer

// Budget CoT:
"Think step by step in under 3 steps, then give the final answer on a new line starting with ANSWER:"
// Output: 100-200 tokens of reasoning + answer

4. Negative instructions (what NOT to do). Telling the model what to exclude is often more token-efficient than describing what to include. "Do not include disclaimers, caveats, or preambles" (10 tokens) prevents 50–200 tokens of unwanted output per response.

5. Reference-based prompting. Instead of describing the desired output format in words (expensive in prompt tokens), provide a single reference example and say "match this format exactly." One example often replaces hundreds of tokens of format description.

System prompt optimization is the single highest-leverage activity in prompt engineering cost management. Because the system prompt is included in every request, even small reductions compound into massive savings at scale. Here is a systematic approach to optimizing system prompts for cost.

Step 1: Audit your current system prompt. Measure the exact token count using tiktoken or your provider's tokenizer. Catalog each section of the prompt by purpose: role definition, behavior instructions, output format, examples, domain knowledge, and guardrails. Calculate the daily cost: token_count × daily_requests × price_per_input_token.

Step 2: Eliminate redundancy. Most system prompts contain redundant instructions — saying the same thing in different ways for emphasis. The model does not need emphasis; it needs clarity. Remove duplicated instructions, combine related guidance, and delete any instruction the model would follow by default (e.g., "respond in English" when all inputs are English).

Step 3: Compress instructions. Replace verbose natural language with concise directives:

Step 4: Move static examples to fine-tuning. If your system prompt includes 5+ few-shot examples that never change, consider fine-tuning the model on those examples instead. This moves the cost from per-request input tokens (paid every time) to a one-time training cost (paid once). A system prompt with 10 examples at 100 tokens each adds 1,000 tokens per request. At 100,000 requests per day on GPT-4o, that is $250/day in input tokens. Fine-tuning to eliminate those examples costs $25 once.

Step 5: Enable prompt caching. After compressing your system prompt, enable prompt caching (Anthropic: 90% discount, OpenAI: 50% discount on cached prefixes). The cache covers the static prefix of your prompt, which is typically the system prompt. A 500-token system prompt cached at 90% discount costs only 50 effective tokens per request. Compression + caching together can reduce system prompt cost by 95%+.

Step 6: A/B test aggressively. After each round of optimization, run the compressed prompt against your evaluation set to verify quality is maintained. Track both quality metrics and cost metrics in CostHawk. If quality drops below your threshold, add back the minimum instructions needed to restore it.

Prompt engineering is an iterative process, and the iteration itself costs money. Every test run of a prompt variant consumes tokens at production rates. Without discipline, prompt testing can become a significant expense — especially when testing against large evaluation datasets or expensive models.

The cost of testing:

// Testing a prompt variant against 500 eval examples on GPT-4o:
// Average input: 800 tokens, average output: 200 tokens
// Input cost: 500 × 800 / 1,000,000 × $2.50 = $1.00
// Output cost: 500 × 200 / 1,000,000 × $10.00 = $1.00
// Total per variant: $2.00
// Testing 10 variants: $20.00
// Testing 10 variants across 3 models: $60.00

// Same test on GPT-4o-mini:
// Input cost: 500 × 800 / 1,000,000 × $0.15 = $0.06
// Output cost: 500 × 200 / 1,000,000 × $0.60 = $0.06
// Total per variant: $0.12
// Testing 10 variants: $1.20
// Testing 10 variants across 3 models: $3.60

Testing on GPT-4o-mini is 16x cheaper than GPT-4o. For initial prompt iteration, always test on the cheapest model first to narrow down candidates before validating on the target model.

Cost-efficient testing practices:

• Use a small, representative eval set. 50–100 carefully selected examples that cover your main use cases and edge cases are sufficient for initial screening. Reserve the full 500+ example eval set for final validation of the top 2–3 candidates.
• Test on cheap models first. Iterate on GPT-4o-mini or Gemini Flash until you have a strong prompt, then validate on your production model. Prompt patterns that work on cheap models generally transfer to expensive models.
• Version control your prompts. Track every prompt variant, its eval results, and its token count. Use a structured format (YAML, JSON, or a prompt management tool) so you can compare versions objectively.
• Measure token counts, not just quality. A prompt variant that scores 2% higher on quality but uses 50% more tokens may not be worth the cost increase. Track both metrics side by side.
• Set a testing budget. Allocate a fixed dollar amount per prompt optimization cycle (e.g., $50). This forces disciplined experimentation and prevents the common trap of endless iteration that costs more in testing than it saves in production.

CostHawk's per-key analytics let you create a dedicated API key for prompt testing, isolating test costs from production costs. This gives you clear visibility into how much you are spending on iteration versus how much you are saving in production.

Prompt engineering and model routing are complementary strategies that, combined, can reduce AI API costs by 80–95% compared to using a single expensive model with unoptimized prompts. The combination works because prompt engineering maximizes quality per token, while model routing directs each request to the cheapest model that meets the quality bar.

How they work together:

Consider a customer support system that handles three types of queries: simple FAQ lookups (60% of traffic), moderately complex troubleshooting (30%), and escalation-worthy complaints (10%).

// Daily cost comparison (100,000 queries/day):
// Naive: 100,000 × $0.0125 = $1,250/day ($37,500/month)
// Optimized: (60,000 × $0.0003) + (30,000 × $0.0006) + (10,000 × $0.0050)
//         = $18 + $18 + $50 = $86/day ($2,580/month)
// Savings: $34,920/month (93% reduction)

The 93% cost reduction comes from two layers: prompt engineering (reducing tokens per request) and model routing (using cheaper models for simpler tasks). Neither alone achieves the full savings.

Implementing prompt-aware routing:

• Task classification prompt: Use a tiny, cheap classifier (GPT-4o-mini with a 50-token prompt, cost: $0.00003) to categorize each incoming request by complexity. Route simple requests to a cheap model with a minimal prompt, and complex requests to an expensive model with a detailed prompt.
• Prompt variants per tier: Maintain different system prompts for different model tiers. The GPT-4o-mini prompt can be shorter and more directive (the model follows simple instructions well). The GPT-4o prompt can include more nuance and context for complex cases.
• Fallback chains: Try the cheap model first. If the output fails a quality check (confidence score, format validation, keyword presence), retry with the expensive model. Most requests succeed on the cheap model, and the retry cost for the minority that fail is lower than routing everything to the expensive model.

CostHawk's per-model and per-key analytics are essential for measuring the effectiveness of routing strategies. Compare cost-per-query and quality metrics across model tiers to find the optimal routing thresholds. Adjust thresholds over time as models improve and prices change — what required GPT-4o six months ago may be handled by GPT-4o-mini today.