Embedding

At its core, an embedding is a translation of human-readable text into a mathematical representation that machines can compare, search, and cluster. When you send text to an embedding API, the model processes the input through multiple transformer layers and outputs a single vector — an array of floating-point numbers — that represents the meaning of the entire input.

Consider these three sentences:

1. "The quarterly revenue exceeded projections by 12%"
2. "Q3 earnings beat analyst estimates, up 12 percent"
3. "The cat sat on the windowsill"

An embedding model would place sentences 1 and 2 close together in vector space (high cosine similarity, ~0.92) because they convey the same meaning, despite using different words. Sentence 3 would be far away from both (low similarity, ~0.15) because it is semantically unrelated.

This property — that semantic similarity maps to geometric proximity — is what makes embeddings so powerful for search and retrieval. Traditional keyword search fails when users use different terminology than what appears in documents. Embedding-based search succeeds because it operates on meaning, not exact word matches.

How embedding models differ from generative models:

• Input only. Embedding models accept text input and return a vector. There is no text generation, no output tokens, and no autoregressive decoding.
• Fixed-size output. Regardless of input length, the output vector has a fixed number of dimensions (e.g., 1,536 for text-embedding-3-small). A 10-word query and a 500-word document both produce 1,536-dimensional vectors.
• Batch-friendly. Embedding APIs accept batches of inputs in a single request (OpenAI supports up to 2,048 inputs per batch), enabling efficient bulk processing at reduced latency.
• No temperature, no randomness. Embeddings are deterministic — the same input always produces the same vector (within floating-point precision).

The vector dimensionality is a key cost tradeoff. Higher-dimensional embeddings (3,072-d from text-embedding-3-large) capture more nuance and generally produce better retrieval quality, but they cost more to store in a vector database and more to compute similarity over. Lower-dimensional embeddings (512-d from voyage-3-lite) are cheaper to store and search but may sacrifice retrieval precision for some use cases.

Embedding model pricing is straightforward — you pay per input token with no output token charge — but the total cost depends heavily on your workload pattern. There are two distinct cost components: indexing cost (embedding your document corpus) and query cost (embedding user queries at runtime).

Indexing cost example: Embedding a corpus of 500,000 support articles averaging 1,500 tokens each:

Total tokens: 500,000 × 1,500 = 750,000,000 (750M tokens)

text-embedding-3-small: 750 × $0.02 = $15.00
text-embedding-3-large: 750 × $0.13 = $97.50
voyage-3:              750 × $0.06 = $45.00
embed-v3:              750 × $0.10 = $75.00

Query cost example: 200,000 user queries per day averaging 40 tokens each:

Daily tokens: 200,000 × 40 = 8,000,000 (8M tokens)

text-embedding-3-small: 8 × $0.02 = $0.16/day ($4.80/month)
text-embedding-3-large: 8 × $0.13 = $1.04/day ($31.20/month)
voyage-3:              8 × $0.06 = $0.48/day ($14.40/month)

Query costs are typically small. The real expense comes from re-indexing — which happens when you upgrade embedding models, change chunking strategies, or add new documents to a growing corpus. Budget for 2–4 full re-indexes per year as embedding models improve and your corpus evolves.

The dimensionality of an embedding vector — the number of floating-point numbers in the array — has cascading effects on storage cost, search latency, and retrieval quality. Choosing the right dimensionality is a cost-quality tradeoff that depends on your use case and scale.

Storage cost by dimensionality:

At 10 million vectors, the storage difference between 512-d and 3,072-d embeddings is 96 GB — which translates to significant vector database costs. Pinecone, Weaviate, Qdrant, and pgvector all charge based on storage and/or compute, and higher dimensionality increases both.

Search latency impact: Cosine similarity computation is proportional to vector dimensionality. Searching 3,072-d vectors is roughly 6x more compute-intensive than searching 512-d vectors. For real-time applications with strict latency requirements (sub-100ms), lower dimensionality enables faster retrieval without scaling up database compute.

Quality impact: Higher dimensionality generally captures more semantic nuance. On standard retrieval benchmarks (MTEB, BEIR), text-embedding-3-large (3,072-d) outperforms text-embedding-3-small (1,536-d) by 2–5% on average. Whether that quality difference matters depends on your use case. For a customer support chatbot, 2% better retrieval may not be perceptible. For a legal document search system where missing a relevant precedent has serious consequences, it may be critical.

Dimensionality reduction: OpenAI's text-embedding-3 models support a dimensions parameter that truncates the output vector to a specified size. You can request 256-d vectors from text-embedding-3-large, getting most of the model's quality at a fraction of the storage cost. This technique — called Matryoshka Representation Learning (MRL) — works because the most important information is packed into the first dimensions. In benchmarks, 512-d truncated vectors from text-embedding-3-large often outperform full 1,536-d vectors from text-embedding-3-small, giving you better quality at lower storage cost. This is one of the highest-leverage embedding cost optimizations available.

Retrieval-Augmented Generation (RAG) is the most common production use case for embeddings, and it introduces a multi-stage cost structure that teams must understand to optimize effectively. A RAG pipeline has two phases, each with distinct embedding cost dynamics.

Phase 1: Indexing (offline, batch)

During indexing, your document corpus is chunked into segments (typically 200–1,000 tokens each), and each chunk is embedded and stored in a vector database. The cost drivers are:

• Corpus size: More documents = more tokens to embed. A 100,000-document knowledge base at 500 tokens per chunk = 50M tokens.
• Chunking strategy: Smaller chunks produce more embeddings (higher cost) but may improve retrieval precision. Larger chunks produce fewer embeddings (lower cost) but may include irrelevant content.
• Overlap: Overlapping chunks (e.g., 100-token overlap between consecutive chunks) improve retrieval at the boundary between chunks but increase total token count by 10–30%.
• Re-indexing frequency: Every time you change your embedding model, chunking strategy, or add significant new content, you must re-embed. Budget for this.

Phase 2: Query (online, real-time)

When a user sends a query, the RAG pipeline embeds the query, searches the vector database for the most similar chunks, and passes those chunks as context to a generative model. The embedding cost at query time is typically negligible (a single 40-token query costs fractions of a cent), but the downstream generative cost is not. Each retrieved chunk becomes part of the generative model's input, consuming input tokens at the generation model's rate.

The hidden cost multiplier: RAG pipelines create a cost coupling between embedding and generation. If your retrieval step returns 5 chunks of 500 tokens each, that is 2,500 additional input tokens on every generative request. At GPT-4o's $2.50/1M input rate, that is $0.00625 per query — 300x more expensive than the embedding itself. The embedding cost is a rounding error; the cost of using the embeddings (as retrieved context in generation) is where the real expense lies.

This means optimizing your RAG embedding pipeline is not just about reducing embedding API costs — it is about reducing the volume and size of retrieved chunks to minimize downstream generation costs. Better embeddings that retrieve fewer, more relevant chunks can actually increase embedding costs while decreasing total pipeline costs by reducing the context stuffed into the generation step.

Embedding costs can be optimized across five dimensions: batching, caching, dimensionality reduction, model selection, and chunking strategy. Here are the practical techniques and their expected savings.

1. Batch API requests. Embedding APIs accept multiple inputs per request. OpenAI supports up to 2,048 inputs per batch call. Batching reduces HTTP overhead, improves throughput, and — for providers that offer batch pricing — can reduce per-token costs. OpenAI's Batch API provides a 50% discount ($0.01/1M instead of $0.02/1M for text-embedding-3-small) for asynchronous batch jobs that complete within 24 hours. If your indexing is not time-sensitive, always use batch mode.

2. Cache embedding results. Embedding the same text twice is pure waste. Implement an embedding cache keyed on a hash of the input text + model name. For query embeddings, a Redis or in-memory cache with a TTL of 1–24 hours catches repeated queries (which are common — many users ask similar questions). For document embeddings, store vectors alongside document hashes and only re-embed when the content changes. A well-implemented cache typically reduces embedding API calls by 20–40% for query workloads and eliminates redundant re-embedding for unchanged documents.

3. Use dimensionality reduction. As discussed in the previous section, OpenAI's dimensions parameter lets you request shorter vectors from text-embedding-3 models. But you can also apply post-hoc dimensionality reduction to any embedding model using PCA (Principal Component Analysis) or random projection. Reducing 1,536-d vectors to 512-d typically retains 95%+ of retrieval quality while cutting storage costs by 67% and speeding up vector search proportionally.

4. Choose the right model tier. Not every embedding task needs the best model. For internal search over well-structured documents with clear keywords, text-embedding-3-small ($0.02/1M) performs within 3% of text-embedding-3-large ($0.13/1M) — a 6.5x cost difference. Reserve the larger model for use cases where retrieval precision is critical (legal, medical, compliance). Run A/B tests on your actual data to quantify the quality difference before paying the premium.

5. Optimize chunking. Smaller chunks produce more embedding API calls but may improve retrieval. Larger chunks produce fewer calls but may dilute relevance. The sweet spot depends on your content: 300–500 tokens per chunk works well for most FAQ and documentation corpora. For long-form content (research papers, legal filings), 500–1,000 token chunks with 100-token overlap balances cost and quality. Avoid the common mistake of chunking too small (under 100 tokens) — this produces many low-context embeddings that increase both embedding and vector database costs without improving retrieval.

Embedding costs are easy to overlook because individual requests are cheap, but they accumulate silently — especially during indexing operations, model migrations, and traffic spikes. Effective monitoring requires tracking embedding usage as a distinct cost category, separate from generation costs.

Key metrics to monitor:

• Daily embedding token volume: Track total tokens sent to embedding APIs per day, broken down by model. Sudden spikes indicate unexpected re-indexing, runaway batch jobs, or application bugs that embed content redundantly.
• Embedding cost per query: For RAG pipelines, track the embedding cost component of each user query. This is typically tiny (sub-cent), but it establishes a baseline for detecting anomalies.
• Indexing cost per run: Log the total cost of each indexing operation. Compare against previous runs to detect scope creep (growing corpus size), chunking changes that increase token volume, or accidental full re-indexes when incremental updates were intended.
• Cache hit rate: If you implement embedding caching, monitor the hit rate. A healthy query cache should achieve 20–40% hit rate for diverse user queries and 60–80% for applications with repetitive query patterns. A dropping hit rate may indicate cache misconfiguration or a shift in query patterns.
• Cost per document indexed: Normalize indexing costs by document count to detect changes in average document length, chunking strategy, or overlap settings.

Alerting thresholds:

• Alert if daily embedding token volume exceeds 2x the 7-day moving average — this catches runaway indexing jobs.
• Alert if embedding spend exceeds 10% of total AI API spend — for most workloads, embedding should be a small fraction of total cost, and exceeding this threshold suggests inefficiency.
• Alert on new embedding models appearing in usage logs — unauthorized model changes can indicate configuration drift or unauthorized experimentation.

CostHawk tracks embedding API calls as a dedicated category in your usage dashboard, with per-model and per-key breakdowns. You can set budget alerts specifically for embedding spend, separate from generation spend, ensuring that indexing operations do not silently consume your budget. The anomaly detection engine flags unusual embedding volume patterns, catching issues like accidental full corpus re-indexes before they complete and bill.