Tokenization

Tokenization is the mandatory first step in every interaction with a large language model. Before an LLM can read your prompt, reason about it, or generate a response, your raw text must be converted into a sequence of integers — each integer representing a token from the model's fixed vocabulary. This process is deterministic: the same text always produces the same tokens with a given tokenizer, and it is reversible — tokens can be decoded back to the original text without information loss.

The need for tokenization arises from how neural networks operate. Transformers process fixed-dimensional vectors, not variable-length strings. The tokenizer bridges this gap by mapping every possible text input to a sequence of vocabulary indices, each of which is then looked up in an embedding table to produce the dense vectors the model actually computes on.

Why not just use characters or words? Character-level tokenization would require extremely long sequences (a 1,000-word paragraph is roughly 5,000 characters), making attention computation prohibitively expensive since transformer attention scales quadratically with sequence length. Word-level tokenization cannot handle misspellings, neologisms, or morphological variations without an impossibly large vocabulary. Sub-word tokenization — the approach used by BPE, WordPiece, and SentencePiece — strikes the optimal balance: common words become single tokens, uncommon words are split into recognizable fragments, and the vocabulary stays manageable (typically 32,000–200,000 entries).

The tokenization process is invisible to most API users — you send text, you get text back — but it operates behind every call and directly determines the token count that appears on your invoice. A 1,000-character English paragraph might become 250 tokens with one tokenizer and 290 with another. That 16% difference is pure cost overhead if you are on the less efficient tokenizer for your content type.

Understanding tokenization transforms cost management from guesswork to engineering. Instead of wondering why a seemingly short prompt costs more than expected, you can inspect the token breakdown, identify inefficiencies (unnecessary whitespace, verbose JSON keys, non-Latin characters that tokenize poorly), and make targeted optimizations that reduce your bill without changing what the model actually does.

Three tokenization algorithms dominate the LLM landscape, each with distinct mechanics and cost implications. Understanding their differences helps explain why the same text produces different token counts — and different costs — across providers.

BPE in detail: Byte-Pair Encoding starts by treating every byte (or Unicode character) as an individual token. It then scans the training corpus to find the most frequently occurring adjacent pair of tokens, merges them into a new single token, and repeats. After 100,000+ merge operations, common English words like "the", "and", and "function" are single tokens, while rare terms are split into sub-word fragments. OpenAI's o200k_base tokenizer uses approximately 200,000 merge operations, producing a large vocabulary that compresses English very efficiently — roughly 4.0 characters per token on average.

SentencePiece in detail: Unlike BPE, which builds its vocabulary bottom-up, SentencePiece's unigram model works top-down. It starts with a massive candidate vocabulary (often 1 million+ entries) and iteratively prunes tokens whose removal has the least impact on the overall probability of the training corpus. This approach tends to produce more uniform tokenization across languages because it optimizes globally rather than greedily. For multilingual workloads, SentencePiece-based models often produce 5–15% fewer tokens for non-English text compared to BPE-based models.

Cost implication: The algorithm determines the vocabulary, the vocabulary determines token counts, and token counts determine cost. Choosing a provider whose tokenizer is most efficient for your content type can yield meaningful savings before you even consider per-token pricing differences.

Because each provider uses a different tokenizer, the same input text produces different token counts — and therefore different costs — depending on which API you call. This is not a theoretical concern; the differences are measurable and financially material at scale.

Here is a real-world comparison of token counts for identical text samples across major tokenizers:

Key observations from this data:

• GPT-4o's 200K vocabulary gives it the best compression for English prose and code. Its larger vocabulary means more common multi-character sequences are captured as single tokens.
• Gemini's SentencePiece tokenizer excels on non-Latin scripts. The Chinese sample produces 16% fewer tokens on Gemini than GPT-4o, which translates to 16% lower input costs at identical per-token pricing.
• JSON and structured data shows the most variability. Tokenizers handle punctuation-heavy content differently, and the range can be 10–15% between the most and least efficient tokenizer.
• Code tokenization is consistently less efficient than prose across all tokenizers. Expect 30–40% more tokens per character for source code compared to natural language.

To illustrate the financial impact: a team processing 100 million tokens per day of mixed English and JSON content on Claude 3.5 Sonnet ($3.00/1M input) pays $300/day. If GPT-4o produces 8% fewer tokens for the same content at $2.50/1M input, the equivalent cost is $230/day — a saving of $70/day, or $25,550 per year, from tokenizer efficiency and pricing combined.

CostHawk normalizes token counts across providers in its cost comparison reports, showing you the effective price-per-character (not just price-per-token) so you can make apples-to-apples comparisons that account for tokenizer efficiency differences.

Tokenizer efficiency varies dramatically across languages because most tokenizers are trained primarily on English-heavy corpora. Languages with non-Latin scripts, agglutinative morphology, or limited representation in training data produce significantly more tokens per semantic unit, directly inflating costs for multilingual applications.

Why does this happen? BPE and similar algorithms learn merge rules from training data. Since training corpora are overwhelmingly English-dominant (estimates range from 50–80% English for most foundation models), English text benefits from the most merge operations. Common English words and phrases are single tokens. Chinese characters, by contrast, appear less frequently in training data, so the tokenizer learns fewer merges for Chinese character sequences — meaning each character is often split into two or three byte-level tokens.

The cost impact is substantial. A customer support chatbot serving Chinese-speaking users will pay 2.5–3x more per conversation in token costs compared to an identical English chatbot, all else being equal. For a high-volume multilingual application processing 1 million conversations per month across 10 languages, the tokenization efficiency gap can represent $10,000–$50,000/month in additional costs for non-English traffic.

Mitigation strategies for multilingual workloads:

• Evaluate SentencePiece-based models (Gemini) for non-Latin content. SentencePiece's language-agnostic approach often produces 10–20% fewer tokens for CJK languages compared to BPE-based models.
• Pre-translate to English when feasible. If your pipeline involves sending large context blocks (e.g., documents for summarization), translating to English first and then processing with a BPE model can actually be cheaper than processing the original language directly — translation costs may be offset by tokenization savings.
• Monitor per-language costs with CostHawk's tag-based analytics. Tag requests by language to see exactly how much each language costs relative to English and identify the highest-ROI optimization targets.
• Use shorter prompts in high-cost languages. If Chinese tokenizes at 2.6x the cost of English, every word in your Chinese system prompt costs 2.6x more. Keep system prompts minimal for high-cost languages and rely more on few-shot examples that fit the model's training distribution.

Source code and structured data formats like JSON, XML, and YAML are among the most token-inefficient content types, yet they are ubiquitous in AI-powered developer tools, code assistants, and data processing pipelines. Understanding why — and how to mitigate the cost — is critical for teams building these applications.

Why code tokenizes poorly:

• Punctuation density. Code contains far more punctuation (braces, brackets, semicolons, colons, operators) than natural language. Each punctuation character is typically its own token or shares a token with adjacent whitespace, producing a low characters-per-token ratio.
• Variable and function names. Identifiers like calculateMonthlyRevenue or user_authentication_middleware are domain-specific strings that do not appear in the tokenizer's training data as single entries. They are split into multiple sub-word tokens: ["calculate", "Monthly", "Revenue"] or ["user", "_authentication", "_middleware"].
• Indentation. Whitespace-sensitive languages like Python pay a token cost for every level of indentation. A function nested 4 levels deep with 4-space indentation consumes 16 characters of whitespace per line — roughly 4 tokens per line just for indentation.
• Repetitive boilerplate. Import statements, type annotations, and framework boilerplate add tokens that carry structural but not semantic novelty.

JSON is particularly expensive:

// This JSON payload is 164 characters and ~62 tokens:
{
  "user_id": "usr_12345",
  "subscription_plan": "enterprise",
  "monthly_cost": 299.99,
  "features": ["analytics", "alerts", "api_access"]
}

// The same data as compact text is 95 characters and ~24 tokens:
user usr_12345 enterprise $299.99 analytics,alerts,api_access

That is a 2.6x token reduction for the same information by switching from JSON to a compact text format. At scale — say, 100,000 requests per day each including a 500-token JSON context block — switching to compact text could save 38 million tokens per day, or $95/day on GPT-4o input pricing ($34,675/year).

Optimization strategies for structured data:

• Minify JSON before including it in prompts. Remove pretty-printing, collapse whitespace, and use short key names. {"n":"John","a":30} uses fewer tokens than {"name": "John", "age": 30}.
• Use CSV or TSV for tabular data. Column headers plus comma-separated rows produce 40–60% fewer tokens than the equivalent JSON array of objects.
• Summarize instead of including raw data. Instead of sending 50 rows of database results as JSON, send a natural-language summary: "The top 5 users by spend are: Alice ($4,200), Bob ($3,800)..." This conveys the same information in far fewer tokens.
• Strip comments and docstrings from code context. If you are sending source code as context for a code review or refactoring task, remove comments and docstrings that the model does not need to see. This can reduce code token counts by 15–25%.
• Use tree-sitter or AST-based extraction to send only the relevant code structures (function signatures, class definitions) rather than entire files.

CostHawk's per-request analytics show token counts alongside request metadata, making it straightforward to identify which requests contain token-heavy structured data and quantify the savings from format optimization.

Model selection is usually driven by capability, latency, and per-token price — but tokenization efficiency deserves a seat at the table. Two models with identical per-token pricing can have materially different effective costs if one tokenizer compresses your content more efficiently than the other.

Framework for tokenizer-aware model selection:

Step 1: Profile your content. Take a representative sample of your production prompts (at least 1,000 requests) and run them through each candidate model's tokenizer. Record the token count for each sample across all tokenizers. Tools for this include OpenAI's tiktoken library, Anthropic's @anthropic-ai/tokenizer, and Google's countTokens API endpoint.

Step 2: Calculate effective cost per character. This is the metric that accounts for tokenizer efficiency:

effective_cost_per_char = (price_per_million_tokens / 1,000,000) × (tokens / characters)

// Example for English prose:
// GPT-4o: ($2.50 / 1M) × (250 tokens / 1000 chars) = $0.000000625/char
// Claude 3.5 Sonnet: ($3.00 / 1M) × (268 tokens / 1000 chars) = $0.000000804/char
// Gemini 1.5 Pro: ($1.25 / 1M) × (280 tokens / 1000 chars) = $0.000000350/char

Step 3: Factor in output tokenization. Output tokens matter too. If one model's tokenizer produces more compact output (fewer tokens for equivalent semantic content), your output costs decrease. This is harder to measure because output content varies, but you can estimate by comparing the average output token count for identical prompts across models.

Step 4: Weight by content mix. If 70% of your traffic is English prose and 30% is JSON, calculate a weighted effective cost that reflects your actual content distribution. The model that is cheapest for English prose may not be cheapest for JSON-heavy workloads.

Practical recommendations by content type:

The takeaway: do not just compare price-per-token across providers. Compare price-per-character (or price-per-semantic-unit) to account for tokenizer efficiency. CostHawk's provider comparison reports show effective cost per request — normalizing for tokenizer differences — so you can see which provider is genuinely cheapest for your specific workload, not just cheapest on a per-token sticker price.