LLM Token Counting and Cost Estimation: BPE Mechanics, Context Budgets, and Real Math

Sun, 06 Sep 2026 00:00:00 +0000

QUICK ANSWER RFC SPECIFICATION SUMMARY

100% Client-Side

A token is a subword unit produced by the model's tokenizer, typically trained with byte-pair encoding, so cost and context limits are measured in tokens rather than words. English prose averages roughly four characters per token, but code, JSON, and non-Latin scripts run far higher. API cost equals input tokens multiplied by the input rate plus output tokens multiplied by the output rate, where output is normally three to five times more expensive. In multi-turn chat the whole history is resent every turn, so cumulative input grows roughly quadratically with turn count — which is why prompt caching and batch processing cut bills more than shortening prompts does.

Difficulty: Intermediate Time: 8 Mins Privacy: 100% Client-Side RAM Standard: Byte-Pair Encoding (Sennrich et al., 2016)

Interactive Workbench LIVE

LLM Token Counting and Cost Estimation: BPE Mechanics, Context Budgets, and Real Math

Open Fullscreen ↗

Initializing Workbench...

100% Client-Side RAM Sandbox

🔒 Private Execution: Computations run in local browser memory. Zero server uploads. Zero server uploads.

Launch Fullscreen →

Free In-Browser Tool TOOLBOX UTILITY

LLM Token Budget & Cost Matrix

Paste a prompt to estimate token counts and projected spend across model tiers. Proprietary prompts and system instructions stay in browser memory.

Open Token & Cost Calculator →

Why Tokens, Not Words

Language models do not read characters or words. They read integers drawn from a fixed vocabulary — typically 100,000 to 300,000 entries — where each entry maps to a chunk of bytes. That mapping is learned, not designed, and it is why token accounting feels arbitrary until you see the algorithm.

Byte-Pair Encoding (BPE) builds the vocabulary greedily. Start with individual bytes, count adjacent pairs across a training corpus, merge the most frequent pair into a new symbol, and repeat until the vocabulary reaches its target size. Frequent sequences — the, ing, function, https — collapse into single tokens. Rare sequences stay fragmented.

Two consequences follow directly:

Common English is cheap; unusual strings are expensive. A UUID such as f47ac10b-58cc-4372-a567-0e02b2c3d479 has essentially no frequent substrings, so it tokenizes close to one token per few characters — often 20 or more tokens for 36 characters. Generate them with the and you will see the cost of putting many in a prompt.
Structure is not free. Every {, ", : and newline in a JSON payload is billed. The same data as compact prose can cost meaningfully fewer tokens than as pretty-printed JSON.

What you get from this guide: an accurate estimation method, a corrected model of multi-turn spend, and the specific levers that lower cost without degrading output.

Counting Tokens Properly

The heuristic and its limits

Content type	Approximate tokens
English prose	~1 token per 4 characters (~0.75 words)
Source code	~1 token per 2.5-3.5 characters
Minified JSON	~1 token per 2.5-3 characters
CJK / Devanagari / Cyrillic	Frequently 1-2 tokens per character
Hashes, UUIDs, Base64	Near one token per 2-3 characters

The non-Latin row is the one that breaks budgets. A Hindi or Japanese support corpus can cost several times what an English corpus of the same visual length costs, which matters for anyone building for Indian or East Asian markets.

Count exactly, for real budgets

Every major provider exposes a counting path — use it rather than a third-party tokenizer that may not match the model you are calling:

// Anthropic: authoritative count for the exact model you will call.
const { input_tokens } = await client.messages.countTokens({
 model: "claude-opus-5",
 system: SYSTEM_PROMPT,
 messages,
});

Anthropic’s own guidance is explicit that tokenizers differ across model generations: the tokenizer introduced with Opus 4.7 can consume roughly 1x to 1.35x the tokens of the earlier one for identical input, so a migration re-baselines every budget you built.

The Cost Formula

Cost = (input_tokens / 1,000,000) × input_rate
 + (output_tokens / 1,000,000) × output_rate

Output is the expensive side — commonly three to five times the input rate — because it is generated sequentially. On models with reasoning or “thinking” enabled, internal reasoning tokens are billed as output even when the text is not returned to you, which is the single most common surprise on a first bill.

Published rates, checked September 2026

Per million tokens, standard (non-batch) processing. Verify against the provider’s own pricing page before committing a budget — these change, and several current rates are explicitly promotional.

Provider	Model	Input	Output	Note
Anthropic	Claude Opus 5	$5.00	$25.00	1M context
Anthropic	Claude Sonnet 5	$2.00	$10.00	1M context
Anthropic	Claude Haiku 4.5	$1.00	$5.00	200K context
OpenAI	gpt-6-astra	$10.00	$50.00	—
OpenAI	gpt-5.6-sol	$4.00	$20.00	Promotional pricing published at least through 21 Nov 2026
OpenAI	gpt-5.6-terra	$2.00	$12.00	—
OpenAI	gpt-5.5	$5.00	$30.00	Context under 272K
Google	Gemini 3.8 Flash	$0.75	$3.75	Rate published through 31 Dec 2026
Google	Gemini 3.1 Pro Preview	$2.00	$12.00	Rate varies by prompt length band

Note the pattern rather than the exact numbers: a frontier tier near $5-10 input, a workhorse tier near $2, and a fast tier under $1. Price your workload against the tier, then confirm the current figure.

A worked example

A support assistant: 1,200-token system prompt, 400-token user message, 300-token reply, 50,000 conversations per month, on a $2 / $10 model.

Input per call : 1,600 tokens → 1,600 × 50,000 = 80,000,000 tokens
Output per call : 300 tokens → 300 × 50,000 = 15,000,000 tokens

Input cost : 80 × $2.00 = $160.00
Output cost : 15 × $10.00 = $150.00
Total = $310.00 / month

Now cache the 1,200-token system prompt. If cache reads bill at roughly a tenth of the input rate, that portion drops from $120 to about $12, taking the total to roughly $200 — a 35% cut with no change to the prompt’s content or the model’s behaviour.

The Multi-Turn Multiplier

The single largest estimation error in production LLM apps: the API is stateless. There is no server-side conversation. Every turn resends the full history.

Turn	New input	Total input billed this turn
1	500	500
2	500	1,000
3	500	1,500
…	…	…
10	500	5,000

Cumulative input across ten turns is 27,500 tokens, not 5,000. Cost grows with roughly the square of the turn count. Three mitigations, in order of preference:

Cache the stable prefix. Order the request as tools, then system, then messages, and keep everything volatile — timestamps, request IDs, the current question — after the last cache breakpoint. A single changed byte in the prefix invalidates everything after it.
Summarise or compact old turns rather than resending raw transcripts, once history exceeds what the task actually needs.
Trim what was never needed. Full tool schemas, entire retrieved documents, and verbose few-shot examples are often resent every turn for value that decayed after turn two.

Context Windows Are a Separate Constraint

Cost and capacity are different limits. A 1M-token context window does not mean filling it is wise:

Filling the window is expensive. One million input tokens at $2 per million is $2 for a single request.
Retrieval quality can degrade with very large, weakly relevant context; a focused 8K prompt frequently outperforms a padded 200K one.
Output caps are separate. A model with a 1M input window may cap a single response far lower, so “it fits in context” says nothing about how much it can write back.

Budget three numbers independently: input tokens per call, output tokens per call, and calls per month. Optimising the wrong one is why prompt-shortening exercises so often fail to move the bill.

Step-by-Step: Building a Token Budget with Toolbox

Collect real inputs, not samples you wrote by hand — export ten to twenty actual prompts including system text and tool definitions.
Open the tool: visit the and paste them one at a time.
Record the median and the 90th percentile, not just the average. Tail requests drive both cost spikes and context overflow errors.
Multiply by real call volume and add the multi-turn factor: for an n-turn conversation, cumulative input is roughly n(n+1)/2 times the per-turn increment.
Confirm with the provider’s own counter before signing off a budget, since only that number matches what you will be billed.
Re-baseline after any model change. A tokenizer change alone can move counts by up to a third.

Outcome: a defensible monthly cost projection with a named optimisation order — cache the prefix, batch what can wait, and only then touch model tier or reasoning effort.

Before pasting production prompts into any estimator, remove secrets and personal data — see the .

Related guides: · ·

FAQ

Frequently Asked Questions

Why do the same words cost a different number of tokens in different models? ▼

Each model family ships its own tokenizer vocabulary, trained by byte-pair encoding or a SentencePiece variant on its own corpus. A term seen often during training becomes one token; another vocabulary may split it into three. Counts therefore differ across providers and even across generations from one provider — Anthropic notes the tokenizer introduced with Opus 4.7 can use roughly 1x to 1.35x the tokens of its predecessor.

Is the rule that one token equals four characters reliable? ▼

Only as a rough first pass for English prose, where about four characters or 0.75 words per token holds. It fails for code and JSON, where punctuation and field names fragment heavily, for non-Latin scripts that can cost several tokens per character, and for hashes or UUIDs that tokenize nearly per character. Budget from measured counts.

Why is my bill higher than my per-request estimate suggested? ▼

Chat APIs are stateless, so every turn resends the whole conversation as input and cumulative input grows roughly with the square of the turn count. Add system prompts, tool definitions, retrieved documents, and reasoning tokens billed as output, and resent context usually dominates spend rather than new output.

What is the cheapest safe way to cut LLM cost without hurting quality? ▼

Prompt caching first, because it changes price rather than behaviour: keep a stable prefix of system prompt, tool definitions, and reference documents ahead of the volatile part of the request. Then use asynchronous batch processing for non-urgent work, commonly at half price. Only after that consider lower reasoning effort or a smaller model, since those can change output quality.

Llm | Toolbox Guides & Articles

LLM Token Counting and Cost Estimation: BPE Mechanics, Context Budgets, and Real Math

LLM Token Counting and Cost Estimation: BPE Mechanics, Context Budgets, and Real Math

LLM Token Budget & Cost Matrix

Why Tokens, Not Words

Counting Tokens Properly

The heuristic and its limits

Count exactly, for real budgets

The Cost Formula

Published rates, checked September 2026

A worked example

The Multi-Turn Multiplier

Context Windows Are a Separate Constraint

Step-by-Step: Building a Token Budget with Toolbox

Frequently Asked Questions