LLM Token Counting and Cost Estimation: BPE Mechanics, Context Budgets, and Real Math

Why Tokens, Not Words

Language models do not read characters or words. They read integers drawn from a fixed vocabulary — typically 100,000 to 300,000 entries — where each entry maps to a chunk of bytes. That mapping is learned, not designed, and it is why token accounting feels arbitrary until you see the algorithm.

Byte-Pair Encoding (BPE) builds the vocabulary greedily. Start with individual bytes, count adjacent pairs across a training corpus, merge the most frequent pair into a new symbol, and repeat until the vocabulary reaches its target size. Frequent sequences — the, ing, function, https — collapse into single tokens. Rare sequences stay fragmented.

Two consequences follow directly:

1. Common English is cheap; unusual strings are expensive. A UUID such as f47ac10b-58cc-4372-a567-0e02b2c3d479 has essentially no frequent substrings, so it tokenizes close to one token per few characters — often 20 or more tokens for 36 characters. Generate them with the UUID generator and you will see the cost of putting many in a prompt.
2. Structure is not free. Every {, ", : and newline in a JSON payload is billed. The same data as compact prose can cost meaningfully fewer tokens than as pretty-printed JSON.

What you get from this guide: an accurate estimation method, a corrected model of multi-turn spend, and the specific levers that lower cost without degrading output.

Counting Tokens Properly

The heuristic and its limits

The non-Latin row is the one that breaks budgets. A Hindi or Japanese support corpus can cost several times what an English corpus of the same visual length costs, which matters for anyone building for Indian or East Asian markets.

Count exactly, for real budgets

Every major provider exposes a counting path — use it rather than a third-party tokenizer that may not match the model you are calling:

// Anthropic: authoritative count for the exact model you will call.
const { input_tokens } = await client.messages.countTokens({
  model: "claude-opus-5",
  system: SYSTEM_PROMPT,
  messages,
});

Anthropic’s own guidance is explicit that tokenizers differ across model generations: the tokenizer introduced with Opus 4.7 can consume roughly 1x to 1.35x the tokens of the earlier one for identical input, so a migration re-baselines every budget you built.

The Cost Formula

Cost = (input_tokens / 1,000,000) × input_rate
     + (output_tokens / 1,000,000) × output_rate

Output is the expensive side — commonly three to five times the input rate — because it is generated sequentially. On models with reasoning or “thinking” enabled, internal reasoning tokens are billed as output even when the text is not returned to you, which is the single most common surprise on a first bill.

Published rates, checked September 2026

Per million tokens, standard (non-batch) processing. Verify against the provider’s own pricing page before committing a budget — these change, and several current rates are explicitly promotional.

Note the pattern rather than the exact numbers: a frontier tier near $5-10 input, a workhorse tier near $2, and a fast tier under $1. Price your workload against the tier, then confirm the current figure.

A worked example

A support assistant: 1,200-token system prompt, 400-token user message, 300-token reply, 50,000 conversations per month, on a $2 / $10 model.

Input  per call : 1,600 tokens → 1,600 × 50,000  =  80,000,000 tokens
Output per call :   300 tokens →   300 × 50,000  =  15,000,000 tokens

Input  cost : 80 × $2.00  = $160.00
Output cost : 15 × $10.00 = $150.00
Total                      = $310.00 / month

Now cache the 1,200-token system prompt. If cache reads bill at roughly a tenth of the input rate, that portion drops from $120 to about $12, taking the total to roughly $200 — a 35% cut with no change to the prompt’s content or the model’s behaviour.

The Multi-Turn Multiplier

The single largest estimation error in production LLM apps: the API is stateless. There is no server-side conversation. Every turn resends the full history.

Cumulative input across ten turns is 27,500 tokens, not 5,000. Cost grows with roughly the square of the turn count. Three mitigations, in order of preference:

1. Cache the stable prefix. Order the request as tools, then system, then messages, and keep everything volatile — timestamps, request IDs, the current question — after the last cache breakpoint. A single changed byte in the prefix invalidates everything after it.
2. Summarise or compact old turns rather than resending raw transcripts, once history exceeds what the task actually needs.
3. Trim what was never needed. Full tool schemas, entire retrieved documents, and verbose few-shot examples are often resent every turn for value that decayed after turn two.

Context Windows Are a Separate Constraint

Cost and capacity are different limits. A 1M-token context window does not mean filling it is wise:

• Filling the window is expensive. One million input tokens at $2 per million is $2 for a single request.
• Retrieval quality can degrade with very large, weakly relevant context; a focused 8K prompt frequently outperforms a padded 200K one.
• Output caps are separate. A model with a 1M input window may cap a single response far lower, so “it fits in context” says nothing about how much it can write back.

Budget three numbers independently: input tokens per call, output tokens per call, and calls per month. Optimising the wrong one is why prompt-shortening exercises so often fail to move the bill.

Step-by-Step: Building a Token Budget with Toolbox

1. Collect real inputs, not samples you wrote by hand — export ten to twenty actual prompts including system text and tool definitions.
2. Open the tool: visit the Toolbox LLM Token Calculator and paste them one at a time.
3. Record the median and the 90th percentile, not just the average. Tail requests drive both cost spikes and context overflow errors.
4. Multiply by real call volume and add the multi-turn factor: for an n-turn conversation, cumulative input is roughly n(n+1)/2 times the per-turn increment.
5. Confirm with the provider’s own counter before signing off a budget, since only that number matches what you will be billed.
6. Re-baseline after any model change. A tokenizer change alone can move counts by up to a third.

Outcome: a defensible monthly cost projection with a named optimisation order — cache the prefix, batch what can wait, and only then touch model tier or reasoning effort.

Before pasting production prompts into any estimator, remove secrets and personal data — see the prompt redaction guide.

Related guides: Prompt PII and secret redaction · JSON formatting and validation · UUID v4 vs v7