Our token budget said 7k, the API rejected the request at 8k: counting with the wrong tokenizer

Our document summarisation service sends contracts and reports to an LLM API. To stay within the model's input limit and control cost, it trims each document to a budget before sending. The budget was set well below the limit for the configuration we used. Yet about 2% of requests failed with an error saying the prompt exceeded the maximum allowed, and costs per request were consistently about 15% higher than our own estimates.

The application log for a failing request:

summarise doc=ctr_88123 budget=7000 counted_input_tokens=6912 model=... status=400
error="prompt is too long: 8247 tokens > 8192 maximum"

Our counter said 6,912 tokens. The API said 8,247. A difference of 19%.

How we counted

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def count_tokens(text: str) -> int:
    return len(enc.encode(text))

def build_prompt(doc_text: str) -> str:
    return trim_to_budget(doc_text, budget=7000, counter=count_tokens)

The service had started with one provider's model years earlier and later switched to another provider's model family. The counting code stayed. cl100k_base is the tokenizer of an older family of one provider's models; the model we called used a different tokenizer.

Measuring the gap

We sent 300 sample documents to the provider's token counting endpoint and compared with our local count:

Different tokenizers split text differently, and the gap was larger for non-English text and for text full of numbers and table characters.

The other missing pieces

Even with the right tokenizer, the counter only counted the document. The request contained more:

• the system prompt (about 600 tokens);
• the chat formatting the API applies around messages (role markers and separators), a few tokens per message;
• a tool definition for structured output, whose JSON schema the provider converts into prompt tokens (about 300 tokens);
• and the configured output limit: some limits apply to input plus requested output, so reserving max_tokens matters.

What we ruled out

• Hidden characters in documents. Normalising whitespace changed counts by under 1%.
• Duplicate content from retries. Each failing request contained the document once.
• Provider-side bugs. The provider's count endpoint agreed with the error message within a few tokens.

Root cause

We estimated tokens with a tokenizer from a different model family and counted only the document text. The model's real tokenizer produced 10-19% more tokens for the same text, and the system prompt, message formatting and tool schema added several hundred more. Documents that our counter put just under budget were over the limit in reality.

Tokenizers are model-specific

A tokenizer maps text to the token ids a specific model was trained on. Different model families, and sometimes different generations from the same provider, use different vocabularies. The same sentence can be 20 tokens for one and 26 for another. There is no universal token count.

What counts towards the limit

Who is affected

Anyone who switched model providers or families and kept their token counting code, anyone counting with a generic tokenizer library "close enough" for budgets, and applications with tools or long system prompts. Non-English and number-heavy content diverges the most.

Environment

Count with the provider

Most major providers expose a token counting endpoint, or an SDK method, for their own models, which applies the real tokenizer, message formatting and tool schemas. We count the full request exactly as we will send it:

def count_request_tokens(client, model, system, messages, tools) -> int:
    resp = client.messages.count_tokens(model=model, system=system, messages=messages, tools=tools)
    return resp.input_tokens

Counting calls add a small amount of latency. For trimming, we do not call it repeatedly: we use a local estimate to trim roughly, then one exact count to confirm, and trim further only if needed.

Calibrate the local estimate

For the rough trimming step, we kept a cheap local estimate but calibrated it per language and document type using the measured ratios, with a margin:

RATIO = {"en": 1.12, "de": 1.22, "default": 1.25}

def estimated_tokens(text: str, lang: str) -> int:
    return int(len(enc.encode(text)) * RATIO.get(lang, RATIO["default"]))

The ratios are recalculated monthly from logged provider counts, because they drift with document mix.

Budget the whole request

CONTEXT_LIMIT = 8192          # input limit for this deployment
OUTPUT_RESERVE = 1024
OVERHEAD = count_request_tokens(client, MODEL, SYSTEM, [{"role": "user", "content": ""}], TOOLS)
DOC_BUDGET = CONTEXT_LIMIT - OUTPUT_RESERVE - OVERHEAD - 200     # safety margin

The overhead is measured once at startup with an empty document, so changes to the system prompt or tool schema automatically shrink the document budget.

Use the usage numbers the API returns

Every response includes the actual input and output token counts. We log them and compare with our estimate; an alert fires if the estimate is more than 5% low on average, which catches a model or prompt change that breaks the calibration.

Cost reporting

The same mistake had been understating costs in our monthly reports, because they multiplied our own estimates by the price per token. Reports now use the usage figures returned by the API, aggregated per feature and per customer, which matched the provider's invoice. Product decisions about which documents to summarise automatically changed once the real cost per document was visible.

Alternatives and trade-offs

• A larger context model reduces how often trimming matters but costs more per token and does not fix wrong estimates.
• Chunk and summarise hierarchically for long documents instead of trimming, which also improves quality for documents far above the budget.

How to test the fix

A test sends the 300 sample documents through the new trimming path in a staging environment and asserts that none exceed the limit and that the average gap between estimate and actual is under 3%.

Confirming in production

Context length errors dropped to zero. Cost estimates matched invoiced usage within 2%, which finance had asked about for months. Average document coverage improved slightly, because the calibrated estimate trimmed less aggressively than a blanket safety margin would have.

Regression guard

• Token counting goes through one module that knows the model; changing the model requires changing it.
• The estimate-versus-actual alert described above.
• A unit test fails if the configured model and the counter's model family do not match.

Security implications

None directly. Trimming user-provided documents to a budget should keep the system prompt and instructions intact and cut only document content, so that oversized inputs cannot push instructions out of the request. Our trimming always preserves the system prompt and tool definitions.

Prevention checklist

• Count tokens with the target model's tokenizer or the provider's counting API.
• Include system prompt, message formatting, tool schemas and output reservation in the budget.
• Calibrate local estimates per language and content type, with a margin.
• Log actual usage from responses and alert on estimate drift.
• Revisit token counting whenever you change models or providers.

References

• Anthropic, Token counting: https://docs.claude.com/en/docs/build-with-claude/token-counting
• OpenAI, tiktoken: https://github.com/openai/tiktoken
• Hugging Face, Tokenizers summary: https://huggingface.co/docs/transformers/tokenizer_summary
• Google, Gemini token counting: https://ai.google.dev/gemini-api/docs/tokens