All are priced at $5, yet the final bill differs by 30%. A senior OpenAI executive notes that tokens have never been directly comparable in terms of pricing.

The same piece of text is fed to two different models, one splits it into 766 tokens, and the other splits it into 1170 tokens.

The person who released this set of figures is Tibo, head of OpenAI Codex.

His exact words are: One token from OpenAI does not equal one token from another model. A lower price per token does not necessarily mean a lower total bill.

Everyone is comparing prices by "how many dollars per million tokens", as if token is a standard unit like gram or kilowatt-hour, but it is not.

To make it easier for people to understand, he also told a story about pizza.

Two identical pizzas.

The first shop cuts each pizza into 8 slices at $2 per slice. The second shop cuts each pizza into 16 slices at $1.25 per slice. The second shop advertises a lower unit price, but the whole pizza costs $20, while the first one only costs $16.

He added a remark: Your stomach doesn't care how many slices you just ate.

Each slice is cheaper, but the whole pizza costs more. Different slicing methods make unit prices no longer comparable.

Token is the smallest unit for model billing, you can understand it as the "slicing technique" the model uses to cut text.

For the same paragraph, different slicing techniques will result in different numbers of pieces. You are charged based on the number of pieces cut out. The more pieces you get, the more expensive the bill will be.

This comparison covers English text, technical documents, multilingual content and numerical content.

The tokenizer of GPT-5.6 Sol outputs 766 tokens, while the estimated result of Claude Opus 5 is 1170 tokens.

For the same piece of text, the slicing technique of GPT-5.6 Sol produces 34.5% fewer tokens.

And the input price of both models is $5 per million tokens.

The unit prices are exactly the same, but the number of pieces is 30% less, so the input cost is also 30% lower accordingly.

The trouble also lies right here.

If even the definition of "how big one token is" cannot be aligned between two vendors, then is the API price comparison table that everyone circulates every day still valid?

Why do two different token counts come out for the same piece of text?

This is because the unit of token has no unified measurement standard at all.

Each vendor trains its own tokenizer, and decides on its own how large the fragments of the text will be cut into.

For common words, the tokenizer takes the whole word as a single token; for rare words, one word will be split into three or four fragments.

Taking English as an example is the most intuitive. Words that appear frequently such as "the", "and" and "is" are assigned a unique dedicated ID by the tokenizer, so one word equals one token.

For a long word like "unbelievable", it has to be split into several fragments such as "un", "believ" and "able", so one word takes up three tokens.

The logic is simple: The tokenizer is generated by statistics from the training corpus. Any character combination that appears frequently will occupy a separate position as a single token. The remaining combinations can only be assembled from small fragments.

Therefore, "how many tokens a paragraph has" is essentially asking "how common the content in this paragraph is in the vendor's training corpus".

And English prose is exactly the type of content with the smallest difference in token counts across different models. For code, JSON, and long strings of numbers, the token count results from different vendors will differ even more significantly.

Even the token counts of the vendor's own old and new models are not interoperable

This is not a problem exclusive to any single vendor.

Anthropic's official documentation clearly states: Token counts are estimates, and the actual number of input tokens consumed when creating a message may have minor discrepancies.

They also provided a specific figure.

Models after Claude 4.7 use a new tokenizer, the same input text will generate about 30% more tokens than earlier models, and the exact increase depends on the content and workload type.

Anthropic official documentation: Models from Claude 4.7 onwards use a new tokenizer, the same text will produce about 30% more tokens, do not reuse the token count measured by older models.

For the same company and the same piece of text, the token count increases by 30% after the model is updated.

Therefore, the official suggestion is: To know how different your workload's token count is, run the same request on both models to count the tokens, and compare the returned input_tokens values.

Do not use the token count measured on earlier models to estimate costs.

Even between two generations of models from the same vendor, the token counts cannot be reused. Comparing "unit price per million tokens" directly across different vendors is far from standardized.

At the same $5 unit price, the bill differs in four aspects

With the same unit price and the same input, where exactly does the difference in the final bill come from?

The first aspect is the tokenization efficiency mentioned earlier. For the same piece of text, different numbers of tokens are generated, multiplied by the same unit price, the total cost you pay will naturally be different.

The second aspect is caching.

The cached input price of GPT-5.6 Sol is $0.50 per million tokens, which is only one tenth of the standard input price. For workloads with many repeated prefixes, this item alone can completely change the structure of the total bill.

The third aspect is output.

The output price of GPT-5.6 Sol is $30 per million tokens, while Claude Opus 5 starts at $25 per million tokens.

In real agent workflows, the proportion of output tokens in total cost is often higher than that of input tokens.

That means the 34.5% cost saved in the input side may very likely be offset by the extra output cost.

The fourth aspect, which is the easiest to be ignored, is written on OpenAI's own model page. When the input of GPT-5.6 Sol exceeds 272K tokens, the input of the entire request is charged at twice the standard rate, and the output is charged at 1.5 times the standard rate.

GPT-5.6 Sol official model page: Input at $5, cached input at $0.50, output at $30, the small note below states the surcharge rule for content exceeding 272K tokens.

It is not the excess part that is charged at a higher rate, but the entire request is subject to the higher multiplier.

For the same piece of code, if you query it in a 270,000-token context versus a 280,000-token context, the unit price will jump to a higher tier.

This restriction comes from the official pricing page. The longer the context, the faster the overhead of attention and video memory increases. The long context window is never free.

With the million-token window enabled, your money is slowly draining away

Tibo then posted a second post teaching people how to manually open the full context window in Codex.

Open ~/.codex/config.toml, add the following three lines before all section headers:

model = "gpt-5.6-sol"

model_context_window = 1000000

model_auto_compact_token_limit = 900000

The first line selects the model, the second line sets the context budget to 1 million tokens, and the third line makes the automatic compression trigger around 900,000 tokens to leave a small margin.

Save the file, restart the client, and start a new session for the configuration to take effect.

If you don't want to modify the default values, you can also temporarily override them only in a single CLI session:

codex -m gpt-5.6-sol

-c model_context_window=1000000

-c model_auto_compact_token_limit=900000

These two parameters can be found in the official Codex configuration reference, and their functions are exactly as described by Tibo.

model_context_window: The number of tokens in the usable context window of the current model.

model_auto_compact_token_limit: The threshold that triggers automatic history compression.

But the documentation only defines the meaning of the parameters, and does not list the values "1,000,000 / 900,000" as a general recommendation.

Tibo also added a note at the end of his post: The default values have been carefully tuned by the team.

Then why do so many people want to modify it manually?

A user's actual test report on GitHub explains the reason.

This test report in the openai/codex repository: The Codex directory limits the window to 372K, with an effective value of 353.4K, while the model specification states 1.05M.

Under specific versions of the Codex client and ChatGPT Pro accounts, the window marked for gpt-5.6-sol in the model directory is 372K, converted to 95% availability, the actual usable window is 353.4K, while the official model page claims it is 1.05M.

You paid for a million-token window, but you can only use one third of it.

This report is limited to specific versions and account types, and cannot be regarded as the situation of all users. Tibo's configuration post was published even later.

One more thing to clarify: Changing the configuration to 1 million tokens will not immediately generate a 1 million-token cost. Billing is always based on the actual processing volume.

However, pushing the compression threshold to 900,000 means that a long session will carry an increasingly long history, and every round of request will reprocess this entire history.

The larger the window and the later the compression is triggered, the more likely the request will hit the 272K threshold mentioned earlier.

In short conversations, the minor difference caused by different tokenizers is negligible. But when the session extends to hundreds of thousands of tokens, the history is carried forward repeatedly, multiplied by a higher billing multiplier, the negligible decimal difference will become a significant integer-level difference.

Money is not spent all at once, it accumulates round by round.

The next billing unit is "per successful outcome"

There is another sentence in Tibo's post that is overshadowed by all the figures: What really matters is the price per successful outcome.

He also provided a method. Benchmark tests can be used as a starting point, but to know if the service is expensive, you have to run your own real workload through it.

This sentence changes the anchor point of price comparison: from "how much does one million tokens cost" to "how much does it cost to complete the same task".

To figure out which vendor is cheaper for your specific use case, just run a test yourself.

Use the same original text, the same language ratio, and the same tool definition, call the official token counting interfaces of both vendors respectively to get the real token count, then take into account cache hit rate, output length, inference length and long context billing multiplier, and finally compare which vendor costs less to complete the task.

Tokenization efficiency is only the first link in this chain. A model with more efficient tokenization may still end up with a higher total bill if it produces verbose inference results and requires many reworks.

In the future, the question we should ask is no longer "how much does one million tokens cost", but "how much does it cost to fix this bug".

References:

https://x.com/thsottiaux/status/2089082893804896524?s=20

https://x.com/thsottiaux/status/2088866513008873560?s=20 https://github.com/openai/codex/issues/31860

This article is from the WeChat official account "AI Era", Author: ASI Revelation, published with authorization from 36Kr.