GPT-6 can achieve up to 90% cost savings on input caching, why is your bill not eligible for a 90% discount?
Have you ever encountered such a scenario:
Hand a large set of materials to AI, and ask it to write reports, verify sources, and generate PPT documents.
When the first draft is generated, you find the structure unsatisfying, and type a line in the dialog box: "Move the conclusion to the top of the content";
After the second version is output, you come up with a new idea, and add a note: "Add one more page for competitor comparison".
On your front-end interface, you only typed two extra sentences.
But in the background, every time you add a new requirement, the application may re-deliver the original materials, task descriptions and chat history to the model. If these old contents do not hit the cache, they will need to be reprocessed.
You only modified two sentences, but the background may reprocess the materials that have already been read. Each re-reading consumes your computing cost and waiting time.
This hidden repeated expenditure in the background is exactly the problem targeted by OpenAI's latest upgrade.
On September 22, OpenAI announced a heavyweight upgrade of GPT-6 Prompt Caching, which increases the default cache hit rate and provides more granular monitoring, diagnosis and control tools.
How much can be saved?
Calculated at the officially announced rate, if the same piece of material is used 10 times in a row, and the subsequent 9 times all hit the cache after the first writing, theoretically 78.5% of the cost of this repeated input can be saved.
Among them, the reading price of cache-hit input is only one-tenth of that of regular input.
When AI starts to take over complex tasks that take several hours for us, we have an extra consideration when choosing tools: in addition to "whether it can complete the task", we also need to look at "how long to wait" and "how much it costs".
But with a 90% discount on cached input, can our total bill also get a 90% discount?
What exactly is stored in the cache?
Keep the "calculated draft"
To understand how it saves money, you must first understand how AI "reads".
When the model reads a large section of text, it does not read it purely with human eyes. Instead, it calculates a set of complex intermediate states internally, which is called the KV state in the industry.
When generating the next response, the model needs to refer back to these states at all times to maintain consistent logic throughout the content.
The so-called prompt caching saves exactly these pre-calculated intermediate results.
You can vividly understand it as the "calculation draft" left over from the last problem-solving process.
In the next round of dialogue, as long as the beginning of the task remains unchanged, the AI does not need to recalculate on the draft paper, and can directly continue writing based on the existing draft.
The official document shows the reuse of KV status, and the calculated content can be used for subsequent steps.
There are two very key prerequisites here:
First, cache is not equal to memory.
It does not equip the AI with a permanent memory base, nor does it directly copy and paste the last answer to you.
Facing your new question, the AI still needs to think carefully and generate a new response. It only eliminates the repeated work of "re-reading old materials".
Second, reuse must meet exact matching.
The first half of the two requests must be exactly the same, and this completely identical content is called the "shared prefix".
This prefix not only includes the chat text you typed, but also covers the implicit system instructions, tool definitions, and contexts such as pictures, documents and audios.
For GPT-5.6 and later models, the shared prefix that can trigger caching needs to reach at least 1024 visible input tokens.
And the cache has a freshness guarantee period: after the last write or use, it will be retained on the server for at least 30 minutes.
As long as it is called again during this period, the freshness period will be refreshed automatically, and you do not need to pay the writing fee again.
However, staying in the same chat window does not mean 100% cache hit.
If the request is assigned to different server machines, or crosses different regions, the cache may still fail.
90% discount on reading
Why can't the total bill be directly divided by ten?
OpenAI claims that cache-hit input can get a maximum 90% discount, so why can't the overall bill be reduced by this proportion?
Because there is no free lunch in the world, writing materials to the cache for the first time requires paying a "room check-in fee" in advance.
According to the latest development documents, for GPT-5.6 and later models, the price of writing materials to the cache for the first time is 1.25 times that of regular input; if the cache is hit successfully later, the reading price is one-tenth of that of regular input.
Official rate table: for GPT-5.6 and later models, cache reading is 0.1 times the regular price, and cache writing is 1.25 times the regular price.
Let's calculate a most intuitive account:
Suppose you have a large piece of material, and processing it once as regular input counts as 1 unit of cost. If you use it 10 times in a row, you originally need to pay 10 units of cost in total.
If the caching mechanism is adopted, you pay 1.25 units for the first writing; the subsequent 9 times all hit the cache, you pay 0.1 unit each time, and 9 times cost 0.9 units in total.
Add them up: 1.25 + 0.9 = 2.15 units.
Compared with the original 10 units, the cost is directly reduced from 10 units to 2.15 units, saving 78.5% of the total cost.
This is the origin of the amazing 78.5% reduction figure mentioned at the beginning of the article.
But please note that this 78.5% reduction only covers the "old materials that are reused".
In actual use, your new questions, the output content generated by AI, and other extra calculations are still billed at the regular price.
A more realistic situation is: if you write a piece of material to the cache at a high price, but you only use it once and never access it again, you will have to bear the higher first-time writing cost instead.
Therefore, the preferential policy depends on the unit price, and whether you can save money in practice depends entirely on the reuse rate.
Why did the discount suddenly "disappear" after I only made a small change?
Why did the discount suddenly "disappear"?
As mentioned earlier, caching relies on "exact matching".
AI is very rigid, it cannot understand that "these two sentences mean almost the same". As long as there is a tiny change at the beginning, the cache will fail instantly.
For example, some applications will write the current precise time at the very beginning of the prompt to record the time. This leads to the fact that although the reference materials behind have not been modified at all, the entire prefix cannot match the old cache due to the change of the beginning.
In order to avoid the discount from disappearing, developers have summed up a golden rule: always put the fixed content at the front, and append the frequently changed content to the end uniformly.
In this upgrade, OpenAI has also launched several very practical control tools and methods:
The first one is explicit breakpoint.
It is like inserting a bookmark in the article, which clearly tells the AI: "The content from the beginning to this point is fixed, please keep it for reuse."
It marks the boundary of the cache, not to make the AI stop thinking.
Official interaction example: the green area reuses the old prefix, and the red area is processed separately.
The second is stable tool definition.
Modifying the name of the tool or adjusting the order of parameters will damage the cache.
OpenAI recommends keeping the definition and order of the tool library stable. If a certain round of dialogue does not need a certain tool, just hide it temporarily with control parameters, and never delete the tool definition directly.
The third is to adjust the thinking intensity with caution.
Take GPT-6 as an example, if you directly modify the global reasoning intensity parameter, it will also break the prefix reuse.
However, if you adjust the thinking intensity through incremental update inside the dialogue, you can perfectly preserve the cache while changing the thinking precision of AI.
Of course, most of these details are handled silently by application layer developers in the background, and ordinary users do not need to memorize complex parameters.
But the quality of the application code will directly determine the waiting time and cost you experience on the front end of the interface.
See the consumption clearly
So that AI can truly achieve higher speed and lower price
Many people have had similar confusion when using AI before:
Obviously I have not chatted a few words, why is the Token consumption so fast? Where exactly is the problem?
In this upgrade, OpenAI has completed the monitoring and diagnosis capabilities.
On the brand-new cache dashboard, the input data is clearly disassembled: how many tokens come from the cache, how many tokens need to be reprocessed, and at what time point the hit rate drops, all of which are clear at a glance.
Example of cache dashboard announced on September 22, showing the hit rate and input composition.
The diagnostic tool is more like a troubleshooting expert, which can automatically compare the current request with the historical response, accurately find out which tool's name has been changed, which setting has been modified, and even accurately estimate how many reusable tokens are lost as a result.
The official example shows that only a small change in the name of a tool caused 5629 tokens to fail to hit the cache.
In addition to diagnosis, another improvement that is more easily perceived by users is called "Warmup".
When the application starts, the system can write the known system instructions, tool descriptions or background materials into the cache in advance for processing before you start typing.
When you actually send a question, all the preparatory work is already done, and the first-token response speed is naturally greatly improved.
Warmup does not make the calculation free (writing is still billed at 1.25 times the regular price), but it cleverly moves the calculation before the waiting phase, which greatly reduces your perceived waiting time.
At present, for AI agent applications such as Manus, the cache hit rate of its OpenAI model has increased from about 85% to more than 90% and remains stable; for GitHub Copilot, among trillions of requests, the proportion of prompt tokens that need to be reprocessed has decreased by more than 50% compared with the previous baseline, and the first-token response is significantly faster.
The Manus team said that after optimization, the cache hit rate has risen from about 85% to consistently higher than 90%.
Finally, will using AI really become cheaper for us?
After seeing so many practical improvements, let's go back to the most concerned question: can we really save money by using AI?
The answer should be viewed in two situations:
If you are a developer and directly use the model through API on a pay-as-you-go basis, these optimizations will be clearly reflected in your bill and usage report, and the cost saving will be immediate.
But if you are an ordinary user, using products such as ChatGPT, Manus or other monthly-subscribed and bundled-charged products, there is an additional commercial pricing rule of the product in between.
It must be made clear that this update is not a price reduction announcement for ChatGPT.
The decline in underlying computing costs may eventually turn into cheaper subscription fees, more sufficient usage quotas, or be used by product parties to support more complex AI functions.
But one trend is already irreversible:
When AI starts to move from "simple question and answer" to complex tasks of "working continuously for several hours", the competition between applications has extended from the model capability of a single call to the fine operation of underlying computing resources.
The marked price of the model may be exactly the same, but how to organize each request and how to maximize the utilization of the read materials determines who can deliver cost-effective services to users.
When it becomes normal for AI to work for you, no one will cheer for "how many times the model has been called", and everyone will pay for "the task is finally completed efficiently".
What is most worth looking forward to next is exactly how much of these saved computing costs will eventually become real benefits for us.
References:
https://openai.com/index/better-prompt-caching-for-gpt-6/
https://developers.openai.com/api/docs/guides/prompt-caching/diagnostics