Prompt Caching stores the result of a prompt’s tokenization and model processing so that identical or similar requests can reuse the work, cutting latency and cost.
01What it is and how it works
When you send a prompt to a model, the service first tokenizes the text, then runs the tokens through the neural network to produce embeddings and a response. Prompt Caching captures the tokenized representation and the intermediate activations for a given prompt. On a later request that uses the same prompt (or a prompt that matches a cached entry), the service skips the tokenization and forward pass, pulling the cached result instead. The cache is identified by a hash of the prompt text and optional parameters, so only exact matches are reused. This reduces the number of compute cycles the model must perform, which directly lowers the per‑request token cost and shortens response time.
It saves the work a model does on a prompt so you can reuse it later.
02What to do about it
- Identify high‑frequency prompts in your workflow, such as product descriptions or FAQ queries.
- Enable caching in the API call by adding the
cache_controlflag as described in the OpenAI docs. - Group similar prompts under a single template so they can share the same cache entry.
- Monitor your usage dashboard for the "cached tokens" metric and adjust your cache‑eligible prompts accordingly.
03How it is measured or noticed
OpenAI’s usage reports include a line for "cached prompt tokens" and "cache hits". A rising cache‑hit ratio indicates that the same prompts are being reused. You can also look at latency: cached calls typically return in under 200 ms, whereas uncached calls may take 500 ms or more. If you see a sudden drop in cost without a change in request volume, it often means caching is now active.
04Common mistakes
- Assuming similar but not identical prompts will hit the cache – only exact text matches are stored.
- Turning caching on for prompts that contain user‑specific data, which can leak information across sessions.
- Forgetting to set the same
temperatureand other generation parameters; a change creates a new cache key. - Relying on caching for one‑off queries – the overhead of storing the entry may outweigh any benefit.
05Limits
Prompt Caching works only with models that support the feature (e.g., GPT‑4o and later). It does not apply to streaming responses, because each token is generated in real time. The cache size is bounded by the service; older entries are evicted based on least‑recently‑used policy. Caching is also distinct from response caching – it saves the prompt processing step, not the final generated text.
06Worked example
`curl https://api.openai.com/v1/chat/completions \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "Give me a 2‑sentence brand tagline."}],
"cache_control": {"type": "ephemeral"}
}'`
The first call tokenizes and processes the tagline prompt. A second call with the identical JSON payload will return almost instantly, and the usage report will show the tokens counted under "cached prompt tokens".
Frequently asked questions
How is prompt caching different from response caching?
It is different because prompt caching stores the tokenized input and the model's internal processing, not the final generated response. Response caching saves the output text, while prompt caching lets the model skip tokenization and inference for identical prompts.
Should I enable prompt caching for all of my API calls?
It depends on the models you use and how similar your requests are. Enable it when you frequently send identical prompts to supported models, otherwise it adds little benefit.
How does the service decide that two prompts are similar enough to reuse a cached result?
It matches the exact token sequence produced by the tokenizer; only prompts that tokenize to the same series of tokens count as a hit. Slight changes, even whitespace, create a different token stream and bypass the cache.
Does prompt caching still work if I add a trailing space to my prompt?
No, adding a trailing space changes the tokenization, so the request will not match a cached entry. The cache only hits on identical token sequences.
What happens if I assume caching is happening but it isn’t?
You will notice higher latency and higher token usage because the model processes each request from scratch. Your usage reports will show zero "cached prompt tokens" and no cache hits.
When will cache hits appear in my usage reports?
They appear immediately after each request and are visible in the next usage‑report refresh. You can monitor the "cache hits" line to see the effect in real time.
Which models support prompt caching?
Currently only models that explicitly advertise the feature, such as GPT‑4o and later, support prompt caching. Older models will ignore the cache‑related headers and process every prompt normally.
Asked out loud
spoken, not typedThe same term in the words somebody uses speaking to an assistant rather than typing into a box — written from the situation, which is why each one carries the situation it came from.
Yes, it can speed things up if you are sending the same prompt repeatedly, because the model can reuse the tokenized input and skip re‑processing. Make sure you are using a model that supports the feature.
Usually the platform handles caching automatically, but you can reduce latency by grouping identical queries together so the service can hit the cache. If the model doesn't support caching, you’ll need to wait for the response each time.
You can disable prompt caching for a specific request by setting the appropriate header or by slightly changing the prompt text so it tokenizes differently. This forces the model to process the input fresh and avoids unintended reuse.