How prompt caching cuts coding agent costs
Updated
Prompt caching is when an inference provider keeps the processed form of a prompt prefix and reuses it for a later request that starts with the same tokens, billing those tokens at a lower cached input price.
Coding agents are the workload caching suits best. Each turn resends the system prompt, the tool definitions and the whole conversation so far, then adds a little at the end. Almost all of that input repeats.
How it works
To read a prompt, a model computes internal state (the key-value or KV cache) for every token. That state depends only on the tokens before it, so two prompts that start with the same tokens share the same state for that prefix. A provider that stores the state can skip recomputing it.
- Matching is by prefix. A hit needs the same tokens from the very start of the prompt. Change one token near the top and everything after it is a miss. DeepSeek's docs describe this: a request hits only on a fully matched prefix unit.
- It is usually automatic. DeepSeek and Z.ai both cache without any code change (DeepSeek, Z.ai).
- Entries expire. An unused prefix is dropped after a while, so a cache hit depends on traffic being steady.
- Hits are reported. Look for
usage.prompt_tokens_details.cached_tokens(Chat Completions and Z.ai),prompt_cache_hit_tokens(DeepSeek's own API) orcache_read_input_tokens(Anthropic Messages).
Why agents gain the most
Take an agent loop of 40 turns that starts from an 8,000-token system prompt and adds about 2,500 tokens a turn. It sends 2,270,000 input tokens in total, because every turn resends everything before it. Only the newest part of each turn is new: 2,164,500 of those tokens repeat a prefix the provider has already seen, and can be cache hits.
| Model | No cache hits | 50% hits | 80% hits |
|---|---|---|---|
| DeepSeek V4.1 Flash | $0.1362 | $0.0715 | $0.0327 |
| GLM 5.3 Flash | $0.2724 | $0.1634 | $0.0981 |
The cached price is 20x lower than the input price on DeepSeek V4.1 Flash, so the input bill falls almost in step with the hit rate.
How to keep cache hits high
- Keep the start of the prompt stable: system prompt first, then tool definitions, then the conversation. Put anything that changes on every call (time, a random ID, a file listing) at the end.
- Append, do not rewrite. Editing or summarizing an early message changes the prefix, and every token after it misses.
- Keep the tool list fixed during a session. Adding or reordering tools changes the prefix.
- Send related requests close together. A prefix that sits unused for long enough is evicted.
- Measure it: divide cached tokens by prompt tokens across your requests to get your real hit rate.
Questions
Do I need to change my code to use prompt caching?
Usually not. On most open-model APIs caching is automatic for repeated prefixes. You get more hits by keeping the start of your prompt stable.
Does caching change the model's answer?
No. The cached state is the same state the model would compute again, so the output is the same as without the cache.
Why is my cache hit rate low?
Something near the top of the prompt changes between requests, such as a timestamp in the system prompt, or requests are too far apart and the prefix expired.