LLM Context Caching: Cut Repeat Token Costs

Many AI requests repeat a large block of instructions, examples, policies, or reference material. Context caching can reduce the work required to process that repeated prefix, improving cost or latency when a provider and model support it. The benefit is real only when requests reuse an identical eligible segment and the application measures the provider’s actual cache accounting.

Why this decision matters

Caching is easy to misunderstand because it is not the same as storing the final answer. The model still processes the current request and generates a new response. The reusable portion usually has strict ordering, size, lifetime, and model-version rules. Small edits near the beginning may invalidate everything that follows. More importantly, a cache key or shared prefix can become a privacy boundary. Tenant-specific records should never be made reusable across customers merely to improve a hit rate.

A practical workflow

1. Map repeated prefixes. Inspect production request shapes and identify stable instructions, schemas, examples, and public reference text. Keep current user data and frequently changing facts outside the reusable segment.
2. Order content deliberately. Place the most stable shared material first and volatile data later. Canonicalize insignificant formatting only if it does not change meaning or signatures.
3. Define the isolation key. Include provider, model version, policy version, tenant boundary, data classification, and relevant feature settings. Conservative isolation is better than accidental reuse.
4. Measure provider-reported usage. Capture cache creation and cache read tokens when available. Calculate the effective request cost from billed categories rather than assuming every repeated token received a discount.
5. Plan expiry and invalidation. Change a version when policy, examples, permissions, or source material changes. Test how the application behaves after a cold start and when the cache service is unavailable.

Work through a realistic example

A coding assistant sends a 25,000-token public style guide and tool schema with each request. User code and the specific task change each time. The team places the stable guide and schema first, attaches a policy version, and keeps repository content later. It tests one cold request followed by several identical-prefix requests and reads the billing fields. When the tool schema changes, the version changes too. A separate cache namespace is used for any private organization instructions, preventing one customer from benefiting from or inferring another customer’s prefix.

What to measure and record

Use hit rate, eligible tokens, read tokens, creation tokens, end-to-end latency, time to first token, and effective cost per successful task. Segment by model and request class. A high hit rate can still produce little savings if the reusable prefix is small or output generation dominates the bill. Track cold-start behavior because a short cache lifetime or widely distributed traffic may reduce reuse. Compare response quality before and after restructuring; moving examples or instructions for cacheability can change model behavior even if the text is unchanged.

Common traps

• Caching mutable facts: A policy, price, inventory count, or security rule can become stale during the cache lifetime.
• Cross-tenant prefixes: Combining private instructions for efficiency creates confidentiality and deletion problems.
• Assuming a visual match: Whitespace, ordering, hidden parameters, and model versions may affect eligibility. Verify with usage data.
• Optimizing tokens but not outcomes: A cheaper request that causes more retries or poorer results can raise total cost per completed task.

Review questions

• Which prefix bytes are identical across qualifying requests?
• What private data appears before the cache boundary?
• How quickly must changed policy take effect?
• Do billing fields prove cache reads on the chosen model?
• What happens when a request is routed to another region or version?

A 30-day implementation plan

Begin with one bounded case and an owner who can make a decision. The first milestone is map repeated prefixes. Write down the current state, the intended result, and the evidence that will count as complete. Keep the initial scope small enough to review in one working session, but realistic enough to expose operational friction.

During the second week, run the workflow with a colleague who did not design it. Ask them to answer: “Which prefix bytes are identical across qualifying requests?” Record where they need undocumented knowledge, which data is unavailable, and which step depends on a person or system that has no backup. Fix those gaps before increasing volume or authority.

By the end of the month, repeat the process under a failure condition related to caching mutable facts. Compare the observed result with the original acceptance criteria, assign unresolved actions, and set the next review date. Preserve the decision record beside the operational documentation. A modest control that is used, measured, and improved is more valuable than an ambitious design that exists only in a policy file.

Put the result into routine operations

Create a small dashboard that shows cost per successful task with and without cache reads. Alert on sudden hit-rate drops after deployments. Document which source owns each cached block and how a change triggers invalidation. Keep deletion obligations in mind when private content is cached by a provider. Test a disabled-cache mode so an outage does not stop essential work. If the prefix changes on nearly every request, simplify the prompt or use retrieval rather than forcing a caching design that will rarely qualify.

Related Duck Cloud reading

Pair cache accounting with AI Agent Cost Controls.

Conclusion

Context caching works best for large, stable, safely shareable prefixes. Structure the request around that reality, isolate private material, version changes, and use actual billing evidence. The goal is not the highest hit rate; it is a lower cost for correct completed work without weakening data boundaries.