AI Prompt Caching: How to Measure Savings Instead of Guessing

One handbook, fifteen questions, one reusable prefix

First, measure the shared material with the provider's token counter instead of estimating from page count. In this scenario, suppose the instructions, handbook, examples, and response schema total 24,000 input tokens. The individual questions average 120 tokens. The task is not to cache the answers. It is to avoid processing the same 24,000-token opening from scratch for every question when the API supports reuse.

Freeze a version of that opening before the test. Give it a simple identifier such as biology-handbook-v3 and store the exact bytes, message order, image settings, tool definitions, and schema used to build the request. A changed space may not matter on every platform, but changed content, ordering, tools, or media settings can change tokenization or break an exact prefix match.

Keep the output task narrow: answer from the handbook, cite the relevant section label, identify uncertainty, and write one follow-up question for the learner. That makes it possible to compare caching behavior without confusing it with a change in answer quality.

Caching reuses prompt work, not the answer

OpenAI's prompt-caching guide says cache hits depend on matching prompt prefixes. Static instructions and examples should come first; variable user information should come last. The guide also makes an important distinction: the model still generates a new response. A cache hit does not guarantee identical wording, because cached prompt processing is different from reusing a saved completion.

Google gives similar layout advice for Gemini implicit caching: place large common content at the beginning and send requests with similar prefixes close together. Anthropic exposes cache controls and breakpoints, but the reusable unit is still a stable block of prompt context rather than a finished answer.

This distinction matters for learning software. A cached handbook can support a fresh explanation for each student question. If the application needs an identical answer for an identical question, ordinary application-level response caching may be the better tool. Prompt caching is for repeated context that the model must still reason over.

Put stable context ahead of the question

A weak request layout begins with a timestamp, student ID, or today's question, then appends the handbook. Because the opening changes, the provider may not see a reusable prefix even though most of the request is identical.

A stronger layout begins with the fixed tutor rules, the unchanged handbook, the worked examples, and the response schema. The student-specific question, permitted preferences, and request timestamp come after that shared block. On APIs with explicit breakpoints or cache keys, place them according to the current provider instructions rather than copying another provider's syntax.

The improvement is structural, not rhetorical. The same educational guardrails and source material are present in both versions; the improved request simply separates stable context from changing context so the service has something consistent to reuse.

• Weak layout: changing student data -> changing question -> shared handbook -> output rules.
• Improved layout: fixed tutor rules -> shared handbook -> fixed examples and schema -> changing question.
• Expected evidence: later requests report cache reads for much of the shared prefix.

Read the usage object like a receipt

Do not use a faster response as proof of a cache hit. Network conditions, model load, output length, and routing can all change latency. Use the response's token fields. OpenAI reports cache reads in cached_tokens and, for GPT-5.6 and later families, writes in cache_write_tokens. Anthropic separates cache_creation_input_tokens, cache_read_input_tokens, and uncached input_tokens. Google's Interactions API reports cache hits in usage.total_cached_tokens.

The field names are different, but the questions are the same: how many tokens were written, how many were later read, how many remained uncached, and how many output tokens were generated? Keep the raw usage object with the request ID, model name, prompt version, timestamp, and elapsed time. A dashboard total alone is too coarse for diagnosing one prefix.

For the handbook example, calculate a simple cache-read ratio for requests after the first: cached input tokens divided by the measured shared-prefix tokens. The ratio will not always be 100 percent because providers use minimums, increments, breakpoints, and their own matching rules. What matters is that the observed read volume is large, repeatable, and consistent with the prefix you intended to reuse.

Run two lanes and compare the receipts

Create two five-request lanes with the same model, output limit, handbook, questions, and timing. In Lane A, keep the 24,000-token prefix byte-for-byte stable and put each question at the end. In Lane B, deliberately place a changing timestamp near the beginning or reorder one fixed block on every request. Lane B is a diagnostic control, not a recommended production design.

Run the first request in each lane as a likely write or miss, then send the remaining requests at a steady interval allowed by the provider. Record total input, cache-write input, cache-read input, output tokens, time to first token when available, and total latency. Re-run once so a single routing event does not decide the result.

Compare total cost using the live price page for the exact model and processing tier. Do not multiply every input token by one rate: some services charge cache writes, cache reads, storage, long-context requests, batch jobs, or regional processing differently. The test passes only when Lane A shows repeatable read volume and lower measured cost for the complete sequence, not merely a promising first hit.

• Same task and model in both lanes.
• One stable prefix lane and one intentionally changing control lane.
• Raw usage and latency captured for every request.
• Cost calculated from the current rate card after the run.

Five misses that masquerade as a cache problem

The request may be below the minimum cacheable length. OpenAI documents a strict 1,024-token minimum prefix for GPT-5.6 and later families, while earlier families can vary from 1,024 to 2,048 tokens. Google's July 2026 Interactions API documentation lists model-specific minimums of 2,048 or 4,096 tokens for the models shown there. Check the exact model rather than generalizing from one product.

The prefix may drift. A current date, random identifier, reordered tool list, modified image detail setting, or changed structured-output schema near the front can turn a visually similar request into a cache miss. Log a hash of the intended stable prefix so you can distinguish an application change from a provider-side miss.

The entry may have expired or been evicted. Anthropic documents five-minute and one-hour cache durations with different write multipliers. OpenAI documents model-family-specific retention behavior, and Google distinguishes implicit caching from explicit cached-content workflows. A test that waits too long between requests may be measuring cold starts rather than reuse.

A cache hit may still be uneconomic. Current OpenAI and Anthropic documentation charges more than the base input rate for certain cache writes, while reads are cheaper. If a large prefix is written once and never read, caching can add cost. Break-even depends on the provider, model, TTL, storage terms, and number of successful reads.

Finally, caching is not a privacy control. A cache can change how prompt computation is reused without changing the sensitivity of the content. Review current retention, data residency, and zero-data-retention rules for the endpoint and cache mode before sending private material.

Set a decision gate before production traffic

Write the acceptance rule before viewing the totals. For this study app, an example rule is: adopt prompt caching only if the stable lane produces cache reads on at least four of the final five requests, the median cache-read ratio clears a chosen threshold such as 70 percent of the shared prefix, answer quality remains unchanged on a fixed rubric, and total measured cost is lower than the uncached control.

That 70 percent figure is a local engineering threshold, not a provider promise. A smaller shared prefix or a very cheap model may justify a different gate. A latency-sensitive tutor may also require a time-to-first-token improvement, while a batch workflow may care only about cost.

Keep the experiment with the prompt version and repeat it after a model migration, API change, large handbook revision, tool-schema change, or price update. As of August 5, 2026, the three providers documented materially different controls and accounting fields. A working cache strategy is therefore a measured property of one workload, not a permanent feature of the prompt.

The useful habit is to inspect the receipt

Prompt caching rewards a simple design habit: stable information first, changing information last. But layout alone is not evidence. The usage record must show that the intended prefix was read from cache, and the full sequence must cost less or respond faster under the metric that matters.

Start with one repeated document and a handful of questions. Freeze the prefix, keep a deliberately unstable control lane, record every usage field, and calculate the result from the current rate card. If the cache does not win, the experiment has still taught you something useful about request size, prefix drift, timing, or provider behavior.

The goal is not to maximize cached tokens. It is to make repeated AI work observable enough that a savings claim can survive inspection.

Continue learning on JoyfulGrid

Frequently asked questions

Sources

1. Prompt cachingOpenAI API Documentation

   Used for current prefix-matching behavior, GPT-5.6 cache writes and reads, minimum lengths, breakpoints, usage fields, retention notes, and the distinction between cached input and newly generated output.

2. Data controls in the OpenAI platformOpenAI API Documentation

   Used for the warning that cache modes can interact with application-state retention, data residency, and Zero Data Retention eligibility.

3. Prompt cachingClaude Platform Documentation

   Used for cache breakpoints, five-minute and one-hour durations, write and read multipliers, usage accounting fields, and cache-performance tracking.

4. PricingClaude Platform Documentation

   Used to confirm that prompt-cache writes and reads are priced differently and that other processing modifiers can stack with caching.

5. Context cachingGoogle AI for Developers

   Used for Gemini Interactions API implicit-caching behavior, model-specific minimums, stable-prefix guidance, and total_cached_tokens measurement.

6. Context caching for the Generate Content APIGoogle AI for Developers

   Used for explicit cached-content use cases, token and storage billing factors, TTL considerations, and usage metadata.

7. Gemini Developer API pricingGoogle AI for Developers

   Used for the instruction to calculate savings from the current model, processing tier, cache-read, storage, and tool prices rather than a stale copied rate.