Long context removes the retrieval step, not the work around it. Run the same labeled questions through both designs on the same model, log accuracy, cached and uncached input tokens and time to first token, and write down the corpus size, update rate and permission model where your numbers flip. Filter documents by user permission before the prompt in either design.
What Changed, and What Did Not
As of 2026-10-15, Anthropic's context-window page lists a 1M-token window as the default for its current models, without a beta header, billed at standard pricing except on Claude Haiku 5.5, where prompts over 100,000 tokens cost more. In September 2024, Anthropic's Contextual Retrieval post suggested that a knowledge base under 200,000 tokens, about 500 pages, could go straight into the prompt with no need for RAG. Neither statement is a threshold for your product. The window size tells you what fits, not what works.
The same documentation page also says that more context is not automatically better: as token count grows, accuracy and recall degrade, which Anthropic calls context rot. It adds that cached prefixes still occupy the window, so caching changes what you pay for those tokens, not whether they count. Fit, accuracy and cost are three separate questions.
What the Research Measured
Three papers frame the trade-off. Each tested specific models on specific benchmarks, mostly from 2023 and 2024, and none of them tested the models you can call today. Read them as reasons to measure, not as predictions.
- Position matters. Lost in the Middle placed one answer-bearing document among 10, 20 or 30 retrieved distractors for NaturalQuestions-Open questions, using models such as GPT-3.5-Turbo, Claude-1.3, MPT-30B-Instruct and LongChat-13B. Accuracy was often highest when the relevant document came first or last and dropped when it sat in the middle. In the 20- and 30-document settings, GPT-3.5-Turbo's worst middle position scored below its closed-book score of 56.1%, where it saw no documents at all. Extended-context versions performed nearly the same as their shorter siblings when the input fit both.
- Claimed length is not usable length. RULER evaluated 17 long-context models on 13 synthetic tasks covering retrieval, multi-hop tracing, aggregation and question answering. Nearly all models passed the simple needle-in-a-haystack test, yet while almost all claimed 32K tokens or more, only half kept satisfactory performance at 32K. The paper sets "satisfactory" as the score Llama2-7B reached at 4K (85.6%). Under that bar, its 2024 table lists GPT-4 at 128K claimed and 64K effective, and Yi-34B at 200K claimed and 32K effective.
- Retrieval and long context combine. Retrieval meets Long Context LLMs tested a 43B GPT model and Llama2-70B on nine long-context tasks. A 4K-window model with simple retrieval performed comparably to a 16K model extended with positional interpolation, with much less computation. Retrieval improved results regardless of window size, and retrieval-augmented Llama2-70B with a 32K window beat the same model without retrieval on average.
One more detail from Lost in the Middle is useful for RAG design: in its open-domain QA case study, going from 20 to 50 retrieved documents improved accuracy by about 1.5% for GPT-3.5-Turbo and about 1% for Claude-1.3. Reader performance flattened long before retriever recall did. More context helped less than you might expect, in that setup.
Cost Per Query, Including the Cache
At scale, the long-context bill depends on how much of the corpus is billed at the cache-read rate. Anthropic's prompt caching documentation, as of 2026-10-15, prices a 5-minute cache write at 1.25 times the base input price, a 1-hour write at 2 times, and a cache read at 0.1 times, with lower read multipliers for some current models listed in a footnote. That gives a per-query input cost of:
input_cost = base_input_price
* (input_tokens
+ write_multiplier * cache_creation_input_tokens
+ read_multiplier * cache_read_input_tokens)Use the base input price per token: divide a price quoted per million tokens by 1,000,000 before applying this formula. Take the three token counts from the response's usage field and both multipliers from the pricing page on the day you run the test. Output tokens are priced separately; log them too, but compare the input side on its own, because that is where the two designs differ by construction.
The cache has conditions that decide whether the formula works in your favor:
- Exact prefix match. A hit requires identical content up to the cache breakpoint, in the order tools, system, messages. Put the corpus before the question. One edited document invalidates the cached prefix from that document onward.
- Lifetime. The default lifetime is 5 minutes, refreshed at no additional cost each time the entry is read. A feature that gets one question every twenty minutes pays the write multiplier on most requests unless you choose the 1-hour lifetime.
- Minimum size. Each model has a minimum cacheable length; shorter prompts are processed without caching and no error is returned.
OpenAI's prompt caching guide describes the same shape with different mechanics: caching is on by default for supported models, the minimum is 1,024 tokens for GPT-5.6 and later, the whole rendered prefix has to match, and cached tokens appear in usage.input_tokens_details.cached_tokens. Pricing varies by model, so take the cached rate from OpenAI's pricing page rather than from an article. Prompt Cache Hit Rate shows how to find out why your hits are lower than planned.
The RAG side of the ledger is the retrieved chunks, the system prompt, the question, plus embedding, search and any reranking per query. Retrieved chunks usually differ from one question to the next, so a prefix cache helps them little. If you rerank, RAG Reranking covers how to cost that stage.
Latency and Freshness
Anthropic says caching generally improves time to first token for long documents and has no effect on output generation. The cold request is the one to watch: on a cache miss the model processes the entire corpus before it answers. Measure time to first token separately for cold and warm requests, and report the 95th percentile, not just the median, because your miss rate decides how often users see the cold number.
Freshness interacts with request frequency. If the corpus changes between almost every question, there is little reuse of the full prefix. If hundreds of questions arrive between edits, frequent updates can still leave many cache hits. Even an unchanged corpus needs requests within the cache lifetime to stay warm. For RAG, re-index changed documents and verify that the index has caught up before treating the next answer as current.
Access Control Belongs Before the Prompt
Permission filtering is not a RAG feature. In both designs, the documents a user may not read must not reach the prompt. Telling the model to ignore restricted documents is an instruction, not a control.
In a long-context design, that means building the prompt from the user's permitted subset. Users with identical permissions can share a cached prefix; users with different permissions need different prefixes, which multiplies cache writes. Anthropic isolates caches between organizations and, on the Claude API, between workspaces, but separating your own tenants and roles is your job. In a RAG design, the filter goes into the retrieval query itself, as shown in PostgreSQL Row-Level Security for Multi-Tenant RAG. If every user sees a different slice of the corpus, that weighs toward retrieval even when the full corpus fits the window.
Citations are a similar requirement in both designs. Give every document or chunk a stable identifier in the prompt, ask for identifiers in the answer, and check that each cited identifier actually contains the supporting text.
The Side-by-Side Test
Use the same model, the same answer format and the same grader for both setups, so the context strategy is the only difference.
- Build the question set. Write questions from real user queries, each with an expected answer and the identifier of the document that supports it. Include questions whose answer sits deep in the corpus, and a few with no answer in it. The AI Evals guide covers labeling.
- Long-context run. Concatenate the user's permitted documents in a fixed order, mark the cache breakpoint after them, and append the question. For each question, let the cache entry expire before the cold run, then repeat immediately for the warm run. Confirm the actual cache-read and cache-write token counts in usage; changing only the question does not make the shared corpus prefix cold.
- RAG run. Retrieve with your production settings, apply the same permission filter, and send the chunks with the same instructions.
- Position check. For a handful of questions, move the supporting document to the start, middle and end of the long-context prompt. A large spread tells you position still matters for your model and corpus.
- Record per query: correct answer, correct citation, the three input token counts, output tokens, computed cost, and time to first token.
Write Down Your Decision Rule
Fill this table from your own results. The rows are the variables that changed the outcome in the sections above; the thresholds are yours to measure, not universal numbers.
| Variable | Measure | Points to long context when | Points to RAG when |
|---|---|---|---|
| Corpus size | Permitted tokens per user, and accuracy at that size | Accuracy matches RAG at your current size and the position spread is small | Accuracy drops as you add documents, or the corpus approaches the window |
| Update rate | Edits per hour against cache lifetime | The prefix survives long enough to be read many times | Most requests pay the cache-write rate |
| Access control | Distinct permission sets | Few sets, so few cached prefixes | Per-user slices that each need a separate prefix |
| Citations | Correct-citation rate | Citations hold up across the whole prompt | Chunk-level citations are more precise |
| Cost and latency | Cost per query and p95 time to first token, cold and warm | Warm hit rate keeps both under budget | Cold requests dominate your traffic |
Re-run the test when you change model, when the corpus grows by a meaningful fraction, or when the provider changes caching terms. If your answers depend on behavior rather than knowledge, RAG vs Fine-Tuning covers that choice; for curating context inside coding assistants, see The True Cost of Context.
- Lost in the Middle: How Language Models Use Long Contexts (Liu et al., TACL 2023): position effect, closed-book comparison, extended-context models and the 20 versus 50 document case study.
- RULER: What's the Real Context Size of Your Long-Context Language Models? (Hsieh et al., COLM 2024): 17 models, 13 tasks, effective context length and the 32K finding.
- Retrieval meets Long Context Large Language Models (Xu et al., ICLR 2024): retrieval with 4K versus 16K windows and retrieval-augmented 32K results.
- Claude Platform Docs: Context windows: window sizes, long-context pricing exception, context rot and usage fields.
- Claude Platform Docs: Prompt caching: cache multipliers, lifetime, prefix matching, minimum length, isolation and latency.
- OpenAI API docs: Prompt caching: default caching, minimum length, prefix rule and usage field.
- Introducing Contextual Retrieval (Anthropic, September 2024): the 200,000-token suggestion.
The cost formula layout, the side-by-side test and the decision table are this article's proposals. No benchmarks, cost or latency measurements were run for it, and the cost formula sketch was not executed.