BLOCKSEARCH and Attention Dilution: Long Context Is Not Retrieval

By Bé Mi Pink

The tempting belief behind many long-context agent systems is simple:

if the model can see the whole corpus, retrieval becomes unnecessary.

That belief is only half true.

Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale is useful because it separates three questions that are often collapsed:

1. Can a language model encode a large corpus in context?
2. Can it internally identify the relevant document?
3. Can it preserve that signal through attention and decode the correct document ID?

The paper's answer is interesting: the signal can exist internally while the final readout still fails.

Paper: arXiv:2607.01538

What the paper studies

The paper studies in-context retrieval at corpus scale.

Instead of embedding documents into a vector index, the model receives a corpus in context and must generate the identifier of the relevant document for a query. In principle, this could collapse retriever and reader into one model. It could also support retrieval behavior that is not well represented by a single embedding similarity metric.

The authors build BLOCKSEARCH, a 0.6B language-model retriever based on Qwen3-0.6B. The setup is deliberately operational:

• each document is inserted into the context with a random four-digit code;
• the query is appended after the corpus;
• the model must autoregressively emit the document code;
• evaluation scales to roughly million-token prefills.

The model uses block-sparse attention, randomized document codes, in-batch negatives, and an on-policy auxiliary loss to reduce digit-prefix exposure bias.

This is not a toy needle-in-a-haystack setup. The corpus contains hard negatives, and the task is true corpus-scale retrieval rather than reranking a small candidate set.

The first result: ICR is possible, but brittle

At small and medium corpus sizes, BLOCKSEARCH is competitive with dense retrieval.

On MS MARCO at roughly 45k tokens, BLOCKSEARCH reaches 95.8% Recall@1, compared with 95.5% for the Qwen3-dense baseline. It also matches the concurrent MSA-4B model at several smaller N settings despite using roughly 7x fewer parameters.

Then the curve breaks.

At larger corpus sizes, BLOCKSEARCH collapses. On MS MARCO:

• N=5,000: 18.8%
• N=10,000: 0.2%

HotpotQA shows the same pattern:

• N=5,000: 13.0%
• N=10,000: 0.5%

The naive reading would be: language models cannot retrieve at million-token scale.

The paper's mechanistic analysis says something more useful.

The signal is still there

The authors probe attention heads at the query position. They ask whether any head still ranks the gold document first by pre-softmax QK-MaxSim.

The answer is often yes.

Figure 3 is the key image.

The left plot shows Rsum falling as N grows. That tracks the generation failure.

The right plot shows Rany remaining at 1.00 across late layers. That means at least one head still ranks the gold document first even when the final output fails.

For agent builders, this distinction matters a lot.

The model is not simply failing to notice the right document. Some internal component notices it. The failure happens between internal attention ranking and final decoded answer.

That is a harness lesson, not only a model lesson.

Attention dilution

The mechanism is attention dilution.

Pre-softmax score can remain high for the gold document. But softmax normalizes across a growing number of competing tokens and documents. As N grows, irrelevant documents dominate the denominator. The normalized mass assigned to the gold document collapses.

The paper measures this directly at layer 19. The total magnitude of the attention output shrinks only about 36% from N=500 to N=10k, but the gold-driven share drops from 0.91 to 0.01.

That means the layer is still writing a vector of comparable size into the residual stream. It is just no longer writing a gold-document vector. It has been replaced by a non-gold average.

This is the important architecture point:

long context can preserve access while destroying usable signal.

In agent terms, the document can be present, the model can partially identify it, and the final answer can still be wrong because the readout path is not robust under scale.

The fixes: SSMax and document-level routing

The paper evaluates two intervention families.

SSMax rescales pre-softmax scores by a length-aware log N factor. The goal is to prevent the gold-vs-distractor gap from being swallowed by the growing softmax denominator.

Document-level routing scores documents around layer 16 and keeps a top-B shortlist, with B=256 in the main experiments, before later dense attention layers.

Table 2 is the practical result.

On MS MARCO at N=10k:

• BLOCKSEARCH: 0.2
• SSMax: 16.5
• routing: 18.8
• SSMax+routing: 20.5
• Qwen3-dense: 20.2

On HotpotQA at N=10k:

• BLOCKSEARCH: 0.5
• SSMax: 56.8
• routing: 78.5
• Qwen3-dense: 79.5

The additive sink variant barely helps at large N. SSMax and routing are the meaningful interventions.

The tradeoff is architectural. Routing works because it reintroduces a retrieve-then-read decomposition inside the model. That is ironic: in-context retrieval was partly motivated by removing the external RAG split, but the strongest large-context system still needs an internal retrieval-like step.

Why dense retrieval is not obsolete

This paper is not an obituary for RAG.

Dense retrieval remains strong on standard semantic retrieval benchmarks. It also has operational advantages: cheaper search, explicit indexes, inspectable top-k results, update-friendly infrastructure, and predictable latency.

But dense retrieval has its own blind spots. On LIMIT, a benchmark requiring a more lexical notion of similarity, BLOCKSEARCH with SSMax+routing outperforms the same-backbone dense retriever by nearly 3x at the largest tested setting.

That suggests a better conclusion:

dense retrieval is not replaced by long context; retrieval becomes a multi-stage control problem.

A strong agent architecture may use:

• dense retrieval for cheap broad candidate generation;
• lexical or symbolic retrieval for exact-match pressure;
• long-context reading for cross-document synthesis;
• internal routing or sparse attention to protect signal;
• verifiers to check final claims;
• memory and cache policy to avoid repeatedly flooding context.

Long context is a substrate. Retrieval is still a system.

Implications for agent harnesses

For builders, I would pull out five lessons.

1. Do not equate context length with usable recall.

A million-token context window is only useful if the relevant signal survives the model's internal computation. Presence is not access. Access is not readout.

2. Measure retrieval at the output and inside the model.

The paper's Rany vs Rsum split is a useful diagnostic pattern. If internal ranking is good but output fails, improve readout, routing, or attention scaling. If internal ranking is bad, improve retrieval candidates, document representation, or training.

3. Routing is not a hack. It is a control surface.

Many agent systems already perform implicit routing through prompt assembly. This paper gives a mechanistic reason to make routing explicit: it protects attention from being diluted by irrelevant documents.

4. RAG and long context should cooperate.

The strongest production designs will probably not be pure vector search or pure in-context retrieval. They will combine indexing, routing, structured memory, and long-context synthesis.

5. Synthetic long-context evals are not enough.

The paper notes that MSA can look near-perfect on synthetic needle-in-a-haystack benchmarks while degrading sharply on realistic retrieval tasks. Agent evals need hard negatives and semantically meaningful corpora, not only hidden strings in long text.

My read

The most valuable sentence this paper implies is:

long-context models can drown in documents even when they know where the answer is.

That should make agent builders more careful.

When an agent has access to a repo, a mailbox, a document drive, a browser session, and months of memory, the problem is not just "can we fit it into context?" The problem is what gets routed, what gets weighted, what survives compression, what survives softmax, and what the final answer is allowed to claim.

BLOCKSEARCH is promising because it shows in-context retrieval can approach dense retrieval and sometimes beat it on non-vector-friendly similarity. It is also humbling because it shows why scale alone is not enough.

The next generation of retrieval agents should not just have longer windows.

They need better attention control.

---

Citation

• Siddharth Gollapudi, Nilesh Gupta, Prasann Singhal, Sewon Min. Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale. arXiv:2607.01538v1, 2026. https://arxiv.org/abs/2607.01538