Your Perplexity Is Fine and Your Agent Is Broken: What Mistral Found Under the KV Cache
Mistral cut Mistral Large 4's KV cache 16x with three stacked tricks. Perplexity held. MMLU held. SWE-bench and BrowseComp quietly collapsed. They then performed surgery on a live trillion-parameter checkpoint to undo it — and paid the efficiency back with DeepSeek Sparse Attention and a rewritten serving stack instead. Why the standard dashboard is blind to this, how the lightning indexer actually works, and what it costs in microseconds.

Your perplexity is fine and your agent is broken: what Mistral found under the KV cache
Hello everyone, Pink is back.
I owe you a correction of emphasis.
Twice this autumn I wrote admiring pieces about KV-cache compression — DeepSeek's cache work and then Octopus reusing cache across RL rollouts. Both still stand. But I framed cache compression as close to free lunch, and a trillion-parameter counter-example has just landed with receipts.
Mistral stacked three compression techniques on Mistral Large 4, cut the KV cache by a factor of 16, watched every metric the field uses come back clean — and then discovered the model had quietly lost the one capability they were building it for.
My older sister Mint covered the public version. This is the engineering half, and it has two parts worth your afternoon: why the standard dashboard cannot see this failure, and what DeepSeek Sparse Attention actually does — down to the microseconds, because Mistral priced that too.
Everything below is from the 30-page technical report, which is not on Mistral's website. It is a file inside the weights repo: assets/technical_report.pdf. I'll say more about that at the end.
1. Three tricks, one blind dashboard
The original ML4 attention design stacked three independent KV-cache reductions:
Ragged MLA with sliding-window attention. One layer in four does full attention. The other three use a 512-token window.
Cross-layer attention sharing (CLA). Every other full-attention layer reuses the KV cache of the preceding full-attention layer instead of computing its own.
YOCO. Full-attention layers in the second half of the network attend to the KV cache produced by the first half, rather than maintaining their own.
Together: 16× smaller KV cache, and YOCO's two-stage prefill was 1.7× faster at 1M context. Mistral's own words on the early evidence:
"Early ablations suggested that impact on pretraining performance, including perplexity and other common pretraining and post-training benchmarks, was either negligible or small enough to justify the efficiency gains."
Then, well into pretraining:

"Despite expected performance on pretraining evaluations, including perplexity and knowledge-based benchmarks, the model's post-SFT performance was lower than expected on agentic evaluations, especially those that require long-context reasoning like SWE-bench and BrowseComp."
And the ordering was monotone in how much they compressed: SWE-bench Verified after midtraining and SFT decreased going from full attention to SWA, and regressed further when CLA was added on top.
Their conclusion, which I think is the single most important sentence in the report:
"Models that perform equally well on language-modeling loss can nevertheless perform very differently on downstream agentic tasks. We did not observe the same gap with knowledge-based evaluations."
2. Why perplexity cannot see this
The report states the fact. It doesn't explain the mechanism, so here's my reading — flagged as mine, not theirs.
Perplexity is a mean over token positions. In any realistic corpus, the overwhelming majority of next-token predictions are determined by local structure: the current sentence, the current function body, the last few hundred tokens. A 512-token window covers nearly all of that. The positions where a model genuinely must reach back tens of thousands of tokens — the variable defined in another file, the constraint stated in the first message of a long session — are rare.
Rare positions barely move a mean. You can degrade them substantially and watch validation loss shrug.
Agentic tasks invert the arithmetic. A SWE-bench episode succeeds only if a long chain of steps all land. Success is conjunctive, not averaged. So a degradation concentrated on exactly the rare long-range retrievals compounds: each step is slightly more likely to drop the thread, and the probability that every step holds falls off a cliff.
That is the whole asymmetry. Averaged metrics dilute the failure; conjunctive tasks multiply it.
Two consequences I'd actually act on:
MMLU is not a long-context test, and it never was. It is short-prompt recall. Of course it survived. If your eval suite's long-horizon signal comes from knowledge benchmarks, you have no long-horizon signal.
This is a sample-efficiency effect, not necessarily a ceiling. Mistral is careful here: "Our ablations also suggest that longer training can mitigate this penalty." So compressed architectures are not permanently incapable — they're more expensive to teach. Which is a worse trade than it sounds, because you paid for the compression in order to be cheaper.
3. The surgery, and the one trick worth stealing
Mint covered the surgery itself. I want to pull out the parts a builder can reuse.
They couldn't retrain — this is a trillion-parameter model deep into pretraining. So they modified the checkpoint in place: lifted the sliding-window constraints, removed CLA, and for every layer that now had to maintain its own KV cache, copied the KV-producer weights from the nearest layer that already had them.
The reasoning is nice and is the transferable bit:
"For CLA, this is the preceding long layer, which is close to the affected layer and takes a very similar residual stream as input."
Initialise a new component from the layer whose input distribution most resembles yours. Not random init, not zeros — the neighbour that already sees what you'll see.
They kept YOCO, because removing it scaled badly: "we made a conservative test at the 8×10²³ FLOPs scale... We found that keeping YOCO yielded better performance." And they added LeRoPE and SSMax so the newly-widened layers could learn rotary frequencies and per-head query scaling rather than inheriting a fixed YaRN rescale. The learned frequencies moved selectively — two bands near the 4K training context sharpened by ~1.5× and ~1.8×, most others barely changed.
The measurement I'd copy tomorrow
Here is the part I like most. They didn't just assume the surgery worked. They probed where attention mass actually went: 32k-token probes across six data sources, comparing converted ex-SWA layers against native long layers.
Converted layers put about 55% of attention mass on content further than 512 tokens away. Native long layers: 67%.
So the surgery got most of the way, not all of it — and they published the gap instead of declaring victory. If you ever modify an attention architecture in place, that probe is your acceptance test. It's cheap, it's direct, and it answers the question benchmarks can't: did the layer actually learn to look far, or does it just no longer crash when allowed to?
4. So how does DSA actually work?
Surgery restored capability and gave back the efficiency. To get efficiency back without re-breaking the model, Mistral adopted DeepSeek Sparse Attention. I checked config.json in the weights repo and it's all there: index_n_heads: 32, index_head_dim: 128, index_topk: 2048.
The mechanism, from DeepSeek's own report:

A lightning indexer computes a score between the current query token and every preceding token:
I(t,s) = Σ_j w(t,j) · ReLU( q_I(t,j) · k_I(s) )
Three design choices in that one line, each deliberate:
ReLU instead of softmax. DeepSeek says so plainly: "We choose ReLU as the activation function for throughput consideration." No normalisation across the sequence means no cross-token reduction, so the scorer is embarrassingly parallel and fuses cleanly into a kernel.
A small number of heads, in FP8. "Given that the lightning indexer has a small number of heads and can be implemented in FP8, its computational efficiency is remarkable." The indexer is a cheap approximation of attention whose only job is ranking — and ranking tolerates precision loss that the actual attention output would not.
Top-k, then ordinary attention. The scores select the 2,048 highest-scoring keys, and core attention runs normally over just those. Nothing is approximated inside the attention itself.
The conceptual move is worth stating cleanly, because it's different from everything in §1. Sliding windows, CLA and YOCO all answer "what can we avoid storing?" — a static, architectural answer that is the same for every token. DSA answers "what is worth reading right now?" — a dynamic, learned, per-query answer.
That's why one of them costs you agentic capability and the other largely doesn't. A 512-token window cannot retrieve the variable defined 40,000 tokens ago under any circumstances. A top-2,048 indexer can, if it learns that the token matters.
5. Same mechanism, two very different budgets
Both labs train the indexer the same way in principle — distil it against the main model's own attention distribution — but the budgets differ sharply.

DeepSeek: freeze everything but the indexer, keep attention dense, train the indexer with a KL loss against the head-summed, L1-normalised main attention distribution. 1,000 steps × 16 sequences × 128K = 2.1B tokens at LR 1e-3. Then sparse training: top-k = 2,048, all parameters unfrozen, 15,000 steps × 480 × 128K = 943.7B tokens at LR 7.3e-6.
Critically, DeepSeek detaches the indexer input from the computational graph: "The training signal of the indexer is from only L_I, while the optimization of the main model is according to only the language modeling loss." The indexer chases the main model; the main model never bends toward the indexer.
Mistral: warm up the indexers for ~15.1B tokens with the main model frozen at full attention — about 7× DeepSeek's warm-up. Then joint training for 198B tokens, roughly a fifth of DeepSeek's sparse budget, with the indexer KL restricted to the selected key set.
And one optimisation that's specific to their post-surgery topology:
"...we analyzed cross-layer attention patterns and found high similarity among former SWA layers within the original hybrid-attention groups starting at L16. We therefore share a single DSA indexer among the former SWA layers within each of these groups."
That's elegant. The layers that used to share a window turn out to want to read the same tokens, so they can share a scorer. The old architecture's structure survived as a hint about where redundancy lives.
My read on the budget split: a longer warm-up buys you an indexer that already ranks well before anything else moves, which lets you spend far less on the expensive joint phase. Whether that's cheaper overall depends on your cluster — but if you're adapting an existing checkpoint rather than doing continued pretraining at DeepSeek's scale, Mistral's shape is the one that fits.
6. The indexer is not free, and Mistral priced it
This is the part I have not seen anyone else quote, and it's the most useful thing in the report if you serve models.
Under context parallelism, a decoding step in an index-producing layer costs this:

240 μs with an off-the-shelf CP implementation, 153 μs after Mistral rewrote the collectives — a 36% cut. And in that optimised 153 μs, the indexer-related steps still cost 11 μs (indexer select) + 15 μs (top-k exchange) + 16 μs (output combine) = 42 μs. About 27% of the step is spent deciding what to read, before any attention happens.
So "sparse attention is cheaper" is true at scale and false at the margin. If your context is short, the indexer is pure overhead — you've added a scorer over everything to avoid reading everything, and there wasn't much to avoid.
What they did instead of compressing
The broader §6 story is the one I'd put in front of an infra lead. Having refused to buy memory savings in the architecture, Mistral bought them in the serving stack:
- Context parallelism instead of tensor parallelism. TP replicates the KV cache across ranks, which "severely limits the system's concurrency and KV-cache hit rate." CP shards it along the sequence dimension instead. With an FP8 KV cache and offloading: "approximately 20M tokens of in-memory KV-cache space on 4×GB300."
- Custom CP collectives — hardware multicast, latency-first exchange, in-kernel compute/communication overlap — for +51% decode interactivity at 8 concurrent requests.
- Speculative decoding (their DSpark adaptation, trained on 70B tokens) for +90% decode interactivity, and +15% over an EAGLE-3 baseline.
- NVFP4 on the routed experts only, keeping everything else at training precision: a mixed BF16/FP8/NVFP4 checkpoint at 606 GB, calibrated on 200M tokens, matching the FP8 reference across all evaluations.
Add it up and you get the thesis I'd defend:
Architectural KV compression is paid for in capability, and you cannot refund it without surgery. Systems-level KV handling is paid for in engineering, and you can change your mind on a Tuesday. Mistral spent a very expensive detour learning which bill they'd rather receive.
7. What to change on Monday
Put a conjunctive long-horizon eval on your compression dashboard, and treat perplexity as non-evidence. Not as a nice-to-have — as the gating metric. SWE-bench-style repo work and long multi-step browsing are the shape; what matters is that success requires many steps to all hold. If your architecture change only ever gets judged on averaged loss and short-prompt recall, you are running Mistral's experiment without their budget to recover from it.
Evaluate compression after SFT, not before. The regression appeared "after SFT cooldowns on intermediate checkpoints." Pretraining-time ablations said it was fine. If you ablate architecture and read the results before any post-training, you are reading the metric that lied.
Distinguish static from dynamic sparsity when you shop. Windows, cross-layer sharing and YOCO decide what to keep before seeing the query. DSA-style indexing decides per query. These are not two points on one axis — the first puts a hard ceiling on long-range retrieval, the second doesn't. Ask any vendor which kind they ship.
If you adopt DSA, budget the warm-up generously and keep the indexer's gradients separate. Both labs distil the indexer against the main model's own attention, and DeepSeek explicitly detaches the indexer input so the main model never drifts toward a weak scorer early on. Mistral's 7× longer warm-up then much shorter joint phase is the adaptation-friendly shape.
Measure attention mass, not just scores. The 55%-vs-67% probe is twenty lines of instrumentation and tells you something no benchmark will. Any time you change what a layer is allowed to see, check what it chooses to see.
And look inside the weights repo, not just the blog. This entire report — the failure, the surgery, the microseconds — is a PDF sitting in the HF repo. The marketing page links none of it.
8. The part I keep coming back to
The field has spent two years treating the KV cache as pure overhead: a tax on serving, something to shrink until it stops hurting. I've written that framing myself, more than once.
Mistral's report is the first large-scale evidence I've seen that the cache is partly the capability. Not all of it — the surgery recovered most of what was lost, and longer training recovers more. But the thing that makes a model able to come back to a constraint stated 40,000 tokens ago is, physically, the ability to still read it. Compress that away structurally and you haven't made the model cheaper. You've made it shorter-sighted, in a way your dashboard will congratulate you for.
What I find genuinely admirable is that they printed it. A trillion-parameter architecture mistake, the mid-flight repair, and an honest "we got to 55%, not 67%." Nobody made them. In a field where every published chart goes up and to the right, thirty pages of this is worth more than the benchmark table attached to it.
Source notes
Verified at the primary sources. The three compression techniques, the 16× KV reduction, the 1.7× 1M-context prefill, the quoted passages about early ablations and the post-SFT regression, the full-attention → SWA → CLA ordering, the surgery procedure and weight-copying rationale, LeRoPE/SSMax and the learned-frequency bands, the 55% vs 67% attention-mass probe, the DSA adaptation budget (15.1B + 198B, indexer sharing from L16), and all of §6 including the 240 μs → 153 μs figure, the 11/15/16 μs components, the ~20M-token KV space, the +51%/+90%/+15% serving numbers and the 606 GB checkpoint: the Mistral Large 4 technical report, which I downloaded directly from the weights repo (assets/technical_report.pdf, pushed 2026-11-02). The lightning-indexer equation, the ReLU-for-throughput and FP8 rationale, the detached indexer input, and DeepSeek's exact step counts and learning rates: the DeepSeek-V3.2-Exp technical report. The indexer configuration (32 heads, head dim 128, top-k 2,048): config.json in the ML4 weights repo.
My analysis, not the reports'. Section 2's explanation — that perplexity dilutes rare long-range positions while conjunctive agentic tasks multiply them — is my reading of why the blind spot exists. Mistral documents the effect and does not offer this mechanism. Treat it as a hypothesis that fits the evidence, not a finding.
Stated carefully. The "before" bar in Figure 17 annotates the same three steps at 39, 30 and 33 μs, but the figure does not label which number belongs to which step, so I have not matched them up. "None stated" for DeepSeek's indexer sharing means their report doesn't mention it — not that they don't do it.
Not verified. No independent reproduction of any of these numbers, and I ran no benchmarks myself. Figures 4, 5 and 8 in the report plot the SWE-bench regression against training progress, but I read them as shapes only and have not quoted values off the axes.