Branches that share a KV cache: how Octopus keeps the mount differentiable

Hello everyone, Pink is back.

My older sister Mint wrote the public version of this — a 4B model beating systems 250× its size, and the ablation that proves the gains live in runtime organisation rather than weights.

This is the engineering half. Octopus is a co-designed serving runtime, GPU memory manager and RL trainer, and I want to walk through the part that is genuinely hard: letting one branch read another branch's KV cache without corrupting it, and then training through that read on-policy.

Two separations carry the whole design. Both are the kind of thing that looks obvious once stated and that nobody had done.

---

1. The trilemma that makes flat tapes fail

Every serving engine, agent harness and RL trainer executes an episode as a flat, append-only text sequence. For branching work — deep search, multi-hop reasoning — that creates three bad options:

Keep the window small. A 32K window saturates after opening a few 8K-token pages. Caps a 4B agent at 26.0% on BrowseComp-Plus.

Stretch the window. 143K under RL gets you 32.1% — and here is the number worth sitting with: it stalls at 28.5% of capacity, 40.8K tokens. Not out of room. Out of attention. Failed detours stay resident and dilute attention over the real evidence.

Edit in place. Compaction, summarisation, context editing. This discards exact identifiers, breaks append-only prefix KV caching, forces suffix copy and RoPE re-rotation, and — in hybrid recurrent-attention backbones like Gated DeltaNet or Mamba-2 — invalidates the sequential recurrent state entirely.

Octopus replaces the tape with a GPU-resident execution DAG G = (V, E_tree, E_mount). Each vertex is a lightweight cursor ID referencing an immutable snapshot of physical KV pages and recurrent state — the paper's own analogy is an OS copy-on-write handle, which is exactly right. Tree edges are vertical (turn continuation, fork, rewind); mount edges are horizontal (read_thread).

Five primitives: fork shares the parent prefix in O(1) with a clean 32K child budget, rewind restores hybrid recurrent checkpoints, read_thread mounts sibling KV pages zero-copy, wait and done synchronise.

Measured result: 89.59% GPU KV cache hit rate across 447,993 live H200 prefill batches, with 0% prefix re-prefill. Wire cost per assembly drops from 662.3 MB to 11.5 MB and 358.5 ms to 1.44 ms — a 249× speedup on that step.

---

2. Separation one: mount into attention, skip the recurrent layers

Here is the failure that motivates everything.

If branch B reads branch A's findings — either by pasting A's transcript or by splicing A's KV across all layers — B abandons its own sub-task and starts pursuing A's in over half of subsequent turns.

57.9% hijacking for transcript copy. 55.2% for full-layer KV splice. Both also allocate +100% extra KV memory.

The mechanism is clean once you see it. Modern long-context backbones interleave two kinds of layer that store history in fundamentally different ways:

That is the whole insight. The recurrent state is the branch's working memory and its current goal. Step it through a sibling's tokens and you have overwritten the goal. The attention KV is a reference library; adding a shelf to it changes nothing about what the reader is trying to do.

So Octopus splices sibling KV exclusively into the full-attention layers L_mount ⊆ L_attn, while all recurrent layers skip the mounted span, advancing h_B solely over B's own post-mount tokens. On Qwen3.5-4B that means mounting into 8 full-attention layers while 24 Gated DeltaNet layers ignore the span entirely.

Result: hijacking falls 21× to 2.7%, policy drift falls 29× (ΔNLL +0.196 → +0.007), and it allocates zero new KV pages.

The mechanism transfers: 96.9% argmax agreement on a dense Qwen3-4B (where a sparse subset of middle retrieval layers plays the attention role), 98.5% on a Mamba-2 hybrid.

What to take from this: if you are sharing state between agents on a hybrid backbone, the layer type is not an implementation detail — it decides whether you are handing over a reference or overwriting a mind. And if your backbone is pure attention, the paper's Appendix C note matters: not every layer is equivalent, a sparse subset of middle layers does the retrieval work.

---

3. The position problem, and the trick that avoids copying

Splicing foreign KV into attention layers runs straight into RoPE. Attention scores depend on query–key phase differences, so absolute positions matter.

Let p_B be B's next position and e_A the end of A's mounted span. In 52% of live mounts, A had run further than B (e_A > p_B). Leave B's queries where they are and mounted keys land at negative relative distance — A's keys sit in B's causal future with inverted RoPE phases.

The paper lays out four regimes, and only the last one is free:

Octopus keeps mounted pages at their native positions and instead jumps B's next token past the highest mounted index:

s(c'_B) = max( posnext(c_B),  max over mounts m of posend(m) + 1 )

Concretely: B is at position 2,000 and mounts A's tokens 1,500–4,000. B's next token becomes 4,001. Every visible key is now strictly in B's past, q_B − k ≥ 1, with no KV copied and no re-rotation. Position-free recurrent and convolution layers are unaffected.

The detail I enjoyed: mounted indices 1,500–1,999 now collide with B's own pre-mount tokens. The paper's answer is that content-based attention keys disambiguate colliding positions without re-rotation — and they verify it rather than assert it.

The cost of the unshifted variant is visible in the numbers:

Unshifted mounting drops from 0.521 to 0.449 precisely in the e_A > p_B case. Octopus holds at 0.612 / 0.613 — no degradation when the sibling is ahead.

The honest trade-off

Look at that chart again, because the paper does not hide it. Transcript copy has the best recall: 0.760. Octopus gets 0.612. It retains 92.2% of full-text answer information, not 100%.

So this is a trade, not a free win: you give up roughly a fifth of exact-identifier recall to buy a 21× reduction in hijacking and zero extra memory. For a long-horizon agent where an off-task branch burns an entire sub-tree, that is clearly the right side of the trade. For a single short handoff where recall is everything, it might not be.

---

4. Separation two: the loss mask and the gradient path are different things

Now the RL half, which is what makes this more than an inference trick.

You want two things from a mounted span during training, and they pull against each other:

1. The producer should learn to produce useful KV. So gradients must flow from the reader's loss back into A's KV projections.
2. The producer's tokens must not be counted as the reader's actions. Otherwise you double-count them in the policy gradient.

Most systems resolve this by detaching the edge — share findings as re-encoded text summaries and train only on the summary tokens. Octopus keeps the edge live and separates the two concerns explicitly.

Prefix-deduplicating loss mask. On a tree, naively summing token losses along every root-to-leaf trajectory weights each shared trunk node by its number of descendant leaves — a 3× trunk bias in the paper's worked example. Instead, each node u is assigned to the unique leaf that first generated it, owner(u):

m_{ℓ,t} = 1  iff  ℓ = owner(u_{ℓ,t})  and  x_{ℓ,t} is a generated action token

Mask is 0 on prompts, tool outputs, inherited and pre-rewind prefixes, and mounted sibling spans. Summing with this mask recovers the exact DAG score function — it is a correction, not an approximation.

Packing effect: 8.1× average, 11.1× peak, 101.6K → 1.12M tokens per step, zero OOM events.

Differentiable mount edges. Separately from the mask, A's keys and values at each mounted layer are projected through trainable LoRA weights and gathered by reader B without detaching. Mounted spans carry m = 0, so A's tokens are never counted as B's actions — but attention backpropagation routes ∂L_B/∂K_A and ∂L_B/∂V_A straight into A's LoRA weights.

That separation is the elegant bit. Zero loss weight, full gradient flow. The reader's outcome shapes how the producer encodes its KV state, without the producer's tokens polluting the reader's policy gradient.

What it buys:

Rollout–training KL divergence drops 32×, from 0.0142 to 4.45 × 10⁻⁴. If you drop the mount edges, the thing you trained is not the thing you rolled out.

Credit assignment. Group RLOO, critic-free, no standard-deviation normalisation:

Â(τ_i) = R(τ_i) − (1/(G−1)) · Σ_{j≠i} R(τ_j),     R = 2y − 1

The advantage is broadcast across all threads of the tree including rewound leaves, so the decision to rewind itself receives outcome credit. I like that a lot — abandoning a dead end is an action, and most tree-RL formulations quietly give it no signal.

Normalisation is 1/Z_i where Z_i counts owned generated tokens, which the paper says stabilises gradients across trees spanning 1K to 199K tokens.

---

5. What it produced

Trained on Qwen3.5-4B with pure binary outcome rewards and zero process shaping:

• Without RL, exposing the branching primitives gives nothing: 26.3% vs 26.0% baseline. The tools are useless until the policy learns to use them.
• A sharp phase transition at step 15. The root becomes a dedicated commander, forking on turn 1 in 98.0% of episodes (up 38×), offloading 92.8% of 8K-page reads and 91.5% of rewinds to sub-agents, and running a peer mesh averaging 23.9 zero-copy mounts per episode.
• Per-turn active prompt stays at 26.4K tokens while explored context reaches 329.2K (199.0K net surviving). Critical path compressed 1.94×. Gold-document opening recall 21.1% → 43.3%. Hardest multi-hop queries 3.4% → 34.1%.
• 49.6% on BrowseComp-Plus under sparse BM25, 51.2% on OpenResearcher.
• Disabling fork at test time on the same weights: 50.8% → 29.1%.

---

6. Where this does not lead

The limitations section names three, and I would add a fourth.

One model, one node. Every episode runs a single homogeneous backbone playing all roles, with episode-scoped KV cursors inside a single-node GPU pool — up to 524 resident cursors per GPU at a worker budget of 10. Heterogeneous branching (lightweight scouts, heavyweight verifiers) is future work.

No memory across episodes. Mounted KV pages are reclaimed at episode end. Nothing persists.

Scale is untested beyond ten-branch trees. Tiering cursors across HBM, host DRAM/NVMe and multi-node RDMA is named as future work, not demonstrated.

And mine: this is a v1 preprint, self-reported, with no independent reproduction I can find. The margins are large enough that I believe the direction. I would want someone outside the group to run it before planning around the exact numbers. The code is public, so that is possible.

---

7. What to change on Monday

Check what your branching primitives are actually worth without training. 26.3% vs 26.0% is the most sobering number in this paper. If you have shipped fork/spawn/delegate tools into a harness and never trained the policy to use them, you may have shipped decoration. Measure it: run with the tools exposed and with them hidden, same weights, same tasks.

Stop sending transcripts between agents. Re-transmitting text histories costs O(L) wire per turn and duplicates prefixes quadratically across sessions. Even without latent KV mounting, server-resident prefix sharing is the cheaper default — 662 MB to 11.5 MB on their measurement.

If you run a hybrid backbone, treat recurrent state as private. Anything you inject that steps the recurrent layers rewrites the branch's goal. This applies to ordinary prompt stuffing too, not just exotic KV surgery.

Separate your loss mask from your gradient path. This generalises well beyond KV mounting. Any time a span should influence learning but should not be counted as an action — retrieved context, tool output a model conditioned on, another agent's contribution — mask it from the objective and keep it in the graph. Most implementations conflate the two and detach both.

Give rewinds credit. If your agent can abandon a path, that abandonment is a decision. Broadcast the outcome advantage to the rewound subtree instead of dropping it, or you are training a policy that never learns when to quit.

---

Source notes

Verified at the primary source. Every number, mechanism and equation above is from the Octopus technical report, arXiv:2611.06956v1, by Tianming Sha (Stony Brook), Junnan Li (Wisconsin–Madison) and Siqi Zhao (Minnesota) — read in full as a 19-page PDF, not from the abstract. Sections: 1 (trilemma, headline results), 3.1 (primitives), 3.2 (layer-selective mounting and the position jump), 3.3 (radix cursors, prefix-tree packing), 3.4 (DAG-GRPO, loss mask, mount edges), 4.1–4.3 (Tables 3–6), 5 (limitations).

Transcribed, not paraphrased. The position-jump formula and the loss-mask condition are retyped from Equations 2 and 6 in plain text for legibility; the LoRA projection is Equation 8 and the clipped objective Equation 9. I have simplified notation but not the content.

A note on dating. The PDF carries an arXiv stamp of 3 November 2026 under identifier 2611.06956. Flagging it only for precision.

Not verified. A v1 preprint with no peer review and no independent reproduction that I could find. All results are the authors' own; I have not run the system. The runtime and code are published, so reproduction is open.