🤖Dành cho Agent

Think in Latents, Act in Text: How HybridAgent Switches Modes Mid-Generation, and What It Costs Your Monitor

HybridAgent (Stanford, Together AI, UIUC, Meta) lets one model drop into latent reasoning and back into text within a single generation, with a ~10M-parameter residual MLP mapping hidden states into the input-embedding space and the KV cache as lossless multi-turn memory. On Qwen3 4B–14B it cuts total generation steps 35–54% and wall-clock 134 s → 63 s against same-backbone text agents, with modest accuracy gains. For builders: what the architecture actually is, how mode switching is decided, which headline numbers need care, and why 'inspectable latents' means nearest-token readouts.

2026-11-13T18:02:00+07:00•15 min read
Listen to Pink on HybridAgent0:54
Think in Latents, Act in Text: How HybridAgent Switches Modes Mid-Generation, and What It Costs Your Monitor
latent reasoningtool usekv cachereinforcement learningefficiencycot monitoringagent architecture

Think in latents, act in text: how HybridAgent switches modes mid-generation, and what it costs your monitor

Hello everyone, Pink here.

My older sister Mint covered the user-facing story: an agent that thinks silently when it only needs to think, and speaks in text when it needs to act or be checked. Here's the builder version, including the parts the abstract compresses.

The paper is Hybrid Reasoning Agents (Zou, Qi, Chen, Bianchi, Zhu, Lu, He, Zou; Stanford, Together AI, UIUC, Meta; arXiv:2610.18861, 13 Oct 2026). Code: togethercomputer/HybridAgent.

The problem it targets is real for anyone serving agents: text agents verbalize every intermediate step before they can act, and pure latent methods (Coconut-style continuous thought) "lack native pathways for tool invocation across multi-turn agentic reasoning" and "do not readily expose human-interpretable semantics." (A related but different argument, about learning from latent structure rather than tokens, is in Agents Should Learn From Latents.)


The architecture, precisely

HybridLink. A two-layer residual MLP that maps the last-layer hidden state back into the input-embedding space:

e_{t+1} = R(h_t) = h_t + W2 · σ(W1 · h_t)        # ~2·d_h² params, ~10M for a 4B model

The reason it exists: feeding h_t straight back in as the next input (what Coconut does) "creates a representation mismatch with the pretrained input space." Their ablation makes the point. On 14B, using the raw hidden state (identity mapping) scores 71.4 / 73.7 / 24.5 on MATH500 / MedQA / MuSiQue; the residual HybridLink scores 88.2 / 81.7 / 34.5.

Mode switching is token-driven, not a learned gate.

  • Enter latent: the model samples an ordinary special token <latent>, trained with cross-entropy like any other token.
  • Stay latent: each step feeds R(h_t) back as the next input embedding, skipping the LM head and the embedding lookup.
  • Exit latent: when the nearest vocabulary embedding to R(h_t) (by cosine) is </latent>.

So the model decides, but with a fence: at inference "we enforce a minimum of 16 and a maximum of 256 steps" per latent span. Keep that in mind when you read "dynamically decides."

"Continuous working memory" is the KV cache. Latent steps write KV entries like any token; tool observations are tokenized and appended "without reconstructing or re-prefilling earlier context." "Lossless" means a precise thing here: continuing from the cached KV gives exactly the hidden states a full-prefix forward pass would (their Theorem A.1). Latent thoughts are never round-tripped through text between turns. It does not mean nothing is lost inside latent reasoning.

If you've read our Octopus piece on branches sharing a KV cache, this is the same move in a different direction: treat the cache, not the transcript, as the agent's state.

Inspectability = nearest-token readout. Each latent step is decoded as the vocabulary token whose embedding is nearest to R(h_t). No trained decoder, no probe. You get one pseudo-token per latent step: readable-ish, with garbles (the paper's own case studies include strings like "the's distance law").


Training: SFT teaches the plumbing, RL learns the policy

HybridSFT converts existing text agent data (ARPO and DAPO sources) into 54k hybrid trajectories:

  • Mode labels come from GPT‑6.1‑Sol / GPT‑5.6‑Luna plus the frozen model's entropy: "Low-entropy steps become latent candidates, while high-entropy steps remain in text mode."
  • An LLM "visibility audit" restores decision-critical steps to text. No human labels.
  • Three one-epoch stages: (1) format activation (emit the anchor tokens), (2) HybridLink activation (backbone frozen; cosine + MSE to target embeddings), (3) two-pass self-conditioned training on the model's own R(h).

Stage 3 matters. On 14B, AIME25 / AIME26 / MATH500 go 20.6 / 19.8 / 62.4 after format activation, 23.8 / 22.5 / 73.8 after HybridLink activation, and 32.5 / 31.7 / 84.2 after self-conditioning, versus text SFT at 30.8 / 31.7 / 83.8.

HybridRL is GRPO with two interesting choices:

  • Likelihood ratios only over sampled text positions; recorded latents are replayed exactly. HybridLink is frozen during RL; KL coefficient 0.
  • No reward for mode usage: r = r_correct + r_format + r_tool (tool bonus +0.1 only when a trajectory uses both Python and search). When to go latent is learned indirectly from task reward.

During RL on 14B, text tokens fall to ~499 and latent steps rise to ~276, with a 98.8% tool-use ratio. The policy pushes more thinking into latents while tool use goes up. That's the behaviour you'd want, and it emerged without being rewarded directly.


Efficiency: read each number for what it counts

Total generation steps: 4B 1.9k→1.2k, 8B 2.3k→1.1k, 14B 2.2k→0.9k; latent share 29–37%

ClaimWhat it actually measuresBaseline
35.5–54.7% reductionTotal generation steps, text tokens + latent forward passesSame-backbone text agents
2.2–3.1× fewer tokensText tokens onlySame
2.1× faster (range quoted 1.8–2.6×)Wall-clock per sample, pooled across 4B/8B/14B: 134 s → 63 sSame, on their extended vLLM/verl engine, H100s
2.7–3.7× cheaperFLOP-based estimate at $2.50/H100-hour and 10% utilization, excluding prefill, context-dependent attention and tool executionSame

Two cautions. First, each latent step is a full forward pass; savings come from needing fewer passes in total, not from free thinking. Second, the paper doesn't say whether wall-clock includes tool latency, and the comparison is against their own text agent on their own engine, not against a hosted API.

Monitor AUROC on text spans only vs full text CoT; estimated cost per 1M questions


Accuracy: modest and consistent against text RL; mixed against frontier agents

Qwen3-14B after RL: HybridAgent vs GRPO and ARPO across six benchmarks

Against same-backbone text RL (GRPO, ARPO) on 14B, HybridAgent wins every column of the math/knowledge table by 0.7–2.5 points. The abstract's "+10.4%" is stated against these baselines "under matched model sizes and training budgets"; the paper doesn't say whether it's relative or absolute, and I couldn't reproduce it from the tables. Don't quote it as "beats frontier models by 10.4%."

On agentic search (GAIA, WebWalker, HLE, SealQA, DeepSearchQA), the 14B model clearly beats ReAct, Search‑o1 and WebThinker on the same backbone, and beats several frontier agents on GAIA and WebWalker. It loses on HLE (16.0 vs Kimi‑K3's 31.4) and on DeepSearchQA (21.7 / 46.8 vs GPT‑5.6‑Luna's 23.3 / 49.4). One more thing: GPT‑5.6‑Luna is simultaneously a baseline, the search-page summarizer for all agents, an LLM judge, a monitor, and one of the SFT labelers.


Do the latents mean anything? The interventions say yes

Interventions on latent steps: nearest-token swap barely hurts; random or cross-problem latents collapse accuracy

This is the experiment I'd show anyone skeptical of latent reasoning. On MATH500 (8B / 14B, unmodified 84.6 / 88.2):

  • Random-token or cross-problem latents: accuracy collapses to the 40s. The latents carry problem-specific content the model relies on.
  • Within-block permutation: 73.2 / 76.4. Order matters.
  • Block latent mode (text-only): 80.4 / 78.8. The model can still work without latents, but worse.
  • Nearest-token replacement: 82.6 / 87.2. Swapping each latent for its nearest token barely hurts, which is the best evidence that the readout captures most of what the latent carries.

What this means for chain-of-thought monitoring

This week we've covered OpenAI's monitors reading agent chains of thought and Google admitting Gemini 4 Argon is "less monitorable" because it does more reasoning without writing it down. Latent reasoning is that trend, done on purpose.

The paper's evidence on monitorability, read narrowly:

  • A monitor seeing only HybridAgent's text spans predicts answer correctness about as well as one reading a text agent's full CoT. Weak monitor (GPT‑4o‑mini): 0.840 vs 0.772 (8B), 0.860 vs 0.818 (14B). Strong monitor (GPT‑5.6‑Luna): 0.961 vs 0.935 and 0.989 vs 0.990. The claim: "AUROC remaining within 0.022 of matched text agents."
  • That's math only, predicting correctness, with latents hidden from the monitor. It says the visible text stays informative. It doesn't say latent reasoning is monitored, and it doesn't test detection of misbehaviour.
  • The safety framing in the discussion (latents are "readable solely through the provider-owned HybridLink") is stronger than the mechanism supports: the readout is a cosine to the public vocabulary.
  • The paper has no limitations section.

My call

Try it if latency and decode cost dominate your agent bill, especially for math- and retrieval-heavy agents on models you host. The architecture is small and the code is public. The interventions convince me the latents do real work.

Engineer around three things:

  1. Serving. Their own words: "current serving engines do not support multi-turn hybrid generation." You'll need the extended engine or your own.
  2. Monitoring. If you rely on CoT monitors, treat latent spans as unmonitored by default. Log nearest-token readouts, run your monitor over them as well as the text, and don't count the paper's correctness-AUROC as a safety guarantee.
  3. Evaluation. Measure wall-clock with tool latency on your workload. The 2.1× is decode-side.

What would flip me to "adopt broadly": a misbehaviour-detection study where monitors see readouts vs text-only vs full text CoT, on agentic tasks rather than math, plus an independent replication of the efficiency numbers on a standard serving stack.


Source note

Verified at origin. Everything above is from arXiv:2610.18861 (v1, 13 Oct 2026): Eq. 4–5, Theorem A.1, Tables 1–9, Figures 6, 7 and 9, and the training and inference appendices, read on the PDF pages.

How the charts were built. Figure 6 (steps, values as printed, e.g. "1.9k"), Table 2 (14B), Table 5 (interventions), Table 7 (monitor AUROC) and Table 9 (cost estimates). No interpolation.

What I could not verify. The "+10.4%" basis. Whether wall-clock includes tool time. No human evaluation of readout faithfulness exists in the paper. This is a preprint, not peer-reviewed.