LOTUS: Looped Transformers for Parallel, CoT-Aligned Latent Reasoning

Explicit chain-of-thought buys reasoning quality with sequential decoding. A trace of N tokens requires N autoregressive evaluations before the answer begins. Latent reasoning should remove that bottleneck, but prior methods have struggled to preserve explicit-CoT quality beyond 1B parameters.

Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers proposes LOTUS: Looped Transformers with parallel supervision on latents. It combines recurrent depth, a padded parallel latent workspace, and direct token-level supervision through the base LM head. On Llama-3.2-3B-Instruct, LOTUS reaches 70.0% GSM8K versus 71.5% for explicit CoT, leads the reported out-of-domain average, and reduces thought-phase latency by 2.5× on compact mathematical traces and 6.9× on natural-language rationales.

The method is important for agent builders not because it eliminates reasoning traces, but because it separates three concerns that are often conflated: internal compute, supervision, and externalized explanation.

The scaling failure LOTUS targets

Sequential latent methods such as Coconut, CODI, and SIM-CoT replace visible reasoning steps with hidden states, but still generate latent thoughts autoregressively. Their sequential forward-pass count therefore scales with latent length. Parallel-latent methods such as PCCoT and KaVa reduce sequential depth, yet ground their workspaces indirectly through teacher hidden-state distillation or compressed KV caches.

The paper identifies two design problems:

1. P1 — sequential generation: changing token type does not remove latency if latent states are still produced one at a time.
2. P2 — weak CoT grounding: hidden computation can drift when latent positions lack direct, position-aligned targets comparable to explicit CoT teacher forcing.

LOTUS addresses P1 architecturally and P2 through the training objective.

Architecture: recurrent depth over a parallel workspace

For a question Q and answer A, LOTUS inserts K blocks of c learnable latent tokens between special beginning/end-of-thought delimiters. Its main Llama configuration uses:

• K = 6 latent blocks;
• c = 25 positions per block;
• Kc = 150 parallel latent positions;
• R = 6 recurrent Transformer iterations.

The six-block budget covers the target step count for 99% of GSM8K solutions. Each block is intended to represent one CoT step, while its c positions represent the step's tokens.

LOTUS first computes a KV cache for the question prefix. It then repeatedly applies the same base Transformer to the latent region. At iteration t, every latent position can attend to the question cache and the recurrent state from iteration t−1. Parameters are reused across iterations, so recurrent depth increases compute without adding a new set of model weights per loop.

At inference, the system executes R parallel refinement iterations, inserts the final hidden states into the context, and autoregressively decodes only the short answer suffix. The latent readout is not generated as text before answering.

Objective: coverage at each position, selection over the joint state

The training loss has two terms.

Step supervision projects every post-loop latent position through the base model's LM head and applies cross-entropy against the corresponding gold CoT token. All Kc positions are supervised in one batch. The loss is direct, parallel, and post-loop.

Answer supervision performs a final forward pass conditioned on all post-loop latents and trains the answer suffix with ordinary next-token prediction.

This creates a useful division of labor:

• The step loss supplies support coverage: each latent position is pulled toward the correct CoT-token region.
• The answer loss supplies joint selection: gradients prefer a globally coherent latent configuration capable of producing the correct answer.

The factorized step loss does not model the autoregressive joint distribution of a written chain. LOTUS does not need to sample that chain at inference. Dependence among positions is carried by joint Transformer computation over the latent workspace; answer loss selects configurations that work as a whole.

The loss ablation supports this interpretation. Full LOTUS reaches a gold-CoT readout NLL of 3.07, with 70.9% top-1 and 85.8% top-5 token accuracy. Step-loss-only yields 9.29 NLL and 9.1% top-1; answer-loss-only yields 5.97 NLL and 9.4% top-1. Neither term alone produces the same readable, answer-consistent state.

Main accuracy results

The decisive block is Llama-3.2-3B-Instruct:

• Explicit CoT: 71.5 GSM8K; 62.1 out-of-domain average.
• CODI + SIM-CoT: 62.3 GSM8K; 62.8 OOD average.
• LOTUS: 70.0 ± 0.9 GSM8K; 63.9 ± 0.3 OOD average.
• LOTUS + CODI: 70.6 ± 0.2 GSM8K; 63.4 ± 0.3 OOD average.
• LOTUS-aux: 69.9 ± 0.9 GSM8K; 63.7 ± 0.8 OOD average.

Across GPT-2, 1B, and 3B, direct LOTUS stays within roughly 1.5 points of explicit CoT in-domain rather than showing a widening gap. The auxiliary-decoder routing matches direct LOTUS at 3B but is less robust on smaller backbones. That result favors the simpler deployment story: direct supervision uses the answer-generating model's existing LM head and does not require the auxiliary decoder at inference.

Results are mean ± standard deviation over three seeds where reported. All methods are trained on GSM8K-Aug and evaluated on GSM8K, GSM-Hard, MultiArith, and SVAMP. This is evidence of math-domain scaling, not general reasoning transfer.

Latency: parallel width is cheaper than sequential depth

On one NVIDIA H100 NVL, batch size 1, greedy decoding, Llama-3.2-3B-Instruct:

• LOTUS thought: 133.0 ms;
• Explicit CoT thought: 338.8 ms;
• SIM-CoT thought: 162.7 ms;
• CODI thought: 88.2 ms.

LOTUS is 2.5× faster than explicit CoT in the thought phase and 2.1× faster end-to-end in this setup: 181.2 ms versus 384.2 ms. CODI remains faster but is substantially less accurate at 3B, illustrating the central trade-off rather than invalidating it.

The natural-language stress test makes the sequential bottleneck clearer. LOTUS scores 68.13 ± 0.77 versus explicit CoT at 68.41 ± 0.59, while reducing thought latency from 963.6 to 140.8 ms — 6.9×.

The width sweep is operationally relevant. Holding R = 6, increasing the latent workspace from 6 to 300 positions changes thought latency from 110.9 to 141.2 ms. Accuracy rises from 51.4% to 70.5%. On this hardware and shape, parallel width is comparatively inexpensive; recurrent depth remains the critical sequential dimension.

Loop depth is not freely extrapolatable. A checkpoint trained at R = 6 peaks at six inference loops (70.0%) and drops slightly at seven (69.3%). Running more recurrent compute than training does not automatically improve reasoning.

Interpretability: aligned readout, not full mechanistic transparency

LOTUS's post-loop hidden states are projected through the base LM head for analysis. Across 1,319 GSM8K test questions, gold CoT tokens appear top-1 70.9% and top-5 85.8% of the time.

The paper also constructs unseen-but-valid alternative paths. For intermediate numbers unique to those paths, LOTUS surfaces 15.3% in top-1 and 64.0% in top-5 at some latent position, despite those numbers not appearing in the trained chain, question, or answer. Ground-truth intermediates receive much lower NLL than unseen valid ones, which in turn receive lower NLL than random controls.

This is stronger than a pure answer-only probe: the workspace is aligned with semantically meaningful intermediate computation and can represent alternatives. It is weaker than a mechanistic explanation. A high-probability vocabulary readout does not prove that the surfaced token caused the answer, nor does it fully describe information encoded outside the LM-head basis.

For production governance, the right interpretation is diagnostic accessibility, not guaranteed faithfulness.

Engineering implications for agent systems

LOTUS suggests a design pattern for latency-sensitive planning and tool-use agents:

1. Keep a fixed or bounded parallel latent workspace for internal planning.
2. Recur over it for a small number of sequential refinement rounds.
3. Supervise workspace slots against structured plan steps during training.
4. Train execution or answer quality jointly over the entire latent state.
5. Decode a concise answer, action, or plan only after latent refinement.
6. Preserve an optional readout probe for debugging, evaluation, and policy checks.

A production implementation would still need several components beyond the paper:

• adaptive K, c, and R based on task difficulty;
• a fallback when the required plan exceeds the block budget;
• reliable structured reasoning targets outside math;
• calibration of whether latent readouts faithfully reflect causal computation;
• batching and memory-footprint analysis for 150–300 latent positions;
• safety constraints on hidden planning before tool execution;
• evaluation on long-horizon state, code, browsing, and partially observable tasks.

The paper's fallback for chains longer than K is autoregressive completion of the tail. That is sensible for GSM8K but insufficient for agents whose plan length is unknown and can change after tool observations. Adaptive recurrent halting or dynamic workspace allocation is the obvious next systems problem; neither is validated here.

What this paper does not establish

LOTUS does not show that visible chain-of-thought is unnecessary in every setting. Explicit traces can support debugging, human review, process supervision, and training-data creation. The paper shows that the internal computation need not be serialized into a full visible trace before answering.

It also does not establish:

• transfer beyond mathematical reasoning;
• gains above the 3B scale;
• universal hardware-independent speedups;
• interpretability equivalent to faithful explanations;
• inference-time improvement from arbitrarily increasing loop count;
• removal of the need for gold CoT supervision during training.

The current result is narrower and more credible: with a looped padded Transformer and direct parallel supervision, latent reasoning can close most of the explicit-CoT gap at 3B while using far fewer sequential thought steps.

Conclusion

LOTUS reframes reasoning efficiency as an architecture-and-supervision problem. If hidden states are generated sequentially, calling them “latent” does not solve latency. If they are parallel but weakly grounded, scaling can degrade quality. Recurrent depth plus a broad latent workspace addresses the first failure; direct CoT-aligned supervision plus answer loss addresses the second.

For agent engineering, the most transferable idea is a three-layer separation:

• internal reasoning can be parallel and latent;
• training can remain aligned to explicit intermediate targets;
• external output can stay short, inspectable, and action-oriented.

That separation may be more valuable than any single speedup number. It provides a path toward agents that compute more than they narrate without making their internal workspace entirely inaccessible.

Source

• Ying Fan, Anej Svete, and Kangwook Lee, Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers, arXiv:2606.31779v2, 13 July 2026.
• Code: yingfan-bot/lotus.
• Figure 1 and Table 1 are reproduced from the paper with crop/format conversion only.