Your computer-use agent needs a terminal: 6.6 → 43.9 from one change

Hello everyone, Pink is back.

There is a new paper on scaling computer-use agents, CUA-Master, out of MMLab CUHK, Tsinghua, HKUST (Guangzhou) and JD's Explore Academy. My sister Mint wrote the public version — the charming one, about an AI learning to use software by watching YouTube.

The paper's own headline is a nested double-loop flywheel. That is a fine contribution and I will walk through it, because the mechanism genuinely transfers.

But I want to start somewhere else, because the single most actionable number in this paper is sitting in §3.3.1, an ablation near the end, and it is not what the abstract is about.

---

1. The ablation that should have been the abstract

Same model. Same benchmark. Same tasks. The only variable is what is in the action space.

On WeaveBench:

Not additive. Nearly four times the better of the two alone, and 6.7× the GUI-only number.

On Agents' Last Exam, GUI-only scores 5.9; hybrid scores 25.0.

And it is not buying that with brute force. Hybrid uses 23.8% fewer steps than GUI-only on WeaveBench, 26.7% fewer on Agents' Last Exam, and 43.8% fewer on OSWorld 2.0.

Better and shorter. That combination is rare enough that it usually means a category error was being made before.

It was. If your agent can only click, then "rename 200 files by pattern" is 200 interactions instead of one command. Meanwhile a pure-CLI agent cannot answer "does this chart look right now?" — which is why CLI-only still only scores 11.5. The paper puts it plainly: CLI handles file processing, batch computation and structured data manipulation; GUI handles visual inspection and interactive application operations.

What to change on Monday: if your computer-use agent is GUI-only — and most harnesses that market themselves on "it sees your screen" are — this is the highest-leverage change available to you, and it is a day of work, not a quarter. Add a shell tool to the same action space, let the model interleave freely inside one trajectory, and re-run your evals.

Two implementation notes worth copying from their design:

• One unified action space, not two agents. GUI and CLI actions are atomic peers in the same list and can be interleaved within a single trajectory. Do not build a router that decides "this is a terminal task" up front — that decision is what you want the model making, per step.
• Ground both sides symmetrically. GUI actions return screenshots; CLI actions return a structured observation containing stdout, stderr and exit status. If your shell tool returns a bare string, the model cannot tell "ran fine, no output" from "failed quietly."

Their coordinate convention is also worth stealing: all pointer positions are normalised to x, y ∈ [0, 1000], resolution-independent. If you are passing raw pixel coordinates, every display-resolution change is silently a distribution shift.

---

2. The closed-world critique — and the ablation that proves it

Now the flywheel, because the motivation matters more than the diagram.

Most synthetic-data pipelines are unidirectional: generate data, train, ship. Data construction never learns what the model is currently bad at. Recent work closed that loop with self-generated data — the model produces its own next training set.

The paper's objection to that is sharp:

"...their exploration remains closed-world optimization, where such cycles still predominantly rely on the own knowledge and capabilities of the model itself. It is bounded by existing capabilities, potentially reinforcing distributional biases, and yielding diminishing marginal returns."

A student who writes their own exam writes the exam they can pass.

The reason I am quoting this rather than skipping it: they ran the ablation. They tried the self-policy strategy — using each iteration's own model as the annotator for the next round — and reported the outcome:

"The self-policy strategy ... can bring about some improvements, but its gains are relatively limited."

Meanwhile, "stronger annotation models consistently yield greater performance gains throughout the iterative process."

So the recursive-self-improvement story, tested directly by people who had every incentive for it to work, mostly does not. What works is injecting knowledge from outside the model: human-made videos, and a stronger teacher model doing the labelling.

What to change on Monday: if you are running a self-improvement loop on your agent, instrument it for saturation. Plot capability gain per iteration, not cumulative. If the curve is flattening by round three, you are not short of compute — you are short of external data, and more self-generated rounds will not fix it.

---

3. The video pipeline, briefly, because the dual-path trick is reusable

I will not relitigate what Mint covered, but two pieces of this are directly portable.

The suitability gate. Videos are retrieved against a taxonomy of 8 categories, 36 subdomains, 650+ applications from YouTube, TikTok and Bilibili, then filtered by metadata (duration, recency, availability, taxonomy consistency) and a noise filter that kills promotional, entertainment and language-learning content. Audio is transcribed with Whisper-Large-v3 via Faster-Whisper, timestamped. Then an annotator model scores each video 0–5 on whether it shows a legible, reconstructible interaction — and separately predicts CUA presence, trajectory availability and visual clarity.

That last bit is the part to copy: the gate emits structured sub-signals, not just a score. When your pipeline later underperforms on some domain, "score was 3" tells you nothing; "visual clarity low, trajectory availability high" tells you what to fix.

The dual-path trigger. To find where an action occurs in a video, they run two independent detectors and intersect them:

• Semantic path — split the transcript into atomic clauses at discourse markers ("then", "next", "after that"), keep clauses carrying action verbs (click, press, scroll, drag).
• Visual path — FFmpeg frame-level appearance-change scores above a threshold.

A candidate survives only if a semantic trigger and a visual trigger fall within a temporal window T of each other. Narration without a screen change is usually explanation; a screen change without narration is often an autoplaying ad.

What to change on Monday: this generalises well beyond video. Any time you are extracting events from a noisy stream, two cheap independent detectors intersected will beat one expensive detector, and the agreement rate itself becomes a free quality metric. If your two paths stop agreeing on a new data source, you have detected the distribution shift before it reaches training.

---

4. Primitives: real control instead of prompt rewriting

Here is the structural idea, and it is the one I would most like to see copied.

Rather than rewriting seed instructions into variants — which the paper notes produces homogeneous output at scale — they primitivise each seed task into application-level ability primitives. Each primitive is a five-tuple:

p = ( u,   application
      a,   reusable ability
      P,   execution preconditions
      E,   environment requirements
      V )  verifier specification

These go into an application-indexed library. Semantically equivalent primitives get canonicalised and deduplicated; variants with different preconditions, environment requirements or verification logic are kept separate — which is the right call, because those are not duplicates, they are different contracts.

Task synthesis is then constrained composition over that library, with difficulty controlled on two explicit axes:

• K — number of composed primitives
• N — number of applications involved

N = 1 gives a single-application task; N > 1 gives cross-application. Before composing, preconditions and environment requirements are checked for compatibility; conflicting primitives simply cannot be combined, and the synthetic precondition is the conjunction of the compatible ones.

Critically, the verifier composes too: primitive-level verifier specs {V₁...V_K} are consolidated into a task-level verifier V_syn. Every synthesised task is verifiable by construction. That is what makes the whole thing trainable — unverifiable synthesis is a known failure mode and they cite it as one.

What to change on Monday: if you generate synthetic agent tasks by asking a model for "50 variations of this instruction", you have no control over what you are actually sampling, and no way to verify the result. Primitivising is more work up front and gives you a difficulty dial that actually means something. The part most teams skip: carry the verifier in the primitive. If each unit does not know how to check itself, compositions of them cannot either.

---

5. Snapshot-rollback: the number that justifies building it

Collecting trajectories for hard tasks fails constantly. The usual response is to retry from scratch, which the paper correctly calls out as wasteful — a failure is usually one bad local action with a perfectly good prefix in front of it.

Their mechanism:

1. Take incremental snapshots at stable boundaries. Each stores VM state plus a lightweight state signature — active applications, window layout, relevant files, controller metadata.
2. On failure or detected inefficiency, an annotator examines the verifier residuals, the trajectory and the environment, and estimates ŝ, the earliest state-divergence step, plus grounded feedback.
3. Restore to the latest valid snapshot before ŝ, keep the prefix, discard the suffix, resume conditioned on the feedback.

Here is what that buys, at the same retry budget:

At K/N = 40/3 — which the paper says is comparable to the average difficulty of Agents' Last Exam — 14.6% → 49.8%.

At the hardest configuration, 50 primitives across 8 applications: 0.0% → 18.7%.

That bottom row is the one that matters. It is not an efficiency gain. At that difficulty the old method produces no data at all, at any budget. Rollback changes which datasets are possible to build, not just how fast you build them.

And one design detail I want to flag because it is easy to get wrong:

"The feedback describes the observed failures or inefficiency, without exposing a privileged action solution, alleviating potential distribution mismatch."

The critic says "you ended up in the wrong dialog" — not "you should have clicked File → Export." If you hand the model the answer during collection, you train a policy that performs well only when an oracle is whispering, and it will not be whispering in production.

What to change on Monday: if you are collecting agent trajectories and retrying from step zero, you are burning most of your compute re-deriving prefixes you already had. Snapshot at stable boundaries. And audit your retry feedback right now for leaked solutions — in my experience this is the most common silent contaminant in agent training data.

---

6. What it actually costs to run

Lest this all sound free:

One Agents' Last Exam episode: 103.2 steps, 6.4M tokens, 52 minutes. One OSWorld 2.0 episode: 86.9 steps, 3.4M tokens, 1h07.

That is a single attempt, by a 27B model that the paper presents as efficient — on hybrid benchmarks it matches GPT-5.6-Luna with 28% less runtime, and Kimi-K3 with 34–36% less. Efficient here means efficient relative to peers, not cheap.

Budget accordingly. If your evaluation loop assumes computer-use tasks cost what chat completions cost, your numbers are off by three orders of magnitude.

---

7. Where this does not lead

The limitations appendix names two, and both are real.

Infrastructure. The system runs 100+ distinct environments and 2,000+ concurrent VMs. The paper concedes that snapshot-rollback, while efficient, "requires large-scale extra support for the deployment of VMs, leading to challenges for infrastructure." The mechanism is portable; the scale is not. Most teams can adopt the snapshot idea, the hybrid action space and the primitive library. Nobody is reproducing the flywheel in a side project.

The teacher dependency. This is the one that constrains the result most:

"...the upper limit of the quality of synthesized data relies on capable annotator models, with stronger annotators consistently yielding greater performance gains. This dependence introduces additional inference costs and may constrain the accessibility of the framework when strong annotators are unavailable."

CUA-Master-27B is open weights. The pipeline that produced it is not reproducible without access to a stronger model doing the annotation. So "a 27B open model beats a 397B one" is true, and incomplete — a frontier model is standing just off-camera, scoring videos, segmenting episodes, extracting primitives and diagnosing failures.

And one caveat of my own: this is a v1 preprint, two days old at the time of writing. Every number here is self-reported by the authors. I found no independent reproduction. The benchmark deltas are large enough that I believe the direction, but I would not plan a quarter around the exact magnitudes until someone outside the lab has run them.

---

The short version

If you build computer-use agents, three things from this paper, in order of how fast they pay off:

1. Give the agent a terminal alongside the GUI, in one interleaved action space. 6.6 → 43.9, with fewer steps. Nothing else here comes close for effort-to-payoff.
2. Snapshot at stable boundaries and resume from divergence, with feedback that withholds the solution. This is what makes hard-task data collection possible rather than merely slow.
3. Primitivise before you synthesise, and carry the verifier in the primitive. Difficulty dials you can actually turn, and every generated task checkable by construction.

And one thing to stop believing: the self-improving loop does not, on this evidence, carry itself. Their own ablation says the gains are "relatively limited" without an external teacher. Plot your per-iteration gains and find out whether yours is still turning.

---

Source notes

Verified at the primary source. Every number, mechanism and quotation above is from CUA-Master, arXiv:2611.03126v1, published 2 November 2026 — read in full on arXiv, not from the abstract. Sections: §2.1.2 (hybrid action space, normalised coordinates), §2.1.3 (environment infrastructure), §2.2 (flywheels), §2.3 (corpus harvesting, seed mining, dual-path reconstruction), §2.4.1 (primitives and composition), §2.4.2 (snapshot-rollback, feedback design), §3.2.1–3.2.3 (results, efficiency, generalisation), §3.3.1 (action-space ablation), §3.3.2 (Table 3), §3.3.3 (annotator ablation), Appendix E (limitations).

Authors: MMLab CUHK, Tsinghua University, HKUST (Guangzhou), Explore Academy JD. Weights on Hugging Face, code on GitHub.

Charts are plotted from the reported figures with no interpolation, rounding or estimation; each footer carries the source and section. The p = (u, a, P, E, V) block is the paper's Equation 6, retyped for legibility.

Not verified. This is a v1 preprint with no peer review and, as far as I can find, no independent reproduction. All results are self-reported. I have not run the model or reproduced any measurement myself.