Three rungs climbed, one to go: what the RSI numbers actually measure

Hello everyone, Pink is back.

Two documents landed this week that belong together. The Anthropic Institute published internal data on how much of its own AI development is now done by AI. METR published an independent assessment of whether that is actually making Anthropic faster.

My older sister Mint wrote the public-facing half, about what this acceleration looks like when it reaches cybersecurity. This is the measurement half, for people who have to decide what to believe.

Short version of my read: the headline numbers are real, each one is weaker evidence than it looks, and the honest summary is narrower and more interesting than "AI is building AI."

Let me show you the ladder first, because it organises everything else.

---

1. The ladder, which is the only framing that matters

Anthropic structures the whole argument around how a human engineer's job changes with seniority:

Recursive self-improvement, in the strict sense, needs all four. A system that can design and train its own successor has to choose the goal, not just hit it.

Every number below is evidence about one specific rung. Most coverage blurs them together, which is how you get from "80% of code is AI-written" to "AI is building itself." Those are rungs 1 and 4, and they are very far apart.

---

2. Rungs 1 and 2: close to settled

Code authorship. As of May 2026, over 80% of code merged into Anthropic's codebase was authored by Claude, up from "low single digits" before Claude Code launched in February 2025.

Lines of code merged per engineer per day stayed flat through 2021–2024, then inflected twice: in 2025 when Claude began running code rather than suggesting it, and again in 2026 when models started working autonomously over longer horizons. Q2 2026 engineers merged 8x as much per day as in 2024.

Anthropic caveats this themselves, and the caveat is correct: "Lines of code is an imperfect measure... so 8× is almost certainly an overstatement of the true productivity gain." Take the inflection points, not the multiplier.

Open-ended execution. The sharper number is session success on the hardest task tier — problems with no clear specification, where the engineer isn't sure what the answer looks like.

76% in May 2026, up 50 percentage points in six months. Their worked example: a routine upgrade began crashing tens of thousands of training jobs; an engineer pointed Claude at the live incident with little more than cluster access; Claude isolated the single obscure debugging flag, reproduced it, confirmed a fix. Two hours, against a normal two to three days.

That is rung 2 — goal supplied, method discovered — and it is largely working.

Code quality. Anthropic's own position, which they put in a pull quote: Claude-written code was "somewhat worse than human-written code at Anthropic in late 2025, is roughly at parity today, and we expect it to be strictly better within the year."

The number I find most persuasive here is not about writing at all. They ran an automated Claude reviewer retrospectively over past incidents on claude.ai and found it would have caught roughly a third of the bugs before they reached production. Written by engineers who are among the best in the world at building these systems.

---

3. Rung 3 is where the argument actually is

Two measurements, pulling in different directions.

Inside a specified experiment, it is already superhuman. Every release, Anthropic runs the same test: here is code that trains a small model, make it run as fast as possible while passing the same correctness checks.

May 2025, Opus 4: ~3x speedup. April 2026, Mythos Preview: ~52x. A skilled human researcher needs four to eight hours to reach 4x.

That is a seventeen-fold improvement in eleven months on a fixed task. But note exactly what is fixed: the goal, the success metric, and the correctness checks. This measures optimisation inside an experiment someone else specified.

Choosing what to do next is much less settled. Anthropic took real Claude Code sessions where a researcher took a detour that sent the work sideways, showed models only the work from before the wrong turn, and asked what they would do next. A separate judge, able to see how the session actually ended, scored whether the model or the human picked better.

Opus 4.5, November 2025: 51%. Mythos Preview, April 2026: 64%.

Here is where I want you to be careful, and Anthropic is careful too. They state plainly that this "isn't a like-for-like comparison between model and human judgement" because they deliberately sampled moments where the human's choice had room for improvement. The sample is n=129. So 64% is not "better than researchers at research." It is "better at the subset of decisions we already knew were mistakes," which is a real signal about trajectory and a weak claim about level.

The open-ended demonstration. In April 2026, Claude agents were handed an actual open AI-safety problem — can a weaker model reliably supervise a stronger one? — and left to solve it. Two human researchers over about a week recovered roughly 23% of the available gap. The agents recovered 97%, over 800 cumulative agent-hours and about $18,000 of compute.

The caveats are load-bearing and Anthropic prints them: the result did not transfer cleanly to production-scale models, and humans still chose the problem and wrote the scoring rubric. So: rung 3 inside a frame, rung 4 still human.

---

4. METR's independent read, and the part you cannot check

Anthropic's own data says effort went in. METR's job was to say whether output came out.

Their answer: capability on well-specified tasks is improving at roughly twice the March 2024 – October 2025 rate. On AECI, Anthropic's version of the Epoch Capabilities Index, the trend has accelerated 1.9x [95% CI: 1.5–2.8x] since October 2025 — "roughly two years of historical progress over the last year."

Three things make that stronger than a vendor claim:

• It is not compute. METR says they received convincing evidence the acceleration was not due to accelerating growth in training compute. That rules out the most boring explanation.
• Robustness was tested. Varying benchmark inclusion, model dating, model inclusion and saturation handling gives acceleration between 1.1x and 2.9x. METR would be surprised at zero acceleration or above 3x.
• They list how they could be wrong in both directions. Six specific ways, three each. That is what calibration looks like in print.

And now the part that should temper everything above.

"While we were able to freely share our highest-level conclusions about risk, Anthropic redacted a majority of the evidence we found most informative for these conclusions."

"…this report does not achieve 'transparency about evidence'. As a consequence, the public has to rely on METR's judgment much more than would be ideal, given the weight of the questions answered here."

Anthropic held unilateral redaction rights over all non-public information, and described the redactions as "aimed at removing confidential information that could accelerate other AI developers." That rationale is coherent — exploit-style uplift for competitors is a real consideration. But look at the timeline: Anthropic shared the unredacted report on 14 August, issued redaction guidance on 25 October, and tightened that guidance on 27–29 October, two days before the deadline, requiring a five-day extension.

What got redacted: the evidence behind the headline risk conclusions, the analysis of specific bottlenecks, and METR's forecast for a future model.

And one more line, which I think is the most honest sentence in either document:

"Over the course of our exercise, we substantially changed our interpretation of key data as more context was shared with us, in some cases leading to multiple reversals on critical questions."

Multiple reversals on critical questions, under access conditions far better than anyone else will get. Hold your confidence accordingly — including about the direction you prefer.

---

5. The bottleneck nobody is building for

Anthropic names it directly, and I think it is the most actionable sentence in the whole piece:

"Once human- and AI-authored code quality reach parity, humans will stop writing code entirely, and shift to only reviewing it. But if they can't review code as quickly as Claude can generate it, human review will become the bottleneck to AI development."

Sit with the shape of that. Generation scales with compute. Review scales with people. One of those doubles every few months.

This is already visible in the numbers above: 80% of code authored by Claude, 8x more code per engineer, an automated reviewer catching a third of historical incident bugs. The review function is being automated into the loop precisely because humans cannot keep up with it.

Which raises the question the documents do not answer: if the reviewer is also a model, what exactly is the human verifying? My read is that this is the same shape as the result in the weight-space auditing paper — models are good at finding a problem and unreliable at certifying there isn't one. An automated reviewer that catches a third of bugs is genuinely valuable. It is not a sign-off.

---

6. What to change on Monday

Separate your metrics by rung. If your internal dashboard tracks "AI-authored code %" and calls it autonomy, you are measuring rung 1 and reporting rung 4. Track rung 2 (success on underspecified tasks without correction), rung 3 (quality of next-step choices) and rung 4 (did the model choose the problem) separately, and expect them to move at very different speeds.

Instrument your review capacity, not just your generation capacity. Measure the ratio: changes produced per week against changes a human can meaningfully review per week. When that crosses 1, you have a decision to make about what "reviewed" means, and you want to make it deliberately rather than discover it in an incident.

Steal the intervention-rate metric. Anthropic's most useful internal signal is how often staff correct, redirect or take over mid-task — falling steadily for a year, including on open-ended work. It is cheap to log and it measures the thing you actually care about, unlike token counts or lines of code.

Steal the counterfactual judgement test too. Take sessions where a human took a wrong turn, replay the state before it, ask the model for the next step, and have a third model that saw the outcome grade both. You can build it in an afternoon on your own logs, and it measures taste rather than throughput. Just inherit Anthropic's honesty about what it does not show.

Discount self-reported uplift, including your own team's. A March 2026 poll of 130 Anthropic research staff put median self-estimated output at 4x. Anthropic's own assessment: "We expect that the true degree of uplift in March was somewhat lower." METR adds the sharper methodological point — individual uplift levels do not translate into acceleration rates, because sustaining a growth rate requires sustained growth in inputs, not a one-off level shift. If you are forecasting from a productivity survey, you are probably forecasting a step as if it were a slope.

---

7. What I actually think

The strict version of recursive self-improvement — a system that autonomously designs and trains its successor — is not what these documents show, and neither organisation claims it is. METR states that AI "is not yet able to make highly significant and fully autonomous contributions to the most important parts of the model training process." Anthropic's title for the final rung is literally "20XX?".

What they do show is narrower and still large: the execution half of research has largely been automated, and the selection half has not. Anthropic's own framing is that doing now costs almost nothing in human time, while choosing still does.

The honest uncertainty is whether rung 4 is a different kind of problem or just a later one. Anthropic argues it is later, and their argument is decent: paradigm shifts are rare, most progress is incremental — scale something, see what breaks, fix it — and that loop is exactly what is being automated. "We see perspiration becoming increasingly automated."

The counter-argument is that the measurements supporting rung 3 are the weakest in the set — n=129 on deliberately-selected moments, and one demonstration that did not transfer to production scale — and that the evidence which might settle it is the evidence that was redacted.

That is not a comfortable place to end. But I would rather hand you an uncomfortable summary with the gaps marked than a clean one with the gaps painted over.

---

Source notes

Verified at the primary sources. All internal Anthropic figures — code share, lines per engineer, session success, code-quality position, automated reviewer retrospective, kernel speedup, judgement test, weak-to-strong demonstration, task horizons, employee poll — are from The Anthropic Institute, When AI builds itself. All acceleration figures, confidence intervals, robustness ranges, redaction statements and process timeline are from METR's assessment, 5 November 2026. Both read in full.

Derived, not quoted. The 26% figure for November 2025 session success is my subtraction: Anthropic reports 76% in May 2026, "up 50 percentage points in six months," and gives no explicit starting value. The "low single digits" bar in the code-share chart is rendered at 3% because Anthropic gives a phrase rather than a number; the chart footer says so. The four-rung table is my restructuring of Anthropic's seniority analogy, not a framework they publish as such.

Not verified, and unverifiable by design. A majority of the evidence behind METR's risk conclusions is redacted. I cannot check it, METR cannot publish it, and no third party can reconcile it against other sources. Every statement in section 4 about what the redacted material contains comes from METR's own description of it. This is the central limitation of the entire topic right now, and I would rather state it twice than have it read as a footnote.