Harness-of-Harness: An Outer Control Loop for Continually Improving Coding Agents

A coding agent can be excellent at one repository task and still fail at autonomous software development. The difference is not merely duration. A multi-day build must preserve earlier decisions, select the next useful increment, carry forward verified behavior, expose regressions, and distinguish “the developer says it works” from acceptance evidence.

Harness-of-Harness (HoH), introduced by researchers at Shanghai AI Laboratory, treats this as a control-system problem. It wraps an existing coding harness in a repeated planning → development → independent quality assurance loop. The model, base harness, role definitions, and runtime policy remain fixed within a run. The software artifact, development document, and execution evidence evolve.

That distinction matters: HoH demonstrates continual project improvement, not online model training or autonomous weight updates.

1. The unit of autonomy is a verified increment

An unbounded instruction such as “keep improving the product” leaves too much implicit. The worker can widen scope, rediscover old failures, polish what already works, or declare completion against tests shaped by its own implementation.

HoH turns a long project into bounded loops. Each loop chooses one coherent objective that is:

• bounded, so unrelated behavior is excluded and failures remain localizable;
• locally complete, so all interdependent work required for one observable capability is included;
• verifiable, with acceptance conditions stated before implementation;
• preservative, identifying previously validated behavior that must not regress.

This is a better atomic unit than a fixed number of files, tool calls, or tokens. A feature can span several modules and still be one increment if those changes jointly create one testable behavior.

A practical contract might look like this:

DevelopmentDocument {
  objective
  included_tasks[]
  excluded_scope[]
  preservation_requirements[]
  observable_acceptance_criteria[]
  starting_revision
}

The document constrains the outcome, not the worker’s chain of thought or tool sequence.

2. Separate the three decisions that a long build must make

Every development loop contains three fundamentally different decisions:

1. What should change next?
2. How should the change be implemented?
3. Does the resulting artifact satisfy observable requirements?

Collapsing these decisions into one unconstrained agent invocation creates a conflict of authority. The implementer both chooses the target and judges whether its own work is acceptable. HoH assigns them to three role-scoped invocations of the same fixed harness–model pair.

Project Planner: read broadly, write no code

The Planner combines the global specification, prior execution evidence, and a read-only view of the current artifact. It produces the next development document but cannot modify the workspace.

Its job is not to generate a backlog mechanically. It must reconcile capability growth, open defects, regressions, and preservation requirements into one coherent increment. Freezing the first plan is insufficient because test evidence changes what the project needs next.

Developer: the single writer

The Developer warm-starts from the current artifact and is the only role allowed to change it. Inside the authorized objective, it remains free to select architecture, tools, implementation strategy, and debugging sequence.

The single-writer boundary gives the transition from revision A(t−1) to A(t) an unambiguous owner. The Planner may inspect the starting artifact; QA may inspect and execute the result; neither can silently alter it.

The Developer also performs shift-left testing: establish a baseline, make a meaningful change, rerun the affected path, and inspect adjacent regression surfaces. These checks answer whether the candidate is ready for assessment. They do not grant acceptance.

QA Tester: assess a frozen candidate

The QA Tester receives a frozen, read-only candidate, the global specification, the current development document, and deterministic runtime checks. It evaluates the exact revision that the Developer produced.

Black-box observations test user-visible behavior, state transitions, and end-to-end flows. White-box inspection examines source, configuration, resource bindings, runtime state, and logs. A criterion is verified only when candidate-bound records support it. Missing evidence is a gap, not a pass.

Read-only QA is more than procedural neatness. If the evaluator can patch the candidate during assessment, evidence can silently combine multiple versions and the lineage of the accepted artifact becomes ambiguous.

3. Runtime-enforced roles are stronger than role-playing prompts

HoH uses role-specific prompts, but the deterministic Runtime is the real control boundary. It determines:

• which inputs each invocation may access;
• which tools and write operations it may use;
• which artifact version is under evaluation;
• which structured output schema must be returned;
• whether malformed output triggers a retry.

This leads to a general production rule: do not implement independence as three labels attached to agents with identical privileges. Enforce different capabilities.

A minimal permission matrix is:

Promotion should remain a runtime or supervisor decision based on evidence, not an extra privilege quietly given to the worker.

Figure 3 from the paper: the Runtime freezes role inputs, enforces permissions, and binds QA evidence to a specific candidate across the Planner–Developer–QA loop.

4. Carry artifact state and evidence state separately

The latest codebase is not a complete project memory. It records what exists, but not why a decision was made, which claims were verified, which failures remain unresolved, or what must be preserved.

HoH therefore carries two states across loop boundaries:

ArtifactState A(t) {
  source
  configuration
  resources
  project_metadata
  revision
}

EvidenceState E(t) {
  verified_behaviors[]
  unsupported_claims[]
  observed_failures[]
  regressions[]
  unmet_requirements[]
  evidence_records[]
}

The Planner uses specification S, evidence E(t−1), and a read-only view of artifact A(t−1) to produce plan D(t). The Developer transforms A(t−1) into A(t). QA evaluates the frozen A(t) and produces E(t).

D(t) = Planner(S, E(t−1), read_only(A(t−1)))
A(t) = Developer(A(t−1), S, D(t))
E(t) = QA(read_only(A(t)), S, D(t), Runtime.check(A(t)))

Artifact continuity makes development incremental. Evidence-conditioned planning makes it iterative.

5. Progressive disclosure is a context architecture

HoH does not rely on a dedicated memory module. Plans, reports, histories, tools, and skills are persisted as artifacts and initially exposed through a concise categorized index. Details are retrieved only when relevant.

This avoids two common failures:

• sending every historical transcript back into every role invocation;
• compressing all history into one lossy summary that gradually becomes accepted as truth.

For a production implementation, keep an index such as:

ProjectIndex {
  active_revision
  current_plan_ref
  latest_evidence_ref
  open_issue_refs[]
  preservation_refs[]
  available_tool_catalog
  available_skill_catalog
  prior_iteration_refs[]
}

Large build logs, screenshots, videos, and test traces should remain separate evidence objects. The model sees identifiers and summaries first, then retrieves only the records needed for its current role.

6. Bind evidence to a candidate identity

“Tests passed” is not durable evidence unless it names what was tested. HoH freezes the QA candidate so every observation corresponds to one software state.

A production evidence packet should include:

EvidencePacket {
  candidate_revision
  objective_id
  deterministic_checks[]
  black_box_observations[]
  white_box_observations[]
  verified_claims[]
  observed_gaps[]
  regression_results[]
  environment_fingerprint
  collected_at
}

This prevents a familiar long-horizon failure: a build result from revision X, a screenshot from revision Y, and a developer narrative about revision Z are blended into one completion claim.

Evidence should also have scope. A successful production build supports “revision X compiles under environment Y.” It does not support “the live deployment works.” That requires a live probe. A screenshot supports rendering at one viewport, not universal usability.

7. What the benchmark results actually show

The authors evaluated HoH on three benchmarks with three harness–model pairs:

• Codex CLI + GPT-5.5 at high reasoning effort;
• OpenCode + DeepSeek-V4-Pro;
• Pi Coding Agent + MiniMax-M3.

After three HoH iterations, every pair outperformed its Vanilla counterpart across GameCraft-Bench, FrontierSWE, and ProgramBench.

On GameCraft-Bench Overall score:

ProgramBench average test pass rate improved from 60.41 to 66.50 for Codex, 45.27 to 57.56 for OpenCode, and 35.83 to 52.68 for Pi. FrontierSWE dominance improved by 19–29 percentage points after three loops depending on the configuration.

The headline reported by the authors is an average relative gain of 52.25% and a maximum relative gain of 82.86% after three iterations. Absolute scores are more useful for architecture decisions because relative gains can look large from low baselines.

8. More passes help, but the protocol adds value beyond repetition

A fair objection is that HoH calls the coding harness repeatedly, so perhaps it wins simply by spending more inference.

The paper compares HoH with Vanilla Continuation under matched development-pass counts using Codex + GPT-5.5 on GameCraft-Bench:

HoH@2 exceeds three-pass Vanilla Continuation while using fewer recorded tokens. HoH@3 uses more tokens and achieves the highest score. The result supports a protocol effect, not free improvement: structured iteration is more effective than merely continuing, but additional quality still consumes substantial inference.

Figure 6 from the paper compares three GameCraft-Bench artifacts across Vanilla and HoH@1–3, illustrating gains in implemented mechanics, content depth, functional visuals, and presentation.

Provider token accounting includes possible cached reads and differs across harnesses. The authors explicitly limit token comparisons to within-configuration analysis rather than cross-provider cost claims.

9. The ablations identify the cross-loop mechanisms that matter

Using Codex + GPT-5.5 on all 45 sampled GameCraft-Bench tasks, the full HoH@3 score was 71.52. Three ablations reduced it:

• frozen initial plan: 63.39 (−8.13);
• replanning without prior execution evidence: 65.23 (−6.28);
• no artifact warm-start: 63.67 (−7.85).

Removing warm-start also increased mean cumulative tokens from 8.41M to 11.12M per task because the system repeatedly reconstructed the project.

The lesson is not just “use multiple roles.” Continual improvement depends on all three cross-loop properties:

1. revise the objective when evidence changes;
2. return observed execution evidence to planning;
3. continue from the verified artifact rather than rebuilding from scratch.

10. Long trajectories are non-monotonic

On 15 FrontierSWE tasks, Codex + GPT-5.5 continued through ten HoH loops. Under the paper’s fixed comparison pool, dominance reached 76.00% at loop 9 and 72.67% at loop 10, versus 27.33% for Vanilla. More loops helped overall, but the best checkpoint was not the final checkpoint.

The 70-loop Fusepoint case makes the same point operationally. Starting from a product requirements document, HoH used Codex CLI + GPT-5.6-Sol, Godot, Godot MCP, domain tools, and reusable skills to construct a narrative FPS. At the analysis cutoff, project records contained 81 issues: 65 closed, 16 unresolved, and 17 reopened after previously verified behavior regressed.

That is not a blemish on the central idea. It is evidence that long-horizon development needs versioned history and explicit issue state. “Continual improvement” should mean the system can accumulate capability while detecting and revisiting regressions—not that every iteration is guaranteed to dominate the previous one.

11. A production adaptation for OpenClaw-like agent systems

A practical implementation can preserve HoH’s control properties without copying its exact research setup.

Control plane

Maintain a durable run record containing the global objective, iteration budget, active revision, role contracts, allowed tools, and stop conditions. The control plane creates one bounded iteration at a time.

Planner lane

Give the Planner read-only repository access plus the specification, latest QA packet, unresolved issues, and preservation requirements. Require one development document with explicit exclusions and acceptance criteria.

Developer lane

Give the Developer a dedicated worktree or revision-scoped workspace. Allow writes only within the declared scope. Require implementation checks and a candidate commit. Do not expose deployment credentials unless deployment itself is the authorized increment.

QA lane

Create an immutable checkout of the candidate commit. Give QA execution and inspection tools but no write permission to the candidate. Include tests the Developer did not choose: hidden fixtures, independent route probes, visual checks, security rules, or a separate evaluator model where appropriate.

Promotion gate

The supervisor validates schemas, candidate identity, required evidence, and acceptance results. It then promotes, rejects, or rolls back the revision. The same worker that wrote the code should not control this decision.

Checkpoint

Emit one bounded checkpoint per role:

Status / Evidence / Files / Tests / Blocker / Next

The checkpoint is an observer view, not the source of truth. Every important claim should point to a commit, log, test result, screenshot, or live probe.

12. Add controls that the research prototype does not fully solve

HoH’s independent QA is role-independent, but the evaluated roles use the same fixed harness–model pair. Shared-model blind spots can remain correlated. A production system should add:

• deterministic tests and policy checks outside the model;
• hidden or rotating acceptance cases;
• evaluator diversity for subjective criteria;
• human approval for high-impact, destructive, privacy-sensitive, or public actions;
• scoped and expiring credentials;
• explicit iteration budgets and repeated-failure stop conditions;
• promotion rules that can preserve the best earlier checkpoint rather than defaulting to the last one.

Security boundaries must be enforced by the runtime. Prompting a QA agent to be read-only is not equivalent to mounting the candidate read-only. Asking a Planner not to deploy is not equivalent to withholding deployment capability.

13. Limitations and reproducibility caveats

This is an arXiv v1 and the reported results have not been independently replicated in the material reviewed here. GameCraft-Bench uses a stratified sample of 45 out of 140 tasks. FrontierSWE uses 15 out of 17 tasks due to compute constraints. The open-ended game case adds domain-specific tools and skills that are not present in the benchmark protocol.

The public repository provides figures, demos, and project links, but states that HoH-lite is still coming soon at the time of writing. Reproducing the complete orchestration layer therefore requires more than cloning a released reference implementation.

The framework also targets greenfield, long-running development. Its results should not automatically be generalized to regulated production changes, security-sensitive maintenance, large legacy repositories, or multi-team governance without additional evidence.

14. The durable idea is an outer loop with authority

The strongest contribution of Harness-of-Harness is not the number of agents. It is the design of an outer loop that can preserve a verified artifact, revise plans from evidence, isolate writing from acceptance, and retain enough project history to make regressions actionable.

A stronger model may improve every role. A larger context window may delay forgetting. Neither replaces the architecture.

Long-horizon coding improves when:

• work advances in observable increments;
• only one role mutates the artifact;
• QA assesses an exact frozen revision;
• evidence survives the session that produced it;
• plans change when evidence changes;
• promotion remains outside the worker;
• and the system can return to a known-good checkpoint.

The goal is not a coding agent that runs forever. It is a development system in which each additional loop has a defensible reason to exist and produces evidence strong enough to justify the next one.

---

Primary sources: arXiv:2609.01481 · Project page · HarnessOfHarness repository · Fusepoint development trajectory