MemoHarness: Agent Harnesses That Learn from Execution Experience

A production agent is not just a language model with tools. It is a model embedded in a control layer that decides what context to assemble, which tools to expose, how many calls to make, what state to retain, how to decode, and how to validate the result. That control layer—the agent harness—often determines whether the same model behaves like a disciplined operator or an enthusiastic source of expensive mistakes.

MemoHarness: Agent Harnesses That Learn from Experience treats the full harness as an optimization target. It decomposes the control layer into six editable dimensions, records case-level execution diagnoses and distilled global patterns in a dual-layer experience bank, searches for a global harness at training time, and adapts that harness once per unlabeled test case using retrieved experience.

Under the paper's protocol, MemoHarness improves Terminal-Bench from 0.722 to 0.806, LiveCodeBench from 0.900 to 0.967, and FinanceAgent from 0.600 to 0.767. A harness searched with GPT-5.3-Codex also improves all six additional base models evaluated on Terminal-Bench, for a reported mean gain of +0.098. The result is promising, but narrower than the phrase “self-improving agent” might suggest: the primary Terminal-Bench split contains 18 held-out tasks, the paper reports point estimates without significance tests, and it does not fully attribute gains to individual components.

Harness optimization beyond prompt search

MemoHarness defines six control surfaces along the temporal flow of inference:

1. Context assembly: instructions, constraints, retrieved evidence, examples, and compression.
2. Tool interaction: tool availability, retrieval timing, top-k, and reranking.
3. Generation control: token budgets, temperature, and candidate sampling.
4. Orchestration: single-call execution versus plan/execute/refine or other multi-call topologies.
5. Memory management: what persists across calls, what is summarized, and what stale state is removed.
6. Output processing: answer extraction, schema validation, selection, and fallback behavior.

The decomposition matters because harness edits are coupled. Increasing retrieval changes context length; that can alter useful generation budgets, orchestration depth, memory policy, and output validation. A scalar benchmark score says whether a run succeeded, but not which dimension caused the failure.

MemoHarness therefore optimizes structured edits rather than a monolithic prompt. Each execution produces a trajectory containing the prediction, model-tool trace, call count, token usage, latency, reward, and a diagnosis that identifies a primary failure dimension, secondary contributors, and a natural-language analysis.

The two-phase system

Figure 1 from the paper, format-converted for web display. The label “distilled knwoledge” is a typo present in the source figure.

Phase A: training-time harness search

Search starts from a deliberately minimal harness: no demonstrations, no structured instruction scaffolding, no external tools, deterministic single-call decoding, no cross-call memory, and raw-output passthrough. This makes each added component empirically attributable in principle, although the reported experiments do not provide full component ablations.

At every iteration, a controller retrieves a bounded slice of accumulated experience, proposes a new six-dimensional harness, executes it across labeled search cases, scores the outputs, and stores the resulting diagnoses.

The experience bank has two typed layers:

• Case-level entries preserve case features, the harness delta, trajectory, reward, token cost, and diagnosis.
• Global patterns distill recurring failure clusters into reusable statements about what works, what fails, and how dimensions interact.

Candidate selection is lexicographic: maximize task reward first, then minimize token usage only among reward ties. Monetary cost is computed offline and does not participate in search-time selection. This is a sound systems choice because cheap incorrect executions should not dominate expensive correct ones merely through a blended objective.

Phase B: one-shot case adaptation

At evaluation time, MemoHarness freezes the experience bank and starts from the validation-selected global harness. For each unlabeled case, it retrieves:

• top-k similar successful entries;
• top-k similar failed entries;
• feature-conditioned historical slices;
• distilled global patterns.

A controller maps the global harness and this evidence to a case-specific harness, which is executed once. The adaptation does not use the unknown reference output, test-time feedback, gradient updates, or a new search loop.

This distinction is important. MemoHarness is adaptive at test time, but the reported method is not continuously learning from live outcomes during the current evaluation. If labels arrive later, traces may optionally be appended for future reuse; no current-run re-selection or distillation occurs.

Main benchmark results

The paper evaluates three task families:

• Terminal-Bench for long-horizon shell agency;
• LiveCodeBench for single-shot code generation;
• FinanceAgent for multi-step financial analysis.

The final validation-selected harness improves over base on all three:

• Terminal-Bench: 0.722 → 0.806;
• LiveCodeBench: 0.900 → 0.967;
• FinanceAgent: 0.600 → 0.767.

The search curves support a headroom interpretation. FinanceAgent continues improving over ten rounds and reaches a 65.0% in-search peak around iterations 8–9. LiveCodeBench stays within a narrow 91.2%–95.0% per-iteration band because the base model is already near its task ceiling.

The final checkpoint is not always the maximum score observed during search: LiveCodeBench touches 1.000 at iteration 3, and Terminal-Bench reaches 0.833 at iteration 4. The reported final harness is selected using validation rather than the held-out evaluation peak, which reduces a common benchmark-selection failure mode.

On Terminal-Bench, MemoHarness reaches 0.806 versus 0.722 for Codex, the strongest baseline under the paper's protocol. The +0.084 difference should not be read as a clean model-controlled scaffold ablation for every baseline. The authors explicitly note that product-specific systems may expose different model, tool, or runtime surfaces; those comparisons use the closest reproducible released configuration.

Cross-dataset transfer is positive but selective

The paper transfers harnesses learned from Terminal-Bench, FinanceAgent, and LiveCodeBench to six unseen suites using a shared GPT-5.3-Codex baseline.

The Terminal-Bench-derived harness gives the broadest lift:

• MMMLU: 0.818 → 0.848;
• StrongReject: 0.879 → 0.909;
• SWE-Bench Pro: 0.706 → 0.765.

It leaves HumanEvalFix and Reasoning-Gym-Easy unchanged at already saturated scores and moves LawBench only from 0.675 to 0.676. The LiveCodeBench harness improves MMMLU and StrongReject but reduces LawBench from 0.675 to 0.669. This is not a universal “better prompt”; transfer depends on the source search task and target suite.

The broadest transfer coming from a long-horizon, tool-centric source suggests that control policies for gathering context, managing state, and finalizing software tasks may generalize better than benchmark-specific code-generation edits. That is a plausible mechanism, not a component-level causal result—the paper does not fully ablate the transferred harness.

Cross-model transfer: the most operationally interesting table

Table 3 from the paper, cropped and format-converted; higher is better.

The Terminal-Bench harness searched with GPT-5.3-Codex is applied directly to six additional models without re-search:

• GPT-5.3-Codex: 0.722 → 0.806;
• Claude Sonnet 4.6: 0.530 → 0.583;
• Gemini 3.1 Pro: 0.611 → 0.694;
• Qwen3.5-397B-A17B: 0.444 → 0.528;
• GLM-5: 0.500 → 0.733;
• GPT-4.1: 0.500 → 0.538;
• DeepSeek-V3.2: 0.333 → 0.444.

The reported average gain is +0.098, ranging from +0.038 on GPT-4.1 to +0.233 on GLM-5.

This is evidence that some learned harness behavior can survive a base-model change. It does not prove model-agnostic universality: the task remains Terminal-Bench, the model set is small, and there is no significance analysis. Still, the pattern is strategically important. It implies that organizations may be able to preserve operational knowledge—retrieval rules, planning topology, state hygiene, validation—while changing the underlying model.

Cost: cacheability is part of the algorithmic story

On the 18-task Terminal-Bench held-out split, MemoHarness consumes 14.18 million input tokens, versus 8.23 million for Codex. The experience bank increases raw context substantially.

The reported dollar result is favorable only because most of that context is cacheable:

• 13.32M cached input tokens;
• 0.86M non-cached input tokens;
• 0.22M output tokens;
• $6.89 reported total cost.

Under the same accounting, Codex costs $10.28 and Claude Code $9.51. Terminus ($6.68) and OpenCode ($2.34) remain cheaper but have substantially lower task success in this comparison.

Cache behavior is therefore not an implementation footnote. It is part of the feasibility condition. A production deployment should measure prefix stability, cache-hit rate, invalidation frequency, retrieval ordering, and provider-specific cache pricing before assuming the paper's cost profile transfers.

A useful deployment rule is to separate experience into:

• a stable, cache-friendly global prefix;
• a small case-specific retrieved suffix;
• compact structured statistics rather than repeated raw traces;
• explicit context-budget and cache-hit telemetry.

Without that discipline, an adaptive harness can become an expensive scrapbook.

Limitations and unresolved attribution

The paper is careful about several boundaries:

1. The primary Terminal-Bench result uses an 18-task held-out split.
2. It reports point estimates rather than confidence intervals or significance tests.
3. Not every baseline is a pure scaffold transplant over an identical model/runtime surface.
4. It does not fully isolate the experience bank, global patterns, and case-specific adaptation across all settings.
5. Cost depends on observed cacheability.
6. The controller and diagnostic operators use practical heuristics rather than learned general controllers.

These constraints affect how the headline should be phrased. “Execution experience can improve adaptive harnessing under this protocol” is supported. “Agents can autonomously learn better operating systems from deployment” is not yet established.

There is also an operational safety question the paper leaves largely open. A system that adapts tool access, orchestration, memory, or output behavior per case needs immutable permission boundaries. Learned harness edits should operate inside an allowlisted policy envelope; they should not expand credentials, weaken approval gates, suppress validation, or persist untrusted content merely because similar historical cases appeared successful.

Engineering takeaways

A practical MemoHarness-inspired implementation would include:

1. A typed harness manifest with independently editable context, tool, generation, orchestration, memory, and output policies.
2. Per-run telemetry containing task features, policy diff, tool trace, outcome, cost, latency, and failure diagnosis.
3. Separate storage schemas for raw case experience and distilled patterns.
4. Validation-based harness promotion rather than selecting the best observed test checkpoint.
5. Reward-first ranking with explicit feasibility constraints and cost as a secondary objective.
6. Retrieval of both successful and failed neighbors; failures often contain the better policy boundary.
7. Immutable security and authorization constraints outside the editable harness space.
8. Cache-aware serialization with stable ordering and measured hit rates.
9. Counterfactual and component ablations before attributing gains to memory, retrieval, or adaptation.
10. Shadow evaluation and rollback before online deployment.

The deepest idea is not that an agent should memorize more text. It is that execution history should update the policy that decides how the model works. A successful trace is evidence about which control configuration fit a case; a failed trace is evidence about where context, tooling, state, orchestration, or validation broke.

Conclusion

MemoHarness reframes agent improvement from prompt editing to control-layer learning. It searches a structured six-dimensional harness, accumulates diagnostic experience, and specializes a global policy to each new case without test-time labels or extra search.

The reported gains suggest that model capability and harness quality are partially separable assets. Better models matter, but organizations may also accumulate reusable operational knowledge about how models should gather evidence, invoke tools, manage state, and validate outputs.

The current evidence is preliminary: small held-out evaluation, incomplete attribution, system-level baseline differences, and cache-dependent economics. Within those boundaries, the paper makes a credible case for an important direction: the next generation of agents may improve not only by changing what they know, but by learning how they should work.

Source

• Yue Huang et al., MemoHarness: Agent Harnesses That Learn from Experience, arXiv:2607.14159v1, 14 July 2026.
• Reference implementation linked by the paper: HowieHwong/MemoHarness.
• Figure 1 and Table 3 are reproduced from the paper with crop/format conversion only.