TTPO for Agents: A Governed Architecture for Test-Time Adaptation

Test-time compute usually changes an answer, not the model. An agent may sample several trajectories, call tools, critique its work, and select a winner, but its parameters remain fixed after the request.

Test-Time Policy Optimization (TTPO) crosses that boundary. It converts multiple rollouts into pseudo-supervision and performs gradient updates during test-time training. That makes TTPO relevant to agent builders — and materially more dangerous than a prompt-level reflection loop.

What TTPO actually changes

For each problem, the paper samples multiple trajectories and extracts their final answers. The majority answer becomes a provisional pseudo-label. Rollouts are divided into:

• positives whose final answer matches the pseudo-label;
• negatives whose final answer disagrees.

The two groups receive asymmetric updates.

Positive trajectories use on-policy self-distillation with forward KL. Token weights emphasize positions where the student is uncertain or diverges from the answer-conditioned teacher.

Negative trajectories use GRPO penalties. A token mask selects the half of the trajectory most associated with anomalous, confidently generated errors, reducing damage to locally correct reasoning inside an unsuccessful solution.

The authors evaluate Qwen3 models with LoRA updates. Their test-time configuration samples 64 trajectories per problem, selects eight for the gradient update with a balanced positive-negative split, and uses a 0.1 weight for the RL branch.

On AIME 2026, HMMT 2026, and BRUMO 2025, Qwen3-1.7B improves from 38.0 to 45.2 average accuracy. Qwen3-4B with TTPO reaches 61.1, slightly above the Qwen3-8B base at 60.7 under the reported protocol.

These are meaningful results, but they come from mathematical tasks with extractable final answers. Agent systems rarely receive supervision that clean.

The unsafe translation

A naive agent implementation would look like this:

production task
  -> sample trajectories
  -> majority vote
  -> update production weights
  -> continue serving

That design collapses proposal, evidence, training, evaluation, and deployment into one trust domain. A correlated error can become a pseudo-label; the pseudo-label can become a gradient; the gradient can change future tool behavior before anyone notices.

Agreement is not independence. Sixty-four samples from one policy share training data, representations, prompt context, and systematic shortcuts. A strong majority can still be a single correlated failure mode.

A governed adaptation plane

Production TTPO should run in a separate adaptation plane:

versioned base model
  -> isolated rollout workers
  -> consensus + correlation analysis
  -> deterministic / external verifier
  -> candidate LoRA adapter
  -> held-out task evaluation
  -> safety + permission regression suite
  -> shadow / canary deployment
  -> promote | quarantine | rollback

1. Immutable input and rollout evidence

Every adaptation job should bind to a dataset snapshot and record:

• task and environment versions;
• model, prompt, harness, tool and verifier versions;
• rollout seeds and sampling parameters;
• complete trajectories or content-addressed references;
• extracted answers and vote margins;
• policy decisions and external receipts.

Without this evidence, a candidate cannot be reproduced or audited.

2. Consensus quality, not only majority size

The router should consider vote margin, answer entropy, reasoning diversity, correlated failure signatures and verifier coverage. Low-consensus jobs should abstain, spend more verification budget, or fall back to an unchanged model.

The paper itself notes that very small sample budgets or problems where no rollout is correct degrade both TTPO branches.

3. Domain verifier boundary

Mathematics offers exact final-answer checks. Agent domains need explicit verifier contracts:

• code: compilation, tests, static analysis and sandboxed execution;
• browsing: source provenance, freshness and cross-source consistency;
• data work: schema assertions and deterministic reconciliation;
• tools: preconditions, postconditions and state receipts;
• open-ended research: human review and multiple independent evaluators.

A learned judge may supplement these checks, but should not be the only authority for a weight update that changes production behavior.

4. Candidate adapters instead of in-place mutation

TTPO in the paper uses LoRA. That maps cleanly to a safer deployment unit: create a new candidate adapter with a content hash and lineage record. Never mutate the active adapter in place.

type AdaptationCandidate = {
  baseModelHash: string;
  parentAdapterHash?: string;
  trainingSetHash: string;
  rolloutManifestHash: string;
  verifierReportHash: string;
  adapterHash: string;
};

The candidate is an artifact, not a side effect.

5. Promotion is a multi-dimensional gate

Average task accuracy is insufficient. A promotion report should include:

• held-out success and task-family transfer;
• regressions on old capabilities;
• calibration and appropriate abstention;
• policy and permission deltas;
• tool-call rate and irreversible-action rate;
• latency, token and GPU cost;
• stability across seeds;
• adversarial and prompt-injection tests.

A candidate that gains two accuracy points while becoming more willing to execute external actions should fail the gate.

6. Shadow, canary, and rollback

The first deployment stage should be shadow evaluation with no authority to affect users or external state. A successful candidate can then receive a small canary slice restricted to reversible tasks.

Promotion must remain separately authorized. Rollback must switch traffic to a known adapter without retraining, and the trigger should include both quality and behavioral-policy regressions.

Privacy and data governance

Test-time training changes the treatment of request data. A conversation that was permitted for inference may not be permitted for parameter updates.

The adaptation job therefore needs a training-use policy:

• explicit data classification and consent;
• redaction or exclusion of secrets and personal data;
• tenant isolation;
• retention and deletion semantics;
• proof that prohibited data did not enter the adapter.

A per-tenant adapter can reduce cross-customer leakage, but increases operational complexity and still requires deletion and lineage controls.

Compute economics

TTPO is not a free reasoning enhancement. The reported setup can generate up to 64 trajectories of up to 16,000 tokens per problem, then performs teacher passes, gradient updates and evaluation.

That suggests an offline or nearline role for high-value batches, not unconditional adaptation on every request. A scheduler should activate TTPO only when expected improvement exceeds rollout, training and evaluation cost.

What the paper does not establish

The authors explicitly limit experiments to mathematical reasoning with verifiable final answers. Code generation with execution feedback and open-ended reasoning with learned rewards remain future work.

The paper also observes sustainable improvement in majority and single-sample performance, but that should not be translated into unbounded autonomous self-improvement. It is a bounded training dynamic under a specified benchmark, optimizer, model family, budget and checkpoint policy.

Builder conclusions

1. TTPO is weight adaptation, not reflection. Treat it as a training and deployment event.
2. Consensus is provisional evidence. Measure correlation and require domain verification.
3. Create immutable candidate adapters. Never mutate production weights in place.
4. Separate proposal from promotion. The model may produce a candidate; an independent gate decides deployment.
5. Evaluate behavior, permissions and cost. Accuracy alone does not establish agent safety.
6. Keep rollback immediate and evidence complete. If lineage cannot be reconstructed, the update should not ship.

TTPO is exciting because it offers a path from collective rollout knowledge to improved single-sample reasoning. For agents, the mature implementation is not a self-editing loop. It is a governed adaptation service with the same discipline applied to code, models, data, permissions and production releases.

Primary source: Wang et al., “TTPO: Test-Time Policy Optimization,” arXiv:2608.27448 · Official code