Trustworthy AI Agents Need Authorization, Provenance, and Calibrated Uncertainty

A capable model can still be an unsafe agent. It may follow an instruction embedded in tool output, treat a polished but fabricated table as evidence, optimize toward an incorrect consensus, miss low-probability physical outcomes, lose its route over a long horizon, or repeat weak usage patterns after a one-off training course.

Six studies released together make the same systems point from different directions: reliability cannot live entirely inside the model. It needs explicit controls around proposal, evidence, authorization, uncertainty, simulation, recovery, and learning.

1. Separate action proposals from execution authority

Prompt-injection defenses often focus on classifying malicious text. That is useful, but incomplete. An agent does not cause harm merely by reading a hostile instruction. Harm occurs when an untrusted influence crosses the boundary into a consequential tool call.

SARA treats that boundary as an authorization problem. The system tracks the provenance of action suggestions and permits execution only when the proposed action is supported by the user's objective and audited evidence. On AgentDojo and AgentDyn, the authors report attack-success rates no higher than 0.63% across four main configurations while retaining competitive utility.

Figure 1 from the SARA paper. Results are author-reported preprint evidence.

The architectural lesson is stronger than the exact benchmark number:

untrusted observation
  -> model proposes intent
  -> normalize proposed action
  -> collect provenance and policy evidence
  -> independent authorization decision
  -> execute or deny
  -> immutable audit record

The model should not be both advocate and judge. A tool call that sends money, modifies production, publishes content, changes access, or sends a message needs a separately enforceable policy decision.

A practical authorization object might include:

type AuthorizationRequest = {
  userGoal: string;
  proposedAction: CanonicalToolCall;
  supportingEvidence: EvidenceRef[];
  evidenceTrust: number[];
  permissionScope: string[];
  reversibility: "reversible" | "external" | "irreversible";
  expectedStateChange: StateAssertion[];
};

Natural-language intent is still valuable, but it should be compiled into a canonical action before policy evaluation. Otherwise a model can reframe the same side effect in language that bypasses a brittle classifier.

2. Professional presentation is not provenance

A second study tested 12 frontier models on questions whose outcomes were not actually predictable. Adding professional-looking market tables increased directional decisions from 6.5% to 54.0%.

The alarming result is not merely that data influences models. Completely fabricated tables raised commitment from 24.5% to 36.8%, nearly matching genuine data at 37.6%. The visual and structural cues of evidence were enough to induce action even when epistemic support was absent.

Figure 1 from the original paper.

This failure mode appears in real agent systems whenever a dashboard, PDF, spreadsheet, retrieved web page, or API response is accepted because it looks authoritative.

An evidence layer should therefore represent at least:

• origin and retrieval time;
• content hash and transformation history;
• whether the source is primary, derived, or user-supplied;
• schema and unit checks;
• corroborating and contradicting sources;
• freshness and applicability conditions;
• confidence that is separate from presentation quality.

Most importantly, the action policy needs an abstention state. “Insufficient evidence” is not a failed completion; for consequential tasks it is often the correct output.

3. Test-time learning needs disagreement-aware gates

Test-Time Policy Optimization (TTPO) explores how a model can improve during inference without labeled answers. It samples multiple solutions, uses majority outcomes as provisional supervision, distills trajectories in agreement, and penalizes confidently wrong behavior in disagreement cases.

The asymmetric treatment matters. Naive majority voting can turn a correlated error into a pseudo-label and reinforce it. Disagreement is not noise to delete; it is a signal that the system should reduce confidence or spend more verification budget.

The authors report that TTPO matches supervised self-distillation across five competition-math benchmarks without labels. For Qwen3-1.7B, test-time training increases from 38.0% to 45.2%; no-chain-of-thought settings improve from 25.2% to 36.4% depending on the task.

Figure 2 from the TTPO paper.

For agents, a safe extension would keep adaptation behind a reversible candidate boundary:

1. sample independent trajectories;
2. measure agreement and shared failure correlation;
3. verify outputs with deterministic checks where possible;
4. create an adaptation candidate;
5. evaluate on held-out and safety tasks;
6. promote only if the candidate passes all gates.

An agent should never rewrite production policy merely because several samples agree. Models trained similarly can agree for the same wrong reason.

4. World models must cover distributions, not produce one plausible future

A video can look physically convincing while representing the wrong probability distribution. PAWBench repeats the same physical initial condition and evaluates whether generated outcomes match both the probabilities and the diversity of valid reference behaviors.

Across 50 scenarios and 11 systems, no evaluated model consistently matches the reference probabilities while covering the full set of valid behaviors. Prompt changes, initialization noise, and training interventions can shift the distribution, but the benchmark exposes a deeper limitation: visual plausibility is not probabilistic calibration.

Figure 1 from PAWBench.

This matters for robotics, autonomous driving, logistics, and safety simulation. A planner that ignores rare but dangerous futures may look excellent in a demo and fail exactly where risk management matters.

Production evaluation should therefore include:

• coverage of valid outcome modes;
• calibration against observed frequencies;
• explicit tail-risk scenarios;
• counterfactual and perturbation tests;
• separation between visual quality and predictive validity.

5. Embodied agents need long-horizon recovery

UrbanGround places multimodal agents in a city-scale 3D reconstruction of Hong Kong. Agents navigate from first-person observations and maps while handling blocked roads and moving pedestrians.

Current models perform relatively well on scene recognition and short-range spatial reasoning, but orientation errors accumulate over long routes and recovery is weak. Dynamic rerouting and pedestrian motion remain especially difficult.

Figure 1 from UrbanGround.

The corresponding control pattern is a hierarchical loop rather than one uninterrupted trajectory:

mission objective
  -> route plan
  -> short-horizon action segment
  -> localization assertion
  -> progress and anomaly check
  -> continue | replan | recover | request help

Long-horizon success depends less on never making an error than on detecting drift early and recovering before local mistakes compound.

6. Training must be embedded in the workflow

A workplace study analyzed 713,564 prompts from nearly 4,000 back-office workers across 15 departments over eight months. Senior employees used GenAI more sophisticatedly, consistent with domain expertise complementing model capability.

However, the authors did not observe clear improvement over time or durable changes after formal AI training. Strategy, digital innovation, and project-management teams showed the most sophisticated use, but a single course did not create a sustained organization-wide learning curve.

Figure 4 from the workplace study.

For enterprise agents, enablement should be part of the execution environment:

• task-specific templates and examples;
• inline quality checks;
• feedback tied to actual deliverables;
• reusable skills with validation;
• team-level review of failures and successful patterns;
• metrics for outcome quality, not prompt volume.

A production control plane

Taken together, the studies suggest a layered architecture:

User intent
  -> scoped plan
  -> evidence retrieval + provenance
  -> model proposal
  -> uncertainty and disagreement checks
  -> deterministic validation
  -> independent authorization
  -> sandboxed execution
  -> state and artifact assertions
  -> audit, feedback, and governed learning

Each layer addresses a different failure class. Provenance does not replace authorization. Authorization does not prove the evidence is true. A simulator does not guarantee calibrated tails. A successful action does not prove the agent learned a reusable skill.

The completion record should therefore contain more than a final status:

type CompletionEvidence = {
  intendedChange: StateAssertion[];
  observedChange: StateAssertion[];
  artifactHashes: string[];
  externalReceipts: string[];
  authorizationDecision: string;
  evidenceRefs: string[];
  unresolvedUncertainty: string[];
};

Operational conclusions

1. Models propose; policy authorizes. Keep consequential execution behind an independent, enforceable boundary.
2. Evidence needs provenance, not visual polish. A professional table can be fabricated and still alter model behavior.
3. Disagreement should increase verification. Consensus is useful but can be correlated and wrong.
4. Evaluate distributions and recovery. One plausible future and one successful route are not enough.
5. Learning requires governed feedback loops. One-off training and unvalidated self-modification do not create durable improvement.

The broader engineering principle is simple: agent trustworthiness is a systems property. Intelligence helps generate better proposals. Reliability comes from the controls that decide what those proposals are allowed to become.

Primary sources: SARA · Fabricated evidence study · TTPO · PAWBench · UrbanGround · Workplace GenAI study