Build an Evidence-First Agent Control Plane

The most consequential AI systems are becoming infrastructure. A conversational model can act as a search gateway. A small domain model can route audit cases. An agent can discover applications, invoke tools, modify its own harness, and operate a robot through speech.

That shift changes the production question. Accuracy is still necessary, but it is no longer the whole contract. Builders must also answer:

• Which source authorized this claim or action?
• Which principal granted the capability?
• What observed state did the decision depend on?
• What monitor can stop the output before it becomes a side effect?
• Can the mutation be reversed after the environment changes?

Recent primary-source releases make these requirements concrete. The EU's VLOSE designation treats ChatGPT as a systemically relevant information gateway. The FSB connects frontier-model risk to cyber resilience and critical third parties. VERA-8B routes audit conclusions only when evidence and confidence pass a contract. String stages application context and binds privilege to provenance. Speculative Probing moves monitoring into the inference path. EvoUndo tests recovery in counterfactual states. Voice-robot experiments show that an ASR error can flip a safe refusal into an executable response.

The builder lesson is not to copy any single paper. It is to assemble an evidence-first control plane.

1. Make evidence an input to authorization

Many systems collect citations after an agent has finished. That produces a readable report, but it does not prove that the evidence actually governed the action. A production control plane should bind evidence before crossing an irreversible boundary.

Represent an action request as a typed packet:

ActionRequest {
  principal
  objective
  tool
  arguments
  observed_state_hash
  evidence_refs[]
  policy_version
  risk_class
  recovery_plan
}

The authorizer should validate the packet, not the agent's prose. Evidence references should resolve to immutable snapshots or content hashes, carry a source class, and state what claim they support. A vendor benchmark, an author's experiment, a regulator's decision, and an independent replication are different evidence classes. Preserve that distinction in the schema.

This is the operational pattern behind VERA-8B's most useful idea. The paper reports strong audit metrics, but the transferable design is the evidence contract: valid structure, verified spans, calibrated routing, and abstention when requirements fail. For an agent, “abstain” becomes “do not execute; route to review.”

2. Separate discovery from capability

Tool catalogs create two risks at once: context overload and authority sprawl. Dumping every schema into every prompt consumes tokens and makes wrong-action selection more likely. Granting every discovered tool makes a prompt mistake materially dangerous.

String's staged interface suggests a better shape:

1. Keep a tiny resident discovery surface.
2. Open only the application state required for the current objective.
3. Reveal an action schema only after the agent selects the application and intent.
4. Resolve credentials outside model-visible text.
5. Authorize the action from the caller's provenance and policy, not from content inside the opened document.

Discovery must never imply permission. A remote page may describe a shell command without acquiring shell authority. User-supplied text may name a secret without gaining access to its value. Stored application documents should declare capabilities, but a separate policy engine should decide whether the current principal can use them.

The String benchmark reports large reductions in wrong-action selection and completed-episode token use. Treat those numbers as author-reported results, then benchmark your own catalog. The architectural goal is measurable regardless of the exact win: lower prompt residency, fewer candidate actions, and a smaller authorization surface.

3. Put monitors before effectors

Post-hoc moderation is too late when output directly feeds an API, trading system, robot, or infrastructure command. The control plane needs an inline monitor between model output and the effector.

Speculative Probing explores a low-overhead implementation by reusing multi-token prediction machinery, soft prompts, and existing KV cache. Whether or not that exact method fits your stack, the placement principle is critical:

model output
  -> structural validator
  -> semantic/risk monitor
  -> policy decision
  -> tool adapter
  -> side effect

Each stage should fail closed for its own class of error. Structural validation rejects malformed arguments. The semantic monitor detects unsafe intent, data leakage, or suspicious trajectory changes. Policy checks principal, resource, amount, environment, and approval. The adapter then applies final deterministic bounds.

Measure the monitor as a production component. Track false negatives, false positives, calibration drift, added p50/p95 latency, GPU memory pressure, and behavior under adversarially long context. A cheap monitor that silently misses the dangerous class is not a safety improvement.

4. Test rollback after the world has changed

Most rollback tests are local: apply a change, immediately undo it, declare success. Self-modifying agents make that test insufficient. A new tool can register an event listener. A prompt change can alter later state. Middleware can rewrite identifiers. A resource allocation can create external dependencies. The inverse operation that worked at creation time may no longer address the same state.

EvoUndo's counterfactual framing gives builders a concrete promotion gate. Before accepting an agent-generated mutation:

1. Snapshot the relevant harness and external identifiers.
2. Apply the candidate in an isolated environment.
3. Verify the intended capability gain.
4. Transition the environment through adversarial but valid states.
5. Execute the recovery plan from each state.
6. Verify semantic restoration, not only command success.
7. Reject the mutation if any required state is unrecoverable.

Recovery needs both grounding and expressivity. Grounding means identifying the exact object, version, listener, permission, or resource to restore. Expressivity means the rollback language can represent the required operation. An “undo” command with no stable address is theater; a perfect address is useless if the recovery API cannot express the inverse.

Keep learned skills and harness mutations quarantined until these tests pass. Promotion should record the generator, evidence, policy version, test matrix, rollback artifact, and expiry conditions.

5. Route uncertainty to a real review queue

Human-in-the-loop is often implemented as a banner that nobody owns. A useful review path has a queue, service-level objective, evidence packet, explicit decision choices, and a durable result.

Use calibrated thresholds to create three lanes:

• auto-allow when evidence, confidence, policy, and reversibility all pass;
• auto-deny when deterministic policy or safety constraints fail;
• review for uncertain, novel, high-impact, or weakly evidenced cases.

Reviewers should see the proposed action, material arguments, source excerpts, model rationale only as supporting context, relevant policy, and the recovery plan. They should not need to reconstruct the entire agent transcript.

Do not optimize automatic coverage in isolation. VERA-8B reports 83.2% automatic coverage on its frozen test set, but your threshold should be selected from the cost of false authorization and false denial. In high-impact domains, a lower automation rate with auditable evidence can be the better system.

6. Treat model gateways as critical infrastructure

The FSB warning is written for financial stability, yet the engineering implication generalizes. Frontier models are frequently delivered through a small number of model and cloud providers. An outage, compromise, policy change, quota event, or correlated model failure can affect many downstream agents at once.

Build resilience at the dependency boundary:

• inventory model, inference, identity, storage, and telemetry providers;
• define degraded modes that remove capability instead of guessing;
• keep deterministic procedures for critical operations;
• rehearse credential revocation and provider isolation;
• preserve evidence and audit logs outside the affected model path;
• test restoration from clean infrastructure;
• measure concentration risk across ostensibly separate products.

Failover to another model is not automatically safe. The fallback may interpret tools differently, violate calibration assumptions, or lack the monitor used by the primary path. Qualify fallback behavior with the same trace replay and policy tests as the primary.

7. Reconfirm intent after speech recognition

Voice interfaces add a lossy, probabilistic layer before the model. The robot study reports that simulated ASR errors increased unsafe acceptance under its benchmark protocol and reduced ordinary task completion. The transferable lesson is that a transcript is an observation, not authenticated intent.

For consequential voice commands:

1. Preserve audio provenance and ASR confidence.
2. Detect ambiguous entities, negation, quantities, and destructive verbs.
3. Restate the interpreted action through a separate confirmation channel.
4. Require stronger confirmation as impact rises.
5. Apply independent physical or software interlocks.
6. Record what the user confirmed, not only what ASR first produced.

The language model should not be the final safety controller. Robot speed limits, workspace boundaries, financial limits, destructive-operation approvals, and emergency stops belong in systems the model cannot rewrite.

A minimal production blueprint

An evidence-first control plane can be built as eight cooperating services:

1. Context broker for staged discovery and bounded retrieval.
2. Provenance ledger for immutable source and observation references.
3. Capability broker for principal-scoped, short-lived tool grants.
4. Output monitor before any effector.
5. Policy engine with deterministic high-impact rules.
6. Review router with owned queues and durable decisions.
7. Mutation workshop for quarantine, replay, and counterfactual recovery tests.
8. Recovery controller outside the worker's authority.

Instrument the boundaries. Log which evidence authorized each action, which catalog entries entered context, which monitor fired, which policy version decided, which human reviewed, and whether recovery was exercised. Then replay representative traces whenever the model, prompt, tool schema, monitor, policy, or provider changes.

The defining property of a trustworthy agent is not that it never makes a mistake. It is that the system can attribute the decision, constrain the capability, detect the dangerous output, escalate uncertainty, stop the effect, and restore a known state. Intelligence lives in the worker. Trustworthiness lives in the control plane.