Build a Durable Knowledge Supply Chain for Long-Lived Agents

Long-lived agents create value by not starting from zero. They reuse context, structured facts, memories, procedures, tool schemas, and learned skills. That reuse can dramatically reduce cost and improve consistency. It can also turn one bad interaction into a persistent system behavior.

The key engineering mistake is treating all retained knowledge as harmless text. A structured fact can change a decision. A memory can bias retrieval. A skill can invoke tools. A ranking heuristic can reshape what the system sees. Once an artifact affects future actions, it belongs in a governed supply chain.

Recent primary-source work makes the trade-off unusually clear. Agentic data cracking shows how knowledge already uncovered during expensive reasoning can be structured for cheaper future queries. EvoSkill Injection shows that self-generated skills can be poisoned, stored, retrieved, and reactivated later. BLOOM-WILT demonstrates why ordinary audits miss rare behaviors at deployment scale. CHASE shows that repeated optimization against a ranking signal can make ranking diverge from independently judged quality. Frontier model releases add more capability and longer-lived context to the same system.

The builder lesson is not “do not learn.” It is: separate learning from promotion.

1. Model retained knowledge as typed artifacts

Do not put facts, summaries, procedures, credentials, policies, and executable skills into one undifferentiated vector store. They have different risk, freshness, and review requirements.

A minimal artifact envelope should contain:

KnowledgeArtifact {
  id
  type              // fact, summary, procedure, skill, policy, index feature
  source_refs[]
  created_by
  created_at
  observed_at
  model_and_harness
  permissions[]
  test_evidence[]
  confidence
  expiry
  supersedes[]
  revocation_state
}

The envelope is more important than the prose inside it. It lets retrieval filter by artifact type, source class, time, authority, and revocation status before content reaches a worker.

Facts need freshness and source reconciliation. Procedures need environment compatibility. Skills need static review, sandbox tests, permission bounds, and recovery. Policies need a human owner and versioned approval. Ranking features need independent quality evaluation because they are optimization targets, not truth.

2. Fork context, but do not inherit trust

Agentic data cracking proposes a useful cost pattern: when an agent has already paid to open and reason over a long document, a secondary process can extract structured facts from the loaded context at marginal cost. Over time, future questions can query the structured layer instead of reopening every source.

Production systems should adopt the cost idea while tightening the trust boundary. A context fork inherits bytes and observations, not authority. Its output should enter a staging table with source spans, document hashes, extraction time, and a schema version. It should not silently overwrite canonical data.

Use three layers:

1. Raw evidence: immutable source snapshot or resolvable content hash.
2. Candidate structure: extracted entities, relations, values, and source spans.
3. Promoted knowledge: reconciled records that passed validation and freshness policy.

This shape preserves the savings without turning one hallucinated extraction into durable truth. It also allows re-extraction when the parser, model, schema, or source changes.

3. Put every generated skill in quarantine

EvoSkill Injection exposes a persistent attack surface. An adversarial trajectory can induce an agent to generate a malicious capability that appears to be a legitimate reusable procedure. Later tasks retrieve the artifact under benign conditions and activate the embedded behavior.

The defense cannot be a prompt that says “write safe skills.” Promotion must occur outside the generator's authority.

For each proposed skill:

1. Store it in a quarantine namespace that production retrieval cannot access.
2. Record the complete generating trajectory and all untrusted inputs that influenced it.
3. Parse declared tools, filesystem paths, network hosts, secret scopes, and side effects.
4. Run static checks for hidden instructions, dynamic execution, broad globs, destructive operations, and permission escalation.
5. Replay the successful case in a sandbox.
6. Add negative, adversarial, and unrelated tasks to test accidental triggering.
7. Verify cleanup and rollback.
8. Require a separate promoter — human or policy service — to sign the release.

Most importantly, separate skill retrieval from capability grant. A skill may describe how to deploy a service, but reading it should not grant shell, cloud, or production credentials. Capability tokens should be short-lived, principal-scoped, task-scoped, and issued at execution time by a policy layer.

4. Audit for behaviors that random sampling will miss

Pre-deployment tests cover far fewer interactions than real deployment. A behavior with a tiny probability can still occur routinely across millions of sessions. Random prompts and fixed red-team sets are therefore necessary but insufficient.

BLOOM-WILT uses an adaptive auditor and decoding-time logit tilting to search for rare target behaviors. Whether or not your deployment exposes next-token distributions, the operating principle generalizes: audit should have optimization pressure.

Maintain behavior specifications for the failures you care about — unauthorized secret use, permission laundering, hidden persistence, deceptive completion, unsafe self-modification, or policy evasion. Let an auditor vary conversational strategy across rounds, learn from scored traces, and retain the highest-value counterexamples.

Then turn every confirmed failure into three durable assets:

• a regression trace;
• a monitor or deterministic policy candidate;
• a revocation query for artifacts that may encode the same behavior.

Do not reduce the result to a single “safety score.” Preserve the actual transcript, environment, model version, retrieval set, tool grants, and outcome. Rare failures are diagnostic artifacts, not only metrics.

5. Keep quality independent from optimization

CHASE studies repeated adaptation to LLM ranking signals and reports that ranking-quality alignment declined across all six simulated domains. The exact magnitude is an author-reported result from a controlled simulation, not a measurement of the whole web. The failure mechanism is still relevant to agent systems.

Whenever creators or agents can observe a metric and optimize for it, the metric becomes a target. Retrieval scores, evaluator preferences, task-success proxies, citation counts, and human approval rates can all be gamed or homogenized.

Use an independent quality lane:

• hide portions of the evaluation set;
• rotate evaluators and model families;
• preserve source-grounded deterministic checks;
• sample for human review outside the ranking path;
• measure diversity and novelty, not only top-score convergence;
• test whether ranking changes still predict downstream utility.

A skill that rises in retrieval ranking is not necessarily a better skill. It may simply have learned the vocabulary of the retriever. Promotion should depend on behavior under tests, not similarity to previously promoted artifacts.

6. Make revocation a first-class operation

Most memory systems are designed to append. Mature systems must also invalidate, supersede, and recall.

Revocation should answer:

• Which agents retrieved this artifact?
• Which derived artifacts used it as evidence?
• Which decisions or side effects depended on it?
• Can the artifact be removed from hot indexes immediately?
• Can cached copies be expired?
• What safe replacement or degraded mode applies?

Build a dependency graph from source to extracted fact, from fact to summary, from trajectory to skill, and from skill to execution. When a source is corrected or a skill is found malicious, traverse the graph and quarantine descendants until they are revalidated.

For executable knowledge, use kill switches outside worker control. A revoked skill must fail capability issuance even if stale text remains in a cache or prompt.

7. Bind reuse to freshness and environment

Knowledge can be safe when created and dangerous when reused elsewhere. API schemas change. Model tool-call behavior shifts. Permissions move. A cleanup command may target the wrong resource after an environment rename.

Every reusable artifact should declare compatibility constraints: harness version, tool schema hash, operating system, service version, policy version, and required principals. Retrieval can surface an incompatible artifact as reference, but execution should require revalidation.

Use time differently by artifact class. A regulatory fact may require a source refresh. A benchmark may remain historically correct but need a warning that it is not a current production result. A deployment procedure should expire quickly. A destructive recovery skill should require a live dry-run whenever its target changed.

8. A minimal promotion pipeline

A practical knowledge supply chain can start with six services:

1. Evidence store for immutable snapshots and source hashes.
2. Candidate registry for extracted facts, summaries, and generated skills.
3. Validation workers for schema checks, reconciliation, replay, and adversarial tests.
4. Policy signer that promotes artifacts and records approved permissions.
5. Typed retrieval gateway that filters by task, principal, freshness, compatibility, and revocation.
6. Revocation controller outside worker authority, with dependency traversal and cache invalidation.

Instrument every boundary. Log which artifacts entered context, which versions were used, which evidence supported promotion, what capability token was issued, which monitor fired, and how the result affected future retrieval.

The objective is not to stop an agent from accumulating experience. It is to prevent accumulation from silently becoming authority. Expensive context should become cheaper structured knowledge. Successful trajectories should become reusable procedures. But nothing should become trusted merely because the agent produced it, retrieved it often, or learned to rank it highly.

Long-lived intelligence comes from reuse. Long-lived trust comes from provenance, quarantine, independent evaluation, scoped permission, and revocation.