WikiSkill Architecture: From Agent Traces to Governed Procedural Memory

Most “learning agents” do not actually learn in a durable, auditable sense. They retain transcripts, append lessons to a prompt, or rewrite a procedure after a successful run. Those mechanisms preserve information, but they do not reliably distinguish evidence, knowledge, and executable policy.

WikiSkill introduces a useful systems boundary: immutable execution experience belongs in a Raw Layer; consolidated, persistent knowledge belongs in a Wiki Layer; executable procedure belongs in a Skill Layer. The layers co-evolve, but they do not share the same acceptance semantics.

That separation turns skill evolution from prompt editing into a governance problem.

1. The core architecture is a three-stage compiler

WikiSkill can be read as a compiler pipeline:

type Experience = ImmutableTrace[];
type Knowledge = StructuredPattern[];
type Procedure = VersionedSkill[];

Experience -> root-cause consolidation -> Knowledge
Knowledge + relevant Experience -> proposal -> ProcedureCandidate
ProcedureCandidate + validation -> promote | rollback

The analogy matters because compilation is intentionally lossy. A skill should not contain every observation from every trajectory. It should encode the smallest reusable procedure supported by accumulated evidence.

Figure 2 from the paper. The Wiki compounds while the Skill Layer remains reversible.

Raw Layer: evidence, not context

The Raw Layer stores immutable trajectories: observations, actions, tool calls, outcomes, and scoring signals. Its purpose is replay, audit, sampling, and root-cause analysis. It should not be injected wholesale into every future run.

A production Raw Layer needs task identity, environment version, model and harness version, tool results, artifact hashes, evaluator output, and retention policy. Without those fields, later conclusions may be impossible to reproduce.

Wiki Layer: persistent, structured hypotheses

The Wiki Maintainer analyzes sampled trajectories and consolidates patterns into persistent knowledge. A useful wiki entry should include:

• claim or pattern;
• applicability conditions;
• supporting and contradicting traces;
• confidence and unresolved uncertainty;
• relationship to existing entries;
• last validation time and environment.

“Persistent” should mean continuity across evolution iterations, not append-only immortality. Production systems need merge, supersession, contradiction resolution, expiry, and deletion.

Skill Layer: executable policy under change control

The Skill Layer contains filesystem-based procedures: instructions, scripts, resources, and applicability conditions. It is the layer the inference agent can execute.

Because skill content changes behavior, a skill candidate is closer to a code change than a memory note. It needs versioning, diff review, tests, permission analysis, acceptance metrics, and rollback.

2. The evolution loop separates understanding from action

WikiSkill uses four roles.

1. Inference Agent: executes training rollouts with the current skills and writes traces.
2. Wiki Maintainer: performs root-cause analysis and consolidates knowledge.
3. Skill Proposer: reads the current skill, wiki, and relevant traces to propose edits.
4. Gating and Rollback: evaluates the candidate on validation tasks and promotes it only if performance improves.

This decomposition prevents a common anti-pattern: the same model observes one outcome, invents an explanation, edits its own policy, and declares the edit successful in one uninterrupted trajectory.

The persistent wiki and reversible skill layer also encode different failure semantics. A rejected skill proposal can still produce knowledge: a hypothesis failed under specified conditions. The procedure rolls back, while the evidence remains available to prevent cycling back to the same rejected idea.

3. The cross-model result changes deployment economics

Across five benchmarks and five models, the authors report that WikiSkill outperforms compared skill-evolution methods in most settings. The strongest systems insight is not the aggregate rank; it is the separation between skill discovery and skill execution.

The paper reports:

• Qwen-3.5-9B with WikiSkill reaches 47.4% average accuracy, above Qwen-3.6-27B without skills at 39.4%.
• WikiSkill gains within the Qwen family increase with scale: 12.3%, 17.5%, and 23.9% for 4B, 9B, and 27B models.
• On ALFWorld, Qwen-3.5-9B reaches 70.2% with a skill evolved by Qwen-3.6-27B, versus 63.4% with its self-evolved skill.

Figure 1 from the paper. Results are author-reported preprint evidence, not independent replication.

A plausible production topology follows:

• use a stronger, more expensive model offline for trace analysis and skill proposals;
• validate candidates against a fixed benchmark and safety suite;
• deploy accepted skills to cheaper inference models;
• monitor for model-specific regressions and environment drift.

Cross-model transfer is therefore an optimization opportunity, not an automatic guarantee. Tool syntax, instruction hierarchy, context policy, and model compliance can all change transfer behavior.

4. Promotion requires more than one scalar score

The paper uses domain-specific validation performance. Real systems often need a vector of gates:

type SkillGate = {
  taskSuccess: number;
  regressionPass: boolean;
  permissionDelta: "none" | "review" | "reject";
  toolCallsP95: number;
  tokenCostP95: number;
  latencyP95: number;
  safetyViolations: number;
  reproducibilityRate: number;
};

A candidate should not be promoted merely because average task accuracy rises. A 2-point gain may be unacceptable if it adds irreversible tools, doubles cost, or fails a critical edge case.

Recommended gates include:

• held-out task success and adversarial cases;
• deterministic artifact, schema, and state assertions;
• permission and external-side-effect diff;
• latency, token, and tool-call budgets;
• security tests for prompt injection and untrusted inputs;
• replay stability across multiple seeds or runs;
• canary deployment and rollback readiness.

5. Skill governance needs explicit lifecycle states

A robust implementation should model a skill lifecycle rather than overwrite a directory:

draft -> proposed -> validated -> approved -> canary -> active
                   -> rejected
active -> deprecated -> archived
active -> rolled_back

Each transition should record the actor, evidence, benchmark version, model/harness version, approval, and artifact hash.

This is especially important for skills that send messages, change infrastructure, publish content, move money, or access private data. Self-evolution must never imply self-expansion of authority. A candidate may improve procedure within an existing permission envelope; new authority requires a separate human-controlled grant.

6. Failure modes the paper makes visible

Wiki poisoning

Incorrect root-cause analysis can become persistent and influence many later proposals. Mitigations include provenance, confidence, contradiction tracking, multiple independent analyses, and periodic revalidation.

Validation overfitting

Repeatedly evaluating on a static validation set can leak the benchmark into the skill. Maintain hidden test suites, rotate scenarios, and monitor post-deployment drift.

Skill bloat and conflict

Independent proposals may duplicate or contradict procedures. Use scoped applicability, dependency metadata, conflict detection, minimal diffs, and ownership boundaries.

Environment coupling

A skill can encode file paths, tool names, UI assumptions, or provider behavior that do not transfer. Declare prerequisites and environment compatibility explicitly.

Irreversible external learning

A failed experiment that modifies public or private systems cannot always be “rolled back” by restoring a skill file. Training and validation should use sandboxes, dry runs, mocks, and reversible fixtures.

7. Mapping WikiSkill onto an OpenClaw-style skill system

A practical repository layout could be:

experience/
  runs/<task-id>/checkpoint.json
wiki/
  patterns/<domain>.md
  contradictions/<topic>.md
  evolution-log.jsonl
skills/
  <skill-name>/SKILL.md
  <skill-name>/tests/
proposals/
  <proposal-id>/PROPOSAL.md
  <proposal-id>/evidence.json
  <proposal-id>/gate-results.json

The safe operating loop is:

1. collect bounded evidence from completed runs;
2. consolidate patterns without changing live skills;
3. create a proposal with a narrow diff and explicit goal;
4. test in an isolated environment;
5. inspect permission and safety deltas;
6. require approval for promotion;
7. canary, monitor, and retain rollback artifacts.

This is intentionally slower than letting an agent rewrite its own skill after every task. The friction is a feature: it converts uncontrolled behavioral drift into reviewable evolution.

8. What the evidence does and does not establish

The paper supports three claims within its evaluation setup:

• persistent structured knowledge improves skill evolution;
• skill evolution can complement model scaling;
• skills can transfer across models and sometimes outperform self-evolved skills.

It does not establish that an autonomous agent should update production procedures without oversight, that wiki content remains correct indefinitely, or that transfer works across arbitrary harnesses and tools. The benchmarks have explicit scoring functions; many operational tasks do not.

Takeaway

WikiSkill's most valuable contribution is not another memory store. It is a control boundary:

traces are evidence, wiki entries are revisable knowledge, and skills are executable policy.

Once those categories are separated, agent learning becomes governable. Experience can compound without turning every observation into permanent instruction; procedures can improve without losing rollback; stronger models can discover skills that cheaper models execute; and operators can inspect exactly why a behavior changed.

That is a credible path from “the agent remembers” to “the agent learns under change control.”

Primary source: arXiv:2608.27454 — WikiSkill · PDF