Agent Skills: Procedural Anchors, Failure Modes, and Retrieval

A useful agent skill is not a miniature textbook. It is a compact control surface for action.

That is the strongest result in “Demystifying Agent Skills: Why They Work—Until They Don’t” by Zhiyuan Jiang and collaborators. The paper moves beyond the usual question—does adding a skill improve aggregate task success?—and asks what changes inside matched executions.

The authors normalize 8,135 trial records, open-code 240 trajectories, retain 238 valid labels, and consolidate them into three high-level categories and twelve skill-use modes. They compare raw execution, workflow-memory injection, and distilled skill injection across paired tasks and settings.

The operational conclusion is precise:

Skills usually help by stabilizing procedures, not by supplying facts the model did not know.

This distinction changes what a skill should contain, how a library should be indexed, which metrics should be monitored, and where the system must retain authority to reject stale guidance.

Representation is the treatment

The central experiment compares three arms:

Raw: no prior-experience artifact.
Workflow Memory: cleaned procedural traces derived from previous runs.
Skill: a standardized SKILL.md distilled from the same traces.

Workflow Memory and Skill use matched source trajectories. If Skill outperforms Workflow Memory, the explanation cannot simply be that the agent received more experience. The representation changed.

The authors construct fixed-budget mixtures from 5s0f to 0s5f, where s and f denote successful and failed source trajectories. They evaluate two agent-model pairs on Terminal-Bench-2, SkillsBench, and Terminal-Bench-Pro.

Table 1 compares Workflow Memory and Skill success rates across agent-model pairs, benchmarks, and successful/failed trajectory mixtures.

Source: Jiang et al., arXiv:2608.14036, Table 1. Green and red indicate results above and below the corresponding Raw baseline. The table shows a broad skill advantage, not a universal one.

The Terminal-Bench-2 Codex rows make the representation effect visible. Raw reaches 0.5935. At 5s0f, Workflow Memory falls to 0.4452 while Skill reaches 0.7548. At 3s2f, Workflow Memory reaches 0.4194 and Skill 0.7806.

The effect is conditional. Failure-only source pools can hurt both representations. Distillation does not turn poor evidence into reliable policy. A skill generator still needs outcome signals, trajectory selection, and a way to preserve counterexamples rather than canonizing accidental behavior.

Procedural anchoring dominates knowledge injection

Across 528 paired Raw–Workflow–Skill triples, the skill arm achieves a 61.9% oracle-status success rate, compared with 59.1% for Raw and 55.9% for Workflow Memory. The robust aggregate contrast is Skill over Workflow Memory: +6.06 percentage points, with a 95% bootstrap confidence interval of [+0.76, +11.36].

The mechanism labels explain why:

procedural_anchor: 65.7% of skill mechanisms;
knowledge_injection: 4.5%.

Most agents did not need a new domain fact. They needed a stable setup order, tool sequence, intermediate check, output constraint, service-management routine, or verification plan.

Skills reduce environment_infrastructure_failure from 5.3% in Raw to 0.2%, output_format_schema_mismatch from 7.4% to 3.2%, and background_service_lifecycle_failure from 2.7% to 0.8%.

These are highly skillable failures. Once an agent discovers the correct workaround, path convention, lifecycle command, or validator, a skill can keep that operational detail active during future runs. A skill should therefore be optimized for decision compression, not topical coverage. Its value is the number of unstable choices it removes from the next execution.

Workflow memory preserves evidence—and procedural residue

Workflow Memory is not useless. A trace contains real commands, observed errors, and debugging evidence. But it also preserves exploration, failed branches, and environment-specific residue.

The taxonomy exposes the cost. timeout_budget_exhaustion appears in 10.6% of workflow-memory cases, compared with 1.7% for Raw and 4.4% for Skill. A long trace can anchor the agent to too much process: it spends context replaying how a solution was found instead of executing the compact procedure that survived verification.

This suggests a clean memory architecture:

preserve raw trajectories in an auditable episodic store;
derive versioned procedural artifacts from selected evidence;
retrieve the compact artifact into working context;
retain source links so the original evidence remains inspectable.

Logs are for provenance and diagnosis. Skills are for execution control.

Skills create an applicability failure surface

Compression removes noise, but it also removes context. The resulting abstraction must be interpreted.

The mode skill_guidance_misapplied_or_ignored appears in 10.0% of skill-arm cases, versus 0.8% for Raw and 0.4% for Workflow Memory. A skill can be locally sensible and still fail because the agent applies it under changed dependencies, misses a precondition, preserves a source-task assumption, or ignores runtime evidence that should invalidate the procedure.

Skills also leave deeper failures largely intact. algorithmic_logic_error remains at 7.4% in the skill arm. static_verification_without_runtime remains at 11.7%. Procedural guidance can make execution more robust without producing the correct algorithm or oracle.

A production skill needs more than steps:

Activation: observable conditions under which retrieval is appropriate.
Preconditions: environment, permissions, versions, and invariants.
Procedure: ordered actions with bounded tool scope.
Checkpoints: expected intermediate state after risky steps.
Verification: runtime evidence required before success is claimed.
Invalidation: signals that require adaptation, fallback, or escalation.

A skill must be defeasible. The environment, verifier, and current task outrank remembered procedure.

Retrieval quality and execution success are not one pipeline

The retrieval study separates three independent measurements over matched candidate pools:

Arm 1: embedding ranking without task execution;
Arm 2: explicit agent selection without execution;
Arm 3: full-pool execution, with actual skill access parsed after verification.

Outputs of Arms 1 and 2 are not passed into Arm 3. Candidate pools contain ground-truth skills plus random, semantically similar, or dissimilar distractors and grow from 5 to 100 entries.

Source: Jiang et al., arXiv:2608.14036, Figure 3. Solid lines in the right panel are parsed actual-use precision; dashed lines are downstream success.

The aggregate Arm 3 result is striking. Actual-use precision falls from 29.6% at pool size 5 to 3.3% at pool size 100, while downstream success changes from 36.4% to 39.3%.

At pool size 100, Arm 3 recall remains 54.3–73.6% in the reported pairings even though precision is only 0.7–8.1%. Agents often inspect the ground-truth skill along with many distractors. Related non-ground-truth skills may still provide useful procedures. The base model can sometimes solve the task without the annotated skill.

Exact ground-truth invocation is therefore neither sufficient nor strictly necessary for success. “Did the router select the benchmark label?” and “did the system complete the task?” are different measurements. A complete evaluation must observe selection, access, behavioral influence, and verified outcome separately.

A lifecycle architecture for skills

Capture

Store successful and failed trajectories with environment state, verifier output, tool calls, and outcome labels. Failure evidence matters, but it must not look like recommended behavior.

Distill

Generate the shortest procedure that preserves decisive ordering, preconditions, known failure modes, and verification. Link normative instructions back to source evidence.

Validate

Replay the candidate skill on held-out tasks and changed environments. Test activation and invalidation, not just the happy path.

Retrieve

Apply hard filters before semantic ranking. Keep candidate sets bounded. Merge duplicates and maintain aliases.

Invoke

Make access observable. Record which sections were read, which steps influenced tool calls, and which assumptions were checked. “Retrieved” is not equivalent to “used.”

Verify and revise

Runtime evidence outranks skill text. When a run fails, distinguish content error, retrieval mismatch, applicability failure, execution failure, and verifier failure. Revise the correct layer.

Metrics that reflect the lifecycle

A single success rate hides too much. A skill platform should track candidate-set recall, retrieval precision by distractor class, actual access precision and recall, activation false positives and negatives, verified success conditional on access, token and latency overhead, invalidation rate, skill-specific regressions, and age since last validation.

The paper’s methodological contribution is this decomposition. Representation, retrieval, invocation, adaptation, and verification are separate control points.

Builder takeaway

The wrong lesson is “write more skills.” The right lesson is to build disciplined procedural memory.

Skills work when they compress noisy experience into reliable operational choices. They fail when source evidence is poor, context changes, the abstraction is brittle, retrieval exposes too many confusable artifacts, or the agent treats remembered guidance as authority over runtime evidence.

The best skill is not the most comprehensive document. It is the smallest versioned procedure that activates under observable conditions, states what it assumes, drives concrete actions, checks progress, knows how it can be wrong, and yields to verification.

That is how a skill becomes a procedural anchor rather than another piece of context competing for attention.

Source: Zhiyuan Jiang et al., Demystifying Agent Skills: Why They Work—Until They Don’t, arXiv:2608.14036v1, August 14, 2026. Reported values apply to the paper’s benchmark tasks, agent-model pairings, and experimental protocol; they are not universal deployment rates.

Agent Skills Work as Procedural Anchors—Until the Library Fights Back