Repositories Are Not Agent Knowledge Until They Become Verified Operating Context

An agent can have access to a repository and still know almost nothing about how to use it well.

Access is not operational competence. A checkout gives the agent source files. Search gives it fragments. A large context window may let it read more of the codebase. None of those mechanisms guarantees that the agent will identify the right interface, select an appropriate workflow, respect version-specific constraints, validate the output, or recover from a known failure before the task budget is gone.

Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills frames this gap as missing operational knowledge. Its DisCo system turns repositories and papers into compact, provenance-grounded, verified skill graphs. The reported AREX-Skill snapshot contains 5,353 skills distilled from 1,000 machine-learning repositories, indexed across 20 areas and 178 capability families.

The most useful way to read the paper is not as a new prompt-engineering technique. It is an architectural proposal:

Research agent = Model + Harness + Operational Knowledge

The model supplies general reasoning. The harness supplies execution control. The operational layer supplies the versioned, task-relevant know-how that tells the agent what to consider before it starts rediscovering a repository through trial and error.

This article turns that idea into an implementation blueprint for teams building agent systems.

1. Separate operating knowledge from control logic

Many agent stacks place too much knowledge inside the harness. Package instructions end up embedded in system prompts, tool descriptions, orchestration code, retry handlers, or one-off workflows. That coupling creates three problems:

1. Domain knowledge becomes hard to inspect. Operators cannot easily tell which repository assumptions guided a run.
2. Updating knowledge risks changing control behavior. A package upgrade may require editing the same prompt that governs planning or permissions.
3. Learning does not transfer cleanly. A workaround discovered in one task remains trapped in its trajectory or harness implementation.

Repo-To-Skill draws a cleaner boundary.

This distinction matters for safety as much as performance. A skill should inform the agent; the harness should continue to govern what the agent is allowed to do. Loading a deployment skill must not silently grant deployment credentials. Reading a data-migration procedure must not bypass an approval gate. Operational knowledge and execution authority are different system objects.

Figure 1: The operational layer and the reported benchmark impact of AREX-Skill — source: Repo-To-Skill, arXiv:2609.02749.

2. Define a skill as a contract, not a document

DisCo represents a skill with three layers:

skill/
├── SKILL.md       # knowledge interface and usage policy
├── references/    # evidence and deeper technical material
└── scripts/       # execution interface with defined inputs and outputs

That structure is simple, but its semantics are stronger than “put instructions in Markdown.” Each layer has a different contract.

SKILL.md: the knowledge interface

The entry file should help an agent answer five questions quickly:

• Does this skill apply to my current objective?
• What capability does it expose?
• Under which conditions should I use or reject it?
• What procedure should I follow?
• Which checks prove completion?

A useful SKILL.md is compact because it is read first. It routes deeper rather than copying every API detail into the initial context.

references/: the knowledge substrate

References preserve the material needed for difficult decisions: public API signatures, data formats, algorithmic details, version constraints, configuration matrices, failure explanations and source locations. They are opened only when the current step needs them.

scripts/: the execution interface

Scripts wrap deterministic, repetitive or error-prone actions behind explicit inputs and outputs. They should reduce reinvention, not hide side effects. A script without a validation rule is merely automation; a script with a contract can become part of verified operating context.

The resulting pattern separates capability from usage policy. References and scripts make an operation possible. SKILL.md tells the agent when and how that operation should be used.

3. Build skill graphs around future tasks, not source folders

A repository rarely maps cleanly to one skill. It may support data preparation, training, inference, evaluation, serving and troubleshooting. Packing all of that into a single entry file creates a miniature documentation dump and defeats progressive disclosure.

DisCo therefore produces a skill graph: one router-like entry skill connected to bounded component skills. The split follows likely agent tasks rather than the repository's directory tree.

repository-entry
├── setup-and-interface-selection
├── data-preparation
├── training-or-inference
├── evaluation-and-output-checks
└── troubleshooting-and-recovery

Links can encode routing, dependency or composition. The agent starts at the entry point, loads one relevant branch and leaves unrelated branches unopened.

This design has a practical implication: do not organize agent knowledge around where maintainers happened to place files. Organize it around the decisions and workflows an agent must execute.

A directory called utils/ is not a capability. “Validate a generated configuration before launching a costly training run” is.

4. Use a four-stage distillation pipeline

The paper formalizes every DisCo construction run as:

Anchor → Scope → Ground → Construct → Verify

The anchor can be a source, such as a repository or paper, or a concrete task. The same four stages apply in both cases.

Stage 1 — Scope: decide what deserves to become operational knowledge

Start by fixing the evidence boundary. Include the repository roots, public documentation, examples, tests, scripts and configuration that describe reusable package behavior. Exclude generated files, caches, vendored dependencies, local environments, build outputs and unrelated development internals unless the intended workflow genuinely depends on them.

Then identify capabilities. The important output is not an inventory of files; it is a bounded set of future agent tasks.

A good scope record includes:

source:
  repository: owner/project
  commit: <immutable revision>
  package_version: <if available>
  dirty_state: false
capabilities:
  include:
    - select the supported public interface
    - prepare valid input data
    - run bounded inference
    - validate the produced artifact
    - recover from documented runtime failures
  exclude:
    - repository release administration
    - undocumented internal APIs
    - production deployment

The explicit exclusion list prevents a broad repository from turning into an unsafe, ambiguous skill.

Stage 2 — Ground: attach every operational claim to evidence

Documentation establishes intent, but source and live inspection may be needed to confirm current behavior. Tests reveal invariants. Examples expose expected composition. Configuration files reveal defaults and compatibility constraints.

Grounding should produce an internal claim–evidence map:

Claims that cannot be grounded should be omitted, narrowed, or marked as unresolved. A plausible instruction is not enough.

Stage 3 — Construct: package knowledge for selective use

Write the smallest entry skill that can route the agent correctly. Move API tables, long explanations and version matrices into references. Add scripts only for operations supported by the evidence boundary.

Each procedural step should end with a checkable completion criterion. For example:

1. Read the package version and select the matching reference table; complete when the version range has one compatible public interface.
2. Run the tiny-input validator before the full job; complete when the output schema and numeric checks pass.
3. On error X, apply recovery Y once, then rerun only the failed check; stop when the same failure repeats.

This is operational writing. It replaces “try this command” with “perform this bounded action, observe this evidence, and stop under these conditions.”

Stage 4 — Verify: test the skill as an artifact

The paper's central methodological rule is worth preserving verbatim in spirit: verification is what separates distillation from summarization.

Verification can combine:

• assertion-backed usability cases;
• safe repository-native tests or examples;
• import, version and CLI checks;
• tiny-fixture execution;
• smoke scripts with explicit outputs;
• static validation of metadata, links, provenance and local-path leakage;
• recovery tests for known failure modes.

If a failure is attributable to the skill, repair the smallest affected area and rerun the relevant checks. Do not erase the evidence of failed checks. The accepted graph should ship with a construction record containing its source boundary, evidence, checks performed and unresolved gaps.

Figure 2: Scope, ground, construct and verify in creator mode; selective operating context in researcher mode — source: Repo-To-Skill, arXiv:2609.02749.

5. Support both source-first and task-first construction

DisCo uses two complementary modes.

Task-agnostic distillation

The anchor is a source. The system asks which reusable capabilities a repository, paper or tutorial can provide. Construction happens offline, before a downstream task requests the skill.

This mode is appropriate when:

• the repository is widely reused;
• many future tasks share its setup or failure modes;
• version-specific behavior is expensive to rediscover;
• the cost of construction can be amortized across repeated use.

The reported AREX repository snapshot contains 5,353 skills across 1,000 repository graphs. Construction used GPT-5.5 and GPT-5.6-sol with high reasoning effort and an average allocation of about $40 per repository. That number makes reuse economics part of the design: not every repository deserves precomputed skill coverage.

Task-oriented distillation

The anchor is a problem. The system decomposes the task, identifies capability gaps, searches for appropriate sources and constructs the required skills on demand.

This mode is appropriate when:

• the task is novel or highly specialized;
• the relevant repositories are not known in advance;
• the operating knowledge is defined partly by the benchmark or task interface;
• a bounded diagnostic signal can verify the resulting procedure.

Task-oriented output should still be reusable for a class of related problems. Otherwise it is a solution trace, not a skill.

6. Route progressively and preserve a no-skill fallback

A library with thousands of skills cannot be injected wholesale. AREX uses a two-level taxonomy and router: area → capability family → repository graph. The agent opens only the relevant branch.

A production router should optimize for more than recall. An irrelevant skill can actively reduce performance by steering the agent away from a task-specific strategy. In the paper's PaperBench results, 18 of 20 tasks improve, but two regress. The authors identify retrieval precision as a plausible explanation.

That suggests three routing requirements:

1. Relevance threshold. Do not force a skill route when no close match exists.
2. Visible rationale. Record why a graph was selected and which capability it is expected to provide.
3. Unguided fallback. Let the agent return to base reasoning when skill guidance conflicts with direct evidence or repeatedly fails verification.

A skill should be a defeasible prior, not an absolute command. The paper's PassNet construction analysis illustrates why: an initially sensible warning against reimplementing vendor-optimized operators was too broad and blocked a valid optimization. Evidence from a missed case led the authors to revise the rule from a ban into a default that admits exceptions.

7. Evaluate skill value without hiding its cost

The paper compares the same GPT-5.5 Codex setup with and without distilled skills. It reports:

These are strong results, but a production evaluation should separate four quantities:

Total value = downstream quality gain
            + failures avoided
            + exploration time saved
            - construction cost
            - retrieval overhead
            - maintenance cost
            - harm from stale or mismatched guidance

The paper matches downstream execution constraints and constructs skills before evaluation, but skill-construction budgets are separate from runtime budgets. On FrontierCS, skill-equipped runs also use more tokens, steps and tool calls overall, although the authors report that per-task score gains are essentially uncorrelated with the additional usage.

Do not evaluate a skill platform with answer quality alone. Track:

• route precision and route abstention;
• task success delta against the same agent without skills;
• time and tokens spent before the first valid result;
• invalid or incomplete outcomes;
• recovery success rate;
• construction and maintenance cost;
• number of successful reuses per skill version;
• regressions caused by stale or mismatched guidance.

8. Add lifecycle governance before production use

Repo-To-Skill focuses on producing and evaluating operational knowledge. A production platform also needs a control plane around that knowledge.

A safe lifecycle looks like this:

Versioned source
    ↓
Candidate skill graph
    ↓
Evidence + sandbox verification
    ↓
Approval and risk classification
    ↓
Scoped runtime retrieval
    ↓
Outcome monitoring
    ↓
Keep / revise / quarantine / revoke

Minimum provenance record

{
  "skill_id": "package-x-inference",
  "source": {
    "repository": "owner/package-x",
    "commit": "immutable-revision",
    "package_version": "1.4.2"
  },
  "evidence": [
    "docs/inference.md",
    "src/package_x/api.py",
    "tests/test_inference.py"
  ],
  "verification": {
    "passed": ["import", "tiny-fixture", "output-schema"],
    "skipped": ["gpu-throughput"],
    "unresolved": ["Windows backend not tested"]
  },
  "permissions": ["read-workspace", "execute-sandboxed-script"],
  "review_after": "2026-12-01"
}

Lifecycle triggers

Reverification should occur when:

• the upstream commit or package version changes;
• a public interface is deprecated;
• a recovery action starts failing;
• outcome monitoring detects a negative success delta;
• security policy changes;
• a skill requests broader permissions than its previous version;
• the review horizon expires.

Stale skills should not remain silently routable. Quarantine is better than confident misuse.

9. An implementation checklist

For a first production pilot, keep the scope narrow.

Choose the source

• Select one frequently reused repository with expensive setup or recurring failure modes.
• Pin an immutable commit and package version.
• Define capabilities and exclusions before generation begins.

Build the graph

• Create one router-like entry skill.
• Split component skills by future workflows, not folders.
• Keep the entry context small through progressive disclosure.
• Attach each important operational claim to inspectable evidence.

Verify the artifact

• Run a tiny successful case.
• Run at least one negative or recovery case.
• Validate metadata, links, paths and script interfaces.
• Store passed, skipped and unresolved checks.

Control runtime use

• Require a relevance threshold.
• Keep permissions in the harness, outside the skill.
• Preserve a no-skill fallback.
• Log the selected skill version and routing rationale.

Measure reuse

• Compare against the same agent and harness without the skill.
• Track first-valid-result time, failures, quality and total cost.
• Revise or revoke when evidence changes.

10. The deeper shift: from repository access to institutional memory

The old assumption was that an agent with code search and a large context window could reconstruct whatever it needed. Repo-To-Skill challenges that assumption. Reconstruction is possible, but it repeatedly spends task budget and loses the result after the run.

Verified skill graphs offer a different model: the organization can retain operational discoveries as explicit, portable and reviewable artifacts. A successful recovery does not have to remain buried in a trajectory. A maintainer's hard-won configuration rule can become reusable operating context. A repository upgrade can trigger targeted reverification rather than another cycle of silent rediscovery.

That is why the paper's contribution is larger than repository summarization. It describes a path from public software artifacts to institutional memory for agents.

The opportunity is substantial, but so is the responsibility. An unverified skill scales error as efficiently as a verified skill scales expertise. The winning systems will not be those with the largest folders of SKILL.md files. They will be those that can answer, for every piece of operational guidance:

Where did this come from, when does it apply, what proved it, what can it do, and how do we revoke it when it stops being true?

Primary sources: Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills — arXiv:2609.02749 · AREX-Skill Library — GitHub