Skills and tools

Dong et al. went looking for the cases where a skill made an agent worse. They ran agents on SkillsBench and SWE-Skills-Bench twice per task, once with a loaded skill and once without it or with a semantically matched reference skill, and wherever the skill-guided run did worse they traced the regression back to the skill ¹. They found 307 skill-induced failures: 125 functional failures, where the task broke, and 182 efficiency regressions, where the task succeeded at a higher cost. Irrelevant skills were a minor cause; seemingly relevant skills caused most of the functional failures ¹.

The efficiency categories show how that happens. The largest, Excessive Verification, has 67 cases, and the next, Heavy Implementation Pipeline, has 30 ¹. In both, the skill was on topic and its advice was reasonable in general. The agent read a validation checklist as a required procedure and ran every check, or read a construction recipe (subprocess workflows, multi-stage conversion, runtime simulation) as the only acceptable path and built all of it for a task that needed a fraction.

In the authors' words, skills "often turn validation checklists and construction recipes into mandatory work" ¹. An irrelevant skill gets ignored. A relevant skill gets obeyed, including the parts that do not apply.

The obvious worry with a skill library is retrieval: load the wrong skill and the agent goes astray, so you improve retrieval. The 307-failure study says a large share of the damage comes from loading the right skill and following it too literally, and better retrieval makes that worse, because it loads relevant skills more often. If that holds, a team building a skill library should not start by improving retrieval. Within six weeks two papers had responded with the same design, treating "is this skill relevant?" and "should this skill run now?" as separate decisions and learning the second one from matched with/without evidence ^2,3.

Failures like these only appear once libraries get large, shared and long-lived. Before 2026 the skills literature was about getting the loop to work at all: can an agent write reusable procedures, verify them, and get better by reusing them? It can, and the way that loop was built explains why its failures showed up late.

A skill, in the sense used here, is procedural knowledge an agent writes to an external artifact that a later run of the same or a different agent can retrieve and execute: a code function, a tool wrapper, an MCP server, or a folder with a SKILL.md file. The weights never change; what changes is the set of procedures the agent can load. Experiential memory stores what happened, while a skill stores how to do something. Harness edits change the control flow that runs the agent, while a skill changes what the agent knows how to do.

The idea is older than the file format. Voyager (2023) wrote JavaScript skills in Minecraft, admitted only code that passed execution and self-verification checks, retrieved skills by embedding, and collected 3.3× more unique items than prior agents ⁴. LATM (2023) split tool making from tool use: GPT-4 wrote tools that GPT-3.5 called at comparable accuracy and lower cost ⁵.

In 2025, SkillWeaver's web agents distilled practice into API functions that lifted weaker agents by up to 54.3% relative on WebArena ⁶, and Alita generated MCP servers at runtime to reach 75.15% pass@1 on GAIA validation ⁷. Two design choices from these systems are still standard: executable skills can be gated by execution, and an agent retrieves from a library and never loads all of it. One gap came along with them: no system of that era deleted, retired or re-validated a skill.

Until late 2025 every system invented its own skill format, so nothing one agent learned could be used by another agent's harness. Anthropic's Agent Skills, launched on October 16, 2025, fixed the format ⁸. A skill is a folder with a required SKILL.md file that opens with YAML frontmatter (a name and a description), plus optional scripts/, references/ and assets/ directories. Loading uses progressive disclosure: the agent reads only names and descriptions at startup, loads the full instructions when a skill activates, and pulls resources on demand.

Anthropic published the format as an open standard in December 2025 ⁹, and Codex, Gemini CLI, Cursor, Windsurf and GitHub Copilot adopted it ¹⁰. A research system that emits a SKILL.md now produces something every major coding agent can load, and a registry can index skills without knowing which agent wrote them. Progressive disclosure made large libraries possible in principle, because at startup each skill costs only its description. It also put the whole selection decision on that one description field, and several of the failures below start there.

The standard is thin on purpose. It says where instructions go and nothing about whether they are correct, current, safe or worth loading. In a June post, "Skill Libraries Need CI, Not More Prompts", I argued that once skills can be installed, auto-selected and updated, they behave like governed persistent state, and that the commit gate is more important than the proposer ¹⁰. That post covered lifecycle tooling (SkillOps, PACE, SkillRevise, Workflow-to-Skill, MemoRepair and others). It did not cover how agents write and improve skills for themselves, or the evidence from June through September about what happens when those libraries get large.

Every system described below gathers evidence, proposes a new or revised skill, tests it, commits or rejects it, and retrieves it later. They differ in what they mine, how they propose and which check makes the commit decision.

Mining repositories: Code2Skill

The newest extraction work skips agent experience entirely. Code2Skill, a September paper, turned code units from 19,769 popular, actively maintained GitHub repositories into skill records (atomic operations, composite workflows, recurring patterns). It verified each record by reconstructing it without seeing the source body and then comparing against the source, and kept 1,006,822 accepted records with provenance metadata ¹¹.

Across 72 protocol-matched evaluations covering nine model settings and eight benchmarks, retrieving from that bank improved models by 11.7% on average over matched baselines and beat them in 57 cases; under a shared interface it also beat trajectory-derived skill banks on all seven shared benchmarks ¹¹. Skills mined from repositories solve the cold-start problem that trace mining has, because they exist before the agent has done anything. They also tie a million records to code that will change, which makes the staleness problem covered below more pressing.

Consolidating many traces: Trace2Skill

Trace mining still has the strongest transfer evidence. Early extraction was sequential: finish an episode, write or edit a skill, move on, which overfits to whichever episode came last. Trace2Skill (March 2026, v5 June 2026) consolidates many trajectories in parallel. It induces recurring patterns across a broad set of traces, resolves conflicts between them and writes a unified skill directory, and it outperforms both sequential skill editing and ReasoningBank-style retrieval of raw experience ¹².

I would point people to the transfer result. Skills that Qwen3.5-35B evolved on its own trajectories improved a Qwen3.5-122B agent by up to 57.65 absolute percentage points on WikiTableQuestions, with no weight update ¹². A smaller model's procedural lessons, written as files, made a larger model much better in the same domain.

The comparison stays within one model family, "up to" marks a best case, and the paper is labeled work in progress. Within those limits it shows that a lesson written as text can still work after a change of model. It also fits the pattern report 12 traces across the series: code and structured procedure transfer across models where tuned prose often does not.

Learning from failures: EvoSkill and SkillRevise

A successful trace shows one path that worked. A failed trace shows where the agent's default behavior breaks, which is what a skill ought to fix.

EvoSkill (Sentient AI and Virginia Tech, March 2026) analyzes execution failures, proposes new skills or edits, and keeps only those on a Pareto frontier of held-out validation performance, with the model frozen throughout ¹³. Exact match rose from 60.6% to 67.9% on OfficeQA and from 26.6% to 38.7% on SealQA, and skills learned on SealQA transferred zero-shot to BrowseComp for a 5.3% gain ¹³. Merging libraries from independent runs beat any single run (67.9% against 64.5%), and gains flattened as the training fraction grew ¹³.

SkillRevise (HKUST, May 2026, v4 September 2026) starts from one imperfect skill. It diagnoses defects from execution evidence, retrieves general repair principles, applies edits anchored to the observed failure, and keeps the best re-executed candidate. On SkillsBench, base-agent success rose from 36.05% to 61.63% ¹⁴.

Optimizing skill text: SkillOpt and its successors

SkillOpt (Microsoft, May 2026) treats skill text as trainable external state ¹⁵. It proposes bounded edits, keeps a history of rejected edits so it does not cycle back to them, and commits an edit only when held-out validation improves. Across 6 benchmarks, 7 target models and 3 harnesses (direct chat, Codex, Claude Code), it was best or tied in all 52 cells; with GPT-5.5 it added 23.5 points in direct chat, 24.8 inside Codex and 19.1 inside Claude Code ¹⁵.

Within four months SkillOpt was the baseline to beat, and three September papers went after it. COBRA-Skills casts skill optimization as budgeted sequential search, with a contextual bandit deciding which candidates deserve an expensive evaluation; it reported the strongest average across six benchmarks and three target models at 55–58% lower optimization cost than SkillOpt, using 50 optimization examples per benchmark ¹⁶.

GraphSkillEvo represents a skill as a graph of steps and transitions and evolves a population with mutation and crossover, beating SkillOpt by 4.01% average accuracy on GPT-5.4-nano and 1.76% on GPT-5.4 across five benchmarks ¹⁷. SkillAdam adds an optimization memory and a volatility-driven edit budget, functional analogues of Adam's two moments, so that iteration-local feedback stops overwriting earlier corrections ¹⁸.

Each makes the proposer cheaper, more structured or more stable, and all three keep the same acceptance rule: an edit is kept because a validation score went up, with no statistical control on that decision. The next improvement should come from the acceptance rule, for reasons the PACE results below make concrete.

Who judges a new skill

The August evidence raises a harder question, which is whether explicit skills are doing the work at all. ContinualSkillBench built five domains of 100 interconnected subtasks each, ordered by difficulty with chances for cross-task reuse ¹⁹. Sequential execution generally improved performance, but plain in-context learning performed comparably to explicit skill maintenance on average, "suggesting that much of the improvement arises from adaptation to prior context and feedback rather than reusable skill abstraction alone" ¹⁹.

Explicit skills helped selectively, on tasks needing reusable procedures or precise outputs, and less capable models accumulated larger, more fragmented collections of task-specific skills ¹⁹. It is the skills version of CL-Bench's finding for memory systems, discussed in 07. I take it seriously: if in-context learning matches explicit skills on average, some of the gains in the table further down probably come from the agent having seen related tasks rather than from the skills it wrote.

Earlier 2026 benchmarks point the same way. SkillLearnBench (CMU and Amazon AGI; COLM 2026) found that every continual skill-learning method beat the no-skill baseline but none led consistently, that stronger backbones did not reliably produce better skills, and that self-feedback alone induced recursive drift while repeated external feedback produced real improvement ²⁰.

A May 2026 lifecycle study found that a model can be a strong skill extractor and a weak skill consumer, and that model-generated skills show non-trivial negative transfer even when they help on average ²¹. Where no ground-truth tests exist, CoEvoSkills (COLM 2026) evolves a Surrogate Verifier alongside the Skill Generator without showing it test content, and beat five baselines on SkillsBench for both Claude Code and Codex ²². The ordering matches what the rest of the series finds: external checks beat co-evolved judges, and co-evolved judges beat self-judgment.

Every system so far has a fixed improvement procedure written by its authors: how to analyze a failure, which traces to retrieve, how much budget to spend, how to phrase an edit. MetaSkill-Evolve (July 2026) makes that procedure a skill too ²³. Its improvement pipeline has five roles (Analyzer, Retriever, Allocator, Proposer, Evolver), all running on the same frozen backbone, and a meta-skill describes how each should operate. Task skills evolve on a fast loop from execution traces. The meta-skill evolves on a slower loop, "under the same pipeline applied to itself, with no additional model or objective" ²³.

Over the raw backbone, MetaSkill-Evolve added 23.54 points on OfficeQA, 16.09 on SealQA and 1.92 on ALFWorld, and it beat no-skill, static-skill and single-level evolution baselines on all three ²³. The comparison with single-level evolution is the one I would look at first, because it shows that evolving the procedure adds something beyond evolving skills under a fixed procedure.

The recursion is real but bounded. The meta-skill improves the improvement procedure and is improved by that same procedure; there is no third level, and nothing in the paper shows the two-level process converging or continuing to improve beyond the reported runs. The spread across benchmarks (23.54 points on OfficeQA, 1.92 on ALFWorld) suggests the meta-level helps most where the base skill loop has the most room.

Other reports in the series pair fast exploration in context with slow consolidation into weights. Here the slow store is still text, a meta-skill, so the whole recursive stack can be read and rolled back. Whether that counts as recursive self-improvement is the debate taken up in report 13.

Across the 2026 papers, gains are large when skills target a specific domain and are checked by execution, and smaller and more variable when the skill set is broad or the check is weak.

Three caveats apply to nearly every row. Most gains are measured in distribution: skills are learned and tested on the same benchmark family, and only EvoSkill's BrowseComp transfer, SkillOpt's cross-benchmark check and Code2Skill's breadth give out-of-distribution evidence. Few papers report a matched-budget baseline that gives the no-skill agent the extra tokens and calls the skill loop consumed, and ContinualSkillBench suggests that baseline would absorb much of the gain. And most rows are single-group preprints. The positive results have not been independently replicated at the scale the negative results below have.

In large libraries, loading and applying skills cause far more failures than writing them does. Go back to the 307-failure study ¹. Neither Excessive Verification nor Heavy Implementation Pipeline comes from a bug in the skill. Both come from a mismatch between a skill's generality and a task's needs.

A skill written to be safe across many tasks includes checks and scaffolding that most individual tasks do not need, and a model trained to follow instructions treats "consider validating X" as "validate X". Because 182 of the 307 failures were tasks that passed at a higher cost, a with/without eval that scores only pass rate would record them as neutral or positive.

Two papers from August and September put a gate between retrieval and execution. RADEG (August 2026) learns a cheap surrogate that predicts whether running a retrieved skill bundle is worth it. It is trained on matched rollouts in which one skill of the bundle is deleted, added or replaced, which isolates the effect of composition on verifier reward, and on 288 held-out rollouts it beat relevance-based and random gating across execution budgets ².

SkillApt (September 2026) makes the decision per skill, choosing LOAD or ABSTAIN from matched with/without runs in similar historical states. On its SRA-Bench evaluation it matched BM25 top-1 accuracy (0.838 against 0.838) while cutting skill activation from 100% of the time to 31.5% and mean token use by 74.3% ³. Its authors also found that both skill utility and how learnable the activation boundary is vary across base models ³.

How a skill is invoked also changes the outcome. Running skill packages as subagents in fresh context windows beat loading them into the main context in a September study, when the packages expose clear input–output contracts, at the cost of coordination tokens ²⁸.

Wix reached a similar design in production. Its August paper describes the selection pipeline for Helpmate, its customer-care assistant: a semantic matcher, then a deterministic gate that removes skills whose hard-stop conditions already hold for the account, then the LLM's choice ³⁵.

Across 756.6K user messages, the gate removed 59.4% of the skill–message pairs that had passed semantic matching, and together the two stages cut skill-description context by 90.5% against exposing all 10 skills. In a replay of 1,000 risk-enriched conversations with every skill exposed, the model picked a skill production would have blocked in 78 of them (7.8%) ³⁵. When a relevant skill that can't be executed for that account is visible, it steers the model anyway, which is the 307-failure mechanism measured on production traffic.

If relevant skills sometimes hurt, the natural remedy is to measure each skill and prune the harmful ones. ASSAY (June 2026) shows why an aggregate measurement can't do that ²⁹. It uses randomized masking, running tasks with random subsets of the library switched off and estimating each skill's causal contribution, and it found "pervasive causal heterogeneity": individual skills "routinely help on some task types while hurting on others, yet their opposing effects cancel in aggregate, making them invisible to global curation methods" ²⁹.

A skill with zero net effect may be very good for one task type and quite bad for another, and global pruning loses either way, whether it keeps the skill or drops it. ASSAY's per-task suppression beat prior curation methods across seven base models from four providers on AppWorld and tau-bench ²⁹.

The catch is cost. Randomized masking needs many runs per skill per task type, and an ecosystem registry holds tens of thousands of skills. SkillApt and RADEG are cheaper relatives of the ASSAY idea, since they estimate a skill's effect in the current state instead of on average.

Every skill system above keeps an edit because some score went up. PACE (June 2026) formalized why that decision is unreliable when it is repeated ³⁰. Applying "keep if the score improved" hundreds of times against a noisy development set is adaptive multiple testing with no correction.

On Qwen2.5 agents from 0.5B to 3B parameters, greedy acceptance committed 30–42% false edits and 10–33% harmful ones when real gains were available. When none were, 72–100% of commits were false (13–21 spurious self-modifications per run), and the most fragile agent lost 4.9 points ³⁰. PACE replaces the greedy rule with an e-process, a sequential test that accumulates paired evidence for candidate over incumbent on identical instances and commits only past a threshold; false commits fell to about zero at roughly 18% lower evaluation cost ³⁰.

PACE's experiments are on prompt edits in small models, so its specific rates don't transfer to skill libraries. The structure of the problem does. SkillOpt, EvoSkill, SkillRevise and the September optimizers all use some form of keep-if-better, and COBRA-Skills' bandit makes evaluation cheaper without changing the commit rule ¹⁶. A skill commit is a hypothesis test, and most skill loops run it without error control. For a skill loop running against real users, the acceptance rule is the first component I would replace.

Repo2Skill-Evo (August 2026) treats each software release as a skill-maintenance task: given skills written for version 1 and the patch to version 2, update what is obsolete and keep what still holds ²⁵. Across 57 real repositories and 105 release transitions, every transition invalidated part of the version-1 skill set, and six frontier agents reached only 29.9%–69.7% avg@3 macro F1 at the repair ²⁵. The errors went two ways: agents left stale content untouched, or edited too broadly and deleted guidance that was still correct.

A stale skill does not throw an error. It gives confident, specific, outdated instructions, and because it is relevant, the agent follows them. Put that next to the 307-failure finding and the problem gets worse: the skills most likely to be loaded are the ones most likely to mislead when the code under them changes.

In practice, humans still do the maintenance, when anyone does. A September longitudinal study mined five public AI-skill repositories (873 commits, 143 skill files, 254 substantive post-creation edits from October 2025 to June 2026) and found that every substantive edit was authored or merged through a named human account, while 62% had an AI co-author trailer ³¹. Across 18,463 registry skills and 23,199 skills in 5,876 GitHub repositories, a July study reports that 53% of reused skills were never modified after adoption ³². Public skill maintenance is a human-governed, AI-assisted process, and for most reused skills it never happens.

The line between skills and weights is blurring from both sides. SKT (August 2026) synthesized 27,164 verified trajectories from 2,000 public skills and fine-tuned models on them, which consistently improved skill use across benchmarks and harnesses ³³. A skill-internalization subfield now trains skills into weights and removes them from context. Prime Intellect's Prime Agent (August 2026) exposes prompts, skills, memory and sub-agents to the agent through one create/read/update/delete interface driven by its own trajectories ³⁴, so in practice the boundary between "skill", "tool" and "harness" is getting thin.

The shipped version of agents that write their own skills is open source and mostly human-reviewed. OpenClaw (390,396 GitHub stars on September 24, 2026) treats skills as packages: a CLI to install, search, update and publish them, per-agent allowlists, and a public registry, ClawHub, which lists about 67,300 skills ^36,37.

A September study of three ClawHub snapshots describes a boom that has already peaked. The observable stock nearly doubled in 91 days in the first half of 2026, and monthly listing creation was falling from its spring peak by the end of the window ²⁴. Attention is concentrated (the top 10% of skills received 46.93% of downloads) and scrutiny is thin: 77.86% of skills have zero stars and zero comments, while 85.06% of readable skills contain evidence of privileged actions ²⁴.

Nous Research's Hermes Agent (248,668 stars, MIT license) builds the self-improvement loop into the product. Its README describes "a built-in learning loop" that "creates skills from experience, improves them during use, nudges itself to persist knowledge, searches its own past conversations, and builds a deepening model of who you are across sessions" ³⁸. The README describes what the loop writes and does not describe a test that decides what gets kept.

A shared format that any agent can load is also a distribution channel for attackers. Between February and May 2026, the ClawHavoc campaign pushed malicious skills onto ClawHub; a Cloud Security Alliance research note on June 24, 2026 put the confirmed count at 1,184 ³⁹.

Code scanners catch much of the skill malware that ships with code: VirusTotal detected 72.8% of confirmed malicious skills that bundled code ⁴⁰. The 0% detection figure that circulated applies to payload-less Semantic Compliance Hijacking, where the entire attack is natural-language instructions ⁴¹. Scanners also disagree with each other: on the 61,990 ClawHub skills that three scanners all cover, they disagreed on 23,702, and after human adjudication their weighted sensitivity ranged from 21.67% to 61.06% ²⁴.

SkillShift, from early September, showed a subtler class of attack: skills that keep their declared task and valid outputs while covertly steering decisions. They reached attacker-favored selection rates of 81.33% in agentic commerce and 63.33% in software-dependency choices with utility fully preserved, and the scanners evaluated did not detect them ⁴².

For self-improving agents, the attack I would worry about most is self-poisoning. EVOMAL (August 2026) plants malicious skills in a shared library without ever invoking them. A self-evolving coding agent that writes new tools by imitating retrieved skills then copies the payload into its own new skill, stores it and runs it ²⁶.

Across six models on 153 SWE-bench Verified tasks, 20.3%–41.8% of tasks added a newly authored malicious skill, and poisoned libraries ended up holding 4.9–9.0 times as many malicious skills as were planted. After the planted skills were removed, Qwen3 still showed a 68% self-poisoning rate at round five, because its own copies remained ²⁶.

Those copies get past defenses that check attacker-submitted names, code and signatures, because the agent wrote them. A counter-prompt that discourages template-style copying cut the rate to at most 6.7% without significant loss in task completion ²⁶. Package provenance and signed catalogs protect installation. A library the agent writes for itself also needs provenance of its own, down to which retrieved skill each new one imitated.

Can an agent tell that a skill has gone stale when there is no release to diff against? Repo2Skill-Evo hands agents the patch and asks them to repair the skills, and even then the best agent reaches 69.7% F1 ²⁵. In deployment most drift arrives without a clean diff: an API changes behavior, a website is redesigned, a dependency deprecates a flag. Code2Skill-scale banks tie a million records to moving code ¹¹. Detecting that a skill has gone stale before it misleads a run is unsolved, and the harmful-skills result means the cost of missing it falls mostly on the skills used most.

Can per-task attribution be made cheap enough for a registry? ASSAY shows that a skill has to be evaluated per task type ²⁹, and SkillApt shows the effect can be estimated per state from matched runs ³. Both need with/without evidence that is expensive to collect for a registry of tens of thousands of skills. A cheap estimator learned from routine logs would let per-state activation move from benchmarks into registries.

Does meta-improvement converge, and how should a meta-level edit be accepted? MetaSkill-Evolve shows that evolving the improvement procedure helps, by 23.54 points on one benchmark and 1.92 on another ²³. I couldn't find evidence on whether the two-timescale process converges, oscillates or overfits its procedure to the benchmark that drives it, or on how to gate a meta-level edit whose effect shows up only after many task-level cycles. PACE-style sequential testing exists for task-level commits; nothing equivalent exists yet for procedure-level commits.

How should an agent record where its own skills came from? EVOMAL's worm keeps spreading because nothing records that a new skill was modeled on a retrieved one ²⁶. No system covered here tracks lineage for agent-written skills or uses acceptance tests that check behavior instead of signatures.