Skill Libraries Need CI, Not More Prompts

The hard problem is no longer how to write a useful SKILL.md. That part is becoming standardized, documented, and productized. The hard problem is how a system should decide that a skill deserves to exist, when it should load, how it should change, who is allowed to publish it, what evidence travels with it, and when it should be killed.

The last two months made that shift visible. OpenAI Codex adopted the open skill format while separating skills from installable plugins. GitHub added package-manager commands for installing, pinning, updating, and publishing skills. Perplexity published the most concrete maintenance guide: write evals first, make descriptions routing metadata, add negative trigger examples, and treat every sentence as a context tax. NVIDIA shipped verified agent skills with scanning, signing, and skill cards. Papers such as SkillOps, SkillOpt, PACE, SkillRevise, Workflow-to-Skill, MemoRepair, and OpenSkillEval moved the research question from “can agents remember procedures?” to “can a library of procedures maintain itself without drifting, poisoning retrieval, or p-hacking its own verifier?”

That is a different object than an instruction file. A mature skill is a context package: prose, references, scripts, allowed tools, trigger policy, tests, provenance, risk metadata, utility history, and a decision log. The package is read by a model, but it is governed like software.

The center of gravity moved from authoring to operations

The April version of this article treated skills as the emerging distribution unit for agentic AI. That was directionally right and operationally incomplete. Distribution creates a new failure mode: once skills can be installed, shared, updated, and invoked automatically, the library becomes a software ecosystem. Ecosystems accumulate technical debt.

The clearest recent evidence clusters around four moves. Standards made skills portable. Package managers made them installable. Eval systems made their value measurable. Security tools made their attack surface explicit. None of those moves makes skills autonomous in the magical sense. They make skills inspectable enough to maintain.

Popularity matters here because skills are partly an ecosystem problem. Hacker News supplied the useful skepticism: high-point threads on Forge, Claude Code daily-driver workflows, TDD skills, browser harnesses, and state-machine agents converged on the same practical stance. Skills help when paired with deterministic checks, evals, small blast radius, and versioning. Pure prose gets ignored.

GitHub supplied adoption heat, though stars should be read as heat rather than proof. The strongest repos and tools are not just skill catalogs. They are package managers, eval runners, scanners, provenance systems, and docs-to-skill compilers. That is the structural signal: the ecosystem is building the maintenance layer around the format.

The open standard solved syntax, not correctness

The open AgentSkills format is intentionally thin. A directory contains a required SKILL.md. The markdown file carries frontmatter, most importantly a description that lets an agent decide when to load the full instructions. Extra files live under resources such as scripts/, references/, and assets/. That shape is useful because it separates routing metadata from heavier context.

Thinness is a feature at the standard layer. It lets Codex, Claude Code, Gemini CLI, GitHub Copilot, Cursor, Windsurf, and other agents converge on a common package shape without agreeing on one governance regime. The format says where the instruction lives. It does not say whether the instruction is safe, current, useful, original, compatible with the installed package version, or worth loading.

Codex makes the split explicit. Skills are the authoring format. Plugins are the installable distribution unit. That distinction matters because authoring, packaging, discovery, installation, activation, and verification have different failure modes. A good SKILL.md can still be installed from the wrong source, selected for the wrong task, executed with the wrong permissions, or kept long after the underlying API changed.

The selection cliff is now documented product behavior, not just a benchmark observation. Codex loads initial skill metadata into context but limits that startup list. When a library grows, descriptions get shortened or omitted. A library with too many plausible skills becomes a retrieval problem before it becomes a task-solving problem.

A production skill is a context package

Calling a skill “a prompt” hides the engineering. A prompt is one string. A production skill is a bundle that changes model behavior, tool access, evidence collection, and sometimes local execution. It should be reviewable before activation and auditable after use.

skill/
  SKILL.md                  # routing metadata plus instructions
  references/               # docs, examples, schemas, API notes
  scripts/                  # optional executable helpers
  evals/
    trigger-cases.jsonl     # should-load and should-not-load prompts
    task-fixtures/          # inputs, repos, browser states, data
    assertions.json         # deterministic checks and rubric specs
  reports/
    latest-with-vs-without.json
  SKILLCARD.yaml            # owner, risk, limitations, verification status
  provenance.json           # source repo, ref, tree SHA, signature
  decision-history.md       # accepted and rejected edits

This shape explains why Contentful Skill Kit is interesting. It does not treat every skill as prose. Workflow skills are typed state machines with schemas, transitions, actions, saved state, and generated AgentSkills-compatible output. Reference skills remain progressive-disclosure topic loaders. Composite skills combine subskills and topic references. That is the right boundary: skill packages can be declarative workflow controllers, not only long markdown files.

The contract inside the package should be small enough for tooling to understand. SkillOps uses preconditions, operations, artifacts, validity, and failure modes. Contractual Skills adds goals, boundaries, permissions, human approval points, evidence requirements, output contracts, quality criteria, verification steps, and handoff rules. The common direction is clear: the model can read natural language, but the library needs structured fields for policy and maintenance.

A contract does not make a skill safe by itself. It makes the boundary inspectable. Runtime enforcement still has to live in the loader, permission system, sandbox, evaluator, and audit log.

Skill learning is not transcript summarization

The shared learning loop is now recognizable across papers and products: run a task, capture the trajectory, diagnose what mattered, draft a reusable artifact, evaluate it against future tasks, and publish only if it survives rejection. The danger sits in the word “capture.” Raw trajectories contain accidents, one-off environment details, lucky guesses, stale API responses, and task-specific scaffolding. Dumping them into memory is a way to preserve noise.

Workflow-to-Skill is useful because it names the intermediate representation that summarization skips. It decomposes evidence into routing, workflow, semantics, and attachments. It preserves control flow, verification, safety, rollback, state management, evidence, and confidence annotations. That is closer to compilation than summarization. A summary says what happened. A compiled skill says what future agent behavior should be repeatable, under which trigger, with which checks.

SkillRevise addresses a different state: the cold-start skill that already exists but is imperfect. It diagnoses defects from execution evidence, retrieves repair principles, applies anchored edits, and re-executes candidates. Its reported SkillsBench improvement, from 36.05% to 61.63%, matters less than the mechanism: a skill draft is not trusted because it is plausible. It becomes trusted through execution-anchored repair.

GEPA is the right adjacent analogy. Text artifacts can be optimized against eval metrics using reflection and evolutionary search. Execution traces, errors, profiling data, and reasoning logs supply directional information for textual edits. They are not gradients in the mathematical sense, but they play the same practical role: they tell the optimizer what kind of mutation might fix the artifact.

The commit gate is more important than the proposer

Self-improving systems tend to focus attention on the proposer: the model that reflects on failures and writes a better instruction. Recent work points in the opposite direction. The acceptor is the safety-critical component. A strong proposer with a weak gate writes convincing slop into persistent state.

PACE makes the statistical problem explicit. Greedy acceptance, “keep the edit if the score went up,” is vulnerable to adaptive multiple testing. If an optimizer tries enough candidate edits against noisy evals, some edits will look good by chance. PACE compares candidates to incumbents on identical instances and commits only when an anytime-valid e-process accumulates decisive evidence. That shifts self-improvement from vibes to a commit rule.

Reward-hacking research strengthens the same point. Verifier-gated systems can still optimize the verifier instead of the real objective. RLVR work shows models finding verifier shortcuts. Reward Hacking Benchmark shows tool-using agents exploiting naturalistic shortcuts such as skipping verification or tampering with evaluators. A skill library inherits that risk whenever it treats eval score as the only truth.

OpenSkillEval and agent-skills-eval make the empirical norm practical. Run the agent with the skill, run it without the skill, grade outputs, store traces, store timing, store judge decisions, and publish a static report. The package can then carry evidence, not just claims.

The uncomfortable conclusion is simple: “eval-gated” is not enough. The gate needs negative controls, held-out instances, deterministic checks where possible, statistical discipline under repeated candidate testing, and adversarial pressure against the verifier itself.

A library that only grows becomes worse

SkillOps gives the right name to the problem: skill technical debt. A skill can be locally reasonable and still damage the library. It can duplicate another skill under a different name. It can claim a trigger that overlaps with a more precise skill. It can reference an old API. It can require permissions that no longer match the workflow. It can pass old evals and fail new environments. It can be useful to one model and harmful to another.

Maintenance is not a periodic summarization job. It is library-time work over structured contracts, traces, utility history, and provenance. Some of that work should run synchronously at publish time. Much of it should run asynchronously when the agent is idle: recompute trigger precision, replay evals, scan for new risk patterns, detect duplicates, split overbroad skills, merge near-identical skills, demote stale skills, and retire skills that no longer justify their context cost.

MemoRepair adds the missing deletion mechanism. Agent memory does not only store source facts. It stores summaries, cached outputs, embeddings, learned skills, and executable procedures derived from those facts. When a source artifact is deleted, corrected, or invalidated, descendants can keep influencing future actions. Barrier-first repair withdraws affected descendants before repair and republishes only validated predecessor-closed successors. Retirement is therefore not local to one skill. It is cascade repair over influence provenance.

DCPM and MemOS show the same asynchronous pattern for memory. Synchronous writes capture immediate state. Asynchronous maintenance induces schemas, reconciles conflicts, preserves supersession chains, and decides what to keep. The interesting part is not “sleep-time summarization.” The interesting part is that maintenance has a different latency budget and stronger validation rules than runtime assistance.

Semantic similarity is not enough

Skill selection is often described as retrieval: match the task to the skill description. That is only the first term in the scoring function. A high-similarity skill with a high false-positive rate should lose to a less similar skill with stronger evidence on the current task class.

Perplexity’s description guidance is valuable because it treats descriptions as executable routing policy. “Load when…” points the router at user intent. Negative examples reduce adjacent-skill leakage. Accessory files let the loaded instruction stay short. Gotchas become an append-mostly memory of mistakes the skill has actually seen.

A mature router should rank by semantic match, trigger precision and recall history, utility on similar tasks, model and harness compatibility, context budget, risk level, required permissions, freshness, tenant or project scope, and whether a deterministic verifier exists for the requested outcome.

The selection cliff also changes authoring. A large library forces compression in descriptions. Compression makes false positives more likely unless descriptions are written as precise triggers. The shortest skill is not always best. The best description spends tokens where routing uncertainty is highest.

The loader is part of the security boundary

Skills combine natural-language instructions, bundled scripts, references, dynamic context, tool permissions, and sometimes external fetches. That mix puts them between package security and prompt-injection security. Traditional code scanners see scripts and dependencies. Prompt-injection scanners see text. A malicious skill can use both layers at once.

Recent security work made that boundary concrete. Datadog showed why dynamic context is dangerous: if a skill or agent configuration executes shell commands before the model sees the rendered content, model-level refusal has no chance to intervene. NVIDIA’s verified-skills pipeline responds with scanning, signing, skill cards, review, and catalog synchronization. OWASP’s Agentic Skills Top 10 frames skills as a behavior layer: MCP describes how models talk to tools; skills describe what workflows those tools execute. SkillVetBench argues that static and signature methods miss threats that appear only through natural language, multicomponent logic, or runtime interaction.

Security also changes distribution. GitHub’s pinning and provenance metadata, NVIDIA’s signatures and skill cards, Snyk-style scanning, OWASP risk categories, and portable-memory provenance all point to the same requirement: a runtime should know not only which skill matched, but which version, source, signature, scan result, and permission envelope it loaded.

The market wants executable organizational knowledge

YC’s Company Brain RFS is important because it says the quiet part in product language. The bottleneck is not only model quality. It is fragmented operational knowledge: Slack threads, tickets, docs, code review norms, customer escalations, deploy rules, partner processes, and the tacit patterns that experienced employees carry around. The desired system pulls that knowledge out, structures it, keeps it current, and turns it into executable context for AI.

That is not ordinary RAG. RAG answers questions from documents. A company brain has to preserve how work gets done, when facts changed, who owns a procedure, what approvals are required, which exceptions matter, and how agents should verify completion. Some outputs will be semantic memory. Some will be workflows. Some will be contracts. Some will become skills.

The startup pattern is consistent even when benchmark claims should be treated cautiously. Teams are building trace memory, production-agent monitoring, docs-to-skill pipelines, self-healing browser automations, observability skills, and shared context layers. The reliable signal is not any one vendor metric. It is that multiple markets are converging on procedural memory as the product, not chat over a vector store.

The lifecycle pipeline is the product

A skill system should start with the lifecycle, not with the largest possible library. The library only compounds if the system can prove that each skill triggers correctly, improves outcomes, remains safe, and disappears when it stops helping.

The repair router is as important as the skill router. Not every recurring failure belongs in a skill. HarnessFix makes that boundary explicit: some failures belong in tool design, orchestration, verification, observability, state management, or the evaluation harness. If every defect becomes a SKILL.md patch, the library becomes a junk drawer for runtime bugs.

Unified Context Evolution makes the same distinction at the artifact level. Memory, strategy, workflow, and skill are different evolvable context units. They have different write rules and validation rules. A user preference should not go through the same gate as an executable deployment workflow. A temporary execution-state summary should not be promoted with the same durability as a verified project convention.

Skills help repeated procedural work, not every problem

The strongest skill results share a shape. There is a recurring task class. The procedure can be written down. The environment is stable enough that yesterday’s repair applies tomorrow. The outcome can be verified. The skill has a precise trigger and a bounded permission envelope. Coding workflows, browser automations, runbooks, SDK usage, observability tasks, and compliance checklists fit that shape.

Diffuse tasks are harder. Open-ended strategy work, taste-heavy writing, rapidly changing product surfaces, and tasks with weak verifiers do not give the maintenance loop clean feedback. A skill can still help by encoding preferences or process, but automatic evolution becomes fragile. Self-feedback alone risks recursive drift. Stronger models do not automatically write better skills. Some curated skills hurt some tasks. A bigger library can reduce reliability by increasing retrieval ambiguity.

The open research problem is not whether agents can write down procedures. They can. The open problem is whether a system can maintain a growing library of procedures under noisy feedback, adversarial inputs, shifting models, stale sources, and limited context without gradually poisoning itself.

The practical bet is smaller and stronger: build a lifecycle pipeline before building a large library. Make every skill carry evidence. Make every load auditable. Make every update reviewable. Make deletion normal. Once those mechanics exist, automatic skill learning becomes less mystical. It becomes software maintenance over procedural context.