← Blog
13 min read

Fifteen Skills and the Tests That Measure Them

Two months after the eleven skills, three of them are gone and seven have arrived. What changed is that a skill gets measured now: whether it fires, whether a weaker model can follow it, and whether it does better than no skill at all.

Two months ago I wrote up the eleven skills in my Claude Code setup, and the policy file that fires them without my having to ask. The setup is airon, and it has fifteen skills now, which sounds like four were added, and is not what happened. Three of the eleven are gone, one skill arrived and was deleted again inside two weeks, and the cost scanner that the first post gave a whole section to has been removed, along with everything that depended on it.

What changed is that a skill gets measured now. In September a skill earned its place by having a line in the policy file. Now it is measured on whether it fires when it should, whether a weaker model can follow it, and whether it does better than no skill at all, and three of the original eleven would have failed the first of those every time.

What happened to the eleven

Eight survived, which are kiss, tdd, scientificmethod, investigate, plan, orchestrate, adversarial-review and wikikb. The three that went all went for the same reason, which is that nothing used them.

temporal-planning went first, in September. Across a month of sessions nothing invoked it, including the plan and orchestrate runs that were told to hand off to it, and the one timeline that it did produce overestimated every step that it later measured. model-routing and optimizer followed in October, after six weeks of transcripts in which neither one was invoked at all. Routing's effect didn't show up anywhere, and the ledger that the scanner wrote its findings into had no reader other than the optimizer, which never ran.

That is the part worth stopping on. The first post argued that a skill nobody remembers to invoke is a bookmark, and that the policy file is what fixes it. All three of these had a trigger in the policy file, and they still never fired, which makes a rule that fires nothing a bookmark too, and nothing in the setup could tell me that it was happening.

kept
8 of 11
kisstddscientificmethodinvestigateplanorchestrateadversarial-reviewwikikb
removed
3 + 1
temporal-planningSep 25not invoked in a month of sessions, even by the runs told to hand off to it
model-routingOct 23not invoked once in six weeks of transcripts, and its effect never appeared
optimizerOct 23not invoked once, and the scanner's ledger had no other reader
reverse-engineerOct 19 – Nov 1found no more than an unaided model did, at 2.4× the cost; its practices moved into investigate
added
7
comment-reviewskill-hardeningskill-shootoutrecallsuspend-resumescheduled-jobstaskforge
Eleven became fifteen, and the count hides most of it. Each of the three that went had a trigger in the policy file and still never fired. A fourth arrived in October and was gone thirteen days later, after a benchmark. Two of the seven that arrived exist to test the others.

The fourth deletion was the first one measured on whether it helped rather than on whether it ran. reverse-engineer was a design-recovery skill, for working out how an existing system behaves when no design document survives, and it was added in October. Within two weeks a blind benchmark had run it against an unaided model on the same design-recovery task, three runs each, judged pairwise by reviewers who could read the code. It found nothing that the unaided model missed, it was wrong less often, with 1 of 30 sampled claims wrong against 4 of 30, and it cost about 2.4 times as much per run. So the four practices that bought the accuracy moved into investigate, and the skill was deleted.

The seven that arrived are comment-review, skill-hardening, skill-shootout, recall, suspend-resume, scheduled-jobs and taskforge, which used to be a CLI that the policy named and is a skill of its own now. Two of them exist to test the others.

A weaker model reads each skill

skill-hardening came out of carlink, the tool I built for diagnosing my cars, whose skill was written for the weakest agent that might ever run it. The method is to give a weaker model, which is Haiku, a realistic practice job and nothing but the skill, and then to judge what it actually did and said against a checklist that was written before it started. The skill is fixed in one batch per round, the jobs that failed are run again alongside one new one, and the rounds stop when only wording problems are left. On carlink that took four rounds and turned up 19 findings, three of them high, and in the first round two of the three agents presented the simulator's made-up readings to the person as though they had come from the real car.

On November 8th all of airon's skills went through it, which was eleven again, although not the same eleven, and the trials found things that rereading would not have:

  • Asked for five findings, the adversarial-review trial manufactured five, so the skill has a causation test now, and a rule against filling a quota that the requester set
  • kiss learned that a patch location named in a bug report is part of the symptom rather than the scope
  • An earlier fix to tdd said that each rule the person states needs a test, and in round 3 the trial wrote all 12 tests and the whole implementation in a single Green, so the clause was removed rather than answered with another one

That last one is an instance of a rule that now runs through all of this, and it gets its own section below.

Racing a skill against no skill

Hardening shows that a skill can be followed, and it doesn't show that following it helps, which is what skill-shootout is for. It runs each arm, which is up to four skills plus a no-skill arm that it adds itself, fresh and at the same time, on the same cases, with the same model on the same machine. An isolated judge ranks the results under shuffled labels, and an exact sign test over the paired runs calls an improvement, a regression or no detectable change. With three arms the 5% is shared across nine tests, so over nine paired runs nothing short of nine to nothing counts.

skill-hardening
A weaker model gets a realistic job and nothing but the skill, judged against a checklist written before it starts.
rounds until only wording is left
skill-shootout
Every arm runs fresh and at once, a no-skill arm always among them, ranked by a judge who cannot tell which is which.
exact sign test
commit
The verdict goes in the commit message, and the change goes to review.
no CI behind it
Against no skill, so far · won–lost–tied over paired runs
orchestrateADDS VALUE9–0–0
taskforgeADDS VALUE9–0–0
skill-hardeningADDS VALUE9–0–0
scheduled-jobsADDS VALUE14–1–0over fifteen runs
suspend-resumeNOTHING DETECTABLE7–2–09–0–0 on its first cases
recallNOTHING DETECTABLE7–1–1its first cases never made it fire
planNOTHING DETECTABLE4–4–1and more tokens
skill-shootoutNOTHING DETECTABLE6–2–1no medium defects where no skill made 7
wikikbNOTHING DETECTABLE—fired in 2 of 9 runs
not raced yetkisstddscientificmethodinvestigateadversarial-reviewcomment-review
Both checks are a MUST for any change to what a skill tells an agent to do. Below them, the verdict against no skill for every skill raced so far. With three arms the 5% is shared across nine tests, so over nine paired runs only nine to nothing counts. Nothing detectable means the skill may not be needed, or the cases miss what it adds.

The verdicts are not flattering, and that is what makes them worth having. The reworked orchestrate, taskforge and skill-hardening each beat no skill on all nine runs, and scheduled-jobs beat it on 14 of 15. plan and wikikb showed no detectable value over no skill, and neither did skill-shootout itself, although on the case where it had to set up a comparison the no-skill arm made seven medium mistakes, among them no baseline and no cost cap, to its none. The skill is careful about what that verdict means, which is that either the skill is not needed or the cases miss what it adds, and suspend-resume shows why the second half matters, because it beat no skill nine to nothing on its first cases and showed nothing detectable on a later set.

A shootout loads no policy file, so a skill there fires only when its own description matches what was asked, and that makes the count of runs in which it fired the most telling number in the report. wikikb fired in two of nine runs, and its previous version in three, because its description does not trigger on a plain "make this change" request, and recall's first cases never made it fire at all, so that first verdict tested nothing and the cases were rewritten. That is the bookmark problem from the first post again, seen from the other side. Without the policy file a description is all that a skill has to fire on, which is why the policy file still matters.

None of this is free. A shootout costs somewhere between one and ten dollars, and a skill costs tokens each time it runs: the new orchestrate used a median of 158,528 tokens per run, against 63,181 with no skill at all. The policy makes both checks a MUST now for any change to what a skill tells an agent to do, with the verdict written into the commit message.

Knowing when to stop fixing

The first post made code-review a MUST because these repos have no CI. Since then review has gained a stopping rule, which is that two consecutive rounds in which my own fix introduces a new finding of medium or high severity means stop and re-scope, rather than fix again.

It arrived alongside a set of rules for scoping a guard, meaning a gate, a watcher or a pre-commit check, which fails open on any input it does not handle, so the review rounds on one end only when its input space does. The rules make that space finite before anything is written: gate on an identity rather than a grammar, never hand-roll a parser for a format that already has one, keep each invariant in exactly one place, prefer a loud refusal, and ask whether prose already does the job.

It fires often, and when it does, the answer has usually been to take something out. On the installer's ownership logic, three review rounds running each found a defect in the previous round's fix, so the design was cut back to the simplest version instead of being patched a fourth time. A column for plan, meaning a stream that can start now but has to merge after another, went through nine review runs and three designs that each failed to converge, and the version that shipped has orchestrate treat it as an ordinary dependency, with the head start that it was meant to give written up as a proposal instead. On carlink the rule ended review five times in a single eight-hour hardening pass, and the first of those was settled by deleting logic rather than adding it.

comment-review belongs with it. An agent writes comments while it is reasoning, so its comments record the reasoning rather than the code, and one migration shipped 2,610 added comment lines across 188 files, most of them chain-of-thought that was already in the commit messages. The test it applies to each one is whether a future agent needs it and could discover it no other way.

What survived of the cost model

The scanner is gone, but the finding from it that mattered most survived, which was that always-loaded context is the most expensive thing there is, because every session pays for it on every turn. It changed the structure twice.

The first time, the skill policy turned out to be loading on each turn of every session on a machine that already loads an equivalent policy from another package, which was around 40 KB of near-duplicate standing context per turn, at a point when cache reads were about 95% of the bill. The second time, the rules became short stubs that hold only their triggers and prohibitions, with the full text read on demand, and the set that loads in every session went from 31,279 bytes to 8,642. It has crept back up to 9,669 since, which is what the test that fails above 12,000 is there for.

global-CLAUDE.md
1,339 bytes
machine-wide instructions, and how to read a pointer
rules/skill-policy.md
6,765 bytes
reference/skill-policy.md
read when a trigger fires
26,118 bytes
rules/rust-first-tooling.md
1,055 bytes
rules/scoped/rust-first-tooling.md
loads when a Rust file is touched
8,049 bytes
rules/memory.md
510 bytes
nothing to cut
31,279
bytes in every session before the split
8,642
right after it
9,669
today, against a test that fails above 12,000
Every session pays for what it loads, on every turn. The rules that load are stubs now, holding the triggers, the prohibitions and a pointer to the full text, and rule-probes checks in a throwaway install that each one still loads, alongside a control that has to answer NOT-LOADED.

Shrinking what loads has an obvious failure, which is that something stops loading, so a tool called rule-probes installs the setup into a throwaway home directory and asks a fresh session a question that only one file can answer, alongside a control that has to come back NOT-LOADED. Run against the old tree, the probes that check the full texts stay out of a new session failed, because there the full texts loaded into each session, and that is how I know those probes can fail.

The fleet lost the most. The budget governor, which the first post called the primary control, priced the workers by importing the scanner, so it went too, and the commit that removed it says plainly that this removes a real cap. I said at the time that the governor was inferred rather than measured. What paces a swarm of workers now is a spawn stagger, a live memory check and an optional cap on live workers per board, which is the control I called the weakest, and the orchestrate skill says in its own description that nothing bounds a wave's spend. Cost is reported per stream now, with a total for the fleet, but nothing enforces it.

Everything personal moved out

In October everything about me left the repo. Memory, personal rules, settings, the investigations worth keeping and the list of scheduled jobs live in a private overlay repo now, which the installer layers on top. airon's own rules and skills name no person, and a test fails if the repo tracks a memory or names a home-directory path.

Memory follows me between machines, and it picked up a rule of its own. The index is the only part of memory that loads into a session, so a memory that is missing from the index is a file that nothing will ever recall, and I found one of mine in that state. Writing a memory never touches the index, because several machines appending to the same file collide when they sync and the loser's line is lost, so a separate pass catches the index up, from one machine at a time.

The rest of it runs the machine itself. recall searches what earlier Claude Code, Codex and Kiro sessions did, redacted. suspend-resume records every tmux and agent session before a reboot and brings them back afterwards. scheduled-jobs runs a list of recurring jobs from the overlay through one launchd agent, and a new session is told when one of them failed. airon has a version number now, which is 1.4.0, and a changelog, and a session is told when a newer version is out or when it is running an older one than the one installed. Codex shares the same rules and skills, and the skills that make sense on claude.ai are packaged as zips for upload.

Two smaller changes are worth recording. A stream used to wait at merged for somebody to accept it, and tasks sat there for days, so a merged review is the acceptance now. And taskforge is moving from its CLI onto GitHub issues, so that agents on different machines share one record, which is built and not yet verified, and the skill carries a banner that forbids real use until a verification run against a sandbox project has been done. Until then every machine keeps using the CLI.

What it buys now

The first post ended by saying that none of this makes the agent smarter, it makes it consistent. That is still true, and the two months since have added the other half, which is that consistency is only worth having when the thing done consistently is worth doing. In September I believed I could make a skill fire without being asked. Now I can find out whether it does, and whether it should.

The failure that the first post protected against was a plausible answer, delivered confidently, that nobody checked. Skills fail the same way, as a plausible skill, installed confidently, that nobody measured, and three of my eleven were exactly that, which I had written a post about.