Fifteen Skills and the Tests That Measure Them
Two months after the eleven skills, three of them are gone and seven have arrived. What changed is that a skill gets measured now: whether it fires, whether a weaker model can follow it, and whether it does better than no skill at all.
Two months ago I wrote up the eleven skills in my Claude Code setup, and the policy file that fires them without my having to ask. The setup is airon, and it has fifteen skills now, which sounds like four were added, and is not what happened. Three of the eleven are gone, one skill arrived and was deleted again inside two weeks, and the cost scanner that the first post gave a whole section to has been removed, along with everything that depended on it.
What changed is that a skill gets measured now. In September a skill earned its place by having a line in the policy file. Now it is measured on whether it fires when it should, whether a weaker model can follow it, and whether it does better than no skill at all, and three of the original eleven would have failed the first of those every time.
What happened to the eleven
Eight survived, which are kiss, tdd, scientificmethod, investigate, plan, orchestrate,
adversarial-review and wikikb. The three that went all went for the same reason, which is that
nothing used them.
temporal-planning went first, in September. Across a month of sessions nothing invoked it, including
the plan and orchestrate runs that were told to hand off to it, and the one timeline that it did produce
overestimated every step that it later measured. model-routing and optimizer followed in October,
after six weeks of transcripts in which neither one was invoked at all. Routing's effect didn't show up
anywhere, and the ledger that the scanner wrote its findings into had no reader other than the
optimizer, which never ran.
That is the part worth stopping on. The first post argued that a skill nobody remembers to invoke is a bookmark, and that the policy file is what fixes it. All three of these had a trigger in the policy file, and they still never fired, which makes a rule that fires nothing a bookmark too, and nothing in the setup could tell me that it was happening.
The fourth deletion was the first one measured on whether it helped rather than on whether it ran.
reverse-engineer was a design-recovery skill, for working out how an existing system behaves
when no design document survives, and it was added in October. Within two weeks a blind benchmark had
run it against an unaided model on the same design-recovery task, three runs each, judged pairwise by
reviewers who could read the code. It found nothing that the unaided model missed, it was wrong less
often, with 1 of 30 sampled claims wrong against 4 of 30, and it cost about 2.4 times as much per run.
So the four practices that bought the accuracy moved into investigate, and the skill was deleted.
The seven that arrived are comment-review, skill-hardening, skill-shootout, recall,
suspend-resume, scheduled-jobs and taskforge, which used to be a CLI that the policy named and is a
skill of its own now. Two of them exist to test the others.
A weaker model reads each skill
skill-hardening came out of carlink, the tool I built for diagnosing my cars,
whose skill was written for the weakest agent that might ever run it. The method is to give a weaker
model, which is Haiku, a realistic practice job and nothing but the skill, and then to judge what it
actually did and said against a checklist that was written before it started. The skill is fixed in one
batch per round, the jobs that failed are run again alongside one new one, and the rounds stop when only
wording problems are left. On carlink that took four rounds and turned up 19 findings, three of them
high, and in the first round two of the three agents presented the simulator's made-up readings to
the person as though they had come from the real car.
On November 8th all of airon's skills went through it, which was eleven again, although not the same eleven, and the trials found things that rereading would not have:
- Asked for five findings, the
adversarial-reviewtrial manufactured five, so the skill has a causation test now, and a rule against filling a quota that the requester set kisslearned that a patch location named in a bug report is part of the symptom rather than the scope- An earlier fix to
tddsaid that each rule the person states needs a test, and in round 3 the trial wrote all 12 tests and the whole implementation in a single Green, so the clause was removed rather than answered with another one
That last one is an instance of a rule that now runs through all of this, and it gets its own section below.
Racing a skill against no skill
Hardening shows that a skill can be followed, and it doesn't show that following it helps, which is what
skill-shootout is for. It runs each arm, which is up to four skills plus a no-skill arm that it adds
itself, fresh and at the same time, on the same cases, with the same model on the same machine. An
isolated judge ranks the results under shuffled labels, and an exact sign test over the paired runs
calls an improvement, a regression or no detectable change. With three arms the 5% is shared across nine
tests, so over nine paired runs nothing short of nine to nothing counts.
The verdicts are not flattering, and that is what makes them worth having. The reworked orchestrate,
taskforge and skill-hardening each beat no skill on all nine runs, and scheduled-jobs beat it on 14
of 15. plan and wikikb showed no detectable value over no skill, and neither did skill-shootout
itself, although on the case where it had to set up a comparison the no-skill arm made seven medium
mistakes, among them no baseline and no cost cap, to its none. The skill is careful about what that
verdict means, which is that either the skill is not needed or the cases miss what it adds, and
suspend-resume shows why the second half matters, because it beat no skill nine to nothing on its
first cases and showed nothing detectable on a later set.
A shootout loads no policy file, so a skill there fires only when its own description matches what was
asked, and that makes the count of runs in which it fired the most telling number in the report.
wikikb fired in two of nine runs, and its previous version in three, because its description does not
trigger on a plain "make this change" request, and recall's first cases never made it fire at all, so
that first verdict tested nothing and the cases were rewritten. That is the bookmark problem from the
first post again, seen from the other side. Without the policy file a description is all that a skill
has to fire on, which is why the policy file still matters.
None of this is free. A shootout costs somewhere between one and ten dollars, and a skill costs tokens
each time it runs: the new orchestrate used a median of 158,528 tokens per run, against 63,181 with no
skill at all. The policy makes both checks a MUST now for any change to what a skill tells an agent to
do, with the verdict written into the commit message.
Knowing when to stop fixing
The first post made code-review a MUST because these repos have no CI. Since then review has gained a
stopping rule, which is that two consecutive rounds in which my own fix introduces a new finding of
medium or high severity means stop and re-scope, rather than fix again.
It arrived alongside a set of rules for scoping a guard, meaning a gate, a watcher or a pre-commit check, which fails open on any input it does not handle, so the review rounds on one end only when its input space does. The rules make that space finite before anything is written: gate on an identity rather than a grammar, never hand-roll a parser for a format that already has one, keep each invariant in exactly one place, prefer a loud refusal, and ask whether prose already does the job.
It fires often, and when it does, the answer has usually been to take something out. On the installer's
ownership logic, three review rounds running each found a defect in the previous round's fix, so the
design was cut back to the simplest version instead of being patched a fourth time. A column for plan, meaning a stream that can start now but has
to merge after another, went through nine review runs and three designs that each failed to converge,
and the version that shipped has orchestrate treat it as an ordinary dependency, with the head start
that it was meant to give written up as a proposal instead. On carlink the rule ended review five times
in a single eight-hour hardening pass, and the first of those was settled by deleting logic rather than
adding it.
comment-review belongs with it. An agent writes comments while it is reasoning, so its comments record
the reasoning rather than the code, and one migration shipped 2,610 added comment lines across 188
files, most of them chain-of-thought that was already in the commit messages. The test it applies to
each one is whether a future agent needs it and could discover it no other way.
What survived of the cost model
The scanner is gone, but the finding from it that mattered most survived, which was that always-loaded context is the most expensive thing there is, because every session pays for it on every turn. It changed the structure twice.
The first time, the skill policy turned out to be loading on each turn of every session on a machine that already loads an equivalent policy from another package, which was around 40 KB of near-duplicate standing context per turn, at a point when cache reads were about 95% of the bill. The second time, the rules became short stubs that hold only their triggers and prohibitions, with the full text read on demand, and the set that loads in every session went from 31,279 bytes to 8,642. It has crept back up to 9,669 since, which is what the test that fails above 12,000 is there for.
Shrinking what loads has an obvious failure, which is that something stops loading, so a tool called
rule-probes installs the setup into a throwaway home directory and asks a fresh session a question
that only one file can answer, alongside a control that has to come back NOT-LOADED. Run against the
old tree, the probes that check the full texts stay out of a new session failed, because there the full
texts loaded into each session, and that is how I know those probes can fail.
The fleet lost the most. The budget governor, which the first post called the primary control, priced
the workers by importing the scanner, so it went too, and the commit that removed it says plainly that
this removes a real cap. I said at the time that the governor was inferred rather than measured. What
paces a swarm of workers now is a spawn stagger, a live memory check and an optional cap on live workers
per board, which is the control I called the weakest, and the orchestrate skill says in its own
description that nothing bounds a wave's spend. Cost is reported per stream now, with a total for the
fleet, but nothing enforces it.
Everything personal moved out
In October everything about me left the repo. Memory, personal rules, settings, the investigations worth keeping and the list of scheduled jobs live in a private overlay repo now, which the installer layers on top. airon's own rules and skills name no person, and a test fails if the repo tracks a memory or names a home-directory path.
Memory follows me between machines, and it picked up a rule of its own. The index is the only part of memory that loads into a session, so a memory that is missing from the index is a file that nothing will ever recall, and I found one of mine in that state. Writing a memory never touches the index, because several machines appending to the same file collide when they sync and the loser's line is lost, so a separate pass catches the index up, from one machine at a time.
The rest of it runs the machine itself. recall searches what earlier Claude Code, Codex and Kiro
sessions did, redacted. suspend-resume records every tmux and agent session before a reboot and brings
them back afterwards. scheduled-jobs runs a list of recurring jobs from the overlay through one launchd
agent, and a new session is told when one of them failed. airon has a version number now, which is
1.4.0, and a changelog, and a session is told when a newer version is out or when it is running an older
one than the one installed. Codex shares the same rules and skills, and the skills that make sense on
claude.ai are packaged as zips for upload.
Two smaller changes are worth recording. A stream used to wait at merged for somebody to accept it, and
tasks sat there for days, so a merged review is the acceptance now. And taskforge is moving from its
CLI onto GitHub issues, so that agents on different machines share one record, which is built and not
yet verified, and the skill carries a banner that forbids real use until a verification run against a
sandbox project has been done. Until then every machine keeps using the CLI.
What it buys now
The first post ended by saying that none of this makes the agent smarter, it makes it consistent. That is still true, and the two months since have added the other half, which is that consistency is only worth having when the thing done consistently is worth doing. In September I believed I could make a skill fire without being asked. Now I can find out whether it does, and whether it should.
The failure that the first post protected against was a plausible answer, delivered confidently, that nobody checked. Skills fail the same way, as a plausible skill, installed confidently, that nobody measured, and three of my eleven were exactly that, which I had written a post about.