Most of us treat Claude Skills as free upgrades. Install a few, and your agent gets better. There are whole lists of “the best Claude Skills” built on that idea.
I wanted a number for it, so I raced the same agent against itself on one coding task, once without a Skill and once with one, five runs a side, on real API bills.
📺 Watch the video: Claude Skills: Do They Actually Make Your Agent Better? I Measured It
How a Skill loads
A Skill is a folder with a SKILL.md in it. The agent doesn’t read the whole file up front. At the start it sees one line per Skill, the name and the description, and it loads the rest only when a task needs it. Scripts in the folder run without the agent reading them.
Think of a cook with a drawer of recipe cards. The cook can read the tabs at a glance and pulls a card only when the order calls for it. A long system prompt is the whole cookbook pinned to the wall above the stove, paid for on every order.
The Skill in the race holds a small till repo’s house rules. Here is its front matter:
---
name: till-house-rules
description: The till's house rules for any change to this repo (money in whole pence, where a new block goes on the receipt, everything a new flag has to touch, tests), and a script that checks a change against them. Use before you add or change a flag, the receipt or the JSON, and run the script before you say you're done.
---
The setup
Both sides used the Claude Agent SDK with Read, Edit, Bash and the Skill tool, on Claude Sonnet 5.5 at the default effort. The task was the same for both: add paying cash to a shop’s till, from the spec. Each run started from a fresh copy of the repo, and the sides took turns. A hidden check with 16 tests graded every run, and 5 of those tests were house rules. The rules were also written in the repo’s own docs, so the side without the Skill had a fair chance to find them.
The only difference between the sides was whether the Skill folder was there.
Three tests before the race
1. What an unused Skill costs. The first call’s prompt was 4,036 tokens with no Skill, 4,153 with one (+117) and 5,023 with ten (+987). Those tokens are cached, so 1,000 calls cost about a cent. When I asked for something none of the ten Skills fit, the agent loaded none of them (2 of 2 runs).
2. A bad Skill. I changed one rule to an out-of-date one: new receipt lines “straight after TOTAL”, while the repo’s docs say after the VAT block. It still passed 3 of 3. The agent read the docs and went with the repo.
3. The same rules three ways. I put the same text in CLAUDE.md, in the system prompt and in a Skill, three runs each. All 9 passed.
| Where the rules live | Cost per run |
|---|---|
| CLAUDE.md | $0.092 |
| System prompt | $0.094 |
| Skill | $0.100 |
The race
| Without the Skill | With the Skill | |
|---|---|---|
| Passed | 4 of 5 | 5 of 5 |
| Median time | 37 s | 35 s |
| Median turns | 8 | 6 |
| Skill loaded | turn 1, every run | |
| Cost per passing run | $0.1216 | $0.0921 |
Cost per passing run is everything a side spent divided by the runs that passed. The run without the Skill that failed was round 1, where it skipped the CLI test the house rules ask for. With the Skill, each passing run cost about a quarter less (1.3x).
Everything in the video, all the tests and the race, came to $2.36 on the API, and the Anthropic Console agreed to the cent.
What I took from it
Put rules you need on some tasks in a Skill, and rules you need on every task in CLAUDE.md. Keep the source of truth in your repo, so a bad Skill gets caught the way it did in test 2.
This was one task on one model, so measure a Skill on your own task before you believe anyone’s list of the best ones, mine included.
Try it on your own Skill
I built skill-check for that. It’s free and open source. Point it at your project, your task and your Skill, and it runs the same agent with and without the Skill in fresh copies of your project, then prints the pass rate, time, tokens, when the Skill loaded and the cost per passing run. It uses your own API key, so give it a Console workspace with a spend limit.
git clone https://github.com/thecoderlog/skill-check
cd skill-check
npm install
export ANTHROPIC_API_KEY=...
node skill-check.mjs --repo ~/code/my-app \
--skill ~/code/my-app/.claude/skills/house-rules \
--task "Add a --dry-run flag to the deploy command. Add tests." \
--check "npm test" --runs 3
Run it on your favourite Skill and tell me in the comments whether it actually helped.
Code & video
- 📺 Full video: Claude Skills: Do They Actually Make Your Agent Better? I Measured It
- ⚡ The race as a Short: youtube.com/shorts/7AQasZpIRRs
- 💻 skill-check: github.com/thecoderlog/skill-check
- 📚 Agent Skills docs: Agent Skills overview, Skills in the Agent SDK, Skills in Claude Code
- 🔁 Related: Claude Agent SDK vs Claude Code: same engine, who wins the race? and the prompt caching race (why an unused Skill is nearly free)
If this was useful, subscribe to The Coderlog on YouTube for the next race.