Methodology
How every skill is tested: three arms, 18 rule-scored runs on Codex, a security review, and a written verdict.
What gets listed
A skill is listed only after we run it. A repository of SKILL.md files is not a review. Example pages in this repository are drafts and stay unpublished. Skills we judged unsafe to run are not run.
The protocol
Each skill is run on three small tasks in three arms: no skill (control), no skill but with the skill's core rule written as one sentence in the prompt (one-sentence prompt), and the skill installed (treatment). The prompts never name the skill. Each task and arm runs twice, so a full review is 18 graded runs. Runs use a fresh temporary project, temporary HOME and caches, and no network.
Scoring is rule-based: hidden unit tests, planted-issue recall, constraint checks, or render and OCR checks, 0 to 5 per run. No language model decides a score. Differences under about 1 point are noise at n=2.
The score on every page is a total out of 30: the six dimensions each run from 0 to 5 and are added up. The dimensions are output quality, lift over no skill, how reliably the skill triggers, install and portability, token and time cost (5 is cheap and fast), and safety hygiene. Next to the total, the page shows the lift: the mean task score with the skill minus the mean without it, on the 0 to 5 task scale.
The verdict is stated by the test pipeline and uses the numbers shown on the page. Recommended needs all three: a total of at least 22 out of 30, output quality of at least 3.5, and a lift of at least +1.0. Not recommended applies when the total is below 14 out of 30, output quality is below 2.5, or the skill scores a full point worse than no skill. Everything else is Situational. A skill can therefore show 24 out of 30 and still be Situational, because its lift is under +1.0; each page says which criterion it misses. The one-sentence arm shows whether the skill beats pasting its rule as one line; it does not enter the score.
Every page lists what was changed after the fact, for example checks that were loosened and re-scored, runs that were not completed, and skills tested at an older commit.
Tiers
Every skill is tested on Codex. A pass there also covers Claude Code. We do not copy that result to other agents.
| Tier | Tested with | A pass also covers | | ----------- | ----------- | ------------------ | | SOTA agents | Codex | Claude Code |
A second tier on a cheaper model (DeepSeek Flash) is being re-tested and will be added when it is done. Until then it shows as not tested.
The review page shows the outputs side by side: files, screenshots, before and after excerpts, and video with a player. Video generation is a top-level category.
Security review
Each skill also gets a security review from the static audit and from what the sandboxed run actually did.
- Static audit: what the skill file asks the agent to do.
- Risky permissions: names the audit flagged, such as network or reads outside the workspace.
- Network and file access: what we observed while the skill ran.
- Badge: Safe, Caution, or Unsafe. The same badge is on the list card.
What is public
The verdict, scores, written review, and the outputs embedded on the page are public. Transcripts, downloadable run files, and the raw score of every run are Pro.
Adding a review
Real reviews come from the test pipeline. See the README for the directory layout. A review stays draft until the runs are real.