test-driven-development (obra/superpowers)

Name: test-driven-development (obra/superpowers) review
Item: test-driven-development (obra/superpowers)
Rating: 24
Author: Agent Skills

superpowers test-driven-development: makes the agent write the test first, and costs about 1.6x the tokens

RecommendedSafe

24.0 / 30

Lift over no skill: +1.00

Recommended: the total 24.0/30 is at least 22, output quality 5.0 is at least 3.5, and the lift +1.00 is at least +1.0.

SOTA

Pass

Tested with Codex.

Covers Codex, Claude Code

Codex CLI 0.166 (SOTA tier): 18 graded runs. A pass here also covers Claude Code under our tier rule; it says nothing about the second tier.

Scores

Each dimension is 0 to 5 and the six add up to the total out of 30. 5 is better on every dimension. Cost 5 means cheap and fast. Install 5 means easy.

Trigger reliability: 5.0
Lift over no skill: 3.0
Output quality: 5.0
Token and time cost: 2.0
Install and portability: 4.0
Safety: 5.0

Security review

Safe

Static scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Manual review: Safe to run in a no-network sandbox (no findings). A clean scan does not prove a skill safe.

Install

npx skills@latest add obra/superpowers --skill test-driven-development

Source

Pinned commit: 2ce8aa15d05bd57213e6f746f953aafb7ded7a8b

Tested: 2026-11-02

What we tested

Implement a function from a precise spec — Implement slugify(text, max_length=None) in slug.py (Python 3, standard library only). Spec: lowercase the text; replace each run of characters that are not ASCII letters or digits with a single '-'; strip leading and trailing '-'; an empty result becomes 'n…
Implement a small stateful class — Write ratelimiter.py (Python 3, standard library only) with class RateLimiter(limit, window_seconds, clock=time.monotonic): a sliding-window limiter per key. allow(key) returns True and records the call if fewer than limit calls were recorded in the last…
Fix a rounding bug — Customers say cart_total in cart.py is sometimes a cent off compared with our accounting system, which rounds half-up to the cent at each step (each line total, the discounted subtotal, and the tax). For example cart_total([(1.005, 1)]) should return 1.09 an…

Pros

Lifts the mean score from 4.0 to 5.0 over no skill (+1.00).
The skill was picked up without being named: Skill file read in 6/6 treatment runs.
Beats a one-sentence version of its own rule by +0.33.
High absolute quality with the skill: 5.0 out of 5.

Cons

Uses 1.65x the tokens of no skill.
Test caveat: The one-sentence arm (6 runs) was added later on a different machine from the control and treatment arms.
n=2 per task per arm; differences under about 1 point are noise.

Review

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a Mac test machine, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill. Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.

Three-arm comparison

Arm	Mean score (0-5)	Tokens per run
Without the skill	4.0	65k
One-sentence prompt	4.67	127k
With the skill	5.0	108k

Measured with Codex CLI over 18 runs, scored by rule-based checks with no LLM judge. Total 24.0 / 30, verdict Recommended.

Disclosure: The one-sentence arm (6 runs) was added later on a different machine from the control and treatment arms.

Verdict: Recommended (24.0 / 30). Source: obra/superpowers, skills/test-driven-development, pinned at commit 2ce8aa1. Install: npx skills@latest add obra/superpowers --skill test-driven-development. Licence: MIT (repo LICENSE).

What it is

The test-driven-development skill from obra/superpowers. Its description says "use when implementing any feature or bugfix, before writing implementation code". About 1,500 words of Markdown (the red-green-refactor 'iron law', a table of excuses for skipping the test, a verification checklist) plus one extra file on writing good tests. No scripts.

How we tested

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a Mac test machine, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill.

Implement a function from a precise spec — Implement slugify(text, max_length=None) in slug.py (Python 3, standard library only). Spec: lowercase the text; replace each run of characters that are not ASCII letters or digits with a single '-'; strip leading and trailing '-'; an empty result becomes 'n…
Implement a small stateful class — Write ratelimiter.py (Python 3, standard library only) with class RateLimiter(limit, window_seconds, clock=time.monotonic): a sliding-window limiter per key. allow(key) returns True and records the call if fewer than limit calls were recorded in the last…
Fix a rounding bug — Customers say cart_total in cart.py is sometimes a cent off compared with our accounting system, which rounds half-up to the cent at each step (each line total, the discounted subtotal, and the tax). For example cart_total([(1.005, 1)]) should return 1.09 an…

Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.

Results

The one-liner given to the third group: "Follow strict red-green-refactor: write a failing test first and watch it fail for the right reason, write the minimal implementation to pass it, then refactor; no production code without a failing test first."

Task	Control score	One-liner	With skill	Control tokens	One-liner tokens	With-skill tokens	Skill read
t1-slugify	4.0	5.0	5.0	62k	134k	108k	2/2
t2-ratelimiter	4.0	5.0	5.0	63k	147k	97k	2/2
t3-cart-bugfix	4.0	4.0	5.0	69k	97k	116k	2/2

Mean score: 4.0 without vs 5.0 with the skill (delta +1.0). Tokens per run: 65k vs 108k (1.65x); time 55s vs 51s. One-liner group: 4.67 (the skill's gain over the one-liner: +0.33; one-liner's own gain over no skill: +0.67); one-liner tokens 127k per run, so the skill costs 0.85x the one-liner.

What happened

Correctness was identical: the hidden test suites passed in all 12 runs of both groups (12/12, 10/10 and 11/11 tests). On these small tasks the skill does not change whether the code works.
What it does change is the order of work. In all 6 treatment runs the test file was written in an earlier step than the implementation; in all 6 control runs the agent wrote tests and implementation in the same patch (or fixed the bug first and added the test afterwards in the cart task). In the treatment run we read through (slugify), the agent wrote the test, ran it to see it fail, then wrote the code.
It triggers from the description alone: the skill file was read in 6/6 treatment runs although no prompt mentioned TDD.
It costs more: about 1.65x the tokens per run (107k vs 65k) at similar wall-clock time (51 s vs 55 s), because the agent reads the skill and its companion file and runs the loop several times.
One-sentence version (strict red-green-refactor, no production code without a failing test): mean 4.67, between control (4.0) and the skill (5.0), at 127k tokens per run against 65k for control (2.0x) and 108k for the skill (1.2x the skill).
Tests were written before the implementation in 0 of 6 control runs, 4 of 6 one-liner runs and 6 of 6 skill runs; the missing two one-liner runs are on the cart bug-fix task. The skill is the only group that did it every time, and it did so with fewer commands (10 per run) than the one-liner (19).

Dimension scores

Dimension	Score	Why
1_output_quality	5.0	Mean score 5.00/5 with the skill (programmatic checks, see per-task table).
2_lift_vs_no_skill	3.0	+1.00 points vs 4.00 without the skill (n=2 per task; under ~1 point is noise).
3_trigger	5.0	Skill file read in 6/6 treatment runs.
4_instructions_portability	4
5_efficiency	2	1.65x tokens (108k vs 65k per run) and 0.93x time (51s vs 55s).
6_safety_hygiene	5

Limits of this test

The hidden tests all pass for both groups (ceiling), so the measured lift is the process check (test written before implementation) which we compute from the event log, not an outcome difference.
Tasks are small single-file Python exercises; larger repos, other languages and test quality (for example mutation testing) were not tested.
n = 2 per group per task: differences under about 1 point are noise.
Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.

Security

See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.

Bottom line

Does what it says: it is the only group that wrote the failing test first in every run (6/6, against 0/6 for control and 4/6 for the one-sentence version), at 1.7x the control's tokens and 0.85x of the one-sentence version's. Final correctness on these small tasks was the same in all groups, so the gain is the process, not the code.

Transcript

Run log for test-driven-development: Codex CLI, network off, per-run temp HOME/TMPDIR, one row per graded run.
Columns: task, arm-run, score (0-5), tokens, seconds, checks passed, skill files the agent opened.

t1-slugify control-1: score 4.0, tokens 62017, secs 55.5, checks 2/3, opened nothing
t1-slugify control-2: score 4.0, tokens 62139, secs 53.8, checks 2/3, opened nothing
t1-slugify oneliner-1: score 5.0, tokens 135535, secs 52.9, checks 3/3, opened nothing
t1-slugify oneliner-2: score 5.0, tokens 134448, secs 46.7, checks 3/3, opened nothing
t1-slugify treatment-1: score 5.0, tokens 118034, secs 56.3, checks 3/3, opened test-driven-development
t1-slugify treatment-2: score 5.0, tokens 97974, secs 48.3, checks 3/3, opened test-driven-development
t2-ratelimiter control-1: score 4.0, tokens 63363, secs 46.1, checks 2/3, opened nothing
t2-ratelimiter control-2: score 4.0, tokens 63431, secs 98.9, checks 2/3, opened nothing
t2-ratelimiter oneliner-1: score 5.0, tokens 140042, secs 60.8, checks 3/3, opened nothing

18 runs, 3 artifacts

Sample output

SOTA · With the skill: test_slug.py (t1-slugify, run 1, Codex)

import unittest from slug import slugify class SlugifyTests(unittest.TestCase): def test_normalizes_text(self): cases = [ ('Hello, World!', 'hello-world'), (' --ABC__123\t\nxyz!! ', 'abc-123-xyz'), ('café déjà vu １２٣', 'caf-d-j-vu'), ('İ X', 'i-x'), ('already-a-slug', 'already-a-slug'), ] for text, expected in cases: with self.subTest(text=text): self.assertEqual(slugify(text), expected) def test_empty_result_uses_fallback(self): for text in ('', '--- !\t', '中文', '😀'): with self.subTest(text=text): self.assertEqual(slugify(text), 'n-a') def test_length_limits_respect_word_boundaries(self): cases = [ ('Hello, World!', 8, 'hello'), ('Hello, World!', 11, 'hello-world'), ('Hello, World!', 20, 'hello-world'), ('Hello, World!', 5, 'hello'), ('Hello, World!', 6, 'hello'), ('one two three', 7, 'one-two'), ('one two three', 9, 'one-two'), ('Supercalifragilistic', 5, 'super'), ('Supercalifragilistic next', 5, 'super'), ('abc', 1, 'a'), ('abc', 0, ''), ('!!!', 1, 'n'), ('!!!', 2, 'n'), ('!!!', 3, 'n-a'), ] for text, limit, expected in cases: with self.subTest(text=text, limit=limit): self.assertEqual(slugify(text, limit), expected) def test_explicit_none_leaves_length_unlimited(self): self.assertEqual(slugify('Hello, World!', None), 'hello-world') if __name__ == '__main__': unittest.main()

See Pro