test-driven-development (obra/superpowers)
test-driven-development：对照组 4.0、一句话提示 4.67、装 skill 5.0（0–5 分，Codex 实测，n=2/任务/组）。结论：推荐，相对不装 skill +1.00。
24.0 / 30
相对不装 skill 的提升：+1.00
推荐：总分 24.0/30 不低于 22，产出质量 5.0 不低于 3.5，提升 +1.00 不低于 +1.0。
前沿
通过使用 Codex 测试。
覆盖 Codex, Claude Code
Codex CLI 0.166（SOTA 档）：18 次评分运行。按我们的分档规则，这档通过同时覆盖 Claude Code；它不代表第二档的结果。
分数
每一维 0 到 5 分，六维相加是满分 30 的总分。每一维都是 5 更好。成本 5 表示又便宜又快。安装 5 表示容易。
- 触发可靠性
- 5.0
- 相对不装的提升
- 3.0
- 产出质量
- 5.0
- 令牌与时间成本
- 2.0
- 安装与可移植性
- 4.0
- 安全
- 5.0
安全审查
安全静态扫描：0 CRITICAL / 0 HIGH / 0 MEDIUM。人工审查：可在断网沙箱中运行。扫描没有命中不等于绝对安全。
安装
npx skills@latest add obra/superpowers --skill test-driven-development
固定提交: 2ce8aa15d05bd57213e6f746f953aafb7ded7a8b
测试日期: 2026-11-02
我们测了什么
- Implement a function from a precise spec — Implement slugify(text, max_length=None) in slug.py (Python 3, standard library only). Spec: lowercase the text; replace each run of characters that are not ASCII letters or digits with a single '-'; strip leading and trailing '-'; an empty result becomes 'n…
- Implement a small stateful class — Write ratelimiter.py (Python 3, standard library only) with class RateLimiter(limit, window_seconds, clock=time.monotonic): a sliding-window limiter per key. allow(key) returns True and records the call if fewer than limit calls were recorded in the last…
- Fix a rounding bug — Customers say cart_total in cart.py is sometimes a cent off compared with our accounting system, which rounds half-up to the cent at each step (each line total, the discounted subtotal, and the tax). For example cart_total([(1.005, 1)]) should return 1.09 an…
优点
- 平均分从不装时的 4.0 提到 5.0（+1.00）。
- 不点名也会被触发：Skill file read in 6/6 treatment runs.
- 比把它的规则压成一句话的做法高 +0.33。
- 装 skill 后绝对质量高：5.0 / 5。
缺点
- token 消耗是不装 skill 时的 1.65 倍。
- 测试说明：一句话提示组（6 次）是后来在另一台机器上补跑的，和对照组、安装组不是同一台。
- 每个任务每组 n=2，差距在 1 分以内属于噪声。
评测
在 Codex CLI（SOTA 档）上跑 3 个任务、对照、一句话提示、装 skill 三组，共 18 次运行，由规则脚本打分（没有 LLM 评委），断网沙箱、每次运行独立临时目录。具体做法见下方英文评测原文。
三组对比
| 组别 | 平均分（0–5） | 每次 token |
|---|---|---|
| 不装 skill | 4.0 | 65k |
| 一句话提示 | 4.67 | 127k |
| 装了 skill | 5.0 | 108k |
Codex CLI 实测，18 次运行，规则脚本打分，没有 LLM 评委。总分 24.0 / 30，结论 推荐。
测试披露： 一句话提示组（6 次）是后来在另一台机器上补跑的，和对照组、安装组不是同一台。
下面是评测原文，英文。
Verdict: Recommended (24.0 / 30). Source: obra/superpowers, skills/test-driven-development, pinned at commit 2ce8aa1. Install: npx skills@latest add obra/superpowers --skill test-driven-development. Licence: MIT (repo LICENSE).
What it is
The test-driven-development skill from obra/superpowers. Its description says "use when implementing any feature or bugfix, before writing implementation code". About 1,500 words of Markdown (the red-green-refactor 'iron law', a table of excuses for skipping the test, a verification checklist) plus one extra file on writing good tests. No scripts.
How we tested
Codex CLI 0.166 (gpt-6-astra, low reasoning) on a Mac test machine, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill.
- Implement a function from a precise spec — Implement
slugify(text, max_length=None)in slug.py (Python 3, standard library only). Spec: lowercase the text; replace each run of characters that are not ASCII letters or digits with a single '-'; strip leading and trailing '-'; an empty result becomes 'n… - Implement a small stateful class — Write ratelimiter.py (Python 3, standard library only) with class
RateLimiter(limit, window_seconds, clock=time.monotonic): a sliding-window limiter per key.allow(key)returns True and records the call if fewer thanlimitcalls were recorded in the last… - Fix a rounding bug — Customers say
cart_totalin cart.py is sometimes a cent off compared with our accounting system, which rounds half-up to the cent at each step (each line total, the discounted subtotal, and the tax). For example cart_total([(1.005, 1)]) should return 1.09 an…
Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.
Results
The one-liner given to the third group: "Follow strict red-green-refactor: write a failing test first and watch it fail for the right reason, write the minimal implementation to pass it, then refactor; no production code without a failing test first."
| Task | Control score | One-liner | With skill | Control tokens | One-liner tokens | With-skill tokens | Skill read |
|---|---|---|---|---|---|---|---|
| t1-slugify | 4.0 | 5.0 | 5.0 | 62k | 134k | 108k | 2/2 |
| t2-ratelimiter | 4.0 | 5.0 | 5.0 | 63k | 147k | 97k | 2/2 |
| t3-cart-bugfix | 4.0 | 4.0 | 5.0 | 69k | 97k | 116k | 2/2 |
Mean score: 4.0 without vs 5.0 with the skill (delta +1.0). Tokens per run: 65k vs 108k (1.65x); time 55s vs 51s. One-liner group: 4.67 (the skill's gain over the one-liner: +0.33; one-liner's own gain over no skill: +0.67); one-liner tokens 127k per run, so the skill costs 0.85x the one-liner.
What happened
- Correctness was identical: the hidden test suites passed in all 12 runs of both groups (12/12, 10/10 and 11/11 tests). On these small tasks the skill does not change whether the code works.
- What it does change is the order of work. In all 6 treatment runs the test file was written in an earlier step than the implementation; in all 6 control runs the agent wrote tests and implementation in the same patch (or fixed the bug first and added the test afterwards in the cart task). In the treatment run we read through (slugify), the agent wrote the test, ran it to see it fail, then wrote the code.
- It triggers from the description alone: the skill file was read in 6/6 treatment runs although no prompt mentioned TDD.
- It costs more: about 1.65x the tokens per run (107k vs 65k) at similar wall-clock time (51 s vs 55 s), because the agent reads the skill and its companion file and runs the loop several times.
- One-sentence version (strict red-green-refactor, no production code without a failing test): mean 4.67, between control (4.0) and the skill (5.0), at 127k tokens per run against 65k for control (2.0x) and 108k for the skill (1.2x the skill).
- Tests were written before the implementation in 0 of 6 control runs, 4 of 6 one-liner runs and 6 of 6 skill runs; the missing two one-liner runs are on the cart bug-fix task. The skill is the only group that did it every time, and it did so with fewer commands (10 per run) than the one-liner (19).
Dimension scores
| Dimension | Score | Why |
|---|---|---|
| 1_output_quality | 5.0 | Mean score 5.00/5 with the skill (programmatic checks, see per-task table). |
| 2_lift_vs_no_skill | 3.0 | +1.00 points vs 4.00 without the skill (n=2 per task; under ~1 point is noise). |
| 3_trigger | 5.0 | Skill file read in 6/6 treatment runs. |
| 4_instructions_portability | 4 | |
| 5_efficiency | 2 | 1.65x tokens (108k vs 65k per run) and 0.93x time (51s vs 55s). |
| 6_safety_hygiene | 5 |
Limits of this test
- The hidden tests all pass for both groups (ceiling), so the measured lift is the process check (test written before implementation) which we compute from the event log, not an outcome difference.
- Tasks are small single-file Python exercises; larger repos, other languages and test quality (for example mutation testing) were not tested.
- n = 2 per group per task: differences under about 1 point are noise.
- Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.
Security
See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.
Bottom line
Does what it says: it is the only group that wrote the failing test first in every run (6/6, against 0/6 for control and 4/6 for the one-sentence version), at 1.7x the control's tokens and 0.85x of the one-sentence version's. Final correctness on these small tasks was the same in all groups, so the gain is the process, not the code.
记录
Run log for test-driven-development: Codex CLI, network off, per-run temp HOME/TMPDIR, one row per graded run. Columns: task, arm-run, score (0-5), tokens, seconds, checks passed, skill files the agent opened. t1-slugify control-1: score 4.0, tokens 62017, secs 55.5, checks 2/3, opened nothing t1-slugify control-2: score 4.0, tokens 62139, secs 53.8, checks 2/3, opened nothing t1-slugify oneliner-1: score 5.0, tokens 135535, secs 52.9, checks 3/3, opened nothing t1-slugify oneliner-2: score 5.0, tokens 134448, secs 46.7, checks 3/3, opened nothing t1-slugify treatment-1: score 5.0, tokens 118034, secs 56.3, checks 3/3, opened test-driven-development t1-slugify treatment-2: score 5.0, tokens 97974, secs 48.3, checks 3/3, opened test-driven-development t2-ratelimiter control-1: score 4.0, tokens 63363, secs 46.1, checks 2/3, opened nothing t2-ratelimiter control-2: score 4.0, tokens 63431, secs 98.9, checks 2/3, opened nothing t2-ratelimiter oneliner-1: score 5.0, tokens 140042, secs 60.8, checks 3/3, opened nothing
18 次运行、3 个产物
样例产物
前沿 · 装了 skill：test_slug.py（t1-slugify，第 1 次，Codex）
import unittest from slug import slugify class SlugifyTests(unittest.TestCase): def test_normalizes_text(self): cases = [ ('Hello, World!', 'hello-world'), (' --ABC__123\t\nxyz!! ', 'abc-123-xyz'), ('café déjà vu １２٣', 'caf-d-j-vu'), ('İ X', 'i-x'), ('already-a-slug', 'already-a-slug'), ] for text, expected in cases: with self.subTest(text=text): self.assertEqual(slugify(text), expected) def test_empty_result_uses_fallback(self): for text in ('', '--- !\t', '中文', '😀'): with self.subTest(text=text): self.assertEqual(slugify(text), 'n-a') def test_length_limits_respect_word_boundaries(self): cases = [ ('Hello, World!', 8, 'hello'), ('Hello, World!', 11, 'hello-world'), ('Hello, World!', 20, 'hello-world'), ('Hello, World!', 5, 'hello'), ('Hello, World!', 6, 'hello'), ('one two three', 7, 'one-two'), ('one two three', 9, 'one-two'), ('Supercalifragilistic', 5, 'super'), ('Supercalifragilistic next', 5, 'super'), ('abc', 1, 'a'), ('abc', 0, ''), ('!!!', 1, 'n'), ('!!!', 2, 'n'), ('!!!', 3, 'n-a'), ] for text, limit, expected in cases: with self.subTest(text=text, limit=limit): self.assertEqual(slugify(text, limit), expected) def test_explicit_none_leaves_length_unlimited(self): self.assertEqual(slugify('Hello, World!', None), 'hello-world') if __name__ == '__main__': unittest.main()