tdd (Matt Pocock)
tdd：对照组 4.67、一句话提示 4.67、装 skill 4.67（0–5 分，Codex 实测，n=2/任务/组）。结论：视情况而定，相对不装 skill +0.00。
22.7 / 30
相对不装 skill 的提升：+0.00
视情况而定：提升 +0.00 低于 +1.0。推荐要求总分 22/30、产出质量 3.5、提升 +1.0。
前沿
通过使用 Codex 测试。
覆盖 Codex, Claude Code
Codex CLI 0.166（SOTA 档）：18 次评分运行。按我们的分档规则，这档通过同时覆盖 Claude Code；它不代表第二档的结果。
分数
每一维 0 到 5 分，六维相加是满分 30 的总分。每一维都是 5 更好。成本 5 表示又便宜又快。安装 5 表示容易。
- 触发可靠性
- 5.0
- 相对不装的提升
- 2.0
- 产出质量
- 4.7
- 令牌与时间成本
- 2.0
- 安装与可移植性
- 4.0
- 安全
- 5.0
安全审查
安全静态扫描：0 CRITICAL / 0 HIGH / 0 MEDIUM。人工审查：可在断网沙箱中运行。扫描没有命中不等于绝对安全。
安装
npx skills@latest add mattpocock/skills --skill tdd
固定提交: 4ae2317ddb22b046e26455e4f9c9b3aaaf62212d
测试日期: 2026-11-02
我们测了什么
- Implement a function from a precise spec — Implement slugify(text, max_length=None) in slug.py (Python 3, standard library only) using TDD. Spec: lowercase the text; replace each run of characters that are not ASCII letters or digits with a single '-'; strip leading and trailing '-'; an empty result…
- Implement a small stateful class — Write ratelimiter.py (Python 3, standard library only) test-first (red-green-refactor) with class RateLimiter(limit, window_seconds, clock=time.monotonic): a sliding-window limiter per key. allow(key) returns True and records the call if fewer than limit…
- Fix a rounding bug — Customers say cart_total in cart.py is sometimes a cent off compared with our accounting system, which rounds half-up to the cent at each step (each line total, the discounted subtotal, and the tax). For example cart_total([(1.005, 1)]) should return 1.09 an…
优点
- 不点名也会被触发：Skill file read in 6/6 treatment runs.
- 装 skill 后绝对质量高：4.67 / 5。
缺点
- 相对不装 skill 没有可测量的提升（+0.00，在噪声范围内）。
- 一句话提示得 4.67，装 skill 得 4.67（差 +0.00）。
- token 消耗是不装 skill 时的 1.7 倍。
- 测试说明：一句话提示组（6 次）是后来在另一台机器上补跑的，和对照组、安装组不是同一台。
- 每个任务每组 n=2，差距在 1 分以内属于噪声。
评测
在 Codex CLI（SOTA 档）上跑 3 个任务、对照、一句话提示、装 skill 三组，共 18 次运行，由规则脚本打分（没有 LLM 评委），断网沙箱、每次运行独立临时目录。具体做法见下方英文评测原文。
三组对比
| 组别 | 平均分（0–5） | 每次 token |
|---|---|---|
| 不装 skill | 4.67 | 86k |
| 一句话提示 | 4.67 | 178k |
| 装了 skill | 4.67 | 146k |
Codex CLI 实测，18 次运行，规则脚本打分，没有 LLM 评委。总分 22.7 / 30，结论 视情况而定。
测试披露： 一句话提示组（6 次）是后来在另一台机器上补跑的，和对照组、安装组不是同一台。
下面是评测原文，英文。
Verdict: Situational (22.7 / 30). Source: mattpocock/skills, skills/engineering/tdd, pinned at commit 4ae2317. Install: npx skills@latest add mattpocock/skills --skill tdd. Licence: MIT (repo LICENSE).
What it is
The tdd skill from Matt Pocock's skills repo (engineering/tdd): the red-green loop plus two reference files on what a good test is and when to mock. About 570 words in SKILL.md; pure Markdown, no scripts. It triggers when the user wants test-first work.
How we tested
Codex CLI 0.166 (gpt-6-astra, low reasoning) on a Mac test machine, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).
Because this skill's description only triggers when the user asks for TDD / test-first, the prompts say so ("using TDD", "test-first (red-green-refactor)", "Do it test-first"). The same words are given to the control group, so the test is whether the skill improves on a plain request for TDD.
- Implement a function from a precise spec — Implement
slugify(text, max_length=None)in slug.py (Python 3, standard library only) using TDD. Spec: lowercase the text; replace each run of characters that are not ASCII letters or digits with a single '-'; strip leading and trailing '-'; an empty result… - Implement a small stateful class — Write ratelimiter.py (Python 3, standard library only) test-first (red-green-refactor) with class
RateLimiter(limit, window_seconds, clock=time.monotonic): a sliding-window limiter per key.allow(key)returns True and records the call if fewer thanlimit… - Fix a rounding bug — Customers say
cart_totalin cart.py is sometimes a cent off compared with our accounting system, which rounds half-up to the cent at each step (each line total, the discounted subtotal, and the tax). For example cart_total([(1.005, 1)]) should return 1.09 an…
Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.
Results
The one-liner given to the third group: "Work test-first in vertical slices: write one failing test through the public interface, confirm it fails, write the minimum code to pass it, repeat, and refactor only when green; do not write all the tests up front."
| Task | Control score | One-liner | With skill | Control tokens | One-liner tokens | With-skill tokens | Skill read |
|---|---|---|---|---|---|---|---|
| t1-slugify | 5.0 | 5.0 | 5.0 | 79k | 184k | 135k | 2/2 |
| t2-ratelimiter | 5.0 | 5.0 | 5.0 | 99k | 183k | 169k | 2/2 |
| t3-cart-bugfix | 4.0 | 4.0 | 4.0 | 79k | 164k | 133k | 2/2 |
Mean score: 4.67 without vs 4.67 with the skill (delta +0.0). Tokens per run: 86k vs 146k (1.70x); time 50s vs 67s. One-liner group: 4.67 (the skill's gain over the one-liner: +0.0; one-liner's own gain over no skill: +0.0); one-liner tokens 178k per run, so the skill costs 0.82x the one-liner.
What happened
- With the prompt already asking for test-first, the control group did it too: on the two build tasks every run of both groups passed all checks, including 'test written before implementation'. The skill added nothing measurable there.
- On the bug-fix task (cart rounding) all four runs, with and without the skill, fixed the bug and passed the hidden tests, and all four failed the same process check: the fix or the test arrived in the same step as the other, rather than a failing test first. The skill did not change that.
- The skill was read in 6/6 treatment runs. It made the agent do far more work (24-26 commands vs 8-10 on the build tasks) for the same result: about 1.7x the tokens.
- Hidden-test correctness was 100% in both groups.
- One-sentence version (work test-first in vertical slices, one failing test at a time): mean score 4.67, identical to control and skill, at 178k tokens per run against 86k for control (2.1x) and 146k for the skill. The one-liner agents ran 27 commands per run on average against 9 for control and 23 for the skill, because they ran the test loop repeatedly.
- Every group wrote its own tests in all runs and passed all hidden tests; tests-before-implementation was met in 4 of 6 runs in all three groups, so the prompt-level request 'test first' already did what the skill is for.
Dimension scores
| Dimension | Score | Why |
|---|---|---|
| 1_output_quality | 4.67 | Mean score 4.67/5 with the skill (programmatic checks, see per-task table). |
| 2_lift_vs_no_skill | 2.0 | +0.00 points vs 4.67 without the skill (n=2 per task; under ~1 point is noise). |
| 3_trigger | 5.0 | Skill file read in 6/6 treatment runs. |
| 4_instructions_portability | 4 | |
| 5_efficiency | 2 | 1.70x tokens (146k vs 86k per run) and 1.34x time (67s vs 50s). |
| 6_safety_hygiene | 5 |
Limits of this test
- Each test prompt itself asked for TDD, which gives the control group most of the benefit; a user who does not ask would see the skill's description not trigger at all.
- Small Python tasks only; test quality was not scored.
- n = 2 per group per task: differences under about 1 point are noise.
- Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.
Security
See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.
Bottom line
On top of a test-first request this skill produces no extra gain on small tasks: control, the one-sentence version and the skill all scored 4.67, while the skill used 1.7x and the sentence 2.1x the tokens of control. It may help keep tests behaviour-focused over a longer session, which we did not measure.
记录
Run log for tdd: Codex CLI, network off, per-run temp HOME/TMPDIR, one row per graded run. Columns: task, arm-run, score (0-5), tokens, seconds, checks passed, skill files the agent opened. t1-slugify control-1: score 5.0, tokens 78904, secs 50.3, checks 3/3, opened nothing t1-slugify control-2: score 5.0, tokens 79205, secs 47.1, checks 3/3, opened nothing t1-slugify oneliner-1: score 5.0, tokens 199600, secs 74.7, checks 3/3, opened nothing t1-slugify oneliner-2: score 5.0, tokens 168650, secs 68.2, checks 3/3, opened nothing t1-slugify treatment-1: score 5.0, tokens 144538, secs 57.6, checks 3/3, opened tdd t1-slugify treatment-2: score 5.0, tokens 126047, secs 50.7, checks 3/3, opened tdd t2-ratelimiter control-1: score 5.0, tokens 99439, secs 60.2, checks 3/3, opened nothing t2-ratelimiter control-2: score 5.0, tokens 99611, secs 54.3, checks 3/3, opened nothing t2-ratelimiter oneliner-1: score 5.0, tokens 159969, secs 70.9, checks 3/3, opened nothing
18 次运行、3 个产物
样例产物
前沿 · 装了 skill：test_slug.py（t1-slugify，第 1 次，Codex）
import unittest from slug import slugify class SlugifyTests(unittest.TestCase): def test_hard_cuts_first_word_when_it_exceeds_limit(self): self.assertEqual(slugify('Supercalifragilistic', 5), 'super') def test_truncates_at_word_boundary(self): self.assertEqual(slugify('Hello, World!', 8), 'hello') def test_normalizes_text(self): self.assertEqual(slugify('Hello, World!'), 'hello-world') def test_preserves_only_ascii_letters_and_digits(self): self.assertEqual(slugify(' -- CAFÉ_42\t中文 / Test!! '), 'caf-42-test') def test_lowercases_before_filtering(self): self.assertEqual(slugify('\u212aELVIN'), 'kelvin') def test_keeps_slug_that_fits_limit(self): for limit in (11, 20, None): with self.subTest(limit=limit): self.assertEqual(slugify('Hello, World!', limit), 'hello-world') def test_keeps_complete_words_without_trailing_separator(self): for limit, expected in ((3, 'one'), (4, 'one'), (6, 'one'), (7, 'one-two'), (8, 'one-two')): with self.subTest(limit=limit): self.assertEqual(slugify('one two three', limit), expected) def test_hard_cuts_long_first_word_even_with_later_words(self): self.assertEqual(slugify('Supercalifragilistic hello', 5), 'super') def test_applies_limit_to_fallback(self): self.assertEqual(slugify('', 2), 'n') def test_empty_normalized_text_has_fallback(self): for text in ('', ' !!! ', '中文'): with self.subTest(text=text): self.assertEqual(slugify(text), 'n-a') if __name__ == '__main__': unittest.main()