writing-plans (obra/superpowers)

评测

在 Codex CLI（SOTA 档）上跑 3 个任务、对照、一句话提示、装 skill 三组，共 18 次运行，由规则脚本打分（没有 LLM 评委），断网沙箱、每次运行独立临时目录。具体做法见下方英文评测原文。

三组对比

Codex CLI 实测，18 次运行，规则脚本打分，没有 LLM 评委。总分 25.3 / 30，结论 推荐。

下面是评测原文，英文。

Verdict: Recommended (25.3 / 30). Source: obra/superpowers, skills/writing-plans, pinned at commit 2ce8aa1. Install: npx skills@latest add obra/superpowers --skill writing-plans. Licence: MIT (repo LICENSE).

What it is

The writing-plans skill (obra/superpowers): turn a spec into a bite-sized implementation plan: exact file paths, a failing test first, the exact run command with expected result, minimal implementation, a commit step per task, checkbox steps, no TBD placeholders.

How we tested

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).

Three small Python projects (todo CLI, report module, notes API), each with a spec.md. The prompt only asks for an implementation plan in plan.md and forbids writing code. It does not describe the plan format. The one-liner describes the whole format: bite-sized 2-5 minute steps, exact paths, failing test first with complete code, exact command with expected output, minimal implementation, git commit, no placeholders.

1. Plan due dates for a CLI — spec.md describes a feature for this small todo CLI. Write the implementation plan for it to plan.md in this folder. Do not write any code in the project, the plan only.
2. Plan a CSV export — spec.md describes a feature for this report module. Write the implementation plan for it to plan.md in this folder. Do not write any code in the project, the plan only.
3. Plan API rate limiting — spec.md describes a feature for this notes API. Write the implementation plan for it to plan.md in this folder. Do not write any code in the project, the plan only.

Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.

Results

The one-liner given to the third group: "Write the plan as bite-sized tasks (each step 2-5 minutes): exact file paths, the failing test first with complete code, the exact command with its expected output, the minimal implementation with complete code, then a git commit step; no placeholders like TBD or 'add validation'."

Mean score: 1.56 without vs 4.43 with the skill (delta +2.87). Tokens per run: 88k vs 119k (1.34x); time 66s vs 102s. One-liner group: 4.21 (the skill's gain over the one-liner: +0.22; one-liner's own gain over no skill: +2.65); one-liner tokens 127k per run, so the skill costs 0.93x the one-liner.

What happened

• Control 1.56, skill 4.43, one-liner 4.21 (n=6 each). Control's plans are short prose outlines (46-71 lines): no failing-test-first steps, no expected output and no commit steps in any of 6 runs, and one run-command each.
• Skill plans are 110-142 lines in a checkbox format with a failing-test step, a run step marked 'Expect failure' or 'Expect all tests to pass', and a task-level commit in the CSV task. They still have gaps: 4 of 6 runs have no git commit step, and code blocks appear in 2-3 steps rather than every step.
• One-liner plans are about 350 lines with complete code in each step and all required commits and commands, but no checkboxes in 6/6 (a skill-specific format) and in 3 of 6 runs fewer than 4 run commands, since they used PowerShell wording.
• Tokens: skill 119k, one-liner 127k, control 88k per run (1.35x control). The skill was read in 6/6 treatment runs.

Dimension scores

Limits of this test

• Rule-based checks on the text of plan.md: counts of paths, commands, commits, checkboxes and code blocks. Plan quality, whether the plan would really work, was not judged.
• I changed the checks after seeing the first scoring and re-scored every run under the same new rules: (1) the 'no placeholders' check matched the word TODO, which is the product name in the todo-CLI task (removed TODO from the pattern); (2) 'failing test first' and 'expected output' patterns were widened to accept 'Expect failure' and 'Expect all tests to pass'; (3) the commit threshold dropped from 3 to 2 (the skill plans have two tasks) and the checkbox threshold from 12 to 8. Before this change the skill scored 2.98 and the one-liner 4.04; after it, 4.43 and 4.21. Control stayed at 1.4-1.6 either way.
• The checkbox check counts a format the skill uses and the one-liner never asked for, which favours the skill.
• n=2 per cell; small fixtures.
• n = 2 per group per task: differences under about 1 point are noise.
• Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.

Security

See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.

Bottom line

Without help Codex writes a loose outline; with the skill it writes real TDD steps. A paragraph naming the same format does nearly as well and produces fuller code per step, so the skill is mainly a convenient bundle of that paragraph.