writing-plans (obra/superpowers)

Name: writing-plans (obra/superpowers) review
Item: writing-plans (obra/superpowers)
Rating: 25.3
Author: Agent Skills

writing-plans: turns a short feature spec into a step-by-step TDD plan (control 1.6, skill 4.4, one-liner 4.2) for 1.35x the tokens; a one-sentence prompt of the same format scores almost as well

RecommendedSafe

25.3 / 30

Lift over no skill: +2.87

Recommended: the total 25.3/30 is at least 22, output quality 4.4 is at least 3.5, and the lift +2.87 is at least +1.0.

SOTA

Pass

Tested with Codex.

Covers Codex, Claude Code

Codex CLI 0.166 (SOTA tier): 18 graded runs. A pass here also covers Claude Code under our tier rule; it says nothing about the second tier.

Scores

Each dimension is 0 to 5 and the six add up to the total out of 30. 5 is better on every dimension. Cost 5 means cheap and fast. Install 5 means easy.

Trigger reliability: 5.0
Lift over no skill: 4.9
Output quality: 4.4
Token and time cost: 3.0
Install and portability: 4.0
Safety: 4.0

Security review

Safe

Static scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Manual review: Safe to run in a no-network sandbox (no findings). A clean scan does not prove a skill safe.

Install

npx skills@latest add obra/superpowers --skill writing-plans

Source

Pinned commit: 2ce8aa15d05bd57213e6f746f953aafb7ded7a8b

Tested: 2026-11-02

What we tested

Plan due dates for a CLI — spec.md describes a feature for this small todo CLI. Write the implementation plan for it to plan.md in this folder. Do not write any code in the project, the plan only.
Plan a CSV export — spec.md describes a feature for this report module. Write the implementation plan for it to plan.md in this folder. Do not write any code in the project, the plan only.
Plan API rate limiting — spec.md describes a feature for this notes API. Write the implementation plan for it to plan.md in this folder. Do not write any code in the project, the plan only.

Pros

Lifts the mean score from 1.56 to 4.43 over no skill (+2.87).
The skill was picked up without being named: Skill file read in 6/6 treatment runs.

Cons

Test caveat: We changed three checks after seeing the first scores and re-scored all 18 runs under the new rules: the placeholder check no longer matches the word TODO, which is the product name in one task; the failing-test-first and expected-output patterns accept 'Expect failure' wording; the commit and checkbox thresholds were lowered. Before the change the skill scored 2.98 and the one-sentence prompt 4.04; after it, 4.43 and 4.21. Control stayed near 1.5 either way.
n=2 per task per arm; differences under about 1 point are noise.

Review

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below). Three small Python projects (todo CLI, report module, notes API), each with a spec.md. The prompt only asks for an implementation plan in plan.md and forbids writing code. It does not describe the plan format. The one-liner describes the whole format: bite-sized 2-5 minute steps, exact paths, failing test first with complete code, exact command with expected output, minimal implementation, git commit, no placeholders.

Three-arm comparison

Arm	Mean score (0-5)	Tokens per run
Without the skill	1.56	88k
One-sentence prompt	4.21	127k
With the skill	4.43	119k

Measured with Codex CLI over 18 runs, scored by rule-based checks with no LLM judge. Total 25.3 / 30, verdict Recommended.

Disclosure: We changed three checks after seeing the first scores and re-scored all 18 runs under the new rules: the placeholder check no longer matches the word TODO, which is the product name in one task; the failing-test-first and expected-output patterns accept 'Expect failure' wording; the commit and checkbox thresholds were lowered. Before the change the skill scored 2.98 and the one-sentence prompt 4.04; after it, 4.43 and 4.21. Control stayed near 1.5 either way.

Verdict: Recommended (25.3 / 30). Source: obra/superpowers, skills/writing-plans, pinned at commit 2ce8aa1. Install: npx skills@latest add obra/superpowers --skill writing-plans. Licence: MIT (repo LICENSE).

What it is

The writing-plans skill (obra/superpowers): turn a spec into a bite-sized implementation plan: exact file paths, a failing test first, the exact run command with expected result, minimal implementation, a commit step per task, checkbox steps, no TBD placeholders.

How we tested

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).

Three small Python projects (todo CLI, report module, notes API), each with a spec.md. The prompt only asks for an implementation plan in plan.md and forbids writing code. It does not describe the plan format. The one-liner describes the whole format: bite-sized 2-5 minute steps, exact paths, failing test first with complete code, exact command with expected output, minimal implementation, git commit, no placeholders.

Plan due dates for a CLI — spec.md describes a feature for this small todo CLI. Write the implementation plan for it to plan.md in this folder. Do not write any code in the project, the plan only.
Plan a CSV export — spec.md describes a feature for this report module. Write the implementation plan for it to plan.md in this folder. Do not write any code in the project, the plan only.
Plan API rate limiting — spec.md describes a feature for this notes API. Write the implementation plan for it to plan.md in this folder. Do not write any code in the project, the plan only.

Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.

Results

The one-liner given to the third group: "Write the plan as bite-sized tasks (each step 2-5 minutes): exact file paths, the failing test first with complete code, the exact command with its expected output, the minimal implementation with complete code, then a git commit step; no placeholders like TBD or 'add validation'."

Task	Control score	One-liner	With skill	Control tokens	One-liner tokens	With-skill tokens	Skill read
t1-todo-due	1.51	4.21	4.28	94k	132k	111k	2/2
t2-csv-export	1.71	4.47	4.8	84k	129k	122k	2/2
t3-rate-limit	1.45	3.95	4.21	85k	120k	122k	2/2

Mean score: 1.56 without vs 4.43 with the skill (delta +2.87). Tokens per run: 88k vs 119k (1.34x); time 66s vs 102s. One-liner group: 4.21 (the skill's gain over the one-liner: +0.22; one-liner's own gain over no skill: +2.65); one-liner tokens 127k per run, so the skill costs 0.93x the one-liner.

What happened

Control 1.56, skill 4.43, one-liner 4.21 (n=6 each). Control's plans are short prose outlines (46-71 lines): no failing-test-first steps, no expected output and no commit steps in any of 6 runs, and one run-command each.
Skill plans are 110-142 lines in a checkbox format with a failing-test step, a run step marked 'Expect failure' or 'Expect all tests to pass', and a task-level commit in the CSV task. They still have gaps: 4 of 6 runs have no git commit step, and code blocks appear in 2-3 steps rather than every step.
One-liner plans are about 350 lines with complete code in each step and all required commits and commands, but no checkboxes in 6/6 (a skill-specific format) and in 3 of 6 runs fewer than 4 run commands, since they used PowerShell wording.
Tokens: skill 119k, one-liner 127k, control 88k per run (1.35x control). The skill was read in 6/6 treatment runs.

Dimension scores

Dimension	Score	Why
1_output_quality	4.43	Mean score 4.43/5 with the skill (programmatic checks, see per-task table).
2_lift_vs_no_skill	4.87	+2.87 points vs 1.56 without the skill (n=2 per task; under ~1 point is noise).
3_trigger	5.0	Skill file read in 6/6 treatment runs.
4_instructions_portability	4	SKILL.md in the superpowers format; refers to other superpowers skills for execution.
5_efficiency	3	1.34x tokens (119k vs 88k per run) and 1.54x time (102s vs 66s).
6_safety_hygiene	4	Prose only; see security.md.

Limits of this test

Rule-based checks on the text of plan.md: counts of paths, commands, commits, checkboxes and code blocks. Plan quality, whether the plan would really work, was not judged.
I changed the checks after seeing the first scoring and re-scored every run under the same new rules: (1) the 'no placeholders' check matched the word TODO, which is the product name in the todo-CLI task (removed TODO from the pattern); (2) 'failing test first' and 'expected output' patterns were widened to accept 'Expect failure' and 'Expect all tests to pass'; (3) the commit threshold dropped from 3 to 2 (the skill plans have two tasks) and the checkbox threshold from 12 to 8. Before this change the skill scored 2.98 and the one-liner 4.04; after it, 4.43 and 4.21. Control stayed at 1.4-1.6 either way.
The checkbox check counts a format the skill uses and the one-liner never asked for, which favours the skill.
n=2 per cell; small fixtures.
n = 2 per group per task: differences under about 1 point are noise.
Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.

Security

See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.

Bottom line

Without help Codex writes a loose outline; with the skill it writes real TDD steps. A paragraph naming the same format does nearly as well and produces fuller code per step, so the skill is mainly a convenient bundle of that paragraph.

Transcript

Run log for writing-plans: Codex CLI, network off, per-run temp HOME/TMPDIR, one row per graded run.
Columns: task, arm-run, score (0-5), tokens, seconds, checks passed, skill files the agent opened.

t1-todo-due control-1: score 1.38, tokens 102848, secs 60.8, checks 3/10, opened nothing
t1-todo-due control-2: score 1.64, tokens 86724, secs 67.6, checks 3/10, opened nothing
t1-todo-due oneliner-1: score 4.47, tokens 133771, secs 130.1, checks 9/10, opened nothing
t1-todo-due oneliner-2: score 3.95, tokens 131546, secs 112.8, checks 8/10, opened nothing
t1-todo-due treatment-1: score 4.34, tokens 123893, secs 104.1, checks 8/10, opened writing-plans
t1-todo-due treatment-2: score 4.21, tokens 99712, secs 99.6, checks 8/10, opened writing-plans
t2-csv-export control-1: score 1.71, tokens 85021, secs 70.6, checks 4/10, opened nothing
t2-csv-export control-2: score 1.71, tokens 84700, secs 64.8, checks 4/10, opened nothing
t2-csv-export oneliner-1: score 4.47, tokens 129967, secs 119.6, checks 9/10, opened nothing

18 runs, 3 artifacts

Sample output

SOTA · With the skill: plan.md (t1-todo-due, run 1, Codex)

# Todo Due Dates Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Add optional due dates, chronological listing, and an overdue-only CI check without migrating existing data. **Architecture:** Keep the existing JSON dictionary storage and implement date validation, ordering, and rendering in `todo/cli.py`. Capture the local date once per list invocation and pass it to a pure overdue predicate. Preserve the existing `main(argv=None)` entry point and argparse error handling. **Tech Stack:** Python standard library: argparse, datetime, json, unittest, unittest.mock, tempfile; no new dependencies. **Spec:** `spec.md` (read alongside this plan). ## Global Constraints - `todo add "title" --due 2026-03-01` stores an optional ISO due date with the item. - `todo list` prints overdue items first (marked `!`), then by due date, then undated items. - `todo list --overdue` prints only overdue items; exit code 1 if there are any (so CI can use it). - Invalid dates are rejected with a clear message and exit code 2. - Items already stored without a due date must keep working (no migration step for the user). ## Decisions for details not specified - Accept exactly `YYYY-MM-DD` with a real Gregorian calendar date; reject whitespace, compact dates, timestamps, and unpadded components. - An item is overdue only when unfinished and its due date is strictly before the current local date. Due today is not overdue. - Sort overdue items by ascending due date, then remaining dated items by ascending due date, then undated items. Preserve insertion order within equal keys. Completed items remain visible in ordinary listings but are never overdue. - Preserve `[ ] title` and `[x] title` for undated items. Append ` (due YYYY-MM-DD)` for dated items and prefix overdue lines with `! `, e.g. `! [ ] rent (due 2026-03-01)`. - Ordinary listing returns 0, even with overdue items. Overdue-only listing returns 0 with no output when none qualify. Listing never rewrites storage. - Store dates as strings under `due`; omit the key when not supplied. Treat an absent key or `null` as undated. Arbitrary corrupt JSON or malformed dates manually inserted into storage are outside this feature. ## Review Focus 1. Calendar boundaries: leap day is accepted only in leap years; impossible dates fail before storage changes (Task 1). 2. ISO-looking variants: compact dates, timestamps, whitespace, and unpadded dates are rejected (Task 1). 3. Midnight/today boundary: due today is not overdue; use one captured local date per listing (Task 2). 4. Completed items with past dates: remain visible without `!` and are excluded from overdue filtering (Tasks 2–3). 5. Legacy and tied items: missing/null dates work, ties retain insertion order, and listing leaves the JSON file untouched (Tasks 2–3). ## File Map - Modify `todo/cli.py`: add argument validation, optional date persistence, overdue predicate, sorting, rendering, and filter exit status. - Create `tests/test_cli.py`: isolated CLI tests u [... excerpt: first 3200 of 11517 characters]

See Pro