receiving-code-review (obra/superpowers)

Review

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below). Three small Python modules (pricing, report, inventory), each with a REVIEW.md of comments that mix correct items with wrong or unclear ones (for example deleting a function that is still used, a Redis cache, 'make this more robust', numpy where it is not available, a rename that breaks callers). The prompt is only 'Address the review feedback in REVIEW.md' plus the test command. The one-liner states the evaluate-before-acting principle.

Three-arm comparison

Measured with Codex CLI over 18 runs, scored by rule-based checks with no LLM judge. Total 23.9 / 30, verdict Situational.

Verdict: Situational (23.9 / 30). Source: obra/superpowers, skills/receiving-code-review, pinned at commit 2ce8aa1. Install: npx skills@latest add obra/superpowers --skill receiving-code-review. Licence: MIT (repo LICENSE).

What it is

The receiving-code-review skill (obra/superpowers): how to act on code-review feedback: verify each comment against the code, implement only what is correct, push back with a concrete reason on wrong or unnecessary items, ask when something is unclear, and no performative agreement. It states 'STOP - do not implement anything yet' when any item is unclear.

How we tested

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).

Three small Python modules (pricing, report, inventory), each with a REVIEW.md of comments that mix correct items with wrong or unclear ones (for example deleting a function that is still used, a Redis cache, 'make this more robust', numpy where it is not available, a rename that breaks callers). The prompt is only 'Address the review feedback in REVIEW.md' plus the test command. The one-liner states the evaluate-before-acting principle.

1. Review with a dead-code claim that is wrong — Address the review feedback in REVIEW.md for pricing.py. Tests: python -m unittest.
2. Review with a wrong numpy and rounding suggestion — Address the review feedback in REVIEW.md for report.py. Tests: python -m unittest.
3. Review with a rename and a premature lock — Address the review feedback in REVIEW.md for inventory.py. Tests: python -m unittest.

Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.

Results

The one-liner given to the third group: "Evaluate each review comment technically before acting: check it against the code, implement only what is correct, push back with a concrete reason on anything wrong or unnecessary, ask about anything unclear, and do not open with performative agreement such as 'you're absolutely right'."

Mean score: 3.38 without vs 3.66 with the skill (delta +0.29). Tokens per run: 84k vs 86k (1.02x); time 41s vs 34s. One-liner group: 4.21 (the skill's gain over the one-liner: -0.55; one-liner's own gain over no skill: +0.83); one-liner tokens 90k per run, so the skill costs 0.95x the one-liner.

What happened

• Control 3.38, skill 3.66, one-liner 4.21 (n=6 each). The score mixes hidden tests for the valid items, whether the wrong item was applied, and whether the final message pushes back with a technical reason.
• Inventory task: control applied the bad rename in 2/2 runs (11 matches of the broken pattern each) and passed 0 of 6 hidden tests; the skill and the one-liner scored 3.5 in both runs and did not apply it. This is the clearest benefit.
• Pricing task: control 4.0 and 5.0, one-liner 4.5, skill 2.86 in both runs. The skill's final message correctly says the legacy function is still used by invoice.py and Redis is not needed, but because an item ('make this more robust') was unclear it followed the skill's 'STOP - do not implement anything yet', changed no files, and asked a question. The hidden tests for the valid items then passed only 4 of 7.
• Report task: skill 4.25 and 5.0, one-liner 4.25 and 5.0, control 3.5 and 4.25. Pushing back with a reason on the wrong items is the weak spot for every arm (numpy was missed in most runs).
• No arm opened with performative agreement. Tokens are about equal (skill 86k, one-liner 90k, control 84k per run). The skill was read in 6/6 treatment runs.

Dimension scores

Limits of this test

• The pricing score punishes the skill's deliberate 'stop and ask' behaviour; a human might prefer that behaviour, and the check is simply whether valid items are implemented in one pass.
• Recall checks on the final message and hidden unit tests for the valid items; the code quality of the changes was not judged.
• Each fixture's pushback regexes may miss other valid wording.
• n=2 per cell; small modules.
• n = 2 per group per task: differences under about 1 point are noise.
• Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.

Security

See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.

Bottom line

The skill prevents applying wrong comments (inventory) but makes the agent stop in the pricing case, so net gain over no guidance is small, and a sentence stating the same principle scores higher here. Worth it if you want the agent to ask instead of guess; not if you want one pass.