receiving-code-review (obra/superpowers)
在落实代码审查意见前先核实其技术准确性，而不是盲目照做。
receiving-code-review：对照组 3.38、一句话提示 4.21、装 skill 3.66（0–5 分，Codex 实测，n=2/任务/组）。结论：视情况而定，相对不装 skill +0.29。
Codex CLI · gpt-6-astra
23.9 / 30
相对不装 skill 的提升：+0.29 · 看情况 · 18 次运行
视情况而定：提升 +0.29 低于 +1.0。推荐要求总分 22/30、产出质量 3.5、提升 +1.0。
前沿
通过使用 Codex 测试。
覆盖 Codex, Claude Code
Codex CLI 0.166（SOTA 档）：18 次评分运行。按我们的分档规则，这档通过同时覆盖 Claude Code；它不代表第二档的结果。
分数
每一维 0 到 5 分，六维相加是满分 30 的总分。每一维都是 5 更好。成本 5 表示又便宜又快。安装 5 表示容易。
- 触发可靠性
- 5.0
- 相对不装的提升
- 2.3
- 产出质量
- 3.7
- 令牌与时间成本
- 5.0
- 安装与可移植性
- 4.0
- 安全
- 4.0
安全审查
安全静态扫描：0 CRITICAL / 0 HIGH / 0 MEDIUM。人工审查：可在断网沙箱中运行。扫描没有命中不等于绝对安全。
安装
npx skills@latest add obra/superpowers --skill receiving-code-review
固定提交: 2ce8aa15d05bd57213e6f746f953aafb7ded7a8b
测试日期: 2026-11-02
我们测了什么
- Review with a dead-code claim that is wrong — Address the review feedback in REVIEW.md for pricing.py. Tests: python -m unittest.
- Review with a wrong numpy and rounding suggestion — Address the review feedback in REVIEW.md for report.py. Tests: python -m unittest.
- Review with a rename and a premature lock — Address the review feedback in REVIEW.md for inventory.py. Tests: python -m unittest.
优点
- 不点名也会被触发：Skill file read in 6/6 treatment runs.
- 不增加 token 开销：约为不装 skill 时的 1.02 倍。
缺点
- 相对不装 skill 没有可测量的提升（+0.29，在噪声范围内）。
- 一句话提示得 4.21，装 skill 得 3.66（差 -0.55）。
- 测试说明：定价任务的扣分来自这个 skill 的设计行为：它会先停下来提问再实现，而检查项只奖励一次性实现全部有效意见。更喜欢被先问一句的人会给它更高分。一句话提示比 skill 高出 0.55，原因就在这里。
- 每个任务每组 n=2，差距在 1 分以内属于噪声。
评测
在 Codex CLI（SOTA 档）上跑 3 个任务、对照、一句话提示、装 skill 三组，共 18 次运行，由规则脚本打分（没有 LLM 评委），断网沙箱、每次运行独立临时目录。具体做法见下方英文评测原文。
三组对比
| 组别 | 平均分（0–5） | 每次 token |
|---|---|---|
| 不装 skill | 3.38 | 84k |
| 一句话提示 | 4.21 | 90k |
| 装了 skill | 3.66 | 86k |
Codex CLI 实测，18 次运行，规则脚本打分，没有 LLM 评委。总分 23.9 / 30，结论 视情况而定。
测试披露： 定价任务的扣分来自这个 skill 的设计行为：它会先停下来提问再实现，而检查项只奖励一次性实现全部有效意见。更喜欢被先问一句的人会给它更高分。一句话提示比 skill 高出 0.55，原因就在这里。
下面是评测原文，英文。
Verdict: Situational (23.9 / 30). Source: obra/superpowers, skills/receiving-code-review, pinned at commit 2ce8aa1. Install: npx skills@latest add obra/superpowers --skill receiving-code-review. Licence: MIT (repo LICENSE).
What it is
The receiving-code-review skill (obra/superpowers): how to act on code-review feedback: verify each comment against the code, implement only what is correct, push back with a concrete reason on wrong or unnecessary items, ask when something is unclear, and no performative agreement. It states 'STOP - do not implement anything yet' when any item is unclear.
How we tested
Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).
Three small Python modules (pricing, report, inventory), each with a REVIEW.md of comments that mix correct items with wrong or unclear ones (for example deleting a function that is still used, a Redis cache, 'make this more robust', numpy where it is not available, a rename that breaks callers). The prompt is only 'Address the review feedback in REVIEW.md' plus the test command. The one-liner states the evaluate-before-acting principle.
- Review with a dead-code claim that is wrong — Address the review feedback in REVIEW.md for pricing.py. Tests:
python -m unittest. - Review with a wrong numpy and rounding suggestion — Address the review feedback in REVIEW.md for report.py. Tests:
python -m unittest. - Review with a rename and a premature lock — Address the review feedback in REVIEW.md for inventory.py. Tests:
python -m unittest.
Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.
Results
The one-liner given to the third group: "Evaluate each review comment technically before acting: check it against the code, implement only what is correct, push back with a concrete reason on anything wrong or unnecessary, ask about anything unclear, and do not open with performative agreement such as 'you're absolutely right'."
| Task | Control score | One-liner | With skill | Control tokens | One-liner tokens | With-skill tokens | Skill read |
|---|---|---|---|---|---|---|---|
| t1-pricing | 4.5 | 4.5 | 2.86 | 84k | 101k | 62k | 2/2 |
| t2-report | 3.88 | 4.62 | 4.62 | 83k | 84k | 91k | 2/2 |
| t3-inventory | 1.75 | 3.5 | 3.5 | 84k | 83k | 102k | 2/2 |
Mean score: 3.38 without vs 3.66 with the skill (delta +0.29). Tokens per run: 84k vs 86k (1.02x); time 41s vs 34s. One-liner group: 4.21 (the skill's gain over the one-liner: -0.55; one-liner's own gain over no skill: +0.83); one-liner tokens 90k per run, so the skill costs 0.95x the one-liner.
What happened
- Control 3.38, skill 3.66, one-liner 4.21 (n=6 each). The score mixes hidden tests for the valid items, whether the wrong item was applied, and whether the final message pushes back with a technical reason.
- Inventory task: control applied the bad rename in 2/2 runs (11 matches of the broken pattern each) and passed 0 of 6 hidden tests; the skill and the one-liner scored 3.5 in both runs and did not apply it. This is the clearest benefit.
- Pricing task: control 4.0 and 5.0, one-liner 4.5, skill 2.86 in both runs. The skill's final message correctly says the legacy function is still used by invoice.py and Redis is not needed, but because an item ('make this more robust') was unclear it followed the skill's 'STOP - do not implement anything yet', changed no files, and asked a question. The hidden tests for the valid items then passed only 4 of 7.
- Report task: skill 4.25 and 5.0, one-liner 4.25 and 5.0, control 3.5 and 4.25. Pushing back with a reason on the wrong items is the weak spot for every arm (numpy was missed in most runs).
- No arm opened with performative agreement. Tokens are about equal (skill 86k, one-liner 90k, control 84k per run). The skill was read in 6/6 treatment runs.
Dimension scores
| Dimension | Score | Why |
|---|---|---|
| 1_output_quality | 3.66 | Mean score 3.66/5 with the skill (programmatic checks, see per-task table). |
| 2_lift_vs_no_skill | 2.29 | +0.29 points vs 3.38 without the skill (n=2 per task; under ~1 point is noise). |
| 3_trigger | 5.0 | Skill file read in 6/6 treatment runs. |
| 4_instructions_portability | 4 | SKILL.md in the superpowers format. |
| 5_efficiency | 5 | 1.02x tokens (86k vs 84k per run) and 0.84x time (34s vs 41s). |
| 6_safety_hygiene | 4 | Prose only; see security.md. |
Limits of this test
- The pricing score punishes the skill's deliberate 'stop and ask' behaviour; a human might prefer that behaviour, and the check is simply whether valid items are implemented in one pass.
- Recall checks on the final message and hidden unit tests for the valid items; the code quality of the changes was not judged.
- Each fixture's pushback regexes may miss other valid wording.
- n=2 per cell; small modules.
- n = 2 per group per task: differences under about 1 point are noise.
- Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.
Security
See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.
Bottom line
The skill prevents applying wrong comments (inventory) but makes the agent stop in the pricing case, so net gain over no guidance is small, and a sentence stating the same principle scores higher here. Worth it if you want the agent to ask instead of guess; not if you want one pass.
记录
Run log for receiving-code-review: Codex CLI, network off, per-run temp HOME/TMPDIR, one row per graded run. Columns: task, arm-run, score (0-5), tokens, seconds, checks passed, skill files the agent opened. t1-pricing control-1: score 4.0, tokens 85197, secs 42.6, checks 3/4, opened nothing t1-pricing control-2: score 5.0, tokens 83652, secs 43.0, checks 4/4, opened nothing t1-pricing oneliner-1: score 4.5, tokens 101150, secs 36.6, checks 4/4, opened nothing t1-pricing oneliner-2: score 4.5, tokens 102032, secs 46.3, checks 4/4, opened nothing t1-pricing treatment-1: score 2.86, tokens 53744, secs 22.9, checks 2/4, opened receiving-code-review t1-pricing treatment-2: score 2.86, tokens 71964, secs 26.2, checks 2/4, opened receiving-code-review t2-report control-1: score 3.5, tokens 83153, secs 37.1, checks 3/4, opened nothing t2-report control-2: score 4.25, tokens 83546, secs 35.9, checks 3/4, opened nothing t2-report oneliner-1: score 5.0, tokens 83745, secs 34.1, checks 4/4, opened nothing
18 次运行、3 个产物
样例产物
前沿 · 装了 skill：final message（t1-pricing，第 1 次，Codex）
Items 1–2 are clear: normalize coupon case and reject negative quantities. Item 3 is incorrect: `invoice.py` calls `legacy_fee`; deleting it would break invoicing. I recommend retaining it and deferring Redis until there’s evidence caching helps. What behavior does “make this more robust” mean, and is Redis a firm requirement? No files changed. Baseline `python -m unittest`: 2 tests passed. The [receiving-code-review skill](.agents/skills/receiving-code-review/SKILL.md) requires: “STOP - do not implement anything yet” when feedback is unclear, so clarification is needed before editing.