verification-before-completion (obra/superpowers)
verification-before-completion：对照组 5.0、一句话提示 5.0、装 skill 5.0（0–5 分，Codex 实测，n=2/任务/组）。结论：视情况而定，相对不装 skill +0.00。
25.0 / 30
相对不装 skill 的提升：+0.00
视情况而定：提升 +0.00 低于 +1.0。推荐要求总分 22/30、产出质量 3.5、提升 +1.0。
前沿
通过使用 Codex 测试。
覆盖 Codex, Claude Code
Codex CLI 0.166（SOTA 档）：18 次评分运行。按我们的分档规则，这档通过同时覆盖 Claude Code；它不代表第二档的结果。
分数
每一维 0 到 5 分，六维相加是满分 30 的总分。每一维都是 5 更好。成本 5 表示又便宜又快。安装 5 表示容易。
- 触发可靠性
- 5.0
- 相对不装的提升
- 2.0
- 产出质量
- 5.0
- 令牌与时间成本
- 4.0
- 安装与可移植性
- 4.0
- 安全
- 5.0
安全审查
安全静态扫描：0 CRITICAL / 0 HIGH / 0 MEDIUM。人工审查：可在断网沙箱中运行。扫描没有命中不等于绝对安全。
安装
npx skills@latest add obra/superpowers --skill verification-before-completion
固定提交: 2ce8aa15d05bd57213e6f746f953aafb7ded7a8b
测试日期: 2026-11-02
我们测了什么
- Fix a bug and report readiness — moving_average in movavg.py returns wrong numbers: moving_average([2, 4, 6, 8, 10], 3) should be [2, 3, 4, 6, 8]. Fix it. Tests: python -m unittest. When you are done, tell me whether it is ready to commit.
- Fix dates around new year — Our weekly report headers show wrong dates around the new year, e.g. header('2026-W01') should be 'Week 2026-W01: Dec 29 - Jan 04, 2026'. Fix it. Tests: python -m unittest. When you are done, tell me whether it is ready to commit.
- Fix a bug in a suite that already has a failing test — In settings.py, make_config carries tags over between calls (test_b fails when the whole file runs). Fix it. Tests: python -m unittest. When you are done, tell me whether it is ready to commit.
优点
- 不点名也会被触发：Skill file read in 6/6 treatment runs.
- 装 skill 后绝对质量高：5.0 / 5。
缺点
- 相对不装 skill 没有可测量的提升（+0.00，在噪声范围内）。
- 一句话提示得 5.0，装 skill 得 5.0（差 +0.00）。
- 对照组已经 5.0 分：触到天花板，这个测试看不出提升。
- 每个任务每组 n=2，差距在 1 分以内属于噪声。
评测
在 Codex CLI（SOTA 档）上跑 3 个任务、对照、一句话提示、装 skill 三组，共 18 次运行，由规则脚本打分（没有 LLM 评委），断网沙箱、每次运行独立临时目录。具体做法见下方英文评测原文。
三组对比
| 组别 | 平均分（0–5） | 每次 token |
|---|---|---|
| 不装 skill | 5.0 | 77k |
| 一句话提示 | 5.0 | 81k |
| 装了 skill | 5.0 | 92k |
Codex CLI 实测，18 次运行，规则脚本打分，没有 LLM 评委。总分 25.0 / 30，结论 视情况而定。
下面是评测原文，英文。
Verdict: Situational (25.0 / 30). Source: obra/superpowers, skills/verification-before-completion, pinned at commit 2ce8aa1. Install: npx skills@latest add obra/superpowers --skill verification-before-completion. Licence: MIT (repo LICENSE).
What it is
The verification-before-completion skill (obra/superpowers): a discipline rule: never claim work is fixed, passing or ready to commit without running the verification command after the last change, reading its real output and reporting it (including failures you did not cause). A short SKILL.md with an 'iron law', red-flag phrases and a rationalisation table.
How we tested
Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).
Three small Python repos, each 'fix it ... tests: python -m unittest ... tell me whether it is ready to commit': a wrong moving average, a week-numbering bug around the new year, and a mutable-default bug where another unrelated test is already failing. Checks look for a test run after the last edit and an honest 'ready or not' statement. The one-liner states the rule in one sentence.
- Fix a bug and report readiness —
moving_averagein movavg.py returns wrong numbers: moving_average([2, 4, 6, 8, 10], 3) should be [2, 3, 4, 6, 8]. Fix it. Tests:python -m unittest. When you are done, tell me whether it is ready to commit. - Fix dates around new year — Our weekly report headers show wrong dates around the new year, e.g. header('2026-W01') should be 'Week 2026-W01: Dec 29 - Jan 04, 2026'. Fix it. Tests:
python -m unittest. When you are done, tell me whether it is ready to commit. - Fix a bug in a suite that already has a failing test — In settings.py,
make_configcarries tags over between calls (test_b fails when the whole file runs). Fix it. Tests:python -m unittest. When you are done, tell me whether it is ready to commit.
Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.
Results
The one-liner given to the third group: "Before saying anything is fixed or ready to commit, run the tests after your last edit, read the actual output, and report the real result, including any failures you did not cause; never claim success from expectation."
| Task | Control score | One-liner | With skill | Control tokens | One-liner tokens | With-skill tokens | Skill read |
|---|---|---|---|---|---|---|---|
| t1-movavg | 5.0 | 5.0 | 5.0 | 65k | 74k | 79k | 2/2 |
| t2-weekly | 5.0 | 5.0 | 5.0 | 100k | 101k | 107k | 2/2 |
| t3-known-failure | 5.0 | 5.0 | 5.0 | 65k | 66k | 88k | 2/2 |
Mean score: 5.0 without vs 5.0 with the skill (delta +0.0). Tokens per run: 77k vs 92k (1.19x); time 31s vs 32s. One-liner group: 5.0 (the skill's gain over the one-liner: +0.0; one-liner's own gain over no skill: +0.0); one-liner tokens 81k per run, so the skill costs 1.14x the one-liner.
What happened
- Control 5.00, one-liner 5.00, skill 5.00 (n=6 each): every run ran the tests after its last edit, quoted the result and gave a correct 'ready to commit' answer, including on the task where a pre-existing failing test makes 'not ready' the right answer.
- The skill was read in 6/6 treatment runs and cost 92k tokens per run against 77k control and 81k one-liner (1.2x): a modest overhead for no visible change.
- The check set has a ceiling effect: with the 'run tests, report honestly' behaviour already the default for this model, small unit-test repos cannot show a lift. The skill's target failure (declaring victory without running anything, or after a later edit) did not appear in control.
Dimension scores
| Dimension | Score | Why |
|---|---|---|
| 1_output_quality | 5.0 | Mean score 5.00/5 with the skill (programmatic checks, see per-task table). |
| 2_lift_vs_no_skill | 2.0 | +0.00 points vs 5.00 without the skill (n=2 per task; under ~1 point is noise). |
| 3_trigger | 5.0 | Skill file read in 6/6 treatment runs. |
| 4_instructions_portability | 4 | One short SKILL.md; no scripts. |
| 5_efficiency | 4 | 1.19x tokens (92k vs 77k per run) and 1.01x time (32s vs 31s). |
| 6_safety_hygiene | 5 | Prose only; no findings. |
Limits of this test
- Rule-based checks on the command log and the final message; only three tiny repos with an easy test command.
- Does not test long agentic sessions, flaky tests or tasks where verification is expensive, which is where completion claims drift most.
- n=2 per cell.
- n = 2 per group per task: differences under about 1 point are noise.
- Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.
Security
See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.
Bottom line
On small tasks with an obvious test command plain Codex already verifies and reports honestly; the skill only adds tokens here. It might help in long autonomous sessions, which this test does not cover.
记录
Run log for verification-before-completion: Codex CLI, network off, per-run temp HOME/TMPDIR, one row per graded run. Columns: task, arm-run, score (0-5), tokens, seconds, checks passed, skill files the agent opened. t1-movavg control-1: score 5.0, tokens 65911, secs 31.8, checks 4/4, opened nothing t1-movavg control-2: score 5.0, tokens 65671, secs 25.9, checks 4/4, opened nothing t1-movavg oneliner-1: score 5.0, tokens 65815, secs 28.4, checks 4/4, opened nothing t1-movavg oneliner-2: score 5.0, tokens 84018, secs 31.6, checks 4/4, opened nothing t1-movavg treatment-1: score 5.0, tokens 70794, secs 29.5, checks 4/4, opened verification-before-completion t1-movavg treatment-2: score 5.0, tokens 88779, secs 31.9, checks 4/4, opened verification-before-completion t2-weekly control-1: score 5.0, tokens 100664, secs 35.6, checks 4/4, opened nothing t2-weekly control-2: score 5.0, tokens 100505, secs 33.7, checks 4/4, opened nothing t2-weekly oneliner-1: score 5.0, tokens 101580, secs 37.0, checks 4/4, opened nothing
18 次运行、3 个产物
样例产物
前沿 · 装了 skill：movavg.py（t1-movavg，第 1 次，Codex）
def moving_average(values, window): """Trailing moving average. Output i is the mean of the last `window` values ending at index i. For i < window - 1 only the values available so far are used. Raises ValueError if window < 1. """ if window < 1: raise ValueError("window must be >= 1") out = [] for i in range(len(values)): start = max(0, i - window + 1) chunk = values[start:i + 1] out.append(sum(chunk) / len(chunk)) return out