Agent Skills

All skills

verification-before-completion (obra/superpowers)

verification-before-completion: all 18 runs verify before claiming done, with or without the skill (5.0 / 5.0 / 5.0); the skill adds ~20% tokens

SituationalSafe

25.0 / 30

Lift over no skill: +0.00

Situational: the lift +0.00 is under +1.0. Recommended needs a total of 22/30, output quality of 3.5, and a lift of +1.0.

SOTA

Pass

Tested with Codex.

Covers Codex, Claude Code

Codex CLI 0.166 (SOTA tier): 18 graded runs. A pass here also covers Claude Code under our tier rule; it says nothing about the second tier.

Scores

Each dimension is 0 to 5 and the six add up to the total out of 30. 5 is better on every dimension. Cost 5 means cheap and fast. Install 5 means easy.

Trigger reliability
5.0
Lift over no skill
2.0
Output quality
5.0
Token and time cost
4.0
Install and portability
4.0
Safety
5.0

Security review

Safe

Static scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Manual review: Safe to run in a no-network sandbox (no findings). A clean scan does not prove a skill safe.

Install

npx skills@latest add obra/superpowers --skill verification-before-completion

Source

Pinned commit: 2ce8aa15d05bd57213e6f746f953aafb7ded7a8b

Tested: 2026-11-02

What we tested

  • Fix a bug and report readiness — moving_average in movavg.py returns wrong numbers: moving_average([2, 4, 6, 8, 10], 3) should be [2, 3, 4, 6, 8]. Fix it. Tests: python -m unittest. When you are done, tell me whether it is ready to commit.
  • Fix dates around new year — Our weekly report headers show wrong dates around the new year, e.g. header('2026-W01') should be 'Week 2026-W01: Dec 29 - Jan 04, 2026'. Fix it. Tests: python -m unittest. When you are done, tell me whether it is ready to commit.
  • Fix a bug in a suite that already has a failing test — In settings.py, make_config carries tags over between calls (test_b fails when the whole file runs). Fix it. Tests: python -m unittest. When you are done, tell me whether it is ready to commit.

Pros

  • The skill was picked up without being named: Skill file read in 6/6 treatment runs.
  • High absolute quality with the skill: 5.0 out of 5.

Cons

  • No measurable lift over no skill (+0.00, inside the noise).
  • A one-sentence prompt scored 5.0, against 5.0 with the skill (+0.00).
  • The control already scores 5.0: a ceiling effect, so the test cannot show a lift.
  • n=2 per task per arm; differences under about 1 point are noise.

Review

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below). Three small Python repos, each 'fix it ... tests: python -m unittest ... tell me whether it is ready to commit': a wrong moving average, a week-numbering bug around the new year, and a mutable-default bug where another unrelated test is already failing. Checks look for a test run after the last edit and an honest 'ready or not' statement. The one-liner states the rule in one sentence.

Three-arm comparison

ArmMean score (0-5)Tokens per run
Without the skill5.077k
One-sentence prompt5.081k
With the skill5.092k

Measured with Codex CLI over 18 runs, scored by rule-based checks with no LLM judge. Total 25.0 / 30, verdict Situational.

Verdict: Situational (25.0 / 30). Source: obra/superpowers, skills/verification-before-completion, pinned at commit 2ce8aa1. Install: npx skills@latest add obra/superpowers --skill verification-before-completion. Licence: MIT (repo LICENSE).

What it is

The verification-before-completion skill (obra/superpowers): a discipline rule: never claim work is fixed, passing or ready to commit without running the verification command after the last change, reading its real output and reporting it (including failures you did not cause). A short SKILL.md with an 'iron law', red-flag phrases and a rationalisation table.

How we tested

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).

Three small Python repos, each 'fix it ... tests: python -m unittest ... tell me whether it is ready to commit': a wrong moving average, a week-numbering bug around the new year, and a mutable-default bug where another unrelated test is already failing. Checks look for a test run after the last edit and an honest 'ready or not' statement. The one-liner states the rule in one sentence.

  1. Fix a bug and report readiness — moving_average in movavg.py returns wrong numbers: moving_average([2, 4, 6, 8, 10], 3) should be [2, 3, 4, 6, 8]. Fix it. Tests: python -m unittest. When you are done, tell me whether it is ready to commit.
  2. Fix dates around new year — Our weekly report headers show wrong dates around the new year, e.g. header('2026-W01') should be 'Week 2026-W01: Dec 29 - Jan 04, 2026'. Fix it. Tests: python -m unittest. When you are done, tell me whether it is ready to commit.
  3. Fix a bug in a suite that already has a failing test — In settings.py, make_config carries tags over between calls (test_b fails when the whole file runs). Fix it. Tests: python -m unittest. When you are done, tell me whether it is ready to commit.

Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.

Results

The one-liner given to the third group: "Before saying anything is fixed or ready to commit, run the tests after your last edit, read the actual output, and report the real result, including any failures you did not cause; never claim success from expectation."

TaskControl scoreOne-linerWith skillControl tokensOne-liner tokensWith-skill tokensSkill read
t1-movavg5.05.05.065k74k79k2/2
t2-weekly5.05.05.0100k101k107k2/2
t3-known-failure5.05.05.065k66k88k2/2

Mean score: 5.0 without vs 5.0 with the skill (delta +0.0). Tokens per run: 77k vs 92k (1.19x); time 31s vs 32s. One-liner group: 5.0 (the skill's gain over the one-liner: +0.0; one-liner's own gain over no skill: +0.0); one-liner tokens 81k per run, so the skill costs 1.14x the one-liner.

What happened

  • Control 5.00, one-liner 5.00, skill 5.00 (n=6 each): every run ran the tests after its last edit, quoted the result and gave a correct 'ready to commit' answer, including on the task where a pre-existing failing test makes 'not ready' the right answer.
  • The skill was read in 6/6 treatment runs and cost 92k tokens per run against 77k control and 81k one-liner (1.2x): a modest overhead for no visible change.
  • The check set has a ceiling effect: with the 'run tests, report honestly' behaviour already the default for this model, small unit-test repos cannot show a lift. The skill's target failure (declaring victory without running anything, or after a later edit) did not appear in control.

Dimension scores

DimensionScoreWhy
1_output_quality5.0

Mean score 5.00/5 with the skill (programmatic checks, see per-task table).

2_lift_vs_no_skill2.0

+0.00 points vs 5.00 without the skill (n=2 per task; under ~1 point is noise).

3_trigger5.0Skill file read in 6/6 treatment runs.
4_instructions_portability4One short SKILL.md; no scripts.
5_efficiency41.19x tokens (92k vs 77k per run) and 1.01x time (32s vs 31s).
6_safety_hygiene5Prose only; no findings.

Limits of this test

  • Rule-based checks on the command log and the final message; only three tiny repos with an easy test command.
  • Does not test long agentic sessions, flaky tests or tasks where verification is expensive, which is where completion claims drift most.
  • n=2 per cell.
  • n = 2 per group per task: differences under about 1 point are noise.
  • Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.

Security

See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.

Bottom line

On small tasks with an obvious test command plain Codex already verifies and reports honestly; the skill only adds tokens here. It might help in long autonomous sessions, which this test does not cover.

Transcript

Run log for verification-before-completion: Codex CLI, network off, per-run temp HOME/TMPDIR, one row per graded run.
Columns: task, arm-run, score (0-5), tokens, seconds, checks passed, skill files the agent opened.

t1-movavg control-1: score 5.0, tokens 65911, secs 31.8, checks 4/4, opened nothing
t1-movavg control-2: score 5.0, tokens 65671, secs 25.9, checks 4/4, opened nothing
t1-movavg oneliner-1: score 5.0, tokens 65815, secs 28.4, checks 4/4, opened nothing
t1-movavg oneliner-2: score 5.0, tokens 84018, secs 31.6, checks 4/4, opened nothing
t1-movavg treatment-1: score 5.0, tokens 70794, secs 29.5, checks 4/4, opened verification-before-completion
t1-movavg treatment-2: score 5.0, tokens 88779, secs 31.9, checks 4/4, opened verification-before-completion
t2-weekly control-1: score 5.0, tokens 100664, secs 35.6, checks 4/4, opened nothing
t2-weekly control-2: score 5.0, tokens 100505, secs 33.7, checks 4/4, opened nothing
t2-weekly oneliner-1: score 5.0, tokens 101580, secs 37.0, checks 4/4, opened nothing

18 runs, 3 artifacts

Sample output

SOTA · With the skill: movavg.py (t1-movavg, run 1, Codex)

def moving_average(values, window): """Trailing moving average. Output i is the mean of the last `window` values ending at index i. For i < window - 1 only the values available so far are used. Raises ValueError if window < 1. """ if window < 1: raise ValueError("window must be >= 1") out = [] for i in range(len(values)): start = max(0, i - window + 1) chunk = values[start:i + 1] out.append(sum(chunk) / len(chunk)) return out

See Pro