systematic-debugging (obra/superpowers)
systematic-debugging：对照组 4.45、一句话提示 4.86、装 skill 4.45（0–5 分，Codex 实测，n=2/任务/组）。结论：视情况而定，相对不装 skill +0.00。
22.4 / 30
相对不装 skill 的提升：+0.00
视情况而定：提升 +0.00 低于 +1.0。推荐要求总分 22/30、产出质量 3.5、提升 +1.0。
前沿
通过使用 Codex 测试。
覆盖 Codex, Claude Code
Codex CLI 0.166（SOTA 档）：18 次评分运行。按我们的分档规则，这档通过同时覆盖 Claude Code；它不代表第二档的结果。
分数
每一维 0 到 5 分，六维相加是满分 30 的总分。每一维都是 5 更好。成本 5 表示又便宜又快。安装 5 表示容易。
- 触发可靠性
- 5.0
- 相对不装的提升
- 2.0
- 产出质量
- 4.5
- 令牌与时间成本
- 3.0
- 安装与可移植性
- 4.0
- 安全
- 4.0
安全审查
安全静态扫描：0 CRITICAL / 0 HIGH / 0 MEDIUM。人工审查：可在断网沙箱中运行。扫描没有命中不等于绝对安全。
安装
npx skills@latest add obra/superpowers --skill systematic-debugging
固定提交: 2ce8aa15d05bd57213e6f746f953aafb7ded7a8b
测试日期: 2026-11-02
我们测了什么
- Off-by-one in a small function — moving_average in movavg.py returns wrong numbers: moving_average([2, 4, 6, 8, 10], 3) should be [2, 3, 4, 6, 8] but the later values come out wrong. Find the cause and fix it. Tests: python3 -m unittest.
- Cross-file date bug — Our weekly report headers show wrong dates around the new year, e.g. header('2026-W01') should be 'Week 2026-W01: Dec 29 - Jan 04, 2026'. Find the cause and fix it.
- Order-dependent test failure — Tests in tests/test_settings.py pass on their own but test_b fails when the whole file runs, and configs seem to carry tags from earlier calls. Diagnose and fix it.
优点
- 不点名也会被触发：Skill file read in 6/6 treatment runs.
缺点
- 相对不装 skill 没有可测量的提升（+0.00，在噪声范围内）。
- 一句话提示得 4.86，装 skill 得 4.45（差 -0.42）。
- 测试说明：一句话提示组（6 次）是后来在另一台机器上补跑的，和对照组、安装组不是同一台。
- 每个任务每组 n=2，差距在 1 分以内属于噪声。
评测
在 Codex CLI（SOTA 档）上跑 3 个任务、对照、一句话提示、装 skill 三组，共 18 次运行，由规则脚本打分（没有 LLM 评委），断网沙箱、每次运行独立临时目录。具体做法见下方英文评测原文。
三组对比
| 组别 | 平均分（0–5） | 每次 token |
|---|---|---|
| 不装 skill | 4.45 | 75k |
| 一句话提示 | 4.86 | 98k |
| 装了 skill | 4.45 | 99k |
Codex CLI 实测，18 次运行，规则脚本打分，没有 LLM 评委。总分 22.4 / 30，结论 视情况而定。
测试披露： 一句话提示组（6 次）是后来在另一台机器上补跑的，和对照组、安装组不是同一台。
下面是评测原文，英文。
Verdict: Situational (22.4 / 30). Source: obra/superpowers, skills/systematic-debugging, pinned at commit 2ce8aa1. Install: npx skills@latest add obra/superpowers --skill systematic-debugging. Licence: MIT (repo LICENSE).
What it is
The systematic-debugging skill from obra/superpowers: 'use when encountering any bug, test failure or unexpected behaviour, before proposing fixes'. Four phases (investigate root cause, find patterns, hypothesis, fix), a rule against stacking fixes, and companion notes (root-cause tracing, defense in depth, condition-based waiting) plus test-pressure scenarios and one shell script (find-polluter.sh, a local bisect over test files).
How we tested
Codex CLI 0.166 (gpt-6-astra, low reasoning) on a Mac test machine, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill.
- Off-by-one in a small function —
moving_averagein movavg.py returns wrong numbers: moving_average([2, 4, 6, 8, 10], 3) should be [2, 3, 4, 6, 8] but the later values come out wrong. Find the cause and fix it. Tests:python3 -m unittest. - Cross-file date bug — Our weekly report headers show wrong dates around the new year, e.g. header('2026-W01') should be 'Week 2026-W01: Dec 29 - Jan 04, 2026'. Find the cause and fix it.
- Order-dependent test failure — Tests in tests/test_settings.py pass on their own but
test_bfails when the whole file runs, and configs seem to carry tags from earlier calls. Diagnose and fix it.
Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.
Results
The one-liner given to the third group: "Debug systematically: reproduce the failure first, investigate the root cause and form one hypothesis at a time with evidence before changing code, make the smallest fix for the root cause (no stacked guesses), and verify with the tests."
| Task | Control score | One-liner | With skill | Control tokens | One-liner tokens | With-skill tokens | Skill read |
|---|---|---|---|---|---|---|---|
| t1-moving-average | 4.17 | 4.58 | 4.58 | 60k | 62k | 69k | 2/2 |
| t2-weekly-dates | 4.17 | 5.0 | 4.17 | 78k | 104k | 109k | 2/2 |
| t3-shared-state | 5.0 | 5.0 | 4.58 | 85k | 127k | 118k | 2/2 |
Mean score: 4.45 without vs 4.45 with the skill (delta +0.0). Tokens per run: 75k vs 99k (1.33x); time 34s vs 45s. One-liner group: 4.86 (the skill's gain over the one-liner: -0.42; one-liner's own gain over no skill: +0.42); one-liner tokens 98k per run, so the skill costs 1.01x the one-liner.
What happened
- Outcome: the hidden tests passed in every run of both groups, so correctness did not differ.
- Process check 'ran the tests before editing the code' (reproduce first): control 4/6 runs missed it, treatment 4/6 runs missed it. The skill's first phase is 'reproduce', but the agents with the skill still often patched before running anything.
- The skill was read in 6/6 treatment runs, so triggering is not the problem; it simply did not change behaviour on these small bugs (the bugs were findable by reading the code).
- Cost: roughly 1.3x tokens per run (89k vs 67k).
- One-sentence version (reproduce first, one hypothesis at a time, smallest fix, verify): mean 4.86 against 4.45 for control and 4.45 for the skill, at 98k tokens per run against 75k (1.3x) and 99k for the skill (1.0x).
- The difference is in running the tests before editing: 5 of 6 one-liner runs did so against 2 of 6 control runs and 2 of 6 skill runs. The skill did not change that behaviour; the sentence naming it did. Every group explained the root cause in all runs and passed every hidden test, with small diffs.
Dimension scores
| Dimension | Score | Why |
|---|---|---|
| 1_output_quality | 4.45 | Mean score 4.45/5 with the skill (programmatic checks, see per-task table). |
| 2_lift_vs_no_skill | 2.0 | +0.00 points vs 4.45 without the skill (n=2 per task; under ~1 point is noise). |
| 3_trigger | 5.0 | Skill file read in 6/6 treatment runs. |
| 4_instructions_portability | 4 | |
| 5_efficiency | 3 | 1.33x tokens (99k vs 75k per run) and 1.32x time (45s vs 34s). |
| 6_safety_hygiene | 4 |
Limits of this test
- Three small planted bugs (off-by-one, cross-file date logic, order-dependent test pollution) — all easy enough that the baseline fixes them.
- The skill is designed for hard, multi-attempt bugs; we did not test those.
find-polluter.shwas not exercised.- n = 2 per group per task: differences under about 1 point are noise.
- Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.
Security
See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.
Bottom line
On three small bugs the skill added about 1.3x tokens and no measurable change: control and skill both scored 4.45 and both ran the tests before editing in only 2 of 6 runs. The one-sentence version that says 'reproduce first' did better (4.86, 5 of 6). It might pay off on harder bugs, which we did not test.
记录
Run log for systematic-debugging: Codex CLI, network off, per-run temp HOME/TMPDIR, one row per graded run. Columns: task, arm-run, score (0-5), tokens, seconds, checks passed, skill files the agent opened. t1-moving-average control-1: score 4.17, tokens 60337, secs 30.6, checks 3/4, opened nothing t1-moving-average control-2: score 4.17, tokens 60293, secs 24.1, checks 3/4, opened nothing t1-moving-average oneliner-1: score 4.17, tokens 62543, secs 30.2, checks 3/4, opened nothing t1-moving-average oneliner-2: score 5.0, tokens 62033, secs 32.0, checks 4/4, opened nothing t1-moving-average treatment-1: score 5.0, tokens 69716, secs 31.0, checks 4/4, opened systematic-debugging t1-moving-average treatment-2: score 4.17, tokens 69385, secs 26.6, checks 3/4, opened systematic-debugging t2-weekly-dates control-1: score 4.17, tokens 78575, secs 38.8, checks 3/4, opened nothing t2-weekly-dates control-2: score 4.17, tokens 78005, secs 39.5, checks 3/4, opened nothing t2-weekly-dates oneliner-1: score 5.0, tokens 96141, secs 41.5, checks 4/4, opened nothing
18 次运行、3 个产物
样例产物
前沿 · 装了 skill：movavg.py（t1-moving-average，第 1 次，Codex）
def moving_average(values, window): """Trailing moving average. Output i is the mean of the last `window` values ending at index i. For i < window - 1 only the values available so far are used. Raises ValueError if window < 1. """ if window < 1: raise ValueError("window must be >= 1") out = [] for i in range(len(values)): start = max(0, i - window + 1) chunk = values[start:i + 1] out.append(sum(chunk) / len(chunk)) return out