code-review (Matt Pocock)
code-review：对照组 4.37、一句话提示 4.37、装 skill 4.45（0–5 分，Codex 实测，n=2/任务/组）。结论：视情况而定，相对不装 skill +0.08。
23.5 / 30
相对不装 skill 的提升：+0.08
视情况而定：提升 +0.08 低于 +1.0。推荐要求总分 22/30、产出质量 3.5、提升 +1.0。
前沿
通过使用 Codex 测试。
覆盖 Codex, Claude Code
Codex CLI 0.166（SOTA 档）：18 次评分运行。按我们的分档规则，这档通过同时覆盖 Claude Code；它不代表第二档的结果。
分数
每一维 0 到 5 分，六维相加是满分 30 的总分。每一维都是 5 更好。成本 5 表示又便宜又快。安装 5 表示容易。
- 触发可靠性
- 5.0
- 相对不装的提升
- 2.1
- 产出质量
- 4.5
- 令牌与时间成本
- 3.0
- 安装与可移植性
- 4.0
- 安全
- 5.0
安全审查
安全静态扫描：0 CRITICAL / 0 HIGH / 0 MEDIUM。人工审查：可在断网沙箱中运行。扫描没有命中不等于绝对安全。
安装
npx skills@latest add mattpocock/skills --skill code-review
固定提交: 4ae2317ddb22b046e26455e4f9c9b3aaaf62212d
测试日期: 2026-11-02
我们测了什么
- Review a small service module — Review orders_service.py before I merge it. List the problems you find, most serious first. Do not change any files.
- Review an auth helper — Review auth_handlers.py: I just wrote it for our login flow. List the problems, most serious first. Do not change any files.
- Review the latest commit against written standards and spec — Review my latest commit (the changes since HEAD~1) against CODING_STANDARDS.md and SPEC.md. List every deviation, most serious first. Do not change any files.
优点
- 不点名也会被触发：Skill file read in 6/6 treatment runs.
缺点
- 相对不装 skill 没有可测量的提升（+0.08，在噪声范围内）。
- 每个任务每组 n=2，差距在 1 分以内属于噪声。
评测
在 Codex CLI（SOTA 档）上跑 3 个任务、对照、一句话提示、装 skill 三组，共 18 次运行，由规则脚本打分（没有 LLM 评委），断网沙箱、每次运行独立临时目录。具体做法见下方英文评测原文。
三组对比
| 组别 | 平均分（0–5） | 每次 token |
|---|---|---|
| 不装 skill | 4.37 | 78k |
| 一句话提示 | 4.37 | 77k |
| 装了 skill | 4.45 | 114k |
Codex CLI 实测，18 次运行，规则脚本打分，没有 LLM 评委。总分 23.5 / 30，结论 视情况而定。
下面是评测原文，英文。
Verdict: Situational (23.5 / 30). Source: mattpocock/skills, skills/engineering/code-review, pinned at commit 4ae2317. Install: npx skills@latest add mattpocock/skills --skill code-review. Licence: MIT (repo LICENSE).
What it is
The code-review skill from mattpocock/skills (engineering/code-review). It reviews changes since a given ref on two axes: does the code follow the repo's documented coding standards (and common smells), and does it implement what the spec asked; it is written for repos with standards/spec docs and for use with sub-agents. Markdown only, no scripts.
How we tested
Codex CLI 0.166 (gpt-6-astra, low reasoning) on a Mac test machine, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).
Two single-file reviews (an order service with 8 planted problems, three of them not security-related; an auth helper with 8 planted problems) and one review of the latest commit against CODING_STANDARDS.md and SPEC.md (8 planted deviations), which is the skill's intended use. All prompts are read-only ('list the problems, most serious first, do not change files').
- Review a small service module — Review orders_service.py before I merge it. List the problems you find, most serious first. Do not change any files.
- Review an auth helper — Review auth_handlers.py: I just wrote it for our login flow. List the problems, most serious first. Do not change any files.
- Review the latest commit against written standards and spec — Review my latest commit (the changes since HEAD~1) against CODING_STANDARDS.md and SPEC.md. List every deviation, most serious first. Do not change any files.
Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.
Results
The one-liner given to the third group: "Review on two axes: whether the code follows the repo's documented coding standards (plus common code smells), and whether it implements what the spec asked, and cite the file and line for each finding."
| Task | Control score | One-liner | With skill | Control tokens | One-liner tokens | With-skill tokens | Skill read |
|---|---|---|---|---|---|---|---|
| t1-orders | 3.82 | 3.59 | 3.82 | 79k | 79k | 94k | 2/2 |
| t2-auth | 4.29 | 4.53 | 4.53 | 76k | 70k | 93k | 2/2 |
| t3-diff | 5.0 | 5.0 | 5.0 | 78k | 80k | 153k | 2/2 |
Mean score: 4.37 without vs 4.45 with the skill (delta +0.08). Tokens per run: 78k vs 114k (1.46x); time 44s vs 57s. One-liner group: 4.37 (the skill's gain over the one-liner: +0.08; one-liner's own gain over no skill: +0.0); one-liner tokens 77k per run, so the skill costs 1.49x the one-liner.
What happened
- No measurable gain: mean score 4.37 without the skill, 4.37 with the one-sentence version, 4.45 with the skill. Planted-issue recall per run: 5.0 of 8 control, 5.0 one-liner, 5.1 skill.
- On the task the skill was written for (review a commit against a standards file and a spec) every group found all 8 deviations in 6 of 6 runs. Plain Codex already reads CODING_STANDARDS.md and SPEC.md and compares the diff to them when asked to.
- On the two single-file reviews every group found 5-7 of 8 planted problems. The misses were the same across groups (float money, a secret written to the log, no rate limit on login), and nothing the skill adds changed that.
- The skill was read in 6/6 runs and cost 114k tokens per run against 78k (1.5x); on the commit-review task 154k against 79k (about 2x), the heaviest case. The one-liner cost the same as the control. No run changed any files.
- A caveat on the scoring: items are matched by regex, so a differently worded finding can count as a miss.
Dimension scores
| Dimension | Score | Why |
|---|---|---|
| 1_output_quality | 4.45 | Mean score 4.45/5 with the skill (programmatic checks, see per-task table). |
| 2_lift_vs_no_skill | 2.08 | +0.08 points vs 4.37 without the skill (n=2 per task; under ~1 point is noise). |
| 3_trigger | 5.0 | Skill file read in 6/6 treatment runs. |
| 4_instructions_portability | 4 | Written for repos that have standards/spec docs and for agents that can spawn sub-agents; works as plain instructions elsewhere. |
| 5_efficiency | 3 | 1.46x tokens (114k vs 78k per run) and 1.30x time (57s vs 44s). |
| 6_safety_hygiene | 5 | Markdown only; no scripts, no commands. |
Limits of this test
- The skill is designed to farm out the two axes to sub-agents and to read repo docs; a non-interactive Codex run with one small repo does not show whether that helps on a large diff.
- Planted problems in tiny files; quality of explanations, severity ordering and false positives were not scored.
- The oneliner restates the skill's two axes; a team without documented standards would have nothing for the first axis.
- n = 2 per group per task: differences under about 1 point are noise.
- Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.
Security
See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.
Bottom line
When asked to review a commit against written standards and a spec, plain Codex already found every planted deviation, and the skill added tokens (1.5x to 2x) without finding more. It may still help a team that wants every review to follow the same two-axis format, which this test does not measure.
记录
Run log for code-review: Codex CLI, network off, per-run temp HOME/TMPDIR, one row per graded run. Columns: task, arm-run, score (0-5), tokens, seconds, checks passed, skill files the agent opened. t1-orders control-1: score 3.59, tokens 79037, secs 43.8, checks 2/2, opened nothing t1-orders control-2: score 4.06, tokens 79487, secs 50.4, checks 2/2, opened nothing t1-orders oneliner-1: score 3.59, tokens 97051, secs 50.6, checks 2/2, opened nothing t1-orders oneliner-2: score 3.59, tokens 61642, secs 42.5, checks 2/2, opened nothing t1-orders treatment-1: score 4.06, tokens 65668, secs 44.1, checks 2/2, opened code-review t1-orders treatment-2: score 3.59, tokens 123428, secs 56.0, checks 2/2, opened code-review t2-auth control-1: score 4.53, tokens 76436, secs 43.2, checks 2/2, opened nothing t2-auth control-2: score 4.06, tokens 76339, secs 43.5, checks 2/2, opened nothing t2-auth oneliner-1: score 4.53, tokens 61176, secs 49.9, checks 2/2, opened nothing
18 次运行、3 个产物
样例产物
前沿 · 装了 skill：final message（t1-orders，第 1 次，Codex）
Found six problems, most serious first: 1. **High — Missing cancellation authorization** ([line 13](orders_service.py:13)). `current_user` is never checked. Anyone able to call this function can cancel another user’s order by supplying its ID. Verify ownership or administrative permission before updating. 2. **High — SQL injection** ([line 9](orders_service.py:9)). `order_id` is interpolated directly into SQL. A value such as `"0 OR 1=1"` changes the query and can return another order. Use a parameterized query. 3. **High — Credential exposure** ([line 4](orders_service.py:4), [line 48](orders_service.py:48)). A live-looking API key is hardcoded, and the successful charge path prints it to logs. Load credentials from secure configuration and remove the logging. If this key is real, rotate it. 4. **High — Charging always fails silently** ([lines 45–55](orders_service.py:45)). `send_to_gateway()` always raises `NotImplementedError`; `charge()` catches it and returns `None`. No payment can succeed, and callers receive no explicit failure. Implement the gateway and propagate or explicitly report failures. 5. **Medium — Discount thresholds are incorrect** ([line 24](orders_service.py:24)). The documented “or more” thresholds require `>=`. Currently, a total of 100 receives no discount, and 500 receives only 10%, yielding 100 and 450 instead of 90 and 400. 6. **Medium — Export file is never explicitly closed** ([line 39](orders_service.py:39)). Buffered output and resource cleanup depend on garbage collection, including on exceptions. Use a `with open(...)` block to ensure flushing and closing. Static review only; no files changed.