pr (mattpocock/skills)
pr：对照组 1.67、一句话提示 4.95、装 skill 5.0（0–5 分，Codex 实测，n=2/任务/组）。结论：推荐，相对不装 skill +3.33。
29.0 / 30
相对不装 skill 的提升：+3.33
推荐：总分 29.0/30 不低于 22，产出质量 5.0 不低于 3.5，提升 +3.33 不低于 +1.0。
前沿
通过使用 Codex 测试。
覆盖 Codex, Claude Code
Codex CLI 0.166（SOTA 档）：18 次评分运行。按我们的分档规则，这档通过同时覆盖 Claude Code；它不代表第二档的结果。
分数
每一维 0 到 5 分，六维相加是满分 30 的总分。每一维都是 5 更好。成本 5 表示又便宜又快。安装 5 表示容易。
- 触发可靠性
- 5.0
- 相对不装的提升
- 5.0
- 产出质量
- 5.0
- 令牌与时间成本
- 5.0
- 安装与可移植性
- 4.0
- 安全
- 5.0
安全审查
安全静态扫描：0 CRITICAL / 0 HIGH / 0 MEDIUM。人工审查：可在断网沙箱中运行。扫描没有命中不等于绝对安全。
安装
npx skills@latest add mattpocock/skills --skill pr
固定提交: 4ae2317ddb22b046e26455e4f9c9b3aaaf62212d
测试日期: 2026-11-02
我们测了什么
- PR body for a cache — I'm opening a pull request. Write the PR body for the change in change.diff to PR.md. Before/after test output is in tests_before.txt and tests_after.txt.
- PR body for a column drop — Please write up my pull request. Write the PR body for the change in change.diff to PR.md. Before/after test output is in tests_before.txt and tests_after.txt.
- PR body for a copy change — Write a PR description. Write the PR body for the change in change.diff to PR.md. Before/after test output is in tests_before.txt and tests_after.txt.
优点
- 平均分从不装时的 1.67 提到 5.0（+3.33）。
- 不点名也会被触发：Skill file read in 6/6 treatment runs.
- 不增加 token 开销：约为不装 skill 时的 1.0 倍。
- 装 skill 后绝对质量高：5.0 / 5。
缺点
- 一句话提示得 4.95，装 skill 得 5.0（差 +0.05）。
- 每个任务每组 n=2，差距在 1 分以内属于噪声。
评测
在 Codex CLI（SOTA 档）上跑 3 个任务、对照、一句话提示、装 skill 三组，共 18 次运行，由规则脚本打分（没有 LLM 评委），断网沙箱、每次运行独立临时目录。具体做法见下方英文评测原文。
三组对比
| 组别 | 平均分（0–5） | 每次 token |
|---|---|---|
| 不装 skill | 1.67 | 82k |
| 一句话提示 | 4.95 | 89k |
| 装了 skill | 5.0 | 82k |
Codex CLI 实测，18 次运行，规则脚本打分，没有 LLM 评委。总分 29.0 / 30，结论 推荐。
下面是评测原文，英文。
Verdict: Recommended (29.0 / 30). Source: mattpocock/skills, skills/engineering/pr, pinned at commit 4ae2317. Install: npx skills@latest add mattpocock/skills --skill pr. Licence: MIT (repo LICENSE).
What it is
The pr skill (mattpocock/skills): turns a diff into a pull-request body with a fixed three-part shape: Summary (the smallest useful diagram, pseudocode or call tree), Evidence (Before: / After: with the test output) and Merge Danger (Door: one-way or two-way, plus a one-word Blast Radius:), written in the project's glossary words. A single short SKILL.md.
How we tested
Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).
Three tasks, each 'write the PR body for change.diff to PR.md' with tests_before.txt / tests_after.txt next to it: a cache change (two-way door), a migration that drops a column (one-way door) and a copy change. Control gets only that sentence. The one-liner group gets one added sentence that spells out the whole template. The skill group is not told the format either.
- PR body for a cache — I'm opening a pull request. Write the PR body for the change in change.diff to PR.md. Before/after test output is in tests_before.txt and tests_after.txt.
- PR body for a column drop — Please write up my pull request. Write the PR body for the change in change.diff to PR.md. Before/after test output is in tests_before.txt and tests_after.txt.
- PR body for a copy change — Write a PR description. Write the PR body for the change in change.diff to PR.md. Before/after test output is in tests_before.txt and tests_after.txt.
Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.
Results
The one-liner given to the third group: "Write the PR body as three short sections: Summary (the smallest useful diagram, pseudocode or call tree, no preamble), Evidence (Before: and After: with the test output), and Merge Danger (Door: one-way or two-way, and a one-word Blast Radius:); use the glossary's words."
| Task | Control score | One-liner | With skill | Control tokens | One-liner tokens | With-skill tokens | Skill read |
|---|---|---|---|---|---|---|---|
| t1-cache | 1.67 | 5.0 | 5.0 | 82k | 83k | 79k | 2/2 |
| t2-migration | 1.67 | 4.83 | 5.0 | 82k | 91k | 79k | 2/2 |
| t3-copy | 1.67 | 5.0 | 5.0 | 81k | 91k | 87k | 2/2 |
Mean score: 1.67 without vs 5.0 with the skill (delta +3.33). Tokens per run: 82k vs 82k (1.00x); time 37s vs 34s. One-liner group: 4.95 (the skill's gain over the one-liner: +0.05; one-liner's own gain over no skill: +3.28); one-liner tokens 89k per run, so the skill costs 0.92x the one-liner.
What happened
- Control 1.67, one-liner 4.95, skill 5.00 (n=6 each). Control wrote clear, accurate prose PRs but in none of the 6 runs used the three sections, a diagram or pseudocode block, the Before/After labels, a Door or a Blast Radius. It is a format the model cannot guess, so any group that is told the format scores high; the check set measures format compliance, not writing quality.
- The skill triggered without being named (6/6 reads) and the one-sentence prompt carries the same information: the difference between them is 0.05 points (one run missed a template-section regex: the migration one-liner run had 2 of 3 sections matched). Tokens per run were 82k control, 89k one-liner, 82k skill: no overhead.
- Judgement content looks right where it counts: on the migration the skill run called the door one-way and said that reverting code cannot bring the dropped column back, and named the blast radius; the cache and copy tasks were two-way. The control's migration PR also warned to apply 0041 first, but gave no door or radius, so a reviewer would not find it by scanning.
- The skill is tiny (a few lines), so loading it costs nothing; its value is that nobody has to paste the template every time.
Dimension scores
| Dimension | Score | Why |
|---|---|---|
| 1_output_quality | 5.0 | Mean score 5.00/5 with the skill (programmatic checks, see per-task table). |
| 2_lift_vs_no_skill | 5 | +3.33 points vs 1.67 without the skill (n=2 per task; under ~1 point is noise). |
| 3_trigger | 5.0 | Skill file read in 6/6 treatment runs. |
| 4_instructions_portability | 4 | One short SKILL.md with a template, no scripts. |
| 5_efficiency | 5 | 1.00x tokens (82k vs 82k per run) and 0.93x time (34s vs 37s). |
| 6_safety_hygiene | 5 | Prose only; no findings. |
Limits of this test
- Rule-based format checks only (sections present, Before/After, Door value correct, Blast Radius, diagram/pseudocode block); they do not judge whether the diagram is the 'smallest useful' one or the writing is good.
- The control group is bound to fail format checks by construction: the result shows the skill removes the need to restate a house format, not that it makes the model write better PRs.
- Three small fixtures; no real repository or git history.
- n = 2 per group per task: differences under about 1 point are noise.
- Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.
Security
See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.
Bottom line
If you want this PR format, the skill delivers it on the first try for free (82k tokens, same as no skill), and it is equal to pasting the template as a one-liner. Install it for consistency across a team; do not expect better prose.
记录
Run log for pr: Codex CLI, network off, per-run temp HOME/TMPDIR, one row per graded run. Columns: task, arm-run, score (0-5), tokens, seconds, checks passed, skill files the agent opened. t1-cache control-1: score 1.67, tokens 82698, secs 35.7, checks 4/9, opened nothing t1-cache control-2: score 1.67, tokens 82765, secs 30.9, checks 4/9, opened nothing t1-cache oneliner-1: score 5.0, tokens 83654, secs 32.5, checks 9/9, opened nothing t1-cache oneliner-2: score 5.0, tokens 84007, secs 38.4, checks 9/9, opened nothing t1-cache treatment-1: score 5.0, tokens 70378, secs 33.8, checks 9/9, opened pr t1-cache treatment-2: score 5.0, tokens 88915, secs 35.0, checks 9/9, opened pr t2-migration control-1: score 1.67, tokens 81748, secs 33.8, checks 4/9, opened nothing t2-migration control-2: score 1.67, tokens 82290, secs 36.5, checks 4/9, opened nothing t2-migration oneliner-1: score 5.0, tokens 100647, secs 37.7, checks 9/9, opened nothing
18 次运行、3 个产物
样例产物
前沿 · 装了 skill：PR.md（t1-cache，第 1 次，Codex）
## Summary Cache free slots by Room and day to avoid repeated database queries. Creating a Reservation invalidates the matching cache entry. ```text free_slots(Room, day) cache hit → return cached free slots cache miss → query Reservations → compute and cache free slots create Reservation insert Reservation → invalidate cache for Room and day ``` ## Evidence - **Before:** `tests/test_availability.py::test_free_slots_not_queried_twice` failed: `db.query called 2 times, expected 1`. Suite: 1 failed, 14 passed in 0.9s. **After:** Supplied test output reports 15 passed in 0.8s. ## Merge Danger **Door:** two-way Reverting restores a database query on each availability request; no schema or stored-data changes. **Blast Radius:** availability Cached free slots affect Members checking Room availability. Invalidation covers Reservation creation in the current process; other mutation paths or processes can leave cached results stale. The cache has no size limit or expiry.