pr (mattpocock/skills)
pr: a pure format skill: control never produces the three-section PR body (1.7/5), the skill and a one-sentence prompt both do (5.0 and 4.95), at no extra tokens
29.0 / 30
Lift over no skill: +3.33
Recommended: the total 29.0/30 is at least 22, output quality 5.0 is at least 3.5, and the lift +3.33 is at least +1.0.
SOTA
PassTested with Codex.
Covers Codex, Claude Code
Codex CLI 0.166 (SOTA tier): 18 graded runs. A pass here also covers Claude Code under our tier rule; it says nothing about the second tier.
Scores
Each dimension is 0 to 5 and the six add up to the total out of 30. 5 is better on every dimension. Cost 5 means cheap and fast. Install 5 means easy.
- Trigger reliability
- 5.0
- Lift over no skill
- 5.0
- Output quality
- 5.0
- Token and time cost
- 5.0
- Install and portability
- 4.0
- Safety
- 5.0
Security review
SafeStatic scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Manual review: Safe to run in a no-network sandbox (no findings). A clean scan does not prove a skill safe.
Install
npx skills@latest add mattpocock/skills --skill pr
Pinned commit: 4ae2317ddb22b046e26455e4f9c9b3aaaf62212d
Tested: 2026-11-02
What we tested
- PR body for a cache — I'm opening a pull request. Write the PR body for the change in change.diff to PR.md. Before/after test output is in tests_before.txt and tests_after.txt.
- PR body for a column drop — Please write up my pull request. Write the PR body for the change in change.diff to PR.md. Before/after test output is in tests_before.txt and tests_after.txt.
- PR body for a copy change — Write a PR description. Write the PR body for the change in change.diff to PR.md. Before/after test output is in tests_before.txt and tests_after.txt.
Pros
- Lifts the mean score from 1.67 to 5.0 over no skill (+3.33).
- The skill was picked up without being named: Skill file read in 6/6 treatment runs.
- No token overhead: 1.0x the tokens of no skill.
- High absolute quality with the skill: 5.0 out of 5.
Cons
- A one-sentence prompt scored 4.95, against 5.0 with the skill (+0.05).
- n=2 per task per arm; differences under about 1 point are noise.
Review
Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below). Three tasks, each 'write the PR body for change.diff to PR.md' with tests_before.txt / tests_after.txt next to it: a cache change (two-way door), a migration that drops a column (one-way door) and a copy change. Control gets only that sentence. The one-liner group gets one added sentence that spells out the whole template. The skill group is not told the format either.
Three-arm comparison
| Arm | Mean score (0-5) | Tokens per run |
|---|---|---|
| Without the skill | 1.67 | 82k |
| One-sentence prompt | 4.95 | 89k |
| With the skill | 5.0 | 82k |
Measured with Codex CLI over 18 runs, scored by rule-based checks with no LLM judge. Total 29.0 / 30, verdict Recommended.
Verdict: Recommended (29.0 / 30). Source: mattpocock/skills, skills/engineering/pr, pinned at commit 4ae2317. Install: npx skills@latest add mattpocock/skills --skill pr. Licence: MIT (repo LICENSE).
What it is
The pr skill (mattpocock/skills): turns a diff into a pull-request body with a fixed three-part shape: Summary (the smallest useful diagram, pseudocode or call tree), Evidence (Before: / After: with the test output) and Merge Danger (Door: one-way or two-way, plus a one-word Blast Radius:), written in the project's glossary words. A single short SKILL.md.
How we tested
Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).
Three tasks, each 'write the PR body for change.diff to PR.md' with tests_before.txt / tests_after.txt next to it: a cache change (two-way door), a migration that drops a column (one-way door) and a copy change. Control gets only that sentence. The one-liner group gets one added sentence that spells out the whole template. The skill group is not told the format either.
- PR body for a cache — I'm opening a pull request. Write the PR body for the change in change.diff to PR.md. Before/after test output is in tests_before.txt and tests_after.txt.
- PR body for a column drop — Please write up my pull request. Write the PR body for the change in change.diff to PR.md. Before/after test output is in tests_before.txt and tests_after.txt.
- PR body for a copy change — Write a PR description. Write the PR body for the change in change.diff to PR.md. Before/after test output is in tests_before.txt and tests_after.txt.
Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.
Results
The one-liner given to the third group: "Write the PR body as three short sections: Summary (the smallest useful diagram, pseudocode or call tree, no preamble), Evidence (Before: and After: with the test output), and Merge Danger (Door: one-way or two-way, and a one-word Blast Radius:); use the glossary's words."
| Task | Control score | One-liner | With skill | Control tokens | One-liner tokens | With-skill tokens | Skill read |
|---|---|---|---|---|---|---|---|
| t1-cache | 1.67 | 5.0 | 5.0 | 82k | 83k | 79k | 2/2 |
| t2-migration | 1.67 | 4.83 | 5.0 | 82k | 91k | 79k | 2/2 |
| t3-copy | 1.67 | 5.0 | 5.0 | 81k | 91k | 87k | 2/2 |
Mean score: 1.67 without vs 5.0 with the skill (delta +3.33). Tokens per run: 82k vs 82k (1.00x); time 37s vs 34s. One-liner group: 4.95 (the skill's gain over the one-liner: +0.05; one-liner's own gain over no skill: +3.28); one-liner tokens 89k per run, so the skill costs 0.92x the one-liner.
What happened
- Control 1.67, one-liner 4.95, skill 5.00 (n=6 each). Control wrote clear, accurate prose PRs but in none of the 6 runs used the three sections, a diagram or pseudocode block, the Before/After labels, a Door or a Blast Radius. It is a format the model cannot guess, so any group that is told the format scores high; the check set measures format compliance, not writing quality.
- The skill triggered without being named (6/6 reads) and the one-sentence prompt carries the same information: the difference between them is 0.05 points (one run missed a template-section regex: the migration one-liner run had 2 of 3 sections matched). Tokens per run were 82k control, 89k one-liner, 82k skill: no overhead.
- Judgement content looks right where it counts: on the migration the skill run called the door one-way and said that reverting code cannot bring the dropped column back, and named the blast radius; the cache and copy tasks were two-way. The control's migration PR also warned to apply 0041 first, but gave no door or radius, so a reviewer would not find it by scanning.
- The skill is tiny (a few lines), so loading it costs nothing; its value is that nobody has to paste the template every time.
Dimension scores
| Dimension | Score | Why |
|---|---|---|
| 1_output_quality | 5.0 | Mean score 5.00/5 with the skill (programmatic checks, see per-task table). |
| 2_lift_vs_no_skill | 5 | +3.33 points vs 1.67 without the skill (n=2 per task; under ~1 point is noise). |
| 3_trigger | 5.0 | Skill file read in 6/6 treatment runs. |
| 4_instructions_portability | 4 | One short SKILL.md with a template, no scripts. |
| 5_efficiency | 5 | 1.00x tokens (82k vs 82k per run) and 0.93x time (34s vs 37s). |
| 6_safety_hygiene | 5 | Prose only; no findings. |
Limits of this test
- Rule-based format checks only (sections present, Before/After, Door value correct, Blast Radius, diagram/pseudocode block); they do not judge whether the diagram is the 'smallest useful' one or the writing is good.
- The control group is bound to fail format checks by construction: the result shows the skill removes the need to restate a house format, not that it makes the model write better PRs.
- Three small fixtures; no real repository or git history.
- n = 2 per group per task: differences under about 1 point are noise.
- Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.
Security
See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.
Bottom line
If you want this PR format, the skill delivers it on the first try for free (82k tokens, same as no skill), and it is equal to pasting the template as a one-liner. Install it for consistency across a team; do not expect better prose.
Transcript
Run log for pr: Codex CLI, network off, per-run temp HOME/TMPDIR, one row per graded run. Columns: task, arm-run, score (0-5), tokens, seconds, checks passed, skill files the agent opened. t1-cache control-1: score 1.67, tokens 82698, secs 35.7, checks 4/9, opened nothing t1-cache control-2: score 1.67, tokens 82765, secs 30.9, checks 4/9, opened nothing t1-cache oneliner-1: score 5.0, tokens 83654, secs 32.5, checks 9/9, opened nothing t1-cache oneliner-2: score 5.0, tokens 84007, secs 38.4, checks 9/9, opened nothing t1-cache treatment-1: score 5.0, tokens 70378, secs 33.8, checks 9/9, opened pr t1-cache treatment-2: score 5.0, tokens 88915, secs 35.0, checks 9/9, opened pr t2-migration control-1: score 1.67, tokens 81748, secs 33.8, checks 4/9, opened nothing t2-migration control-2: score 1.67, tokens 82290, secs 36.5, checks 4/9, opened nothing t2-migration oneliner-1: score 5.0, tokens 100647, secs 37.7, checks 9/9, opened nothing
18 runs, 3 artifacts
Sample output
SOTA · With the skill: PR.md (t1-cache, run 1, Codex)
## Summary Cache free slots by Room and day to avoid repeated database queries. Creating a Reservation invalidates the matching cache entry. ```text free_slots(Room, day) cache hit → return cached free slots cache miss → query Reservations → compute and cache free slots create Reservation insert Reservation → invalidate cache for Room and day ``` ## Evidence - **Before:** `tests/test_availability.py::test_free_slots_not_queried_twice` failed: `db.query called 2 times, expected 1`. Suite: 1 failed, 14 passed in 0.9s. **After:** Supplied test output reports 15 passed in 0.8s. ## Merge Danger **Door:** two-way Reverting restores a database query on each availability request; no schema or stored-data changes. **Blast Radius:** availability Cached free slots affect Members checking Room availability. Invalidation covers Reservation creation in the current process; other mutation paths or processes can leave cached results stale. The cache has no size limit or expiry.