codebase-design (mattpocock/skills)

评测

在 Codex CLI（SOTA 档）上跑 3 个任务、对照、一句话提示、装 skill 三组，共 18 次运行，由规则脚本打分（没有 LLM 评委），断网沙箱、每次运行独立临时目录。具体做法见下方英文评测原文。

三组对比

Codex CLI 实测，18 次运行，规则脚本打分，没有 LLM 评委。总分 23.6 / 30，结论 视情况而定。

下面是评测原文，英文。

Verdict: Situational (23.6 / 30). Source: mattpocock/skills, skills/engineering/codebase-design, pinned at commit 4ae2317. Install: npx skills@latest add mattpocock/skills --skill codebase-design. Licence: MIT (repo LICENSE).

What it is

The codebase-design skill (mattpocock/skills): a way to review architecture by module depth: deep modules with a small interface hiding a lot of behaviour; flag shallow pass-through modules, single-adapter seams that are only hypothetical, duplicated logic and behaviour tested only through helpers; say where a seam belongs so behaviour is tested through the interface.

How we tested

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).

A small shop backend with six planted friction points (shallow validators and money wrappers, a pass-through inventory service, a single payment adapter, duplicated logic, and others). Tasks: write an HTML report of architecture improvements, a report limited to payment and inventory, and 'why is place_order hard to test and how would you fix it' (answer in chat). Prompts do not name any design terms. The one-liner states the depth idea and the things to flag.

1. Whole-repo architecture scan — Review this small shop backend for architecture improvements: where would restructuring make the code easier to change and easier to test? Write your findings as a self-contained HTML report named report.html in this folder (do not open a browser or start a se…
2. Payments and inventory only — Look only at the payment and inventory parts of this backend and tell me which restructurings would pay off. Write your findings as a self-contained HTML report named report.html in this folder (do not open a browser or start a server) and do not change any co…
3. Why is place_order hard to test? — It feels painful to write a proper test for place_order in this backend. Why, and what restructuring would fix that? Do not change any files; answer in your final message.

Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.

Results

The one-liner given to the third group: "Judge modules by depth: a deep module has a small interface hiding a lot of behaviour; flag shallow pass-through modules, hypothetical single-adapter seams, duplicated logic and behaviour that is only tested through helpers, and say where a seam belongs so behaviour is tested through the interface."

Mean score: 3.21 without vs 3.92 with the skill (delta +0.71). Tokens per run: 119k vs 133k (1.12x); time 154s vs 158s. One-liner group: 4.54 (the skill's gain over the one-liner: -0.63; one-liner's own gain over no skill: +1.34); one-liner tokens 114k per run, so the skill costs 1.17x the one-liner.

What happened

• Control 3.21, skill 3.92, one-liner 4.54 (n=6 each). Without guidance, the HTML reports used almost none of the architecture vocabulary (0-2 uses of deep/shallow/seam/interface, where the check wants 6) in 4 of 4 report runs. The skill reports used it in all 4 runs, as did the one-liner.
• Whole-backend scan: skill 4.24 and 3.41, one-liner 3.41 and 3.86, control 2.88 and 2.2. Both guided arms found 3-4 of the 6 planted points; the shallow validators wrappers and the single payment adapter were missed in several runs by every arm.
• Focused payment and inventory task: skill 3.75 in both runs, control 2.5 in both, one-liner 5.0 in both. All skill and control runs found only one of the two points (the hypothetical single payment adapter was missed); the one-liner found both.
• Why is place_order hard to test: all arms are close (control 4.17 and 5.0, skill 4.17 and 4.17, one-liner 5.0 and 5.0); the skill runs missed one cause (helper-only tests) in one run and pass-through layers in another.
• Tokens: skill 133k, one-liner 114k, control 119k per run (1.12x control). The skill was read in 6/6 treatment runs.

Dimension scores

Limits of this test

• Rule-based recall checks on the report text (keywords near file names) and a vocabulary count; whether the proposed restructurings are sound was not judged and the reports were not opened in a browser.
• The vocabulary check rewards the skill's own terms, which the one-liner also names, and favours guided arms.
• 'Before/after' check shows up for 2 runs failing in the scan task only; not a differentiator.
• n=2 per cell; one small codebase.
• n = 2 per group per task: differences under about 1 point are noise.
• Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.

Security

See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.

Bottom line

The skill clearly gives the review the module-depth framing that a bare prompt lacks, with a small score gain over control, but the same idea in one sentence finds more of the planted problems for fewer tokens, so the gain is mostly stylistic.