improve-codebase-architecture (mattpocock/skills)

评测

在 Codex CLI（SOTA 档）上跑 3 个任务、对照、一句话提示、装 skill 三组，共 18 次运行，由规则脚本打分（没有 LLM 评委），断网沙箱、每次运行独立临时目录。具体做法见下方英文评测原文。

三组对比

Codex CLI 实测，18 次运行，规则脚本打分，没有 LLM 评委。总分 23.0 / 30，结论 视情况而定。

下面是评测原文，英文。

Verdict: Situational (23.0 / 30). Source: mattpocock/skills, skills/engineering/improve-codebase-architecture, skills/engineering/codebase-design, pinned at commit 4ae2317. Install: npx skills@latest add mattpocock/skills --skill improve-codebase-architecture. Licence: MIT (repo LICENSE).

What it is

The improve-codebase-architecture skill from mattpocock/skills. It looks for 'deepening opportunities' in a codebase: shallow modules whose interface is as complex as their implementation, pass-through layers, hypothetical single-adapter seams, duplicated logic, behaviour only tested through helpers. It writes an HTML report with a before/after per candidate and a top recommendation. It leans on a sibling skill, codebase-design, for the vocabulary. Markdown only, no scripts.

How we tested

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).

Three tasks on a small Python shop backend with planted friction (shallow validators/money helpers, an inventory service that only forwards to its repo, a single payment adapter, discount logic duplicated in cart and orders, tests that only cover helpers): a whole-codebase architecture review written as report.html, the same limited to payment and inventory, and 'why is place_order painful to test' answered in chat. The skill group's prompts start with $improve-codebase-architecture (explicit invocation); control and one-sentence prompts do not name it.

1. Whole-repo architecture scan — Review this small shop backend for architecture improvements: where would restructuring make the code easier to change and easier to test? Write your findings as a self-contained HTML report named report.html in this folder (do not open a browser or start a se…
2. Payments and inventory only — Look only at the payment and inventory parts of this backend and tell me which restructurings would pay off. Write your findings as a self-contained HTML report named report.html in this folder (do not open a browser or start a server) and do not change any co…
3. Why is place_order hard to test? — It feels painful to write a proper test for place_order in this backend. Why, and what restructuring would fix that? Do not change any files; answer in your final message.

Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.

Results

The one-liner given to the third group: "Look for deepening opportunities: shallow modules whose interface is as complex as their implementation (apply the deletion test: would deleting it concentrate complexity or just move it?), pass-through layers, single-adapter seams that are only hypothetical, logic duplicated across modules, and logic that is only tested through helpers instead of the real interface; present each candidate with files, problem, solution, benefits, a before/after, and name a top recommendation."

Mean score: 3.2 without vs 4.12 with the skill (delta +0.92). Tokens per run: 115k vs 208k (1.80x); time 112s vs 135s. One-liner group: 3.91 (the skill's gain over the one-liner: +0.2; one-liner's own gain over no skill: +0.72); one-liner tokens 120k per run, so the skill costs 1.73x the one-liner.

What happened

• Whole-codebase review is where it shows: the skill and the one-sentence version found 6/6 planted points with architecture vocabulary, before/after and a top recommendation in all four runs; plain Codex found 2/6 in one run and 5 of 6 checks in the other, and neither run had a clear 'top recommendation'.
• The focused review (payment and inventory only) is hard for everyone: the best group found one of the two planted points on average. Control 2/4 checks, one-liner 3/4, skill 3/4. The skill did not do better than the sentence.
• On 'why is place_order hard to test' all three groups found the planted causes (hidden PaymentAdapter, shared module-level stock, duplicated discount logic, tests that stop at helpers). The skill's answer used the skill's vocabulary ('interface', 'locality'); the control's was equally correct in plain words.
• Mean score 3.20 control, 3.91 one-sentence, 4.12 skill. The skill adds 0.21 over the sentence at 1.7x the tokens (208k vs 120k per run) because it also reads the sibling skill; the one-sentence group costs the same as the control (115k).
• Because the skill installs two skill folders, the treatment runs created extra files in the project's .agents folder; they were excluded from the 'no code files changed' check (a first pass wrongly failed them and a Windows-specific bug in the check was fixed before the final numbers).

Dimension scores

Limits of this test

• One small Python fixture with textbook planted problems; whether the report is useful on a large real repo is not measured.
• Report quality, false positives and the HTML styling (it loads Tailwind and Mermaid from CDNs when opened in a browser) were not scored; the sub-agent behaviour the skill describes was not exercised.
• The one-sentence prompt restates the skill's own checklist, so it is an upper bound for a short prompt.
• n = 2 per group per task: differences under about 1 point are noise.
• Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.

Security

See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.

Bottom line

Compared with doing nothing the skill clearly improves a whole-codebase review, mostly by forcing the report structure and vocabulary, and costs 1.8x tokens. A single sentence that lists the same smells gets most of the gain, so it is worth installing mainly if you want the identical report format every time.