prototype (mattpocock/skills)

Review

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a Mac test machine, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below). Three tasks: model a subscription lifecycle with payment failures (state question), compare options for a static pricing section (look question, real plan content must stay), and let the user try discount-stacking rules before coding (logic question, save into prototype/). Rule checks include self-contained HTML, five states shown, buttons per event, guided scenarios, marked as prototype, no test files, and for the pricing task at least three variants, a ?variant= URL parameter and a fixed floating switcher.

Three-arm comparison

Measured with Codex CLI over 18 runs, scored by rule-based checks with no LLM judge. Total 27.2 / 30, verdict Recommended.

Verdict: Recommended (27.2 / 30). Source: mattpocock/skills, skills/engineering/prototype, pinned at commit 4ae2317. Install: npx skills@latest add mattpocock/skills --skill prototype. Licence: MIT (repo LICENSE).

What it is

The prototype skill from mattpocock/skills: builds a throwaway prototype that answers a design question. Logic or state questions get a single HTML file with a state panel, free-play buttons and guided walkthroughs; look questions get several radically different variants on one route, switched with a URL parameter and a floating bottom bar. Rules: clearly marked as throwaway, trivial to run, no persistence, no polish, no tests. SKILL.md plus LOGIC.md and UI.md, about 2,600 words in total; Markdown only.

How we tested

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a Mac test machine, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).

Three tasks: model a subscription lifecycle with payment failures (state question), compare options for a static pricing section (look question, real plan content must stay), and let the user try discount-stacking rules before coding (logic question, save into prototype/). Rule checks include self-contained HTML, five states shown, buttons per event, guided scenarios, marked as prototype, no test files, and for the pricing task at least three variants, a ?variant= URL parameter and a fixed floating switcher.

1. Prototype a subscription state model — I can't tell whether our subscription lifecycle makes sense when payments fail. States: trial, active, past_due, paused, canceled. Events: trial ends, payment succeeds, payment fails, user pauses, user resumes, user cancels, user reactivates. Rules: two failed…
2. Prototype pricing page variants — On pricing.html of this small static site (three plans) I'm unsure how the plans should be presented. Make a few different design options of that section that I can compare in the browser; the real pricing content must stay. No build step.
3. Prototype discount stacking rules — Does stacking of discounts feel right? Rules so far: coupon SAVE10 is 10% off, coupon HALF is 50% off, orders over $50 ship free, loyalty members get an extra 5% off. I can't decide what order they apply in or which may stack. I want to try cases and watch wha…

Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.

Results

The one-liner given to the third group: "Build a throwaway prototype that answers one stated question: for logic or state questions a single self-contained HTML file with a visible statement of the question, a readable current-state panel re-rendered after every click, one button per action, and guided walkthrough scenarios (happy path, edge case, an illegal move); for look questions three radically different variants on one page switched by a ?variant= URL parameter and a floating bottom bar. No tests, no polish."

Mean score: 3.71 without vs 4.95 with the skill (delta +1.23). Tokens per run: 133k vs 136k (1.02x); time 186s vs 149s. One-liner group: 4.72 (the skill's gain over the one-liner: +0.23; one-liner's own gain over no skill: +1.01); one-liner tokens 89k per run, so the skill costs 1.52x the one-liner.

What happened

• Mean score 3.71 for control, 4.72 for the one-sentence version, 4.95 for the skill. The skill is +1.24 over control and +0.23 over the one-liner at 136k tokens per run against 133k (1.02x) for control and 89k for the one-liner (1.5x).
• The look task shows the effect: the skill produced three variants with a floating bottom switcher and URL-switchable variants in both runs (5.0 and 5.0 by our checks); control scored 4.0 and 1.0: no floating bar in either run, three layouts with a switch and a URL parameter in one run and not in the other, and in one run a plan price was missing from the page.
• On the two logic tasks control already built interactive HTML (4.4-4.8 on the subscription model, 4.0 on discount stacking); the skill adds the walkthrough scenarios and the prototype label: scenarios appear in 4 of 4 skill runs against 2 of 4 control runs.
• The one-sentence version did as well on the pricing task; it lost a little on the logic tasks by not always marking the file as a prototype (2 of 4 runs).

Dimension scores

Limits of this test

• Rule-based checks of features (variants, floating bar, scenarios); the quality of the designs and whether the prototype answers the question were not judged by a person.
• The scoring was fixed twice during analysis: variants may live in separate CSS/JS files, and a layout switch counts as a variant; the first pass undercounted those, so earlier numbers are not used.
• The skill's advice to commit the prototype to a throwaway branch and leave a pointer on an issue cannot be tested in a scratch folder.
• n = 2 per group per task: differences under about 1 point are noise.
• Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.

Security

See security.md. Scan: 1 CRITICAL / 1 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.

Bottom line

The skill clearly makes Codex build the kind of prototype it describes (variants switchable by URL, floating bar, guided scenarios) where a plain request does not, at no extra token cost. A one-sentence summary gets most of the same behaviour, but the skill adds the logic-walkthrough and prototype-labelling details.