theme-factory (Anthropic)
theme-factory：对照组 3.33、一句话提示 3.33、装 skill 5.0（0–5 分，Codex 实测，n=2/任务/组）。结论：推荐，相对不装 skill +1.67。
27.7 / 30
相对不装 skill 的提升：+1.67
推荐：总分 27.7/30 不低于 22，产出质量 5.0 不低于 3.5，提升 +1.67 不低于 +1.0。
前沿
通过使用 Codex 测试。
覆盖 Codex, Claude Code
Codex CLI 0.166（SOTA 档）：18 次评分运行。按我们的分档规则，这档通过同时覆盖 Claude Code；它不代表第二档的结果。
分数
每一维 0 到 5 分，六维相加是满分 30 的总分。每一维都是 5 更好。成本 5 表示又便宜又快。安装 5 表示容易。
- 触发可靠性
- 5.0
- 相对不装的提升
- 3.7
- 产出质量
- 5.0
- 令牌与时间成本
- 5.0
- 安装与可移植性
- 4.0
- 安全
- 5.0
安全审查
安全静态扫描：0 CRITICAL / 0 HIGH / 0 MEDIUM。人工审查：可在断网沙箱中运行。扫描没有命中不等于绝对安全。
安装
npx skills@latest add anthropics/skills --skill theme-factory
固定提交: ca45fc496a3436251f191354091d557da31c15b5
测试日期: 2026-11-02
我们测了什么
- Apply a named preset theme — Apply the 'Ocean Depths' theme to deck.html. Edit the file in place.
- Apply another preset theme — Apply the 'Tech Innovation' theme to deck.html. Edit the file in place.
- Pick a fitting preset — Pick the preset theme that best fits a children's science fair announcement, state which one you chose, and apply it to deck.html in place.
优点
- 平均分从不装时的 3.33 提到 5.0（+1.67）。
- 不点名也会被触发：Skill file read in 6/6 treatment runs.
- 不增加 token 开销：约为不装 skill 时的 0.66 倍。
- 比把它的规则压成一句话的做法高 +1.67。
- 装 skill 后绝对质量高：5.0 / 5。
缺点
- 测试说明：对照组凭记忆就认识这些命名主题，只有装了 skill 才能从它的清单里挑，这解释了和一句话提示组之间的大差距。
- 每个任务每组 n=2，差距在 1 分以内属于噪声。
评测
在 Codex CLI（SOTA 档）上跑 3 个任务、对照、一句话提示、装 skill 三组，共 18 次运行，由规则脚本打分（没有 LLM 评委），断网沙箱、每次运行独立临时目录。具体做法见下方英文评测原文。
三组对比
| 组别 | 平均分（0–5） | 每次 token |
|---|---|---|
| 不装 skill | 3.33 | 111k |
| 一句话提示 | 3.33 | 126k |
| 装了 skill | 5.0 | 73k |
Codex CLI 实测，18 次运行，规则脚本打分，没有 LLM 评委。总分 27.7 / 30，结论 推荐。
测试披露： 对照组凭记忆就认识这些命名主题，只有装了 skill 才能从它的清单里挑，这解释了和一句话提示组之间的大差距。
下面是评测原文，英文。
Verdict: Recommended (27.7 / 30). Source: anthropics/skills, skills/theme-factory, pinned at commit ca45fc4. Install: npx skills@latest add anthropics/skills --skill theme-factory. Licence: Apache-2.0 / Anthropic terms (see LICENSE.txt in skill).
What it is
The theme-factory skill from anthropics/skills: 10 preset colour-and-font themes (Ocean Depths, Sunset Boulevard, Forest Canopy, Tech Innovation and so on), each a short markdown file with hex palette and font pairing, plus a theme showcase PDF, for styling slides, docs or pages. SKILL.md tells the agent to show the showcase, let the user choose, then apply the theme's colours and fonts.
How we tested
Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).
Three tasks on the same plain HTML slide deck (deck.html): apply the named theme 'Ocean Depths', apply the named theme 'Tech Innovation', and 'pick the preset theme that best fits a children's science fair, state which one, and apply it'. The one-liner group adds one sentence telling the agent to apply a preset theme's palette and font pairing and say which one; it contains no theme data.
- Apply a named preset theme — Apply the 'Ocean Depths' theme to deck.html. Edit the file in place.
- Apply another preset theme — Apply the 'Tech Innovation' theme to deck.html. Edit the file in place.
- Pick a fitting preset — Pick the preset theme that best fits a children's science fair announcement, state which one you chose, and apply it to deck.html in place.
Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.
Results
The one-liner given to the third group: "Apply the chosen preset theme's color palette (background, text, accent) and its header/body font pairing consistently to every element, and say which theme you applied."
| Task | Control score | One-liner | With skill | Control tokens | One-liner tokens | With-skill tokens | Skill read |
|---|---|---|---|---|---|---|---|
| t1-ocean | 5.0 | 5.0 | 5.0 | 70k | 103k | 69k | 2/2 |
| t2-tech | 5.0 | 5.0 | 5.0 | 73k | 102k | 79k | 2/2 |
| t3-choose | 0.0 | 0.0 | 5.0 | 187k | 171k | 71k | 2/2 |
Mean score: 3.33 without vs 5.0 with the skill (delta +1.67). Tokens per run: 111k vs 73k (0.66x); time 53s vs 44s. One-liner group: 3.33 (the skill's gain over the one-liner: +1.67; one-liner's own gain over no skill: +0.0); one-liner tokens 126k per run, so the skill costs 0.58x the one-liner.
What happened
- On the two named-theme tasks all three groups score 5.0: control (no skill, no theme files in the folder) applied the exact Ocean Depths hexes (#1a2332, #2d8b8b, #a8dadc, #f1faee) and Tech Innovation palette from memory. The preset themes are public enough that the model already knows them; the skill adds nothing on a request that names the theme.
- The skill matters where the theme list itself is needed: on the 'pick the best preset' task control scored 0/2 (one run said it chose a preset called 'Sky', which is not one of the ten; the other spent 247k tokens), the one-liner scored 0/2 (one run replied that no preset list was available in the workspace and asked the user for it), and the skill scored 2/2 (picked 'Sunset Boulevard' and applied its palette).
- Mean 3.33 control, 3.33 one-liner, 5.00 skill. The one-liner cannot substitute because the value is the data in the skill's theme files, not an instruction.
- Tokens: 73k per run for the skill against 111k control and 126k one-liner (0.66x and 0.58x), because on the open task control and one-liner search and guess while the skill reads a small theme file.
- Treatment runs copy the skill's themes folder and showcase PDF into the project (14 new files per run on average), which pollutes the working folder.
Dimension scores
| Dimension | Score | Why |
|---|---|---|
| 1_output_quality | 5.0 | Mean score 5.00/5 with the skill (programmatic checks, see per-task table). |
| 2_lift_vs_no_skill | 3.67 | +1.67 points vs 3.33 without the skill (n=2 per task; under ~1 point is noise). |
| 3_trigger | 5.0 | Skill file read in 6/6 treatment runs. |
| 4_instructions_portability | 4 | SKILL.md of 43 lines, 10 theme files, a PDF showcase; no scripts. |
| 5_efficiency | 5 | 0.66x tokens (73k vs 111k per run) and 0.83x time (44s vs 53s). |
| 6_safety_hygiene | 5 | Prose, data and a PDF only. |
Limits of this test
- Rule-based palette checks (preset hex colours used, theme named); not a judgement of how good the restyled deck looks.
- Task 3's check counts a run as correct only if it uses one of the ten themes' hexes and names one, so it favours any group that can see the list.
- n=2 per cell; the skill's showcase step (showing the PDF to the user and waiting for a choice) is interactive and cannot run in a headless test.
- n = 2 per group per task: differences under about 1 point are noise.
- Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.
Security
See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.
Bottom line
Useful when you want one of the ten specific presets chosen or listed: it removes guessing and saves tokens. If you just name a preset, plain Codex already does the same, so the skill is not needed for that.
记录
Run log for theme-factory: Codex CLI, network off, per-run temp HOME/TMPDIR, one row per graded run. Columns: task, arm-run, score (0-5), tokens, seconds, checks passed, skill files the agent opened. t1-ocean control-1: score 5.0, tokens 74123, secs 43.0, checks 2/2, opened nothing t1-ocean control-2: score 5.0, tokens 66741, secs 48.1, checks 2/2, opened nothing t1-ocean oneliner-1: score 5.0, tokens 112572, secs 60.5, checks 2/2, opened nothing t1-ocean oneliner-2: score 5.0, tokens 94515, secs 51.3, checks 2/2, opened nothing t1-ocean treatment-1: score 5.0, tokens 69081, secs 47.0, checks 2/2, opened theme-factory t1-ocean treatment-2: score 5.0, tokens 70216, secs 33.6, checks 2/2, opened theme-factory t2-tech control-1: score 5.0, tokens 77803, secs 56.7, checks 1/1, opened nothing t2-tech control-2: score 5.0, tokens 69182, secs 57.9, checks 1/1, opened nothing t2-tech oneliner-1: score 5.0, tokens 119920, secs 57.6, checks 1/1, opened nothing
18 次运行、3 个产物
样例产物
前沿 · 装了 skill：deck.html（t1-ocean，第 1 次，Codex）
<!DOCTYPE html> <html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1"><title>Q3 Update</title> <style> :root { color-scheme: dark; --navy: #1a2332; --teal: #2d8b8b; --seafoam: #a8dadc; --cream: #f1faee; } * { box-sizing: border-box; } body { margin: 0; background: var(--navy); color: var(--cream); font-family: "DejaVu Sans", sans-serif; font-size: clamp(18px, 2.2vw, 28px); line-height: 1.6; } section { max-width: 1200px; min-height: 100vh; margin: 0 auto; padding: clamp(32px, 7vw, 96px); display: flex; flex-direction: column; justify-content: center; border-top: 6px solid var(--teal); } h1, h2 { font-family: "DejaVu Sans", sans-serif; font-weight: 700; line-height: 1.15; margin: 0 0 0.6em; } h1 { font-size: clamp(48px, 7vw, 88px); } h2 { color: var(--seafoam); font-size: clamp(36px, 5vw, 60px); } p { margin: 0 0 1em; } section:first-child p { color: var(--seafoam); } ul { margin: 0; padding-left: 1.3em; } li { padding: 0.25em 0; } li::marker { color: var(--seafoam); } table { width: 100%; border-collapse: collapse; border: 2px solid var(--teal); margin-top: 1em; } th, td { padding: 0.65em 1em; text-align: left; border: 1px solid var(--teal); } th { background: var(--seafoam); color: var(--navy); font-weight: 700; } @media print { section { min-height: 0; height: 100vh; break-after: page; } section:last-child { break-after: auto; } body { print-color-adjust: exact; -webkit-print-color-adjust: exact; } } </style> </head> <body> <section><h1>Q3 Update</h1><p>Prepared for the leadership team</p></section> <section><h2>Highlights</h2><ul><li>Revenue up 12%</li><li>Two launches shipped</li><li>Support backlog cleared</li></ul></section> <section><h2>Next quarter</h2><p>Focus: reliability, onboarding, pricing review.</p><table border="1"><tr><th>Metric</th><th>Q3</th></tr><tr><td>Revenue</td><td>$1.2M</td></tr><tr><td>Churn</td><td>2.1%</td></tr></table></section> </body></html>