caveman-review (JuliusBrussee/caveman)
caveman-review：对照组 3.26、一句话提示 4.83、装 skill 4.16（0–5 分，Codex 实测，n=2/任务/组）。结论：视情况而定，相对不装 skill +0.90。
25.1 / 30
相对不装 skill 的提升：+0.90
视情况而定：提升 +0.90 低于 +1.0。推荐要求总分 22/30、产出质量 3.5、提升 +1.0。
前沿
通过使用 Codex 测试。
覆盖 Codex, Claude Code
Codex CLI 0.166（SOTA 档）：18 次评分运行。按我们的分档规则，这档通过同时覆盖 Claude Code；它不代表第二档的结果。
分数
每一维 0 到 5 分，六维相加是满分 30 的总分。每一维都是 5 更好。成本 5 表示又便宜又快。安装 5 表示容易。
- 触发可靠性
- 5.0
- 相对不装的提升
- 2.9
- 产出质量
- 4.2
- 令牌与时间成本
- 5.0
- 安装与可移植性
- 4.0
- 安全
- 4.0
安全审查
安全静态扫描：0 CRITICAL / 0 HIGH / 0 MEDIUM。人工审查：可在断网沙箱中运行。扫描没有命中不等于绝对安全。
安装
npx skills@latest add JuliusBrussee/caveman --skill caveman-review
固定提交: 9b68f7ed12b05e7efaff18e65c5a3fc2849fd0a4
测试日期: 2026-11-02
我们测了什么
- Review a small service module — Review orders_service.py before I merge it. List the problems you find, most serious first. Do not change any files.
- Review an auth helper — Review auth_handlers.py: I just wrote it for our login flow. List the problems, most serious first. Do not change any files.
- Review the latest commit against written standards and spec — Review my latest commit (the changes since HEAD~1) against CODING_STANDARDS.md and SPEC.md. List every deviation, most serious first. Do not change any files.
优点
- 平均分从不装时的 3.26 提到 4.16（+0.90）。
- 不点名也会被触发：Skill file read in 6/6 treatment runs.
- 不增加 token 开销：约为不装 skill 时的 1.02 倍。
缺点
- 一句话提示得 4.83，装 skill 得 4.16（差 -0.67）。
- 每个任务每组 n=2，差距在 1 分以内属于噪声。
评测
在 Codex CLI（SOTA 档）上跑 3 个任务、对照、一句话提示、装 skill 三组，共 18 次运行，由规则脚本打分（没有 LLM 评委），断网沙箱、每次运行独立临时目录。具体做法见下方英文评测原文。
三组对比
| 组别 | 平均分（0–5） | 每次 token |
|---|---|---|
| 不装 skill | 3.26 | 69k |
| 一句话提示 | 4.83 | 60k |
| 装了 skill | 4.16 | 70k |
Codex CLI 实测，18 次运行，规则脚本打分，没有 LLM 评委。总分 25.1 / 30，结论 视情况而定。
下面是评测原文，英文。
Verdict: Situational (25.1 / 30). Source: JuliusBrussee/caveman, skills/caveman-review, pinned at commit 9b68f7e. Install: npx skills@latest add JuliusBrussee/caveman --skill caveman-review. Licence: MIT (repo LICENSE).
What it is
The caveman-review skill (JuliusBrussee/caveman): a terse code-review style: one finding per line as L<line>: <problem>. <fix>. with an optional severity prefix (bug, risk, nit, q), exact symbol names in backticks, a concrete fix, and no hedging, praise or restating what the code does.
How we tested
Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).
Three review tasks on small Python files with 8 planted issues each: an orders service (SQL injection, missing authorization, hard-coded key logged, float money and others), an auth handler file, and a review of the latest commit against CODING_STANDARDS.md and SPEC.md. Prompts ask for a review, most serious first, no changes. The one-liner states the line format and the no-hedging rule.
- Review a small service module — Review orders_service.py before I merge it. List the problems you find, most serious first. Do not change any files.
- Review an auth helper — Review auth_handlers.py: I just wrote it for our login flow. List the problems, most serious first. Do not change any files.
- Review the latest commit against written standards and spec — Review my latest commit (the changes since HEAD~1) against CODING_STANDARDS.md and SPEC.md. List every deviation, most serious first. Do not change any files.
Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.
Results
The one-liner given to the third group: "Write each finding as one line L<line>: <problem>. <fix>. with an optional severity prefix (bug, risk, nit, q), exact symbol names in backticks and a concrete fix; no hedging, praise or throat-clearing, and do not restate what the code does."
| Task | Control score | One-liner | With skill | Control tokens | One-liner tokens | With-skill tokens | Skill read |
|---|---|---|---|---|---|---|---|
| t1-orders | 3.06 | 4.66 | 4.15 | 74k | 64k | 61k | 2/2 |
| t2-auth | 3.19 | 4.92 | 3.31 | 73k | 64k | 88k | 2/2 |
| t3-diff | 3.53 | 4.92 | 5.0 | 59k | 50k | 60k | 2/2 |
Mean score: 3.26 without vs 4.16 with the skill (delta +0.9). Tokens per run: 69k vs 70k (1.02x); time 56s vs 54s. One-liner group: 4.83 (the skill's gain over the one-liner: -0.67; one-liner's own gain over no skill: +1.57); one-liner tokens 60k per run, so the skill costs 1.17x the one-liner.
What happened
- Control 3.26, skill 4.16, one-liner 4.83 (n=6 each). Finding the planted issues is similar for all arms (5-7 of 8, with the missing-rate-limit issue missed by almost every run); the differences are mainly in format and length.
- Format: control did not use the
L<n>:line format in any of its 6 runs (its orders review is a numbered list with a bold severity). The skill met the format in the commit-review task (5.0 in both runs) and in one orders run, but in the auth task its lines begin with a number and a bold severity, so the format check failed in 2/2 and the score stayed at 3.3. The one-liner passed in nearly all. - Brevity: control reviews ran 233-332 words on the orders and auth tasks (full marks at 220 words); the skill's auth reviews were also about 300 words, while the one-liner stayed under the limit in nearly all runs.
- No arm used hedging phrases. Tokens: skill 70k, one-liner 60k, control 69k per run (1.01x control). The skill was read in 6/6 treatment runs.
Dimension scores
| Dimension | Score | Why |
|---|---|---|
| 1_output_quality | 4.16 | Mean score 4.16/5 with the skill (programmatic checks, see per-task table). |
| 2_lift_vs_no_skill | 2.9 | +0.90 points vs 3.26 without the skill (n=2 per task; under ~1 point is noise). |
| 3_trigger | 5.0 | Skill file read in 6/6 treatment runs. |
| 4_instructions_portability | 4 | SKILL.md only. |
| 5_efficiency | 5 | 1.02x tokens (70k vs 69k per run) and 0.96x time (54s vs 56s). |
| 6_safety_hygiene | 4 | Prose only; see security.md. |
Limits of this test
- Rule-based checks on the final message: recall of planted issues, a line-start regex for the L<n> format, a word count, and a hedging-phrase list. The quality of the fixes was not judged.
- The format regex is strict about the start of each line; a bullet such as '- L17 - High: ...' counts as matching but a numbered list with bold severity does not.
- One check ('did not modify files') passed everywhere and is not shown.
- n=2 per cell; small files.
- n = 2 per group per task: differences under about 1 point are noise.
- Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.
Security
See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.
Bottom line
The skill moves a review towards the terse one-finding-per-line style and keeps the same coverage at no extra cost, but it does not always keep to the format. A single sentence describing the format worked better here, so the benefit is over no guidance, not over a good prompt.
记录
Run log for caveman-review: Codex CLI, network off, per-run temp HOME/TMPDIR, one row per graded run. Columns: task, arm-run, score (0-5), tokens, seconds, checks passed, skill files the agent opened. t1-orders control-1: score 3.02, tokens 65430, secs 62.1, checks 4/5, opened nothing t1-orders control-2: score 3.1, tokens 82622, secs 61.8, checks 4/5, opened nothing t1-orders oneliner-1: score 4.66, tokens 64873, secs 59.1, checks 5/5, opened nothing t1-orders oneliner-2: score 4.66, tokens 64681, secs 49.6, checks 5/5, opened nothing t1-orders treatment-1: score 4.83, tokens 50257, secs 52.3, checks 5/5, opened caveman-review t1-orders treatment-2: score 3.47, tokens 72971, secs 62.1, checks 4/5, opened caveman-review t2-auth control-1: score 3.3, tokens 82305, secs 67.1, checks 4/5, opened nothing t2-auth control-2: score 3.08, tokens 64757, secs 55.3, checks 4/5, opened nothing t2-auth oneliner-1: score 5.0, tokens 64677, secs 57.0, checks 5/5, opened nothing
18 次运行、3 个产物
样例产物
前沿 · 装了 skill：final message（t1-orders，第 1 次，Codex）
- L17 — **High: Missing authorization.** `cancel_order` ignores `current_user`, allowing cancellation of another user’s order by ID (CWE-862). Require ownership or explicit administrative permission before updating. - L9 — **High: SQL injection.** `order_id` is interpolated directly into SQL; input such as `0 OR 1=1` changes the query and can retrieve an unrelated order (CWE-89). Use `WHERE id = ?` with bound parameters. - L4, L48 — **High: Credential exposure.** `API_KEY` is hardcoded and included in the successful-charge log, exposing it through source access and logs (CWE-798/CWE-532). Load it from a secret store, remove it from logs, and rotate it if genuine. - L47–55 — **High:** Every `charge` call reaches `NotImplementedError`, which is swallowed into `None`; no payment can succeed. Implement `send_to_gateway` and surface explicit payment failures. - L24, L26 — **Medium:** Discount thresholds contradict the docstring: exactly 100 receives no discount, and exactly 500 receives only 10%. Change both comparisons to `>=`. - L39–41 — **Medium:** Opening the export with `"w"` truncates an existing file before querying or serializing succeeds. Prepare the export first, then write through a temporary file and replace the destination. - L39–42 — **Medium:** The export file is never explicitly flushed or closed; completion depends on object cleanup. Use `with open(...)` so writes and closure complete before returning.