caveman (JuliusBrussee/caveman, SKILL.md only)

评测

在 Codex CLI（SOTA 档）上跑 3 个任务、对照、一句话提示、装 skill 三组，共 18 次运行，由规则脚本打分（没有 LLM 评委），断网沙箱、每次运行独立临时目录。具体做法见下方英文评测原文。

三组对比

Codex CLI 实测，18 次运行，规则脚本打分，没有 LLM 评委。总分 21.7 / 30，结论 视情况而定。

下面是评测原文，英文。

Verdict: Situational (21.7 / 30). Source: JuliusBrussee/caveman, skills/caveman, pinned at commit 9b68f7e. Install: npx skills@latest add JuliusBrussee/caveman --skill caveman. Licence: MIT (repo LICENSE; hooks and installer in the repo were NOT installed or run).

What it is

The caveman skill (only skills/caveman/SKILL.md; the repo's hooks and installer were not installed). A terse-reply style: answer first, kill greetings and recaps, short words, drop articles but never negations, keep code, paths, numbers and errors verbatim, plain prose for warnings and irreversible actions. It triggers on 'caveman mode', 'be brief', 'less tokens'. About 600 words of Markdown, no scripts.

How we tested

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a Mac test machine, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).

Each prompt is a technical question (an infinite useEffect loop, a slow Postgres query, merge vs rebase on a shared branch) ending in 'Be brief; I'm paying per token.' That phrase is one of the skill's own triggers, and the control group gets the same sentence. So the test asks what the skill adds on top of a plain request to be brief.

1. Explain an infinite useEffect loop — In a React component I have const fetchData = async () => { const r = await fetch('/api/items'); setItems(await r.json()); } and useEffect(() => { fetchData(); }, [fetchData]);. The component re-fetches in an endless loop. Why, and what is the fix? Be brie…
2. Fix a slow Postgres query — Postgres: SELECT id, total FROM orders WHERE customer_id = 42 ORDER BY created_at DESC LIMIT 20; takes seconds on a 50M-row orders table that only has a primary-key index. What should I do? Be brief; I'm paying per token.
3. Merge or rebase on a shared branch — My team works on a shared feature branch that other people have already pulled. I want to bring in the latest main. Should I use git merge or git rebase, and why? Be brief; I'm paying per token.

Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.

Results

The one-liner given to the third group: "Answer tersely: answer first, no greeting, filler, hedging or recap, drop needless articles, keep every technical fact, code, number and negation exact."

Mean score: 4.67 without vs 4.67 with the skill (delta +0.0). Tokens per run: 14k vs 31k (2.20x); time 12s vs 19s. One-liner group: 4.5 (the skill's gain over the one-liner: +0.17; one-liner's own gain over no skill: -0.17); one-liner tokens 14k per run, so the skill costs 2.20x the one-liner.

What happened

• Without the skill the agent already answers in about 59 words (45-70) when asked to be brief; with the skill 52 words (39-64) and with the one-sentence version 54. The 7-word difference is within run-to-run spread. No group used filler openers or closers; all 18 answers kept the code, commands and key facts.
• Fact retention was the same: 3-5 of 5 required technical facts per answer in every group (share of required facts kept: 87% control, 80% one-liner, 87% skill). The one fact the agents often dropped (that a state update re-renders the component and re-runs the effect) was dropped by all three groups equally. The 'do not rebase a shared branch' point was matched by our check in 1 of 2 control, 0 of 2 one-liner and 1 of 2 skill answers; in the answers we read, all of them recommended merge and kept rebase for private or unpublished work, so those misses are partly wording.
• The skill was read in 6/6 runs because the prompt contains its trigger phrase. It costs 31k tokens per run against 14k (2.2x), because the agent loads the skill and runs two commands to read it before answering a question it could answer directly. The one-liner costs the same as the control.
• Reading the answers, the styles are the same: answer first, fragments, a short code block. We could not tell skill answers from control answers.

Dimension scores

Limits of this test

• The test measures the skill on top of an explicit 'be brief' request. The skill's real use is persistent: it stays on for a whole session so you never have to repeat the request, which this three-question test does not exercise.
• The token cost counted here is a one-off skill load per conversation; over a long session with a saved few words per answer it could pay for itself; we did not measure that.
• Only chat answers to three technical questions were checked; commit messages, code review comments and long agent sessions (where the skill has other rules) were not tested.
• The repo's hooks, statusline and other caveman skills were not installed or run.
• n = 2 per group per task: differences under about 1 point are noise.
• Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.

Security

See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.

Bottom line

On top of a plain 'be brief' request, caveman did not make answers shorter or less accurate in a way we could measure, and loading it doubled the tokens of a short answer. If you want your agent terse for a whole session without asking each time, the skill does that; if you only need brevity on a question, say so in the prompt.