Agent Skills

全部技能

caveman-commit (JuliusBrussee/caveman)

caveman-commit：对照组 4.02、一句话提示 4.95、装 skill 5.0（0–5 分，Codex 实测，n=2/任务/组）。结论：视情况而定，相对不装 skill +0.98。

看情况安全

26.0 / 30

相对不装 skill 的提升：+0.98

视情况而定：提升 +0.98 低于 +1.0。推荐要求总分 22/30、产出质量 3.5、提升 +1.0。

前沿

通过

使用 Codex 测试。

覆盖 Codex, Claude Code

Codex CLI 0.166（SOTA 档）：18 次评分运行。按我们的分档规则，这档通过同时覆盖 Claude Code；它不代表第二档的结果。

分数

每一维 0 到 5 分，六维相加是满分 30 的总分。每一维都是 5 更好。成本 5 表示又便宜又快。安装 5 表示容易。

触发可靠性
5.0
相对不装的提升
3.0
产出质量
5.0
令牌与时间成本
5.0
安装与可移植性
4.0
安全
4.0

安全审查

安全

静态扫描：0 CRITICAL / 0 HIGH / 0 MEDIUM。人工审查：可在断网沙箱中运行。扫描没有命中不等于绝对安全。

安装

npx skills@latest add JuliusBrussee/caveman --skill caveman-commit

来源

固定提交: 9b68f7ed12b05e7efaff18e65c5a3fc2849fd0a4

测试日期: 2026-11-02

我们测了什么

  • Commit message for a small fix — I need a commit message. Write the commit message for the change in change.diff and save it to COMMIT_MSG.txt (the message only).
  • Commit message for a new endpoint — Please write a commit. Write the commit message for the change in change.diff and save it to COMMIT_MSG.txt (the message only).
  • Commit message for a breaking change — Write a commit message. Write the commit message for the change in change.diff and save it to COMMIT_MSG.txt (the message only).

优点

  • 平均分从不装时的 4.02 提到 5.0（+0.98）。
  • 不点名也会被触发：Skill file read in 6/6 treatment runs.
  • 不增加 token 开销：约为不装 skill 时的 1.09 倍。
  • 装 skill 后绝对质量高：5.0 / 5。

缺点

  • 一句话提示得 4.95，装 skill 得 5.0（差 +0.05）。
  • 每个任务每组 n=2，差距在 1 分以内属于噪声。

评测

在 Codex CLI（SOTA 档）上跑 3 个任务、对照、一句话提示、装 skill 三组，共 18 次运行，由规则脚本打分（没有 LLM 评委），断网沙箱、每次运行独立临时目录。具体做法见下方英文评测原文。

三组对比

组别平均分（0–5）每次 token
不装 skill4.0253k
一句话提示4.9548k
装了 skill5.058k

Codex CLI 实测，18 次运行，规则脚本打分，没有 LLM 评委。总分 26.0 / 30，结论 视情况而定。

下面是评测原文，英文。

Verdict: Situational (26.0 / 30). Source: JuliusBrussee/caveman, skills/caveman-commit, pinned at commit 9b68f7e. Install: npx skills@latest add JuliusBrussee/caveman --skill caveman-commit. Licence: MIT (repo LICENSE).

What it is

The caveman-commit skill (JuliusBrussee/caveman): a commit-message style guide: Conventional Commits type(scope): imperative summary of at most 50 characters with no trailing period, a body only when the why is not obvious, a BREAKING CHANGE note for breaking changes, and no 'this commit', 'I', 'we', 'now', emoji or AI attribution.

How we tested

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).

Three diffs (a pagination fix, a new /health endpoint, a rename of a config key from timeout to timeout_ms). The prompt only asks for a commit message saved to COMMIT_MSG.txt, with no format given. The one-liner states the Conventional Commits format and its rules in one sentence.

  1. Commit message for a small fix — I need a commit message. Write the commit message for the change in change.diff and save it to COMMIT_MSG.txt (the message only).
  2. Commit message for a new endpoint — Please write a commit. Write the commit message for the change in change.diff and save it to COMMIT_MSG.txt (the message only).
  3. Commit message for a breaking change — Write a commit message. Write the commit message for the change in change.diff and save it to COMMIT_MSG.txt (the message only).

Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.

Results

The one-liner given to the third group: "Write a Conventional Commits message: type(scope): imperative summary of at most 50 characters with no trailing period, a body only when the why is not obvious (breaking changes always get one with a BREAKING CHANGE note), and no 'this commit', 'I', 'we', 'now', emoji or AI attribution."

TaskControl scoreOne-linerWith skillControl tokensOne-liner tokensWith-skill tokensSkill read
t1-fix4.495.05.047k48k49k2/2
t2-feature4.495.05.054k48k55k2/2
t3-breaking3.074.865.056k48k67k2/2

Mean score: 4.02 without vs 5.0 with the skill (delta +0.98). Tokens per run: 53k vs 58k (1.09x); time 42s vs 40s. One-liner group: 4.95 (the skill's gain over the one-liner: +0.05; one-liner's own gain over no skill: +0.93); one-liner tokens 48k per run, so the skill costs 1.19x the one-liner.

What happened

  • Control 4.02, skill 5.0, one-liner 4.95 (n=6 each). Control writes a plain imperative sentence ('Fix pagination offset for one-based page numbers') with no type(scope): prefix in 6/6 runs, so it fails the 'conventional type' rule every time; its other rules mostly passed (one run also had a subject over 50 characters).
  • Breaking change task: control did not mark the breaking change (no BREAKING CHANGE note and no !) in 2/2 runs; its body does mention the migration. The skill scored 5.0 in both runs: feat(config)!: with a BREAKING CHANGE footer. The one-liner did the same in 2/2 but one run's message failed the conventional-type check.
  • Tokens: skill 58k, one-liner 48k, control 53k per run (1.09x control). The skill was read in 6/6 treatment runs.

Dimension scores

DimensionScoreWhy
1_output_quality5.0

Mean score 5.00/5 with the skill (programmatic checks, see per-task table).

2_lift_vs_no_skill2.98

+0.98 points vs 4.02 without the skill (n=2 per task; under ~1 point is noise).

3_trigger5.0Skill file read in 6/6 treatment runs.
4_instructions_portability4SKILL.md only.
5_efficiency51.09x tokens (58k vs 53k per run) and 0.96x time (40s vs 42s).
6_safety_hygiene4Prose only; see security.md.

Limits of this test

  • The checks are rule-based on COMMIT_MSG.txt (type prefix, length, tense, tone, forbidden words, breaking note); whether the message describes the diff well was not judged beyond a check for key terms.
  • Control is penalised for not guessing the Conventional Commits convention that the prompt did not request.
  • n=2 per cell; three small diffs.
  • n = 2 per group per task: differences under about 1 point are noise.
  • Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.

Security

See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.

Bottom line

A reliable way to get Conventional Commits messages with breaking-change flags from a bare 'write a commit message' prompt. A single sentence naming the format gives the same result, so the skill mainly saves you from writing it.

记录

Run log for caveman-commit: Codex CLI, network off, per-run temp HOME/TMPDIR, one row per graded run.
Columns: task, arm-run, score (0-5), tokens, seconds, checks passed, skill files the agent opened.

t1-fix control-1: score 4.49, tokens 47722, secs 20.0, checks 2/3, opened nothing
t1-fix control-2: score 4.49, tokens 47848, secs 20.8, checks 2/3, opened nothing
t1-fix oneliner-1: score 5.0, tokens 47948, secs 29.4, checks 3/3, opened nothing
t1-fix oneliner-2: score 5.0, tokens 48277, secs 21.1, checks 3/3, opened nothing
t1-fix treatment-1: score 5.0, tokens 49540, secs 24.8, checks 3/3, opened caveman-commit
t1-fix treatment-2: score 5.0, tokens 49614, secs 16.5, checks 3/3, opened caveman-commit
t2-feature control-1: score 4.49, tokens 48133, secs 31.0, checks 2/3, opened nothing
t2-feature control-2: score 4.49, tokens 60402, secs 101.3, checks 2/3, opened nothing
t2-feature oneliner-1: score 5.0, tokens 48035, secs 107.1, checks 3/3, opened nothing

18 次运行、3 个产物

样例产物

前沿 · 装了 skill：COMMIT_MSG.txt（t1-fix，第 1 次，Codex）

fix(api): correct offset for one-based pagination

查看 Pro