caveman-commit (JuliusBrussee/caveman)
caveman-commit: writes Conventional Commits messages with a breaking-change footer (control 4.0, skill 5.0, one-liner 4.95) for about the same tokens; control writes a plain sentence, and a one-sentence prompt of the format does as well
26.0 / 30
Lift over no skill: +0.98
Situational: the lift +0.98 is under +1.0. Recommended needs a total of 22/30, output quality of 3.5, and a lift of +1.0.
SOTA
PassTested with Codex.
Covers Codex, Claude Code
Codex CLI 0.166 (SOTA tier): 18 graded runs. A pass here also covers Claude Code under our tier rule; it says nothing about the second tier.
Scores
Each dimension is 0 to 5 and the six add up to the total out of 30. 5 is better on every dimension. Cost 5 means cheap and fast. Install 5 means easy.
- Trigger reliability
- 5.0
- Lift over no skill
- 3.0
- Output quality
- 5.0
- Token and time cost
- 5.0
- Install and portability
- 4.0
- Safety
- 4.0
Security review
SafeStatic scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Manual review: Safe to run in a no-network sandbox (no findings). A clean scan does not prove a skill safe.
Install
npx skills@latest add JuliusBrussee/caveman --skill caveman-commit
Pinned commit: 9b68f7ed12b05e7efaff18e65c5a3fc2849fd0a4
Tested: 2026-11-02
What we tested
- Commit message for a small fix — I need a commit message. Write the commit message for the change in change.diff and save it to COMMIT_MSG.txt (the message only).
- Commit message for a new endpoint — Please write a commit. Write the commit message for the change in change.diff and save it to COMMIT_MSG.txt (the message only).
- Commit message for a breaking change — Write a commit message. Write the commit message for the change in change.diff and save it to COMMIT_MSG.txt (the message only).
Pros
- Lifts the mean score from 4.02 to 5.0 over no skill (+0.98).
- The skill was picked up without being named: Skill file read in 6/6 treatment runs.
- No token overhead: 1.09x the tokens of no skill.
- High absolute quality with the skill: 5.0 out of 5.
Cons
- A one-sentence prompt scored 4.95, against 5.0 with the skill (+0.05).
- n=2 per task per arm; differences under about 1 point are noise.
Review
Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below). Three diffs (a pagination fix, a new /health endpoint, a rename of a config key from timeout to timeout_ms). The prompt only asks for a commit message saved to COMMIT_MSG.txt, with no format given. The one-liner states the Conventional Commits format and its rules in one sentence.
Three-arm comparison
| Arm | Mean score (0-5) | Tokens per run |
|---|---|---|
| Without the skill | 4.02 | 53k |
| One-sentence prompt | 4.95 | 48k |
| With the skill | 5.0 | 58k |
Measured with Codex CLI over 18 runs, scored by rule-based checks with no LLM judge. Total 26.0 / 30, verdict Situational.
Verdict: Situational (26.0 / 30). Source: JuliusBrussee/caveman, skills/caveman-commit, pinned at commit 9b68f7e. Install: npx skills@latest add JuliusBrussee/caveman --skill caveman-commit. Licence: MIT (repo LICENSE).
What it is
The caveman-commit skill (JuliusBrussee/caveman): a commit-message style guide: Conventional Commits type(scope): imperative summary of at most 50 characters with no trailing period, a body only when the why is not obvious, a BREAKING CHANGE note for breaking changes, and no 'this commit', 'I', 'we', 'now', emoji or AI attribution.
How we tested
Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).
Three diffs (a pagination fix, a new /health endpoint, a rename of a config key from timeout to timeout_ms). The prompt only asks for a commit message saved to COMMIT_MSG.txt, with no format given. The one-liner states the Conventional Commits format and its rules in one sentence.
- Commit message for a small fix — I need a commit message. Write the commit message for the change in change.diff and save it to COMMIT_MSG.txt (the message only).
- Commit message for a new endpoint — Please write a commit. Write the commit message for the change in change.diff and save it to COMMIT_MSG.txt (the message only).
- Commit message for a breaking change — Write a commit message. Write the commit message for the change in change.diff and save it to COMMIT_MSG.txt (the message only).
Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.
Results
The one-liner given to the third group: "Write a Conventional Commits message: type(scope): imperative summary of at most 50 characters with no trailing period, a body only when the why is not obvious (breaking changes always get one with a BREAKING CHANGE note), and no 'this commit', 'I', 'we', 'now', emoji or AI attribution."
| Task | Control score | One-liner | With skill | Control tokens | One-liner tokens | With-skill tokens | Skill read |
|---|---|---|---|---|---|---|---|
| t1-fix | 4.49 | 5.0 | 5.0 | 47k | 48k | 49k | 2/2 |
| t2-feature | 4.49 | 5.0 | 5.0 | 54k | 48k | 55k | 2/2 |
| t3-breaking | 3.07 | 4.86 | 5.0 | 56k | 48k | 67k | 2/2 |
Mean score: 4.02 without vs 5.0 with the skill (delta +0.98). Tokens per run: 53k vs 58k (1.09x); time 42s vs 40s. One-liner group: 4.95 (the skill's gain over the one-liner: +0.05; one-liner's own gain over no skill: +0.93); one-liner tokens 48k per run, so the skill costs 1.19x the one-liner.
What happened
- Control 4.02, skill 5.0, one-liner 4.95 (n=6 each). Control writes a plain imperative sentence ('Fix pagination offset for one-based page numbers') with no
type(scope):prefix in 6/6 runs, so it fails the 'conventional type' rule every time; its other rules mostly passed (one run also had a subject over 50 characters). - Breaking change task: control did not mark the breaking change (no BREAKING CHANGE note and no
!) in 2/2 runs; its body does mention the migration. The skill scored 5.0 in both runs:feat(config)!:with a BREAKING CHANGE footer. The one-liner did the same in 2/2 but one run's message failed the conventional-type check. - Tokens: skill 58k, one-liner 48k, control 53k per run (1.09x control). The skill was read in 6/6 treatment runs.
Dimension scores
| Dimension | Score | Why |
|---|---|---|
| 1_output_quality | 5.0 | Mean score 5.00/5 with the skill (programmatic checks, see per-task table). |
| 2_lift_vs_no_skill | 2.98 | +0.98 points vs 4.02 without the skill (n=2 per task; under ~1 point is noise). |
| 3_trigger | 5.0 | Skill file read in 6/6 treatment runs. |
| 4_instructions_portability | 4 | SKILL.md only. |
| 5_efficiency | 5 | 1.09x tokens (58k vs 53k per run) and 0.96x time (40s vs 42s). |
| 6_safety_hygiene | 4 | Prose only; see security.md. |
Limits of this test
- The checks are rule-based on COMMIT_MSG.txt (type prefix, length, tense, tone, forbidden words, breaking note); whether the message describes the diff well was not judged beyond a check for key terms.
- Control is penalised for not guessing the Conventional Commits convention that the prompt did not request.
- n=2 per cell; three small diffs.
- n = 2 per group per task: differences under about 1 point are noise.
- Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.
Security
See security.md. Scan: 0 CRITICAL / 0 HIGH / 0 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.
Bottom line
A reliable way to get Conventional Commits messages with breaking-change flags from a bare 'write a commit message' prompt. A single sentence naming the format gives the same result, so the skill mainly saves you from writing it.
Transcript
Run log for caveman-commit: Codex CLI, network off, per-run temp HOME/TMPDIR, one row per graded run. Columns: task, arm-run, score (0-5), tokens, seconds, checks passed, skill files the agent opened. t1-fix control-1: score 4.49, tokens 47722, secs 20.0, checks 2/3, opened nothing t1-fix control-2: score 4.49, tokens 47848, secs 20.8, checks 2/3, opened nothing t1-fix oneliner-1: score 5.0, tokens 47948, secs 29.4, checks 3/3, opened nothing t1-fix oneliner-2: score 5.0, tokens 48277, secs 21.1, checks 3/3, opened nothing t1-fix treatment-1: score 5.0, tokens 49540, secs 24.8, checks 3/3, opened caveman-commit t1-fix treatment-2: score 5.0, tokens 49614, secs 16.5, checks 3/3, opened caveman-commit t2-feature control-1: score 4.49, tokens 48133, secs 31.0, checks 2/3, opened nothing t2-feature control-2: score 4.49, tokens 60402, secs 101.3, checks 2/3, opened nothing t2-feature oneliner-1: score 5.0, tokens 48035, secs 107.1, checks 3/3, opened nothing
18 runs, 3 artifacts
Sample output
SOTA · With the skill: COMMIT_MSG.txt (t1-fix, run 1, Codex)
fix(api): correct offset for one-based pagination