security-review (everything-claude-code)
security-review：对照组 4.22、一句话提示 4.29、装 skill 4.06（0–5 分，Codex 实测，n=2/任务/组）。结论：视情况而定，相对不装 skill -0.16。
24.9 / 30
相对不装 skill 的提升：-0.16
视情况而定：提升 -0.16 低于 +1.0。推荐要求总分 22/30、产出质量 3.5、提升 +1.0。
前沿
通过使用 Codex 测试。
覆盖 Codex, Claude Code
Codex CLI 0.166（SOTA 档）：18 次评分运行。按我们的分档规则，这档通过同时覆盖 Claude Code；它不代表第二档的结果。
分数
每一维 0 到 5 分，六维相加是满分 30 的总分。每一维都是 5 更好。成本 5 表示又便宜又快。安装 5 表示容易。
- 触发可靠性
- 5.0
- 相对不装的提升
- 1.8
- 产出质量
- 4.1
- 令牌与时间成本
- 5.0
- 安装与可移植性
- 4.0
- 安全
- 5.0
安全审查
安全静态扫描：5 CRITICAL / 6 HIGH / 4 MEDIUM。人工审查：可在断网沙箱中运行。扫描没有命中不等于绝对安全。
安装
npx skills@latest add affaan-m/everything-claude-code --skill security-review
固定提交: b38be6a7f6350964d59328b52f0624e1d1d552d1
测试日期: 2026-11-02
我们测了什么
- Review a small service module — Security-review orders_service.py before I merge it. List the problems you find, most serious first. Do not change any files.
- Review an auth helper — Security-review auth_handlers.py: I just wrote it for our login flow. List the problems, most serious first. Do not change any files.
- Review upload endpoints — Security-review upload_api.py. List the vulnerabilities, most serious first. Do not change any files.
优点
- 不点名也会被触发：Skill file read in 6/6 treatment runs.
- 不增加 token 开销：约为不装 skill 时的 1.03 倍。
缺点
- 相对不装 skill 没有可测量的提升（-0.16，在噪声范围内）。
- 一句话提示得 4.29，装 skill 得 4.06（差 -0.23）。
- 测试说明：任务 t1 里有 3 个并非安全问题的埋点，召回分数的上限只有 3.1 左右。
- 每个任务每组 n=2，差距在 1 分以内属于噪声。
评测
在 Codex CLI（SOTA 档）上跑 3 个任务、对照、一句话提示、装 skill 三组，共 18 次运行，由规则脚本打分（没有 LLM 评委），断网沙箱、每次运行独立临时目录。具体做法见下方英文评测原文。
三组对比
| 组别 | 平均分（0–5） | 每次 token |
|---|---|---|
| 不装 skill | 4.22 | 60k |
| 一句话提示 | 4.29 | 44k |
| 装了 skill | 4.06 | 62k |
Codex CLI 实测，18 次运行，规则脚本打分，没有 LLM 评委。总分 24.9 / 30，结论 视情况而定。
测试披露： 任务 t1 里有 3 个并非安全问题的埋点，召回分数的上限只有 3.1 左右。
下面是评测原文，英文。
Verdict: Situational (24.9 / 30). Source: affaan-m/everything-claude-code, skills/security-review, pinned at commit b38be6a. Install: npx skills@latest add affaan-m/everything-claude-code --skill security-review. Licence: MIT (repo LICENSE).
What it is
The security-review skill from affaan-m/everything-claude-code (ECC): a security checklist for when you add authentication, handle user input, work with secrets, create API endpoints or implement payments. About 1,800 words of Markdown in SKILL.md (secrets management, input validation, SQL injection, authentication and authorization, XSS, CSRF, rate limiting, sensitive data exposure, dependency security, with short good/bad code examples and a pre-deployment checklist) and one extra Markdown file on cloud infrastructure security. No scripts.
How we tested
Codex CLI 0.166 (gpt-6-astra, low reasoning) on a Mac test machine, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).
Three small files with 8 planted problems each (a service module, an auth helper, upload endpoints) and a read-only 'list the problems, most serious first' request. The order service includes three problems that are not security flaws (an off-by-one boundary, a resource leak, float money), which a security review is expected to skip, so on that task a security reviewer can find at most 5 of 8 and the score tops out near 3.1.
- Review a small service module — Security-review orders_service.py before I merge it. List the problems you find, most serious first. Do not change any files.
- Review an auth helper — Security-review auth_handlers.py: I just wrote it for our login flow. List the problems, most serious first. Do not change any files.
- Review upload endpoints — Security-review upload_api.py. List the vulnerabilities, most serious first. Do not change any files.
Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.
Results
The one-liner given to the third group: "Check the code against a security checklist: hardcoded secrets, unvalidated input, SQL/command injection, broken authentication and authorization, unsafe file uploads, sensitive data in errors or logs, and missing rate limiting, citing the exact line for each finding."
| Task | Control score | One-liner | With skill | Control tokens | One-liner tokens | With-skill tokens | Skill read |
|---|---|---|---|---|---|---|---|
| t1-orders | 2.89 | 3.12 | 2.42 | 60k | 44k | 72k | 2/2 |
| t2-auth | 4.77 | 5.0 | 4.77 | 59k | 44k | 61k | 2/2 |
| t3-upload | 5.0 | 4.77 | 5.0 | 59k | 44k | 51k | 2/2 |
Mean score: 4.22 without vs 4.06 with the skill (delta -0.16). Tokens per run: 60k vs 62k (1.03x); time 38s vs 44s. One-liner group: 4.29 (the skill's gain over the one-liner: -0.23; one-liner's own gain over no skill: +0.08); one-liner tokens 44k per run, so the skill costs 1.40x the one-liner.
What happened
- No gain. Mean score 4.22 without the skill, 4.29 with a one-sentence security checklist, 4.06 with the skill. Planted-issue recall: 4.8 of 8 per run without, 4.9 with the one-liner, 4.5 with the skill.
- The skill was a little worse on the order-service task: 2.4 average against 2.9 for control and 3.1 for the one-liner (2 and 3 planted items found per run, against 3-4 for the others). That is within the noise at n=2.
- On the auth helper and the upload endpoints every group found 7 or 8 of 8 planted problems; both are classic textbook flaws (md5, path traversal, pickle, command injection) that plain Codex already knows.
- The skill is read in 6/6 runs. It costs 62k tokens per run against 60k (1.03x) and 44k with the one-liner (1.41x): the one-liner runs were the cheapest, because they read less before answering.
- A scoring caveat: items are matched by regex, so a differently worded finding can count as a miss.
Dimension scores
| Dimension | Score | Why |
|---|---|---|
| 1_output_quality | 4.06 | Mean score 4.06/5 with the skill (programmatic checks, see per-task table). |
| 2_lift_vs_no_skill | 1.84 | -0.16 points vs 4.22 without the skill (n=2 per task; under ~1 point is noise). |
| 3_trigger | 5.0 | Skill file read in 6/6 treatment runs. |
| 4_instructions_portability | 4 | Plain Markdown checklist, mostly JS/TS and Supabase-flavoured examples; applies to other stacks only by analogy. |
| 5_efficiency | 5 | 1.03x tokens (62k vs 60k per run) and 1.17x time (44s vs 38s). |
| 6_safety_hygiene | 5 | Markdown only; scanner hits are code examples (process.env, fetch, template strings) inside the checklist. |
Limits of this test
- Three small single-file reviews with textbook flaws; subtler problems (business-logic authorization, race conditions, supply chain) were not planted.
- The skill's cloud/infrastructure checklist and pre-deployment list were not exercised: no infrastructure files in the test.
- Recall only; false positives and the quality of fix advice were not scored.
- n = 2 per group per task: differences under about 1 point are noise.
- Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.
Security
See security.md. Scan: 5 CRITICAL / 6 HIGH / 4 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.
Bottom line
A reasonable checklist, but plain Codex already finds the same textbook vulnerabilities, and one sentence listing the same categories did as well for about 70% of the tokens. The skill adds nothing measurable on this kind of code review; it may serve as a prompt for a team that wants a consistent checklist.
记录
Run log for security-review: Codex CLI, network off, per-run temp HOME/TMPDIR, one row per graded run. Columns: task, arm-run, score (0-5), tokens, seconds, checks passed, skill files the agent opened. t1-orders control-1: score 3.12, tokens 60493, secs 36.1, checks 1/2, opened nothing t1-orders control-2: score 2.66, tokens 60831, secs 41.9, checks 1/2, opened nothing t1-orders oneliner-1: score 3.12, tokens 44382, secs 32.3, checks 1/2, opened nothing t1-orders oneliner-2: score 3.12, tokens 44393, secs 35.0, checks 1/2, opened nothing t1-orders treatment-1: score 2.19, tokens 73335, secs 69.3, checks 1/2, opened security-review t1-orders treatment-2: score 2.66, tokens 72383, secs 54.7, checks 1/2, opened security-review t2-auth control-1: score 4.53, tokens 59839, secs 37.6, checks 2/2, opened nothing t2-auth control-2: score 5.0, tokens 59894, secs 37.4, checks 2/2, opened nothing t2-auth oneliner-1: score 5.0, tokens 44259, secs 36.9, checks 2/2, opened nothing
18 次运行、3 个产物
样例产物
前沿 · 装了 skill：final message（t1-orders，第 1 次，Codex）
Found six security issues, ordered by severity. Exposure depends on how callers expose these functions; no caller code was provided. 1. **High — SQL injection** — [orders_service.py:9](/var/folders/ln/t0z8fmzd0kz08d3fkz6tskn40000gn/T/asv-security-r-cagonwka/work/orders_service.py:9) `order_id` is interpolated directly into SQL. An input such as `0 OR 1=1` bypasses the intended lookup condition; crafted `UNION` queries could extract other database data. Bind `order_id` as a parameter. 2. **High — Missing cancellation authorization** — [orders_service.py:13](/var/folders/ln/t0z8fmzd0kz08d3fkz6tskn40000gn/T/asv-security-r-cagonwka/work/orders_service.py:13) `current_user` is never used. Anyone able to invoke this function can cancel another user’s order by supplying its ID. Enforce ownership or an authorized administrative role before updating it. 3. **High — Arbitrary file overwrite through exports** — [orders_service.py:39](/var/folders/ln/t0z8fmzd0kz08d3fkz6tskn40000gn/T/asv-security-r-cagonwka/work/orders_service.py:39) `open(path, "w")` immediately truncates any accessible target and follows symlinks. If callers control `path`, they can overwrite files with the service’s permissions—even before the query succeeds. Use server-generated filenames within a controlled export directory and prevent traversal and symlink escapes. 4. **High — Cross-user order export** — [orders_service.py:40](/var/folders/ln/t0z8fmzd0kz08d3fkz6tskn40000gn/T/asv-security-r-cagonwka/work/orders_service.py:40) The export accepts any `user_id` without checking the requester’s identity or permissions. A caller can export another customer’s records. Derive the user ID from authenticated context and explicitly authorize administrative exports. 5. **High if valid — Embedded payment credential and credential logging** — [orders_service.py:4](/var/folders/ln/t0z8fmzd0kz08d3fkz6tskn40000gn/T/asv-security-r-cagonwka/work/orders_service.py:4), [orders_service.py:48](/var/folders/ln/t0z8fmzd0kz08d3fkz6tskn40000gn/T/asv-security-r-cagonwka/work/orders_service.py:48) A live-looking API key is embedded in source and would be printed after successful gateway calls. Its validity is unverified; the current gateway stub always raises. If real, rotate it, load its replacement from secret storage, and remove credential logging. 6. **Medium — Untrusted payment values lack validation** — [orders_service.py:31](/var/folders/ln/t0z8fmzd0kz08d3fkz6tskn40000gn/T/asv-security-r-cagonwka/work/orders_service.py:31), [orders_service.py:45](/var/folders/ln/t0z8fmzd0kz08d3fkz6tskn40000gn/T/asv-security-r-cagonwka/work/orders_service.py:45) Totals trust supplied prices and quantities, while `charge` accepts an amount without matching it to the stored order. If client-controlled, these permit manipulated totals or underpayment. Retrieve authoritative prices, validate quantities and amounts, and compute the charge server-side. No files were changed. This was a static review; the sandbox blocked the attempted in-memory verification.