Agent Skills

全部技能

mcp-builder (anthropics/skills)

mcp-builder：对照组 3.57、一句话提示 4.88、装 skill 4.61（0–5 分，Codex 实测，n=2/任务/组）。结论：视情况而定，相对不装 skill +1.04。

看情况安全

21.6 / 30

相对不装 skill 的提升：+1.04

视情况而定：总分 21.6/30 低于 22。推荐要求总分 22/30、产出质量 3.5、提升 +1.0。

前沿

通过

使用 Codex 测试。

覆盖 Codex, Claude Code

Codex CLI 0.166（SOTA 档）：18 次评分运行。按我们的分档规则，这档通过同时覆盖 Claude Code；它不代表第二档的结果。

分数

每一维 0 到 5 分，六维相加是满分 30 的总分。每一维都是 5 更好。成本 5 表示又便宜又快。安装 5 表示容易。

触发可靠性
5.0
相对不装的提升
3.0
产出质量
4.6
令牌与时间成本
3.0
安装与可移植性
3.0
安全
3.0

安全审查

安全

静态扫描：8 CRITICAL / 13 HIGH / 7 MEDIUM。人工审查：可在断网沙箱中运行。扫描没有命中不等于绝对安全。

安装

npx skills@latest add anthropics/skills --skill mcp-builder

来源

固定提交: ca45fc496a3436251f191354091d557da31c15b5

测试日期: 2026-11-02

我们测了什么

  • Python MCP server for a bookstore API — Write server.py: an MCP server in Python for a bookstore inventory REST API (base URL from the environment, bearer token from the environment). Tools: search books, get one book, update stock for a book, delete a book. Use the official mcp Python SDK (FastMC…
  • TypeScript MCP server for a weather API — Write src/index.ts and package.json: an MCP server in TypeScript for a weather REST API (base URL and API key from the environment). Tools: get current weather for a city, get a 5-day forecast, list saved locations, delete a saved location. Use the official MC…
  • Repair a hastily written MCP server — server.py is an MCP server for our warehouse API that works but was written in a hurry. Fix it so it follows good MCP server practice. Edit server.py in place and keep it talking to the same API. There is no network and no package installation here; just write…

优点

  • 平均分从不装时的 3.57 提到 4.61（+1.04）。
  • 不点名也会被触发：Skill file read in 6/6 treatment runs.
  • 装 skill 后绝对质量高：4.61 / 5。

缺点

  • 一句话提示得 4.88，装 skill 得 4.61（差 -0.27）。
  • token 消耗是不装 skill 时的 1.51 倍。
  • 每个任务每组 n=2，差距在 1 分以内属于噪声。

评测

在 Codex CLI（SOTA 档）上跑 3 个任务、对照、一句话提示、装 skill 三组，共 18 次运行，由规则脚本打分（没有 LLM 评委），断网沙箱、每次运行独立临时目录。具体做法见下方英文评测原文。

三组对比

组别平均分（0–5）每次 token
不装 skill3.5773k
一句话提示4.8875k
装了 skill4.61110k

Codex CLI 实测，18 次运行，规则脚本打分，没有 LLM 评委。总分 21.6 / 30，结论 视情况而定。

下面是评测原文，英文。

Verdict: Situational (21.6 / 30). Source: anthropics/skills, skills/mcp-builder, pinned at commit ca45fc4. Install: npx skills@latest add anthropics/skills --skill mcp-builder. Licence: Proprietary: anthropics/skills LICENSE.txt (source-available; evaluation only, not redistributed).

What it is

The mcp-builder skill from anthropics/skills: a guide for building Model Context Protocol servers (research and plan, implement, review and test, write evaluations). It ships language guides for Python (FastMCP) and TypeScript, MCP best-practice notes and two helper scripts for running evaluations against a server. Proprietary licence (source-available).

How we tested

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).

Three tasks: write a Python MCP server for a bookstore REST API, write a TypeScript MCP server for a weather API, and fix an existing quick-and-dirty server.py to follow good practice. Nothing needs the network (the agent cannot install the MCP SDK; it writes code only). The skill group's prompt is the same as control.

  1. Python MCP server for a bookstore API — Write server.py: an MCP server in Python for a bookstore inventory REST API (base URL from the environment, bearer token from the environment). Tools: search books, get one book, update stock for a book, delete a book. Use the official mcp Python SDK (FastMC…
  2. TypeScript MCP server for a weather API — Write src/index.ts and package.json: an MCP server in TypeScript for a weather REST API (base URL and API key from the environment). Tools: get current weather for a city, get a 5-day forecast, list saved locations, delete a saved location. Use the official MC…
  3. Repair a hastily written MCP server — server.py is an MCP server for our warehouse API that works but was written in a hurry. Fix it so it follows good MCP server practice. Edit server.py in place and keep it talking to the same API. There is no network and no package installation here; just write…

Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.

Results

The one-liner given to the third group: "Design the MCP server like an expert: consistent service-prefixed, action-oriented tool names (e.g. bookstore_search_books), typed input schemas (Pydantic or Zod) with described fields, annotations (readOnlyHint, destructiveHint, idempotentHint), paginated and size-limited results, actionable error messages, secrets only from environment variables, async I/O, and nothing printed to stdout on stdio transport."

TaskControl scoreOne-linerWith skillControl tokensOne-liner tokensWith-skill tokensSkill read
t1-python1.434.644.260k64k97k2/2
t2-typescript4.295.04.6460k55k100k2/2
t3-fix5.05.05.097k106k133k2/2

Mean score: 3.57 without vs 4.61 with the skill (delta +1.04). Tokens per run: 73k vs 110k (1.51x); time 80s vs 89s. One-liner group: 4.88 (the skill's gain over the one-liner: -0.27; one-liner's own gain over no skill: +1.31); one-liner tokens 75k per run, so the skill costs 1.47x the one-liner.

What happened

  • Where it matters, the skill helps: on the two from-scratch servers control missed the basics (tool names with a shared service prefix 2/4 runs, tool annotations such as readOnlyHint 2/2 Python runs, async tools 2/2 Python runs, actionable error messages 2/2 Python runs). Mean score 3.57 control, 4.61 skill.
  • The one-sentence group did better than the skill: 4.88, with one missed check in 18 runs, while the skill group missed 5. The sentence lists the same properties the checks look for, so it is an upper bound for a short prompt, but it costs 75k tokens per run against 110k for the skill and 73k for control.
  • A real defect in the skill group: both TypeScript treatment runs produced a createServer() for the new @modelcontextprotocol/server package but never started a transport (no stdio or Streamable HTTP), so the server file cannot be launched as written; control and one-liner runs did include a transport.
  • On the fix-it-up task all three groups scored the same (all 9/9 checks in all six runs): given an existing server, plain Codex already applied the practices.
  • The skill is heavy to load: SKILL.md plus reference guides; treatment runs left about 12 skill files in the project's .agents folder and read the guides, which is where the 1.5x tokens go. Its helper scripts (evaluation harness) were not run in the offline sandbox.

Dimension scores

DimensionScoreWhy
1_output_quality4.61

Mean score 4.61/5 with the skill (programmatic checks, see per-task table).

2_lift_vs_no_skill3.04

+1.04 points vs 3.57 without the skill (n=2 per task; under ~1 point is noise).

3_trigger5.0Skill file read in 6/6 treatment runs.
4_instructions_portability3

Large reference set, split across language guides; helper scripts need Python packages and a live server.

5_efficiency31.51x tokens (110k vs 73k per run) and 1.11x time (89s vs 80s).
6_safety_hygiene3

Scanner flags 8 CRITICAL / 13 HIGH, all read in context: sample server code and env passing in helper scripts; proprietary licence.

Limits of this test

  • Rule-based checks on code text; none of the generated servers was actually started or connected to a client.
  • No network: the agent could not install or import the MCP SDK, so API details (for example the new v2 TypeScript package) were not validated.
  • The skill's evaluation step (question generation and running them against the server) was not exercised.
  • n = 2 per group per task: differences under about 1 point are noise.
  • Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.

Security

See security.md. Scan: 8 CRITICAL / 13 HIGH / 7 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.

Bottom line

For a from-scratch server the skill clearly beats an unguided agent on naming, annotations, async and error handling, at 1.5x tokens. A short checklist sentence got the same properties for fewer tokens and without the missing-transport defect, so install it for the reference guides and consistency, not for a quality jump over a good prompt.

记录

Run log for mcp-builder: Codex CLI, network off, per-run temp HOME/TMPDIR, one row per graded run.
Columns: task, arm-run, score (0-5), tokens, seconds, checks passed, skill files the agent opened.

t1-python control-1: score 1.79, tokens 51610, secs 53.2, checks 4/9, opened nothing
t1-python control-2: score 1.07, tokens 69259, secs 98.6, checks 3/9, opened nothing
t1-python oneliner-1: score 5.0, tokens 74072, secs 101.3, checks 9/9, opened nothing
t1-python oneliner-2: score 4.29, tokens 54692, secs 126.2, checks 8/9, opened nothing
t1-python treatment-1: score 3.39, tokens 101809, secs 72.0, checks 6/9, opened mcp-builder
t1-python treatment-2: score 5.0, tokens 93275, secs 66.9, checks 9/9, opened mcp-builder
t2-typescript control-1: score 4.29, tokens 52126, secs 59.1, checks 8/9, opened nothing
t2-typescript control-2: score 4.29, tokens 69042, secs 61.5, checks 8/9, opened nothing
t2-typescript oneliner-1: score 5.0, tokens 55412, secs 104.3, checks 9/9, opened nothing

18 次运行、3 个产物

样例产物

前沿 · 装了 skill：server.py（t1-python，第 1 次，Codex）

"""Bookstore MCP server (Python 3.10+, official mcp SDK, httpx, Pydantic 2). Required environment variables: BOOKSTORE_BASE_URL: API root, e.g. https://inventory.example.com/api/v1 BOOKSTORE_API_TOKEN: bearer token (without the 'Bearer ' prefix) Assumed REST contract (adjust paths/fields here to match your API): GET /books?q=...&limit=...&offset=... -> JSON search results GET /books/{book_id} -> JSON book PATCH /books/{book_id}/stock body: {"stock": nonnegative integer} DELETE /books/{book_id} Updates and deletes may return JSON or an empty success response. Search pagination and response shape are supplied by the upstream API. Run with `python server.py` in an environment with the dependencies installed. Transport is stdio; stdout is reserved for MCP protocol messages. """ import os from contextlib import asynccontextmanager from dataclasses import dataclass from typing import Annotated, Any, AsyncIterator from urllib.parse import quote, urlsplit import httpx from mcp.server.fastmcp import Context, FastMCP from mcp.server.fastmcp.exceptions import ToolError from pydantic import Field @dataclass class AppState: client: httpx.AsyncClient @asynccontextmanager async def lifespan(server: FastMCP) -> AsyncIterator[AppState]: base_url = os.environ.get("BOOKSTORE_BASE_URL", "").strip() token = os.environ.get("BOOKSTORE_API_TOKEN", "").strip() if not base_url or not token: raise RuntimeError("Set BOOKSTORE_BASE_URL and BOOKSTORE_API_TOKEN.") try: parsed = urlsplit(base_url) valid = ( parsed.scheme in {"http", "https"} and bool(parsed.hostname) and parsed.username is None and parsed.password is None and not parsed.query and not parsed.fragment ) parsed.port # Validate a supplied port without exposing the URL. except ValueError: valid = False if not valid: raise RuntimeError("BOOKSTORE_BASE_URL must be an HTTP(S) API root without credentials, query, or fragment.") if any(ord(char) < 33 or ord(char) > 126 for char in token): raise RuntimeError("BOOKSTORE_API_TOKEN must contain printable ASCII without whitespace.") async with httpx.AsyncClient( base_url=base_url.rstrip("/") + "/", headers={"Authorization": f"Bearer {token}", "Accept": "application/json"}, timeout=httpx.Timeout(30.0, connect=10.0), follow_redirects=False, trust_env=False, ) as client: yield AppState(client=client) mcp = FastMCP("bookstore_mcp", lifespan=lifespan) BookId = Annotated[str, Field(min_length=1, max_length=256, description="Book identifier, e.g. 'book-123'.")] def book_path(book_id: str) -> str: if not book_id.strip() or book_id in {".", ".."}: raise ToolError("Provide a nonblank book identifier other than '.' or '..'.") return "books/" + quote(book_id, safe="") async def api_request( ctx: Context, method: str, path: str, *, params: dict[str, Any] | None = None, payload: dict[str, Any] | None = None, ) -> dict[str, Any]: state: AppState = ctx.r [... excerpt: first 3200 of 7129 characters]

查看 Pro