mcp-builder (anthropics/skills)

Name: mcp-builder (anthropics/skills) review
Item: mcp-builder (anthropics/skills)
Rating: 21.6
Author: Agent Skills

mcp-builder: lifts MCP servers from 3.6 to 4.6/5 by adding naming, annotations and async, but a one-sentence checklist scores 4.9 for 68% of the tokens

SituationalSafe

21.6 / 30

Lift over no skill: +1.04

Situational: the total 21.6/30 is under 22. Recommended needs a total of 22/30, output quality of 3.5, and a lift of +1.0.

SOTA

Pass

Tested with Codex.

Covers Codex, Claude Code

Codex CLI 0.166 (SOTA tier): 18 graded runs. A pass here also covers Claude Code under our tier rule; it says nothing about the second tier.

Scores

Each dimension is 0 to 5 and the six add up to the total out of 30. 5 is better on every dimension. Cost 5 means cheap and fast. Install 5 means easy.

Trigger reliability: 5.0
Lift over no skill: 3.0
Output quality: 4.6
Token and time cost: 3.0
Install and portability: 3.0
Safety: 3.0

Security review

Safe

Static scan: 8 CRITICAL / 13 HIGH / 7 MEDIUM. Manual review: Safe to run in a no-network sandbox (scanner findings explained; bundled scripts never executed). A clean scan does not prove a skill safe.

Install

npx skills@latest add anthropics/skills --skill mcp-builder

Source

Pinned commit: ca45fc496a3436251f191354091d557da31c15b5

Tested: 2026-11-02

What we tested

Python MCP server for a bookstore API — Write server.py: an MCP server in Python for a bookstore inventory REST API (base URL from the environment, bearer token from the environment). Tools: search books, get one book, update stock for a book, delete a book. Use the official mcp Python SDK (FastMC…
TypeScript MCP server for a weather API — Write src/index.ts and package.json: an MCP server in TypeScript for a weather REST API (base URL and API key from the environment). Tools: get current weather for a city, get a 5-day forecast, list saved locations, delete a saved location. Use the official MC…
Repair a hastily written MCP server — server.py is an MCP server for our warehouse API that works but was written in a hurry. Fix it so it follows good MCP server practice. Edit server.py in place and keep it talking to the same API. There is no network and no package installation here; just write…

Pros

Lifts the mean score from 3.57 to 4.61 over no skill (+1.04).
The skill was picked up without being named: Skill file read in 6/6 treatment runs.
High absolute quality with the skill: 4.61 out of 5.

Cons

A one-sentence prompt scored 4.88, against 4.61 with the skill (-0.27).
Uses 1.51x the tokens of no skill.
n=2 per task per arm; differences under about 1 point are noise.

Review

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below). Three tasks: write a Python MCP server for a bookstore REST API, write a TypeScript MCP server for a weather API, and fix an existing quick-and-dirty server.py to follow good practice. Nothing needs the network (the agent cannot install the MCP SDK; it writes code only). The skill group's prompt is the same as control.

Three-arm comparison

Arm	Mean score (0-5)	Tokens per run
Without the skill	3.57	73k
One-sentence prompt	4.88	75k
With the skill	4.61	110k

Measured with Codex CLI over 18 runs, scored by rule-based checks with no LLM judge. Total 21.6 / 30, verdict Situational.

Verdict: Situational (21.6 / 30). Source: anthropics/skills, skills/mcp-builder, pinned at commit ca45fc4. Install: npx skills@latest add anthropics/skills --skill mcp-builder. Licence: Proprietary: anthropics/skills LICENSE.txt (source-available; evaluation only, not redistributed).

What it is

The mcp-builder skill from anthropics/skills: a guide for building Model Context Protocol servers (research and plan, implement, review and test, write evaluations). It ships language guides for Python (FastMCP) and TypeScript, MCP best-practice notes and two helper scripts for running evaluations against a server. Proprietary licence (source-available).

How we tested

Codex CLI 0.166 (gpt-6-astra, low reasoning) on a dedicated Windows test PC, a fresh temporary project and temporary HOME/TMPDIR/caches per run, skill installed (treatment) or absent (control) or absent with the skill's core requirement squeezed into one sentence appended to the prompt (one-liner group), 2 runs per group per task, network off, /tmp writes blocked. The prompts never mention the skill (except the wording noted below).

Three tasks: write a Python MCP server for a bookstore REST API, write a TypeScript MCP server for a weather API, and fix an existing quick-and-dirty server.py to follow good practice. Nothing needs the network (the agent cannot install the MCP SDK; it writes code only). The skill group's prompt is the same as control.

Python MCP server for a bookstore API — Write server.py: an MCP server in Python for a bookstore inventory REST API (base URL from the environment, bearer token from the environment). Tools: search books, get one book, update stock for a book, delete a book. Use the official mcp Python SDK (FastMC…
TypeScript MCP server for a weather API — Write src/index.ts and package.json: an MCP server in TypeScript for a weather REST API (base URL and API key from the environment). Tools: get current weather for a city, get a 5-day forecast, list saved locations, delete a saved location. Use the official MC…
Repair a hastily written MCP server — server.py is an MCP server for our warehouse API that works but was written in a hurry. Fix it so it follows good MCP server practice. Edit server.py in place and keep it talking to the same API. There is no network and no package installation here; just write…

Scoring is programmatic (hidden tests / planted-issue recall / constraint checks / render+OCR where relevant), 0–5 per run. No LLM judge decided any score.

Results

The one-liner given to the third group: "Design the MCP server like an expert: consistent service-prefixed, action-oriented tool names (e.g. bookstore_search_books), typed input schemas (Pydantic or Zod) with described fields, annotations (readOnlyHint, destructiveHint, idempotentHint), paginated and size-limited results, actionable error messages, secrets only from environment variables, async I/O, and nothing printed to stdout on stdio transport."

Task	Control score	One-liner	With skill	Control tokens	One-liner tokens	With-skill tokens	Skill read
t1-python	1.43	4.64	4.2	60k	64k	97k	2/2
t2-typescript	4.29	5.0	4.64	60k	55k	100k	2/2
t3-fix	5.0	5.0	5.0	97k	106k	133k	2/2

Mean score: 3.57 without vs 4.61 with the skill (delta +1.04). Tokens per run: 73k vs 110k (1.51x); time 80s vs 89s. One-liner group: 4.88 (the skill's gain over the one-liner: -0.27; one-liner's own gain over no skill: +1.31); one-liner tokens 75k per run, so the skill costs 1.47x the one-liner.

What happened

Where it matters, the skill helps: on the two from-scratch servers control missed the basics (tool names with a shared service prefix 2/4 runs, tool annotations such as readOnlyHint 2/2 Python runs, async tools 2/2 Python runs, actionable error messages 2/2 Python runs). Mean score 3.57 control, 4.61 skill.
The one-sentence group did better than the skill: 4.88, with one missed check in 18 runs, while the skill group missed 5. The sentence lists the same properties the checks look for, so it is an upper bound for a short prompt, but it costs 75k tokens per run against 110k for the skill and 73k for control.
A real defect in the skill group: both TypeScript treatment runs produced a createServer() for the new @modelcontextprotocol/server package but never started a transport (no stdio or Streamable HTTP), so the server file cannot be launched as written; control and one-liner runs did include a transport.
On the fix-it-up task all three groups scored the same (all 9/9 checks in all six runs): given an existing server, plain Codex already applied the practices.
The skill is heavy to load: SKILL.md plus reference guides; treatment runs left about 12 skill files in the project's .agents folder and read the guides, which is where the 1.5x tokens go. Its helper scripts (evaluation harness) were not run in the offline sandbox.

Dimension scores

Dimension	Score	Why
1_output_quality	4.61	Mean score 4.61/5 with the skill (programmatic checks, see per-task table).
2_lift_vs_no_skill	3.04	+1.04 points vs 3.57 without the skill (n=2 per task; under ~1 point is noise).
3_trigger	5.0	Skill file read in 6/6 treatment runs.
4_instructions_portability	3	Large reference set, split across language guides; helper scripts need Python packages and a live server.
5_efficiency	3	1.51x tokens (110k vs 73k per run) and 1.11x time (89s vs 80s).
6_safety_hygiene	3	Scanner flags 8 CRITICAL / 13 HIGH, all read in context: sample server code and env passing in helper scripts; proprietary licence.

Limits of this test

Rule-based checks on code text; none of the generated servers was actually started or connected to a client.
No network: the agent could not install or import the MCP SDK, so API details (for example the new v2 TypeScript package) were not validated.
The skill's evaluation step (question generation and running them against the server) was not exercised.
n = 2 per group per task: differences under about 1 point are noise.
Rule-based scoring measures what the checks can see; it cannot judge taste or whether a result is the best possible.

Security

See security.md. Scan: 8 CRITICAL / 13 HIGH / 7 MEDIUM. Verdict: safe to run (no-network sandbox). A clean scan does not prove a skill safe.

Bottom line

For a from-scratch server the skill clearly beats an unguided agent on naming, annotations, async and error handling, at 1.5x tokens. A short checklist sentence got the same properties for fewer tokens and without the missing-transport defect, so install it for the reference guides and consistency, not for a quality jump over a good prompt.

Transcript

Run log for mcp-builder: Codex CLI, network off, per-run temp HOME/TMPDIR, one row per graded run.
Columns: task, arm-run, score (0-5), tokens, seconds, checks passed, skill files the agent opened.

t1-python control-1: score 1.79, tokens 51610, secs 53.2, checks 4/9, opened nothing
t1-python control-2: score 1.07, tokens 69259, secs 98.6, checks 3/9, opened nothing
t1-python oneliner-1: score 5.0, tokens 74072, secs 101.3, checks 9/9, opened nothing
t1-python oneliner-2: score 4.29, tokens 54692, secs 126.2, checks 8/9, opened nothing
t1-python treatment-1: score 3.39, tokens 101809, secs 72.0, checks 6/9, opened mcp-builder
t1-python treatment-2: score 5.0, tokens 93275, secs 66.9, checks 9/9, opened mcp-builder
t2-typescript control-1: score 4.29, tokens 52126, secs 59.1, checks 8/9, opened nothing
t2-typescript control-2: score 4.29, tokens 69042, secs 61.5, checks 8/9, opened nothing
t2-typescript oneliner-1: score 5.0, tokens 55412, secs 104.3, checks 9/9, opened nothing

18 runs, 3 artifacts

Sample output

SOTA · With the skill: server.py (t1-python, run 1, Codex)

"""Bookstore MCP server (Python 3.10+, official mcp SDK, httpx, Pydantic 2). Required environment variables: BOOKSTORE_BASE_URL: API root, e.g. https://inventory.example.com/api/v1 BOOKSTORE_API_TOKEN: bearer token (without the 'Bearer ' prefix) Assumed REST contract (adjust paths/fields here to match your API): GET /books?q=...&limit=...&offset=... -> JSON search results GET /books/{book_id} -> JSON book PATCH /books/{book_id}/stock body: {"stock": nonnegative integer} DELETE /books/{book_id} Updates and deletes may return JSON or an empty success response. Search pagination and response shape are supplied by the upstream API. Run with `python server.py` in an environment with the dependencies installed. Transport is stdio; stdout is reserved for MCP protocol messages. """ import os from contextlib import asynccontextmanager from dataclasses import dataclass from typing import Annotated, Any, AsyncIterator from urllib.parse import quote, urlsplit import httpx from mcp.server.fastmcp import Context, FastMCP from mcp.server.fastmcp.exceptions import ToolError from pydantic import Field @dataclass class AppState: client: httpx.AsyncClient @asynccontextmanager async def lifespan(server: FastMCP) -> AsyncIterator[AppState]: base_url = os.environ.get("BOOKSTORE_BASE_URL", "").strip() token = os.environ.get("BOOKSTORE_API_TOKEN", "").strip() if not base_url or not token: raise RuntimeError("Set BOOKSTORE_BASE_URL and BOOKSTORE_API_TOKEN.") try: parsed = urlsplit(base_url) valid = ( parsed.scheme in {"http", "https"} and bool(parsed.hostname) and parsed.username is None and parsed.password is None and not parsed.query and not parsed.fragment ) parsed.port # Validate a supplied port without exposing the URL. except ValueError: valid = False if not valid: raise RuntimeError("BOOKSTORE_BASE_URL must be an HTTP(S) API root without credentials, query, or fragment.") if any(ord(char) < 33 or ord(char) > 126 for char in token): raise RuntimeError("BOOKSTORE_API_TOKEN must contain printable ASCII without whitespace.") async with httpx.AsyncClient( base_url=base_url.rstrip("/") + "/", headers={"Authorization": f"Bearer {token}", "Accept": "application/json"}, timeout=httpx.Timeout(30.0, connect=10.0), follow_redirects=False, trust_env=False, ) as client: yield AppState(client=client) mcp = FastMCP("bookstore_mcp", lifespan=lifespan) BookId = Annotated[str, Field(min_length=1, max_length=256, description="Book identifier, e.g. 'book-123'.")] def book_path(book_id: str) -> str: if not book_id.strip() or book_id in {".", ".."}: raise ToolError("Provide a nonblank book identifier other than '.' or '..'.") return "books/" + quote(book_id, safe="") async def api_request( ctx: Context, method: str, path: str, *, params: dict[str, Any] | None = None, payload: dict[str, Any] | None = None, ) -> dict[str, Any]: state: AppState = ctx.r [... excerpt: first 3200 of 7129 characters]

See Pro