On this page
An MCP server has a public API that nothing compiles against. The tool listing it returns from tools/list is read at run time by an agent, turned into a prompt, and acted on by a model. Rename a parameter in a REST API and a client build breaks somewhere. Rename a parameter in an MCP tool and every build stays green; the first sign is an agent that starts failing, or one that keeps succeeding while doing something slightly different.
That listing is a contract, and a contract can be tested. This article builds the test for a Python server on the MCP SDK 2.3.0: take a snapshot of everything a client can learn without calling a tool, commit it, and have the pipeline compare each change against the snapshot on the base branch and classify it as breaking, compatible, or in need of a person. Then it puts the gate to work on twenty-one small changes of the kind people make in good faith, and replays recorded calls against each one to see what a client that was not told would experience.
The second part is the reason to build the first. Of the thirteen changes the gate calls breaking, eight fail loudly, with a tool error a model can read. Three return a successful result that no longer fits the schema the client was built against. Two change nothing that a call can observe. A gate that waits for a failing call would have let five of the thirteen through.
The problem
A tool definition carries a name, a description, an input schema, optionally an output schema, and a set of behaviour hints called annotations. In the Python SDK almost all of it is generated: the function name becomes the tool name, the docstring becomes the description, and the type hints, defaults and Field constraints become JSON Schema. That is convenient, and it means an ordinary refactoring is a contract change. Tightening a bound, renaming an argument for readability, or adding a member to a Literal all alter what agents are told.
%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FFF3E0','primaryTextColor':'#232F3E','primaryBorderColor':'#FF9900','lineColor':'#232F3E','clusterBkg':'#F8FAFC','clusterBorder':'#CBD5E1','edgeLabelBackground':'#FFFFFF','nodeTextColor':'#232F3E'}}}%%
flowchart TD
A["Python function:
name, type hints,
defaults, docstring"] --> B["SDK generates
JSON Schema"]
B --> C["tools/list result
with ttlMs and cacheScope"]
C --> D["Client caches
the listing"]
D --> E["Listing becomes part
of the model's prompt"]
E --> F["Model chooses a tool
and writes arguments"]
F --> G2["tools/call"]
G2 --> H["Server validates arguments,
runs the function"]
style A fill:#FFF3E0,stroke:#FF9900,stroke-width:2px,color:#232F3E
style C fill:#EBF8FF,stroke:#0073BB,stroke-width:2px,color:#004C74
style D fill:#F0FFF4,stroke:#1D8102,stroke-width:2px,color:#145202
style E fill:#F3E8FF,stroke:#7C3AED,stroke-width:2px,color:#4C1D95The 2026-07-28 revision of the protocol makes this more pressing, and also easier to test. It removed sessions and the initialize handshake, and it requires that the result of tools/list must not vary per connection. Every listing now carries a ttlMs freshness hint and a cacheScope, and servers are asked to return tools in a deterministic order so that clients can cache the list and model prompt caches keep hitting. A listing that is the same for every caller and designed to be cached is a document. Documents can be put under version control and compared.
| Part of the listing | Who depends on it | What goes wrong when it changes |
|---|---|---|
| Tool name | The model, and any allow-list or approval rule keyed on the name | Calls to the old name fail; approvals no longer match |
| Description | The model: it decides when to call the tool from this text | Behaviour shifts with no error anywhere |
| Input schema | The model, when it writes arguments; the server, when it validates them | Old-shaped calls are rejected, or quietly lose an argument |
| Output schema | Client code and downstream agents that read structuredContent | A field goes missing or a new value appears that nothing handles |
| Annotations | Clients that skip a confirmation prompt for read-only tools | A tool that now writes is still run without asking |
| Capabilities and protocol version | Client transports and SDKs | Features a client relied on disappear |
ttlMs and cacheScope | Client and gateway caches | A change takes longer to reach clients, or a private list is shared |
This sits beside two earlier pieces and does not repeat them. Hosting Remote MCP Servers Securely on AWS and Azure covers who may call a server. Evaluating Agent Performance covers whether an agent does its job. The question here is narrower and comes before both: has the thing the agent reads changed, and did anyone decide that it should?
Test server
The subject is a small server called stockroom, version 1.4.0, with synthetic data held in memory: five tools, one resource, one resource template and one prompt. Two of the tools are enough to show where the contract comes from.
@server.tool(annotations=READ_ONLY)
def get_stock(sku: Sku) -> StockLevel:
"""Return on-hand, reserved and available units for one product."""
return store.level(sku)
@server.tool(annotations=ToolAnnotations(readOnlyHint=False, destructiveHint=False, idempotentHint=False, openWorldHint=False))
def reserve_stock(
sku: Sku,
quantity: Annotated[int, Field(ge=1, le=100, description="Units to hold.")],
order_ref: Annotated[str, Field(min_length=3, max_length=40, description="The caller's order reference.")],
) -> Reservation:
"""Hold stock for an order. Returns status 'rejected' when there is not enough available."""
return store.reserve(sku, quantity)Nothing in that code mentions JSON Schema. The pattern on Sku, the bounds on quantity, the field list of StockLevel and the two docstrings are all published to every client. The annotations are stated by hand: get_stock is read-only, reserve_stock writes but does not destroy.
Contract snapshot
The snapshot tool connects the way a client does, over stdio or Streamable HTTP, and records the server identity, the protocol version, the capabilities, the cache hints, and the four listings. It walks pagination with a page limit so that a listing which never ends cannot hang a build, and it bypasses the SDK client’s own response cache, because a contract has to be read from the server.
async def every_page(call: Callable[..., Awaitable[Any]], attribute: str) -> tuple[list[dict[str, Any]], dict[str, Any]]:
"""Walk a paginated listing. Returns the entries and the cache hint the server attached to it."""
entries, cursor, hint = [], None, {}
for _ in range(MAX_PAGES):
page = await call(cursor=cursor, cache_mode="bypass") # a contract is read from the server, never from a cache
entries.extend(plain(item) for item in getattr(page, attribute))
if not hint:
hint = {key: value for key, value in (("ttlMs", getattr(page, "ttl_ms", None)), ("cacheScope", getattr(page, "cache_scope", None)))
if value is not None}
cursor = getattr(page, "next_cursor", None)
if not cursor:
return entries, hint
raise RuntimeError(f"{attribute}: more than {MAX_PAGES} pages; refusing to continue")
async def take(target: StdioServerParameters | str, timeout: float = 30.0) -> dict[str, Any]:
async with Client(target, read_timeout_seconds=timeout) as client:
capabilities = plain(client.server_capabilities)
if not capabilities.get("experimental"):
capabilities.pop("experimental", None) # 1.x SDKs send an empty object; it carries no promise
info = plain(client.server_info) if client.server_info else {}
snapshot: dict[str, Any] = {
"contractFormat": FORMAT,
"server": {"name": info.get("name"), "version": info.get("version")},
"protocolVersion": client.protocol_version,
"capabilities": capabilities,
"instructions": client.instructions,
"listingCache": {},
"tools": [], "resources": [], "resourceTemplates": [], "prompts": [],
}
cache = snapshot["listingCache"]
if "tools" in capabilities:
tools, cache["tools"] = await every_page(client.list_tools, "tools")
snapshot["tools"] = sorted((pick(t, TOOL_FIELDS) for t in tools), key=lambda t: t["name"])
if "resources" in capabilities:
resources, cache["resources"] = await every_page(client.list_resources, "resources")
templates, cache["resourceTemplates"] = await every_page(client.list_resource_templates, "resource_templates")
snapshot["resources"] = sorted((pick(r, RESOURCE_FIELDS) for r in resources), key=lambda r: r["uri"])
snapshot["resourceTemplates"] = sorted((pick(t, TEMPLATE_FIELDS) for t in templates), key=lambda t: t["uriTemplate"])
if "prompts" in capabilities:
prompts, cache["prompts"] = await every_page(client.list_prompts, "prompts")
snapshot["prompts"] = sorted((pick(p, PROMPT_FIELDS) for p in prompts), key=lambda p: p["name"])
return snapshotTwo details make the output usable as a contract. Entries are sorted by name and keys are sorted, so the file does not change when registration order does. And only declared fields are kept, so per-response metadata stays out. In the run behind this article, a snapshot over stdio, a snapshot over HTTP from a separately started process, and the committed file were byte-identical: 10,019 bytes, one SHA-256. This is the entry for get_stock, up to its output schema:
"description": "Return on-hand, reserved and available units for one product.",
"inputSchema": {
"properties": {
"sku": {
"description": "Stock keeping unit, for example FAS-0012.",
"pattern": "^[A-Z]{3}-\\d{4}$",
"title": "Sku",
"type": "string"
}
},
"required": [
"sku"
],
"title": "get_stockArguments",
"type": "object"
},
"name": "get_stock",
Compatibility rules
The classifier takes two snapshots and returns findings at three levels. Breaking means a client written against the old contract can now fail or be misled. Compatible means old clients are unaffected. Review means a machine cannot tell, and a person has to read the change.
The rules for schemas follow from one question: who has to tolerate whom? An input schema describes what the server accepts, so it may only get wider. Everything the old schema allowed must still be allowed. An output schema describes what the server produces, so it may only get narrower. Everything the new server returns must still fit what old consumers expected. The same edit therefore gets opposite verdicts on the two sides.
| Change | In an input schema | In an output schema |
|---|---|---|
| Add an enum value | Compatible | Breaking |
| Remove an enum value | Breaking | Compatible |
| Tighten a bound (lower maximum, higher minimum) | Breaking | Compatible |
| Loosen a bound | Compatible | Breaking |
integer to number | Compatible | Breaking |
| Add an optional property | Compatible | Compatible, unless the object was closed |
| Add a required property | Breaking | Compatible |
| Remove a property | Breaking | Breaking |
| Required to optional | Compatible | Breaking |
| Change a description, a default or a pattern | Review | Review |
In code, the direction is a single switch. Each comparison reports that the new schema allows less or allows more, and two small methods turn that into a level:
def narrowed(self, where: str, rule: str, detail: str) -> None:
"""The new schema allows less than the old one."""
self.add("breaking" if self.direction == "accepts" else "compatible", where, rule, detail)
def widened(self, where: str, rule: str, detail: str) -> None:
"""The new schema allows more than the old one."""
self.add("compatible" if self.direction == "accepts" else "breaking", where, rule, detail)Properties are the one place where the two sides are not mirror images, so they are written out:
def compare_object(self, old: dict[str, Any], new: dict[str, Any], where: str) -> None:
before, after = old.get("properties", {}), new.get("properties", {})
required_before, required_after = set(old.get("required", [])), set(new.get("required", []))
for name in sorted(before.keys() - after.keys()):
if self.direction == "accepts":
self.add("breaking", f"{where}.{name}", "parameter-removed", "callers that send it are rejected or silently ignored")
else:
self.add("breaking", f"{where}.{name}", "field-removed", "consumers that read it get nothing")
for name in sorted(after.keys() - before.keys()):
if self.direction == "accepts":
if name in required_after:
self.add("breaking", f"{where}.{name}", "required-parameter-added", "existing callers do not send it")
else:
self.add("compatible", f"{where}.{name}", "optional-parameter-added", "existing callers are unaffected")
elif old.get("additionalProperties") is False:
self.add("breaking", f"{where}.{name}", "field-added-to-closed-object", "the old schema forbade extra fields")
else:
self.add("compatible", f"{where}.{name}", "field-added", "consumers that ignore unknown fields are unaffected")
for name in sorted(before.keys() & after.keys()):
path = f"{where}.{name}"
if name in required_after and name not in required_before:
self.narrowed(path, "became-required", "optional -> required")
elif name in required_before and name not in required_after:
self.widened(path, "became-optional", "required -> optional")
self.compare(before[name], after[name], path)
closed_before, closed_after = old.get("additionalProperties") is False, new.get("additionalProperties") is False
if closed_after and not closed_before:
self.narrowed(where, "object-closed", "additionalProperties is now false")
elif closed_before and not closed_after:
self.widened(where, "object-opened", "additionalProperties is no longer false")Removing a property is breaking in both directions. On the output side a consumer loses a field. On the input side the caller’s argument is either rejected or ignored, and the experiment below shows which. Local $ref pointers are inlined before comparing, anyOf branches are matched by type so that adding or removing null is seen for what it is, and any keyword the rules do not know is sent to review, not guessed at.
Outside the schemas the rules are plain. A removed tool is breaking, and a rename is a removal plus an addition. A new read-only tool is compatible. A new tool that can write goes to review, because it widens what every connected agent may do. A changed description goes to review. Losing readOnlyHint is breaking, because clients that skip confirmation for read-only tools will go on skipping it. A removed capability is breaking. A changed protocol version or cache hint goes to review. Forty-six unit tests pin these rules down without starting a server.
The model reads a tool description as guidance, so changing it changes behaviour with no schema difference at all. It is also the surface used in tool poisoning: Invariant Labs showed in April 2025 that instructions hidden in a description can steer a model, and that a server can change a description after the user approved the tool. Their suggested control was a hash of the description. A contract file in version control is that hash with a readable diff attached, which is why the gate reports every description change and never waves one through as cosmetic.
Test results
Rules are easy to agree with in the abstract. To see what they are worth, the lab applies twenty-one single changes to the server, one at a time. Each is a few lines, like this one, which drops an argument the function no longer uses:
@variant("v12", "reserve_stock: remove parameter order_ref")
def _(server):
server.remove_tool("reserve_stock")
@server.tool(annotations=WRITES)
def reserve_stock(sku: Sku, quantity: Quantity) -> Reservation:
"""Hold stock for an order. Returns status 'rejected' when there is not enough available."""
return server.store.reserve(sku, quantity)For every variant the lab does two things. It takes a snapshot and asks the classifier for a verdict against the committed contract. Then it replays nine recorded calls, the calls an agent built against version 1.4.0 makes today, and checks each result against the output schema in the committed contract. The second step stands in for a client that was never told about the change.
| Change | Gate | What the recorded calls did |
|---|---|---|
Rename tool get_stock | Breaking | Tool error: Unknown tool: get_stock |
Remove tool list_warehouses | Breaking | Tool error: unknown tool |
get_stock: add required parameter | Breaking | Tool error: warehouse Field required |
get_stock: add optional parameter | Compatible | All nine as recorded |
get_stock: rename parameter sku | Breaking | Tool error: product_sku Field required |
reserve_stock: maximum quantity 100 to 20 | Breaking | Tool error on a reservation of 30 |
search_products: maximum limit 50 to 100 | Compatible | All nine as recorded |
search_products: remove a category | Breaking | Tool error on a search in that category |
search_products: add a category | Compatible | All nine as recorded |
reserve_stock: quantity integer to string | Breaking | Tool error on both reservations |
search_products: make limit required | Breaking | Tool error on a search without it |
reserve_stock: remove parameter order_ref | Breaking | All nine as recorded. The argument is accepted and dropped |
search_products: default limit 10 to 50 | Review | All nine as recorded |
get_stock result: remove field reserved | Breaking | Call succeeds. Result fails the committed schema: reserved is required |
get_stock result: add field updated_at | Compatible | All nine as recorded |
reserve_stock result: new status backordered | Breaking | Call succeeds. Status is not one of held, rejected |
search_products result: price integer to number | Breaking | Call succeeds. 1265.625 is not an integer |
reserve_stock: description now tells the model to reserve 100 | Review | All nine as recorded |
cancel_reservation: destructiveHint true to false | Review | All nine as recorded |
get_stock: no longer read-only | Breaking | All nine as recorded. |
New tool transfer_stock that moves stock | Review | All nine as recorded |
Thirteen breaking, four for review, four compatible. No change that the gate let through as compatible or review altered the outcome of a recorded call. The thirteen breaking changes divide three ways, and the division is the finding.

Eight are loud. The old call comes back as a tool result with isError set and a validation message that names the field. That is the good case: the specification asks clients to pass such errors to the model so that it can correct itself, and it allows a client to refetch the listing early when a call fails in a way that suggests the listing is out of date. One detail to know before you assert on it: the specification lists an unknown tool as a protocol error with code -32602, and in this run the SDK returned it as a tool result with isError. A test that depends on which of the two arrives will not travel between servers.
Three are quiet. A field removed from a result, a new enum value, and an integer that became a number all came back as successful calls. The SDK client validates structuredContent against the output schema, but it uses the schema from the live listing, and the server changed its schema and its output together. A server that agrees with itself tells you nothing about whether it still agrees with its consumers. Only the replay against the committed schema caught these.
Two are invisible. When order_ref was removed from reserve_stock, the old call still sent it, the server accepted the call, and the reservation was made with the caller’s order reference silently discarded. Nothing was logged. And when get_stock stopped being read-only, no call changed at all; the difference is whether a client that auto-approves read-only tools ought to have asked first. Neither can be found by running anything. They can only be found by comparing declarations.
Five of the thirteen breaking changes produced no failing call when an old client replayed its traffic against the new server’s own schema. End-to-end agent tests would have passed for all five. Compare the declarations, and validate results against the contract you committed, not the one the server is serving.
Build gate
A pull request can be wrong in two different ways, so the gate asks two questions. Is the committed contract fresh, meaning that it matches what the server built from this branch really serves? And is the committed contract compatible with the one on the base branch? The first question forces the author to regenerate the file, which puts the diff in front of the reviewer. The second decides whether the build passes.
%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FFF3E0','primaryTextColor':'#232F3E','primaryBorderColor':'#FF9900','lineColor':'#232F3E','signalColor':'#0073BB','signalTextColor':'#0F172A','actorBkg':'#FFF3E0','actorBorder':'#FF9900','actorTextColor':'#232F3E','noteBkgColor':'#FFF8E7','noteBorderColor':'#FF9900','noteTextColor':'#232F3E'}}}%%
sequenceDiagram
participant Dev as Developer
participant CI as Pipeline
participant Srv as Server from this branch
participant Base as Base branch
Dev->>CI: Pull request with code and contract file
CI->>Srv: Start, then list tools, resources, prompts
Srv-->>CI: Live snapshot
CI->>CI: Live snapshot equals committed file?
Note over CI: No: exit 2, regenerate and commit
CI->>Base: Read the contract file at the base commit
Base-->>CI: Base snapshot
CI->>CI: Classify base against committed
Note over CI: Breaking without a major version: exit 1
CI->>Srv: Replay recorded calls
CI-->>Dev: Verdict and review list in the job summarydef main() -> int:
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("--base", required=True, help="snapshot from the base branch")
parser.add_argument("--head", required=True, help="snapshot committed on this branch")
parser.add_argument("--live", help="snapshot just taken from the running server")
parser.add_argument("--json", help="write the findings to this file")
options = parser.parse_args()
try:
base, head = load(options.base), load(options.head)
live = load(options.live) if options.live else None
except (OSError, json.JSONDecodeError) as error:
print(f"cannot read snapshot: {error}", file=sys.stderr)
return 3
if live is not None and live != head:
drift = compare(head, live)
print("STALE SNAPSHOT: the running server no longer matches the committed contract.")
print(report(drift) if drift else "The difference is outside the compared fields; regenerate the snapshot.\n")
print("Run contract/snapshot.py, commit the result, and let the reviewer see the diff.")
return 2
findings = compare(base, head)
text = report(findings)
print(text)
if options.json:
with open(options.json, "w", encoding="utf-8") as handle:
json.dump([f.__dict__ for f in findings], handle, indent=2)
summary = os.environ.get("GITHUB_STEP_SUMMARY")
if summary:
with open(summary, "a", encoding="utf-8") as handle:
handle.write("## MCP contract\n\n" + text)
if worst(findings) != "breaking":
print(f"PASS: {sum(f.level == 'review' for f in findings)} change(s) to review, none breaking.")
return 0
was, now = major(base["server"].get("version")), major(head["server"].get("version"))
if was is not None and now is not None and now > was:
print(f"PASS: breaking changes are covered by the major version bump {base['server']['version']} -> {head['server']['version']}.")
return 0
print(f"FAIL: breaking change without a major version bump (server version {base['server'].get('version')} -> {head['server'].get('version')}).")
return 1The policy is deliberately small. A breaking change passes only when the server’s major version has gone up, which turns “we decided to break clients” into something that has to be written down. Review findings never fail the build by themselves. They are printed and written to the job summary, so the pull request shows a person exactly which descriptions, defaults and hints to read. In the run, the five situations a pull request can be in produced the five expected exit codes:
| Situation | Exit code | Outcome |
|---|---|---|
| Nothing changed | 0 | Pass |
| Code changed, contract file not regenerated | 2 | Fail: stale snapshot, with the difference listed |
| A description changed | 0 | Pass, with one change listed for review |
| A maximum lowered from 100 to 20, version still 1.4.0 | 1 | Fail: breaking change without a major version bump |
| The same change released as 2.0.0 | 0 | Pass: covered by the version bump |
As a pipeline job it is five steps. This is the job that runs on the sample repository itself:
contract-gate:
runs-on: ubuntu-latest
timeout-minutes: 10
defaults:
run:
working-directory: mcp-contract-testing
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # the base commit's contract has to be readable
- uses: actions/setup-python@v5
with:
python-version: "3.13"
- name: Install
run: python -m pip install --quiet -r requirements.txt
- name: Snapshot the server as built
run: python contract/snapshot.py --stdio "python server/app.py" --out "$RUNNER_TEMP/live.json"
- name: Fetch the contract from the base commit
run: |
base="${{ github.event.pull_request.base.sha || github.event.before }}"
git show "$base:mcp-contract-testing/contract/mcp-contract.json" > "$RUNNER_TEMP/base.json" 2> /dev/null \
|| cp contract/mcp-contract.json "$RUNNER_TEMP/base.json" # no earlier contract: nothing to compare with yet
- name: Gate
run: python contract/gate.py --base "$RUNNER_TEMP/base.json" --head contract/mcp-contract.json --live "$RUNNER_TEMP/live.json"
- name: Replay the recorded calls
run: python contract/replay.py --stdio "python server/app.py" --contract contract/mcp-contract.jsonThe fallback in the fetch step matters for the first commit, when there is no earlier contract to compare with. After that the job has a real base. In the sample’s own history the second commit added the cache hints to the snapshot format, and the job reported four changes for review and passed.
Replay tests
The classifier checks what the server says. The replay checks that it does what it says, using a short file of example calls with the outcome each one should have: a result, or a tool error. It is the MCP version of a consumer-driven contract test, and the examples are best taken from what your agents really send.
async def replay(target: StdioServerParameters | str, contract: dict[str, Any], calls: list[dict[str, Any]],
timeout: float = 30.0) -> list[dict[str, Any]]:
schemas = {tool["name"]: tool.get("outputSchema") for tool in contract["tools"]}
rows: list[dict[str, Any]] = []
async with Client(target, read_timeout_seconds=timeout) as client:
for call in calls:
row = {"id": call["id"], "tool": call["tool"], "expected": call["expect"], "detail": ""}
try:
result = await client.call_tool(call["tool"], call["arguments"])
except MCPError as error: # the protocol refused the request: unknown tool, bad params
row.update(outcome="protocol-error", detail=" ".join(str(error).split())[:160])
except Exception as error: # noqa: BLE001 the SDK's own client-side validation, a dropped connection
row.update(outcome="client-error", detail=f"{type(error).__name__}: {' '.join(str(error).split())[:140]}")
else:
if result.is_error:
row.update(outcome="tool-error", detail=first_text(result))
else:
row["outcome"] = "ok"
schema = schemas.get(call["tool"])
if schema is not None:
problems = sorted(Draft202012Validator(schema).iter_errors(result.structured_content), key=lambda e: list(e.path))
if problems:
where = "/".join(str(p) for p in problems[0].path) or "(root)"
row.update(outcome="contract-violation", detail=f"{where}: {problems[0].message[:120]}")
row["as_recorded"] = row["outcome"] == call["expect"]
rows.append(row)
return rowsRecording the expected errors is as useful as recording the successes. Two of the nine calls expect a tool error, one for a malformed SKU and one for a quantity of zero. If a later change loosens validation, those calls start succeeding and the replay reports it.
SDK upgrades
The contract is generated, so the generator is part of it. To measure that, the lab ran the same server file under four installations of the MCP Python SDK and compared each contract with the one from 2.3.0. The file needs a two-line import fallback to do this, because the class called FastMCP in 1.x is MCPServer in 2.x.
| Installation | Result | Upgrading to 2.3.0 changes |
|---|---|---|
| SDK 1.9.4 with dependencies as of this month (pydantic 2.14.2) | Did not start: the SDK imports a private pydantic name that is gone | Not applicable |
| SDK 1.9.4 with dependencies as of June 2025 (pydantic 2.11.7) | Protocol 2025-03-26, no output schemas | Tool listing differs: five output schemas appear, a MIME type is declared, four capabilities are added. None breaking, one for review |
| SDK 1.26.0 | Protocol 2025-11-25 | Tool listing byte-identical. Four capabilities added, protocol version for review |
| SDK 1.30.0 | Protocol 2025-11-25 | Same as 1.26.0 |
| SDK 2.3.0 | Protocol 2026-07-28 | The committed contract |
Two things come out of this. Moving from a recent 1.x release to 2.3.0 did not change one byte of the tool listing for this server, which is reassuring and is now a measured fact, not a hope. What did change is around the tools: the server reports listChanged for all three listings and subscribe for resources, and a client negotiates the stateless 2026-07-28 revision. Those go to review because they are promises the SDK is now making on your behalf.
The older installation shows the other case. A server written before output schemas were generated gains five of them on upgrade. The gate rates that compatible, which is right, and from then on those five schemas are a commitment that the next refactoring can break. The first row is a reminder of a different kind: a server pinned to an old SDK with unpinned dependencies may not start at all, and a snapshot step that fails with a clear message is a cheap way to find that out in a pull request.
Automated dependency pull requests touch no application code, so a path filter on the server folder will skip them. Trigger the contract job on changes to the lock file as well. An SDK or schema-library upgrade is exactly the change where nobody is looking at the tool definitions.
Rolling out breaking changes
Sometimes the contract has to break. The gate does not forbid that; it makes it a decision. The usual sequence is the one used for database schemas: widen first, move the callers, narrow later.
%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FFF3E0','primaryTextColor':'#232F3E','primaryBorderColor':'#FF9900','lineColor':'#232F3E','clusterBkg':'#F8FAFC','clusterBorder':'#CBD5E1','edgeLabelBackground':'#FFFFFF','nodeTextColor':'#232F3E'}}}%%
flowchart TD
A["Change to a tool"] --> B{"Gate verdict"}
B -- "compatible" --> M["Merge"]
B -- "review" --> R["A person reads the
description, default or hint"]
R --> M
B -- "breaking" --> Q{"Can it be
made additive?"}
Q -- "yes" --> ADD["Add the new tool or
optional parameter beside the old"]
ADD --> DEP["Mark the old one deprecated
in its description"]
DEP --> WAIT["Wait for agents to move,
and for cached listings to expire"]
WAIT --> REM["Remove in the next
major version"]
Q -- "no" --> MAJ["Raise the major version
and tell the consumers"]
style M fill:#F0FFF4,stroke:#1D8102,stroke-width:2px,color:#145202
style R fill:#F3E8FF,stroke:#7C3AED,stroke-width:2px,color:#4C1D95
style ADD fill:#EBF8FF,stroke:#0073BB,stroke-width:2px,color:#004C74
style MAJ fill:#FEF2F2,stroke:#D13212,stroke-width:2px,color:#991B1B
style REM fill:#FEF2F2,stroke:#D13212,stroke-width:2px,color:#991B1BThe waiting step has a number attached. A client may treat a listing as fresh for ttlMs milliseconds, and the specification is explicit that this is a hint about refetching and not a promise that the data will stay the same. In the run the SDK sent ttlMs of 0 and a cacheScope of private on every listing, so a well-behaved client refetches each time it needs the list. If you raise the TTL to cut traffic, you also lengthen the period in which clients work from the old contract after a deployment. That is why the snapshot records the cache hints and the classifier sends any change to them for review.
The protocol can tell a client that a listing has changed, through a list_changed notification or an expired TTL, and a client may refetch after an unexpected error. It has no notion of whether the new listing is compatible with the old one. That judgement is made in your pipeline or it is not made.
Cost
The gate adds seconds, not minutes. On a GitHub-hosted runner the snapshot took 1.59 seconds, almost all of it starting the server process. The comparison took 0.05 seconds and the replay of nine calls 1.68 seconds. It calls no model and no cloud service, so it has no usage cost and no secrets to manage for a server that starts without credentials. The contract file is about 10 KB of JSON in the repository.
The cost that matters is attention. Each review finding asks a person to read a sentence or a hint, and a team that sees twenty of those on every pull request will stop reading them. Keep the list short by keeping descriptions stable, and treat a description rewrite as its own change. The file of recorded calls also needs an owner: it is only worth as much as its resemblance to what agents send.
Production considerations
- Listings that vary by caller. The specification allows the tool set to depend on the authorization presented with the request. If yours does, one snapshot is not the contract. Take one per role or scope set that you support, and gate each.
- Authenticated servers. Snapshot from inside the pipeline with a credential the pipeline already holds, against a build of the branch, not against production. The snapshot contains declarations and no data.
- Tools registered at run time. A server that builds its tool list from a database or a plugin folder has a contract that depends on configuration. Snapshot it with the configuration you ship, and treat a configuration change as a code change.
- Gateways and aggregators. Where a gateway merges several servers, gate each server in its own repository and snapshot the gateway as well, since name collisions and prefixes are decided there.
- Third-party servers. The same snapshot and comparison work against a server you do not own. Run it on a schedule and alert on any difference; for a server outside your control every change deserves a look, including the compatible ones.
- What it cannot see. A tool that keeps its declaration and changes its behaviour passes the classifier. The replay narrows that gap and does not close it. Agent evaluation still has its place.
If an agent’s tools can change clinical, financial or personal data, the contract diff in the pull request is useful change-control evidence: it shows what the agent was permitted to do before and after, and who approved the difference. It is not evidence that the tool behaves safely. That still needs its own testing and, in a clinical setting, a safety assessment.
Decision framework
| Your situation | What to adopt | Why |
|---|---|---|
| One team owns the server and its only agent, released together | Snapshot and freshness check | The diff in review is most of the value when both sides ship at once |
| Other teams or customers build agents against your server | The full gate with the major-version rule, plus replay | You cannot see their code, so the contract is the only thing you can hold steady |
| You consume a server you do not own | Scheduled snapshot and comparison, alert on any change | Description and annotation changes are a security signal as well as a compatibility one |
| Tool list depends on the caller’s scopes | One snapshot per supported role | The listing is per authorization, so the contract is too |
| Server pinned to an SDK you rarely upgrade | Gate dependency pull requests | Generated schemas and declared capabilities move with the SDK |
| Tools change weekly during early development | Snapshot only, no failing gate yet | A gate on a contract nobody depends on yet is friction without benefit |
Key takeaways
- An MCP tool listing is a public contract generated from type hints and docstrings, so ordinary refactoring changes it. Put a canonical snapshot of it in version control.
- Input schemas may only widen and output schemas may only narrow. The same edit, such as adding an enum value, is safe on one side and breaking on the other.
- Of thirteen breaking changes tested, eight failed loudly, three returned results that no longer fit the old output schema, and two changed nothing a call could observe. Compare declarations; do not wait for a failing call.
- Description, default and annotation changes cannot be classified by a machine. Route them to a person and keep the list short enough to be read.
- The gate costs about three seconds a build. Run it on dependency updates as well as code changes, and require a major version for anything breaking.
If you have gated an MCP server in a different way, or found a class of change these rules miss, leave a comment. The awkward cases are the ones worth collecting.
Everything above ran in GitHub Actions on a hosted Linux runner with Python 3.13.18, MCP Python SDK 2.3.0, pydantic 2.14.2 and jsonschema 4.26.0, with unpinned dependencies resolved as they stood on 4 November 2026: 46 unit tests for the rules; snapshots over stdio and Streamable HTTP compared byte for byte with the committed file; nine recorded calls replayed; the gate in five situations with its exit codes; 21 variants of the server, each snapshotted, classified and replayed; and the same server file under SDK 1.9.4, 1.26.0, 1.30.0 and 2.3.0. The contract-gate job ran on the sample repository against its own previous commit. Timings are single measurements on a shared runner. Not run: any model or agent, so nothing here measures how a reworded description changes behaviour; servers built with other SDKs or languages; an authenticated server; a listing that varies by caller. Statements about the protocol come from the 2026-07-28 specification. The run is here.
The complete sample for this article is on GitHub: nithinmohantk/blog-samples-working-repo/mcp-contract-testing. Clone it with git clone https://github.com/nithinmohantk/blog-samples-working-repo.git and open the mcp-contract-testing folder. Its README says how to run it and what has been verified.
Glossary
- Contract snapshotโ A canonical file holding everything a client can learn from an MCP server without calling a tool.
- Input schemaโ The JSON Schema of a tool’s arguments. The server accepts what it describes, so it may only get wider.
- Output schemaโ The JSON Schema of a tool’s
structuredContent. The server produces what it describes, so it may only get narrower. - Tool annotationsโ Hints such as
readOnlyHintanddestructiveHint. The specification says clients must treat them as untrusted unless the server is trusted. - Tool execution errorโ A tool result with
isErrorset, meant to be shown to the model so that it can correct itself. - Protocol errorโ A JSON-RPC error for a request the server could not process at all.
- ttlMsโ How long, in milliseconds, a client may treat a listing as fresh. A hint, not a guarantee.
- cacheScopeโ Whether a cached listing may be shared between callers (
public) or not (private). - Consumer-driven contract testโ A test in which examples of what consumers really send are replayed against the provider.
- Rug pullโ A server changing a tool’s description or behaviour after a user has approved the tool.
- Expand and contractโ Shipping a breaking change in steps: add the new shape, move the callers, remove the old one.
References & Further Reading
- → MCP specification 2026-07-28: Toolsโ Tool definitions, output schemas, annotations and the two kinds of error
- → MCP specification 2026-07-28: Cachingโ What ttlMs and cacheScope mean and how they interact with notifications
- → MCP specification 2026-07-28: Key changesโ The stateless redesign, cacheable listings and deterministic tool order
- → MCP Python SDK: migration guide, v1 to v2โ FastMCP becomes MCPServer, the new Client, and what moved
- → mcp on PyPIโ Release history used to pin SDK versions for the comparison
- → Invariant Labs: Tool Poisoning Attacksโ Hidden instructions in tool descriptions, and descriptions that change after approval
- → Hosting Remote MCP Servers Securely on AWS and Azureโ The authorization side of running an MCP server
- → Sample code for this articleโ Server, snapshot tool, rules, gate, replay, the 21 variants and the CI script
Discover more from C4: Container, Code, Cloud & Context
Subscribe to get the latest posts sent to your email.