mcpscore 1.11.0: smoke-test your MCP tools in CI

The normal mcpscore audit is deliberately non-invasive. It connects, negotiates MCP, collects catalogs, and judges the server’s declared contract without invoking tools. That makes it appropriate for auditing third-party servers, but listing alone cannot prove that a handler still returns what its schema promises.

Smoke mode closes that specific gap for servers you operate. Add --smoke and mcpscore runs the normal scored audit first, then calls eligible tools over the same session. One command now checks both the server’s MCP quality and the runtime behavior most likely to drift during a refactor or dependency upgrade.

# Local Python or Node.js server
uvx mcpscore==1.11.0 ./server.py --smoke

# Any-language stdio server
uvx mcpscore==1.11.0 --smoke --stdio ./my-go-server

This is real execution. Use it in your own development and CI environments—not against someone else’s production server. The web service remains non-invasive and does not offer smoke mode.

First, a tool that declares an outputSchema is called and its real structuredContent is validated against that schema. This catches the quiet contract drift where the handler changes but the schema does not. Even an explicit empty schema still requires structured content to be present.

Second, mcpscore synthesizes deliberately invalid arguments and requires the tool to reject them. Third, it calls a name proven absent from the complete tool catalog and expects a JSON-RPC error. Accepting either request is a failure; hanging, closing the connection, or returning an internal error is a crash, not successful validation.

Inputs are derived deterministically from the schema—defaults, constants, examples, enums, and type zero-values. No LLM chooses arguments, so repeated CI runs remain explainable and reproducible.

By default, smoke mode calls only tools annotated readOnlyHint: true. Unannotated tools are skipped because the MCP default is not read-only. Each skip remains visible, turning missing annotations into actionable coverage information without gambling on side effects.

If the target is an isolated test environment and every tool is safe to execute, --call-all provides a second, explicit level of consent. It is never implied by --smoke. The unknown-tool check is similarly defensive: mcpscore derives a name absent from the collected catalog and skips the check if that catalog is incomplete.

Smoke verdicts are pass, fail, or skip. A valid call that cannot be judged because an upstream is unavailable skips rather than failing your build—someone else’s outage should not be reported as your MCP regression. Failures in behavior the server owns exit with code 4.

uvx mcpscore==1.11.0 ./server.py --fail-under 80 --smoke

The existing score gate keeps exit code 3 and takes precedence when both gates fail. Smoke results live in their own section of the JSON report and never change score or max_score. That separation matters: quality scoring is deterministic, while live tool calls depend on credentials, test data, and upstream services.

The release also fixes a HIGH rule that accidentally let tools with absent or whitespace-only descriptions pass. Those are now treated as missing, matching prompts and resources. Scores can move for affected servers because the most common description defect is finally judged correctly.

The MCP Python SDK moves from 2.0.0 to 2.1.1. Its client fixes let pre-2026 sessions tolerate later cache-hint fields and parse boolean subschemas inside tool properties, turning some former listing failures and insufficient-data skips into real audits.

Modern-only stdio servers can now produce full audits, and three new HIGH rules verify that tools, resources, and prompts do not vary across independent MCP 2026-07-28 connections. The documentation also adopts “Primitives” for the category that covers all four primitive types; report IDs and scores are otherwise unchanged.

Read the complete release notes, inspect the package on PyPI, and use the smoke-mode guide to add the check to CI. Start with read-only tools; opt into --call-all only in an environment designed for it.

For non-invasive checks, keep using the CLI without --smoke or paste a remote endpoint into mcpscore.dev. Our MCP testing tools comparison explains where an audit fits beside Conformance, Inspector, and security scanners.