How to tell if your MCP server is actually good

FastMCP 4 just shipped, and it runs on the modern, stateless protocol. In a recent live session, Nate Nowack, who maintains FastMCP, and Wayne Grigsby, lead solutions engineer at Prefect, worked through what it takes to build a good MCP server on top of it.

What follows is an edited version of that conversation.

What "stateless" actually changes

The protocol is now stateless, and it is worth being precise about what that buys you. MCP behaves more like a typical web standard: you can horizontally scale a server without the weird persistent connections that used to cause so much trouble. FastMCP is on the modern Streamable HTTP protocol, and the change underneath it is that the protocol dropped sessions.

To see why that matters, it helps to remember what the two sides are. An MCP client is something like Claude Code, ChatGPT, or Codex. An MCP server is just a server that encodes some domain-specific logic. The Linear MCP and the GitHub MCP are the familiar examples: someone took part of an API a service already offers and re-presented it through an interface that is friendly for agents.

Historically, when a client connected to a server, it got a session ID, and that ID let the server know who was calling on every subsequent request. If you then wanted to horizontally scale that server, you needed sticky routing so that each caller kept landing on the replica that held their session. The new protocol removes that. It is now incumbent on the client to send everything the server needs to handle a request, so any replica can answer any request.

The clearest place to feel the change is elicitation. Say you ask an agent to book a flight to Europe and something is unspecified, like when you actually want to go. Previously, elicitation paused the bidirectional connection between client and server and passed that question back down to the user, holding the exchange open in the meantime. Now there is nothing to hold open. The server responds that it does not have enough information and ends the request. The client gathers the missing detail, then sends a fresh request carrying all the state it has accrued so far, so that the new request is independently valid on its own. That is one example, but the pattern shows up everywhere once sessions go away.

Why reach for MCP at all instead of a skill

There was a stretch where the honest question in the room was whether MCP was dead. Skills had shown up, and a lot of people decided skills were simply the better option and could not see why anyone would still want MCP. That framing always felt backwards. Skills are supportive of the MCP story, not a replacement for it.

Some of the confusion came from a reasonable-sounding instinct: why put anything in front of an API at all? Why not hand the agent a rich skill that contains all of your documentation, export your OpenAPI spec, drop it in, and let the agent figure out the calls on its own? That made more sense when tokens were cheaper and nobody minded letting an agent read through a giant skill to work out what to do.

What that approach misses is that a skill is not a gap-proof process. An agent will read a skill, agree that it is a great idea, and follow it as best it can. But a skill is not a programmatic way to enforce what happens first and what happens second. It is guidance, not structure, and for a lot of workflows the structure is the point.

MCP is to an agent what a website is to a human

Jeremiah put the useful version of this in a talk a while back. MCP is to an agent what a website is to a human. As people, we do not reach into a raw API to get things done. There is an interface in front of the API, the interface drives some transactions on a backend, and the API eventually writes to a database. Given that, it does not really make sense to hand an agent direct access to the API either. MCP is the interface that sits in that same spot for the agent.

So the verb that matters when you build one is curating. You curate a process, and you stay focused on the outcome you are actually after. Start from the outcome you expect and work backwards. That is the advice we give everyone starting out with FastMCP: figure out how to distill a process, and be willing to collapse a whole set of backend API calls into a single tool call, because what you care about is the outcome that tool represents, not the shape of the API behind it.

"Isn't this just a REST API with extra steps?"

You hear this a lot, and part of why is that the clients have been a bit bad. Outside of Claude Code, which comes from the same people who made MCP, plenty of clients only support tools. They do not support resources or prompts or the other pieces of the protocol, which makes the whole thing look like tool calling with more steps.

There is a second reason the comparison sticks. These models were trained on an enormous amount of humans talking to REST APIs, so that is what they are comfortable doing, and left alone they will keep doing it. But that misses the curation point, and it misses auth and governance too. You would not necessarily apply the same authentication boundaries to an agent service that you apply to your REST API, and once you start thinking about composing several MCP servers together, you are in territory that would not make sense for REST APIs at all. Between the curation, the governance boundary, and hosting providers like Cloudflare and Horizon, the criticism starts to dissolve. The bad clients made it look more substantial than it is, and that is changing as clients get better.

When not to write an MCP server

Right after making the case for MCP, it is worth being honest that there are still plenty of times it is the wrong engineering choice, even as the set of good cases keeps growing.

A personal example: in a normal dev workflow, it is tempting to hand Claude a browser MCP, something like the Chrome DevTools MCP, so it can go poke around a page. In practice the Playwright CLI turned out to be better for that job. It is just a cleaner fit. Another example is uploading a big blob, like an audio or video file, to some platform's API. You do not want to base64-encode the literal audio and push it through an LLM or a tool call. MCP does not have the right shape for that.

The failure mode is treating MCP as the new shiny thing and spamming the button on every use case. When the choice is not attentive to the actual problem, you end up forcing a square peg into a round hole, and that is exactly the kind of thing that makes MCP easy to criticize. If a CLI or a REST API is genuinely the better fit, use it. What is worth reaching for MCP for is the well curated, minimal interface you would give specifically to an agent, one that is deliberately different from your CLI and your REST API.

Which raises a fair question: would you ever write an MCP server as a curation layer on top of a CLI that is good on its own but does not compose a process the way you want? Absolutely. CLIs are sometimes a direct reflection of the API. The Kubernetes CLI is close to a one-to-one image of the REST API. But people also compose API calls inside their CLIs, and the same curation logic applies one level up. Look at the human workflows that grow around a CLI or an API, the sequence of steps a person performs ritualistically, and give the MCP author the chance to collapse that sequence into one interface an agent can drive on the person's behalf. That is the core design question every time: what are humans, now working through agent intermediaries, actually trying to do against my API, and how do I present a concise interface so the agent can do it for them.

Don't map your REST API straight onto MCP, and here is the token bill

Jeremiah wrote a fairly dogmatic piece a while ago: do not map your REST API directly onto an MCP server. It is bad, and you should feel bad. That is not bad advice, and it is a little funny in hindsight, because around the same time FastMCP shipped a utility that converts a FastAPI app straight into an MCP server. That is the natural instinct when you start, but it does none of the curation work, none of the thinking about how people actually compose calls to your API.

There is a public repo that makes this concrete. It defines a few versions of the same server. The first is the thing we recommend against: use the OpenAPI conversion to turn a spec directly into an MCP server. The second is the real Prefect MCP server, where every tool was hand-written for the purpose. There is a third option we will get to later.

If you count the token footprint of each definition, the direct mapping is stark. Naively mapping every endpoint to a tool produces an enormous surface, and that surface is exactly what a client like Claude Code has to read and understand before it can use the interface at all. This is the original criticism of MCP in one measurement: massive token bloat, and the reflex to just hand the agent a CLI instead. Being a little more intentional cuts it way down. The tradeoff is that it becomes your responsibility to expose the functionality people actually want, the outcomes rather than the endpoints. A slimmer set of tools on the server means far fewer tokens for the client to carry, and that is a huge difference.

The practical way to get there is to start from zero tools. Write one, use it for real, then add the next. Adding ten at a time tends to backfire, because you often find the client never touches most of them.

Middleware: controlling what comes back

Middleware is where a self-hosted server gets interesting, because it gives you a level of control over what your server actually returns. You can filter data, and you can restrict what ends up in an agent's context window. This comes up constantly in adoption conversations. You might trust a person with access to Salesforce and still not want to give an agent the same access, because you cannot fully predict what it will request, and once data lands in an agent's context there is no pulling it back.

A concrete case: a team had several Google Drives, and some folders in them needed to stay off limits. When someone hits the Google MCP and asks for a directory listing, middleware keeps the restricted directories out of the response, so they never enter the context window and users who should not see them never do. The team did not want to manage that access in Google. They wanted to enforce it in the MCP layer, and middleware is just Python, so they could. The same idea extends to stripping PHI or PII, or to augmenting responses on the way out.

There is an auth use case too. The Prefect MCP server uses middleware so that a hosted instance can store no authentication information of its own. You set headers in your client with your API token, and the middleware pulls that auth out of the request and forwards it. That way you can run a hosted instance without baking environment variables into the server process, and callers just set headers in their own clients.

Tool bloat and code mode

Code mode was popularized by Cloudflare, and the idea is simple. Instead of a growing pile of tools, you expose roughly two, a search tool and an execute tool. You have a well-typed SDK surface, in TypeScript or similar, that the model can search over to find the right methods. The model then writes code against those methods and passes it to the execute tool, which runs it in a sandbox, and the right things happen. If you have used Cloudflare's MCP, it just does the right thing, quickly.

Code mode is justified there by the sheer complexity and surface area of Cloudflare's API. It can be overkill elsewhere. What it is really fighting is the pull to keep adding tools because each new outcome your users want seems to demand one more. FastMCP has code mode baked in through a code mode import, with Pydantic handling the Python sandboxing under the hood. In TypeScript land, Cloudflare's isolates work well, and there are other options. It is still experimental, so treat the guardrails as guardrails. It is not something many people have adopted yet, and it makes the most sense for the genuinely large surfaces, but it is one more lever for shrinking the token footprint of a server.

Worth a small detour on how to see that footprint. Add the Prefect MCP to Claude Code locally, open it up, and you get a nice view of how context gets consumed, a legend showing how much of the window each thing takes. MCP tools show up as loaded on demand, because good clients do progressive disclosure over toolsets rather than dumping everything up front. It is a genuinely useful diagnostic. If a server feels heavy, run something like this, or have Claude build a small utility to report the token footprint, and you will quickly see where the weight is.

Plugins: shipping the MCP server and the skills together

Plugins combine the two worlds we have been circling. You have an MCP server, and you have skills, plus whatever system dependencies a skill asks the agent to use. The naming from both OpenAI and Anthropic around all this is confusing, but in general a plugin means an MCP server bundled with the skills that belong alongside it.

The Prefect plugin is a good illustration. There is a plugin structure with some metadata, roughly like the classifiers section of a pyproject.toml, and a marketplace file that specifies which skills are included. One of those skills exists to help the agent choose between the read-only MCP server and the CLI. In a client with a terminal, the CLI is often the easier path for mutations, precisely because we did not naively map every endpoint onto the MCP server. So the skill sits next to the server and says, in effect: if you want to read run states and similar things, here are some streamlined read tools; if you want to flip a schedule from paused to active, here is how you do that with the CLI.

The payoff is distribution. When a new teammate joins, you send them a link, they run two commands for Claude or two commands for Codex, and both the MCP servers and the skills install into their harness. The harnesses differ in whether that install is local to a project or global, and there is real minutia there, but the shape is the same: two commands take someone from zero to set up the way your org wants.

One thing worth passing along from submitting the Prefect plugins to the official OpenAI and Anthropic marketplaces: do not have your tools refer to each other. A tool description that says to use some other tool first will get your application categorically denied. We ran into a soft version of this with a single orientation tool meant to help the agent get its bearings, which sat right on the line. Good to know before you submit.

Evals: writing down what the server should get right

Two things push you toward evals. Clients have historically not been great, supporting only tools or ignoring parts of the protocol, and your own server implementation will keep changing over time. Against both, you want a stable set of expectations for what your server does, concrete enough that, for instance, your support team can state plainly what a given tool is supposed to accomplish.

Here is how that turns into an eval. A customer's automation is failing, and the agent cannot figure out why. That is an eval you should write. You are asserting that an agent equipped with your MCP server can diagnose that automation, both today, given how the server is implemented right now, and next week, after you change the tool. Evals are the static list of expectations you hold of an agent when it has your server.

In the repo, that looks like a set of evals you can run across different LLM providers. They range from the simple, an agent can identify the last failing run or list flows, up to the harder cases, why an automation did not fire, or whether it can diagnose rate limiting in a cloud environment. Every one of those started as a support request, a moment where it would have been much nicer to diagnose the problem quickly. Each eval sets up an initial world, a Prefect server with some number of runs in a particular state, runs the agent with the MCP attached, and checks whether it reaches the right conclusion.

The checking has two flavors. Some expectations are deterministic. If it is capture-the-flag and the agent gets the flag, you assert on the flag and you are done, no LLM required. Others are fuzzier, like whether the agent reached its conclusion the right way. There you look over the trace and let a strong model judge it. A weaker model, say Sonnet or Fable, runs the task with the MCP, and a stronger model like Opus reads the trace and decides whether it got there the way you wanted, without being overly prescriptive about the exact tool calls, since those are fungible and will shift as the implementation changes.

This runs in CI. On every pull request against the Prefect MCP server, all the evals run against the updated server. The harness is built on Pydantic AI and emits OpenTelemetry spans to Logfire, so a public trace of a CI run shows the whole thing: roughly a million input tokens and eighteen thousand output tokens across the evaluation, the models used, the tool calls that happened, and a cost of about three dollars for the run. Logfire has an MCP of its own, similar to Datadog, so you can even point an agent at the spans if you want.

The mental model is worth keeping. Imagine dropping an agent, Severance-style, into an empty room with your MCP server and one initial condition, and telling it what you expect it to do. Write those expectations down. Next week you can write new tools, but you had better still satisfy last week's expectations. A lot of MCP engineering now is exactly this, working from outcomes rather than procedures, which is part of a broader shift from a procedural world to a declarative one that has only gotten more interesting since Dagster joined Prefect.

Deployment: getting it into the real world, then locking it down

There are two kinds of agent deployment to keep straight. There are client proxies, the agent living in Claude Desktop or Codex or wherever, and there are standalone agents that run server-side, dropped into an agent workflow. Both need access to your MCP servers, so you have two deployment requirements to satisfy.

The server itself is the straightforward one. You can deploy it over HTTP into any cloud. Spin up a VM and deploy. It has been run on Railway, on Fly, and on AWS, and as long as it is reachable from the public internet, or from within whatever enclosed environment you need, you are in business. That is the first step to getting something into the real world.

Once it is out there, some questions land on you. How do you limit access? You cannot deploy to the public internet, hand someone a URL ending in /mcp, and shrug when that URL leaks, because now everyone has your server. You want to lock it down, and often you want to identify who is calling it, which is where authentication comes in.

A clean way to see this is a standalone agent. Take a Pydantic AI agent and give it an MCP toolset that has been deployed into Horizon, which is Prefect's MCP gateway product and the answer when what you need is governance, since it handles deployment along with a set of governance features. You add the toolset, it is embedded right in the agent, and now the agent can take its instructions, reach the MCP server as needed, and do the job. A nice addition with Pydantic AI is durability. Early on, a tool call that failed would sometimes just stop the agent, or, more maddening, the agent would pretend the call had not failed and answer as though it had the data. Adding durability, retries and backoff on the actual API calls, smooths over those edge cases. The questions to keep asking are the same ones every time: is it durable, is it secure, does it need authentication, does it need to sit on a closed network, and what do you actually need to know about whoever is calling to give them the right experience.

Two smaller things worth stealing. There is a gitless deployment pattern, deploying without git by uploading a zip straight to Horizon, and there is now a skill for it. If you want a new MCP server, a skill encodes your preferred norms and design patterns, you say go make it, and it builds the server, uploads the zip directly to the API, and hands back a URL you can give to people or to your own agents. It is a real quality-of-life improvement.

And auth is genuinely easy to add. A small sample MCP server shows it: wire up a GitHub OAuth provider, and accessing the server routes you through FastMCP's authentication. You sign in, and the resulting credential tokens gate access so you only reach what you are allowed to reach. The same holds for whatever identity provider you use, or for straight token authentication if that is all you need.

So, is your server actually good?

That is the whole point of the exercise. Curate for outcomes instead of mapping endpoints, measure your token footprint, put filtering and auth in middleware, reach for code mode only when the surface earns it, bundle skills alongside the server as a plugin so people install one thing, write down your expectations as evals and run them in CI, and deploy behind real access controls. The public repo, the skill for uploading to Horizon, and a collection of the heuristics from Jeremiah's talk are all worth a look if you want to go deeper. Do these, and you can actually judge whether your MCP server is any good, rather than guessing.