Series: Shipping AI Features Safely - Part 1 of 5. This part sets the foundation; the next four turn it into practice.
Let me describe a launch you may recognize.
Somebody wires a prompt to a model, connects a bit of retrieval, and runs it against twenty example inputs in a notebook. The answers look great. The demo in the sprint review gets a round of applause. Two weeks later the feature is live. Three weeks later a customer posts a screenshot of an answer that is confidently, beautifully wrong - and the team realizes it cannot answer the simplest question anyone will ask in the incident review:
How often does this happen?
Not “can it happen” - everyone knows it can. How often. Is it one answer in ten thousand or one in twenty? Did it start with last Tuesday’s prompt tweak? Is it getting worse? If you cannot answer those questions, you are not running a feature. You are running a slot machine with your brand painted on the front.
The demo proves almost nothing
Classic software gives us a comforting property: if it works for the test cases, it mostly works for the neighbors of those test cases. Logic is deterministic. A sort function that sorts twenty lists correctly is very likely to sort the twenty-first.
LLM features break that intuition in three ways:
- The input space is effectively infinite. Your twenty demo inputs were written by the people who built the feature. Real users phrase things in ways nobody on the team would ever think of.
- The output is probabilistic. The same input can produce different answers on different runs, and a small prompt edit can shift behavior on inputs you never looked at.
- Failures are silent. This is the dangerous one. A model that starts producing subtly wrong advice throws zero exceptions and returns zero
500s. Your error rate dashboard stays perfectly green while quality quietly erodes.
That last point is why the usual production signals - uptime, latency, error rate - are necessary but nowhere near sufficient. They tell you the feature is running. They tell you nothing about whether it is right.
What “production-ready” actually means
Here is the definition I use, and the one this whole series is built on. An AI feature is production-ready when you can do three things:
- Measure its quality - with numbers, on a defined set of cases, before and after every change.
- Trace its behavior - for any single answer, reconstruct what went in, what was retrieved, which tools ran, and what came out.
- Roll back a bad release - in minutes, without a code deploy, when the numbers move the wrong way.
Notice that none of those is about the model. You can swap models, providers, and prompting techniques all day long. Without these three capabilities, every one of those changes is a gamble - and with them, every change becomes an experiment you can evaluate.
The metrics that actually matter
“Accuracy” is the word everyone reaches for, and it is almost useless on its own because it means something different for every feature. Break quality into signals you can actually compute:
- Task success rate - did the output accomplish what the user needed? This is the north star, and it requires you to define success per feature (more on that below).
- Groundedness / citation coverage - for anything built on retrieval: are the claims supported by the sources you provided, and does the answer cite them?
- Schema validity rate - if downstream code consumes the output, how often is it valid, parseable, and inside the allowed values? This one is cheap and brutally honest.
- Refusal rate - in both directions. Refusing legitimate requests is a quality bug, not a safety feature.
- Hallucination proxy - you rarely measure hallucination directly, but you can measure proxies: unsupported claims, invented identifiers, references to entities that do not exist in your data.
- Latency and cost per successful task - not cost per request. A cheap call that fails and gets retried three times is not cheap. (Part 5 goes deep on this one.)
- Retry, fallback, and guardrail trigger rates - the “check engine” lights. When these climb, something upstream changed, even if every individual answer still looks fine.
You do not need all seven on day one. But you need to know which of them define “good” for your feature - and you need at least one of them computed automatically.
Write down what “good” means - in code
Most teams skip this step, and it is the one that makes everything else possible. Before you can measure quality, you need a definition of it that is specific enough to check.
Take a feature that summarizes support tickets for an agent dashboard. “A good summary” is not a definition. This is:
// ticket-summary.contract.js
// The definition of "good" for one AI feature - reviewed like any other code.
export const ticketSummaryContract = {
// Hard rules: violating any of these is a failure, no debate.
schema: {
required: ["summary", "category", "urgency"],
category: ["billing", "bug", "account", "feature-request", "other"],
urgency: ["low", "normal", "high"],
},
checks: [
out => out.summary.length <= 600 || "summary exceeds 600 characters",
out => !/as an ai/i.test(out.summary) || "meta-commentary leaked",
],
// The outcome that actually matters to the user.
success:
"A support agent can triage the ticket correctly without opening the original thread.",
};Two kinds of things live in that file. The hard rules are deterministic - a machine can check them on every output, forever, for free. The success definition is qualitative - it needs human judgment, or a carefully calibrated model judge. Separating them is the single most useful thing you can do, because it tells you exactly which checks belong in CI and which belong in production monitoring.
And note where this lives: in the repository, next to the prompt, reviewed in pull requests. The definition of quality is part of the feature, not a slide in someone’s head.
Why this is an engineering problem, not a data science problem
A common reaction is “evaluation is the ML team’s job.” I disagree, strongly.
Everything above - contracts, regression tests, tracing, gradual rollout, rollback - is the same discipline we already apply to every other part of the system. We would never ship a payment integration whose only test was a demo. We would never deploy a database migration without a way back. The novelty of LLMs is the probabilistic output, not the need for engineering rigor. If anything, they need more of it, because the failures are quieter.
The good news for developers: you already know most of the tools. Test suites, CI gates, structured logging, feature flags, canary releases. This series is mostly about pointing those familiar tools at a new kind of component.
The roadmap for this series
Each part builds on the previous one:
- This part - why the demo proves nothing and which metrics define “good”.
- Build a Golden Set - a curated set of real cases that gates every prompt and model change in CI, deterministic checks first.
- LLM-as-a-Judge - why a model grading a model is not a test, but a very good drift detector for production traffic.
- Trace Every Step - observability for LLM features: one trace per task, spans for every step, and no leaked personal data.
- Prompts Are Deployments - feature flags, canary rollouts, instant rollback, and cost per successful task as the metric that makes AI budgets steerable.
What you can do this week
You do not need a platform or a vendor to start. Three steps, in order:
- Pick one AI feature and write its contract: the hard rules and a one-sentence success definition.
- Collect twenty real inputs from production or support tickets - not ones you invent. Save them with the outputs your feature currently produces.
- Run the hard rules against those outputs and compute a single number: the schema validity rate. It is the cheapest metric there is, and it is almost always lower than the team assumes.
That number is your first instrument. Next time, we turn those twenty inputs into a golden set and wire it into CI.
Flying by instruments
Pilots learn to fly by looking out of the window. On a clear day it works beautifully: you see the horizon, the runway, the other aircraft. That is visual flight - and it is exactly what a demo is. Good conditions, a handful of known inputs, and everything you need to judge is right there in front of you.
Then the aircraft enters a cloud. The horizon disappears, and within seconds the inner ear starts lying - a gentle turn feels like level flight, a descent feels like a climb. Pilots who keep trusting their senses in a cloud get into serious trouble. The ones who survive have trained for the moment they stop believing their eyes and start believing their instruments: altitude, attitude, airspeed, heading.
Production is the cloud. Real traffic, real users, inputs nobody imagined - and your intuition about how the feature behaves is the inner ear: confident, convincing, and wrong. The metrics in this article are the instrument panel. Build it before you fly into the weather, not after the first passenger notices you have been descending for a while.