A pilot proves that something is possible. Production proves that it is dependable. Those are different problems, and most organisations budget only for the first.
In the projects we are called into after a stalled pilot, the model is almost never the blocker. The blockers are consistent enough to be predictable.
There is no definition of correct
A pilot is judged by demonstration. Someone types a question, the output looks impressive, everyone agrees it is promising. Nobody has written down what a correct answer is for the two hundred cases that actually occur.
Without that definition you cannot tell whether a prompt change helped or hurt. Teams end up tuning by anecdote, and confidence erodes with every unexplained regression.
If you cannot measure a change, you are not engineering. You are redecorating.
Nobody owns the failure path
Demos succeed. Production fails, regularly and in ways that need an owner. What happens when retrieval returns nothing? When the model is rate-limited? When a user asks something outside the intended scope?
- Define refusal as a first-class outcome, not an error state
- Route low-confidence cases to a named human, not a generic queue
- Log the full context of every failure so it can be replayed
- Alert on rate-of-refusal, because a silent rise means something upstream broke
The system is not operable
A pilot lives on one engineer’s machine or in a notebook. Production needs deployment, monitoring, cost controls and a runbook that somebody other than the author can follow at two in the morning.
This is ordinary software engineering, and it is the part most AI pilots skip entirely. It is also the part that decides whether the thing survives its first month.
What we do differently
We build the evaluation set before the feature. It is usually a few hundred real cases with an agreed correct outcome, and it runs on every change. It is unglamorous and it is the single highest-leverage artefact in an AI project.
type EvalCase = {
id: string;
input: string;
expected: { intent: string; shouldEscalate: boolean };
};
// A change ships only when it does not regress the suite.
const report = await runSuite(cases, candidate);
if (report.accuracy < baseline.accuracy) {
throw new Error(`Regression: ${report.accuracy} < ${baseline.accuracy}`);
}Everything else follows from that. Once quality is measurable, model choice becomes an optimisation rather than a debate, and the conversation moves from whether the system works to how much of the process it should own.