I see this regularly, and it is rarely the model's fault. A pilot answers one question: can the model do this task? Production asks several others, and most of them were never on the pilot's agenda.
A pilot runs once. Production runs every day.
In a demo, the model gets a handful of well-chosen cases and does well on them. In production it gets thousands, including the strange ones. And it gets them again tomorrow, with a slightly different prompt, slightly different data and possibly a different model version.
Language models are not deterministic. The same question can get a different answer on a second run. So the useful question shifts from "can it do this?" to "how often does it get this right, over many runs?" Nine times out of ten may be fine for a draft email that a person reads before sending. It is not fine for a change to an order in your ERP.
That shift, from one good answer to a reliable rate, changes what you need to build.
Start with the test set
Martin Fowler's catalogue of patterns for GenAI products starts here, and it is the step pilots skip most often. Before you tune anything, collect a fixed set of real cases with answers you agree on. Run every change against it: a new prompt, new data, a new model version.
Without that set, quality is judged by people looking at results and saying "that's good". Every change becomes a gamble, and you find out from complaints. With it, a model upgrade is a routine check instead of a leap. That matters, because providers retire model versions on their own schedule. The model your pilot ran on may not be available in a year.
Building the set is dull work. It is also the one thing that makes everything else in this article manageable.
Autonomy is earned, in stages
Most pilots jump straight from demo to live. There is a better route, and Pramod Sadalage and Prem Chandrasekaran describe it well on Martin Fowler's site: hand over autonomy in stages.
- Shadow mode. The model recommends. People review the recommendation and still do the work themselves.
- Supervised. The model prepares the action. A person approves it before anything happens.
- Autonomous with guardrails. The model acts on its own, within limits you have defined and checks you can see.
- Full autonomy. The model acts, and people check samples.
Each stage produces the evidence for the next one. And each stage is a safe place to stop. A feature that stays supervised for good can still save a great deal of time.
One warning about the supervised stage. Measure how often people change what the model prepared. If that rate drops to zero, either the model is very good or people have stopped reading. Only a regular sample tells you which.
LLM features fail softly
Ordinary software fails loudly: an exception, a timeout, an alert. A failing LLM feature gives a fluent, plausible, wrong answer, and your monitoring sees a 200 OK. That needs a different kind of monitoring: sampling outputs, running the test set on a schedule, watching how often people correct results.
Sometimes it does fail loudly, and pilots rarely meet those cases either. Rate limits, a provider outage, a step that times out halfway through a longer process. Decide up front what happens then. Fall back to another model or provider. Make longer processes resumable from the step that failed, instead of starting over; Thoughtworks' Technology Radar now flags ignoring this as a pitfall for agent workflows. And let failed items land somewhere a person can see them, instead of disappearing.
Cost per task, and an owner
Twenty enthusiastic users make a pleasant invoice. Two thousand users, plus a nightly batch job someone added because it was easy, make a different one. Token prices say little. Cost per handled case, per document, per answer is what a business owner can weigh against the benefit.
And somebody has to own it. Pilots are usually owned by an enthusiast. Production needs a product owner who decides what "good enough" means, and an operations team that knows what failure looks like for this kind of system.
None of this is exotic. It is the production readiness any business-critical system needs: monitoring, recovery, ownership, cost control. What differs is that the usual signals don't fire.
Write the exit criteria down first
A longer pilot won't close the gap. What helps is writing the exit criteria down before the pilot starts:
- the success rate on a fixed set of real cases, over repeated runs;
- the cost per task at expected production volume;
- what the feature does when the model is slow, unavailable or wrong;
- the stage of autonomy it starts in, and what evidence moves it to the next;
- who owns it after go-live.
If you can't write those down, you are running a demo. That is fine, as long as everyone knows it. Demos are good at building enthusiasm and poor at producing decisions.
A pilot proves the model can do the work. Production proves the organisation can live with it.
Back on track
If you have a pilot that has been almost ready for a while, the gap is usually smaller than it feels, and it is seldom the model. Most of the time it comes down to the questions above, answered with numbers. I am happy to go through them with you, on your own case.
