Why most AI pilots stall after the demo

We are often called in after a pilot has stalled. The story is nearly always the same. A small team built something impressive in a few weeks. It was shown to management, who liked it. Then it met the real process, and it has been ’nearly ready’ ever since.

The model is almost never the problem. In the stalled pilots we have reviewed over the past three years, the cause fell into one of five groups.

1. Nobody wrote down the baseline

If you do not know how long the task takes today, how often it goes wrong and what that costs, you cannot say whether the pilot worked. ’People liked it’ is not a result. A pilot without a baseline produces enthusiasm, and enthusiasm does not survive a budget round.

2. The demo ran on clean examples

Pilots are usually tested on the cases the team had to hand, which are the tidy ones. Production is the scanned fax, the supplier who puts the invoice number in the footer, the customer who writes in two languages in one email. A system that handles 95 percent of clean cases may handle 60 percent of real ones.

3. There was no plan for being wrong

Every AI system makes mistakes. The useful design question is what happens next. If the answer is ’someone will notice’, the process owner will, rightly, refuse to sign off. Confidence thresholds, a review queue and a log are what turn a clever demo into something an operations manager can be accountable for.

4. Integration was left for later

A pilot that needs a person to copy text in and out of it has not automated anything. Connecting to the mail server, the ERP and the permission system is slow, unglamorous work, and it is where most of the calendar time goes. Teams that leave it to the end discover the project is three times bigger than they budgeted.

5. It belonged to nobody

Innovation teams build pilots. Operations teams run processes. If the person who owns the process was not in the room from the first week, the pilot has no home to go to.

What to do instead

• Write the baseline before you write any code: volume, handling time, error rate, cost.
• Build a test set from real, messy cases, a few hundred at least, and agree the target with the process owner.
• Design the review queue first. It tells you where the system has to be right and where it can ask.
• Do one integration in the pilot, even a crude one, so the size of the rest is visible early.
• Set a go or no-go date. A pilot that cannot fail cannot succeed either.

None of this is sophisticated. It is ordinary project discipline applied to a technology that makes it tempting to skip. The pilots that reach production are usually the dull ones.