A pilot is proven against the baseline, not in a demo. A manufacturer's stock and reorder exceptions, measured before and after
A demonstration is the seller's best day. The data has been chosen, the examples have been rehearsed, and nobody in the room has to do their normal job while watching it. A pilot is your ordinary week: the data as it really is, the interruptions, the person who knows the work checking every answer. After Pick one AI idea, not twelve, there is one job, one owner and a baseline. This step is about proving, or disproving, that the job is worth keeping.
A typical case
Take a manufacturer that holds raw materials and finished goods across a few sites. Each week a planner reviews the reorder exceptions report: say 1,200 item-and-location rows where stock is below its reorder point, above its maximum, or moving faster or slower than its history says it should. The numbers are illustrative; the shape is common. Most rows need nothing. A few need an order placed today, and a few more point to stock that will sit on the shelf for a year. The planner's skill is telling them apart, and on a busy week some get missed.
Suppose this business ran the step-two test and the job that passed is narrow: rank the exceptions, put the ones that need action at the top, and give a short reason for each: "usage up sharply over four weeks, supplier lead time longer than cover". The planner still decides.
The demonstration runs on last month's data and finds everything. It flags the shortage that stopped a line, and the overstock that was written off. That should not impress anyone. Last month's data is the one set where the answers are already known, and it is easy, without meaning to, to tune a demonstration until it finds them.
What a pilot measures
A pilot runs on work that has not happened yet. Same planner, same weeks, both ways, in a fixed order: each week the planner works the report their own way first and notes what they found, then opens the ranked list, so the list cannot shape the planner's own count. For six weeks, four numbers are recorded.
Exceptions found that needed action, checked afterwards against what actually happened. False alarms, the rows ranked high that needed nothing. Minutes per review, from the planner's own timings. Exceptions only one way caught: those the list caught that the planner missed, and the reverse, each followed up over the six weeks to see whether it became a shortage or a write-off.
Each is counted the same way the baseline was counted in step one. If the baseline came from a tally sheet, the pilot uses the same tally sheet. A different counting method produces a different number, and the difference gets credited to the tool.
Write the verdict rules first
Before the pilot starts, the owner and I write down three outcomes and the numbers that trigger each. Keep: found every exception the planner found, plus kept false alarms to an agreed level, say no more than one in five of the top fifty, and cut review time by a third. Change: found the exceptions but with too many false alarms, so the thresholds are adjusted and the pilot runs again, once. Stop: missed exceptions the planner found, or saved no time.
This is the discipline of a registered clinical trial, where the measure of success is fixed before any results are in. The reason is simple. Once a pilot has run, there is always a number that looks good. If the goalposts move after the fact, every pilot "succeeds", gets rolled out, and then quietly dies when the ordinary weeks show what the chosen number hid.
Stop is a respectable result. Six weeks and a written reason to stop is cheaper than a year of something nobody trusts.
Why most pilots never leave the demo
The GenAI Divide, a report from MIT's NANDA project dated July 2025, drew on interviews with 52 organisations, a survey of 153 senior leaders and a review of over 300 public AI initiatives. It estimated that only 5% of integrated AI pilots were extracting real value, while the vast majority were stuck with no measurable effect on profit and loss. It pointed mostly at the approach rather than the technology, including tools that did not fit day-to-day operations. That finding fits Gartner's July 2024 prediction, cited in step two, that at least 30% of generative AI projects would be abandoned after proof of concept.
In my view the usual causes are plainer still. The pilot ran on demonstration data, so nothing it found was new. There was no baseline, so nobody could say what it was better than. And only the wins were measured: the shortage it caught gets talked about, the thirty false alarms the planner had to clear do not.
Checklist
- The baseline from step one, written down.
- A parallel run on real, current work, by the person who does it.
- The same counting method as the baseline.
- Keep, change and stop rules written before the start.
- A written verdict at the end, whichever way it goes.
This is step three of five. Before it: Pick one AI idea, not twelve. Next: AI belongs inside the workflow, not beside it. All five steps are on the AI page.