Measuring AI ROI without fooling yourself

Almost every AI business case contains the line ’saves X hours per week’. It is the easiest number to produce and the easiest to mislead yourself with. Hours saved turn into value only when something else changes because of them.

Three kinds of benefit

• Cash. Costs that actually leave the business: overtime, temporary staff, outsourcing, penalties, write-offs. These go straight into the case.
• Capacity. Time freed that lets the same team handle more volume or do work that was being skipped. Real, but only if the volume or the skipped work exists.
• Quality. Fewer errors, faster replies, better compliance. Often the largest effect and the hardest to price. Count it through its consequences: fewer credit notes, higher conversion, lower churn.

A sound business case states which of the three it is counting and does not add them together.

Common ways to fool yourself

Counting minutes saved across two hundred people as if they added up to headcount. Twelve minutes a day per person is not five full-time jobs. It is two hundred slightly less rushed people, which may be worth having but is not a saving.

Measuring the pilot team only. Volunteers are your keenest and most capable users. Expect half the effect at full rollout until training catches up.

Leaving out running costs. Model usage, hosting, monitoring, the person who maintains the test set and the time people spend reviewing output all belong in the case.

Comparing against a bad week. Baselines taken during a backlog flatter any intervention. Use at least four ordinary weeks.

What we put in every case

• A baseline measured before the pilot, over at least four weeks.
• One primary metric and at most two secondary ones.
• Full running cost, including review time.
• A payback date, and the name of the person who owns it.
• A measurement at ninety days after go-live, reported whatever it shows.

The last point matters most. Projects whose results are checked at ninety days tend to be the ones that were scoped honestly at the start.