It worked once. That is observed, not proven.
Most costly surprises live in the gap between something being seen to work and something being proven to work. This matters more than ever with anything that has moving parts, or an AI in the loop. A simple rule closes the gap.
A thing that worked once has not been proven. It has been observed. There is a difference, and most costly surprises live in the gap between the two. This matters more than ever now, because anything with moving parts, and especially anything with an AI in the loop, is unpredictable by nature and will happily work perfectly on the demo and fail on the day.
Run it five times before you trust it
The rule is simple. Before you rely on a system, run it five times under real conditions. Five clean successes and you can reasonably assume it works. One or two failures in the five is not bad luck to be waved away, it is a signal that something is still unclear, and the useful move is to find out exactly what rather than ship and hope. The number is a threshold, not magic: enough repetitions to catch an intermittent failure, few enough that you will actually do it.
Why five
One success hides the intermittent failure
The failures that hurt are the ones that happen one time in four. A single test cannot see them. Five can, which is the whole point of the number.
A failure is a question, not a verdict
When one of the five fails, you have not wasted a run, you have found the unclear part while it was still cheap. That is the run doing its job.
This is not just for AI
The same discipline applies to any process you are about to depend on: a new hire's task, a handoff between two people, an automation, a checkout flow. Verify before you trust. And the sister rule for when something breaks: replicate the bug and confirm you understand it before you attempt a fix. A fix for a problem you cannot reproduce is a guess wearing a lab coat.
Observed once is not proven. Before you build on top of anything, put it through five real runs. It is a small tax that buys you out of the expensive category of surprise: the thing that worked in the demo and failed in front of the customer.
Building on something you have not really proven?
Knowing which systems to trust, and how hard to test them before you depend on them, is part of building a business that runs without you. The strategy call is free and it is a real diagnosis. If you are not there yet, the community costs nothing.