The pilot-to-production gap
Many organisations have run an AI pilot. Far fewer have taken one into production. This is a common pattern, not a sign that the technology does not work, and it usually comes down to what changes between a proof of concept and a live system, rather than the model itself.
Why pilots stall
A pilot is often built on clean, curated sample data that does not reflect how messy real operational data actually is. Once the pilot is judged a success, there is frequently no clear owner responsible for taking it further.
Plans for human oversight and error handling are often an afterthought, and the cost and governance implications of running the system continuously are rarely considered until late in the process.
What actually changes in production
Production means real data volume and real data mess, not a curated sample. It means integration with existing systems rather than a standalone demo, ongoing monitoring for drift and failure, and clear accountability for what happens when the system gets something wrong.
The three kinds of risk that matter
Not every AI use case carries the same kind of risk. Some carry decision risk, where a wrong output leads directly to a bad business decision. Some carry reputational risk, where an unexpected or inappropriate output is seen by a customer. Others carry operational risk, where the system fails or degrades and a downstream process depends on it.
Naming which of these applies to a given use case makes it much easier to decide what oversight and monitoring it actually needs, rather than applying the same level of caution everywhere.
What monitoring in production actually means
Monitoring an AI system is not the same as monitoring a conventional application. Alongside uptime and response time, it means tracking whether the quality of outputs is drifting as real-world data changes, watching for inputs that fall outside what the system was designed to handle, and reviewing a sample of real outputs regularly, not just when something is reported as wrong.
Budgeting for the real cost of running AI
The cost of an AI solution does not end at deployment. Ongoing costs include compute for inference, the time spent monitoring and reviewing outputs, periodic retraining or tuning as data and business conditions change, and the human review built into the process for higher-risk decisions.
Organisations that only budget for the build phase are frequently surprised by what it costs to keep a system reliable once it is in daily use.
A practical route to production
Start with a use case that has a clear owner and a measurable outcome, not just an interesting capability. Validate performance on real data, not a cleaned-up sample, and design human oversight in proportion to the risk of the use case.
Plan for ongoing monitoring and improvement as a permanent part of running the system, not a one-off project that ends at go-live.
A practical checklist
- The use case has a named owner accountable for it beyond the pilot stage.
- Performance has been validated on real operational data, not a curated sample.
- The type of risk the use case carries (decision, reputational or operational) has been identified explicitly.
- Human oversight is matched to that risk, not applied as a single blanket standard.
- Ongoing monitoring and the cost of running the system in production are budgeted for, not just the build.