Nov03
Why successful pilots can hide the conditions that fail at scale
A successful AI pilot can create powerful momentum. The technology performs well, users are enthusiastic, the business case looks promising, and leaders start asking how quickly they can expand the solution.
That is exactly when a different question becomes important.
What made the pilot work?
The answer is not always the technology alone.
Pilots often operate under unusually favourable conditions. Experienced people watch them closely. Exceptions receive immediate attention. Missing data is corrected manually. Teams know who to call when something unusual happens. Product owners intervene quickly, technical specialists stay close to the work, and people tolerate temporary inefficiencies because everyone understands the capability is still being proven.
None of this necessarily invalidates the pilot. In many cases, that support is exactly what responsible experimentation requires.
The risk appears when the organisation treats everything that made the pilot successful as evidence that the underlying capability is ready to scale.
A pilot can work because the technology is good. It can also work because people quietly compensate for what the technology, process, or operating model cannot yet do.
Those are very different conditions to industrialise.
A pilot reduces uncertainty.
It gives the organisation a bounded environment in which to test whether a proposed capability can produce useful results under conditions that resemble real work. That is more valuable than a technical demonstration because it starts to show what happens when the technology meets people, data, workflows, exceptions, and operational consequences.
But a successful pilot doesn't mean you should reproduce every part of the arrangement at scale.
Imagine an AI-enabled customer service pilot. The system interprets incoming requests, prepares responses and resolves a large proportion of routine cases. The reported results are impressive. Resolution times fall, users like the interface, and the business case suggests substantial savings if the organisation extends the capability.
Look more closely, however, and several other things may be happening.
A small group of experienced employees is checking difficult cases before customers see the result. A data analyst manually corrects records that do not match the format the system expects. The product team handles unusual exceptions directly in a shared messaging channel. One specialist remembers customer commitments that the underlying systems don't represent properly. When the agent behaves unexpectedly, someone from the technical team can investigate within minutes because the pilot still receives executive attention.
The service works.
The harder question is which of those conditions belong to the future capability.
If the organisation multiplies the pilot, it may discover that some of the most important things it scaled were never visible in the business case.
Before moving from Prove to Move, I find it useful to separate four different things that can become entangled during a pilot: genuine capability, legitimate support, temporary workarounds and unresolved conditions.
The first is the easiest to recognise. Capability is what genuinely works and should be reproduced. The system can reliably interpret a category of requests. The integration works. The workflow is effective. The people using it can achieve the required outcome. These are the things the organisation wants to retain and scale.
The second is legitimate support. Some work around the pilot may look like additional overhead but is actually part of the future operating capability. A service may always require specialist exception handling. A regulated process may always need human review for particular decisions. A production system may need ongoing monitoring, maintenance and periodic recalibration. Those activities do not prove the pilot failed. They are part of the real cost of operating responsibly.
The important step is to recognise them as such. If specialist intervention is a permanent part of the capability, it needs to be designed, staffed, measured and funded. It should not survive as invisible goodwill from a handful of people who helped make the pilot successful.
The third category is the temporary workaround. Pilots often need them. A team may manually transfer information because an integration hasn't been built yet. An engineer may review every output because the organisation is still learning how the service behaves. A temporary spreadsheet may bridge two systems while the design is being proven.
There is nothing inherently wrong with this. The problem arises when a proving mechanism quietly becomes part of steady-state operation.
A workaround that was sensible for fifty cases may become expensive and fragile at fifty thousand. If it was introduced only to enable learning, it needs an explicit retirement condition rather than an indefinite future.
The final category is the most dangerous: the unresolved condition.
This is where people are compensating for something that should be fixed before the capability expands. People are reconstructing missing evidence by hand. Experienced employees are correcting poor recommendations before anyone notices. A team absorbs recurring exceptions because the formal escalation path doesn't work. Customers repeatedly provide the same information because the new service cannot carry context across systems.
At pilot scale, capable people can make these problems appear small.
At enterprise scale, they become operating costs.
This is why pilots can produce a misleading picture of efficiency.
The visible system may be highly automated while a small human organisation sits around it, repairing evidence, interpreting ambiguity and preventing mistakes from becoming consequences.
That organisation is easy to overlook because it is often informal. The people involved may not be assigned to the pilot full-time. Their interventions may not appear in the workflow. They may know that something needs attention and fix it.
The problem is not that people are helping.
Organisations are socio-technical systems. People will always contribute judgement, context, and capability in ways that software execution cannot capture.
The management problem is whether the organisation understands what those people are doing and why.
If an experienced employee intervenes because a customer situation genuinely requires discretion, that may be legitimate support that should remain.
If the same employee intervenes because the system repeatedly loses an earlier customer commitment, the organisation may be compensating for a design defect.
Those two interventions can look identical in a pilot dashboard. They have completely different implications for scale.
This distinction matters commercially and operationally. A pilot can appear cheaper than the future service because the organisation absorbs some of its real operating cost elsewhere. Specialists lend their time. Product teams carry exceptions. Managers provide additional supervision. Customers tolerate friction because the service is visibly experimental.
When the business case excludes those costs, the organisation isn't just scaling the technology. It is scaling an incomplete economic model.
These conditions become harder to remove after scale.
Temporary practices acquire dependencies.
A manual workaround becomes familiar. A specialist team develops expertise around resolving the same recurring problem. A control is added to protect against an exception. Another team builds reporting around the control. Service targets begin to assume that the compensating work exists.
What began as a sensible response to uncertainty becomes embedded in the operating model.
The original pilot may still be described as successful because it achieved the intended output. The organisation now carries additional work, cost and complexity that were never part of the intended future design.
This is how organisational baggage can form even around a successful transformation.
The lesson is not that every workaround is bad or that recurring support proves something is wrong. Necessary resilience, controls, exception handling and specialist capability all have legitimate costs.
The question is whether the burden remains necessary once the capability is understood.
If a feasible, proportionate change can remove the originating problem without sacrificing required protection or service quality, continuing to compensate for it imposes an avoidable burden.
That is why the moment before scale is so valuable. The organisation still has an opportunity to distinguish what should become permanent from what should disappear.
One practical way to expose hidden compensation is to imagine removing the people who made the pilot possible.
Not literally, and not because their contribution is unimportant. The question is diagnostic.
If the pilot team stepped away tomorrow, what would stop working?
Would data quality deteriorate because someone manually repairs inputs? Would exceptions accumulate because the escalation route exists only in a private messaging channel? Would customers receive poorer answers because one experienced employee is carrying relationship history the system cannot see? Would managers lose confidence because they depend on a specialist to interpret the outputs?
Those answers reveal the real operating capability more clearly than a headline productivity measure.
They also prevent a common mistake: treating every human intervention as something that should eventually disappear.
Some interventions reveal defects. Others reveal necessary capability.
If removing the specialist would worsen consequential judgement, the answer may be to design that specialist role properly rather than eliminate it. If removing the analyst exposes a recurring data-quality problem, the answer may be to correct the evidence architecture rather than hire more analysts to compensate.
Scaling should clarify people's roles, not simply reduce their visibility.
This is why I think of proving as more than asking whether the technology works.
A future operating capability is ready to move when the organisation understands the conditions it depends on.
That includes the evidence it requires, the people and systems that contribute to it, the authority needed to act, the exceptions it cannot handle routinely, the support that must remain and the weaknesses that still need correction.
Some uncertainty will always remain. Organisations cannot wait for perfect evidence before scaling anything important.
The discipline is to know which uncertainties have been reduced enough to justify greater dependence and which the current operating arrangement is simply hiding.
That distinction also changes the way leaders interpret a successful pilot.
Instead of asking only whether the target accuracy was achieved, whether users liked the tool or whether the projected return is attractive, they can ask what the organisation had to do to make those results possible.
What did people repair?
What did they interpret?
Which exceptions did they absorb?
Which conditions received unusually close attention?
Which support will still be required when the executive spotlight moves elsewhere?
The answers are not reasons to stop.
They are the information needed to move deliberately.
Good pilots often contain scaffolding.
They should.
When an organisation is learning, temporary support lets it test a new capability without pretending everything is already solved. The mistake is not building the scaffolding. It is failing to distinguish the scaffolding from the building.
Before scaling, leaders should therefore make four different decisions.
What genuine capability should be replicated?
What legitimate support needs to be designed into the future operating model?
Which temporary workarounds need explicit retirement conditions?
Which unresolved problems must be corrected before expansion makes them harder and more expensive to remove?
That is a much more demanding question than asking whether the pilot succeeded.
It is also more useful.
A pilot should not merely tell an organisation that something can work. It should reveal what the capability requires to work dependably when conditions become less forgiving, volumes grow, and the people who created the experiment are no longer standing beside it.
The objective is not to eliminate every human intervention before scale. Nor is it to delay useful technology until every uncertainty has disappeared.
It is to understand what made the pilot work well enough to know what deserves to travel with it.
Because when organisations scale without making that distinction, they can industrialise the very compensation they should have designed out.
Before scale, the question is not only, "Did the pilot work?" It is, "What made the pilot work?"
By Gert Botha
Keywords: AI, Digital Transformation, Business Strategy
Before You Scale an AI Pilot, Ask What Made It Work 
Operational Competence: What Training Records Cannot Prove
A Fair Review
Your Continuity Plan Is Missing One Thing: The Leadership Team
Many Tools, One Contract: Keeping the Whole in View
