Running an AI pilot that can actually reach production

Why pilots stall before production

Most AI pilots succeed on their own terms. The model works on the sample, the demonstration goes well, and then nothing happens. The pilot was set up as an experiment in technology, with no owner in the business, no agreed measure and no plan for the work that follows.

A pilot that can reach production is run as a project with a decision at the end. It has a scope, an owner, a baseline, a team, a rhythm and a date on which someone chooses to scale, change or stop. Those six things shape the outcome more than the choice of technology does.

Scope and a named owner

Scope the pilot to one process, one user group, and one task the AI will perform or one decision it will support. Write a single sentence that says what will be different for those users when it works. If the sentence needs the word and more than once, the scope is too wide. Write down what is out of scope as well: other departments, other document types, other languages, further integrations.

Then name the owner. The owner is a person in the business who runs the process today, feels the problem, has authority to change how the work is done, and will answer for the result after the project team leaves. An owner who sits only in IT or in an innovation team is a warning sign. Add an executive sponsor who controls the budget for the next stage and has agreed to attend the final decision meeting.

• One process, one user group
• A one-sentence outcome
• A written out-of-scope list
• A business owner with authority over the process
• A sponsor who holds the next-stage budget

Agree the baseline and the success measure first

Measure the process as it runs today, before anything is built: volume, time per item, error or rework rate, backlog, and the effort of the people involved. Use the same data sources you will use at the end. Without a baseline, the result of the pilot is an opinion.

Choose one primary success measure that the business already cares about, such as handling time, first-time-right rate or days to close. Model accuracy is a supporting measure. A model can be accurate and the process no faster, because the time went into checking its output.

Set the thresholds in advance, in writing, agreed by the owner and the sponsor: the level at which you scale, the level at which you stop, and the band between them where you change the approach. Add one or two guard measures, such as complaints or error rate, that must hold while the main number improves. Thresholds agreed after the results are in are always met.

A small cross-functional team

Keep the team to a handful of people who can each make decisions in their own area. A large steering group slows a pilot down. A missing role stops it.

Bring your security, privacy and compliance people in during the first week, with a specific description of the data and the use. Agree how much of each person's week the pilot gets, and have their managers confirm it.

• Business owner: priorities and acceptance
• Subject expert: supplies real cases and judges outputs
• Data or AI engineer: builds and tests
• IT or platform contact: access, environments, integration
• Two or three real users of the process

Weekly checkpoints

Hold a short checkpoint every week with the owner present. Half an hour is enough if it follows a fixed agenda: what was shown working this week on real data, the current number against the baseline, what is blocked and who will unblock it, and what will be shown next week.

Insist on a working demonstration at every checkpoint, even a rough one. Slides about progress hide the two things that kill pilots, which are waiting for data access and waiting for a decision. Keep a visible log of blockers with a name and a date against each, and escalate to the sponsor anything open for more than a week.

Put users in front of the system early. Feedback from two real users near the start is worth more than a polished release near the end. Fix the length of the pilot at the outset, and resist extending it. If the evidence is missing on the end date, that is itself a finding.

The decision point: scale, change or stop

End the pilot with a meeting whose only purpose is a decision. The owner presents the baseline, the result, the guard measures, what users said, and an estimate of the production work. The sponsor chooses one of three outcomes.

Scale means the threshold was met, and the budget and team for production are approved in the same meeting. Change means the result fell in the middle band and there is a specific, testable reason to believe a different approach will work: better data, a narrower scope, a different step in the process. Give it one further fixed period. Stop means the threshold was missed, or the cost of production outweighs the gain. Write down what was learned and release the team.

Stopping is a legitimate result of a well-run pilot. The outcome to avoid is a fourth one, where the pilot carries on with no decision because nobody wants to end it.

The production work a pilot usually skips

A pilot proves that the idea works for a few users on sample data. Production means it works for everyone, every day, inside the systems they already use, and that someone answers when it breaks. The gap is mostly ordinary engineering, and it is often larger than the pilot itself. Estimate it before the decision meeting so that scaling is an informed choice.

Plan the handover with the same care. The team that will run the system needs documentation, runbooks, training and a period of working alongside the builders. Data Prophets builds GCC and AI CoE teams on a build-operate-transfer model, and has built and handed over two such centres, each with 80+ people.

• Integration: live connections to source systems in place of manual exports
• Monitoring: quality, usage, cost and failures, with alerts to a named person
• Access control: who may use it, and what data each user can reach
• Support: a route for user problems and an owner for fixes
• Handover: documentation, runbooks and training for the operating team

What to do next

Before your next pilot starts, fill in one page: scope sentence, owner, sponsor, baseline, success thresholds, team, end date and the date of the decision meeting. If any line is blank, fix that first. If you are still choosing a use case, our free AI readiness report starts with six questions.