From pilot to production: how a professional firm gets AI past the demo

Every firm we talk to has a pilot story. A partner saw a demo, an associate built something over a weekend, a vendor ran a trial, and for a few weeks it felt like the future had arrived. Then it quietly stopped being used. Nobody killed it. It just never became a system. So what separates the pilots that become systems from the ones that end up in a folder?

The honest answer is not the model, and it is not the vendor. It is whether the pilot had gates. The MIT NANDA report on the state of AI in business, published in 2025, found that 95 percent of enterprise generative AI pilots delivered no measurable profit-and-loss impact. S&P Global reported the same year that 42 percent of companies had abandoned most of their AI initiatives, up from 17 percent a year earlier. Those are enterprise numbers, with enterprise budgets and engineering teams behind them. A professional firm with no engineers does not beat those odds by trying harder. It beats them by putting three decisions in the right order.

Stage one: the pilot, and the two things that happen before it

A pilot at a firm usually starts with a tool. It should start with a use case that has been scored and a data check that has been passed. Both are boring. Both are where the 95 percent are lost.

Score the use case first. Where does the work come from today? For a firm that is intake calls, the shared inbox, the front desk, the referral log, the document store, and the forms clients fill in. Pick one flow. Name the business owner, meaning the partner or practice manager who will answer for the result. Name the outcome in their words, and put a number on it: hours of paralegal time per month, minutes from first call to first response, the share of new patient calls answered after hours. If nobody will own the outcome, that is the answer, and it cost nothing to learn. Our decision framework for custom versus off-the-shelf and the newer piece on which AI pattern a firm actually needs cover this step.

Then run the data-ready gate. Five questions, asked of the data the system will read, before anyone builds anything. Is it accurate enough to trust? Is it current, or is the "closed" flag six months behind? Do we know where each record came from? Can a system reach it without someone exporting a spreadsheet every Friday? Does one named person own it? At an enterprise this means a warehouse. At a firm it means the practice management system, the document store, the inbox, and whatever lives on the shared drive. Most firms fail at least one of the five, and the honest move is to fix that before the build, not after. This is exactly what our two-week Build Scoping exists to find out, and why the fee credits toward the build rather than standing on its own.

Now build the smallest thing that proves the mechanic. A firm does not need to know the words "retrieval," "agent," or "fine-tune." It needs one of three shapes: a system that answers questions from the firm's own documents, a system that takes an action such as logging a call or drafting a reply, or a system that reads and routes. Pick one. Build the core of it. Nothing else. Our piece on how a private copilot reads a firm's knowledge and the one on when one assistant becomes a team describe the first and second shapes in plain terms.

The golden set: the test that decides whether the pilot passed

This is the step firms skip and the step that matters most. Before the pilot runs, the people who do the work today assemble a golden set: real cases with the correct answer already known. Fifty is a fair start for a narrow flow; a few hundred for anything broad. The pilot runs against every case. A human reviews the results. The pilot passes only if it clears a pass rate the partners agreed to before the build started.

The number matters less than the fact that it exists. Ninety percent is a common starting threshold for a system whose output a person reviews. It rises for anything that reaches a client, a patient, or a regulator with no person in between. What the golden set gives a firm is a decision that is no longer a feeling. "It seems to work" becomes "it passed 47 of 50, and here are the three it missed." The three it missed go back into the set, the system is adjusted, and it runs again. That loop is the whole evaluation discipline, and how to know your firm's AI is actually working walks through it in detail. So does what separates a reliable agent from a demo.

Stage two: the gate

At an enterprise the gate is a security review and a compliance review, with SOC 2 and ISO 42001 on the checklist. At a professional firm the gate has different names and higher stakes, because the rules attach to a license. A law firm's gate is Florida Bar Rule 4-7 and the unauthorized-practice line. A medical practice's gate is HIPAA and the business associate agreement with every vendor in the chain. A wealth firm's gate is the SEC Marketing Rule and the books-and-records obligation. A firm with EU clients adds the AI Act, which we covered in three things US firms should copy from it.

The gate asks four questions of the pilot that passed its golden set. Can a client's prompt trick it into doing something it should not? Does it strip personal information before anything leaves the firm? Where does it remember things, and what must it never remember? And who approves its output before a client sees it? The last question is the approval layer, and we have written it up for law firms, medical practices, and wealth firms, with the general case in the approval layer for regulated firms. The memory question has its own piece: where your firm's AI memory should live.

A pilot that passes the golden set and fails the gate is not a failure. It is a pilot that would have become an incident. Finding that out at the gate, on fifty cases, is the cheapest version of that lesson a firm will ever get. The expensive version is in the AI incident your firm has not planned for.

Stage three: production, one group at a time

Enterprises ship to five percent of users, then fifty, then everyone, with a rollback ready. A firm does the same thing with people instead of percentages. One practice group first. One location first. One partner's matters first. The system runs for real, the people using it know it is new, and the person who built it can turn it off in a minute if it misbehaves. Then the next group.

Production also means the system is watched. Three numbers are enough to start: what it costs per matter or per patient, how long it takes to respond, and how often its output is flagged and sent to a person. The third number is the one that matters. A flagged answer is not a failure; it is the system doing what the approval layer asked. But a rising flag rate is an early warning, and every flagged case goes back into the golden set for the next evaluation run. That closes the loop between stage three and stage one.

The last piece is a weekly look at whether the outcome the business owner named in stage one is actually moving. Not a dashboard for its own sake. One number, reviewed by the person who owns it, against the estimate they made before the build. The real ROI of AI for professional firms lays out what to measure at 30 days, 90 days, and a year.

Who does each part when the firm has no engineers

This is the question the enterprise version never has to answer, because an enterprise has a platform team. A firm does not, and pretending otherwise is how a partner ends up owning a system nobody can maintain. So here is the division we use.

• The firm supplies three people. A business owner who names the outcome and reviews the weekly number. A data owner who can answer the five data-ready questions. Reviewers, the people who do the work today, who assemble and grade the golden set.
• Aday Interactive, Inc. supplies the rest. The scoring, the data-ready gate, the build, the evaluation runs, the compliance gate, the staged rollout, and the monitoring. As a scoped build, that is Build Scoping into a Proof of Concept into the production build. As an ongoing role, it is a Fractional Chief AI Officer who owns the whole path and reports to the partners.

Which one a firm needs is its own decision, and the fractional CAIO versus full-time comparison is the place to make it. Either way, the three stages and two gates do not change. They are the difference between a pilot that becomes a system and a pilot that becomes a story.

Where to start

If a pilot is already running at your firm, ask two questions this week. Does it have a golden set, and did it pass a gate before it touched a client? If the answer to either is no, that is the work, and it is smaller than it sounds. If nothing is running yet, the free readiness assessment below tells you which stage you are actually on, and what the score bands mean explains what to do with the number.