Key takeaways
- A pilot answers "will it work here?", not "can it work?", which requires a baseline, a defined success criterion, and a hard checkpoint.
- The right first use case is high-frequency, measurable, and low-stakes: meeting notes, variance narrative drafting, or a single recurring workflow.
- No 30-day checkpoint means no pilot, and it means a subscription that never gets evaluated.
In this article
- Proof of concept vs. pilot: a critical distinction
- Problem selection criteria: what makes a good first use case
- Specific tools to pilot first
- Baseline capture: what to measure before day 1
- Participant selection: enthusiasts and skeptics both
- The 30/60/90-day checkpoint structure
- Common mistakes that kill AI pilots
- What good pilot documentation looks like
Proof of concept vs. pilot: a critical distinction
Most companies conflate two different things. A proof of concept answers one question: can this technology do what the vendor claims? A pilot answers a different question: will this technology work inside our company, with our people, in our workflows, at scale? The first question is vendor responsibility. The second question is yours.
This distinction matters because the failure mode is different. A proof of concept fails when the technology does not work. A pilot fails when the technology works but nobody uses it, the rollout creates new problems, or there is no measurement to prove value. Most AI failures in the middle market are pilot failures disguised as technology failures.
75% of AI pilot failures are attributable to implementation and change management factors, not technology capability limitations. The technology worked. The deployment did not.
The single most common AI pilot mistake in middle market companies: starting a pilot without capturing a baseline metric. You cannot prove a 60% reduction in meeting prep time if you never measured meeting prep time before the pilot started. Baseline capture is not optional, and it is the entire foundation of the business case.
Illustrative example, a $35M distribution company launched a ChatGPT pilot for drafting supplier communications. After 90 days, the CFO asked whether it was working. Nobody had measured draft time before or after. The pilot was declared "interesting" and quietly abandoned. Three months later, the same company launched a second pilot with a baseline, and documented a 55% reduction in first-draft time.
Problem selection criteria: what makes a good first use case
Choosing the right first use case is the single highest-leverage decision in the entire pilot. Wrong use case selection is the most common reason pilots fail, not the technology, not the people, not the timeline.
AI Pilot Use Case Selection Framework
The sweet spot is a task that happens weekly, takes 20–60 minutes today, has a clear time metric, and involves data that can be entered into the tool without a confidentiality concern. Meeting notes, variance narrative drafting, first-draft communications, research summaries, and job description writing all qualify. Complex financial modeling, legal document review, and strategic recommendations do not.
45 minutes
typical time for manual weekly meeting prep in a $20M–$75M company
15 minutes
target meeting prep time with AI-assisted note-taking and agenda drafting
3–8 people
ideal pilot participant count for a structured 30/60/90-day pilot
Bad success criterion: "saves time." Good success criterion: "reduces meeting prep from 45 minutes to 15 minutes for the weekly ops review." The difference is specificity. A bad criterion cannot be measured. A good criterion produces a yes/no answer at day 30 that drives a go/no-go decision. Every pilot needs a success criterion written down before day 1, not after.
Specific tools to pilot first
Tool selection for a first pilot should prioritize low friction over maximum capability. The goal is not to deploy the most powerful tool, and it is to produce a clear result with the least implementation complexity.
For meeting notes: Granola (Mac and Windows, runs locally, no meeting bot) or Firefly (cross-platform, integrates with Zoom and Teams). Both produce structured summaries with action items. Granola is the lower-friction option for small teams because it does not require calendar integration or a visible bot.
For variance narrative drafting: Claude (Anthropic) or ChatGPT (OpenAI). Both can take a structured data table and produce a clean variance narrative in under 60 seconds. The workflow: paste the variance table, give the model a one-sentence prompt about tone and audience, review and edit the output. Total time: 5–10 minutes vs. 30–45 minutes manually.
For workflow automation: Zapier for a single recurring workflow, not a broad automation strategy. The right first Zapier pilot is one trigger, one action, one measurable output. Example: new row added to a Google Sheet triggers a formatted Slack notification to the ops team. Simple, immediate, measurable.
Companies that start automation with a single high-frequency, low-complexity workflow have a 3x higher likelihood of expanding to 5+ workflows within 12 months, compared to companies that start with a broad automation strategy.
Piloting Granola for meeting notes in a 5-person leadership team at a company with 8 weekly recurring meetings: estimated time saved is 3–5 hours per week across the team. At a fully loaded cost of $150/hour, that is $23,400–$39,000 in annualized labor value, against a Granola subscription cost of $480–$960/year. ROI is not the question. Adoption is the question.
Working through this yourself?
Kolton works directly with founders on M&A readiness, deal structure, and AI implementation — one advisor, not a team of generalists.
Schedule a conversation →Baseline capture: what to measure before day 1
Baseline capture must happen before the pilot starts, not during, not after. The measurement does not need to be rigorous. It needs to be documented and honest.
For a meeting notes pilot: have each participant track their actual meeting prep and note-taking time for two full weeks before the pilot starts. A simple spreadsheet with date, meeting name, and minutes spent is sufficient. Calculate the average across participants.
For a variance narrative pilot: time three consecutive monthly close cycles, how long does the analyst or controller spend writing narrative commentary on variance reports? Track start and stop time. Three data points is enough for a baseline.
For a workflow automation pilot: count how many times the manual process happens per week for four weeks. If you are automating a report distribution process, count how many minutes it takes each time and how many times per week it runs.
Step 1: Define the use case, write one sentence describing the specific task being piloted
Step 2: Write the success criterion, state the specific metric, target value, and measurement method before day 1
Step 3: Capture the baseline, measure the current-state metric for 2–4 weeks before the pilot starts
Step 4: Select participants, choose 3–8 people with a mix of enthusiastic adopters and skeptics
Step 5: Launch and run weekly 15-minute check-ins, structured 15 minutes each week: what worked, what did not, blockers
Step 6: Day 30 go/no-go, compare against success criterion; make a documented decision to continue, pivot, or stop
Pilots with documented baselines are 4x more likely to produce actionable adoption decisions than pilots without baselines. The absence of a baseline does not just make measurement harder, and it makes honest evaluation impossible.
Participant selection: enthusiasts and skeptics both
The most common participant selection mistake is stacking the pilot with enthusiastic early adopters. Enthusiasts will find value in almost any tool, and they are not a representative sample. A pilot that only involves enthusiasts produces a result that does not predict broad adoption.
The right participant mix: 3–8 people total, with roughly half self-selected enthusiasts and half selected skeptics or neutrals. The skeptics are the signal. If the tool produces clear value for a skeptic, it will produce value at scale. If it only works for enthusiasts, you have a self-selection problem, not a technology win.
Skeptics should be people who are not looking for a reason to adopt AI, and they are looking for a reason not to. They will find the friction points the enthusiasts overlook. They will surface the use cases that do not work. That is valuable information before you scale.
In a 6-person pilot team, include at least 2 people who expressed doubt or skepticism about AI tools when the pilot was announced. Their adoption, or their specific objections, and will tell you more about scalability than the 4 enthusiasts combined. Document their feedback at every weekly check-in.
The 30/60/90-day checkpoint structure
The pilot timeline is not arbitrary. Each checkpoint has a specific decision attached to it. Without the decision, the checkpoint is just a calendar meeting.
Day 30, go/no-go: compare actual results against the success criterion written on day 1. Is the metric moving in the right direction? Are participants using the tool without prompting? Are there blockers that are structural (the tool does not do what we need) vs. adoption-based (people are not in the habit yet)? The go/no-go decision is binary: continue the pilot or stop. If the technology is not working at 30 days, extending the pilot rarely produces a different result.
Day 60, scale decision: if the day 30 decision was "go," day 60 answers whether to expand. Who else in the organization has the same use case? What is the estimated cost at full scale? Are there IT or data governance requirements for a broader rollout? The day 60 decision is: expand to full rollout, expand to a second pilot cohort, or continue current scope and revisit at day 90.
Day 90, embedded standard: the pilot should end with the tool either embedded as a standard process (written into the workflow documentation, included in onboarding, assigned a tool owner) or formally retired. "We will keep using it informally" is not a day 90 outcome. It is the path to tool sprawl.
30 days
time to first go/no-go checkpoint in a well-structured AI pilot
60 days
time to scale decision, expand rollout or hold at current scope
90 days
time to embed as standard or retire, the pilot officially ends
Common mistakes that kill AI pilots
Too broad a scope. Piloting "AI for operations" is not a pilot, and it is a wish. A pilot must be scoped to a single use case with a single success criterion. Broad pilots produce vague results that cannot drive a go/no-go decision.
No baseline captured before day 1. If you cannot show the before state, you cannot prove the after state. No baseline means no business case, which means the pilot becomes a feature demo rather than a decision-support tool.
No 30-day checkpoint. Pilots without structured checkpoints drift. Without a checkpoint, a failing pilot limps along for months because nobody wants to be the person who cancels it. The checkpoint forces the decision.
Piloting the wrong use case first. High-stakes, low-frequency, or data-sensitive use cases are not good first pilots, even if they represent the biggest theoretical value. Win a clear, measurable result on a simple use case first. Use that win to fund and justify the harder use cases.
Conflating adoption rate with success. A pilot where 90% of participants use the tool but the success criterion is not met is a failed pilot. A pilot where 50% of participants use the tool and the success criterion is met is a successful pilot. Measure against the criterion, not against adoption.
The #1 predictor of successful AI scaling in organizations is whether the initial pilot had a documented success criterion and a structured go/no-go decision point. Organizations with this structure scale AI 3x faster than those running ad hoc pilots.
What good pilot documentation looks like
Every pilot should produce a single one-page document at the end of the 90-day period. Not a presentation, not a project summary, a one-pager with six fields: use case description, success criterion, baseline metric, actual result at day 90, go/no-go decision and rationale, and recommended next step.
This document serves three purposes. First, it creates institutional memory, the next person who evaluates the same tool category does not start from scratch. Second, it builds the business case for additional AI investment. Third, it builds credibility with the CFO and CEO, who see structured decision-making rather than ad hoc tool adoption.
The one-pager is also the most effective antidote to the "we tried AI and it did not work" dismissal. When a failed pilot is documented with a specific use case and a specific reason for failure, the failure becomes learning. When it is undocumented, it becomes a vague institutional memory that blocks future adoption.
A library of 5–10 completed pilot one-pagers is a strategic asset for a middle market company. It demonstrates to potential acquirers that AI adoption is systematic, not ad hoc, and it demonstrates to employees that investment in pilots produces real decisions, not theater. Build the library from the first pilot forward.
Frequently asked questions
What if participants stop using the tool between check-ins?
Non-use is signal, not failure. At the next weekly check-in, ask what happened specifically, too much friction, output not useful, forgot to use it, or process changed. The answer determines whether the problem is adoption (solvable with habit formation) or fit (the use case was wrong).
What if results are positive but participants resist scaling?
Resistance to scaling is almost always about process disruption, not tool quality. Engage the resistors directly, ask what would need to be true for them to be comfortable. Often the answer is a small process accommodation that does not affect the outcome.
Can a pilot run with fewer than 3 people?
Two-person pilots produce directional signal but not scalability data. If you only have 1–2 people with the relevant use case, run the pilot and document carefully, but weight the day 30 decision conservatively and plan a second cohort before committing to full rollout.
What does a weekly 15-minute check-in look like in practice?
Three questions: What did you use the tool for this week? What worked well? What did not work or felt like friction? Rotate who speaks first. Capture notes in a shared doc. The weekly cadence prevents drift and surfaces blockers before they become reasons to stop using the tool.
Research sources
Disclaimer: Financial figures and case studies in this article are illustrative, based on representative middle market assumptions, and are not guarantees of outcome. Statistical references are drawn from cited third-party research; individual transaction and operational results vary based on business characteristics, market conditions, and deal structure. This content is for informational purposes only and does not constitute legal, financial, or investment advice. Consult qualified advisors for guidance specific to your situation.
Explore adjacent topics
