Start from a KPI you are already accountable for, not from a tool you saw demoed.
Most AI programmes fail the same way. Someone sees a demo, the organisation buys
the tool, and six months later nobody can say whether it helped. The problem is
not the tool. It is that the use case was chosen from the supply side.
Start from a number you already report
Pick a KPI that already appears in a report someone is accountable for. Not a new
metric invented for the project — an existing one, with a history, that a person
would be asked about in a review.
This constraint does more work than it looks like it does. It rules out most
speculative use cases immediately, because they cannot name the number they would
move. It also means you inherit a baseline for free.
Name the decision, not the task
"Summarise support tickets" is a task. "Decide which accounts to call this week"
is a decision. Automating a task saves minutes. Improving a decision changes the
KPI.
Ask: what decision does this information feed? If the answer is "none, it is just
useful to have", the use case will not survive contact with a budget review.
Write the counterfactual down before you start
The single most common failure in AI reporting is attributing to the tool
whatever the number did anyway. Before the pilot begins, write down what you
expect the KPI to do without the intervention, and how you would tell the
difference. A holdout group is best. A pre-agreed seasonal baseline is acceptable.
"We will look at it afterwards and decide" is not.
A worked shape
| Element |
Weak version |
Strong version |
| KPI |
"Productivity" |
Median days-to-quote, already reported monthly |
| Decision |
"Help the team" |
Which quote requests get same-day turnaround |
| Baseline |
None |
Trailing 12 months, same seasonality |
| Owner |
"The AI project" |
The sales operations lead, in their existing review |
If you cannot fill the right-hand column, you are not ready to build. That is a
useful outcome, reached in an afternoon rather than two quarters.
You can ask your own AI assistant about this. Add https://learn.unpyle.com/mcp as a connector in Claude, ChatGPT or Copilot, and it will answer from Unpyle's published guidance rather than from general advice.
If something here is not clear, or you are ready for a virtual coffee, pick a time that suits you. Thirty minutes, no commitment, no pitch deck.
Published by Unpyle, who help teams put AI to work inside their own company. More at unpyle.com.