What to ask in AI demos: a CFO scorecard for separating substance from theatre
In a market full of polished AI demos, finance leaders need a better way to assess what is credible, controllable and capable of delivering real value.
There is no shortage of AI demos right now. Many of them are impressive at first glance.The interface looks slick. The answers come back quickly. The workflow appears effortless. The sales team talks about transformation, autonomy and productivity gains.
But for CFOs, the real question is not whether the demo looks good. It is whether the product will actually work in your finance environment, with your data, your controls, your team, and your level of risk tolerance.
That is why AI demos need a different lens. Not a wow-factor lens. A CFO scorecard lens. Because the job is not to buy the most exciting demo. The job is to identify which solutions are credible, controllable, and capable of delivering value in the real world.
Why this matters now
The market has moved quickly from AI as an assistant to AI as part of the workflow. We are now seeing tools that do more than draft text or summarise documents. They claim to reconcile, forecast, explain variances, chase debtors, review spend, support decisions, and automate finance tasks end to end.
Some of that is real.
Some of it is early.
And some of it is still little more than a polished front end over manual work, brittle rules, or generic large language model prompts.
That does not make the category unhelpful. It does mean CFOs need to get much better at asking the right questions.
The mistake many finance teams make in demos
The most common mistake is to let the vendor control the evaluation criteria. That usually means the demo becomes a guided tour of best-case scenarios:
- clean data
- happy-path workflows
- pre-prepared examples
- the most impressive use case
- very little discussion of controls, implementation effort, or adoption
It tells you what the product can look like when everything lines up perfectly. It tells you far less about what it will take to make it work in your business. A better approach is to score every demo against the same set of questions. Not to catch vendors out. But to compare solutions properly, reduce bias, and make better decisions faster.
A CFO scorecard for AI demos: 15 questions that matter
Here are 15 questions I would want every finance leader to ask when evaluating an AI-native or agentic solution.
1. What data does the product need to work well?
Does it rely on ERP data only? CRM as well? Data warehouse? Emails? PDFs? Contracts? Bank feeds? Manual uploads?
“AI-ready” often turns out to mean “dependent on multiple sources being accessible and reasonably clean”.
2. How much data preparation is really required?
Ask what has to be mapped, cleaned, labelled, or structured before the product becomes useful.
A short implementation can still hide a long data readiness exercise.
3. Where is the AI actually doing the work?
Is the intelligence embedded in the workflow, or is the product mainly generating text on top of human-driven processes?
There is a big difference between an intelligent workflow and a nice-looking copilot.
4. What happens when the model is uncertain?
How does the system handle ambiguity, low confidence, and incomplete data?
Does it escalate, ask for review, stop the workflow, or confidently press ahead?
For finance teams, this is one of the most important design questions of all.
5. What controls are built into the workflow?
Look for approvals, thresholds, exception handling, role-based permissions, audit trails, and segregation of duties.
If the answer is “your team can monitor it”, that is not enough.
6. How is activity logged and evidenced?
Can you see what the system did, what data it used, what recommendation it made, and why?
A finance process that cannot be evidenced cleanly becomes difficult to trust and difficult to defend.
7. How does the product deal with security and sensitive data?
Ask where data is stored, whether customer data is used for model training, how access is controlled, and what security standards the vendor works to.
For CFOs, AI evaluation is not separate from security evaluation.
8. What systems does it integrate with today?
Not what is on the roadmap. What works now?
ERP, CRM, planning tools, banks, procurement tools, billing systems, spreadsheets, document stores, and identity platforms all matter depending on the use case.
9. How much implementation effort falls on our team?
Who configures workflows? Who manages integrations? Who owns data mapping? What internal resources are needed?
A strong product with a weak implementation model can still fail.
10. How quickly can we get to first value?
Ask what a realistic 30, 60, and 90-day journey looks like.
Not the full vision. The first measurable win.
This is where serious vendors usually stand out.
11. What change is required from the finance team?
Does the solution fit existing ways of working, or does it require a major process redesign?
Sometimes redesign is exactly the point. But it needs to be explicit, because adoption risk is often greater than technical risk.
12. What are the best-fit and worst-fit use cases?
A credible vendor should know where their product works well and where it does not.
That answer is often more revealing than the polished demo flow.
13. How do you prove value?
Ask what metrics customers use to measure success:
- time saved
- close cycle reduction
- forecast accuracy
- reduction in errors
- dispute resolution speed
- working capital improvement
- margin protection
- headcount leverage
If value cannot be measured, enthusiasm fades quickly after go-live.
14. Who is already using this successfully?
Ask for referenceability by company size, sector, geography, and use case. Big logos are less useful than relevant examples.
A scale-up CFO should not evaluate a product solely through the lens of a global enterprise deployment, and vice versa.
15. What still requires human judgement?
This is the question that brings the whole evaluation back to reality. In finance, the goal is rarely to remove people altogether.
The goal is usually to improve speed, consistency, insight, and capacity while keeping human accountability where it matters.
Good vendors are comfortable discussing that balance. Weak ones tend to imply the software can do everything.
What good looks like in a demo
A strong AI demo for a CFO audience should do three things.
First, it should show the workflow in context, not just isolated AI features.
Second, it should be honest about controls, dependencies, and implementation effort.
Third, it should connect the product clearly to measurable business value.
That is what separates an interesting technology demo from a credible finance transformation proposition.
One practical step to take this week
Before your next vendor meeting, create a simple scorecard and force every demo through it.
Score each area from 1 to 5:
- data readiness
- security
- controls
- implementation effort
- user adoption
- time to value
- measurable ROI
- customer referenceability
That one discipline will improve the quality of your buying decisions immediately. Because the real risk in AI is not just moving too slowly. It is buying quickly on the basis of a strong demo and discovering later that the hard part was hidden in plain sight.
Final thought
The finance leaders who get most value from AI over the next 12 to 24 months will not necessarily be the ones who move first. They will be the ones who evaluate well.
The ones who know how to tell the difference between assistance and automation, promise and proof, impressive outputs and reliable operating capability.
That is where the advantage will come from. Not from watching better demos. From asking better questions.
Join us at GrowCFO
This is exactly the lens we are bringing to our upcoming GrowCFO Tech Showcase.
We will be exploring AI-native and agentic solutions for the Office of the CFO, with a focus on what finance leaders actually need to know: where the value is, what the implementation reality looks like, and which questions matter most in evaluation.
If you would like to attend, you can reserve your seat here
The market does not need more AI theatre. It needs better conversations between finance leaders and solution providers. That is what we are aiming to create.
