Independent testing for AI on business data
Does your AI give the right numbers?
Assay tests your AI tool and semantic layer against your reconciled books. You get an independent score that says whether your team can trust its answers, and a ranked list of what to fix.
Or email rajen@assayscores.com
Certificate of assay
Sample · illustrative data| Question | AI said | Books say |
|---|---|---|
| Gross margin %, Q3, Buffalo branchTier 1 · rebates not netted | 31.4% | 27.9% |
| Open AR over 90 daysTier 1 · credit memos left out | $412K | $388K |
| Top 10 customers by sales, YTDMatch | 10 / 10 | 10 / 10 |
Accuracy looks strong, but the band follows the weakest dimension. 7 confident wrong answers, 2 of them on margin and AR, hold this tool at Not ready.
The problem
AI answers fast and sounds sure. Nobody checks it against the books.
Wrong sounds like right
A wrong margin number comes back in the same confident sentence as a correct one. Your sales and finance teams can't tell the difference.
The layer drifts
A new branch, a changed rebate rule or an ERP update can quietly break yesterday's correct answer. Nobody notices until a decision goes wrong.
Vendors grade themselves
Tool vendors test their own product, on their own questions. That's useful, but it isn't independent, and it isn't your data.
How it works
Your questions, your books, an independent check.
-
01
Agree the questions
100 to 150 questions your finance, sales and ops people actually ask, ranked by how much a wrong answer costs.
Week 1
-
02
Lock the answer key
Answers come from reports you already reconcile and trust. The key is fixed before your AI tool runs.
Week 1
-
03
Run your AI tool
Your team asks the questions in your own tool, read-only. Reworded versions test consistency.
Week 2
-
04
Score and fix list
Every answer is scored against the key. You get the scorecard, the band, and the failures ranked by impact.
Weeks 2–3
The score
Four measures. One band, set by the weakest.
| Measure | Ready | Conditional | Not ready |
|---|---|---|---|
| CoverageShare of questions the tool can answer at all | ≥ 85% | ≥ 60% | below 60% |
| AccuracyAnswers that match the books, judged on the 95% lower bound | ≥ 85% | ≥ 70% | below 70% |
| Wrong-answer rateConfident answers that are wrong | ≤ 2% | ≤ 5% | above 5% |
| ConsistencySame answer when the question is reworded | ≥ 95% | ≥ 85% | below 85% |
| Tier-1 wrong answersWrong answers on margin, revenue, AR and other high-cost questions | 0 | ≤ 2 | 3 or more |
The band is the lowest of the five. A tool that's 93% accurate but confidently wrong on 7% of answers is not ready. Monthly re-runs track drift and catch regressions after ERP or model changes.
Independence
We score the tool. We don't sell it.
No tool to sell
Assay doesn't sell, resell or take commissions on AI or BI tools. The score has no reason to lean either way.
Key before run
The questions and answer key are locked before your tool answers anything. Nothing is adjusted after the results come in.
Open method
Thresholds, sample sizes and scoring rules are written down in Method v0.1. You can see exactly how every number was reached.
Works with any tool that answers questions about your data: Microsoft Copilot and Power BI, Snowflake Cortex Analyst, Databricks Genie, ERP assistants, or a chatbot your team built. Data can come from Prophet 21, Eclipse, SX.e, Acumatica, NetSuite or any other ERP.
Pricing
Start with a free spot check.
Spot check
Free
- On a call, ask your AI tool 10 finance questions
- Compare each answer to the report you trust
- See where it breaks before you spend anything
Readiness Check
Fixed fee
Quoted after the spot check. Contact us at rajen@assayscores.com
- 100–150 questions scored on all five measures
- Readiness band with 95% confidence intervals
- Every question, answer and answer key
- Failures ranked by business impact
Monitoring
Monthly
Quoted with the Check
- Monthly re-run of your question set
- Drift and regression alerts
- One-page scorecard for leadership
Need the failures fixed? A fixed-fee fix sprint repairs the top issues and re-scores them, quoted after the Check.
Questions
What people ask first.
Do you need access to our ERP?
No. Your team runs the questions in your own tool, read-only. We compare the answers with reports you already reconcile.
We haven't rolled out AI yet. Is this for us?
Yes. A Check before rollout shows which questions Copilot or your ERP's assistant would get wrong, so you fix them first.
How much of our time does it take?
A kickoff call, help picking the questions and pointing to the source reports, and a results review. Usually a few hours across 2–3 weeks.
How is our data handled?
We sign an NDA first, work read-only, keep only what the scoring needs, and delete it when the engagement ends.
What do we actually get?
A scorecard with the band, every question with its answer key and the AI's answer, and a fix list ranked by impact.
Can the vendor fix what you find?
Yes, and we're happy to share results with them. Many fixes are definitions in the semantic layer, like rebates, returns or branch rollups.
Ask your AI ten questions you already know the answer to.
On a free 30-minute call, we run a spot check together. If it passes, great. If it doesn't, you'll know before your team relies on it.
rajen@assayscores.com