How to check numbers an AI gives you: a 7-question checklist

An AI assistant can quickly draft a SQL query or summarise a spreadsheet. It can also confidently count cancelled orders as sales, compare a full week with half a week, or add up a total twice after a join. The query runs, the number looks plausible, and nobody notices until a decision has been made.

Start by knowing what to ask, then check the evidence. Some mistakes become visible in the definition; others require inspecting records or rerunning the calculation. Below are the seven questions we use at Atterna, each with an example from the Meridian practice marketplace (a fictional European marketplace; the numbers are real results on its data for 31 August – 22 September 2026).

Short version: a number is only right for a stated question. Check the definition, the period, the filters, duplicated rows, the denominator, whether an average hides the spread, and whether it agrees with a total you already trust.

1. What exactly was counted?

Ask the AI to state the definition in one sentence: which rows, which status, which amount. Teams sometimes use “revenue” loosely for different money measures. On Meridian, distinguish these explicitly:

These are different measures, not interchangeable accounting definitions of revenue. Marketplace GMV and customer payments are not the platform’s own commission income. Agree the precise label and calculation before using the number.

2. Over which period — and is the last period complete?

An incomplete last period can create a false alarm. On Meridian, paid orders by week look like this:

Ask for “orders by week” and an assistant may report an 88% collapse in the last week. The data simply ends on the morning of 22 September. Always ask: when does the data end, and is every period in the comparison complete?

3. Which rows were filtered in or out?

Test accounts, cancelled orders, internal staff, refunds, other countries, duplicate sign-ups. On Meridian, 249 of 9,320 orders were cancelled. Counting them as sales is a small error here and a large one in a month with a payment outage. Ask the AI to list every filter it applied — and every filter it didn’t.

4. Could a join have multiplied the rows?

This is the classic silent error. An order has several items. Join orders to their items and every order appears once per item. Sum the order total after that join and you count each order several times:

-- Wrong: each order total is repeated once per item line
SELECT SUM(o.items_total_eur)
FROM meridian.orders AS o
JOIN meridian.order_items AS i ON i.order_id = o.order_id
WHERE o.status = 'paid';
-- €1,239,938 — more than double the real GMV of €552,645

The query runs without an error and the result is 2.2 times too high. Ask: did any join produce more rows than the table you are summing? The fix is in our guide to JOIN duplicates.

5. What is the denominator?

“52% conversion” means nothing until you know 52% of what. On Meridian, 3,652 of 7,000 registered buyers paid at least once in the period: 52%. Visitors or checkout starters as the denominator give a different rate for a different question. Whatever you choose, the numerator must be a subset of the denominator, over the same window. Ask the AI to show the numerator and denominator as two separate numbers.

6. Is an average hiding the spread?

The average paid order on Meridian is €60.92. The median is €43.96, and 10% of orders are above €127. A few large orders pull the average up; a typical order is worth much less than the average suggests. Ask for the median and a high percentile next to any average of money, time or quantity.

7. Does it agree with a number you already trust?

The fastest check of all. If the AI says last month’s GMV was €5 million and finance reported €550 thousand, there may be a definition difference, stale data or a calculation error — and it is worth finding out before the meeting, not during it. Keep two or three reference totals in your head: orders per week, active customers, monthly revenue.

How to use this with an AI assistant

Paste the checklist into the conversation. A good prompt after any number is:

Before I use this: state the definition in one sentence, the period and whether the last period is complete, every filter you applied, whether any join changed the row count, the numerator and denominator separately, and the median next to the average.

Compare the explanation with the actual query and data. An assistant can describe a filter it did not apply. Reproduce a total and inspect a few underlying rows before using a consequential result.

Want the questions for a specific answer? Paste it into the free AI Answer Checker: it finds the numbers, the open definitions and the SQL traps, and writes the follow-up for you. The instant check runs in your browser — nothing leaves it unless you choose Check with AI.

Try a claim that sounds convincing

In Atterna’s first case, marketing wants to write “the code grows baskets 2.5×” into the December plan. Does that comparison support the plan? You clarify the request, define the metric, run the comparison and decide what the evidence supports.

Try Bigger baskets? — free, about 17 minutes, no account. The SQL is already written. You decide whether a 2.5× difference belongs in the plan, and what else to check before making a recommendation.

To build these habits from the beginning, follow the data literacy for AI learning path, with a worked campaign example and practice you can try without an account.

FAQ

Can I trust ChatGPT or Claude to analyse my data?

They can help draft queries, tables and charts, but they can make errors in those mechanical steps as well as in the definition. Check the code and calculations as well as whether “customers” means buyers, visitors or accounts. Treat an AI answer like a draft from a fast new colleague: useful, and your responsibility once you forward it.

Do I need to know SQL to check an AI’s analysis?

Not for most of these questions. Definition, period, filters, denominator and a known total need no code. SQL helps with the join question and with reproducing a number yourself, which is why the Atterna path starts with a free game that needs no SQL and continues with SQL from zero.

Which mistakes should I check for first?

Start with the stated definition and whether the compared periods are complete, then check joins and denominators. This is a practical order of checks, not a measured ranking of AI failure rates.

Where can I practise checking numbers?

The free three-question game Wrong Number shows one database giving three different right answers to “how many customers do we have?”. The Atterna Start course teaches the SQL to reproduce numbers yourself, and the Meridian simulator puts it into a job: colleagues ask, you check and decide.