How accurate is ChatGPT at data analysis? A small test on 12 SQL questions

“Is ChatGPT accurate at data analysis?” has no single answer: it depends on the model, the data and above all the question. So we ran a small test of our own. We asked an OpenAI model 12 questions an analyst gets every week, ran each SQL query it wrote on our practice data and compared every number with a reference answer we had written and checked ourselves.

The result in one table

Each question was asked twice. Both runs returned the same number for every question.

In our small test: 8 of 12 matched, 2 used a different money definition, 2 were arithmetically right but misleading. None of the 24 queries failed to run, and none returned a wrong number because of a SQL bug.

How we tested

• Model: gpt-6-luna, a low-cost OpenAI model, called through the API on 1 October 2026. It is not the ChatGPT app, which may use other models and can run Python on uploaded files; we didn’t test that.
• Data: five tables of Meridian, Atterna’s fictional practice marketplace: buyers, sellers, orders, order items, refunds.
• What the model saw: the CREATE TABLE statements, a comment listing the status values, three sample rows per table, and the question. It had to return one DuckDB query and a one- or two-sentence note with any caveat.
• Scoring: we executed the query and compared the result with our reference. Matches means the same number for the same question. Different definition means a defensible number for a different metric. Misleading means the arithmetic is right but the answer would mislead someone who acts on it.

This is not a benchmark. Twelve questions written by us, one model, one clean schema with helpful comments. A messier database, vaguer column names or a long conversation could go worse; a stronger model could do better. We don’t quote general accuracy percentages from other studies here: they depend so much on the setup that they wouldn’t tell you how your own question will go.

What the model got right

Several questions had classic SQL traps built in, and it avoided every one of them, in both runs. It counted people with COUNT(DISTINCT buyer_id), not rows. For “per registered buyer” it put the status filter in the ON of a LEFT JOIN, so buyers with no orders stayed in as zeros (the trap is explained in LEFT JOIN with a WHERE clause). It summed product sales from item lines instead of repeating the order total once per item, which would have crowned the wrong product. It excluded pending refunds from “money actually refunded”. If your idea of AI SQL is a query that doesn’t run, this model, on this schema, did better than that.

Miss 1: a definition the schema doesn’t contain

For commission, the model multiplied the rate by what customers paid after the platform’s discounts. On Meridian, commission is charged on the goods value, and discounts the platform funds are its own cost. Result: €64,189 instead of €67,040. Average order value had the same split: €58.35 on what customers paid, €60.92 on goods value. Both are real metrics; the danger is a report that mixes them.

To its credit, the model wrote which column it used in its note. That note is where you catch it, if you read it.

Practice · check 1 of 2

AI

An AI assistant

“Paid orders totalled €552,645 in gross merchandise value (GMV).”

Can you infer the platform’s own revenue from that number?

Preparing the practice…

Yes — all €552,645 belongs to the platformYes — subtract customer refunds from GMVNo — I need its commission revenue or take rate

Miss 2: data it never looked at

The data ends on Tuesday 22 September at 03:40. Asked how paid orders changed in the latest week, the model compared 406 orders in that half-started week with 3,289 the week before: −2,883. In the first run the note didn’t mention that the week was incomplete. In the second it added that the week “may be incomplete”, without checking or fixing it.

Practice · check 2 of 2

week	paid orders
1	2,625
2	2,751
3	3,289
4	406

Orders fell 88% in week 4. What happened?

Preparing the practice…

Something broke — alert the teamWeek 4 has barely a day of dataThe promotion ended

The 7-day conversion question had the same blind spot, better hidden. Of 1,110 buyers who registered in September, 153 paid within 7 days: 13.8%. But anyone who registered after 15 September at 03:40 hadn’t had 7 days yet when the data ends. Counting only buyers with a full week, it is 95 of 506: 18.8%. Neither run noticed. A dashboard built this way understates conversion for the newest sign-ups every time it refreshes.

The pattern

In our test the mechanics were right and the meaning was where it went wrong: business rules that aren’t in the schema, and facts about the data, like where it ends, that the model never checked. Fluent SQL makes those misses harder to see, not easier. Other models and prompts can also make mechanical mistakes, such as the join that doubles a total; the six silent SQL mistakes show those.

Six checks that would have caught every miss

1. Ask for the definition in one sentence and compare it with your team’s: which amount, which statuses, before or after discounts and refunds.
2. Ask where the data ends and whether the last period, or every cohort’s window, is complete.
3. Read the model’s own caveats. In our test the different definitions were stated in the note.
4. Reproduce one number you already trust, such as last month’s total, with the same query.
5. Count rows before and after any join, and count people with DISTINCT.
6. Separate the number from the story. “Orders fell 88%” and “because of X” are two claims; the second needs evidence the query doesn’t contain.

The full version, with worked examples, is in how to check numbers an AI gives you. To get better answers in the first place, state the metric, period and exclusions in the question; see how to ask AI about data.

Paste an answer you got into the free AI Answer Checker: it marks the numbers, the open definitions and the SQL traps, and writes the follow-up questions for you.

FAQ

How accurate is ChatGPT for data analysis?

There is no single accuracy figure. In our small test, an OpenAI API model answered 8 of 12 analyst questions with exactly our reference number; the other 4 used a different definition or ignored an incomplete period. With a different model, messier data or vaguer questions, the result can be quite different. Check each answer rather than relying on a general rate.

Can I trust ChatGPT to write SQL queries?

Treat its SQL as a draft from a fast colleague. In our test every query ran and the mechanics were mostly right, but two answers used a different business definition and two ignored that the data ended mid-week. Read the query, read its stated assumptions and reproduce one known total before you use the result.

What mistakes does ChatGPT make in data analysis?

The ones in our test: picking a definition the data doesn’t specify, such as before or after discounts, and comparing incomplete periods or cohorts as if they were complete. Other common ones are counting rows instead of people, a join that duplicates rows, and stating a cause the data can’t show.

Is the ChatGPT app the same as the model you tested?

No. We called a low-cost model through the API with a fixed prompt. The ChatGPT app may use other models, can run code on uploaded files and keeps conversation context, so its answers can differ. The checks above apply either way.

How can I check an AI’s analysis if I don’t know SQL?

Most of the checks need no code: ask for the definition, the period and whether it is complete, the filters, and a total you can compare with one you trust. The free Wrong Number game shows in three minutes why the definition changes the number.