Reportman.AI Logo Reportman.AI

← All articles

Why does the AI give you a different figure when you ask again?

· Toni Martir, arquitecto de software

In the morning, someone in sales asks the company's AI tool «how many active customers do we have?» and gets a figure. In the afternoon, someone in accounts types the same question, word for word, and gets one almost four times bigger. Not a single customer was added in between.

The database is the same; what has changed is the SQL statement. The AI translates every question into an SQL statement (the technical instruction that fetches the data), and every time you ask, it translates again. If it writes a different statement, with a different criterion, you get a different figure. The question is why this happens and how to get a figure that repeats.

«How many active customers do we have?», asked twice on the same day

Morning: 412 The statement counts customers with at least one invoice in the last twelve months
One criterion
Afternoon: 1,630 The statement counts the records flagged as active, even if the customer has not bought anything in years
Another criterion
The kept statement: 412 The morning one, checked once and kept. Run again in the afternoon, it gives the same figure
It repeats

An office example with made-up figures. The database is the same in the morning and in the afternoon: what changes is the statement.

Nobody guarantees the same answer twice

The vendors say so themselves. OpenAI explains in its documentation that, by default, «model outputs may differ from request to request». There is a setting to reduce that variation, the temperature, which controls how much randomness goes into the way the model picks each word. But Google and Anthropic both warn that even with the temperature at zero the answer can still vary. And on Anthropic's models released after Claude Opus 4.6, that setting can no longer be changed at all.

So the same question can produce two different statements. And a small difference is enough to change the criterion in a question like this one, because «active customers» can be read more than one way: the customer records flagged as active, or the customers who have bought something in the last year. Both are reasonable, and the AI can take one in the morning and the other in the afternoon. The database runs each statement to the letter: both counts are right; they answer two different questions.

Checking one figure does not settle the next

When a figure matters, you check what the statement behind it does; I explained how to do that without being a programmer in how do you know the AI figure is right. But that check covers that statement, not the question. Ask again the next day and the AI may write a different one, which then needs checking all over again.

For a one-off question, you can live with that. For the figure the management team looks at every month, it means checking the same thing twelve times a year, and risking that one month's figure cannot be compared with the month before.

What repeats is the statement

Every answer from the AI is written from scratch; the statement, by contrast, is fixed text, and the database runs it the same way every time. Run on October's data, it gives October's figure by September's criterion, and the figure only changes when the data does.

So when a question turns into a figure you are going to track, you check the statement once and keep it (in a report, or with your IT people) with its explanation in words alongside, so that whoever reads the figure knows what it counts. That leaves the AI for what it does well: new questions.

So why does it give you a different figure?

Because each time it translates your question afresh and may write a different SQL statement, with a different criterion. The figure you are going to track always comes from the same statement: you check it once and you keep it.


At Reportman AI the SQL statement the AI writes is in plain view and can be reviewed, edited and run. The blind audit, optional and switched on by default, explains in words what each statement does without knowing your question: compare two explanations and you can see whether two answers use the same criterion. Both work with cloud AI and with a model installed on your own machine, with no subscription: you supply the machine.

Sources

Toni Martir, arquitecto de software · · sources checked on