Why AI Gives You a Different Answer Every Time

01The model doesn't pick anything

A language model doesn't produce an answer. It produces a score for every token in its vocabulary, all at once, and those scores get turned into probabilities. On a current model that's a ranked list around a hundred thousand entries long, sometimes closer to two hundred thousand, with a number attached to every one.

Then something outside the model picks one of them. The ranking and the picking are two different jobs, done by two different pieces of software. I had them filed in my head as one.

Temperature is the dial on the second one.

Mechanically it's one division. Every score gets divided by the temperature before it becomes a probability. Divide by a number below 1 and the gaps between scores stretch, so the leaders pull further ahead. Divide by a number above 1 and the gaps compress, so the field bunches up and long shots get a real chance.

Temperature 1 is the neutral setting, because dividing by one changes nothing and you get the model's own distribution exactly as it came out. That's the reference point most explanations leave out. "Higher temperature flattens the distribution" doesn't mean much until you know what it's flattening against.

At 0 the math breaks, since you can't divide by zero, so implementations skip the formula and just take the top-ranked token every time. So 0 isn't the low end of the dial. It's the sampler switched off.

Which is why the same question gives you different answers. Nothing about the model changed between your two tries. The dice did.

02The thing everyone says about it

The standard line, and I've repeated it, is that temperature is a slider between accuracy and creativity. Turn it down for facts, turn it up for writing, and the price of creativity is hallucination.

Half of that holds up. The other half doesn't have much behind it.

A 2024 dialogue benchmark called HalluDial swept temperature and tracked how often the model made things up. The rate sat low and roughly flat through the lower range, then climbed sharply once temperature passed a point. Which is a threshold, not a slider.

The other study went looking for the effect inside the range people actually use. Renze and Guven ran nine models through multiple-choice exams, compared temperatures from 0.0 to 1.0, and found no statistically significant difference in accuracy anywhere in that range.

They also pushed one model past it. GPT-3.5 held around 61 percent accuracy at temperature 1.0 and 56 percent at 1.4. At 1.5 it dropped to 35 percent. At 1.6 it scored 5 percent, with 84 of its 100 answers coming back malformed.

That's one model on one exam, so it's a data point rather than a law. But it puts the cliff somewhere north of 1.4, which is a long way from 0.7.

So the honest version. The tradeoff is real and it mostly lives outside the range you're working in. Between 0 and 1, where nearly every application sits, careful studies struggle to find an effect on correctness at all.

Which makes most temperature tuning theater. Dropping from 0.7 to 0.3 to make a model "more accurate" is adjusting a knob that, in that range, is mostly adjusting variety.

Renze and Guven do recommend 0.0 for problem solving, which sounds like the opposite of what I just said. Their reason is reproducibility, not accuracy. Pin it so the same input gives you the same output, and you're not trading away anything to do that.

03What it's actually good for

Not accuracy. Variety.

Low temperature buys consistency, which matters when the same input has to produce the same output. It costs you range. Models near 0 get repetitive and a little flat, reaching for the same phrasings, because you're always taking the safest available word.

High temperature buys range, which matters when you're generating options and want them to genuinely differ from each other.

What finally made it click for me is that temperature isn't a quality setting, it's a spread setting. It sets how far from the model's first instinct you're willing to let it go.

04The zero that isn't zero

Temperature 0 gets described as deterministic. Same input, same ranking, always take the top one, same output. In theory that holds.

In practice it doesn't, and Anthropic's own API docs say so plainly: even at temperature 0, results won't be fully deterministic.

The usual explanation is floating-point math and GPU concurrency. A 2025 write-up from Thinking Machines Lab called Defeating Nondeterminism in LLM Inference went after that explanation and found it mostly wrong. The real culprit is batching. Production servers group incoming requests together, and the size of that group shifts minute to minute with traffic. Some GPU operations pick a different computation strategy depending on batch size, which produces tiny numeric differences, and when two tokens sit nearly tied for first place a tiny difference is enough to flip which one wins.

So the answer you get can depend on how many other people happened to hit the server at the same moment as you.

05Where that leaves me

The model never chose your answer.

It handed over a ranked list of possibilities and something else reached in. When the output surprises you, that surprise came from the sampling step, not from the thing you think you're talking to.

Small thing to find out, but it changed how I think about what I'm doing every time I open a chat.

Sources

Matthew Renze and Erhan Guven, The Effect of Sampling Temperature on Problem Solving in Large Language Models, Findings of EMNLP 2024 (arXiv:2402.05201). Their null result comes from three narrower comparisons rather than one full grid across every model, prompt technique, and exam, and the numbers above 1.0 are GPT-3.5 alone. HalluDial (2024) for the hallucination rate curve, which is a secondary analysis in that paper rather than its headline result. Thinking Machines Lab, Defeating Nondeterminism in LLM Inference (2025), on batch invariance. Anything wrong here is mine, not theirs.