Alea iacta non est: rerolling the LLM dice

Like everyone else, I talk a lot to various LLMs. And sometimes I wonder how consistent their answers are. I know that LLMs are non-deterministic, but there's a difference between giving wildly different answers and saying the same thing with different words. Sometimes, this is because there are genuinely multiple valid answers, but at other times, the question is underspecified and then the LLM is forced to make guesses as to how to answer.

For example, sometimes I point it towards a certain framework (eg: Demartini Value Determination Framework) and ask it to walk me through it one question at a time. In this case I'm curious how consistently it interprets my answers the same way and if it guides me through that framework consistently.

Sometimes I ask it for some options and recommendations and in this case I'm wondering if those really are the "best" options, or just what came up with the roll of the dice. But the normal chat interface gives me no way to tell the difference.

To test this out, I vibe-coded a little app "reroll" that looks very similar to a normal LLM interface, but it generates every reply 5 times. It sends all 5 replies to Haiku to analyze for similarities and tells me what it thinks about the variation in answers.

Each example below links to a recorded session. The chat shows one of the five replies: the one I continued the conversation with. Under every reply there's a box with Haiku's verdict. Click it to see all five replies side by side, grouped by what Haiku thinks they are doing.

The verdict has two parts. First, how much the five agree, based on the biggest group: uniform (all five do the same thing), mostly uniform (4 of 5), split (3 of 5) or highly divergent (no group bigger than 2). The little bar shows the group sizes. Second, how they differ:

• different next moves: they understood me the same way but went in different directions, like suggesting different stops or asking different questions. Often harmless, and sometimes exactly what you want.
• read you differently: they interpreted my message in different ways, which usually means I didn't give enough context. Haiku then suggests what I could add.
• contradicting each other: they make claims that can't all be true, a hint that at least some of them are making things up.

For example, recently when returning from Croatia, I asked ChatGPT:

I'm close to Zagreb, driving towards Oradea. Can you suggest me the best place to stop along the way?

It recommended me two possible stops, one which didn't look right on Google Maps, so I used the other one, which was a generic gas station stop on the highway. I retried this later with reroll and I couldn't have received more varied answers. It suggested completely different routes, and even when two replies chose the same route, they recommended different stops. And even the distances it gave between cities were different.

Sometimes, refining the prompt helps to narrow it down. If I ask it:

We're looking for some summer seaside trips from Oradea. What would you recommend?

I get 3 clusters in the 5 answers. And it basically assumed driving, with just a small mention at the end about flying. And 4 out of 5 said "Oradea is actually well placed for seaside trips" - what, the closest sea is a 9-hour drive away.

But if I give it some more details:

We're looking for some summer seaside trips from Oradea. What would you recommend? We don't want to drive more than 5-6 hours, but we are open to flying.

In this case, all of them mention Antalya, for example, which has charter flights from Oradea. (Haiku's verdict is a bit off here: it put all five replies in one group but still labeled them "different next moves", and its summary talks about driving to Croatia without mentioning Antalya, which every reply suggests.)

Sometimes, one answer says it doesn't know something, and another one knows about it, so they clearly took very different paths through latent space. For example, asking the following:

"What's are the top 5 maximally different ways to lose weight? Think calorie deficit, exfat150, HCLF, etc"

Leads to 1 reply misunderstanding exfat150, 3 saying they don't know what it is, and one correctly saying what it is (mind you, this is via simple API calls, so the LLM doesn't have access to search the internet). Btw, asking simply for weight loss tricks, even with "top 5 maximally different" in the prompt, makes all the replies give pretty much the same suggestions. But hey, kudos to exfatloss for generating interesting paths through nutrition and through latent space.

One more example: going through the Demartini Value Determination Framework. As expected, it starts off consistently. It's a list of questions. First 3 don't generate almost any variation. But then they start interpreting my answers differently, and sometimes even start going off in a different direction. But when pushing back on something, all of them caved, showing a bit of model sycophancy.

So with this tool, we've seen that:

• Many of the recommendations can be pretty arbitrary. Running it 5 times can give different options, or the same ones but with different distances for example (so some of them must be wrong).
• Giving better context helps the LLM narrow down the answers, leading to less variation.
• Rare knowledge is in the LLMs, but sometimes it doesn't want to come out easily.
• Even a well known and quite rigid framework (such as Demartini's) eventually leads to interpretation and that leads to LLMs making their own choices.

So, reroll is not just a neat debugging tool, but it shows that conventional chat UIs hide an important property of the underlying system. It's a neat little tool, and I'll continue using it to explore the latent space of LLMs and to be a bit more confident about the stability of the answers. It doesn't tell me if they are correct, but it helps tells me about the stability of the interpretation (did the LLM understand what I asked) and of the answer (is it reaching the same conclusions).

One caveat: it's a bit expensive: trying Opus 5 in 16 conversations (which were not even that long), via API, cost me around $8.