On #robot judges: part 6 - in which I suggest that we can never eliminate the hallucination problem.

In part 5,¹ I suggested that a Large Language Model (#LLM) is like a magic machine that generates bricks: much of its output is useful, but it occasionally hallucinates (i.e. makes up facts confidently, which is analogous to the machine generating a dodgy brick).

But surely LLMs will improve over time, such that there will be fewer and fewer hallucinations? Doesn't that justify their widespread use in future, if not now?

In order to answer that question, let's consider what it takes to ensure that an LLM is not hallucinating.

---

Let's go back to the magic brick machine.²

One day, our mysterious benefactor reappears, waves their gizmo or wand over the machine, and proclaims that from now on, the machine will never produce dodgy bricks.

We want to believe them.

But would it be prudent to simply take their word for it, and start using the magic bricks with no safeguards in place?

If we prefer not to be sued, probably not.³

Now, suppose we check every brick that's generated. 1,000 bricks later, we have not come across a dodgy brick. Does this mean that we can be sure that the machine will never generate dodgy bricks? What about after 10,000 bricks? 100,000 bricks?

We can never be sure, unless we:

(a) find out how the machine is generating bricks; and
(b) can confirm that the process will never lead to dodgy bricks being generated.

So we take the magic brick machine apart to see how it works. We discover that at its core, there is an unopenable black box that is integral to the generation process, and we don't know what goes on inside that black box.

So long as we do not know how the machine works, we cannot know for sure whether the machine will, one day, generate a dodgy brick.

---

I therefore suggest that we will never know for sure if an LLM has truly, completely, eliminated hallucinations.⁴

Even if a human operator reviews 100,000 or a million instances of an LLM's output and confirms that there has been zero instances of hallucination,⁵ there is still no guarantee that hallucinations will never be generated.

And that's because we don't know, for sure, why an LLM generates the outputs that it does.⁶

We can't review every line of code in the LLM to identify the precise reasons why a certain prompt generates a certain output. And that's because the rules which the LLM applies are machine-generated and undefined, as opposed to being in the form of classic if-then(-else) code.

---

Now, let me be clear - I am NOT saying we should therefore never use LLMs.

But what I am saying is that before we use a LLM for a particular project, we need to consider:

(a) whether the project can tolerate errors; and
(b) what is the system we have in place to catch errors.⁷

In part 7, we'll finally discuss why I suggest that using an LLM as a robot judge is problematic.