Hallucination: confident and wrong

Ask a chatbot for the population of Sydney and you will probably get a decent answer. Ask it for the five best sources on a niche history topic and you may get a beautifully formatted list in which two of the books do not exist. This is hallucination: fluent, confident, plausible and false. It is the single most important limitation to understand.

Prediction machines produce plausible text, not verified text

Hallucination is not a rare malfunction. It is the normal machinery running as designed. A language model generates whatever text is most plausible given its patterns, and truth is not one of its inputs. Facts repeated millions of times in training data usually come out right, because the true version is also the most statistically likely one. But when the data is thin, the model does not stop and say it does not know. It keeps generating, and what comes out is shaped like a fact without being one.

Why does the model not simply say it does not know? Because saying I do not know is itself just a pattern, and it is a rare one in the training data. The internet is mostly written by people who felt qualified to write, so the overwhelming pattern is a question followed by a confident answer. The model also has no internal meter showing how solid its ground is. From the inside, generating a true sentence and generating a false one are the same operation: both are just the next most plausible words. Newer models are trained to decline more often, and it helps, but it works against the grain of everything else the model learned, which is why a polite refusal is still much rarer than a fluent guess.

Invented citations and quotes

The most dangerous hallucinations are invented references. A model has read millions of citations, so it knows exactly what one looks like: plausible author, plausible title, plausible journal, plausible year. It can generate a perfect specimen for a paper that was never written. The same goes for quotes, court cases and statistics. Lawyers have been sanctioned in real courts for filing documents citing cases an AI invented. The polish is precisely the trap: a fabricated reference looks identical to a real one until you go and check it exists.

The best known example is now taught in law schools. In 2023 a New York lawyer filed a court brief citing six earlier cases to support his client's position, all found using a chatbot. None of the six existed. The model had generated them, complete with convincing names, citations and quotes, and when the lawyer asked the chatbot whether the cases were real, it assured him they were. The court fined the lawyers involved, and the story travelled around the world. Notice the second mistake, because it is the instructive one: asking the model to verify its own output. That is asking the same machinery that produced the fabrication to now produce a judgement about it, and the most plausible continuation of are you sure is yes.

Confidence carries no information

Humans use tone as a signal. Someone who hesitates is unsure. Someone who answers smoothly probably knows. Models break this instinct completely, because they generate correct answers and fabrications with exactly the same fluent, assured voice. The confidence of an AI answer tells you nothing about its truth. Nothing. Judging AI output by how sure it sounds is like judging a book by how firmly the author gripped the pen.

When to be most suspicious

Hallucination risk is not evenly spread. It is highest for specific facts like exact dates, numbers, names and prices, for niche topics that are thin in the training data, for references of any kind, and for anything after the model's training data ends. It is lowest for general explanations of well known ideas, and for tasks where the truth is sitting in your prompt, like summarising text you pasted. The habit that follows: the more specific and checkable a claim, the more it needs checking.

A misconception to clear up: hallucination is not a growing pain that the next model version will cure. Rates have fallen as models improve, and features like web search help by putting real pages into the context window. But the underlying machinery is still plausible continuation, so the floor is not zero, and a model with search can still misread a real source or stitch two real facts into one false claim. Treat every improvement as lowering the odds, never as removing the need to check.

Working with a machine that makes things up

None of this means the tool is useless. It means you have to hold it the right way. Use models for the work where being plausible is the whole job: drafting, rewording, explaining a concept from five angles, generating questions to test yourself with, structuring an argument. Keep the facts on your side of the table: paste in the source material you want it to work from, and treat every specific claim it volunteers, every date, number, name and reference, as unverified until you have found it somewhere independent. For a school assignment the division of labour is clean. Let the model help you understand the topic and organise your thinking, and let your textbook, the library database or another checkable source supply every fact you cite. The model drafts. You verify. That order never reverses.

Hallucination is one of two famous failure modes, and they are siblings. Both come from the same source: a model can only ever reflect the patterns in its data. When the data is thin, you get hallucination. When the data is tilted, you get the subject of the next lesson, bias.