Inside one answer

Sam is 16 and has been using a chatbot for homework help for a few months. One Sunday evening, two questions are on Sam's mind. The first is for a science assignment: why is the sky blue? The second is pure curiosity: who won the local footy grand final yesterday? Sam types both into the same chat, one after the other.

The first answer is excellent. The chatbot explains that sunlight contains all colours, that air molecules scatter shorter wavelengths like blue far more than longer ones like red, that this scattering is why the sky looks blue from every direction, and that the same physics turns sunsets red when light travels through more atmosphere. It is clear, accurate and better structured than the textbook.

The second answer looks just as good. The chatbot names a winning team, gives a final margin of 14 points, and adds a warm line about it being a memorable day for the club. Sam almost repeats it to a mate before remembering that Dad watched the game. Sam checks. The named team lost. The margin is wrong too. The chatbot described a victory that never happened, in the same confident tone it used for the physics.

Why the difference? Trace the first answer. Explanations of why the sky is blue appear thousands of times in the model's training data: textbooks, science sites, forum answers, lesson plans. The physics has not changed in a century. When Sam asked, the model generated the text that such explanations typically contain, token by token, and because the pattern is so common and so consistent, likely text and true text were the same thing. The model was not remembering physics. It was reproducing the shape of a very well worn explanation, and that shape happens to be correct.

Now trace the second answer. Yesterday's local grand final happened after the model's training cutoff, and a suburban footy result was likely never going to be in the training data anyway. The model had no way to know the result. But it was asked for one, and it does not have a reliable sense of the edge of its own knowledge. So it did the only thing it ever does: it generated likely text. Grand final reports in its training data name a winner, give a margin, and strike a celebratory tone. The model produced a perfectly shaped grand final report with fabricated contents. Same mechanism as the sky answer. Completely different outcome.

Notice what did not happen. The model did not lie, because lying requires knowing the truth and hiding it. It did not malfunction, because generating likely text is exactly what it is built to do. The failure was invisible from the outside: both answers were fluent, confident and detailed. The only difference was whether the pattern being reproduced happened to line up with reality. For the sky, it did. For yesterday's game, it could not.

There is a version of this story where the second answer is fine. If Sam's chatbot had a live search feature switched on, it could have looked up yesterday's result, pulled in the real report, and written from that instead of from patterns alone. That is the general fix for the model's blind spot around recent and local facts: do not ask prediction to know what it cannot know, connect it to a source that does. Without that connection, though, the polite thing the model appears to do, giving a definite answer, is exactly the trap, because a definite answer is what a grand final report looks like, whether or not the model has any idea who won.

It also matters that nothing in the second answer looked weaker than the first. There was no stumble in the grammar, no drop in confidence, no small flag saying this part is a guess. Sam only caught it because Sam happened to have a way to check, a Dad who watched the game. Most confident errors slip through for the opposite reason, because the reader has no easy way to verify and the fluent tone does all the convincing. That is why the test Sam lands on is about the type of question, not the feel of the answer, since the feel is identical either way.

Sam now applies a simple test before trusting an answer: is this the kind of thing the training data would contain thousands of consistent examples of, or is it specific, recent or local? Stable, widely documented knowledge tends to come out right. Anything recent, local or precise gets checked against a real source. Sam still uses the chatbot constantly. Sam just knows which of its answers are standing on a mountain of examples, and which are standing on air.