Models, training data and parameters

Every AI headline you will ever read leans on three words: model, data and parameters. News stories throw them around as if everyone knows what they mean, and most people quietly do not. Once you do, a lot of AI coverage becomes much easier to judge.

A model is a trained program, not a database

A model is a computer program whose behaviour was learned from examples rather than written line by line by a programmer. When a chatbot answers your question, it is not searching a stored copy of the internet and pasting back the best page. There is no page. The model generates its answer fresh, word by word, based on patterns it absorbed during training. That is why it can produce sentences nobody has ever written, and also why it can produce sentences that are simply wrong. It is closer to a very well read improviser than to a library.

So how does a program learn from examples? Through an enormous game of guess and check. During training the model is shown a fragment of text with the next word hidden, and it guesses what comes next. When the guess is wrong, the training process nudges the model's internal numbers a tiny amount in the direction that would have made the right word more likely. Then it does this again, and again, billions upon billions of times, across the whole dataset. No single nudge teaches the model anything much. But stack up trillions of tiny corrections and the model becomes very good at predicting what text comes next, which turns out to be a surprisingly powerful skill, because predicting text well forces it to absorb grammar, facts, styles and arguments along the way.

Training data: what the model learned from

Training data is the collection of examples a model learns from. For large language models that means enormous amounts of text: websites, books, code, articles and forum posts, often trillions of words. For image models it means huge collections of pictures paired with descriptions. The data is the model's entire experience of the world. A model has never seen anything, done anything or checked anything. It has only ever processed its data, so the strengths and gaps of that data become the strengths and gaps of the model.

A common misconception is that the model keeps a copy of all that text inside it, like a giant zip file. It does not, and the sizes prove it: the training data can run to hundreds of terabytes while the finished model fits on a single hard drive. What survives training is not the text but the patterns squeezed out of it, the way you can remember the plot of a novel and the way the author writes without being able to recite a single page. This is also why a model usually cannot tell you where it learned something. The fact is not filed under a source. It is smeared across billions of numbers.

Parameters: billions of tiny dials

Inside a model are parameters: adjustable numbers, like billions of tiny dials. Training is the process of turning those dials, a little at a time, until the model's outputs match the patterns in the data. When a company says its new model has 70 billion parameters, it is telling you how many dials there are. All of the model's learned knowledge lives in the settings of those numbers, which is a strange and important idea: there is no list of facts inside, only dials that make some word sequences more likely than others.

Pause on the scale for a moment, because 70 billion is not a number humans picture well. If you typed one parameter per second, day and night, you would finish in a little over 2,000 years. Every one of those numbers started as a random value and was nudged into place by training. And when the model writes a single word of an answer, the calculation runs through all of them. The knowledge is not sitting in any one dial, the same way a song is not sitting in any single groove of a record. It only exists in the whole arrangement.

Is bigger better?

Headlines love parameter counts because big numbers sound impressive. More parameters do generally let a model capture more patterns, but bigger is not automatically better. A smaller model trained on carefully chosen data can beat a giant trained on sloppy data. Bigger models also cost more to run, respond more slowly and use more energy. And for a specific job, like sorting school emails, a small model tuned for that task can outperform a general giant. When a headline treats size as proof of quality, it is skipping the questions that matter: trained on what, and tested how?

Frozen until retrained

One more property matters for reading headlines: a model stops learning the day training ends. That date is called the knowledge cutoff, and everything after it simply does not exist for the model, which is why a chatbot can discuss ancient Rome fluently and know nothing about last week's football results. When a chatbot does seem to know recent news, that is because a search tool has been attached which fetches fresh pages and drops them into the input, not because the model itself learned anything new. And nothing you type in a conversation changes the dials either. The model you finish a chat with is exactly the model you started with, which is a genuine difference from a human tutor, who remembers you tomorrow.

These three words are the foundation for everything else in this topic. The next lesson looks at the other half of the picture: if the model is fixed once trained, then the input you give it is the only lever you hold, and it turns out to be a bigger lever than most people expect.