Bias: where it comes from

A model learns whatever patterns its training data contains. That single sentence explains most of what people call AI bias. The data was written and collected by humans, and humans carry prejudices, blind spots and history. The model absorbs it all without any way to tell wisdom from prejudice. Both are just patterns.

Models learn our patterns, including the ugly ones

If training text mostly describes nurses as women and engineers as men, the model learns that association and reproduces it, in stories, in translations, in the images it generates. Nobody programmed the stereotype in. It arrived the same way everything else did: as a statistical pattern in the data. The model is a mirror of what we wrote, and mirrors do not editorialise.

People sometimes object that a computer cannot be biased, because maths is neutral. And the maths is neutral, in the same way a photocopier is neutral. A photocopier does not care what is on the page, which is exactly why it reproduces whatever is there, smudges included. The unfairness does not live in the algorithm. It lives in the data the algorithm faithfully summarises, so a perfectly correct calculation on tilted data produces tilted answers, with no bug to find anywhere in the code.

Historical data encodes historical unfairness

It gets sharper when AI learns from records of past decisions. Imagine a company trains a model to screen job applications using ten years of its own hiring records. If managers in those years favoured a certain type of applicant, the data does not record that as unfairness. It records it as what success looks like. The model faithfully learns to prefer the same type, and now the old bias runs faster, at scale, wearing a lab coat of objectivity. Learning from history means learning history's unfairness along with everything else.

Walk through an illustrative version with numbers. A company has 1,000 past hires in its records, and 800 of them came from just three universities, because that is where its managers always recruited. A model trained on those records notices that attending one of the three is among the strongest signals of being hired, so it scores those applicants higher. Now a brilliant applicant from a regional university applies. Nothing about her ability has been measured at all. The model marks her down purely because people like her are rare in the record of past decisions, and the recruiter who sees the low score never learns why. And this is not hypothetical. Amazon built an experimental hiring model trained on years of its own applications, discovered it had learned to mark down applications containing the word women's, as in captain of the women's chess club, and scrapped the project.

The bias of missing data

Bias also comes from what is absent. A face analysis model trained mostly on lighter skinned faces performs worse on darker skinned faces. A voice system trained mostly on American accents stumbles on Australian ones, and worse on Indigenous ones. A medical model trained on data from one country can misfire in another. Nobody chose those failures. They are the shadow of a dataset that covered some people thoroughly and others barely at all, and the people missing from the data are usually the ones with the least power to complain.

Why this matters more than a bad chatbot answer

A biased chatbot reply is a small harm. The stakes change when models touch decisions about people: who gets shortlisted for a job, offered a loan, flagged by security, or prioritised for medical care. A biased human makes one decision at a time and can be challenged. A biased model can make thousands of decisions an hour, all tilted the same way, behind a screen that makes the outcome look neutral and mathematical. That is why the question trained on what data belongs in every conversation about AI making decisions, and why removing an obvious column like gender rarely fixes things: the pattern hides in proxies, like which suburb someone lives in or which sports they list.

An Australian warning: Robodebt

Australia has already seen what happens when an automated system makes decisions about people at scale. The Robodebt scheme, which ran in the second half of the 2010s, used an automated calculation to raise welfare debts against hundreds of thousands of people by averaging their annual income across fortnights, a method that produced debts many people never owed. Robodebt was not machine learning, just a simple automated rule, but the lessons transfer directly. The system was wrong in the same way for everyone it touched, the people affected found it nearly impossible to challenge a number produced by a process they could not see, and officials trusted the output because it came from a computer. The scheme was found unlawful, and a Royal Commission examined it in 2023. Now imagine the same dynamics with a model whose reasoning is spread across billions of parameters, and you can see why this lesson keeps insisting on the question of what the system learned from and who checks its output.

Can bias be fixed?

So can bias be fixed? Managed is a better word than fixed. Builders can measure a model's performance separately for different groups instead of quoting one blended accuracy figure, broaden the data so fewer people are missing from it, test the system on the population it will actually serve before switching it on, and keep a human able to review and overturn individual decisions. None of these is a complete cure, and the honest position is that a model trained on human records will always need watching. But there is a large practical difference between a system nobody thought to test for bias and one that is measured, monitored and open to challenge. The first step in every case is the same: someone has to ask the question in the callout below, out loud, before the system goes live.

This lesson completes the vocabulary set: model, training data, parameters, prompt, context, hallucination and bias. Seven terms, and nearly every AI headline you meet is built out of them. The case study that follows puts the whole kit to work on a single headline, because the point of the vocabulary was never the definitions. It is the questions the definitions let you ask.