Training data

People talk a lot about models and little about data. Yet the data decides what a model can learn at all. A model has no source about the world other than its training examples: what is missing there does not exist for the model, and what is wrong there it takes to be true. To understand machine learning you therefore first have to understand where training data comes from and what can go wrong with it.

Collecting and labelling

For supervised learning every example needs a tag with the correct answer, the so-called label. These labels do not fall from the sky: usually people sit down and label image by image, sentence by sentence. That is slow, expensive and error prone, because people often disagree. Is this comment still irony or already an insult? Two labellers give two different answers, and both end up in the dataset. Behind almost every big AI system there is therefore a huge amount of human handiwork that the finished product no longer shows.

Garbage in, garbage out

A model finds the patterns that sit in its data, not the ones we intend. A famous example from research: a model was supposed to tell huskies from wolves and got almost every test image right. Its success had a completely different reason: almost all the wolf photos had snow in the background. The model had learned to spot snow, not wolves. In the same way a model absorbs every imbalance in the data: if a group hardly ever appears in the examples, the model performs poorly for it. If past decisions were unfair, it learns the unfairness right along. The old rule of thumb still holds: garbage in, garbage out. Bad data never makes a good model, no matter how modern the algorithm is.

Training data and test data

How do you check whether a model has really learned something? Definitely not with the examples it saw during training. That would be like an exam whose questions were handed out beforehand together with the solutions: a good grade then only proves a good memory. So the data is split before training, often around 80 percent for training and 20 percent for testing. Which examples land in which part is decided at random, so that both parts look alike. The test data stays locked away until training is finished. Only the result on these held back examples shows whether the model has learned patterns that also fit new cases.

How much data is enough?

More data almost always helps, but volume alone is not enough. Ten thousand nearly identical photos are worth less than a thousand that cover all the important cases: different angles, lighting conditions, rare edge cases. What matters is that the training data looks like the data the model will meet later. A model trained only on studio shots fails on shaky phone photos. Good teams therefore often spend more time on their data than on their model: checking, cleaning, filling gaps, fixing labels. That sounds unspectacular, yet it is the part that decides between success and failure.