Big data

While you read this sentence, millions of new data points are created worldwide: messages, photos, locations, heart rates, search queries. Almost everything you do digitally leaves a trace. Big data means these traces are so huge and arrive so fast that classic tools no longer cope and computers have to search for patterns. If you understand how this works, you can seize the opportunities and spot the risks.

How much is big?

Amounts of data are measured in bytes, and the big units grow in steps of a thousand: 1000 bytes are 1 kilobyte, 1000 kilobytes 1 megabyte, 1000 megabytes 1 gigabyte, 1000 gigabytes 1 terabyte. A phone photo is about 5 megabytes, a movie a few gigabytes. Large platforms, by contrast, process many petabytes per day, each of which is 1000 terabytes. No human can look through such amounts, from here on only algorithms do the work. One note in passing: computers often count internally in steps of 1024 instead of 1000, which is why a 1 terabyte drive shows up as only about 931 gigabytes in the operating system. Here we stick to the round steps of a thousand throughout.

What data reveals about us

Individual data points seem harmless: a like here, a location there. Things get interesting and delicate when many of them come together. From likes you can estimate preferences, moods and even political leanings surprisingly well. Location data builds a movement profile: where you live, work, exercise, whom you visit often. From purchase histories, retailers read off what you might need next. It is not the single trace that gives you away, it is the pattern across thousands of traces.

Patterns in the masses

The great strength of big data: patterns become visible across millions of cases that stay invisible in any single one. Navigation apps detect a traffic jam because hundreds of phones on one road suddenly move slowly. In medicine, rare side effects only show up once data from many clinics flows together. Streaming services recommend shows because people with similar taste liked them. The principle is always the same: many small, individually weak signals combine into a strong one.

Opportunities and downsides

The same technology has two faces. It can detect diseases earlier, smooth traffic and save energy. It can also enable seamless surveillance, sort people into filter bubbles and make prices or loans depend on opaque profiles. And patterns can deceive: if a group barely appears in the data, the algorithm makes worse decisions for that group. Big data is neither good nor evil, what matters is who collects the data, what happens with it and whether you get a say.