Neural networks

Neural networks sound like artificial brains, and that image is exactly what misleads. An artificial neuron has about as much to do with a real nerve cell as a paper plane with an eagle: the basic idea is borrowed, the rest is something entirely different. What an artificial neuron really does fits into one line of maths. The astonishing part is not the single neuron, but what happens when you chain very many of them together.

A mini neuron does the maths

An artificial neuron receives several inputs, for example brightness values of pixels. Each input has a weight, a number that says how important this input is. The neuron multiplies each input by its weight, adds everything up and compares the sum with a threshold: if it is above, the neuron fires and passes a signal on, otherwise it stays silent. A worked example: two inputs with values 1 and 0, the weights are 0.8 and 0.4, the threshold is 0.5. The sum is 1 times 0.8 plus 0 times 0.4, so 0.8. That is above 0.5, the neuron fires. That is all there is: multiply, add, compare.

The weights are learned

Nobody sets the weights by hand. They start with random values, and then the training you already know takes over: the network gets an example, computes its output and compares it with the correct answer. If it is off, all weights are shifted a tiny step in the direction that would have made the error smaller. If a neuron fired when it should not have, the weights of the inputs responsible go down, in the opposite case they go up. After millions of such corrections, weights have formed that work well on the training data. Learning here is once again nothing but shrinking an error, just at very many knobs at once.

From one neuron to layers

A single neuron can only separate very simple patterns, it essentially draws a single dividing line. The power comes from layers: the outputs of the first neurons become inputs of the next ones, whose outputs feed the ones after that. In a face recognition network the early layers respond to simple things like edges and changes in brightness. Middle layers combine these edges into shapes like eyes or noses, and late layers assemble whole faces from them. The baffling part: nobody programmed this division of labour. It emerges on its own, because it is the most effective way to lower the training error. One point that matters for counting: every neuron has its own weights, one for each connection. A layer of 5 neurons that all see the same 10 inputs therefore has 5 times 10, that is 50 weights. By the way, deep learning only means that a network has many such layers, the word deep describes the architecture, not the thoughts.

Big, powerful and still just numbers

Modern networks have millions to many billions of weights, the largest language models even more. Training them consumes huge amounts of data and electricity. The result can translate language, mark tumours on X-ray images and write text. Yet it remains a chain of multiplying, adding and comparing, executed billions of times. A network does not know what it is doing and cannot justify its answers. Even experts often cannot tell from a trained network why it decided a particular case the way it did. Holding both in view at once, the impressive performance and the simple mechanics behind it, is the most honest way to look at this technology.