1Build the network
Before training a network, it helps to know what one is. Its building block, the artificial neuron, is an idea more than 80 years old. This section covers where it came from, how it works, how thousands of them become a few matrix multiplications, and how a network measures its own mistakes. Then you'll build one.
Where the artificial neuron came from
- 1943A neuron as a logic gate. Warren McCulloch and Walter Pitts modelled a brain cell as a simple unit: it adds up incoming signals and “fires” if the total crosses a threshold. Networks of these units could compute logical functions, but every connection had to be set by hand. Nothing was learned.
- 1949Learning as changing connections. Psychologist Donald Hebb proposed that learning happens when connections between neurons that are active together grow stronger. The idea that knowledge lives in connection strengths is still at the heart of every neural network.
- 1958The perceptron learns. Frank Rosenblatt built a neuron that learned its own weights from examples. When it answered wrongly, each weight was nudged toward the right answer. His Mark I Perceptron used a 20×20 grid of light sensors, a crude 400-pixel camera, to learn to tell simple shapes apart. The press predicted machines that would soon walk, talk and think.
- 1969The limits of one layer. In their book Perceptrons, Marvin Minsky and Seymour Papert showed that a single layer can only separate categories with a straight line (in many dimensions, a flat plane). It can't even learn XOR (“one or the other, but not both”). Stacking layers would fix this, but nobody knew how to train the hidden ones. Enthusiasm and funding faded for more than a decade.
- 1986Backpropagation. David Rumelhart, Geoffrey Hinton and Ronald Williams popularised a way to train hidden layers: send the error backwards through the network with the chain rule from calculus, working out each weight's share of the blame. (The underlying maths had appeared earlier, in work by Seppo Linnainmaa and Paul Werbos.) It needed smooth activation functions instead of the perceptron's hard on/off threshold.
- 1989–98Reading handwriting. Yann LeCun and colleagues trained networks to read handwritten ZIP codes and bank cheques. In 1998 LeCun, Corinna Cortes and Christopher Burges released MNIST: 70,000 handwritten digits that became the “hello world” of machine learning. They're the digits on this page.
- 2012–Deep learning. Large datasets, graphics cards (GPUs, very fast at matrix maths) and simpler activations like ReLU made much deeper networks practical. A network called AlexNet won the 2012 ImageNet image-recognition contest by a wide margin. Today's image, speech and language models are still built from the same weighted-sum-plus-activation neuron.
Anatomy of a neuron
A biological neuron collects signals from other cells through branches called dendrites. If the combined stimulation is strong enough, it fires a pulse down its axon to the next cells. Some connections (synapses) are stronger than others, and some inhibit rather than excite. The artificial neuron keeps just this skeleton. The analogy is loose: real neurons are far more complicated.
- Weights. Each input xi is multiplied by a weight wi. A positive weight excites the neuron, a negative one inhibits it, and a larger value means a stronger connection.
- Bias. A number b added on top, setting how easily the neuron fires even before any input arrives.
- Sum. z = w1x1 + w2x2 + … + wnxn + b
- Activation. a = f(z). The activation function f turns the sum into the neuron's output.
The perceptron used a hard step: output 1 if z > 0, otherwise 0. That works for one neuron trained with Rosenblatt's rule, but it blocks backpropagation. The step is flat everywhere, so a small change to a weight usually changes nothing, and there's no slope to say which direction is better. Smooth functions like the sigmoid fixed this. That's what the activation choice below is about.
From one neuron to a layer: matrices
A layer is a group of neurons that all read the same inputs, each with its own weights. Write each neuron's weights as one row, stack the rows, and you have a weight matrix W. All the layer's sums then come from a single matrix–vector multiplication:
z = W x + b, a = f(z)
Row i of W multiplied by the input vector x is exactly neuron i's weighted sum. A whole network is these operations chained: each layer's output vector becomes the next layer's input.
Training libraries go one step further: they stack many images side by side, turning many matrix–vector products into one matrix–matrix product. GPUs were built to do exactly that for 3D graphics, which is a big reason deep learning took off on them. This page processes one image at a time to keep each step visible.
| 0.5 | −1.0 | 2.0 |
| 1.5 | 0.0 | −0.5 |
| 1.0 |
| 0.5 |
| 0.25 |
| 0.1 |
| −0.2 |
| 0.6 |
| 1.175 |
Highlighted row: 0.5×1.0 + (−1.0)×0.5 + 2.0×0.25 + 0.1 = 0.6.
Second row: 1.5×1.0 + 0.0×0.5 + (−0.5)×0.25 − 0.2 = 1.175.
Measuring mistakes: the loss function
The output layer has 10 neurons, one per digit. Their raw sums are turned into probabilities that add up to 1 by the softmax function:
pk = ezk / (ez0 + … + ez9)
To learn, the network needs a single number saying how wrong it was: the loss. For classification the standard choice is cross-entropy, minus the logarithm of the probability given to the correct digit:
L = −ln(pcorrect)
A sure, correct answer (p = 1) costs nothing. A hesitant one costs a little. A confident wrong answer costs a lot, because the logarithm shoots up as p approaches 0. A useful side effect: the error signal at the output works out to simply p minus the right answer (1 for the correct digit, 0 for the rest). That signal is what backpropagation sends back through the network.
Learning: walking downhill
Training means finding weights that make the loss small. Imagine the loss as a landscape with one dimension per weight, tens of thousands of them, that you have to descend in fog. You can't see the valley, but you can feel the slope under your feet. Backpropagation measures that slope for every weight at once: ∂L/∂w, how much the loss would change if the weight were nudged up. Then gradient descent takes a small step downhill:
w ← w − η · ∂L/∂w
η (eta) is the learning rate, the step size, which you can set in step 2. Too small and learning crawls; too large and it overshoots the valley and can blow up. On its way back, the error passes through each neuron's activation slope f′(z). Where that slope is tiny, as in the flat tails of the sigmoid, the signal fades layer by layer. Where it's exactly zero, as in the step function, learning stops entirely. That's why the choice of activation matters so much.
Now build yours
Every hidden neuron below uses the activation you pick here. Each one also shows its slope, the dashed line, which is how much error it lets flow back during learning.
Changing either setting starts over with a fresh, untrained network.
2Watch it learn
- Forward pass. An image's 784 pixels flow left to right. Every neuron weighs its inputs and fires.
- Compare. The 10 outputs are the network's confidence in each digit. The further it is from the right answer, the bigger the loss.
- Backpropagate. The error flows right to left, working out how much each weight is to blame.
- Update. Every weight is nudged slightly in the direction that would have reduced the error. Then the next image arrives.
Drag the slider to browse the training images and see how the network currently responds to each one. Training pauses while you browse, and resumes from the selected image. Hover over any neuron for details.
Where is the knowledge? Every single image changes all the weights at once, each by a tiny amount. No single weight means “loop” or “straight line”. What the network knows is spread across all of them, which is why trained networks are so hard to interpret. Hover over a first-layer neuron to see the pattern its weights have drifted into.
3Try it yourself
Draw a digit below. It goes through the same network you trained above. Keep training while your drawing is here and watch the guess firm up.