Inside a Neural Network

Watch a neural network learn to read handwritten digits one image at a time, then draw your own digit and see which neurons light up. Everything runs in your browser, and the network is written from scratch.

Start training ↓ Draw a digit ↓

Loading 12,000 handwritten digits…

1Build the network

Before training a network, it helps to know what one is. Its building block, the artificial neuron, is an idea more than 80 years old. This section covers where it came from, how it works, how thousands of them become a few matrix multiplications, and how a network measures its own mistakes. Then you'll build one.

Where the artificial neuron came from

  1. 1943
    A neuron as a logic gate. Warren McCulloch and Walter Pitts modelled a brain cell as a simple unit: it adds up incoming signals and “fires” if the total crosses a threshold. Networks of these units could compute logical functions, but every connection had to be set by hand. Nothing was learned.
  2. 1949
    Learning as changing connections. Psychologist Donald Hebb proposed that learning happens when connections between neurons that are active together grow stronger. The idea that knowledge lives in connection strengths is still at the heart of every neural network.
  3. 1958
    The perceptron learns. Frank Rosenblatt built a neuron that learned its own weights from examples. When it answered wrongly, each weight was nudged toward the right answer. His Mark I Perceptron used a 20×20 grid of light sensors, a crude 400-pixel camera, to learn to tell simple shapes apart. The press predicted machines that would soon walk, talk and think.
  4. 1969
    The limits of one layer. In their book Perceptrons, Marvin Minsky and Seymour Papert showed that a single layer can only separate categories with a straight line (in many dimensions, a flat plane). It can't even learn XOR (“one or the other, but not both”). Stacking layers would fix this, but nobody knew how to train the hidden ones. Enthusiasm and funding faded for more than a decade.
  5. 1986
    Backpropagation. David Rumelhart, Geoffrey Hinton and Ronald Williams popularised a way to train hidden layers: send the error backwards through the network with the chain rule from calculus, working out each weight's share of the blame. (The underlying maths had appeared earlier, in work by Seppo Linnainmaa and Paul Werbos.) It needed smooth activation functions instead of the perceptron's hard on/off threshold.
  6. 1989–98
    Reading handwriting. Yann LeCun and colleagues trained networks to read handwritten ZIP codes and bank cheques. In 1998 LeCun, Corinna Cortes and Christopher Burges released MNIST: 70,000 handwritten digits that became the “hello world” of machine learning. They're the digits on this page.
  7. 2012–
    Deep learning. Large datasets, graphics cards (GPUs, very fast at matrix maths) and simpler activations like ReLU made much deeper networks practical. A network called AlexNet won the 2012 ImageNet image-recognition contest by a wide margin. Today's image, speech and language models are still built from the same weighted-sum-plus-activation neuron.

Anatomy of a neuron

A biological neuron collects signals from other cells through branches called dendrites. If the combined stimulation is strong enough, it fires a pulse down its axon to the next cells. Some connections (synapses) are stronger than others, and some inhibit rather than excite. The artificial neuron keeps just this skeleton. The analogy is loose: real neurons are far more complicated.

  1. Weights. Each input xi is multiplied by a weight wi. A positive weight excites the neuron, a negative one inhibits it, and a larger value means a stronger connection.
  2. Bias. A number b added on top, setting how easily the neuron fires even before any input arrives.
  3. Sum. z = w1x1 + w2x2 + … + wnxn + b
  4. Activation. a = f(z). The activation function f turns the sum into the neuron's output.

The perceptron used a hard step: output 1 if z > 0, otherwise 0. That works for one neuron trained with Rosenblatt's rule, but it blocks backpropagation. The step is flat everywhere, so a small change to a weight usually changes nothing, and there's no slope to say which direction is better. Smooth functions like the sigmoid fixed this. That's what the activation choice below is about.

Try it: one neuron Drag the sliders. Blue connections excite, red ones inhibit.

From one neuron to a layer: matrices

A layer is a group of neurons that all read the same inputs, each with its own weights. Write each neuron's weights as one row, stack the rows, and you have a weight matrix W. All the layer's sums then come from a single matrix–vector multiplication:

z = W x + b,   a = f(z)

Row i of W multiplied by the input vector x is exactly neuron i's weighted sum. A whole network is these operations chained: each layer's output vector becomes the next layer's input.

Training libraries go one step further: they stack many images side by side, turning many matrix–vector products into one matrix–matrix product. GPUs were built to do exactly that for 3D graphics, which is a big reason deep learning took off on them. This page processes one image at a time to keep each step visible.

Worked example: a layer of 2 neurons with 3 inputs
W
0.5−1.02.0
1.50.0−0.5
×
x
1.0
0.5
0.25
+
b
0.1
−0.2
=
z
0.6
1.175

Highlighted row: 0.5×1.0 + (−1.0)×0.5 + 2.0×0.25 + 0.1 = 0.6.
Second row: 1.5×1.0 + 0.0×0.5 + (−0.5)×0.25 − 0.2 = 1.175.

Measuring mistakes: the loss function

The output layer has 10 neurons, one per digit. Their raw sums are turned into probabilities that add up to 1 by the softmax function:

pk = ezk / (ez0 + … + ez9)

To learn, the network needs a single number saying how wrong it was: the loss. For classification the standard choice is cross-entropy, minus the logarithm of the probability given to the correct digit:

L = −ln(pcorrect)

A sure, correct answer (p = 1) costs nothing. A hesitant one costs a little. A confident wrong answer costs a lot, because the logarithm shoots up as p approaches 0. A useful side effect: the error signal at the output works out to simply p minus the right answer (1 for the correct digit, 0 for the rest). That signal is what backpropagation sends back through the network.

Try it: how much does an answer cost?

Learning: walking downhill

Training means finding weights that make the loss small. Imagine the loss as a landscape with one dimension per weight, tens of thousands of them, that you have to descend in fog. You can't see the valley, but you can feel the slope under your feet. Backpropagation measures that slope for every weight at once: ∂L/∂w, how much the loss would change if the weight were nudged up. Then gradient descent takes a small step downhill:

w ← w − η · ∂L/∂w

η (eta) is the learning rate, the step size, which you can set in step 2. Too small and learning crawls; too large and it overshoots the valley and can blow up. On its way back, the error passes through each neuron's activation slope f′(z). Where that slope is tiny, as in the flat tails of the sigmoid, the signal fades layer by layer. Where it's exactly zero, as in the step function, learning stops entirely. That's why the choice of activation matters so much.

Now build yours

Every hidden neuron below uses the activation you pick here. Each one also shows its slope, the dashed line, which is how much error it lets flow back during learning.

Changing either setting starts over with a fresh, untrained network.

f(x): neuron output for input x f′(x): slope; how much error flows back

2Watch it learn

  1. Forward pass. An image's 784 pixels flow left to right. Every neuron weighs its inputs and fires.
  2. Compare. The 10 outputs are the network's confidence in each digit. The further it is from the right answer, the bigger the loss.
  3. Backpropagate. The error flows right to left, working out how much each weight is to blame.
  4. Update. Every weight is nudged slightly in the direction that would have reduced the error. Then the next image arrives.
Epoch
1
one pass = 10,000 images
Images seen
0
one at a time
Weight updates
0
Recent accuracy
–
last 500 training images
Test accuracy
–
2,000 images it never trains on

Drag the slider to browse the training images and see how the network currently responds to each one. Training pauses while you browse, and resumes from the selected image. Hover over any neuron for details.

Where is the knowledge? Every single image changes all the weights at once, each by a tiny amount. No single weight means “loop” or “straight line”. What the network knows is spread across all of them, which is why trained networks are so hard to interpret. Hover over a first-layer neuron to see the pattern its weights have drifted into.

Accuracy while training Recent training accuracy Test accuracy (checked each epoch and when paused)

3Try it yourself

Draw a digit below. It goes through the same network you trained above. Keep training while your drawing is here and watch the guess firm up.

What the network sees:
cropped, scaled and centred to 28×28 like MNIST
Network's guess
?
Draw a digit (0–9) in the box