The tiny logic puzzle that stalled neural networks, and the hidden layer that solved it.
XOR looks like the easiest problem in the world. It has two inputs, four cases, and a one-line rule. Yet a single artificial neuron cannot learn it, no matter how long you train it.
Understanding why takes one picture. And the fix explains why every modern neural network has hidden layers.
What XOR does
XOR stands for "exclusive or." It outputs 1 when exactly one of its two inputs is 1.
0 and 0 give 0.
0 and 1 give 1.
1 and 0 give 1.
1 and 1 give 0.
So it's "one or the other, but not both."
The XOR truth table. The output is 1 only when the two inputs differ.
Put the four cases on a grid
Now plot each case as a point, with the first input across and the second input up. Color the points by their answer: blue for 1, hollow for 0.
The two 1s sit on one diagonal. The two 0s sit on the other diagonal.
One neuron, one straight line
A single neuron (a perceptron) decides by drawing one straight line.…