Twin neurons

A neural network learns by turning its dials, the weights. Every dial needs a first value. Why not give them all the same one?

Take a small network: 2 inputs, 4 hidden neurons, 1 output. Set every dial to 0.5 and every bias to 0. Now all four hidden neurons see exactly the same thing, so they compute the same value. They make the same mistake, so training gives each one the same push. After the push they are still identical, so the next step repeats it.

We trained this network for 2,000 steps. The four hidden neurons stayed exactly identical the whole time: the difference between them was 0.0 at every step. They are twins, forever.

Every dial starts at 0.5: the four hidden neurons get the same push at every step. Step 2,000, still twins.

Starting every dial at zero is even worse.…