Menu

1 / 4

Without one bend, 100 layers act like one

DEV Community: deeplearning·Syed Abdul Hanan·16 days ago
#LSEgVqOs
#devtodeeplearning#education#devto#dev#layers
Reading 0:00
15s threshold

Why every neural network needs an activation function, explained with two simple equations.

Deep networks stack layer after layer. But there's a catch hiding in that stack. Without one small ingredient, a hundred layers can do exactly what one layer does, and nothing more.

That ingredient is the activation function. Here's why it matters, in a few lines of algebra.

What a layer does

Every layer in a neural network does two things:

It multiplies and adds: each input is multiplied by a weight, the results are summed, and a bias is added.

It passes the result through an activation function: a small bend.

The first step is simple arithmetic. The second step is the one that's easy to overlook, and it's the one that makes depth worth having.

What happens without the bend

Let's remove the activation function and watch what happens with two tiny layers.

Layer one doubles its input and adds one: h = 2x + 1 .

Layer two triples that and subtracts four: y = 3h − 4 .

Now put them together.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.