How Models Learn

ReLU

Rectified linear unit: outputs z when z is positive and 0 otherwise.

Sigmoid and tanh have a slope everywhere (though it gets tiny far from 0), and ReLU has one wherever z is positive, so a small change shows which way is “better”.
From The artificial neuron, step “Different ways to squash”

See it in action

Try relu hands-on in the lesson The artificial neuron, part of the AI fundamentals course.

Open the lesson