ReLu Function in Python With Examples

Relu or Rectified Linear Activation Function is the most common choice of activation function in the world of deep learning. Relu provides state of the art results and is computationally very efficient at the same time.

The basic concept of Relu activation function is as follows:

We can represent it mathematically as follows:

 the ReLu function.

Relu Function

The pseudo code for Relu is as follows:

In this tutorial, we will learn how to implement our own ReLu function, learn about some of its disadvantages and learn about a better version of ReLu.

Recommended read: Linear Algebra for Machine Learning [Part 1/2]


Let’s get started!

Implementing ReLu function in Python

Let’s write our own implementation of Relu in Python. We will use the inbuilt max function to implement it.

The code for ReLu is as follows :

To test the function, let’s run it on a few inputs.

Complete Code

The complete code is given below :

Output :

Gradient of ReLu function

Let’s see what would be the gradient (derivative) of the ReLu function. On differentiating we will get the following function :

We can see that for values of x less than zero, the gradient is 0. This means that weights and biases for some neurons are not updated. It can be a problem in the training process.

To overcome this problem, we have the Leaky ReLu function. Let’s learn about it next.

Leaky ReLu function

The Leaky ReLu function is an improvisation of the regular ReLu function. To address the problem of zero gradient for negative value, Leaky ReLu gives an extremely small linear component of x to negative inputs.

Mathematically we can express Leaky ReLu as:


  • f(x)=1 (x<0)
  • (αx)+1 (x>=0)(x)

Here a is a small constant like the 0.01 we’ve taken above.

Graphically it can be shown as :

ReLu Function in Python

Leaky ReLu

The gradient of Leaky ReLu

Let’s calculate the gradient for the Leaky ReLu function. The gradient can come out to be:

In this case, the gradient for negative inputs is non-zero. This means that all the neuron will be updated.

Implementing Leaky ReLu in Python

The implementation for Leaky ReLu is given below :

Let’s try it out onsite inputs.

Complete Code

The complete code for Leaky ReLu is given below :

Output :


This tutorial was about the ReLu function in Python. We also saw an improved version of the ReLu function. The Leaky ReLu solves the problem of zero gradients for negative values in the ReLu function.

By admin

Leave a Reply