Theano Python Tutorial With Examples [Latest]

Theano Python

Theano makes it possible to attain high speeds that give a tough competition to hand-crafted C implementations for problems involving large amounts of data. It can take advantage of recent GPUs which makes it perform better than C on a CPU by considerable orders of magnitude under certain circumstances.

Theano has got an amazing compiler which can do various optimizations of varying complexity. A few of such optimizations are:

1. Arithmetic simplification (e.g: --x -> x; x + y - x -> y)
2. Using memory aliasing to avoid calculation
3. Constant folding
4. Merging similar subgraphs, to avoid redundant calculation
5. Loop fusion for elementwise sub-expressions
6. GPU computations

You can see the full list of optimizations here.

Why Theano Python Library?

Typically we manipulate matrices using numpy package, so what make Theano better than any such package!

Theano is a sort of hybrid between numpy and sympy, an attempt is made to combine the two into one powerful library. Let’s have a look at some of its advantages over others:

• Stability Optimization: Theano can find out some unstable expressions and can use more stable means to evaluate them
• Execution Speed Optimization: As mentioned earlier, theano can make use of recent GPUs and execute parts of expressions in your CPU or GPU, making it much faster than Python
• Symbolic Differentiation: Theano is smart enough to automatically create symbolic graphs for computing gradients

Well, enough of theory, let’s start working on the example part.

Theano Tutorial

To start working with Theano, install it using PIP as shown in below image.

Theano Expression into Callable objects

With Theano, we can convert expressions into callable objects. Let’s see a code snippet:

When we run this, we get the following output:

Now, let us try to understand what happened in above program:

• We start by declaring two symbolic floating-point scalars or variables
• Then, we created a simple expression to sum two numbers
• After the expression, we convert the expression into a callable object that takes (x,y) as input and returns a value for z after computation
• Finally, we call the function with some parameters and print the results

Logistic Function

Let’s have a look at rather more elaborate example than just adding two numbers. Let’s try to compute the logistic curve, which is given by:

If we plot a graph for this equation, it will look like:

Logistic function is applied to each element of matrix. Let’s write a code snippet to demonstrate this:

When we run the script, we can see the ouput as:

Everything works fine, the output looks same as expected. Now let’s have a closer look at the functions.

Closer look at Theano Function

Theano functions help in interacting with the symbolic graph. They allow theano to build the computational graph and optimize it.

A typical theano function looks like this:

Here x is the list of input variables, and y is the list of output variables. Let’s check out how this feature is of great use.

Calculating multiple results at once

Let’s say we have to compute the elementwise difference, absolute difference and difference squared between two matrices ‘x’ and ‘y’. Doing this at the same time optimizes program with significant duration as we don’t have to go to each element again and again for each operation.

When we run this program, we can see the output as multiple results being printed:

Using Theano Gradient function

Let’s try some more useful and sophisticated functions as we move towards a minimal training example. Here we’ll try to find out the derivative of an expression with respect to a parameter

We’ll compute the gradient of the logistic function defined above, which can be plotted as:

Let’s demonstrate the working for Gradient with an example:

When we run this program, we can see the output as:

In this way Theano can be used for doing efficient symbolic differentiation (as the expression returned by tensor.grad will be optimized during compilation), even for function with many inputs

Let’s put things together into a simple training example to understand theano better!

Minimal Training Theano Example

Let’s try and train something using theano. We will be using gradient descent to train weights in W so that we get better results from the model than existing (0.9):

When we run this program, we can see the output as:

The second input variable ‘target’ will act as the target value we use for training:

We need a cost function to train the model, which is usually squared distance from target value

Next, we need to calculate partial gradients for the parameters to be updated with respect to the cost function. As we have seen that in earlier example, Theano will do that for us. We simply call the grad function with required arguments:

Now let’s define a variable for the updated version of the parameter. As we know in gradient descent the updated value equals learning rate times gradient subtracted from existing value.

Assuming learning rate(alpha) = 0.1:

Next we have to define a Theano function again, with a couple of changes:

When the function is called, it takes in values for x and target and returns the value for y as output, and Theano performs all the updates in the update list.

Now we repeatedly call the function, in order to train, 10 times in this example to be specific. Typically, training data contains different values, but for sake of this example we use the same values x=[1.0, 1.0] and target=20 each time to check things work correctly.

In the output above, notice how the target value is moving closer to 20 (target value) in each step.

Theano Neural Network Summary

In this post, we discovered the Theano Python library for efficient numerical computation.

We learned that it is a foundation library used for deep learning research and development and that it can be used directly to create deep learning models or by convenient libraries built on top of it, such as Lasagne and Keras.