Python urllib - Python 3 urllib With Examples

Python urllib module allows us to access URL data programmatically.

Python urllib

  • We can use Python urllib to get website content in python program.
  • We can also use it to call REST web services.
  • We can make GET and POST http requests.
  • This module allows us to make HTTP as well as HTTPS requests.
  • We can send request headers and also get information about response headers.

Python urllib GET example

Let’s start with a simple example where we will read the content of Wikipedia home page.

Response read() method returns the byte array. Above code will print the HTML data returned by the Wikipedia home page. It will not be in human readable format, but we can use some HTML parser to extract useful information from it.

Python urllib - Python 3 urllib With Examples

Let’s see what happens when we try to run the above program for JournalDev.

We will get below error message.

It’s because my server doesn’t allow programmatic access to the website data because it’s meant for browsers that can parse HTML data. Usually we can overcome this error by sending User-Agent header in request. Let’s look at the modified program for this.

We are creating request headers using dictionary and then sending it in the request. Above program will print HTML data received from JournalDev home page.

Python urllib REST Example

REST web services are accessed over HTTP protocols, so we can easily access them using urllib module. I have a simple JSON based demo rest web service running on my local machine created using JSON Server. It’s a great Node module to run dummy JSON REST web services for testing purposes.

Notice the console output is printing JSON data.

We can get response headers by calling info() function on response object. This returns a dictionary, so we can also extract specific header data from response.


Python urllib POST

Let’s look at an example for POST method call.

When we call urlopen function, if request has data then it automatically uses POST http method. Below image shows the output of above POST call for my demo service.

Python urllib - Python 3 urllib With Examples

Reference: API Doc

By admin

Leave a Reply