Python requests Library: Sending HTTP Requests with Python

The Python requests library allows you to send HTTP requests, such as GET and POST, and handle responses from web servers. requests is a user-friendly and powerful module for interacting with web resources.

How to Use the requests Module in Python

To work with the requests library, you first need to install it, for example, using pip:

pip install requests

Once installed, you can import the library and send requests to URLs.

The requests library supports multiple HTTP methods, including GET, POST, PUT, and DELETE. Usually, you pick the method based on your desired operation (e.g., fetching data, submitting forms, or updating resources):

• requests.get(): Sends a GET request to the specified URL.
• requests.post(): Creates a POST request, often to submit data to a server.
• requests.put(): Sends a PUT request, updating a resource on the server.
• requests.delete(): The DELETE request deletes a resource on the server.

How to Send a Python Request (GET)

In Python, GET requests using requests.get() retrieve information from a server without modifying any data. GET is the most commonly used HTTP method for accessing web pages, APIs, and other resources. Unlike other methods, the GET request parameters are part of the URL, making it ideal for read-only operations.

import requests

# Send a GET request to a webpage
response = requests.get('<https://example.com>')

# Print the status code of the response
print(response.status_code)

In this example, requests.get() sends an HTTP GET request to a URL while response.status_code provides the HTTP status code of the request (e.g., 200 for success).

How to Send a POST Request in Python

While GET retrieves data, the Python request POST sends data to a server to create or update resources. Typically, POST submits information, such as form data, user input, or API payloads, as part of the request's body.

import requests

url = '<https://api.example.com/data>'
data = {'name': 'John', 'age': 30}

response = requests.post(url, json=data)

# Print the response content
print(response.json())

In this example, requests.post() sends an HTTP POST request to a URL and passes data in JSON format. The response.json() method parses the JSON-formatted response from the server.

When to Use the Python Module requests

The requests library in Python is well-suited whenever you need your Python program to interact with web resources.

Fetching Web Page Content

You can use requests.get() to retrieve and process content from a webpage, also known as web scraping.

import requests

response = requests.get('<https://example.com>')
print(response.text)

This code sends a GET request to fetch the webpage content and prints the HTML response.

Interacting with APIs

The requests Python library is ideal for working with RESTful APIs. The library allows you to send and receive JSON data from APIs without much effort.

import requests

api_url = '<https://api.example.com/users/123>'
response = requests.get(api_url)
data = response.json()

print(data)

This example retrieves data from a user profile on an API and prints the parsed JSON response.

Sending Form Data

You can use requests.post() to send form data to a server, which is helpful for login forms or submitting search queries.

import requests

url = '<https://example.com/login>'
form_data = {'username': 'user123', 'password': 'mypassword'}

response = requests.post(url, data=form_data)
print(response.status_code)

In this example, the data parameter is used to submit form fields to the server.

Examples of Using Python Requests

Web Scraping for Price Comparison

A price comparison platform might import Python requests to scrape prices from different e-commerce websites and compare them.

import requests

url = '<https://example.com/product-page>'
response = requests.get(url)

if response.status_code == 200:
    page_content = response.text
    # Logic to extract price from the HTML content
    print("Page loaded successfully")
else:
    print("Failed to load page")

User Data Automation

Automation scripts can use the requests library to interact with third-party APIs. For example, a company might automate updating user data.

import requests

url = '<https://api.example.com/update_user>'
data = {'user_id': 123, 'new_email': 'new@example.com'}

response = requests.post(url, json=data)
print(response.json())

Website Availability Monitors

Website administrators might use Python requests to create scripts that monitor the availability of the websites and services they manage.

import requests

url = '<https://example.com>'
response = requests.get(url)

if response.status_code == 200:
    print("Website is up!")
else:
    print("Website is down!")

Learn More About Python Requests

Adding Python requests Headers

You can customize the headers by providing a header parameter with the request. A custom header parameter can help you with authorization or other customization purposes.

import requests

url = '<https://api.example.com/data>'
headers = {'Authorization': 'Bearer token123'}

response = requests.get(url, headers=headers)
print(response.status_code)

Handling Python requests Timeouts

You can set a timeout with the timeout parameter to avoid long waits for requests that take too long to complete.

import requests

try:
    response = requests.get('<https://example.com>', timeout=5)
    print(response.status_code)
except requests.Timeout:
    print("Request timed out")

This example sets a timeout of 5 seconds. If the server doesn’t respond within that time, the request raises a Timeout exception.

Using URL HTTP Parameters

You can add query parameters to your requests using the params parameter.

import requests

url = '<https://example.com/search>'
params = {'q': 'python', 'page': 2}

response = requests.get(url, params=params)
print(response.url)

This example sends a GET request with URL parameters (q for a search query and page for pagination).

Sending Files with Python Requests (POST)

Within a POST request, you can use the files parameter to upload files and other file-like objects.

import requests

url = '<https://example.com/upload>'
files = {'file': open('example.txt', 'rb')}

response = requests.post(url, files=files)
print(response.status_code)

Handling JSON Responses

If a server returns JSON data, you can use response.json() to parse the response directly into Python objects.

import requests

url = '<https://api.example.com/data>'
response = requests.get(url)

data = response.json()
print(data)

Authentication with Usernames and Passwords or Tokens

APIs or websites often require authentication through usernames and passwords or, in more modern applications, tokens. You can add credentials to your request using the auth parameter in any requests method.

import requests
from requests.auth import HTTPBasicAuth

url = '<https://example.com/api>'
response = requests.get(url, auth=HTTPBasicAuth('username', 'password'))

print(response.status_code)

Many APIs use tokens instead of passwords, which you can include in the request headers as well.

import requests

url = '<https://api.example.com/data>'
headers = {'Authorization': 'Bearer your_token_here'}

response = requests.get(url, headers=headers)

print(response.status_code)

In this case, the token is sent as part of the Authorization header using the Bearer token scheme, which is standard with API and OAuth tokens.

Handling HTTP Status Codes

HTTP status codes indicate the result of a request. The requests library makes it easy to check these status codes and handle responses based on them.

import requests

response = requests.get('<https://example.com>')

# Check if the request was successful
if response.status_code == 200:
    print("Success!")
elif response.status_code == 404:
    print("Not Found!")
else:
    print(f"Status Code: {response.status_code}")

• 200 OK: The request was successful.
• 201 Created: The request successfully created a new resource.
• 204 No Content: The request was successful, but there’s no content to return.
• 400 Bad Request: The request was invalid or otherwise impossible to process.
• 401 Unauthorized: To access the resource, a username/password, a token, or similar credentials is necessary.
• 403 Forbidden: You're authenticated but lack permission to access the resource.
• 404 Not Found: The requested resource doesn't exist or is no longer available.
• 500 Internal Server Error: The server encountered an error processing the request.

Checking the status code allows you to evaluate whether your HTTP request was successful. In case of an error, the status code hints at what went wrong.

Using urllib for Requests in Python

In Python, HTTP requests can also use the built-in library urllib.

import urllib.request

url = '<https://example.com>'
response = urllib.request.urlopen(url)
html = response.read()

print(html)

In this example, urllib.request.urlopen() sends a GET request to the URL while response.read() reads the response's content.

While urllib can be enough for basic tasks, the requests library is much more straightforward to use. However, you might still encounter urllib in legacy code or lightweight projects that want to avoid external dependencies.