Python HTTP Requests for Humans

Ever try to use Python's standard library for doing a POST? Or a GET, PUT, or DELETE? What about when you have to deal with HTTP Basic Auth?

In a word, ugh.

Let's face it, this is one part of Python that is really not for human consumption. While there are a million things you can do with things like urllib, urllib2, socket, urlparse, the fact of the matter is that implementing anything beyond urllib.urlopen() is a matter of diving into arcane APIs.

Sure, thanks to works like Doug Hellmann's Python Module of the Week and Michael Foord's documentation of urllib2 the problem isn't unsurmountable. Unfortunately, the eclectic mix of libraries and weird APIs means when you have to revisit your code in a few months your code feels like spaghetti.

Do you doubt me?

# This sample gleefully taken from https://gist.github.com/973705

import urllib2

gh_url = 'https://api.github.com'
gh_user= 'user'
gh_pass = 'pass'

req = urllib2.Request(gh_url)

password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_manager.add_password(None, gh_url, gh_user, gh_pass)

auth_manager = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_manager)

urllib2.install_opener(opener)

handler = urllib2.urlopen(req)

print handler.getcode()
print handler.headers.getheader('content-type')

# ------
# 200
# 'application/json'

Really?

This much code to make a simple HTTP GET request with some auth?!?

Really?!?

This is a debugging nightmare! Especially when you have to deal with complex service APIs provided by Paypal, Amazon, Google, Authorize.net, and a million other systems.

I bet I could earn a decent living by charging Pythonistas a buck each time they took the shortcut of doing HTTP actions in the shell via curl or wget.

Anyway, wouldn't it be great if we could just call a single function with the URL and auth data as parameters? And that the same dialogue would exist for GET, POST, PUT, DELETE or whatever? Wouldn't that be just plain wonderful? If only we could have that functionality in Python!!!

Fortunately for us, we do have that functionality courtesy of Kenneth Reitz's Requests library! Our verbose code sample above becomes the wonderfully terse and easy-to-memorize script as shown below:

# This sample joyfully taken from https://gist.github.com/973705

import requests

r = requests.get('https://api.github.com', auth=('user', 'pass'))

print r.status_code
print r.headers['content-type']

# ------
# 200
# 'application/json'

Want to do a post with data? Try this:

# This example cooked up by me!

import requests
post_data = {"amount":10000, "service":"writing blog posts"}

r = requests.post('http://example.com/api', post_data, auth=('user', 'pass'))

print r.status_code
print r.headers['content-type']

# ------
# 200
# 'application/json'

The Requests library is still young, but I've yet to run into any bugs or undocumented edge cases. The documentation is awesome, but you don't really need it at all. The library is intuitive, fun, and and there is clearly one way to do.