Thursday, May 26, 2011

Python HTTP Requests for Humans

Ever try to use Python's standard library for doing a POST? Or a GETPUT, or DELETE? What about when you have to deal with HTTP Basic Auth?

In a word, ugh.

Let's face it, this is one part of Python that is really not for human consumption. While there are a million things you can do with things like urlliburllib2socketurlparse, the fact of the matter is that implementing anything beyond urllib.urlopen() is a matter of diving into arcane APIs.

Sure, thanks to works like Doug Hellmann's Python Module of the Week and Michael Foord's documentation of urllib2 the problem isn't unsurmountable. Unfortunately, the eclectic mix of libraries and weird APIs means when you have to revisit your code in a few months your code feels like spaghetti.

Do you doubt me?

# This sample gleefully taken from

import urllib2

gh_url = ''
gh_user= 'user'
gh_pass = 'pass'

req = urllib2.Request(gh_url)

password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_manager.add_password(None, gh_url, gh_user, gh_pass)

auth_manager = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_manager)


handler = urllib2.urlopen(req)

print handler.getcode()
print handler.headers.getheader('content-type')

# ------
# 200
# 'application/json'


This much code to make a simple HTTP GET request with some auth?!?


This is a debugging nightmare! Especially when you have to deal with complex service APIs provided by Paypal, Amazon, Google,, and a million other systems.

I bet I could earn a decent living by charging Pythonistas a buck each time they took the shortcut of doing HTTP actions in the shell via curl or wget.

Anyway, wouldn't it be great if we could just call a single function with the URL and auth data as parameters? And that the same dialogue would exist for GET, POST, PUT, DELETE or whatever? Wouldn't that be just plain wonderful? If only we could have that functionality in Python!!!

Fortunately for us, we do have that functionality courtesy of Kenneth Reitz's Requests library! Our verbose code sample above becomes the wonderfully terse and easy-to-memorize script as shown below:

# This sample joyfully taken from

import requests

r = requests.get('', auth=('user', 'pass'))

print r.status_code
print r.headers['content-type']

# ------
# 200
# 'application/json'

Want to do a post with data? Try this:

# This example cooked up by me!

import requests
post_data = {"amount":10000, "service":"writing blog posts"}

r ='', post_data, auth=('user', 'pass'))

print r.status_code
print r.headers['content-type']

# ------
# 200
# 'application/json'

The Requests library is still young, but I've yet to run into any bugs or undocumented edge cases. The documentation is awesome, but you don't really need it at all. The library is intuitive, fun, and and there is clearly one way to do.

Monday, May 23, 2011

I love this girl!

In every person's life there are those incredibly memorable experiences that stick with you forever. The entirety of PyCon 2010 is for me that experience. You see, PyCon 2010 saw me introduced to the lovely, talented, and brilliant Audrey Roy.

Moments after I heard her lovely voice for the first time, we looked into each other's eyes. After that moment, I spent every waking moment of the conference finding excuses to spend time with her. Fortunately for me, most of her tastes for talks and social events matched my own. Within days we were dancing in each other's arms.
2010-02-20 23.21.30

By the end of the PyCon we both knew we had something special. So after the conference we both flew back and forth across the country every two weeks for months to keep seeing each other. My phone bill skyrocketed. I became intimately familiar with Skype. It was crazy and magnificent. Finally, on May 5th of 2010 I moved out West to be with her for good.

Audrey may seem shy at first, but she has a fierce heart (FYI, she played Ice Hockey for years and did the bargaining for my car). She has a degree in Electrical Engineering and Computer Science from MIT. She supports me in everything I want to do. If you enjoy Django Packages and follow Open Comparison, she came up with the idea. We cook and eat healthy, and besides our differences on seafood/mac-and-cheese we are a food match. She loves my family and they adore her. She is a talented visual artist in any medium she attempts. As a developer, she learns unbelievably fast and produces high quality code in languages such as Python, C++, JavaScript, Objective-C, and anything else she touches. She hacks the Linux kernel so she can use her preferred peripherals. Her role in the technical community continually grows, and recently she launched the PyLadies advocacy group.
Airborne girlfriend!

An incredible thing about Audrey is that she makes me a better person. I didn't see this at first, but my good friend and mentor Steve Holden pointed it out. She doesn't just bring me joy, she makes me a more tenacious, honest, and compassionate person.

Life is never perfect, but thanks to Audrey and the Python community that brought us together, life is just plain good.

Thursday, May 19, 2011

A couple great Pycon 2011 Talks

Yes, this should have been posted months ago... Anyway, so many great talks at PyCon 2011, here are two great ones! Being an eternal beginner, I try to hit both entry level and advanced talks.

How to write obfuscated Python
Why waste your time with one decorator when 10 of them attached to a function, each named as descriptively as 'X' or 'G' will make your code delightfully hard to interpret? Also included such goodies as 'object = str' which I borrowed for my own Python Worst Practices. In my opinion, Reverend Johnny Healy gave the funniest and obscure talk of the conference, and yet somehow managed to export some useful knowledge in the process.

The Data Structures of Python
Alex Gaynor gave a great 'beginner' talk here. As he went over the basic types in Python, and while he presented well, I wondered if this talk was for me. My hubris was that I thought the talk was beneath me. Then Alex dove into some more sophisticated material like namedtuples and collections and yet again I was reminded how periodically going over the 'basics' is a good thing. Alex kept his talk to a reasonable pace, giving enough time for people to take notes and truly understand what he was teaching. If the whole crazy programming thing doesn't work out for him, he might want to consider becoming an educator.