Redux: Python Things I Don't Like

Back in May of 2009, I wrote about Eight Things I don't like about Python. It was my attempt to come up with things I don't like about my programming language of choice. Consider this my update of that post.

1. Division sucks in Python

In Python 3 this is fixed so that 2 / 3 = 0.6666666666666666 but in Python 2.7.x you have 2 / 3 = 0. You can fix that in Python 2.7.x with doing a from __future__ import division before your division call. Can anyone tell me if a version of 2.7.x will natively support 2 / 3 = 0.6666666666666666 without that import?

Note: Chris Neugebauer pointed out that changing division in Python 2.7.x will break backwards compatibility. However that doesn't change that I don't like it in Python 2.7.x.

2. TKinter blows

Honestly, it doesn't really matter to me anymore. I either use command-line scripts or things delivered to the web. Also, thanks to Brett Cannon, I know if I need to make TKinter look good, I can use TK Themed Widgets right out of the standard library.

3. Lambdas make it easy to obfuscate code

I'm known for not liking lambdas in Python. These days, I do know of use cases for Lambdas, but those are far and few between. I might even try to turn that into a blog post this month - use cases for Lambdas in Python. Fortunately for me, these days I seem to work with people who mostly agree with me on this subject.

4. Sorting objects by attributes is annoying

This is still annoying for me. As I said, "... the snippet of code is trivial. Still, couldn't sorting objects by attributes or dictionaries by elements be made a bit easier? sort and sorted should have this built right in. I still have to look this up each and every time."

I've thought of proposing something easier as a PEP. Imagine that! Me submitting a PEP!

5. Regex should be a built-in function

Before I got to do Python full-time I was a go-to person with regular expressions. Languages without them were weak in my opinion. Since then (2006-ish) my skills have faded somewhat in regards to regular expressions. And you know what? It hasn't been a problem. Python's string functions are fast and useful, and when I really need regular expressions, I import the library and do some research. I'm considering this one closed.

6. Reload could be less annoying

Reload only works on modules. I want to be able to something like reload(my_module), reload(my_class), reload(my_function), or even reload(my_variable):

>>> from my_module import MyClass, my_function, my_variable
>>> mc = MyClass(my_variable)
>>> mc 
5
# I go change something in my_module.MyClass and save the file
>>> reload(MyClass) # reload just MyClass
>>> mc = MyClass(my_variable)
>>> mc 
10

My current fix is to use unittest as my shell as much as possible. And that is probably a good thing.

7. Help doesn't let me skip over the '__' methods

As I said way back when, "Python's introspection and documentation features makes me happy. And yet when I have to scroll past __and__, __or__, and __barf__ each time I type help(myobject), I get just a tiny bit cranky. I want help to accept an optional boolean that defaults to True. If you set it to False you skip anything with double underscores.

The See project is one solution to the issue. A different approach I've used is the Sphinx autodoc feature, but Sphinx is a lot of work and doesn't cover every contigency.

8. Not enough female Pythonistas

These days I know a lot of female Python developers. There is my own fiancee, Audrey Roy. Face-to-face I've met and talked to Christine Cheung, Jackie Kazil, Leah Culver, Katharine Jarmul, Katie Cunningham, Barbara Shaurette, Esther Nam, Sandy Strong, Sophia Viklund, Jessica Stanton Aurynn Shaw, Brenda Wallace, Jen Zajac, and many more I know I'm missing. And there are even more with whom I've had in-depth online conversations.

So why didn't I put a strike-through on this one? Because the numbers still aren't good enough. I know a lot of female Pythonistas, but how many do you know? And even if you know a decent number, what percentage of a meetup group you attend are women?

I can say that things are improving, but they could be better - for women or minorities. Find ways to pitch in, be it PyLadies events, PyStar workshops, or what have you.

One last note on this subject, I've heard some unsubstantiated statements that the .Net world has a higher female-to-male ration then the Open Source world. Are we going to take that kind of thing sitting down?

Loving the bunch class

Warning

This is me playing around with things in Python. It's not anything I use in real projects (except maybe the odd test). Please don't use these in anything important or you'll regret it.

Every play with a bunch class? I love 'em and make them protected or unprotected. I started using them early in my Python career, although it wasn't nearly about 2 years ago that I learned what they were called and the best way to code them. In any case, here is a simple, unprotected Bunch class.

# Simple unprotected Python bunch class
class Bunch(object):

    def __init__(self, **kwargs):
        self.__dict__.update(kwargs)

bunch = Bunch(name='Loving the bunch class')
print(bunch.name)

You can also make protected ones, that don't let pesky developers like me overwrite attributes, methods, and properties by accident:

# Simple protected Python bunch class
class ProtectedBunch(object):
    """ Use this when you don't want to overwrite existing methods and data """

    def __init__(self, **kwargs):
        for k, v in kwargs.items():
            if k not in self.__dict__:
                self.__dict__[k] = v

You can also write them to raise errors when a key is in self.__dict__. Or perhaps merely publish a warning. There are many ways to customize, but generally you want to keep these things as simple as possible. Anyway, let's get back to the main topic...

In the early days of my experiences with Python I found a small, nagging issue with dictionaries and objects. The notation wasn't as handy as what you got with JavaScript and some other languages I was using at the time. For example:

// JavaScript object notation
o = {};
o.name = 'Loving the bunch class';
o.name; // Calling with 'dot' notation
o['name']; // Calling with 'bracket' notation  

Unfortunately, in Python you can't do this with a normal bunch class:

# Python bunch class failing on bracket notation
class Bunch(object):

    def __init__(self, **kwargs):
        self.__dict__.update(kwargs)

bunch = Bunch(name='Loving the bunch class')
print(bunch.name)
print(bunch['name'])
Traceback (most recent call last):
  File "", line 1, in 
TypeError: 'Bunch' object is not subscriptable

The quick answer is found with a little trick I found in the comments of a recipie by Alex Martelli that gives you the ability to do:

# Fancy dictionary/object trick
class Buncher(dict):
    """ Warning: DON'T USE THIS IN REAL PROJECTS """

    def __init__(self,**kw):
        dict.__init__(self,kw)
        self.__dict__.update(kw)

bunch = Bunch(name='Loving the bunch class')
print(bunch.name)
print(bunch['name'])

I'm not the only one who likes Bunch classes. On PyPI I found a really complete implementation.

Of course, in a lot of cases you probably don't want this 'weight of code', right? Dictionaries being lighter than full objects and all that. Nevertheless, it's fun for noodling and playing around with code. Still, I'm thinking it might be a fun little project to take a group of bunch implementations and do performance checks on them versus each other and dictionaries. Maybe the 'performance hit' isn't so bad.

I should also dig into things like defaultdict and other constructs to learn more. Part of the fun of any programming language is the depth of even the 'simplest' components of the language.

PyCodeConf 2011 Report

As my fiancee said, "PyCodeConf is a new kind of Python conference with a radically different format. Speakers are invited to speak about whatever they desire relating to the theme ("The Future of Python"), in front of a room of round tables. In between talks there are long breaks to encourage discussion. As a result, talks are edgier, and you really get to know people and possibly shape the future together."

This summer (2011) I was invited by Chris Wanstrath to be an invited speaker at the first ever PyCodeConf, to be held in Miami, Florida. I've been using Git and Github since early 2009, and more importantly, I've known Chris since DjangoCon 2009. I've always appreciated his interest in not just providing tools for the community, but also his efforts across languages and platforms to improve the lives of developers and those who support developers. And that he and his partners seem to make a pretty penny at it and share what they make (drink-ups and now conferences) is only a good thing in my opinion.

So I accepted Chris' invitation. :-)

The theme for the conference was the future of Python, and I submitted a talk proposal about Collaboration. Chris helped me figure my talk topic, which was awfully nice of him. Shortly afterwards my fiancee, Audrey Roy, received her own invitation to speak.

Between the time that Chris invited me to speak and the start of conference, Chris Williams of JS Conf got involved.

Alright, let's review things...

Accommodations

Everyone stayed in the Epic Hotel, in downtown Miami. The rooms fit the name of the hotel, being Epic in size and having amazing stuff in them. The rooms had free/good internet if you signed up for a hotel mailing list. Since we rehearsed and polished talks the night before the conference we ordered room service and were pleased with what we ate. The hotel had a heated outdoor pool on the 14th floor, which I'll get into later. In any case, the only time I've been in a comparable hotel was the incredible arrangements provided by the PyCon New Zealand folks who put us up in the Museum Hotel for a few days.

Sure, $189/night is high, but 3 nights when you split it with 2 or more people makes it not so bad.

Result: Superb

Conference Meals

If you are serving me food, you can't go wrong with salmon, steak, good cheese, fresh vegetables, coffee, and juice. I can report that PyCodeConf did quite well in this regard.

Outside of the conference I really enjoyed the food. Andiamo was a crazy good pizza place. I also got some really nice grouper which you can only seem to get in Florida.

Result: Superb

The Conference Room

The conference took place on the 14th floor of the Epic Hotel. This was a single track conference, with all the talks were given in the same room. The room was large, but everyone had a comfortable seat at large round tables. That was a nice touch, because it encouraged you to socialize with everyone nearby.

That worked out for the most part, except for a couple of developers sitting at a table who had backs turned to the speakers and were talking loudly while pair programming or something. I asked them to quiet down but 30 seconds later they were back at it. We moved away, but in retrospect considering their rudeness, I should have politely asked them to take it outside.

Anyway, the acoustics were good, the temperature comfortable, and the seats comfortable.

Result: Superb

Speakers

Speaker selection was done magnificently well. There wasn't a dud within the lot of speakers. I normally expect that in any conference you'll get at least one dud talk per day, and pycodeconf didn't have that problem (unless my talk sucked).

Jesse Noller opened things up with a great encouraging talk, Raymond Hettinger gave a talk on Python basics that was so full of nuance that I'm terrified of attending an advanced talk by him, Alex Gaynor filled us with hope for PyPy, Tracy Osborn taught us how to bootstrap entrepreneur projects, Travis Oliphant wants Python core and PyPy to collaborate more with the Scientific Community, Audrey Roy gave up some of her community building secrets, David Beazly explained the issues of the GIL in terms mere mortals such as myself can understand, Gary Bernhardt gave an amazing talk comparing Python and Ruby, and Leah Culver made Django + backbone.js look easy, but if you talk to her you know whatever she does is sophisticated and not for beginners. Dustin Sallings sold me on a neat idea for testing to help catch edge cases and Armin Ronacher opened my eyes on WSGI.

The wonderful thing about these talks is that since everyone knew the upcoming tracks, or had seen the previous ones, we could relate to each other. So David Beazly, Travis Oliphant, and Avery Pennarun raised interesting concerns about PyPy that everyone got the chance to hear about. They weren't show-stopping issues, just raising awareness about things that Alex didn't cover in his talk.

I live-noted the event as much as I could, with notable gaps in Leah's talk (she talks fast and is very technical and wanted to give her my full attention) as well as Armin (his talk shocked me a bit - I'm still a WSGI newbie). You can see my efforts at my PyCodeConf live-notes.

Speaking of live-noting, Josh Bohde also live-noted the event and captured a ton of stuff I missed.

The gaps between talks were also a nice 15 minutes. That meant you could stretch your legs, get a drink, and talk to people.

Result: Superb

Parties

We (me and Audrey) missed the first party (hosted by New Relic at a place called DRB) on account of preparing for our talks. We always do our absolute best on talks, and both like to practice a lot. Also, Mark Pilgrim's disappearance had touched me and I wanted to talk about it. We heard it was a great party, so we'll assume that it was. :)

The next evening the party (hosted by Heroku) was on the 14th floor, which meant it was a pool party! There was great food, good drink, and a latin jazz band playing. One of the pools was heated, so most people stayed in there, and drank many watermelon mojitos served by the staff. The pool was a huge hit because it was comfortable and people just talked freely. No laptops, no phones, just talking. Chris Williams served us drinks himself, Chris Wanstrath got wet, and everyone just relaxed. I have to say, a heated pool party is something EVERY conference should give. MOAR POOL PARTEEZ PLEEZE!

The final night's party was at the News Lounge and was hosted by Github and Droptype. The drinks and people were awesome, and I have good memories of being in a circle listening to Chris Williams and Audrey Roy talk. I did go beyond tipsy, overdid the Capoeira, and tried to convince Chris Wanstrath to give up the whole DVCS hosting thing to do wedding planning. There was also a bunch of us getting kicked out of a Karaoke bar because of the antics of a Python core developer. Ha ha ha. It was a crazy night that took me two days to recover from.

I'm glossing some important discussions that happened while I was still sober at these two parties, and maybe in the future if things play out right I'll go over them.

Result: Superb

People

Part of attending conferences is to meet old friends and make new ones. I got to spend time with Mark Ramm, Jesse Noller, Nick Coghlan, Alex Gaynor, Ben Firshman, Armin Ronacher, Raymond Hettinger, Rachel Hettinger, Chris Wanstrath, and many other excellent people. I also got the chance to meet and befriend Kenneth Reitz, Chris Williams, David Cramer, Wayne Witzel, and Leah Culver. I'm missing at least a dozen more. It was great to put faces to people, and in some cases, hear their side of a story.

Also, I got to gush at programming heroes like David Beazly and Travis Oliphant like a total fanboy.

Believe it or not, I got a bit shy. I'm kicking myself over not introducing myself to more people. Next time!

Result: Superb

Summary

The conference was amazing. Like all conferences it had its own character and fun. Because it was a purely commercial conference I was a bit worried going into it, since I've heard about the corporate feel of these things. Those fears were completely mitigated by the open attitude and decentness of the conference organizers and sponsors. I look forward to attending PyCodeConf again in the future.

If you've got good senior technical staff and you want them to benefit from a conference, this is a good place to send them.

Overall Result: Superb

Conference Talks I want to see

I'm writing this the day after Github's pycodeconf ended. That was an amazing conference, and I'll be blogging it soon (I'll also be writing about PyCon Australia, PyCon New Zealand, and DjangoCon US). With all this conference experience very current in my head, things I've seen and done at them, and the deadline for PyCon US submissions coming up, here are some talks I really want to see happen in the next six months. If not at PyCon US, then please consider these for other forthcoming events!

Note: Couldn't do my preferred 'linkify' as well as I liked thanks to bad hotel internet. I'll clean it up later.

Advanced SQL Alchemy Usage

I think the uber-powerful SQL Alchemy ORM needs the same sort of treatment me and Miguel Araujo gave on Advanced Django Forms Usage. Not a 30 tutorial or overview or 'State of', but tricks and patterns by someone who has used it frequently on more than one project. Multiple projects is important because the speaker should have had the chance to try multiple approaches. Start with something simple like a TimeStampModel all model classes might inherit from, then go into deeper and and more complex technical detail. Finish the talk with something crazy hard from SQL Alchemy that is hard to explain. If that causes you to open a bug/documentation ticket, then you'll know that you've done the talk right.

Advanced Django Models Usage

Following the same pattern as my SQL Alchemy idea above, start with something simple like a TimeStampModel (including South migration of fields), then go into complex looks with Q objects, good patterns for Managers, Aggregation, Transactions, and then finish it with the craziest, hardest thing you can find. When putting together the closing material causes you to open tickets for broken core code/documentation, then you know you've done it right.

Python Code Obfuscation Contest

This certain-to-be-controversial talk idea would be where the speaker would solicit Pythonistas to submit a single arcane Python code module that would have to display the text of "Although that way may not be obvious at first unless you're Dutch." There would be a 'Expert' category which would forbid the eval/exec functions. The "Anything Goes Category" would allow use of eval/exec. The conference talk would be where the speaker announces the winners and comments on the brilliant insanity of submissions.

Django + Flask + Pyramid: A demonstration of useful things you can do with WSGI

At pycodeconf Armin Ronacher showed how with WSGI, he can run Django, Flask, Pyramid all from same server from the same domain. This surprised a lot of people, including me, and I want to see more of what Armin was talking about. I don't want any theory. I don't want anything obscure. I just want meaty bits I can implement the day after I hear the talk.

Zen of Python

Richard Jones gave his version of the talk at PyCon AU, and I want to hear other opinions about it. I'm happy to hear an expert give his view, and I would also be delighted to hear how a beginner (or relative beginner) feels about it.

Websites and OO Design Concepts: A Tutorial

For beginners, I would love to see a talk on a list of OO theories, and as each list item was discussed, examples designed in the context of a web site, how to do things right, plus identified anti-patterns would be presented. The web angle would be a good way to get the incoming Python web crowd to attend and identify with raised issues.

Profiles: Breaking Normalization

In the summer of 2010 I either saw this pattern or cooked it up myself. It is specific to the Django profiles system and helps me get around some of the limitations/features of django.contrib.auth. I like to do it on my own projects because it makes so many things (like performance) so much simpler. The idea is to replicate some of the fields and methods on the django.contrib.auth.model.User model in your user profile(s) objects. I tend to do this usually on the email , first_name , last_name fields and the get_full_name method. Sometimes I also do it on the username field, but then I also ensure that the username duplication is un-editable in any context.

Sure, this breaks normalization, but the scale of this break is tiny. Duplicating four fields each with a max of 30 characters for a total of 120 characters per record is nothing in terms of data when you compare to avoiding the mess of doing lots of profile-to-user joins on very large data sets.

One more thing, I've found that most users don't care about or for the division between their accounts and profiles. They are more than happy with a single form, and if they aren't, well you can still use this profile model to build both account and profile forms.

Alright, enough talking, let me show you how my Profile models tend to look:

from django.contrib.auth.models import User
from django.db import models
from django.utils.translation import ugettext_lazy as _

class Profile(models.Model):
    """ Normalization breaking profile model authored by Daniel Greenfeld """
    
    user = models.OneToOneField(User)
    email = models.EmailField(_("Email"), help_text=_("Never given out!"), max_length=30)
    first_name = models.CharField(_("First Name"), max_length=30)
    last_name = models.CharField(_("Last Name"), max_length=30)

    # username field notes:
    #     used to improve speed, not editable! 
    #     Never changed after original auth.User and profiles.Profile creation!
    username = models.CharField(_("User Name"), editable=False) 

    def save(self, **kwargs):
        """ Override save to always populate changes to auth.user model """
        user_obj = User.objects.get(username=self.user.username)        
        user_obj.first_name = self.first_name
        user_obj.last_name = self.last_name
        user_obj.email = self.email
        user_obj.is_active = self.is_active        
        user_obj.save()
        super(Profile,self).save(**kwargs)

    def get_full_name(self):
        """ Convenience duplication of the auth.User method """
        return "{0} {1}".format(self.first_name, self.last_name)

    @models.permalink
    def get_absolute_url(self):
        return ("profile_detail", (), {"username": self.username})

    def __unicode__(self):
        return self.username

All of this is good, but you have to be careful with emails. Django doesn't let you duplicate existing emails in the django.contrib.auth.model.User model so we want to catch that early and display an elegant error message. Hence this Profile form:

from django import forms
from django.contrib.auth.models import User
from django.utils.translation import ugettext_lazy as _

from profiles.models import Profile

class ProfileForm(forms.ModelForm):
    """ Email validation form authored by Daniel Greenfeld """
        
    def clean_email(self):
        """ Custom email clean method to make sure the user doesn't use the same email as someone else"""
        email = self.cleaned_data.get("email", "").strip()

        if User.objects.filter(email=email).exclude(username=self.instance.user.username):
            self._errors["email"] = self.error_class(["%s is already in use in the system" % email])
            return ""            

        return email

    class Meta:
        
        fields = (
                    'first_name',
                    'last_name',
                    'email',
                    )
        model = Profile

History of my most used shell commands

I ran this a few years back and I'm running it again today.

What is interesting is that compared to the older history, git has replaced svn, pip has replaced easy_install, and virtualenv has now completely subsumed buildout. Oh, how the mighty have fallen!

$ history | awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -rn |head -n 20

209 git
123 python
34 ls
31 mate
18 cd
14 pwd
9 hg
8 touch
7 rm
6 cp
5 pip
5 mv
5 django-admin.py
4 mkvirtualenv
3 mysql
3 mkdir
3 bash
2 deactivate
2 add2virtualenv
1 workon

Quick conferences report: Presentations

My lovely Fiancée, Audrey Roy, was invited to be the opening keynote speaker at both PyCon Australia on Diversity in Python (video) and PyCon New Zealand on Python on the Web.

As for me, I managed to get talks into both of those conferences AND DjangoCon US. I co-presented on three of them, and I share all credit for success with my cohorts. The talks I gave at the conferences were (I'll post videos when they get up):

Confessions of Joe Developer (PyCon Australia, DjangoCon US)
The genesis of this talk was as a lightning talk at I gave at the Hollywood Hackathon. It is a talk about admitting that us mere mortals need to ask questions, take notes, and follow good practices in general. I gave it again at LA Django this summer, extending it to a full length talk complete with lots of technical content. At PyCon Australia I toned down the technical content because I was nervous, and while the response was positive, it  could have been much better. So for DjangoCon I ramped up the tech-talk and it worked much better. I've now given the talk 4 times, and I'm leaning towards retiring it.

 

Python Worst Practices (PyCon New Zealand)
This talk grew out of a SoCal Piggies lightning talk which I gave for the purpose of humor. Often we as Python developers are smug in the clarity of the language that we don't realize just how easily we can obfuscate code. In fact, I contend that Python is fully capable of a code obfuscation contest. This talk rejects a lot of crazy practices I've either done myself or had to debug from other people's work. For New Zealand I added a ton of content and tested things pretty diligently. The variable naming pages stumped some people I really respect and I was quite happy with that result.

 

Django Packages Thunderdome (co-presented with Audrey Roy, DjangoCon US)
Audrey did most of the work for this presentation. In this talk I helped review a horde of Django Packages across 7 different categories. It was nerve wracking because every part of our talk would get judged - but Audrey kept things really positive and made it clear we were providing constructive criticism. I think she got her message across to most people, and more importantly, it got a lot of people thinking about what ought to be normal community standards. I'll probably blog on those community thoughts and statements later, but I think Audrey (with help from me) accomplished what she aimed to do.

View more presentations from Audrey Roy

Advanced Django Form Usage (co-presented with Miguel Araujo)
Some time ago Miguel befriended me and helped resurrect the django-uni-form project. He graciously agreed to help me present on Django Forms and we decided to make the talk as sophisticated as possible. Previous Django form talks have been good, but focused on the fundamentals and we wanted to do something really different. This talk was hard because Miguel and I were on opposite sides of the planet, so we did a lot of github pull/pushes. In both doing research and presenting Miguel did an unbelievably good job and I hope he does more of this in the future. The response was extremely positive and I'm certain that our plan of getting our notes/work/transcript into Django core is well on it's way.

 

Ultimate Django Tutorial Workshop (DjangoCon US)
I got about 10 professional Django experts in a room, including Django core developers, and had them help me coach nearly 20 people through a modified version of the Django tutorial. Students seemed to learn tons, lots of socializing happened thanks to some happy accidents, and the experts got a chance to really see where the Django tutorial needs work. PyLadies organizer Esther Nam spent her sprint days working on something that ties the slides into the Django Tutorial - and for now I'm holding off on sharing my work until she says her work is done.

Summary
These were amazing opportunities to speak and will hopefully make a difference. I wouldn't have traded all of this for the world. It was a lot of work, and I doubt I'll ever go quite at this pace again. My plan is to do fewer talks and make them much better.