New features for Django Packages

Since the Django Dash ended, the Django Packages team has been working to add new features and close out bugs.

"I Use This" added

Since we only want hard metrics on this site, we incorporated an "I Use This" button on the packages. This is so you can identify which packages you use. Please don't press this button for packages that you like, only the ones that are part of your coding efforts.

Added BitBucket support

We are still working out some of the kinks for coming up with stats from BitBucket. Most of the data we collect is fetched via the API, but a little is scraped off individual project pages.

Cache the commits

Originally the commit history was  fetched live. But Github only provides the last 35 commits and BitBucket limits you to the last 50 commits. So now we store the commit history and update it nightly. Which means that the sooner you post your packages the better your commit history will look on Django Packages.

Rebuilt the package updater

Limitations on how many API calls you can make against Github (60 a minute) meant that we had to write some fun code to get around that problem. I think the problem is solved now, but I'm worried I might get to eat my words.

Added a help section

As much as we wanted a completely intuitive site, this will hopefully make it easier for people to figure out how to participate on the site.

Package Add/Edit form refactor

We completely rebuilt the Package add/edit form to make it easier to add packages. So far the response has been entirely positive.

Page cleanup and CSS Reset

We've been slowly cleaning up the HTML and resetting the CSS. Everything is looking prettier. Our goal is to make things more readable, so a lot of the changes are subtle.

Email verification works

It works, and now you get an email to confirm your account.

Django Dash Lessons Learned

Our experience with Django Dash 2010 was that it was an wonderful exercise in classic Django development, cowboy/cowgirl coding, and drinking copious amounts of caffeinated beverages. The result, Django Packages, is something we are happy with, are continuing to improve, and hope will improve the community.

Lesson: Fixtures are a must

Django gives you this amazing Admin control panel. As soon as you get your models in place and are entering test data, start creating fixtures. For the dash we named them initial_data.json and loaded them into the individual app directories. This meant that every time we blew away the database we got a reload with records in place. Sometimes this means you have to hand-edit JSON (or YAML if you swing that way), but the alternative is to waste time re-entering the same data again and again. Don't forget to change the names of your test fixtures before you launch!

From the command-line how to save a fixture pydanny-style:

./manage.py dumpdata package > apps/packages/fixtures/initial_data.json

One nice thing about fixtures is that when you do have the time/need, you can use them to help you write tests. And it makes development easier for contributors.

Lesson: Research ahead of time

In the days before the contest, we researched to see if our target repos (Github, ;BitBucket, and Google Project Hosting) each had an API and a python library to speak to that API. Github has both an API and python library, Bitbucket has an API but no library. And as far as we can tell, Google Project Hosting lacks both API and library (someone please tell me I'm wrong about Google Project Hosting lacking an API).

This meant that when we commenced coding we knew which code base to work with - we weren't trying to look up this or that random package.

We did the same thing for rendering charts.

Lesson: Get it working then optimize

Looking at some of the code makes us wince a bit. But we got it working. Now we can go back and do some code cleanup, maybe use an XML parser instead of regex to try to scrape content from PyPI, and generally feel better about ourselves.

Lesson: Plan out system architecture in advance

In retrospect it was really amusing, but the night of launch the site was serving via the Django runserver command. We were so dead tired and neither of us are crack system administrators that we did what we had to do to score the contest launch point. The next day Audrey got the site running underApache, and next week we'll be giving someone else system access to increase reliability. Next year for the contest we'll probably use something like this or get continuous integration running in the first hour.

Lesson: Don't be afraid to chat with others after the contest starts

Share your ideas, selected packages and frameworks with your competitors. The break from coding helps clear the mind and they might counter with a better idea/package/framework you can use.

Announcing Django Packages!

I'm part of a two person team that just launched that BETA site for http://djangopackages.com, a site that is designed to list all the Django Applications, Frameworks, and Packages created by the Django community. While there are already a few places to look for these things, it is quite easy to argue that they are challenging to navigate, don't give any hard metrics, or are woefully incomplete/unstable/closed. Our goal was to provide an attractive, easy-to-navigate, easy-to-add-data, stable site and share it with the world.

Also, this was our entry into Django Dash 2010, and was the culmination of a few days of brainstorming over paper, a lot of research, and two days of feverish coding/designing. The project was feature complete to our specifications at 5pm the second day, and the rest of the time was spent adding in UI tweaks, usability enhancements, and trying to deploy our creation.

Since then, we've cleaned up a the UI, improved the design, and got the site stable. The code is open source and on github, so fork and contribute!

Design Consideration: No 'Like' button or 'Rate my app' rating systems

We wanted hard metrics. So the package numbers are pulled from the repo sites such as Github, Bitbucket, and Google Code. Otherwise things get weighted funny. Sure, this system can be monkeyed with, but its a good metric for now. We've had suggestions from Django core developers of coming up with a quality check system, things like pypants and/or a formalized approval system.

Design Consideration: Grids

Early on we wanted to duplicate and improve upon the Django CMS Comparison page. There is also a version for Forums, but it would be nice to have a current one for blogs! In addition, recently I heard that 'tag clouds are the mullets of web 2.0'. This really struck a chord in my soul. Since we had metrics on packages, why not compare those metrics, and use those comparisons, which we call 'grids', instead of tags? In fact, we extended our idea and instead of traditional tabs we use grids in the top navigation area as seen below:

Design Consideration: Categories

The site groups packages into three categories, 'Apps' which are individual django applications. 'Frameworks' which are aggregates of apps and python modules. And 'Projects' which are implementations of Apps and Frameworks. We've thought about adding 'Tools' but weren't sure if there was anything that fit that concept, and we are leery about allowing regular Python efforts into the fold.

Design Consideration: Regex vs XML

Slurping data out of Github was easy, especially since I used python-github2. Bitbucket has a RESTful API as well that serves out JSON. I think Google Code does as well. PyPI does not and DOAP on PyPI seems to give little that is useful, so I was forced to do screen scrapes of version numbers and downloads. I'm much faster with Regex and string methods than XML juggling, and speed was of the issue this weekend. I'm not sure what benefit there is to redoing it in HTML5lib or lxml, since what I have works and appears to be stable.

Design Consideration: Leave caching and optimization for later

Besides a tiny bit of memory based template caching on the home page, there is/was no optimization. In time I plan to cache many things using a proper key/value store like redis or memcached. Perhaps not before more design and usability work is done.

Why scared of rabbits?

You wouldn't understand unless you live on the Kansas prairie.

Note: if you have any suggestions, issues, problems with Django Packages please use our issue tracker!

More reasons to go to DjangoCon!

I was thinking some more about DjangoCon next month.

Capoeira - I'm known for doing one-handed cartwheels. But at last year's DjangoCon I got a chance to try my hand at Capoeira. Since then I've managed to get in some actual regular Capoeira training. My hope is to tag up with Oregon Capoeira and see how my Angola matches their Regional. Even if you've never done a cartwheel, it should still be a blast to try out!

Okay, now on to listing some more talks I'm looking forward to having at the conference:

• Why Django sucks and how we can fix it - While Django is the king of Python web framewrks, Eric Florenzano is going to slam it hard. But he isn't going to just troll the conference, he's also going to provide us some possible solutions. The best thing of all, is that since Django is open source, we can all contribute to make it better!
• Pony Pwning - Django is a nicely secure framework thanks to the security focus of the community, but Adam Baldwin shows us how as developers we can make compromising mistakes. If you want to avoid being caught with your pants down on a day where you didn't wear clean underwear, go to this talk!
• State of Pinax - Pinax is a platform for rapidly building websites in Django that I've made contributions to off-and-on since about December 2008. Heck, my talks and tutorial at DjangoCon and Pycon were about Pinax! Anyway, Brian Rosner, arguably the second tallest man in the Pinax community is going to tell us the past, present, and future of this incredible tool set.
• From Slice to Site: Django in 30 minutes - Tired of hearing about how its easier to deploy a <insert-competing-but-sucky-language> site? Katie Cunningham of NASA SMD fame is going to show you how you can do it fast and easy. A great beginner talk from someone skilled at speaking and educating!
• How to sell Django - So your trying to convince the customer/boss that Django is the way to go. But they read somewhere that XML + Shell scripts is how to build an application used by millions or talk to a well-dressed salesperson selling antiquated/broken technology. How do you get past this issue? Well, I'm hosting a panel on how to convince the people who sign the checks that Django is the way to go. I've got Steve Holden, James Tauber, and either Jacob Kaplan-Moss or Frank Wiles, plus another person to discuss this on a panel.
• Large Problems in Django, Mostly Solved - Unfortunately scheduled at the same time as my own talk, my good friend Eric Holscher is giving a talk on some of the best applications written for Django. These are tools that solve a lot of problems for you. Ever wonder what is the best REST application? How to do proper searches? Migrate data? Also, want to make your pet application endorsed by the Django ecosphere? Then attend this talk! 

Getting excited about DjangoCon US!

Djangocon starts in just a month. I'm looking forward to this event because of so many reasons. Lets go over some of them!

Friends - I'll get to meet with old friends and make new ones. Rather than list names I'm going to mark a sheet of paper with the alphabet and check off letters as I meet/make a friend with the first name that matches an unmarked letter. Which makes me wonder if their is a Django app for that in the making...

Portland - Great food and awesome beer at cheap prices. The wonderful thing about Portland is that the base ingredients are really good. I found I liked the simpler/cheaper things there more than fancy foods. The food carts alone are worth visiting the city.

Oregon State - It is a beautiful state and since words can't do justice here is an image:

Me at Mutnomah falls!

 The conference schedule rocks! - The talks they lined up all look really good. They range from the basics to the advanced, and include things that go beyond the technical. Some of my favorite picks from just the first day:

• Typewar: A Case Study - James Tauber is an incredible speaker, not to mention the founder of Pinax and the CEO of Eldarion. His talks are best served live. Typewar is an interesting game but the story of its inception and maintenance is a gem for anyone looking to see how the masters do it.
• So you want to be a core developer? - Russel Keith-Magee, a core developer, lets us know what we need to know and do to work on Django core. He got approved for three talks so you know he has to kick butt as a speaker.
• Maps of Imaginary Lands - Malcom Tredennick, another core developer, shows how to use django.contrib.gis to set up fancy maps of non-existant places. One day I'll have to do this for a place I created in my youth, a place called 'Dannyland'. Anyway, Malcom also has three talks so he's got to have a lot of cool points.
• Django Forms: Tips, tricks, and ways to stay out of trouble - Shawn Rider of PBS fame might not be a core developer but he's another brilliant coder and great speaker. This is a great beginner talk and maybe a decent advanced talk too.

Thats just four great talks described. And its just the tip of the iceberg! This is going to be great!

Edit: Fixed a geography mistake. I am so embarrassed.

Rainbows on the prairie

What is life like on the prairie? Well, under the right conditions it is like this:



About two months ago I rented a nice house a few miles outside Lawrence, Kansas. My original goal was to bike into town a few days every week. However, I'm leery of biking alone down rolling country roads so I bought myself a cheap little car. It gets me to and from Lawrence.

My fun new activity is Capoeira. It keeps me in shape and is something new and exciting for me to do. In order to support the Capoeira, I started up a local meetup. Yeah, at the extreme end of things you see some crazy acrobatics but for most people its the chance to have fun and get in shape. I should post some video online of my antics at Capoeira.

I'm also doing a bit of cooking. The kitchen in our house is really nice and I'm practicing my old culinary skills. My hope is to do a cook-off or a food camp with local friends at some point. Or maybe just relax and just cook for fun.

I tried a ballroom dance class in the area but was not happy with the quality of instruction. Also, that each class we learned a whole new dance form meant that you never really got comfortable with a single method. That said, I did learn some nice steps and hope to do them at a conference and found I really enjoy salsa.

The food here is generally cheap and good. Especially when it comes to core produce and meats. Getting some of the edgier ingredients means paying coastal prices or more. Fish and seafood seems to be hit or miss. I've heard the barbecue is outstanding but haven't been to any of the really notable places.

Life is a lot cheaper here. The dollar stretches amazingly far. My monthly rent on a really big house is a third of the cost of my mortgage or much less than what I was paying for a room in Arlington, Virginia. That means I do a few more of the things I really enjoy and also be able to save at a good rate.

Getting piston forms to play nicely with JSON

Critical Update 2012/05/10!!!
Critical Update 2012/05/10!!!
Critical Update 2012/05/10!!!

Except for a critical security patch, django-piston has been unsupported for about 3 years. That is an eternity, and the number of forks to address multiple issues is cause for alarm. Also, the original author has left the project. Because of that, in it's place at this time I recommend django-tastypie. It is up-to-date, has very good documentation, supports OAUTH, and scored second place in the Django Packages thunderdome (it got nearly 3x as many points!). Another tool to consider is Django Rest Framework, which is as good as django-tastypie but lacks the OAUTH support.

Back to the existing blog post...

A commonly used tool by Djangonauts is django-piston, which is designed to make building a REST API easier. It even works with Django forms to provide easily written PUT/POST validation, which should be pretty darn nice. Unfortunately, if you go with django-piston forms validation it doesn't accomodate the JSON (or XML or YAML) requests and if validation fails it responds in HTML. Even more unfortunate, making validation accept and return JSON with PUT/POST requests is not documented.

While one could argue that it is documented in the django-piston docstrings, in my opinion that is not sufficient.

Fortunately while working on a project for Revolution Systems we worked out a solution:

"""
myapi/resource.py

    author: Daniel Greenfeld
    license: BSD

This assumes your API accepts JSON only.
"""

import json

from piston.decorator import decorator
from piston.resource import Resource
from piston.utils import rc, FormValidationError

def validate(v_form, operation='POST'):
    """ This fetches the submitted data for the form 
        from request.data because we always expect JSON data
        It is otherwise a copy of piston.util.validate.
    """
        
    @decorator
    def wrap(f, self, request, *a, **kwa):
        
        # Assume that the JSON response is in request.data
        # Probably want to do a getattr(request, data, None)
        #   and raise an exception if data is not found
        form = v_form(request.data)

        if form.is_valid():
            setattr(request, 'form', form)
            return f(self, request, *a, **kwa)
        else:
            raise FormValidationError(form)
    return wrap

class Resource(Resource):
    
    def form_validation_response(self, e):
        """
        Turns the error object into a serializable construct.
        All credit for this method goes to Jacob Kaplan-Moss
        """
        
        # Create a 400 status_code response
        resp = rc.BAD_REQUEST
        
        # Serialize the error.form.errors object
        json_errors = json.dumps(
            dict(
                (k, map(unicode, v))
                for (k,v) in e.form.errors.iteritems()
            )
        )
        resp.write(json_errors)
        return resp

Usage in handlers.py:

from django import forms

from piston.handler import BaseHandler

from myapp.models import Article

# We use our custom validate rather than piston's default
from myapi.resource import validate

class ArticleForm(forms.Form):
    """ This is best stored in forms.py but we put 
        here for sake of clarity"""

    author      = forms.CharField(required=True)
    title       = forms.CharField(required=True)
    content     = forms.CharField(required=True)

class ArticleHandler(BaseHandler):

    allowed_methods = ('GET', 'POST', 'PUT', 'DELETE', )
    model = Article
                    
    @validate(ArticleForm)
    def create(self, request):
        # Create/POST code goes here. 

    @validate(ArticleForm)
    def update(self, request, id):
        # Update/PUT code goes here. 

Usage in urls.py:

from django.conf.urls.defaults import *

from piston.authentication import HttpBasicAuthentication as auth

# Import our ArticleHandler
from myapi.handlers import ArticleHandler
# Use our custom Resource class instead of piston's default
from myapi.resource import Resource 

article_handler = Resource(ArticleHandler, authentication=auth)

urlpatterns = patterns('',
   url(
        r'^articles/(?P(\d+))$', 
        article_handler,
        { 'emitter_format': 'json' },
        name='api_article'
       ),   
)

Of course, this assumes you are mapping Create/Read/Update/Delete (CRUD) actions to your API.

I'm interested to see other solutions people have used to handle this in django-piston, and what suggestions people have that could improve on the examples I'm supplying here.