Announcing Garbaginator!

While working on Packaginator at the PyCon 2011 sprints we discovered some serious issues in the way that Django handles garbage collection. After a huge amount of work, we managed to isolate and fix the problem. This 'fix', as it were, was only possible by doing a very sophisticated 'hack' of critical internal components of the Django Web Application framework. We also discovered that similar issues occurred in other existing Python application frameworks such as Pyramid, Flask, Web.py, Web2Py, Grok, Twisted, Tornado, Google App Engine, and Rails.

Since then the Packaginator community has been fiercely debating what we should do with our newly created set of hacks. After a lot of arguments going both ways we've decided to come up with our own application framework and release it to the world under the GPL license.

This brand new application framework ignores the lessons learned from all the other Python frameworks and embraces the cutting edge concept of Not-Invented-Here. It focuses less on features and enhancements over existing systems and much, much more on the critical concept of formal Garbage handling.

Some of the critical modules include:

• RubberGloves (for handling dirty objects)
• Django-Garbaginator
• Flask-Recycling
• Pyramid-Garbaginator
• Web.2.py Garbaginator Bridgerator ('cause people always get Web2Py and web.py confused with each other so we bridged them together)
• BlueBream

Check out the home page at: http://garbaginator.cartwheelweb.com

Fork Garbaginator on GitHub: https://github.com/cartwheelweb/garbaginator

Note: This was an April Fool's Day Joke

Stupid Template Languages

For years I've been absolutely certain that I really prefer stupid template languages any time I'm generating HTML. The less the template language can do the better. Since I spend most of my time coding in Python you might assume this applies just to Python, but I think it also applies to anything where you have the power to readily mix HTML generation and code.

The biggest annoyance I have with smart template languages (Mako, Genshi, Jinja2, PHP, Perl, ColdFusion, etc) is that you have the capability to mix core business logic with your end views, hence violating the rules of Model-View-Controller architecture. While the web can be hard to match to MVC, in general you aren't supposed to do that sort of thing. I've made the mistake of putting core logic in the wrong places in the past, but I'm proud to say I've gotten good at avoiding that particular mistake.

I don't work in a vacuum.

I often work on projects crafted by others, some who decided for arcane/brilliant/idiotic reasons to mix the kernel of their applications in template function/macros. This is only possible in Smart Template Languages! If they were using a Stupid Template Language they would have been forced put their kernel code in a Python file where it applies, not in a template that was supposed to just render HTML or XML or plain text.

What it comes down to is that Smart Template Languages designers assume that developers are smart enough to avoid making this mistake. Stupid Template Languages designers assume that developers generally lack the discipline to avoid creating horrific atrocities that because of unnecessary complexity have a bus factor of 1.

So what is a Smart Template Language?

In my own vernacular, template languages that let you write functions/macros are what I call a Smart Template Language. Some of them are brilliantly executed, the example of Jinja2 comes to mind, but invariably I suffer through abuse of its Macro control structure as implemented by others.

Misery Cubed a.k.a. Genius Template Languages

Next comes Genius Template Languages, which take things a step further. These template languages allow you to not only define functions/macros, but also let you embed unrestricted Python (or Java or Ruby or whatever) in the template. This 'feature' lets you code your entire application in the templates!  In the Python world what comes to mind is Mako and Genshi, but I'm sure there are many other tools with this 'capability'.

I like Stupid Template Languages!

Stupid Template Languages don't let you define functions/macros. They don't let you embed Python code. They barely let you define variables and often have simplistic control architectures.

For Django efforts, which is about 70% of my work, I like the Django Template Language (DTL). Since it is used by a huge community, there are a ton of useful apps which have it as a dependency. Switching away from it would mean cutting myself off from a large ecosphere of tools I can use to not reinvent the wheel.

Back in my Zope/Plone days I really, really enjoyed the Template Attribute Language (TAL) because it was stupid too. If I needed an XML generation template language and could import it easily I might consider using it again, or perhaps Chameleon, which is a new, improved version . The downside is that they come paired with another tool paired with it, METAL, which gave it macros. My own experience with METAL is that it was all too easy to do what we developers do with Smart Template Languages.

But DTL and TAL are slow!

So what?

If you want to boost your performance, first try caching. There are a ton of tools you can use, with Varnish being one I keep seeing in action. Read the docs on your favorite web framework's caching engine and apply what you learn. And Djangonauts should read up on Mike Malone as much as possible.

If after all that the site still delivers slow content and it appears to be a template language issue, then identify the bottleneck content and consider alternatives for that one portion. My favorite response is a bit of AJAX. Use your framework to render the content as JSON and have JavaScript parse it into legible content, a task which JQuery makes trivial.

Getting piston forms to play nicely with JSON

Critical Update 2012/05/10!!!
Critical Update 2012/05/10!!!
Critical Update 2012/05/10!!!

Except for a critical security patch, django-piston has been unsupported for about 3 years. That is an eternity, and the number of forks to address multiple issues is cause for alarm. Also, the original author has left the project. Because of that, in it's place at this time I recommend django-tastypie. It is up-to-date, has very good documentation, supports OAUTH, and scored second place in the Django Packages thunderdome (it got nearly 3x as many points!). Another tool to consider is Django Rest Framework, which is as good as django-tastypie but lacks the OAUTH support.

Back to the existing blog post...

A commonly used tool by Djangonauts is django-piston, which is designed to make building a REST API easier. It even works with Django forms to provide easily written PUT/POST validation, which should be pretty darn nice. Unfortunately, if you go with django-piston forms validation it doesn't accomodate the JSON (or XML or YAML) requests and if validation fails it responds in HTML. Even more unfortunate, making validation accept and return JSON with PUT/POST requests is not documented.

While one could argue that it is documented in the django-piston docstrings, in my opinion that is not sufficient.

Fortunately while working on a project for Revolution Systems we worked out a solution:

"""
myapi/resource.py

    author: Daniel Greenfeld
    license: BSD

This assumes your API accepts JSON only.
"""

import json

from piston.decorator import decorator
from piston.resource import Resource
from piston.utils import rc, FormValidationError

def validate(v_form, operation='POST'):
    """ This fetches the submitted data for the form 
        from request.data because we always expect JSON data
        It is otherwise a copy of piston.util.validate.
    """
        
    @decorator
    def wrap(f, self, request, *a, **kwa):
        
        # Assume that the JSON response is in request.data
        # Probably want to do a getattr(request, data, None)
        #   and raise an exception if data is not found
        form = v_form(request.data)

        if form.is_valid():
            setattr(request, 'form', form)
            return f(self, request, *a, **kwa)
        else:
            raise FormValidationError(form)
    return wrap

class Resource(Resource):
    
    def form_validation_response(self, e):
        """
        Turns the error object into a serializable construct.
        All credit for this method goes to Jacob Kaplan-Moss
        """
        
        # Create a 400 status_code response
        resp = rc.BAD_REQUEST
        
        # Serialize the error.form.errors object
        json_errors = json.dumps(
            dict(
                (k, map(unicode, v))
                for (k,v) in e.form.errors.iteritems()
            )
        )
        resp.write(json_errors)
        return resp

Usage in handlers.py:

from django import forms

from piston.handler import BaseHandler

from myapp.models import Article

# We use our custom validate rather than piston's default
from myapi.resource import validate

class ArticleForm(forms.Form):
    """ This is best stored in forms.py but we put 
        here for sake of clarity"""

    author      = forms.CharField(required=True)
    title       = forms.CharField(required=True)
    content     = forms.CharField(required=True)

class ArticleHandler(BaseHandler):

    allowed_methods = ('GET', 'POST', 'PUT', 'DELETE', )
    model = Article
                    
    @validate(ArticleForm)
    def create(self, request):
        # Create/POST code goes here. 

    @validate(ArticleForm)
    def update(self, request, id):
        # Update/PUT code goes here. 

Usage in urls.py:

from django.conf.urls.defaults import *

from piston.authentication import HttpBasicAuthentication as auth

# Import our ArticleHandler
from myapi.handlers import ArticleHandler
# Use our custom Resource class instead of piston's default
from myapi.resource import Resource 

article_handler = Resource(ArticleHandler, authentication=auth)

urlpatterns = patterns('',
   url(
        r'^articles/(?P(\d+))$', 
        article_handler,
        { 'emitter_format': 'json' },
        name='api_article'
       ),   
)

Of course, this assumes you are mapping Create/Read/Update/Delete (CRUD) actions to your API.

I'm interested to see other solutions people have used to handle this in django-piston, and what suggestions people have that could improve on the examples I'm supplying here.

The end of my Feedfeeder story

Another post about Plone... but this time about me and not about Plone.

For about 18 months I have wrestled with consuming broken RSS feeds to pick up image of the day fields stipulated by customers. These are feeds so broken that no RSS parser, including the masterful Feedparser, can handle them (for example, one image of the day feed usually puts the image in the RSS header and changes that each day - no history is maintained). They aren't actually RSS, they just possess a file name that ends with '.rss'. Plus, periodically the way they are written changes so custom logic fails.

I have forked Reinout van Rees FeedFeeder project, and even proposed complicated logical revisions to handle broken these broken feeds and their shifting implementation. I called it Feedfeeder v2. Reinout always seemed hesitant, and I watched as other people extended on his work and despaired. I knew something was wrong but couldn't put my finger on it. I hesitated to work on it, even though funding for it was readily available.

Then between Spacebook, Pinax, and other efforts I shelved this effort for months, hiding my head in the virtual sand. And yet I knew it needs to be addressed. How could I handle something that broke the otherwise wonderful Feedparser?

During Pycon 2009 I came up with the answer. I took an excellent tutorial on html scraping and learned lots of little tricks to reinforce my skills with BeautifulSoup. You see, screen scraping is a secret pleasure I have. Scraping out a bit of data from a page is like a little puzzle. When I talked about this to someone, in the middle of my discussion with them the answer became clear as day.

The answer was to turn the problem from a RSS interpretation problem to a simple web page scraping puzzle.

1. Fetch via urllib the XML file that pretends to be RSS.
2. Parse it using BeautifulSoup or html5lib.
3. Get all the images listed.
4. Discard all but the largest image.
5. Guess out the meta-data from the XML file and store that for the image.

Problem solved.

Now I just need to make a Plone 3 package to do this for me and my angst is finished.

My apologies Reinout for the time spent on trying to cook a solution via Feedfeeder. Thank you for your insights and your extreme patience. I think you tried to tell me to take a different path.

ZPUG-DC Report 2007/11/07

Got a good presentation on repose.zope2 by Chris McConnough, which lets you deploy easily in virtualenv with just a few commands. Paste is used a lot, of course. This repose thing lets you use WSGI, which means you can use middleware of choice. Chris shared evalerrer with us, which promises to be a great middleware for any developer.

I like this sort of development because the easier it is to install independent versions of things with variations quickly and without a lot of fuss. So you can try new things, discover you don't like them, and try something else. And you don't need to know a lot to make this work.

Sure, knowing the stuff about how it works is important, but this serves as a nice tool for understanding why. It lowers the barrier for entry, which in the Zope (and Plone) world is important.

Afterwards had great discussions with ZPUG at some bar in DC. Talked to Paul Everitt, Matt Bowen, Chris McConnough, Kapil, and some others. I was very sleepy so I was not as sharp as I can be, but I learned some interesting things.

Kapil agreed with a growing suspicion I have about Grok. I'll post about that in the future. Its mostly positive but it does have a couple critiques.

Kapil reinforced my feelings about Pylons vs Django. Another item for a future post.

JSON and Python

I hate XML.
I love JSON.

Now that I've made that clear, lets go into how I want Python to handle my JSON.

1. I want a built-in function to JSONify.
2. I want a built-in function to deJSONify.

Now in the Python world there are multiple JSON libraries. cJSON, simplejson, demjson and more. I've used a few of these and they all seem good. However, most have extraneous methods that I don't care about, or name their json handling methods funny. For example, simplejson has you do loads() and dumps() for loading and dumping of objects/strings, and load() and dump() for loading and dumping of files. Kind of nice, but all I really want is a JSONify and a deJSONify function from my JSON handler. I can do the rest!

So what I've thought about doing is writing a Python JSON package that would do the following:

1. Load a JSON handling package. It would have a list to select from so that if simplejson was not already on your machine, it might check for cjson, demjson, etc and grab the alternate instead.
2. Have a JSONify function that would convert Python objects to JSON.
3. Have a deJSONify function to convert JSON into Python objects.

This looks like a fun and handy little project.