New Year’s Python Meme

I love these blog memes, so I give you my version of Tarek Ziade's New Year's Python Meme.

1. What’s the coolest Python application, framework or library you have discovered in 2011?

For python libraries, that would have to be Kenneth Reitz' python-requests library. I've used it for an amazing amount of stuff and blogged about it. It took the grunge out of doing HTTP actions with Python. The API is clean and elegant, getting out of your way. It embodies the State of the art for API design, which closely matches the Zen of Python.

For applications, djangolint.com is awesome. It has helped me out so much on several projects. I would love to see something like this implemented and maintained for modern Python.

All the Python friendly PaaS efforts that have emerged are changing the landscape for those of us who want to launch projects but don't want to become full time system administrators in the process. Heroku, DjangoZoom, DotCloud, ep.io, gondor.io, and others are making it possible for developers to focus on development not server tooling. Google App Engine paved the way and it is wonderful to see the rest of the universe catch up with material that more closely follow core.

2. What new programming technique did you learn in 2011?

Event based programming! I've touched on it for years, but this year I really got a lot more more into it thanks to Aurynn Shaw kickstarting me and Audrey Roy expanding my knowledge ever since.

3. What’s the name of the open source project you contributed the most in 2011? What did you do?

I participated mostly as co-lead in the Open Comparison project, which amongst other things involved running the largest sprint at PyCon 2011. We maintained Django Packages and launched Pyramid and Plone versions of the project. We hope to launch a Python implementation in 2012.

I took a lot of notes this year at pydanny-event-notes - enough to make a book.

4. What was the Python blog or website you read the most in 2011?

Like Nick Coghlan, that would be http://planet.python.org.

5. What are the three top things you want to learn in 2012?

1. How to use whatever consistently maintained project that replaces PIL that works in Python 2.7.x and Python 3.x.
2. Really advanced Python as taught by Raymond Hettiger.
3. backbone.js

6. What are the top software, app or lib you wish someone would write in 2012?

A tool python-requests, but for shell access. Something like Unipath, but kept up-to-date and with nicely written documentation on Read the Docs.

A PIL replacement that is maintained, works for all modern Pythons, and is close enough to the PIL API to not cause too much confusion.

Something like Django Lint but for Python 2.7.x/3.x.

An open source project that tracks test coverages across PyPI and publishes reports of the results via an API.

Want to do your own list? here’s how:

• copy-paste the questions and answer to them in your blog
• tweet it with the #2012pythonmeme hashtag

Announcing Garbaginator!

While working on Packaginator at the PyCon 2011 sprints we discovered some serious issues in the way that Django handles garbage collection. After a huge amount of work, we managed to isolate and fix the problem. This 'fix', as it were, was only possible by doing a very sophisticated 'hack' of critical internal components of the Django Web Application framework. We also discovered that similar issues occurred in other existing Python application frameworks such as Pyramid, Flask, Web.py, Web2Py, Grok, Twisted, Tornado, Google App Engine, and Rails.

Since then the Packaginator community has been fiercely debating what we should do with our newly created set of hacks. After a lot of arguments going both ways we've decided to come up with our own application framework and release it to the world under the GPL license.

This brand new application framework ignores the lessons learned from all the other Python frameworks and embraces the cutting edge concept of Not-Invented-Here. It focuses less on features and enhancements over existing systems and much, much more on the critical concept of formal Garbage handling.

Some of the critical modules include:

• RubberGloves (for handling dirty objects)
• Django-Garbaginator
• Flask-Recycling
• Pyramid-Garbaginator
• Web.2.py Garbaginator Bridgerator ('cause people always get Web2Py and web.py confused with each other so we bridged them together)
• BlueBream

Check out the home page at: http://garbaginator.cartwheelweb.com

Fork Garbaginator on GitHub: https://github.com/cartwheelweb/garbaginator

Note: This was an April Fool's Day Joke

Why you should go to Pycon

Because you are new to Python
If you want to get better at Python then Pycon is one of the better avenues to learn. Think of it as a nine day data dump of Pythonic knowledge straight to your brain!

When it comes to the fundamentals of Python, the tutorials are a great way to hit the ground running. How do you think I learned? Also, there are dozens of beginner talks designed to introduce you what highly qualified people think you should be doing. Finally, the sprints are a great way to work with real experts for days - because even the most advanced projects needs people to knock out the invariable score of simple bugs, do documentation and write additional tests.

Also, Zed Shaw is going to teach his Learning Python the Hard Way tutorial throughout the conference. (follow that link!) To summarize, besides going over his lessons and some basic conference coaching, he's going to take his class to conference talks and then bring them back to apply what was learned. He won't be doing it alone either - a number of other python folk are going to volunteer their time to help, and I'm going to be one of them.

Because you are old to Python
A good chunk of the tutorials are designed specifically for experienced Pythonistas wishing to expand their knowledge (OMG Raymond Hettinger). And then as we all know the conference itself has scores of advanced talks designed to challenge you. Finally, what better place to work on your favorite project or get people to work on your project then the massive Pycon sprints ?

Want to do Python related technology full-time? What better place to look then where amazing companies and organizations hang out with the intention of hiring people? And invariably some of those hiring groups will be hosting parties too...

Because you are recruiting
Looking for good staff to do Python, Django, Plone, Pyramid, wxPython, NoSQL, Linux admin, Unix admin, BSD admin, Windows admin, Science, or Engineering work? Pycon will be where 1500 of the best and brightest people on the planet will hang out for over a week and some of them will be interested in new opportunities. Sponsorship is a great way to get yourself noticed by hundreds of Pythonistas.

Because it is cheap
Until January 25th Pycon registration is at its early bird prices. Even when it goes up it is still a pretty good deal. Admittedly the biggest expense is the hotel ($159/night) but there are decent and much cheaper hotels ($80/night last I checked) within a few blocks of the conference.

Stupid Template Languages

For years I've been absolutely certain that I really prefer stupid template languages any time I'm generating HTML. The less the template language can do the better. Since I spend most of my time coding in Python you might assume this applies just to Python, but I think it also applies to anything where you have the power to readily mix HTML generation and code.

The biggest annoyance I have with smart template languages (Mako, Genshi, Jinja2, PHP, Perl, ColdFusion, etc) is that you have the capability to mix core business logic with your end views, hence violating the rules of Model-View-Controller architecture. While the web can be hard to match to MVC, in general you aren't supposed to do that sort of thing. I've made the mistake of putting core logic in the wrong places in the past, but I'm proud to say I've gotten good at avoiding that particular mistake.

I don't work in a vacuum.

I often work on projects crafted by others, some who decided for arcane/brilliant/idiotic reasons to mix the kernel of their applications in template function/macros. This is only possible in Smart Template Languages! If they were using a Stupid Template Language they would have been forced put their kernel code in a Python file where it applies, not in a template that was supposed to just render HTML or XML or plain text.

What it comes down to is that Smart Template Languages designers assume that developers are smart enough to avoid making this mistake. Stupid Template Languages designers assume that developers generally lack the discipline to avoid creating horrific atrocities that because of unnecessary complexity have a bus factor of 1.

So what is a Smart Template Language?

In my own vernacular, template languages that let you write functions/macros are what I call a Smart Template Language. Some of them are brilliantly executed, the example of Jinja2 comes to mind, but invariably I suffer through abuse of its Macro control structure as implemented by others.

Misery Cubed a.k.a. Genius Template Languages

Next comes Genius Template Languages, which take things a step further. These template languages allow you to not only define functions/macros, but also let you embed unrestricted Python (or Java or Ruby or whatever) in the template. This 'feature' lets you code your entire application in the templates!  In the Python world what comes to mind is Mako and Genshi, but I'm sure there are many other tools with this 'capability'.

I like Stupid Template Languages!

Stupid Template Languages don't let you define functions/macros. They don't let you embed Python code. They barely let you define variables and often have simplistic control architectures.

For Django efforts, which is about 70% of my work, I like the Django Template Language (DTL). Since it is used by a huge community, there are a ton of useful apps which have it as a dependency. Switching away from it would mean cutting myself off from a large ecosphere of tools I can use to not reinvent the wheel.

Back in my Zope/Plone days I really, really enjoyed the Template Attribute Language (TAL) because it was stupid too. If I needed an XML generation template language and could import it easily I might consider using it again, or perhaps Chameleon, which is a new, improved version . The downside is that they come paired with another tool paired with it, METAL, which gave it macros. My own experience with METAL is that it was all too easy to do what we developers do with Smart Template Languages.

But DTL and TAL are slow!

So what?

If you want to boost your performance, first try caching. There are a ton of tools you can use, with Varnish being one I keep seeing in action. Read the docs on your favorite web framework's caching engine and apply what you learn. And Djangonauts should read up on Mike Malone as much as possible.

If after all that the site still delivers slow content and it appears to be a template language issue, then identify the bottleneck content and consider alternatives for that one portion. My favorite response is a bit of AJAX. Use your framework to render the content as JSON and have JavaScript parse it into legible content, a task which JQuery makes trivial.

Release classifiers in distutils/pypi

Thanks to Doug Napoleone I'm now aware there is already a convention followed for the python and framework versions, but it appears that not enough people are aware of it. This post is pretty much a reposting of the second comment of the post immediately preceding this one and Doug gets full credit for this post. I'm just repeating his message:

The release classifiers in this post should be included in the standard distutils documentation. For the moment, you can see the full list of classifiers here:
http://pypi.python.org/pypi?%3Aaction=list_classifiers

For the python language version the classifier is:

Programming Language :: Python :: x.y.z

With each version on it's own line. That way you can browse the repository by python version (see the bottom of the page):

http://pypi.python.org/pypi?:action=browse&c=214

There is also support for frameworks which you can see on that page as well. There it is done with:

Framework :: Django :: x.y.z

There is Zope, Plone, and a number of other frameworks already there.

In the example you gave the proper, and supported way of writing the metadata is:

Programming Language :: Python
Programming Language :: Python :: 2.4
Programming Language :: Python :: 2.5
Programming Language :: Python :: 2.6
Programming Language :: Python :: 2.7
Framework :: Django
Framework :: Django :: 0.96
Framework :: Django :: 1.0
Framework :: Django :: 1.1
Framework :: Django :: 1.2.1
Framework :: Django :: 1.3

Now it becomes a matter of education and illumination. This should be in the standard distutils documentation and arguably the home page of PyPI (or easily found there). And Django Packages will be supporting this functionality in the near future.

A request for new pypi classifiers

This request is to help enhance Django Packages, PyPM Index, and other projects. This would also help the Python community at large.

Would it be possible that a standard be established for listing in PyPI classifiers which versions of a package is known to operate? Using James Bennett's django-registration at http://pypi.python.org/pypi/django-registration as an example (see my bolded, last two lines to understand what I'm trying to demonstrate):

Development Status :: 5 - Production/Stable
Environment :: Web Environment
Framework :: Django
Intended Audience :: Developers
License :: OSI Approved :: BSD License
Operating System :: OS Independent
Programming Language :: Python
Topic :: Utilities
Python Versions :: 2.4, 2.5, 2.6, 2.7
Django Versions :: 0.96, 1.0, 1.1, 1.2.1, 1.3

The metadata system I'm writing about in this blog post is specified on the distutils documentation page.

I picked a Django package but this could be for Zope, Pyramid, PyQT, or anything.

If we had something like this in place then people could quickly identify on PyPI and other resources if a tool can be of use to them or if it needs to be updated to the latest code base. If this already exists, then can someone point me at the existing specification so I can promote it?

Edit: Noah Kantrowitz suggested I take a look at the 'requires' keyword which is part of the distutils spec. However, this does not show up in the PyPI API (http://wiki.python.org/moin/PyPiXmlRpc) and so doesn't fit our needs.

Leaving NASA

This has been a hard post to write.

I was delighted that on January 3rd, 2005 I started my first day working for the National Aeronautics and Space Administration (NASA). While I wasn't working on science efforts, I was at least contributing to the cause. In 2005 I was introduced by co-worker Chris Shenton to Python, which became my favorite programming language ever. I also learned tools like Zope, Plone, and Django. Over the past five years, I've met a lot of fascinating people in and around the agency, a list that seems endless in size and scope. That includes astronauts, scientists, engineers, developers, managers, and so much more.

This meant so much to me, and maybe because my first memories of television as a child were the moon landings of the early 1970s. I dreamed as a child of being an astronomer or astronaut, and sometimes I plot how I would redo my life to fit these dreams if I got a second childhood.

In the past year I've had some incredible opportunities present themselves to me. I've been presenting frequently on Django and Pinax. I've had the singular honor of writing course material for Holdenweb, LLC on behalf of the O'Reilly School of Technology. Representing NASA as a contractor to the Python and related communities has been both enjoyable and a great honor.

Yet all things, even good ones, must come to an end.

I've decided to become an independent consultant. My first project will be working with Revolution Systems (Jacob Kaplan-Moss and Frank Wiles) on a neat stealth project that looks very promising and once launched will help people. The project will be Python/Django/Linux based, and the client insists on accessibility, testing, and quality work. We'll be exploring the boundaries of what has been done with those tools and besides what must remain proprietary, a lot of our work will end up going back to the community. Sounds like my kind of thing!

My last day is April 1, 2010. I'm both excited to explore this new project, and saddened that my professional world for the past five years is coming to an end. Yet the overlap in technology and the participation of the NASA SMD python group in the open source world means that my work with NASA isn't coming to an end, its just transforming into something different.

Nevertheless, this is the end of an era for me.

Which is partly why I'm happy that I'll still be in touch with my fellow NASA SMD Python contractors such as Katie Cunningham, Chris Shenton, Chris Adams, James Saint-Rossy, and others. I also plan to be real friendly with the awesome Ames Research Center Python/Django/FOSS groups such as the intrepid Mark Friedenbach and the entire incredibly awesome Nebula team.

I'll miss having the pleasure of working with Leslie Cahoon, Jessy Cowan-Sharp, John Kasmark, Bob Ryan, Candace Solomon, Bill Keeter, Gamble Gilbertson, Meredith Mengel, Malik Ahmad, Jenny Mottar, Mike Brody, Virginia Butcher, Dawayne Pretlor, Michele Montgomery, Jim Consalvi,Siew Chin Hon, Hans Goetzelt, Shannon Lantzy, and many more.

Lastly, I'll miss the honor of serving civil servants such as Gretchen Davidian, Sharron Sample, and Ruth Netting and others.

My pre-history with Plone

In the late 1990s I was working at a job doing a mix of Foxpro development, windows system administration, and help desk. I enjoyed the Foxpro development (laugh if you will) and despised the other stuff. So I was always looking for work in other places where I could be doing pure development. However, lacking much work experience at that time, or knowledge of other programming languages, or a college degree, that made switching positions really hard.

Well, one day, probably in 1999, I stumbled across a job opportunity down in Fredericksburg. I I got an email inviting me down for an interview for Zope Corporation, or maybe its predecessor. The job meant working with an obscure language called Python and a web based development system called Zope. Python seemed pretty interesting, and doing development through the browser seemed fascinating. I even looked up some articles on Yahoo (this was before Google and Wikipedia).

Alas, I did not take the job. Python seemed too different and obscure and I was already stuck in the closet with Foxpro. Fredericksburg would have involved a move from the Washington, DC area and my ex-wife and I were even then scraping the bottom of the barrel for money. We lived on the metro, lacking a vehicle between us, so just getting down to Fredericksburg was nigh impossible. So I stayed with Foxpro, eventually finding work in a few other languages before finding my current home in Python in 2005.

What might have been

In an alternative history had I interviewed and been hired for the Python/Zope job I might have worked with Guido van Rossum during his tenure at Zope Corp! I might still be deeply entrenched with Zope and subsequently Plone. My linux skills would be much better. Fredericksburg would be my home, not Arlington so I probably would not be working for NASA right now. I would have never founded NOVA-Django. Finally, the armies of Iceland would have conquered the world.

A very different universe indeed.

Next: My history with Plone

Learn something new every year

As developers, we all make that pledge, don't we? Some of us learn new languages, new frameworks, new APIs, new methodologies. Rarely do we learn a new operating system.

30 years ago I was using whatever they called the operating system on the Apple ][ series of computers. 20 years ago I learned to use MS-DOS. 17 years ago I added Windows 3.1 to that mix. 15 years ago I ran into Windows 95 and the subsequent set of 98, 2000, Me, and XP. 10 years ago I started to play with Linux and Unix, mostly learning the simple shell commands needed to shuffle files around. 3 years ago I commenced working on Mac OS X.

During my Windows 98 days I was forced in doing help desk and system administration work. I hated it. The pop-up menus were not intuitive. Documenting what you did was hence agony. I begged my job to make me a pure developer but they forced me to continue down the path of Windows administration. Eventually I left that job and became a pure developer.

Over the years I got better at Unix and Linux. I even set up two servers for production use, one running Apache 1.3.x around 2001 and and the other a whole bunch of JBOSS stuff in 2003. Finding documentation was a pain, but the sense of logic compared to the chaos of windows pop-ups was a relief. After that when I would touch Unix or Linux it was basic file management stuff and nothing sophisticated, and my beginner skills never got a chance to grow further.

Flash forward to 2007 and beyond where I did serious Python development for Plone and then Django applications. Development was on Mac OS X and production was Linux. My neophyte level skills no longer where quite enough. Our system administrator and other co-workers helped out or did the work but it limited and frustrated me. Nevertheless, the time to really bone up on Linux never happened.

Well, this past week has been a crash course in ramping up my Linux skills. The next version of SMD Spacebook will be put into production as an RPM. Its been assigned my task to do it. This would involve a lot more in the way of Linux that what I was used to, since I needed to create an environment to run the RPM build commands, and that couldn't be our testing or production environments. That meant a chunk of setup and configuration of a lot of packages, and various proprietary bits and pieces in order to meet government and corporate requirements.

I tried not to bug my co-workers too much except for when their documentation didn't quite explain everything, and then I added to their documentation. I'm still a beginner but I feel like I've finally gotten a chance to spread my wings in this arena a little bit and learn a lot.

This will make it easier for me when I finally get around to getting this blog off blogger and into what will probably be Django-Mingus.

Code I'll reuse

When I'm evaluating a package to use in my work or play I tend to look at five things. I think many of my on-line colleagues look at a similar list. If its missing too many of these things then odds are I'll go somewhere else for my needs or roll my own.

Documentation
Did the author bother with a README file? How about some sphinx documentation? How complete it it? Does it get me started and give a few basic examples?

I'm okay with typos and mistakes. These happen. But I want to see

Licensing
Everyone has their own idea of what they like for licenses. I like the MIT/BSD thing. I can understand the attraction to LGPL and GPL although they aren't for me. What I can't stand and won't use are monstrosities like GPL/Commercial used by such libraries as ExtJs.

Want to make money off your software? Easy... let anyone use it and charge for support. Worked damn well for communities and companies like Python, Django, Plone, various Linux distributions (Redhat anyone?), etc...

Eggification
Is your software constructed so that it can be installed via easy_install or pip? And yes, this is a bit of mild embarrassment for me, so I'm happy enough to eggify other people's work.

Tests
Do you have tests? Even a nearly empty tests file or folder? How about a test application? If you have no tests then your package is suspect. How do I know it will work independently of your personal computer?

Code Quality
Does the code smell bad? Can it be easily extended? If its innovative but the code needs work is it on a DVCS so more people can easily contribute?

22 more posts to go!

Pycon talk proposal reminder

If you want to present a talk at Pycon 2010, you've got a mere fourteen days to make a proposal. Hurry up and get cracking on what you want to present before its too late. And remember, the more proposals we get the better Pycon will be!

More importantly, if you get your proposal accepted then you earn major Python community street creditability. Recruiters will smash down your door in the middle of the night in the hopes of interviewing you. You'll be followed by groupies and a posse of yes-men (and women). And Alex Gaynor will be your friend.

Turning away from CAPTCHA

When I first encountered CAPTCHA I thought it was a grand idea. My opinion has changed.

First of all I believed it allowed humans and machines to be differentiated on the web. Sadly, cracking CAPTCHA is done on a regular basis, and there are white papers on how to do it in lots of different languages (such as this one in Python). Bugs in the submission system or cheap human labor works as well. There are enough ongoing issues that most registration systems still include some sort of email system to help filter out the robots.

Second, CAPTCHA fails on accessibility. Yes, you can provide an audio alternative, but what if your users are blind AND deaf? Well, I've actually been told by accessibility experts that a non-CAPTCHA form should be provided for those people. Yes, when I said 'experts', I meant plural!

So where does that leave us for weeding out the humans from the computers?

Right now I'm a fan of logic based CAPTCHA alternatives. The idea is to provide simple questions that are relatively easy for humans to solve and hard for computers to answer. A good example would be, 'Today is Saturday. Yesterday was ___', and the idea is that you should have hundreds or thousands of questions. In fact, I came up with a Plone widget package called humanator to support this concept. There will be a Django version shortly.

There are some issues to overcome:

• We need to cook up a few thousand questions to make it a bit harder on the brute force people.
  
• Internationalization will require translators from many languages to support the projects.
• There is also the issue of the cultural context of the questions. Since this is supposed to be user friendly we don't want to ask any inappropriate questions. I can police English pretty well, but I'll have little control over what happens in other languages.
• Some people thing the logic method is weaker from a security perspective than CAPTCHA. Both arguably rely on a form of security through obfuscation, and I think with about the same amount of work both can be hacked. But a logic based system is easier to set up. ;)
  

DC Sprints location is changed!

Cosi seemed great. Decent food, internet, pleasant seating.

Except they blocked the really useful ports. SSH, Git, you name it.

So we have switched to the Arlington Central Library, just two blocks away.

http://www.arlingtonva.us/departments/Libraries/about/LibrariesAboutCentral.aspx

DC sprints location is official!

All the details can be had by following this link:

http://groups.google.com/group/NOVA-DUG/web/dc-branch-of-the-eurodjangocon-09-sprints

I'm rather excited by this event. It is the first in what I hope becomes a DC tradition of doing 'local mirrors' of python conference sprints with involvement from all the area python groups. This way, more people get to contribute, our community works better together, and the networking opportunities can abound.

On the other hand, I wish I had found more time to organize this event. And that I had found a good location inside of Washington, DC. I suspect this first effort will have a low turnout, but I won't mind being pleasantly surprised. ;)

My hope is that this year alone we might be able to exploit similar efforts in 2009 for the Euro Pycon, DjangoCon, and Plone Conferences.

With that in mind, I'm going to insist that these local mirror sprints be completely framework and tool neutral. For starters, under the hood, aren't we all just doing python? Plus, I'm of the very firm belief that any major system with initiative has things to teach, be it lessons learned or neat ideas. For example, Jacob Kaplan-Moss uses zc.buildout, which grew out of the Zope community. And while we might not have the ability to listen in on actual conference sessions, think about the chances we'll get to look at each other's tools and methods.

Sweet spots of Plone

Yes, I am posting this with Django and Pinax tags because I think my Django and Pinax friends can learn from the lessons of Plone.

I'm admittedly more interested in doing Django these days yet I keep myself firmly aware of the things for which I think Plone (and its components and relatives) is unsurpassed in the Python framework world. Arguing these things to Djangonauts is interesting, because so many times they just reject things out of hand (just like certain Zope zealots I know). And I think that is to their loss. When you hit Plone's sweet spot, things get interesting.

Standards
I've ranted about this before. In essence Plone follows a subset of the Dublin core. Any database object in Plone has certain fields you can rely on in searches, views, and business logic. You don't have to introspect the objects to find these stock fields. Until I went back to entirely developer defined models with Django, I forgot just how much I had taken this for granted.

Workflow
Years ago some madman integrated DCWorkFlow into Plone. You can build custom workflows either in script or via the UI. The scripts and interface to use it has been unpleasant until recently, but then name a workflow engine capable of handling complicated workflows that is fun/easy to use? Thanks to Martin Aspelli his product called collective.wtf, Plone workflow management has become much, much easier. As far as I know, nothing in Django (or TurboGears or Rails) compares except maybe GoFlow, and I don't speak French.

Object Oriented Database
I like the Django ORM. I like SqlAlchemy. Until I run into the edges of the fact that they are modeling table records as objects. Object Relational Impedance Mismatch anyone? Ouch the pain! I do recognize that relational databases are very useful and powerful, but sometimes I wish things were different. One day, when I'm indepedantly wealthy and own my own private tropical island I plan to get the Django ORM running with the ZODB.

ACLs
Groups and Users permissions in Plone are implemented really well. Object oriented databases handle hierarchies and references very well, so the whole parent-child ownership thing is done easily. This is critical in CMS work - ownership and control. Other frameworks have this built-in, but it is not automatically attached to each content object that you create. You get very used to it being magically done for you in Plone.

Finger method of judging graphic design

Katie Cunningham mentioned this in her blog post today. Since I was asked what that was meant by several people I decided to write it up so you can see one of my rants.

The method is simple:

• Put your design on a screen.
• Your hand goes on the screen with fingers horizontal. Ignore the thumb.
  
• Count how many fingers it takes from the top of the screen to content. If you run out of fingers on one hand that means your wonderful design is going to force users to scroll to content. Maybe not on your huge desktop monitor, but on the sort of desktop monitor or laptop that has become ubiquitous, absolutely!
• I have tiny hands for a guy. So for you people with big hands, try it with two or three fingers.
  

Now lets take a look at how the Python community matches up to my finger method.

• http://python.org is just one of my fingers! Python.org FTW!
• http://djangoproject.org is two of my fingers! Good job Django!
• http://plone.org has a new and improved site that is two to three of my fingers, depending on the location. Plone does good!
• http://www.turbogears.org jumps between three and four fingers. Acceptable.
• http://code.google.com is two and a half fingers to content on project pages. Definately workable!
• https://github.com gets it with three fingers.
• http://pinaxproject.com is over four fingers to get to content? Boo!
  
• http://code.pinaxproject.com makes up for the parent site with two fingers to get to content.

So the Python community does pretty well on average. So does Facebook, Twitter, and the other good social networking sites. Now think about the sites that never went anywhere and how my finger method applies.

How about some some abject failures from outside the python community?

• CSS Zen Garden is lovely. And yet the designs are often unworkable to anyone outside of a group of graphic designers looking to hand each other design awards. You get designs that take ten (10) fingers before you get to content. WTF?
• http://www.apple.com/mac fails at over four fingers.
  
• http://www.microsoft.com/windows/help-how-to.aspx goes over four fingers and includes my thumb.
• http://www.adobe.com is as bad as the stuff on CSS Zen Garden.
  

Is this scientific? Heck no. Yet it is a quick and cheap gauge as to why you rarely get repeat visits to your site or why they only visit your download pages.

The end of my Feedfeeder story

Another post about Plone... but this time about me and not about Plone.

For about 18 months I have wrestled with consuming broken RSS feeds to pick up image of the day fields stipulated by customers. These are feeds so broken that no RSS parser, including the masterful Feedparser, can handle them (for example, one image of the day feed usually puts the image in the RSS header and changes that each day - no history is maintained). They aren't actually RSS, they just possess a file name that ends with '.rss'. Plus, periodically the way they are written changes so custom logic fails.

I have forked Reinout van Rees FeedFeeder project, and even proposed complicated logical revisions to handle broken these broken feeds and their shifting implementation. I called it Feedfeeder v2. Reinout always seemed hesitant, and I watched as other people extended on his work and despaired. I knew something was wrong but couldn't put my finger on it. I hesitated to work on it, even though funding for it was readily available.

Then between Spacebook, Pinax, and other efforts I shelved this effort for months, hiding my head in the virtual sand. And yet I knew it needs to be addressed. How could I handle something that broke the otherwise wonderful Feedparser?

During Pycon 2009 I came up with the answer. I took an excellent tutorial on html scraping and learned lots of little tricks to reinforce my skills with BeautifulSoup. You see, screen scraping is a secret pleasure I have. Scraping out a bit of data from a page is like a little puzzle. When I talked about this to someone, in the middle of my discussion with them the answer became clear as day.

The answer was to turn the problem from a RSS interpretation problem to a simple web page scraping puzzle.

1. Fetch via urllib the XML file that pretends to be RSS.
2. Parse it using BeautifulSoup or html5lib.
3. Get all the images listed.
4. Discard all but the largest image.
5. Guess out the meta-data from the XML file and store that for the image.

Problem solved.

Now I just need to make a Plone 3 package to do this for me and my angst is finished.

My apologies Reinout for the time spent on trying to cook a solution via Feedfeeder. Thank you for your insights and your extreme patience. I think you tried to tell me to take a different path.

What I would change in Plone: Generic Setup

At the Washington, DC 2008 Plone Conference I took Joel Burton's awesome two day tutorial on Plone Theming. I don't do design myself, but I have implemented them from time to time. My CSS skills are weak, but Plone gives us lots out of the box so I figured this would be a great way to ramp up on skinning. Also, from a work perspective it made sense, because many Plonistas seem to care more about the engineering/database side of things rather than on the front end.

Joel's class, to put it succinctly, was magnificent. If you want to learn Plone, take one of his classes. It is well worth the money.

We learned about reorganizing viewlets on the fly, making css changes, tab controls, content well manipulation and lots, lots more. Two days of knowledge that didn't just go in one ear and out the other, but really stuck.

I was one of the tech-heavy guys in the room, and the rest were graphic designers. At first the ZMI terrified them, and the two different types of views did not go over so easily. But thanks to Joel, by the end of the two days the designers were really enjoying what they could do with Plone. Everyone felt confident and ready to move forward. The positive energy in the room was energizing!

Then it came time to 'save' the site design we had been working on. In other words, export it so it could be reused. Or sold even! We fired off Generic Setup to export things, and then...

The positive energy diminished

I'm capable of searching through Generic Setup XML files to figure out what is needed in order to capture a Plone skin. I'm sure anyone bothering to read this blog entry is the same way. Yet graphic designer skills tend to end at Photoshop, sometimes at CSS and JavaScript, rarely at PHP. Asking them to go through a few thousand lines of XML is not the way to garner good will.

There was grumbling, and mumbling. Some that were happy until that moment left the class unhappy. Not because of Joel, but because of the export tool that Plone gives us.

When I brought this up with Joel after the class, he said he was aware of it but it was beyond his control. When I brought it up with others, I got a mix of apathy or rejection of the issue. I was told by several that the designer could hand it off to a developer if this was beyond them. Or that it would be too challenging to address. I mentioned it to Tres Seaver and I believe he said he would look into it.

In retrospect, I should have made a bigger stink about it.

In my opinion, Plone should be as developer friendly as possible. Generic Setup should have an 'Export Skin' function, available via the Plone Control Panel. Not the ZMI.

And I think that this should be part of core Plone. As soon as possible. Preferably Plone 3.3.

Why?

At least half the customers you will ever have care very little about the back end. They just want a pretty and functional site. Which is why graphic designers are often so important to them. For example, how many customers do you have who care more about the comps of the graphic designers than the engineering that publishes it?

By making Plone more accessible and friendly to graphic designers, it means they will stay excited about it. Which will attract more of the graphic designer friendly customers.

It gets better.

This export utility would empower graphic designers. They will be able to create Plone skins and publish them. They can make money doing this, advertising their craft. In fact, I predict dedicated on-line stores for this sort of thing will become popular, making the cottage industry for plone skins much more viable.

Which is great for us developers who lack decent design skills. If exporting skins is easy, then I can find (and purchase) them and modify to suit customer needs. And if I need one from scratch, I can rely on a larger crowd of graphic designers.

Jinja2 in zope

Getting Django Tempates inside of zope is not trivial. Yet getting Jinja2 to run inside of zope is easy! Use easy_install (or whatever) to fetch Jinja2 from PyPi and then from the zope debug shell:

>>> from jinja2 import Template
>>> t = Template('{{ name }} rocks!')
>>> t.render(name='Guido')
u'Guido rocks!'

So what does this mean?

It means that right now you can have Django style templates inside of Plone, Zope, Grok, et al. However, you can't mix TAL and Jinja2 in any way via template inheritence.

Off the top of my head I can think of two possible use cases:

1. You have the HATE for XML based template languages and just need something else for all rendering of all content including HTML. This is actually a very feasible option for Grok and pure Zope application development, yet functionally impossible for Plone. Too much of Plone is woven into TAL to make this work.
2. You are using lots of AJAX via KSS, JQuery, YUI, plain old JavaScript, etc and want something handy to help you write content coming from the server side. This is feasible in Zope, Grok, and even Plone.

The downside of using Jinja2 is that you are adding more complexity to the mix. I like simplicity. I would be very careful about using Jinja2 in Zope products, and would only consider option #2.

Thanks to the the indomitable Ian Bicking for suggesting Jinja2.

What I would change in Plone: Templates

These are my thoughts. My thoughts are not those of the luminaries in any community, but just your average Joe Developer. I formed these thoughts over Pycon 2009, and my experiences with Plone, Zope over 2 odd years and Django over the past 4-5 months. A lot of this is therefore inspired by my recent experiences with Django.

Item #1 - Plone/Zope needs Django Templates out of the box

You heard me.

I like the Template Attribute Language (TAL). TAL is stupid. Stupid in that it is very hard to encrypt my templates with business logic. I like this because I recognize I have no discipline when it comes to breaking the Model-View-Controller paradigm if a framework lets me. I also like TAL because it lets me create XML type documents very easily.

I like the Django Template Language (DTL). DTL is stupid. Stupid in that it is very hard to encrypt my templates with business logic. I like this because I recognize I have no discipline when it comes to breaking the Model-View-Controller paradigm if a framework lets me. I also like DTL because it lets me create non-XML type documents very easily.

I like both languages. Some people do not. Advocates of one template system style say very snarky things about the other template system style. Inevitably it comes down to that one is XML focused and the other is not. And from their point of view, they are completely right.

So I think that a version of DTL should be ported to the Plone/Zope world. And for Plone, it should be made readily available out of the box. This port should do all the default DTL filters and as many of the default tags as possible.

Which would I use? I would use either. More importantly, people who love XML based templates could use what they want and people who hate XML based templates could use what they want.

Everybody wins.

Item #2 - Plone TAL documentation needs a cheat sheet of the good functions

For Django, I can go to the DTL tag/filter page and find a handy reference on all the tags and filters available out of the box with Django.

Where is the same with Plone? The documentation has gotten really freaking good, but where is this sort of handy developer reference?

Yes, I have learned that you can use string methods on strings, and that there are various utility methods you can call if you know the name of the right portal object, but these are not in one easy-to-find location.

We Joe Developers need and love this sort of thing.