Announcing Consumer Notebook!

Need a Python programming language book? Want to see a comparison of the ones I own and use? Check out my Must-Have Python Programming Books comparison grid.

Let's drill down and take a closer look at one of the items on the page, in this case Doug Hellmann's amazing The Python Standard Library by Example. The product detail pages include the ability to add pros and cons and attach said products to comparison grids and specialized lists like 'my wishlist' and 'my possessions'.

Speaking of wishlists, check out my own:

In order to add items, like footy pajamas,  I click on the 'add' button and paste the Amazon (or BestBuy) URL into the form:

At this time we just handle Amazon USA and BestBuy USA. In the future we plan on adding more affiliate providers, including non-USA providers to support our non-USA friends.

There's a lot more than that...

In addition to weekly infographics, comparison grids, lists, and products, Consumer Notebook also awards points, coins, badges, and a growing privilege set to participating users. We even implemented an energy bar which regenerates over time, designed to match the pace of human users and serve as one of the brakes on scripts and bots.

Technology

I built this with Audrey Roy using Python, Django, JQuery, PostGreSQL, Memcached, and RabbitMQ. I'll be blogging in depth about the technical side in an upcoming post.

Genesis

It was the summer of 2010 and we were brainstorming ideas for a coding contest called Django Dash. The one we settled on was a listing and comparison site for Django called Django Packages. The result has been a very useful tool for the Django community. Eventually, with the help of several dozen people, we turned the code into the Open Comparison framework and launched Pyramid and Plone implementations. Time permitting this year, we plan to do Python, Flask, Twisted, Node, JQuery, and other implementations.

Since then we've wanted to do something similar, but in the context of products. And we wanted to do it right - elegant design combined with an ad-free space. So we cooked up Consumer Notebook, launching today!

We'll be adding features and enhancements in the months to come. We've acquired a community manager, and even have a blog. We would love for you to check out the site, share it with your friends and family, and send us your commentary, suggestions, and advice.

Django Dash Lessons Learned

Our experience with Django Dash 2010 was that it was an wonderful exercise in classic Django development, cowboy/cowgirl coding, and drinking copious amounts of caffeinated beverages. The result, Django Packages, is something we are happy with, are continuing to improve, and hope will improve the community.

Lesson: Fixtures are a must

Django gives you this amazing Admin control panel. As soon as you get your models in place and are entering test data, start creating fixtures. For the dash we named them initial_data.json and loaded them into the individual app directories. This meant that every time we blew away the database we got a reload with records in place. Sometimes this means you have to hand-edit JSON (or YAML if you swing that way), but the alternative is to waste time re-entering the same data again and again. Don't forget to change the names of your test fixtures before you launch!

From the command-line how to save a fixture pydanny-style:

./manage.py dumpdata package > apps/packages/fixtures/initial_data.json

One nice thing about fixtures is that when you do have the time/need, you can use them to help you write tests. And it makes development easier for contributors.

Lesson: Research ahead of time

In the days before the contest, we researched to see if our target repos (Github, ;BitBucket, and Google Project Hosting) each had an API and a python library to speak to that API. Github has both an API and python library, Bitbucket has an API but no library. And as far as we can tell, Google Project Hosting lacks both API and library (someone please tell me I'm wrong about Google Project Hosting lacking an API).

This meant that when we commenced coding we knew which code base to work with - we weren't trying to look up this or that random package.

We did the same thing for rendering charts.

Lesson: Get it working then optimize

Looking at some of the code makes us wince a bit. But we got it working. Now we can go back and do some code cleanup, maybe use an XML parser instead of regex to try to scrape content from PyPI, and generally feel better about ourselves.

Lesson: Plan out system architecture in advance

In retrospect it was really amusing, but the night of launch the site was serving via the Django runserver command. We were so dead tired and neither of us are crack system administrators that we did what we had to do to score the contest launch point. The next day Audrey got the site running underApache, and next week we'll be giving someone else system access to increase reliability. Next year for the contest we'll probably use something like this or get continuous integration running in the first hour.

Lesson: Don't be afraid to chat with others after the contest starts

Share your ideas, selected packages and frameworks with your competitors. The break from coding helps clear the mind and they might counter with a better idea/package/framework you can use.

Announcing Django Packages!

I'm part of a two person team that just launched that BETA site for http://djangopackages.com, a site that is designed to list all the Django Applications, Frameworks, and Packages created by the Django community. While there are already a few places to look for these things, it is quite easy to argue that they are challenging to navigate, don't give any hard metrics, or are woefully incomplete/unstable/closed. Our goal was to provide an attractive, easy-to-navigate, easy-to-add-data, stable site and share it with the world.

Also, this was our entry into Django Dash 2010, and was the culmination of a few days of brainstorming over paper, a lot of research, and two days of feverish coding/designing. The project was feature complete to our specifications at 5pm the second day, and the rest of the time was spent adding in UI tweaks, usability enhancements, and trying to deploy our creation.

Since then, we've cleaned up a the UI, improved the design, and got the site stable. The code is open source and on github, so fork and contribute!

Design Consideration: No 'Like' button or 'Rate my app' rating systems

We wanted hard metrics. So the package numbers are pulled from the repo sites such as Github, Bitbucket, and Google Code. Otherwise things get weighted funny. Sure, this system can be monkeyed with, but its a good metric for now. We've had suggestions from Django core developers of coming up with a quality check system, things like pypants and/or a formalized approval system.

Design Consideration: Grids

Early on we wanted to duplicate and improve upon the Django CMS Comparison page. There is also a version for Forums, but it would be nice to have a current one for blogs! In addition, recently I heard that 'tag clouds are the mullets of web 2.0'. This really struck a chord in my soul. Since we had metrics on packages, why not compare those metrics, and use those comparisons, which we call 'grids', instead of tags? In fact, we extended our idea and instead of traditional tabs we use grids in the top navigation area as seen below:

Design Consideration: Categories

The site groups packages into three categories, 'Apps' which are individual django applications. 'Frameworks' which are aggregates of apps and python modules. And 'Projects' which are implementations of Apps and Frameworks. We've thought about adding 'Tools' but weren't sure if there was anything that fit that concept, and we are leery about allowing regular Python efforts into the fold.

Design Consideration: Regex vs XML

Slurping data out of Github was easy, especially since I used python-github2. Bitbucket has a RESTful API as well that serves out JSON. I think Google Code does as well. PyPI does not and DOAP on PyPI seems to give little that is useful, so I was forced to do screen scrapes of version numbers and downloads. I'm much faster with Regex and string methods than XML juggling, and speed was of the issue this weekend. I'm not sure what benefit there is to redoing it in HTML5lib or lxml, since what I have works and appears to be stable.

Design Consideration: Leave caching and optimization for later

Besides a tiny bit of memory based template caching on the home page, there is/was no optimization. In time I plan to cache many things using a proper key/value store like redis or memcached. Perhaps not before more design and usability work is done.

Why scared of rabbits?

You wouldn't understand unless you live on the Kansas prairie.

Note: if you have any suggestions, issues, problems with Django Packages please use our issue tracker!