The end of my Feedfeeder story

Another post about Plone... but this time about me and not about Plone.

For about 18 months I have wrestled with consuming broken RSS feeds to pick up image of the day fields stipulated by customers. These are feeds so broken that no RSS parser, including the masterful Feedparser, can handle them (for example, one image of the day feed usually puts the image in the RSS header and changes that each day - no history is maintained). They aren't actually RSS, they just possess a file name that ends with '.rss'. Plus, periodically the way they are written changes so custom logic fails.

I have forked Reinout van Rees FeedFeeder project, and even proposed complicated logical revisions to handle broken these broken feeds and their shifting implementation. I called it Feedfeeder v2. Reinout always seemed hesitant, and I watched as other people extended on his work and despaired. I knew something was wrong but couldn't put my finger on it. I hesitated to work on it, even though funding for it was readily available.

Then between Spacebook, Pinax, and other efforts I shelved this effort for months, hiding my head in the virtual sand. And yet I knew it needs to be addressed. How could I handle something that broke the otherwise wonderful Feedparser?

During Pycon 2009 I came up with the answer. I took an excellent tutorial on html scraping and learned lots of little tricks to reinforce my skills with BeautifulSoup. You see, screen scraping is a secret pleasure I have. Scraping out a bit of data from a page is like a little puzzle. When I talked about this to someone, in the middle of my discussion with them the answer became clear as day.

The answer was to turn the problem from a RSS interpretation problem to a simple web page scraping puzzle.

1. Fetch via urllib the XML file that pretends to be RSS.
2. Parse it using BeautifulSoup or html5lib.
3. Get all the images listed.
4. Discard all but the largest image.
5. Guess out the meta-data from the XML file and store that for the image.

Problem solved.

Now I just need to make a Plone 3 package to do this for me and my angst is finished.

My apologies Reinout for the time spent on trying to cook a solution via Feedfeeder. Thank you for your insights and your extreme patience. I think you tried to tell me to take a different path.

What I would change in Plone: Generic Setup

At the Washington, DC 2008 Plone Conference I took Joel Burton's awesome two day tutorial on Plone Theming. I don't do design myself, but I have implemented them from time to time. My CSS skills are weak, but Plone gives us lots out of the box so I figured this would be a great way to ramp up on skinning. Also, from a work perspective it made sense, because many Plonistas seem to care more about the engineering/database side of things rather than on the front end.

Joel's class, to put it succinctly, was magnificent. If you want to learn Plone, take one of his classes. It is well worth the money.

We learned about reorganizing viewlets on the fly, making css changes, tab controls, content well manipulation and lots, lots more. Two days of knowledge that didn't just go in one ear and out the other, but really stuck.

I was one of the tech-heavy guys in the room, and the rest were graphic designers. At first the ZMI terrified them, and the two different types of views did not go over so easily. But thanks to Joel, by the end of the two days the designers were really enjoying what they could do with Plone. Everyone felt confident and ready to move forward. The positive energy in the room was energizing!

Then it came time to 'save' the site design we had been working on. In other words, export it so it could be reused. Or sold even! We fired off Generic Setup to export things, and then...

The positive energy diminished

I'm capable of searching through Generic Setup XML files to figure out what is needed in order to capture a Plone skin. I'm sure anyone bothering to read this blog entry is the same way. Yet graphic designer skills tend to end at Photoshop, sometimes at CSS and JavaScript, rarely at PHP. Asking them to go through a few thousand lines of XML is not the way to garner good will.

There was grumbling, and mumbling. Some that were happy until that moment left the class unhappy. Not because of Joel, but because of the export tool that Plone gives us.

When I brought this up with Joel after the class, he said he was aware of it but it was beyond his control. When I brought it up with others, I got a mix of apathy or rejection of the issue. I was told by several that the designer could hand it off to a developer if this was beyond them. Or that it would be too challenging to address. I mentioned it to Tres Seaver and I believe he said he would look into it.

In retrospect, I should have made a bigger stink about it.

In my opinion, Plone should be as developer friendly as possible. Generic Setup should have an 'Export Skin' function, available via the Plone Control Panel. Not the ZMI.

And I think that this should be part of core Plone. As soon as possible. Preferably Plone 3.3.

Why?

At least half the customers you will ever have care very little about the back end. They just want a pretty and functional site. Which is why graphic designers are often so important to them. For example, how many customers do you have who care more about the comps of the graphic designers than the engineering that publishes it?

By making Plone more accessible and friendly to graphic designers, it means they will stay excited about it. Which will attract more of the graphic designer friendly customers.

It gets better.

This export utility would empower graphic designers. They will be able to create Plone skins and publish them. They can make money doing this, advertising their craft. In fact, I predict dedicated on-line stores for this sort of thing will become popular, making the cottage industry for plone skins much more viable.

Which is great for us developers who lack decent design skills. If exporting skins is easy, then I can find (and purchase) them and modify to suit customer needs. And if I need one from scratch, I can rely on a larger crowd of graphic designers.

Northern Virginia Django Users Group lives!

Our acronym is NOVA-DUG.

We have a google group here: http://groups.google.com/group/NOVA-DUG

Me and my co-worker Katie Cunningham, are the co-owners of the group. We insist on ownership because we know exactly what we want out of the group.

• It has to meet on a night we can easily attend and that night is invariably Wednesday.
• It has to meet at a location we both can get to easily.

See you around.

Django Dash worst concept contest

The very scary Tony Hauber mentioned Django Dash on twitter. I was doubting if I would try to do it, because coming up with original ideas is hard and my design skills suck.

Then, via twitter, the wonderful Barbara Shaurette suggested that a clever implementation of a stupid idea + bad design is the only way to go in such a contest. That made me laugh. Then she suggested a little side competition for 'worst concept'.

Oh yeah! The contest is so on!

After a bit of trash talking with each other, it was exposed that she thinks that by living around start-up central in San Francisco she has more stupid ideas to work with than me. I am certain, however, that being in the Washington, DC area, the hundreds of elected representatives and senators around gives me the advantage of more stupidity.

I'm going to be decisive and suggest some rules for our little side 'worst concept' contest:

• Judging of the worst is by any third party we mutually agree on. I suggest a BDFL or framework tyrant, but will go with anyone you agree on.
  
• An implementation must be attempted during Django Dash. You get bonus points for completion of the implementation.
• The loser buys a drink for the winner at the next conference we both attend.

Barbara, are you okay with this side bet?

Jinja2 in zope

Getting Django Tempates inside of zope is not trivial. Yet getting Jinja2 to run inside of zope is easy! Use easy_install (or whatever) to fetch Jinja2 from PyPi and then from the zope debug shell:

>>> from jinja2 import Template
>>> t = Template('{{ name }} rocks!')
>>> t.render(name='Guido')
u'Guido rocks!'

So what does this mean?

It means that right now you can have Django style templates inside of Plone, Zope, Grok, et al. However, you can't mix TAL and Jinja2 in any way via template inheritence.

Off the top of my head I can think of two possible use cases:

1. You have the HATE for XML based template languages and just need something else for all rendering of all content including HTML. This is actually a very feasible option for Grok and pure Zope application development, yet functionally impossible for Plone. Too much of Plone is woven into TAL to make this work.
2. You are using lots of AJAX via KSS, JQuery, YUI, plain old JavaScript, etc and want something handy to help you write content coming from the server side. This is feasible in Zope, Grok, and even Plone.

The downside of using Jinja2 is that you are adding more complexity to the mix. I like simplicity. I would be very careful about using Jinja2 in Zope products, and would only consider option #2.

Thanks to the the indomitable Ian Bicking for suggesting Jinja2.

What I would change in Plone: Templates

These are my thoughts. My thoughts are not those of the luminaries in any community, but just your average Joe Developer. I formed these thoughts over Pycon 2009, and my experiences with Plone, Zope over 2 odd years and Django over the past 4-5 months. A lot of this is therefore inspired by my recent experiences with Django.

Item #1 - Plone/Zope needs Django Templates out of the box

You heard me.

I like the Template Attribute Language (TAL). TAL is stupid. Stupid in that it is very hard to encrypt my templates with business logic. I like this because I recognize I have no discipline when it comes to breaking the Model-View-Controller paradigm if a framework lets me. I also like TAL because it lets me create XML type documents very easily.

I like the Django Template Language (DTL). DTL is stupid. Stupid in that it is very hard to encrypt my templates with business logic. I like this because I recognize I have no discipline when it comes to breaking the Model-View-Controller paradigm if a framework lets me. I also like DTL because it lets me create non-XML type documents very easily.

I like both languages. Some people do not. Advocates of one template system style say very snarky things about the other template system style. Inevitably it comes down to that one is XML focused and the other is not. And from their point of view, they are completely right.

So I think that a version of DTL should be ported to the Plone/Zope world. And for Plone, it should be made readily available out of the box. This port should do all the default DTL filters and as many of the default tags as possible.

Which would I use? I would use either. More importantly, people who love XML based templates could use what they want and people who hate XML based templates could use what they want.

Everybody wins.

Item #2 - Plone TAL documentation needs a cheat sheet of the good functions

For Django, I can go to the DTL tag/filter page and find a handy reference on all the tags and filters available out of the box with Django.

Where is the same with Plone? The documentation has gotten really freaking good, but where is this sort of handy developer reference?

Yes, I have learned that you can use string methods on strings, and that there are various utility methods you can call if you know the name of the right portal object, but these are not in one easy-to-find location.

We Joe Developers need and love this sort of thing.

Forming a Northern Virginia Django User Group

There is already a Django User Group in the Washington, DC area (Django District). However, it looks like they meet in Silver Spring. Yuck. I live in Arlington, VA and I don't drive. I'm sure they are a nice bunch of people, but I want something more accessible.

I am a member of the Zope/Python User's Group of DC, which is organized by awesome Alex Clark. It has been going on for ages. Unfortunately they meet on Tuesdays which does't work for me or Katie Cunningham except maybe once a year.

See, what I want in a user's group is one that meets on a Wednesday or maybe Thursday once a month or so in a metro-accessible place with wifi in Arlington, Virginia. Katie agrees. So we are going to form what I nominally call the Northern Virginia Django User Group (shortened perhaps to Nova-DUG). We'll discuss and show off Django and the odd bit of python and pinax. We will be happy to run periodic events with Django-District, and we'll make sure not to fall in the same week as them. Should be fun for all!

More details as we cook them up.