Sunday, April 27, 2008

Feedparser does not work with Google App Engine

After my laughable mistake of trying to do a import feedparser.py, I sat down yesterday and spent half an hour writing my rss aggregator for Google App Enginer. Critical, of course, was use of the excellent feedparser project. It was easy to get everything working, and while not styled it looked good. Everything except for using Feedparser to parse the incoming rss and atom feeds.

Alas, Feedparser tries to use a few modules that the enterprising folks at Google restricted. I haven't done any research yet, but I wonder if it is in the arena of fetching data from URLs, since app engine has its own library. I'll poke at it tomorrow.

In any case, I was very pleased with Google App Engine. Lets go over why:
  • Database is not a RDBMS. Some people might scream at this issue, but the benefits we get are wonderful. Expando seems really fun to use.
  • Built-in ORM. Sure, its not SQLAlchemy or the Django system, but its not that far different in approach and implementation.
  • Django Templates! If I'm not doing TAL and I'm doing XHTML/XML, then my choice is Django. Templates.
  • Cleanly documented. Clear and simple sentences with good examples that are working code, not doc or CLI tests.
  • Easy and intuitive. This part is critical. The framework is not in the way.
Update: Apparently Feedparser works with GAPE. Either something changed about GAPE (feedparser hasn't been updated in a while) or maybe I had a bug. Thanks to Alex UK and crchemist for pointing this out.

Thursday, April 24, 2008

What I want in a feed aggregator

The list is simple:
  1. One page that displays all the content. Maybe do some pagination, or hide descriptions and just show titles. Otherwise have tags, author, description, and link to original post.
  2. One page with a text area that accepts one feed per line.
  3. Include some sort of authentication.
Ways to get this done
Google App Engine handles #3 for me nicely and gives me free hosting. But feedparser doesn't play well with it and I'm not about to do that kind of debugging. Maybe I ought to try BeautifulSoup?

I'm tempted to try a pure Django system, since that could handle all three, but then I would have to pay for hosting. The same would go for Grok as well. I don't want to pay for hosting yet. Or maybe I ought to just pony up a few bucks a month anyhow...

Of course, I can always write my own simple wxPython client.

What to do... what to do...

Update: Never code on two hours sleep. I'm going with Google app Engine because I realized that when you import of feedparser you can't do this:
import feedparser.py

Wednesday, April 23, 2008

Suggest to me a feed aggregator!

I need a good RSS/Atom aggregator. I'm not having any luck. And this is giving me issues because I can't remember everyone's blogs I like to peruse. Any suggestions?

Tried Google Reader and hated it. Updates happened haphazardly and the interface was confusing. The one google product I can't stand.

Tried Rojo and it would be great but I can't seem to easily add my own @#$% RSS feeds. No dice - instead I have to search for everything. WTF?

Or should I perhaps write my own in Django? Or a local version in wxPython? I've worked with FeedParser a number of times and it is an easy API. So is Beautiful Soup for that matter.

Monday, April 21, 2008

Friday, April 18, 2008

Sick but Ubuntu

I was out of work for a couple days. Stuck at home, the first evening I grabbed an old machine and installed Ubuntu plus a surplus wireless card. This was my second Ubuntu install, the first suffering from a Microsoft wireless card. I was amazed by how fast and easy the install was to do. I kept the old XP on a seperate partition and got on the house network in 30 minutes.

Of course the basic installation doesn't have all the python tools I like to use. That took me another 30 minutes.

The only real downside I discovered was a lack of Mac's Textmate. I can do emacs, but Textmate is my preference. So I toyed a bit with gEdit and was pleased. Its also free!

One thing that Ubuntu beats out Mac and Windoze for is speed. Granted, this is a new install but I have to say I love the speed of the machine. And this machine is about 3.5 years old.

Good times indeed.

Monday, April 14, 2008

Oops... forgot RSS feed

Its not a big thing cause adding RSS feeds to Plone is painfully easy. Nevertheless, it is amusing how NASA Science didn't have RSS feeds listed on the front page on launch.

I think atom feeds are also planned.

Friday, April 11, 2008

Zed Shaw, YAML, Python Community

Zed Shaw posts about something I found annoying about a ways back when I was first exploring Python.

Like Zed, I too have run into the "YAML is not Python so don't use it, but wait someone else is using something YAML which is not Python so lets use that!"statement in the early stages of my Python use. Since Zed and I trade emails I should have warned him this would happen but the past few weeks have been hectic.

So this is an official apology to Zed for not giving the warning.

I wish I had given him the warning because that would have been awesome. Likely he would have ignored me, gone down the same path anyway, and then I would have a handy 'I told you so' to hold over his head.

Anyway, this is one of those few things I don't like about Python or the Python community.

Overview of NASA Science

I cooked something up for non-technical family members asking what I did for the NASA Science project. It gets a bit technical in places, but I'm a geek so that's just too bad.

Anyway, you can read it yourself at So what did I do on the website?

I'm also writing something that gets more technical here. I've got to be careful though cause The Man will make my life hard if I give out anything that can be construed as a security leak. So for example, if I give out the IP address of the site, that may or may not, under various conditions, be up for consideration as a security leak.

I kid you not.

Thursday, April 10, 2008

History of my most used shell commands

Courtesy of me -> flub:

history | awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -rn |head -n 20
107 svn
101 cd
101 ./bin/instance
58 ls
37 python
26 mate
16 grep
11 rm
8 ssh
4 ipython
4 easy_install
3 sqlite3
2 sudo
2 pwd
2 mkdir
2 history
2 django-admin.py
2 django-admin
2 ./bin/instance1
1 wget

Our NASA Plone project launches

http://nasascience.nasa.gov

This represents the work of about 20 people or so in many areas. Some details:
  • Plone, Zope, and Python application stack
  • Served by Apache and cached by Varnish
  • Section 508 compliant and meets usability requirements
  • Valid XHTML, so you can easily write stuff to hit it
  • Lots of custom views.
  • Lots of imported plone products
  • Several custom plone products
  • Hitting the mass media now.

Tuesday, April 8, 2008

This looks interesting...

http://code.google.com/appengine/

And Django comes as a default.

It is a very good time to be a Pythonista.

Friday, April 4, 2008

Issues with zope.formlib

Disclaimer: I am not an expert on Zope 3, much less Five, which lets you integrate Zope 3 technology into your Plone efforts. on the other hand, that arguably lets me speak as joe developer trying to meet a deadline, so I think things balance out.

I was tasked with extending a Five based package to include several extra new forms. This post details my efforts, what pleases me about zope.formlib and what annoys me.

Zope.formlib is a Zope 3 product that lets you create HTMl forms with handy bits of Python code. In general I like this approach, because if done right it means less work (thanks to DRY or Don't repeat yourself), and instead of just hand-coding the same form elements again and again - you utilize easily understood libraries that take advantage of inheritance and polymorphism.

Since we weren't storing any of the data I just extended a few existing interfaces, added a new view, a new response template written in a few lines of TAl, and tied it together with ZCMl. It was very straightforward and I had lots of fun. I had done a few bits before with zope.formlib and things were so easy I was delighted. Zope technology is like this, in that you can accomplish tasks quickly.

The basic form looked good, the form submission worked without a hitch, now I just needed to change some checkboxes to radio buttons. I didn't do it earlier because the documentation didn't have that described, but I figured it would essentially be really fundamental to the bool type in zope.schema (for you Django folks, Zope schemas can be likened to your models).

After many efforts at googling the answer I was nowhere. I dove into my 3rd edition copy of Web Component Development with Zope 3. No answer. I started checking the Zope 3 APIDOCs which has tons of information but nothing on turning a field/attribute into a radio button. I did introspection everywhere and might have stumbled across it eventually but a coworker made a suggestion that answered the problem.

So that was done. Now I wanted to encapsulate my forms in a set of macros that other products could easily use. I wanted to wrap SubPageForms in template macros containing form tags so whoever used these forms just needed to a macro include. Again the documentation was lacking, and for the release going as I write this, we created a temporary work around.

So there are described a couple major annoyances. Another one to list right now would be that anything besides a template change in Zope 3 based technology requires a server restart. Now pardon me for saying this, but I thought modern application servers did refresh.

Now keep in mind that when I wasn't trying to unearth the hidden mysteries of zope 3 and five or waiting for my server to restart yet again, I was incredibly productive. The frustrating thing was enough time was wasted with monkeying around zope.formlib that I could have written the TAl for the forms and then used old style Plone forms and been done much more quickly. Yes, I won't make the same mistake again, but I shouldn't have had to poke around as long as I did.

Alright, it may not be quite that bad. But it's bad enough. In my opinion, every HTMl formlib should have in its core documentation working examples of all the standard HTML input tag types. Text, textarea, checkboxes, radio buttons, and so forth.

For example, with the Django Newforms library, you have clear documentation of how to substitute one widget type for another. You are also clearly told that the forms library does not handle wrapping your input fields with form tags. Zope.formlib, combined with the rest of the Zope 3 API is much more powerful, but the Django documentation makes it much more accessible. Here is the Django newforms documentation if you are curious:

http://www.djangoproject.com/documentation/newforms

And here is the current zope.formlib documentation:

http://pypi.python.org/pypi/zope.formlib/3.4.0

The former is complete while the latter is verbose. That's an important distinction.

Just so it is known, I'm wouldn't mind documenting zope.formlib more and putting it somewhere useful on the web once someone gives me a good suggestion as to where to put it.

In any case soon after these issues were resolved I was essentially done and most of the remaining work is/was dealing with the normal sort of last minute unwritten requirements that makes life 'so much fun'. In fact, I rolled out another form in record time and using zope.formlib. I had fun! Just the learning experience was not.

Tuesday, April 1, 2008

I got a job at McDonalds!

I'm going to be a food champion. Which means preparing delicious culinary treats at the grill. Life is good!