My SOPA boycott

I'm against SOPA and PIPA.

I believe that those bills will kill not just free speech, but also business within the USA. Innovation will wither. I'm also of the belief that those companies trying to get SOPA into place don't realize that no idea is new and if SOPA passes they'll be hammered with an increasing amount of takedowns and suits against them for anything they do. Litigation based on SOPA won't be as easily handled as the current status quo.

I've signed the petitions, I've posted on Twitter, Facebook, and Google. That isn't enough. I have to be willing to make a sacrifice. And in this case I'm going to make the sacrifice my vote.

My vote sacrifice is a boycott. It's directed at any politician, local or otherwise:

My Boycott:

• If you as a politician vote for SOPA/PIPA then you lose my vote. Regardless of whatever other opinions you have or party you belong to, you've lost me as a supporter.
• If SOPA/PIPA passes you can get my vote back by voting for what bill that destroys SOPA/PIPA is nominated.
• If SOPA/PIPA fails you can get my vote back by voting against whatever bills are resurrected to replace SOPA/PIPA.
• I will ignore party boundaries. I will vote against my normal grain simply to get you removed from office.

Why?

Like campaign finance reform, controls of freedom of speech often have unpredictable repercussions. 

The terrible thing about these bills is that their supporters are bi-partisan. While it's wonderful to see political opponents working together, in this case, it's for a terrible cause. 

So I'm going to cross political boundaries too. I'm going to say that as a registered Democrat I'm going to vote Republican if a Democrat candidate at any level votes for SOPA/PIP.

About Me

I don't use pirated software. I don't read, watch, or listen to pirated content. I purchase everything legally or use open source equivalents. I make a pretty decent salary and am pretty much in the direct center of what is called the 'middle-class'.

I'm also pro-business and a rather patriotic citizen of the United States. I believe in our nation and what it represents, and I know these bills are going to be a dagger in the heart of what our founding fathers gave us.

Finally, as I said, I'm a Democrat willing to vote Republican, Green, Libertarian, or whatever to make my point. That's my sacrifice. My vote and role in this nation that took my family in over 100 years ago is now in the hands of politicians.

What's yours?

Evaluating which package to use

In November of 2009 I wrote about which third-party Python Packages I'll use. Here is my modern take on it - much of it inspired by personal experience and the advice of peers and mentors:

Tag and release on PyPI

I really don't like pulling from tags on Github, BitBucket, or whatever. Or being told to pull from a specific commit. That works in early development, but it certainly doesn't fly in production.

I also get frustrated when people release on PyPI but then insist on hosting the release themselves. That is because invariably at some critical point in development when PyPI is up, the host provider is down.

A huge point of frustration is that I shouldn't have to leave the canonical source of Python package versions to hunt down what I should be using. I've seen too many beginning Python developers fall into the trap of using 3 year old packages because they didn't know they should be using trunk. I was guilty of doing it for a 6+ month old release in 2010, and for that I apologize and promise I won't do it again.

This also means your package needs to be pip installable. If you don't know how to do it, please read the The Hitchhiker’s Guide to Packaging.

Documentation

2011 is closing, which means your package needs to have Sphinx Documentation. And those Sphinx Docs should be on Read the Docs. Read the Docs is great because it doesn't just host the rendered HTML, it also lets you easily push to it from a DVCS push - and implements nice search and handy PDFs too.

Yes, I know there is packages.python.org but I don't trust it. It doesn't have the easy push/deploy workflow of Read the Docs, which means often the docs are dated because it's yet another step for developers. Plus, the lack of search outside of Sphinx makes it hard to discover documentation.

The same goes for hosting docs yourself. In fact, that's usually worse because when someone goes on vacation and the docs go down... ARGH!

Please don't mention easy_install in your docs. We are nearly in 2012 and ought to be unified on our package installer, which is pip.

Tests

You should have them. Otherwise any update you put on PyPI puts the rest of us at risk. We can't be sure your updates to the project won't break our stuff. So please write some tests! If you add in coverage.py and some kind of lint checker, it can even be fun! It certainly does earn you bragging rights having a high coverage rating.

Code Quality

Are you using new-style classes or old-style classes? Do you follow PEP-8? Do you keep meta-classes to the absolute minimum? Is the code on an available DVCS so others can fork and contribute? These are things that weigh in my judgement, and certainly the judgement of others.

Redux: Python Things I Don't Like

Back in May of 2009, I wrote about Eight Things I don't like about Python. It was my attempt to come up with things I don't like about my programming language of choice. Consider this my update of that post.

1. Division sucks in Python

In Python 3 this is fixed so that 2 / 3 = 0.6666666666666666 but in Python 2.7.x you have 2 / 3 = 0. You can fix that in Python 2.7.x with doing a from __future__ import division before your division call. Can anyone tell me if a version of 2.7.x will natively support 2 / 3 = 0.6666666666666666 without that import?

Note: Chris Neugebauer pointed out that changing division in Python 2.7.x will break backwards compatibility. However that doesn't change that I don't like it in Python 2.7.x.

2. TKinter blows

Honestly, it doesn't really matter to me anymore. I either use command-line scripts or things delivered to the web. Also, thanks to Brett Cannon, I know if I need to make TKinter look good, I can use TK Themed Widgets right out of the standard library.

3. Lambdas make it easy to obfuscate code

I'm known for not liking lambdas in Python. These days, I do know of use cases for Lambdas, but those are far and few between. I might even try to turn that into a blog post this month - use cases for Lambdas in Python. Fortunately for me, these days I seem to work with people who mostly agree with me on this subject.

4. Sorting objects by attributes is annoying

This is still annoying for me. As I said, "... the snippet of code is trivial. Still, couldn't sorting objects by attributes or dictionaries by elements be made a bit easier? sort and sorted should have this built right in. I still have to look this up each and every time."

I've thought of proposing something easier as a PEP. Imagine that! Me submitting a PEP!

5. Regex should be a built-in function

Before I got to do Python full-time I was a go-to person with regular expressions. Languages without them were weak in my opinion. Since then (2006-ish) my skills have faded somewhat in regards to regular expressions. And you know what? It hasn't been a problem. Python's string functions are fast and useful, and when I really need regular expressions, I import the library and do some research. I'm considering this one closed.

6. Reload could be less annoying

Reload only works on modules. I want to be able to something like reload(my_module), reload(my_class), reload(my_function), or even reload(my_variable):

>>> from my_module import MyClass, my_function, my_variable
>>> mc = MyClass(my_variable)
>>> mc 
5
# I go change something in my_module.MyClass and save the file
>>> reload(MyClass) # reload just MyClass
>>> mc = MyClass(my_variable)
>>> mc 
10

My current fix is to use unittest as my shell as much as possible. And that is probably a good thing.

7. Help doesn't let me skip over the '__' methods

As I said way back when, "Python's introspection and documentation features makes me happy. And yet when I have to scroll past __and__, __or__, and __barf__ each time I type help(myobject), I get just a tiny bit cranky. I want help to accept an optional boolean that defaults to True. If you set it to False you skip anything with double underscores.

The See project is one solution to the issue. A different approach I've used is the Sphinx autodoc feature, but Sphinx is a lot of work and doesn't cover every contigency.

8. Not enough female Pythonistas

These days I know a lot of female Python developers. There is my own fiancee, Audrey Roy. Face-to-face I've met and talked to Christine Cheung, Jackie Kazil, Leah Culver, Katharine Jarmul, Katie Cunningham, Barbara Shaurette, Esther Nam, Sandy Strong, Sophia Viklund, Jessica Stanton Aurynn Shaw, Brenda Wallace, Jen Zajac, and many more I know I'm missing. And there are even more with whom I've had in-depth online conversations.

So why didn't I put a strike-through on this one? Because the numbers still aren't good enough. I know a lot of female Pythonistas, but how many do you know? And even if you know a decent number, what percentage of a meetup group you attend are women?

I can say that things are improving, but they could be better - for women or minorities. Find ways to pitch in, be it PyLadies events, PyStar workshops, or what have you.

One last note on this subject, I've heard some unsubstantiated statements that the .Net world has a higher female-to-male ration then the Open Source world. Are we going to take that kind of thing sitting down?

Github is my resume

I remember the first time I heard that statement - a couple years back Eric Florenzano said it to me on Twitter when I posted my resume publicly and asked for opinions. At the time I laughed at his statement, because it felt like naive arrogance to ditch the idea of a resume and 'traditional' social networking like Facebook and LinkedIn. How wrong I was...

Before I go any further, this isn't to say that education, job history, and references aren't important in getting jobs that utilize a lot of Python. They are important, but I think they go more towards shaping you as a person than getting a job. So if you want access to Python jobs (and possibly other open source languages), you need to be able to show working code. Why is this the case? I can thing of several reasons:

1. It is much harder to forge your style of code, comments, tests, and docs in a repo than it is to make false claims on a resume.
2. Development team managers don't take LinkedIn references seriously because how often we see them gamed.
3. Code gives us a body of work employers (including me) can use in order to help evaluate your skill and ability levels.

Let's summarize that into a bold statement:

Python employers want to review your code in a public repo.

That puts the pressure on you doesn't it? Now you've got to show working code. One extremely unethical way to do that is to copy/paste other people's code into your own repo and claim it as your own. The problem with that is real reviewers know good code doesn't just magically appear in gigantic chunks. Which I'll sum up with another statement:

Python employers are smart enough to read your commit log.

So as a beginner, what can you do? A lot of shops will want to see your code but if you put up your early code, doesn't that mean they'll see your ugly, mistake-ridden work? Yes they will - but if you keep at it with tutorial examples you are working, whatever pet project you cook up, or even submitting patches to various existing projects, they'll see how your code improves. I am much more inclined to hire a person able/willing to learn than a jaded expert who doesn't want to grow - which is why I always try to think like an eternal beginner. Which brings me to my third statement:

Python employers are willing to hire bright, hungry developers willing to learn.

Getting away from employment, let's talk a little about the Python development community. This community is a meritocracy with amazing foresight. Passion for code and/or natural talent is often recognized before skill is achieved - but only if you show the community you are learning. Get your code onto Github, or BitBucket, or SourceForge so it is seen, and keep at it! Try to commit every day and if that isn't possible, then once a week!

Because if you write code every day or every week, over time your code will get better, you'll also be able to demonstrate a consistent body of work, and your passion for software development will be obvious. Also, try to comment your code as much as possible.

One good trick is to put your ongoing notes in a repo. I do it myself at https://github.com/pydanny/pydanny-event-notes. My early notes are very, very different from my later notes. Often embarrassingly so, but to a Python employer I'm pretty certain they are a useful reference into just how I think.

Github, not LinkedIn

LinkedIn (and Facebook, Google Plus, et al) are a place to define your profile and nothing more. That profile should include a link to your code. Python employers will be looking your for links to your code, not for any sort of networking you do on those sites. Employers get annoyed by 'developers' who excessively network but have no links to code samples on Github or other similar sites. If you have no code to find then it means we can't see your work, your thought processes, or your passion.

One common technique you see by a lot of Python developers is posting quick links to their projects and efforts on Github using various social networks. You can and should do the same.

You make connections by showing you want to learn

Python HTTP Requests for Humans

Ever try to use Python's standard library for doing a POST? Or a GET, PUT, or DELETE? What about when you have to deal with HTTP Basic Auth?

In a word, ugh.

Let's face it, this is one part of Python that is really not for human consumption. While there are a million things you can do with things like urllib, urllib2, socket, urlparse, the fact of the matter is that implementing anything beyond urllib.urlopen() is a matter of diving into arcane APIs.

Sure, thanks to works like Doug Hellmann's Python Module of the Week and Michael Foord's documentation of urllib2 the problem isn't unsurmountable. Unfortunately, the eclectic mix of libraries and weird APIs means when you have to revisit your code in a few months your code feels like spaghetti.

Do you doubt me?

# This sample gleefully taken from https://gist.github.com/973705

import urllib2

gh_url = 'https://api.github.com'
gh_user= 'user'
gh_pass = 'pass'

req = urllib2.Request(gh_url)

password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_manager.add_password(None, gh_url, gh_user, gh_pass)

auth_manager = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_manager)

urllib2.install_opener(opener)

handler = urllib2.urlopen(req)

print handler.getcode()
print handler.headers.getheader('content-type')

# ------
# 200
# 'application/json'

Really?

This much code to make a simple HTTP GET request with some auth?!?

Really?!?

This is a debugging nightmare! Especially when you have to deal with complex service APIs provided by Paypal, Amazon, Google, Authorize.net, and a million other systems.

I bet I could earn a decent living by charging Pythonistas a buck each time they took the shortcut of doing HTTP actions in the shell via curl or wget.

Anyway, wouldn't it be great if we could just call a single function with the URL and auth data as parameters? And that the same dialogue would exist for GET, POST, PUT, DELETE or whatever? Wouldn't that be just plain wonderful? If only we could have that functionality in Python!!!

Fortunately for us, we do have that functionality courtesy of Kenneth Reitz's Requests library! Our verbose code sample above becomes the wonderfully terse and easy-to-memorize script as shown below:

# This sample joyfully taken from https://gist.github.com/973705

import requests

r = requests.get('https://api.github.com', auth=('user', 'pass'))

print r.status_code
print r.headers['content-type']

# ------
# 200
# 'application/json'

Want to do a post with data? Try this:

# This example cooked up by me!

import requests
post_data = {"amount":10000, "service":"writing blog posts"}

r = requests.post('http://example.com/api', post_data, auth=('user', 'pass'))

print r.status_code
print r.headers['content-type']

# ------
# 200
# 'application/json'

The Requests library is still young, but I've yet to run into any bugs or undocumented edge cases. The documentation is awesome, but you don't really need it at all. The library is intuitive, fun, and and there is clearly one way to do.

How NOT to interview Pythonistas

There are a lot of firms who complain that all the experienced Python people are taken, and wonder if they should choose a different toolset. On the flip side of the coin, I've heard a growing  number of developers/engineers complain about horrible hiring practices they encounter. So which is it?

I think it is a mix of both. Compared to the current need, there is a shortage of experienced Pythonistas. On the other hand, I've seen really stupid mistakes by otherwise professional firms, stupid mistakes which are NOT caused by recruiters and don't just cost the firm a possible hire, but hurts their reputation. This is because people will complain to each other on IRC and in users groups about how your firm hires people and all of a sudden your firm has a bad reputation.

So that this stops happening, here are three really bad moves I've seen by companies:

1. If cold calling, don't EVER ask technical questions.

Recently I keep hearing about this one and it even happened to me about 8 years ago.

A pythonista is doing something, maybe coding, driving, sleeping, or eating when they get a call. The pythonista answers and are asked if they are interested in working for Company X. The pythonista gives a positive answer. Then the interviewer asks if they could answer some technical questions.

At this point the interviewer has failed. In fact, they have failed hard.

Odds are that being on the spot, the Pythonista will agree. And then, without a day to prep themselves for doing an interview, they are answering questions. There are no metrics for this one but I bet 90% of developers will fail questions they normally could answer in a heartbeat. Afterwards, the developer/engineer will kick themselves because they knew the answer, but because they were flustered they got things wrong.

And then, to really seal the deal, because the developer/engineer has failed the interview, the interviewer will inform them that they are not the sort of material that Company X wants. So not only did the pythonista mess up easy questions, now the lack of respect for their skills and person has been made abundantly clear.

The interviewer is completely at fault here.

The real problem for the interviewer is that if that potential hire that they just rejected talks about it, experienced developers/engineers will hear about it and it will be a unspoken black mark against their firm.

Lesson learned: What the interviewer should have done is email first, or if they called, ask if they could schedule a technical interview, either in person or on the phone. There is no exception to this system.

2. Make it clear that an interview is an interview.

Imagine there is a company you respect and admire. You meet the founders or the senior technical lead at a social event and they invite you to visit their firm. You arrive at the office expecting a tour and instead get handed to the technical staff for a challenging interview. Unprepared you don't do so well, and unsurprisingly the company doesn't hire you.

This is so full of wrong on the part of the hiring firm I don't know where to begin.

Lesson learned: If you are bringing someone into the office for an interview, make it abundantly clear you are interviewing the prospect. Say it in person and confirm it in email.

3. Make crazy requests in the technical interview

A few years ago a very capable friend was asked to provide a list of Fibonacci numbers, but he wasn't allowed to use a function and need to use a database. When he solved that one his effort was then criticized for 15 minutes by four developers. Then he was then asked to do it again, only this time in ECMA script.

Well before node.js existed, someone else I know was asked to use browser JavaScript to write a multithreaded HTTP server. When my friend asked "why would you ever do such a thing?", he was told not to ask that question but to solve the issued problem.

In all these cases the interviewee left annoyed, if not angry. Years have gone by and they still complain about these firms.

Lesson learned: For the hard questions, make them meaningful

Stupid Template Languages

For years I've been absolutely certain that I really prefer stupid template languages any time I'm generating HTML. The less the template language can do the better. Since I spend most of my time coding in Python you might assume this applies just to Python, but I think it also applies to anything where you have the power to readily mix HTML generation and code.

The biggest annoyance I have with smart template languages (Mako, Genshi, Jinja2, PHP, Perl, ColdFusion, etc) is that you have the capability to mix core business logic with your end views, hence violating the rules of Model-View-Controller architecture. While the web can be hard to match to MVC, in general you aren't supposed to do that sort of thing. I've made the mistake of putting core logic in the wrong places in the past, but I'm proud to say I've gotten good at avoiding that particular mistake.

I don't work in a vacuum.

I often work on projects crafted by others, some who decided for arcane/brilliant/idiotic reasons to mix the kernel of their applications in template function/macros. This is only possible in Smart Template Languages! If they were using a Stupid Template Language they would have been forced put their kernel code in a Python file where it applies, not in a template that was supposed to just render HTML or XML or plain text.

What it comes down to is that Smart Template Languages designers assume that developers are smart enough to avoid making this mistake. Stupid Template Languages designers assume that developers generally lack the discipline to avoid creating horrific atrocities that because of unnecessary complexity have a bus factor of 1.

So what is a Smart Template Language?

In my own vernacular, template languages that let you write functions/macros are what I call a Smart Template Language. Some of them are brilliantly executed, the example of Jinja2 comes to mind, but invariably I suffer through abuse of its Macro control structure as implemented by others.

Misery Cubed a.k.a. Genius Template Languages

Next comes Genius Template Languages, which take things a step further. These template languages allow you to not only define functions/macros, but also let you embed unrestricted Python (or Java or Ruby or whatever) in the template. This 'feature' lets you code your entire application in the templates!  In the Python world what comes to mind is Mako and Genshi, but I'm sure there are many other tools with this 'capability'.

I like Stupid Template Languages!

Stupid Template Languages don't let you define functions/macros. They don't let you embed Python code. They barely let you define variables and often have simplistic control architectures.

For Django efforts, which is about 70% of my work, I like the Django Template Language (DTL). Since it is used by a huge community, there are a ton of useful apps which have it as a dependency. Switching away from it would mean cutting myself off from a large ecosphere of tools I can use to not reinvent the wheel.

Back in my Zope/Plone days I really, really enjoyed the Template Attribute Language (TAL) because it was stupid too. If I needed an XML generation template language and could import it easily I might consider using it again, or perhaps Chameleon, which is a new, improved version . The downside is that they come paired with another tool paired with it, METAL, which gave it macros. My own experience with METAL is that it was all too easy to do what we developers do with Smart Template Languages.

But DTL and TAL are slow!

So what?

If you want to boost your performance, first try caching. There are a ton of tools you can use, with Varnish being one I keep seeing in action. Read the docs on your favorite web framework's caching engine and apply what you learn. And Djangonauts should read up on Mike Malone as much as possible.

If after all that the site still delivers slow content and it appears to be a template language issue, then identify the bottleneck content and consider alternatives for that one portion. My favorite response is a bit of AJAX. Use your framework to render the content as JSON and have JavaScript parse it into legible content, a task which JQuery makes trivial.

I hate Mac & Cheese

I'm not a picky eater. I'm willing to eat every meat that is legal in the United States and wouldn't mind traveling to places where the laws are different. I have this strange desire to try pickled insects in a third world nation. I often eat things I don't like, such as eggplant.

However, the truth is that I really loathe mac & cheese. I hate cooking it, I hate eating it, I hate cleaning up after it. I've thrown away pots that have been used to cook it. I claim to be different because I despise this American staple, and silly as you may think it may be it is my preference not to have anything to do with cheesy pasta. Heck, it ain't even healthy.

Note: I sometimes like some cheese on top of tomato based sauces or chili that has pasta mixed into it. This is not the same thing as mac & cheese.

Yet the complication I face is that I have dear friends and family who think that their prize-winning homemade mac & cheese made from imported unpasteurized cheese will be something that I love. That I'll change and they'll get the chance to smugly say that it was the quality of what I've eaten that discouraged me.

That won't happen.

I've been forced to try everything from Kraft mac & cheese to gourmet cheesy pasta prepared by chefs I otherwise respect. I've had it dumped on my plate on picnics and in fancy restaurants. I've had the people proudly insist I try their recipe all while making it clear that they would take it as a personal insult if I didn't try it and would also take it as a personal insult if I didn't like it.

And you know what?

I hated it. Its vile. Its disgusting.

Please stop insisting that I try your concoction. You aren't going to change me.

Conferences are a double edged sword

This post isn't for developers, its for managers.

In the Python world once a year there is Pycon. In other languages, tools, and frameworks they all have that one big conference where all the top people go. There are classes, tutorials, scheduled talks, open spaces, keynotes, and tons of networking. All your developers will want to go, and the ones that do will rave about the time they had. So how do you as a manager handle a big conference?

Conferences are a double edged sword. If handled well they will be a huge boon to you and your organization. Handled poorly and they can be devastating to you, your projects, your team, and your organization. Knowing which way to swing the 'conference sword' in such a way that you don't get cut is a handy tool to have in your pocket.

First, remember that at the big conference there will be lots of networking. Make sure your developers are really happy with you before they go. Also have your developers go with business cards. Ask them to bring back business cards. Remind them of whatever referral policy your company might have. Why pay for a headhunter when your own staff can do the hiring for you?

Second, remember that people will be scoping out your staff for potential hire. Don't anger your developers immediately before, during, or immediately after a conference. Keep deadlines as far away as possible from the conference. Don't begrudge them leaving and task them with petty work. Because what would have been training plus an energizing break for your staff will now be considered job fair by your staff.

Third, try to cover even a portion of the costs of the conference. If your organization doesn't officially do conferences, then offer to pay for their time during the sprints just so long as that open source work even tangentially affects what they do back at the office. That way they don't use up so much leave and odds are they'll have lots of smart comrades around to help them through some of the tricky parts.

Fourth, when the developers come back, ask them what they learned. Ask if they can incorporate these lessons into current or future tasks. They'll gush about twenty new things and most of them will be inapplicable. But there will be one or two things that are a perfect match and you'll both be grinning at each other in regards to challenges now effortlessly overcome or new services you can offer.

Fifth, try to go yourself. In fact, try and organize the whole trip. If you are close enough, rent vehicles and drive down. Set up the hotel rooms. You'll be able to network with other manager types and your staff will rave about you. Hiring will become that much easier.

So lets look at two ways to handle a big conference like Pycon:

The Wrong Way

• Block any attempts the developers make in getting reimbursement for the trip.
• Schedule deadlines around the conference. If anything goes wrong, declare that this is why people should not go.
• Before the conference, grumble about developers going away and then when they come back grumble about time lost.
• Developer time lost at the conference should not be compounded by talking about the conference when they come back.
• Act surprised when your team comes back angry and at low morale levels. Act more surprised when they leave for positions elsewhere.

The Right Way

• Get business cards and fliers for your staff to bring to the conference.
• In the meeting before the conference, go over the company referral system for new hires. Also offer beer or a meal for new hires.
• At least get the developers paid hours for the sprints. Beg and borrow from your own manager to try and cover other parts of the conference. Act surprised with the gratefulness of your staff.
• Afterwards, dedicate a staff meeting to things learned at the conference.
• Go yourself and hang out in the vendor room. Hand out cheap but goofy swag. Bring thousands of business cards and hand them out. Visit a few talks. Invite developers from other companies to meals. Boast about your staff and company. Be amazed by the new hires you pick up.

There was a day...

... just a few years back I got in trouble for getting trac/svn for a critical project I was leading. I had myself, two developers, a project manager, business analyst, and a manager best described as the quintessential pointy haired boss. Our company thought at that point that version control and web based ticketing was crazy, edgy, dangerous stuff. Nope, tickets were to be controlled by sharing Excel spreadsheets and zip was how you handled version control.

I'm glad those days are gone.

Sys Admins: What your Developers want you to know

This is my response to Katie Cunningham's post telling Developers what System Administrators want you to know.

Of course, this is mostly for shops with large groups such as our own. But some of this applies anywhere.

Publish your damned system specifications

If rackspace, webfaction, and the rest of the web hosting world does it for $10/month, then why can't well-paid system administrators do it?

Alas, I've yet to go to a shop where I could find on a wiki page or an otherwise easily accessible document an exact specification of the target production environment. Invariably you have to ask in person, go to another department, ask managers, or go searching through word documents. Word documents?!?

Unnacceptable.

As a developer I should be able to pull up for each server a consistently formatted web page that lists the operating system, operating system specifications, patches, database(s), compilers, shell, disk space, languages, hosted applications, and anything else you as a system administrator can think of adding. If your organization doesn't support a wiki or anything like that, then consider sending out a weekly email. Something as simple as this:

• Production01 (prod.mycompany.com): RHEL Linux, Python 2.4.4, 2.6.1, Java 1.6, Ports Open: 80, 443

• Staging02 (staging.mycompany.com): RHEL Linux, Python 2.4.3, 2.6.3, Java 1.6, Ports Open: 80

Look - discrepancies between servers! What a surprise! Glad this is documented!

Based off of this we can build our development environments to match production environments. The QA group can also build their staging environments to match as well. Also, we might be able to better exploit a service you are providing for us rather than writing something from scratch, which means we end up saving time, energy, and money. Everybody wins!

Communication is king

A system administrator needs to be accessible via phone. It just kills me that in some shops their phone numbers are only accessible via their boss. So if you have a problem and that boss is not around the rest of us are plain screwed. Can't our boss have the system administration contact information? Or maybe even us lowly developers?

Also, squirting off emails that you are starting/finishing things is really useful. Or send an email every 30-60 minutes with status. Or jump on IRC/IM and let the very tense developer/QA/business teams know you are still working.

Stick to the schedule

When you have 15 people waiting on you, start the deployment on time. Besides the simple expense of keeping that many people on the clock, delaying a start on people who have already been there for 8 hours is a good way to earn developer/QA/management enmity. Then you'll wonder why they always blame you right off the bat.

Follow the deployment instructions

When it comes to deployment, the hard truth is that your job as a system administrator is to follow instructions. Don't improvise. If the developer instructions don't work (for example, changing to a non-existent directory), don't fix the problem right then and there by opening files and twiddling stuff. Instead, it's time to call this a failed deployment.

That might make you look bad in the short run. Developers might howl. Sometimes you just have to stand firm. A good way to address this issue is to ask to look at a project's deployment instructions before the deployment date. Which brings us to the next point...

Review the deployment instructions early

Smart developers run their deployment instructions early past a system administrator. Take a good hard look at what they are trying to do and let them know if they are doing anything wrong. Good developers will really appreciate what you are doing for them.

Clone the production environment daily

Its 2009. Shouldn't we have fresh staging servers every morning for continuous integration? Yes, I know that when you built server X it matched server Y, but that was 6 months ago. Things have changed on both servers. A smart system administrator can script this out, or so they always tell me...

Communication is king II

So the deployment had problems. Or maybe it didn't. Maybe you had to do a tiny tweak on the instructions to turn an understandable developer mistake into a shining success. Here is what you can do to help defend yourself when things go wrong and to provide developers with a window into what you are doing.

Share the bash session history by use of the Tee command, the logs, and everything else. Without us asking for it! Dump the data and make it easy for us to find. We'll appreciate it and so will you.

Turning away from CAPTCHA

When I first encountered CAPTCHA I thought it was a grand idea. My opinion has changed.

First of all I believed it allowed humans and machines to be differentiated on the web. Sadly, cracking CAPTCHA is done on a regular basis, and there are white papers on how to do it in lots of different languages (such as this one in Python). Bugs in the submission system or cheap human labor works as well. There are enough ongoing issues that most registration systems still include some sort of email system to help filter out the robots.

Second, CAPTCHA fails on accessibility. Yes, you can provide an audio alternative, but what if your users are blind AND deaf? Well, I've actually been told by accessibility experts that a non-CAPTCHA form should be provided for those people. Yes, when I said 'experts', I meant plural!

So where does that leave us for weeding out the humans from the computers?

Right now I'm a fan of logic based CAPTCHA alternatives. The idea is to provide simple questions that are relatively easy for humans to solve and hard for computers to answer. A good example would be, 'Today is Saturday. Yesterday was ___', and the idea is that you should have hundreds or thousands of questions. In fact, I came up with a Plone widget package called humanator to support this concept. There will be a Django version shortly.

There are some issues to overcome:

• We need to cook up a few thousand questions to make it a bit harder on the brute force people.
  
• Internationalization will require translators from many languages to support the projects.
• There is also the issue of the cultural context of the questions. Since this is supposed to be user friendly we don't want to ask any inappropriate questions. I can police English pretty well, but I'll have little control over what happens in other languages.
• Some people thing the logic method is weaker from a security perspective than CAPTCHA. Both arguably rely on a form of security through obfuscation, and I think with about the same amount of work both can be hacked. But a logic based system is easier to set up. ;)
  

I don't like Integrated Development Environments

I really don't like Integrated Development Environments (IDEs). I don't like code completion, class browsers, object inspectors, class heirarchy diagrams, source control management in whatever I am editing coding with. I find such tools arcane and frustrating.

Why?

Its because I want to be able to feel the design of a module. When I manually introspect things I feel like I am sifting through the sand of the module to see what it gives me. I feel it in my gut that this is how I learn a language and use it best of all. Python makes that very easy for you with its powerful introspection capabilities which you use on the shell. I find switching to the shell lets me separate the capabilities of the module from the code I are working on. Which for some reason I find a lot more comfortable and intuitive. Your own mileage may vary.

If I do need a class hierarchy diagram, I just write a fun little python script which generates some dot notation and run graphviz's dot or neato utility.

If I do need source control management, thats what the command line is for!

Also, there are times I have to go and do things on systems besides my own. I can't expect to have Eclipse or NetBeans there. Or if they are there by some weird chance, they won't be configured the way I want.

Keep in mind I do like code highlighting. So I guess that makes me a text editor fan. And now on to my favorites:

Textmates (Mac only) http://macromates.com/
Kendall Clark introduced me to this tool back in April/May of 2006. I was very quickly hooked. It was much better than the TextWrangler that was giving me grief. And also much less arcane than Emacs. It is the one piece of software I'll actually pay to buy to use on the Mac!

Emacs http://en.wikipedia.org/wiki/Emacs
Back in the 1990s I did Perl for a short time and was introduced to Emacs and Vi. Emacs clicked for me, because even its arcane commands worked better for me than Vi. These days my Emacs skills are not superb, but I can get by on any machine that has it installed. So that means any Mac or Linux machine I stumble across.

Textpad http://textpad.com/
So my dark secret is that until December 2006 I did much of my work in Windows. Yes, my first python work was all developed on Windows! Anyway, I had stumbled across Textpad during a Java job and kept using it across other languages and efforts. It was light, did code highlighting, and kept out of the way. Perfect! Well, maybe not since it crashed about once a day. Still, it was better for me than more sophisticated alternatives.

ColdFusion and deprecated code

Lets take a step back in time to 1999. At that point I had been doing a mix of Foxpro for DOS, Perl, and an obscure language called WebML (don't bother looking it up, it doesn't seem to exist anymore). I admit I liked Foxpro but recognized it was a coder's dead end. Perl gave me regular expressions which I liked but I was uncomfortable with everything else. WebML was okay but it was clear that it was too obscure. Around that time I even talked to some people doing something weird with Python and Zope in Fredericksburg, but that sounded even more obscure.

At my job we had someone throw together a ColdFusion 4.51 application. They needed some help. I took a look and quickly figured out the issues, made corrections, and realized this language was really simple. One thing I noticed while pouring over the built-in documentation of the CF IDE (this was back when I used IDEs) was that there was a list of deprecated functions. You were supposed to stop using parameterExists and use isDefined instead. I shrugged, followed the specification, and moved on.

Years went by. I did Java and ColdFusion. Both were annoying because of spaghetti code. Java was also annoying because of boilerplate and the static typing, ColdFusion was annoying because of the tags, the weird ecmascript implementation, and the insistence of developers on using the deprecated functions.

The most obvious example was the parameterExists vs isDefined issue. The former, paramExists accepts a variable, so you couldn't build dynamic code. Specifically you did something like parameterExists(my_variable) instead of like isDefined("my_variable"). This meant more cut-and-paste coding. Also, because it was a deprecated function, the owners of ColdFusion didn't care about the performance of the code. So instead of looping through an array of a hundred or so variables who existence you wanted to check using isDefined, people would still @#$%ing type out parameterExists a hundred times. Or more commonly, cut-and-paste from one code template to another, across applications. A good number of the parameterExists at the top of a CF page often have nothing to do with the purpose of the page and people would wonder why things ran so slow.

Sadly, this sort of behavior still happens today in the CF community. Worse yet, because in my experience the CF community tenure is more important than anything else, the worst offenders are often the people who have been using the tool the longest.

So lets go through a quick summary of my issues with ColdFusion and deprecation:

1. Deprecated language elements, even if deprecated over 5 releases, are... well... deprecated.
2. Deprecated language elements are likely the last to be examined in a release for performance or security reasons.
3. Deprecated language elements can go away.
   
4. The worst offenders of using deprecated language elements tends to be those who have used the language the longest.

Eight things I don't like about python

Jesse Noller threw down a challenge on twitter. Or rather he said something and I've purposefully taken it out of context and am considering it a challenge. His statement was that you should be able to to find at least five things you don't like about your language or tool of choice.

So why should you read what I have to say on the subject?

This one is easy. I'm not a luminary in the field of python. I'm just Joe Developer. I am the user base. If something annoys me then it could annoy others.

1. Division sucks in Python

This is fixed in Python 3 and right now in the python version I am using I can do from __future__ import division. Nevertheless this should have been fixed ages ago with the release of python 2.4.3 or earlier. Why python 2.4.3? Because it was on that release that I started doing professional python coding.

2. TKinter blows

I've done a tiny bit of TKinter coding. I stopped because it was too ugly. Python needs either an updated, prettier version of TKinter or it needs something in its place as part of core python.

3. Lambdas make it easy to obfuscate code

I remember when I thought Lambdas was overheated excrement. I changed my mind. I found myself obfuscating my code, or trying to stumble through someone else's code that was ridden with lambdas. I suppose they have their place, but it seems like 90% of the time they don't add anything besides reducing your line count by a small amount.

4. Sorting objects by attributes is annoying

Yes, the snippet of code is trivial. Still, couldn't sorting objects by attributes or dictionaries by elements be made a bit easier? sort and sorted should have this built right in. I still have to look this up each and every time.

5. Regex should be a built-in function

Actually I sit on the fence on this one. Sometimes I wish python was like Perl and Ruby in that you didn't need to call in a new module when you needed a regular expression. Other times I am grateful I don't have to wade through the inevitable obfuscated crap we coders all too easily generate with regular expressions.

6. Reload could be less annoying

I am a python coder. I love the shell. Except reload only works on modules. Bah! I want it to work on every object in the stack.

7. Help doesn't let me skip over the '__' methods

Python's introspection and documentation features makes me happy. And yet when I have to scroll past __and__, __or__, and __barf__ each time I type help(myobject), I get just a tiny bit cranky. I want help to accept an optional boolean that defaults to True. If you set it to False you skip anything with double underscores.

8. Not enough female Pythonistas

I'm lucky that I work with a lady pythonista. And I've got an internet friend who is also a lady pythonista.

And that is it.

What a damned shame.

Conclusion

I've just handed you eight things to think about. It was hard coming up with actual meaningful things, which proves that at heart I'm just a gushing Guido Van Rossum fan boy.

Update 2011/11/04

New commentary on this post is at Redux: Python Things I dont like

Finger method of judging graphic design

Katie Cunningham mentioned this in her blog post today. Since I was asked what that was meant by several people I decided to write it up so you can see one of my rants.

The method is simple:

• Put your design on a screen.
• Your hand goes on the screen with fingers horizontal. Ignore the thumb.
  
• Count how many fingers it takes from the top of the screen to content. If you run out of fingers on one hand that means your wonderful design is going to force users to scroll to content. Maybe not on your huge desktop monitor, but on the sort of desktop monitor or laptop that has become ubiquitous, absolutely!
• I have tiny hands for a guy. So for you people with big hands, try it with two or three fingers.
  

Now lets take a look at how the Python community matches up to my finger method.

• http://python.org is just one of my fingers! Python.org FTW!
• http://djangoproject.org is two of my fingers! Good job Django!
• http://plone.org has a new and improved site that is two to three of my fingers, depending on the location. Plone does good!
• http://www.turbogears.org jumps between three and four fingers. Acceptable.
• http://code.google.com is two and a half fingers to content on project pages. Definately workable!
• https://github.com gets it with three fingers.
• http://pinaxproject.com is over four fingers to get to content? Boo!
  
• http://code.pinaxproject.com makes up for the parent site with two fingers to get to content.

So the Python community does pretty well on average. So does Facebook, Twitter, and the other good social networking sites. Now think about the sites that never went anywhere and how my finger method applies.

How about some some abject failures from outside the python community?

• CSS Zen Garden is lovely. And yet the designs are often unworkable to anyone outside of a group of graphic designers looking to hand each other design awards. You get designs that take ten (10) fingers before you get to content. WTF?
• http://www.apple.com/mac fails at over four fingers.
  
• http://www.microsoft.com/windows/help-how-to.aspx goes over four fingers and includes my thumb.
• http://www.adobe.com is as bad as the stuff on CSS Zen Garden.
  

Is this scientific? Heck no. Yet it is a quick and cheap gauge as to why you rarely get repeat visits to your site or why they only visit your download pages.

What I would change in Plone: Generic Setup

At the Washington, DC 2008 Plone Conference I took Joel Burton's awesome two day tutorial on Plone Theming. I don't do design myself, but I have implemented them from time to time. My CSS skills are weak, but Plone gives us lots out of the box so I figured this would be a great way to ramp up on skinning. Also, from a work perspective it made sense, because many Plonistas seem to care more about the engineering/database side of things rather than on the front end.

Joel's class, to put it succinctly, was magnificent. If you want to learn Plone, take one of his classes. It is well worth the money.

We learned about reorganizing viewlets on the fly, making css changes, tab controls, content well manipulation and lots, lots more. Two days of knowledge that didn't just go in one ear and out the other, but really stuck.

I was one of the tech-heavy guys in the room, and the rest were graphic designers. At first the ZMI terrified them, and the two different types of views did not go over so easily. But thanks to Joel, by the end of the two days the designers were really enjoying what they could do with Plone. Everyone felt confident and ready to move forward. The positive energy in the room was energizing!

Then it came time to 'save' the site design we had been working on. In other words, export it so it could be reused. Or sold even! We fired off Generic Setup to export things, and then...

The positive energy diminished

I'm capable of searching through Generic Setup XML files to figure out what is needed in order to capture a Plone skin. I'm sure anyone bothering to read this blog entry is the same way. Yet graphic designer skills tend to end at Photoshop, sometimes at CSS and JavaScript, rarely at PHP. Asking them to go through a few thousand lines of XML is not the way to garner good will.

There was grumbling, and mumbling. Some that were happy until that moment left the class unhappy. Not because of Joel, but because of the export tool that Plone gives us.

When I brought this up with Joel after the class, he said he was aware of it but it was beyond his control. When I brought it up with others, I got a mix of apathy or rejection of the issue. I was told by several that the designer could hand it off to a developer if this was beyond them. Or that it would be too challenging to address. I mentioned it to Tres Seaver and I believe he said he would look into it.

In retrospect, I should have made a bigger stink about it.

In my opinion, Plone should be as developer friendly as possible. Generic Setup should have an 'Export Skin' function, available via the Plone Control Panel. Not the ZMI.

And I think that this should be part of core Plone. As soon as possible. Preferably Plone 3.3.

Why?

At least half the customers you will ever have care very little about the back end. They just want a pretty and functional site. Which is why graphic designers are often so important to them. For example, how many customers do you have who care more about the comps of the graphic designers than the engineering that publishes it?

By making Plone more accessible and friendly to graphic designers, it means they will stay excited about it. Which will attract more of the graphic designer friendly customers.

It gets better.

This export utility would empower graphic designers. They will be able to create Plone skins and publish them. They can make money doing this, advertising their craft. In fact, I predict dedicated on-line stores for this sort of thing will become popular, making the cottage industry for plone skins much more viable.

Which is great for us developers who lack decent design skills. If exporting skins is easy, then I can find (and purchase) them and modify to suit customer needs. And if I need one from scratch, I can rely on a larger crowd of graphic designers.

What I would change in Plone: Templates

These are my thoughts. My thoughts are not those of the luminaries in any community, but just your average Joe Developer. I formed these thoughts over Pycon 2009, and my experiences with Plone, Zope over 2 odd years and Django over the past 4-5 months. A lot of this is therefore inspired by my recent experiences with Django.

Item #1 - Plone/Zope needs Django Templates out of the box

You heard me.

I like the Template Attribute Language (TAL). TAL is stupid. Stupid in that it is very hard to encrypt my templates with business logic. I like this because I recognize I have no discipline when it comes to breaking the Model-View-Controller paradigm if a framework lets me. I also like TAL because it lets me create XML type documents very easily.

I like the Django Template Language (DTL). DTL is stupid. Stupid in that it is very hard to encrypt my templates with business logic. I like this because I recognize I have no discipline when it comes to breaking the Model-View-Controller paradigm if a framework lets me. I also like DTL because it lets me create non-XML type documents very easily.

I like both languages. Some people do not. Advocates of one template system style say very snarky things about the other template system style. Inevitably it comes down to that one is XML focused and the other is not. And from their point of view, they are completely right.

So I think that a version of DTL should be ported to the Plone/Zope world. And for Plone, it should be made readily available out of the box. This port should do all the default DTL filters and as many of the default tags as possible.

Which would I use? I would use either. More importantly, people who love XML based templates could use what they want and people who hate XML based templates could use what they want.

Everybody wins.

Item #2 - Plone TAL documentation needs a cheat sheet of the good functions

For Django, I can go to the DTL tag/filter page and find a handy reference on all the tags and filters available out of the box with Django.

Where is the same with Plone? The documentation has gotten really freaking good, but where is this sort of handy developer reference?

Yes, I have learned that you can use string methods on strings, and that there are various utility methods you can call if you know the name of the right portal object, but these are not in one easy-to-find location.

We Joe Developers need and love this sort of thing.

Changing over to .NET

After a lot of thought and meditation I have decided that it is time for a change. This change has been in the works for a long time. I want to belong to something monolithic and proprietary, and I think that .NET on Windows is the way to go.

The .NET platform empowers me with the ability to choose from a host of languages like C#, asp.NET, vb.NET, Boo, IronPython, IronRuby, and many more. And after long and careful consideration I have decided that my next language of choice will be C#.

C# is under the stalwart auspices of Microsoft itself, instead of some guy from Denmark. It has static typing and thanks to compilation most bugs are caught quickly. The curly braces clearly delineate code blocks, and the semi-colons show me when a statement ends. Partial classes will let me spread my object code across many files, and lambda expressions will let me compress complicated functionality into generic functions. Documentation is done via XML rather than the RestructuredText used across the Python community.

Of course, Visual Studio has a lot of visual elements. I am not sure what that means, but being Visual is obviously superior to the TextMates and EMACS I have used in the past. I can't afford Visual Studio yet, but if I don't buy any groceries for myself, wife, and son, I should be able to pay for my copy in only 2-4 months!

In summary this year looks very exciting.

Update: This was an April Fool's joke.

How to do well at the job

This isn't just for technical folks, it is for everyone. It is amazing how frequently we forget these things.

Don't be a jerk.

• Your co-workers are not medical interns. Treat them with respect, not scorn.
• Don't play unnecessary games of technical semantics.
• When someone comes to you with a question be happy they trust you enough to ask you the question. If they have trouble communicating their question, help them out.
• Trying to actively sabotage a boss/colleague/subordinate's job is a stupidly dangerous game to play.
• Handle the successes of others publicly. Handle the mistakes of others privately.
• Appreciate your system administrators and engineering staff. They only get notice when things blow up, and when the chips are down they are the people who will save your behind.
• Never turn down a project without asking for at least a day to think about it.

Be Seen.

• Everyone should be in IRC/Chat at all times they are working. Otherwise we think you are not.
• If you lead/manage a project you have to be seen. Telecommuting is acceptable one day a week. Two days is pushing it. Three days is too much. If you can't pull off a physical presence then don't lead projects.
• If you meet the customer, dress up.

Meet your deadlines early.

• You'll have time for the issues that always turn up.
• You'll have time to look into other things.

Go the extra mile when it is just a yard.

• A little bit of interpretation can keep customers really happy.
  
• Too many people try to follow requirements to the absolute letter, fearing that they'll get some scope creep if they don't.
• A tiny and shiny bit of UI you toss in is what the customer tends to notice to the exclusion of all the other work you did.

Stay out of the closet.

• Never get blocked into the coder's closet. If you do Python, some some time exploring Zope, Django, Twisted, SciPy, libxml, pygame, and other things. Same goes for Java, .Net, and the rest of the programming world.
• Follow the main technical blogs for your favorite tools. See where trends are going.