Thursday, August 18, 2005

Magical Concurrency Faeries or How I Learned To Stop Worrying and Love Deferreds

Every now and then, a recipe like this one pops up. Wouldn't it be wonderful to live in a world where you could close your eyes, tap your heels together, say "this API is asynchronous" three times and be rewarded with an asynchronous version of a synchronous function? I know my life would be a lot easier in that world.

We live in a world with significantly less magic than people sometimes make out. The programming languages we have available now simply cannot perform the synchronous-to-asynchronous transformation automatically for the general case. There are languages which provide interesting constructs which may be the beginnings of such a feature, but Python isn't one of them. What Python gives you in the way of "magic" asynchronous behavior is what we've had for much more than a decade: platform native, pre-emptive threads. While these can be put to some use, the programmer must understand the implications or the result is randomly buggy software.

If you haven't done much programming with threads, this consequence may not be obvious. Consider the case of a library which uses module-level storage to implement its public API. Now consider the author didn't plan to use it with threads and so didn't protect any access of this shared resource with locks. Now try and imagine what will happen if your program happens to call into this library from two different threads at about the same time. For bonus points, try to imagine how you might write a unit test which illuminated this bug in your software. An example of a library written in such a way is the Python standard library urllib module.

Asynchronous programming feels unnatural to many people. The tendency exists to want to write programs that read like they execute: top to bottom. Perhaps this tendency exists because synchronous programming is what most programmers are first taught, or perhaps it is because synchronous programming is simply a better fit for our brains; regardless of the reason behind the preference, it does not change the fact that asynchronous programs cannot be written as though they were synchronous with today's tools. If you think threads are hot, use threads and don't waste your time with tools like Twisted. If you think asynchronous programming presents serious benefits (as I do), embrace it all the way: it won't be easy (writing software rarely is), and you will often have to take existing synchronous software and rewrite it so as to present an asynchronous API. At these times, grit your teeth and get on with it. You'll be much happier in the long run than if you had tried to cobble something together using threads and end up spending a far greater amount of time tracking down non-deterministic bugs that only appear under load after days of runtime on someone else's computer and which disappear whenever you insert debug prints or run with a debugger enabled.

Oh yea, and as Itamar pointed out, the particular recipe I referenced above is, in addition to being conceptually flawed, implemented in a way which is just plain buggy. Even if you disagree with everything I said above, you shouldn't use it.

Monday, August 15, 2005

On 17 Hour Work Days

They're hard.

What did you think I was going to say?

Friday, August 12, 2005

Retrieving User Idle Time on Freenode

Recently Freenode deployed a code update that removed the "idle" field from the whois command's output. mwh pointed out that you can work around this by issuing whois username username.

Thursday, August 11, 2005

Simple Twisted Application for Fun <strikethrough>and Profit&lt/strikethrough>

I really enjoyed the series Firefly that aired a few years back. I've enjoyed it a few more times since. I'm really looking forward to the upcoming movie. So much so, in fact, that I even wanted to try to get into one of the pre-screenings. There's a website that announces dates and locations for pre-screenings. Being pretty lazy and also forgetful, I can't be bothered to actually check the website frequently enough to notice when new tickets are available before they are all sold out. Fortunately, being a programmer, I can simply bend a minion to my will. I whipped up this quickie to do the checking for me:

import rfc822, md5, sys

from twisted.internet import task, reactor
from twisted.mail import smtp
from twisted.python import log
from twisted.web import client

url = ""

MSG = """\
Subject: The upcoming film, Serenity, is awesome
Date: %(date)s
From: "A Respectable Internet Citizen" <>
Message-ID: %(msgid)s
Content-Type: text/html

Serenity hash changed! Go look!
<a href="%(url)s">%(url)s</a>

def mailinfo():
return {
'date': rfc822.formatdate(),
'msgid': smtp.messageid(),
'url': url}

lastDig = None

def main():
def got(page):
global lastDig
dig = md5.md5(page).hexdigest()
if dig != lastDig:
if lastDig is not None:
sys.argv[1], # SMTP host
sys.argv[2], # sender email address
sys.argv[3:], # recipient email addresses
MSG % mailinfo())
lastDig = dig

task.LoopingCall(main).start(60 * 30)

The took about 10 minutes to write and even worked on the 2nd try (the first try I forget recipient addresses had to be a list and gave it a string instead). Then it ran for 21 days before sending me an email a few days ago! Alas, I don't live in Great Britain, or even Europe. It has also, of course, sent out an email once each time one of the listed theaters sells out. And now they're all sold out. :( Given the short period of time remaining before the probable release date, it seems unlikely there will be any more pre-screenings.

Sunday, August 7, 2005

Twisted Conch Insults

Twisted Conch Insults is cool. I wrote the bulk of it almost a year ago, but lately I've been revisiting it with the intent of actually developing some useful software based on it. Since I've never written any major terminal-based apps before, I decided to start with a lot of tiny attempts to get a feel for how things should work.

One of the first things I tried was a typespeed clone:

insults-based typespeed clone

Next, I tried something vaguely nethackesque (yes, that's unicode):

insults-based maze game

Most recently, I started tackling the problem of developing a windowing library for insults (Itamar and I paired on most of this one):

Text UI

There are a couple others in my sandbox for those who are curious. I'm pretty happy with how easy these were. They varied in difficulty from about half an hour of work to about a day's work. Some of the APIs definitely need to be improved (like having a 2-dimensional buffer API for manipulating large areas of the screen, rather than just exposing fundamental terminal operations), but clearly a lot is possible with what already exists. I think I'm going to continue developing the windowing library for now, and then start another round of experiments based on that.