Wednesday, November 10, 2004

Dream Last Night

Jen and I were out in the country somewhere. She had either bought a new car or was renting one. I was coerced into driving somewhere by myself. On the return trip, there was a bridge of some sort to be crossed. Traveling at a good clip, in the right lane, I drove onto it. Immediately, an exit appeared. I quickly changed to the lane to my left, having to cross the solid lines dividing the exit from the highway. Immediately another exit appeared. Frantically, I unsuccessfully attempted to change lanes again. At typical highway speeds, I barreled down (literally, the exit was situated at an incline from the bridge, which was still ascending) the exit for perhaps 100 meters. At this point, the road climbed sharply for several meters and then ceased. Somehow I managed to get out of the car unharmed, but it went over the edge and into the river below.

After inexplicably returning home, I had to explain how I had lost not only the car, but a backpack and a book which it contained. Each of these losses seemed more significant than the last.

Friday, October 15, 2004

Telnet and IMAP4 incompatible!

Recently on the IMAP4 implementors mailing list, the creator of the IMAP4 protocol had this to say:

You can not telnet to a modern IMAP or POP3 server, since TELNET does not have session encryption to protect the confidentiality of the password.

I guess modern servers only support connections over SSL (and telnet-ssl must not count as "TELNET"), and logging in with anything other than plaintext passwords is impossible by hand. Since I do this all the time, I thought it a bit odd. I suppose Mr. Crispin simply wanted to discourage insecure logins, or perhaps he just thinks it is difficult enough to respond to, say, a Cram-MD5 challenge that it may as well be considered impossible in casual conversation.

I've had this Python utility lying around for a while, this seems like a good opportunity to share it.


import sys
import hmac

def main(args=None):
if args is None:
args = sys.argv[1:]

response = hmac.HMAC(args[1], args[2].decode('base64')).hexdigest()
print (args[0] + ' ' + response).encode('base64')

if __name__ == '__main__':

I use it pretty frequently, since I work with IMAP4, SMTP, and POP3 servers and clients a lot, and it is generally too time consuming to use an actual client, and usually actual clients won't give me the information I want anyway. Here's an example of its usage (italics are things I send to the server, the rest is from the shell or received from the server):

exarkun@boson:~$ telnet 143
Connected to
Escape character is '^]'.
01 authenticate cram-md5
telnet> z

[1]+ Stopped telnet 143
exarkun@boson:~$ hmac username password bk04TXh3SURQSlJVNGlZbENRUkZWVmVMRXFEeEJWUzA=

exarkun@boson:~$ fg
telnet 143
01 OK Authentication successful
02 logout
* BYE Nice talking to you
02 OK LOGOUT successful
Connection closed by foreign host.

Perhaps telnet and IMAP4 are not as incompatible as one may otherwise have been led to believe...

Friday, October 8, 2004

In Theory, My Ass

I hate it when people say "in theory" when they really mean "I have no idea what I'm talking about". Cut it out. Don't say "in theory" unless there is an actual theory which predicts what you are about to say. If you don't know what you're talking about, say "I don't know what I'm talking about".

Thursday, September 30, 2004


<bush> terrorists r bad
<kerry> but their in afganistan lol
<bush> iraq is bad 2
<kerry> yea ok but u did a bad job what about the un this war is bad
<bush> the un is w/ me y do u hate america so much
<kerry> omg i dont this is the wrong war
<bush> dont say that bcoz the soldiers will be sad
<kerry> i was in vietnam
<bush> i am a strong leader
<kerry> u made big mistakes i would have done it better
<bush> its no mistake plus u always change ur mind
<kerry> do u shop at pottery barn
<bush> lol stfu
<kerry> jk but 4 real i can do a better job
<bush> this job is hard we must win i can do it
<kerry> man the facts say diff
<bush> look they attacked us we had 2 fight back
<kerry> dude iraq never attacked us
<bush> lol i know i mean the other guy
<kerry> neway plus nobody trusts u
<bush> there wrong i had to do it whatvr
<kerry> what about plutonium in korea
<bush> dumbass it was uranium
<kerry> oh nvm neway iran 2 u always want 2 do it urself
<bush> i didnt even put sanctions on iran
<kerry> ur girls are cute do they have boyfriends
<bush> ...
<kerry> jk neway u dont pay attention to reality i will if i am prez
<bush> i can pay attention but i just dont wanna change my mind like u its cool dont worry
<kerry> i dont change my mind im strong not wussy like w/ nukes there bad
<bush> i can handle nukes dont worry theirs lots of countries helping 2 plus star wars man its almost done
<kerry> north korea man ur a liar ill do it better
<bush> i GOT north korea mean chill shit
<kerry> ok i gotta go soon so last thing 4 the parents out theyre i will keep ur kids safe not kill 'em peace out
<bush> ok vrybody just remember i am strong and ill win the war go liberty wooo god bless u

Sunday, August 29, 2004

How I Spent My Summer Vacation

My two weeks off from work have come to an end. I didn't go anywhere, as I like to do on my vacations, due to various (uninteresting) concerns (which I will not relate here). I did do a bit of programming for fun, though.

  • Improved telnet implementation for Twisted

    Twisted has had a telnet protocol implementation forever. It is pretty lame, though. It makes you format outgoing IAC and such yourself, it doesn't handle \r\0 properly, it's got all kinds of abstraction leaks, and it has no unit tests. So I wrote a new one. And tests for it. Actually I wrote the tests, then the protocol. Hooray test driven development. The new code is still in my sandbox until the backwards compatibility issues get worked out.

  • A VT102 protocol implementation

    This provides a moderately high-level API for manipulating a vt102-compatible terminal. For example, the cursor can be positioned, scroll regions can be defined, the character set can be changed, etc. Together with the improved telnet implementation, this gives a server a convenient way to manipulate the display of a dumb client.

  • Twisted Manhole replacement

    Twisted has had a subpackage named "manhole" for a long time. A manhole in a Twisted application gives you an interactive Python prompt inside that application, as it runs. The state of the program can be inspected and modified using arbitrary Python code. This is quite powerful, and worked pretty well. One could connect to it via PB using the Gtk client, or via telnet. Unfortunately, the Gtk client is not conducive to shared debugging, and the telnet version was uncomfortable to use, since it lacked line editing features. Using the improved telnet implementation and the vt102 protocol, I wrote a replacement for the telnet part of this that supports line editing, line history, and syntax coloring. All using just a telnet client. It can also be hooked up to stdin, making a reactor-friendly readline-replacement. Since Twisted includes an SSH server, this can all be hooked up to that, too.

  • Gravitation Simulation

    I've been trying to write a particular game for a number of years now. It involves some amount of space travel, so of course I need a way to model bodies moving through space with some accuracy. This involves handling gravitational effects. I've written n-body gravitation simulations before, but they were all in crummy languages I'd rather not use any more, or dead slow. This time, I used the opportunity to learn a few things about Numarray, and implemented the simulation with no Python loops. It's probably still not fast enough to use, but it's a lot closer than any of my other Python attempts. (Also it's just a dumb geometric solution instead of a smart integral solution, oh well).

  • Python Oct Tree

    A simple little octree implementation. radix has been fiddling with 3d graphics lately. He wanted an octree to speed up visibility culling. So I wrote one. It was fun struggling to remember high school geometry. Since I'd just finished playing with numarray, I threw in some uses of it here, too. One thing that was frustrating was that I couldn't find a cross product function in numarray. I know I must be missing something. How could there not be a cross product function?

Friday, August 20, 2004

Things I didn't know Python could do

>>> class x:
... y = 0
>>> for x.y in range(5):
... print x.y,
0 1 2 3 4
>>> x = [None]
>>> for x[0] in range(5):
... print x,
[0] [1] [2] [3] [4]

Monday, August 2, 2004

Python Developers Considered Harmful

This morning I learned that the '@decorator' patch had been applied and would be distributed as part of Python 2.4a2. I fumed and wasted a good couple hours going back and forth about it with the denizens of #twisted. We go nowhere, except frustrated, disappointed, and disillusioned (You thought we were jaded before? You ain't seen nothing yet). Afterwards I went and tried to catch up on all the python-dev discussion of the feature. Not much to be found. Where was '@decorator' discussed? I see one post that mentions it offhand, and no followup. Numerous developers I spoke with felt the same way. So I stewed about it all day.

I took out the trash and went for a run at about 11pm tonight. When I got back I sat down and wrote this.

In brief, it defines a function "decorate" and a metaclass "DecoratableType" which may be used in this manner:

class Demo(object):
__metaclass__ = DecoratableType

def foo(x, y):
print 'Static method foo called with', x, y

This closely parallels the '@decorator' syntax, but requires no changes to the interpreter.

What are its advantages over the '@decorator' syntax?

1) It's pure-python.

2) It requires no syntax changes.

3) It works in Python 2.2, 2.3, and yes, 2.4a1, three times as many Python versions as will support the new syntax (once 2.4 is actually released ;).

4) It supports arbitrary expressions to specify decorators (The '@' patch only supports "dotted names". For example, "@foo" and "" will work, but "@foo or bar" will not)

What are its disadantages (and there sure are some)

1) It depends on source line numbers. It's possible that this information could be unavailable in certain environments (I can't think of any, but I'm sure there are some). The posted version also will not deal with intervening whitespace, comments, or decorate() calls well. This can be fixed without very much difficulty.

2) The posted version only works for classes that use the defined metaclass. It would be possible to remove this restriction, but only by hooking into some other mechanism for performing additional processing to the objects involved. This could be in the form of an import hook or a function which is manually invoked.

Hardly anyone generally comments on my blog posts. I hope that everyone who reads this and thinks a solution along the lines of the one described here will comment and let me know, and everyone who thinks the '@decorator' proposal is better will comment and tell me why I'm wrong.

Monday, July 5, 2004

Python implementation of Smalltalk's "become:"

Dash and Glyph were explaining what "orthogonal persistence" means to someone in #twisted today. It came up that Python doesn't have anything like Smalltalk's "become:". "become:" replaces all references to one object with a reference to a different object. Of course, I couldn't let this situation continue.

It has some problems: it uses hardcoded struct field offsets (I know how to fix this - gather empirical evidence about the proper offsets by twiddling carefully selected values and then examining raw memory - it's just a question of writing out all the right tests); it doesn't support all of Python's types, just enough to make a good example (this is easy to fix, just another question of investing time); it only runs on libc6 systems; and it reads and writes memory directly and manually manages refcounts in some places, so bugs generally lead to segfaults :)

All that said, it works pretty nicely on the types it supports: lists, tuples, dicts (that includes class, instance, and locals), and cells. Not a bad chunk of evil for half an afternoon, I'd say.

Saturday, June 12, 2004

Beautiful day, twelve mile bikeride (n/t)

LiveJournal won't let me leave this empty.

Tuesday, June 1, 2004

Cat vomit and subsequent things

Six of us, some with things packed, some packed as things, driving.
Conversation touching degrees of separation, Jen's background, past trips
and the like. The cats join in, with their plaintive mewls at the swaying
of the SUV. Light traffic, good time.

Nearly there, a gagging sound. Splash. Viscous lapping noises from the
rear now accompanying each small-town twist of the road. Splash, splash.
Stopping to see if Emma is alright. As much as any animal standing in a
pool of its own vomit can be, she is. We drive the last ten minutes to our
destination, set free the two prisoners, one of whom carries with her the
stench, but both of whom seem fine at last.

Night fall. Turkey burgers. Sleep.

In the hallway, slow drips from the ceiling. It is a gray day. Sheets
fall from the morning sky but, on the highway, from twelve foot puddles,
rise up again onto the windshield, to be shoved down by two madly flailing
wipers. Jen and I in borrowed raincoats.

Home Depot, identical to the others, only the faces have been changed.
CAT5 comes in pink, makes a festive belt, and is inexpensive when bought to
fit a waist instead of a corridor. The drapes are cut, we leave the store
behind. Traffic makes up for lost time, drags us to near a standstill. We
hit an old, independent book store. Shelves stuffed with fifty, eighty,
hundred year old books. _Gray's Anatomy_, complete with notes from a
previous owner. Piles of Daniel Steele near the back. I buy _Heart of
Darkness_ for a dollar. Unsated, on to a nationwide chain, with floors and
floors of the shiniest new publications. Near the registers, two rivers
spill from the ceiling. Coconut ice cream on the way home.

Once home, back out again to the grocery. Hectic pace as two cooks seek
out the items of two confusingly overlapping lists. I push the cart.

Home again. Later, on the dock, swaying in the wash of large boats, a
thick fog settling over the yard, maybe over the town, county, state.
Soft, permeable, yet simultaneously impenetrable. The dock and house are
cut off, separated from the world. Crabs climb onto the dock and splash
back into the water.

Curry and rice. A movie. Sleep.

The sun is out. The sky is bluer than the water, cloudless. Kayaks out
into the ocean, through the headwind, to an island. We walk around,
barefoot in the sand. Gulls feast on overturned crabs, armored legs
flailing in protest. We dig holes in the wet sand and quickly fill them in
to cover the animals we find. We paddle back.

Leftover curry. Reading in the sun. Salad, falafel, lemon turkey.

Scrabble, I lose badly. Sleep.

Sunday is even windier than Saturday. Whitecaps out in the channel. A
banana for breakfast, then in to catch a bus, cash or traveler's checks,
please, back to Boston. I forget _Heart of Darkness_ in the pocket of my
raincoat, read Gibson instead.

Monday, March 22, 2004

Two days of PyCon sprinting

Woo PyCon wooooooooooooooooooooooooooo!

Saturday and Sunday was spent hacking on several interesting things, including some cleanup of Twisted's treatment of addresses, a few minor boring error handling edge-case fixes, and most importantly IMAGINATION!

We finished porting the NewReality parser to Imagination's object relation system. We finished porting the NewReality action system to Imagination's object relation system. Sweet, sweet tests for both. Telnet wiring and imagination.tac code updated so a server can actually be started and logged into. radix, dash, glyph and I all worked on this to varying degrees. dash's plans are to start porting some of the old concepts to the new system. I think I am going to continue hacking on some tests and then try and get some movement actions working.

And we still have two full days of sprinting!

Friday, March 5, 2004

Twisted "chat server" in one expression

Tonight's messed up code:

(lambda r,p,b: (r.listenTCP(6665,(type('F',(p.Factory,object),{'protocol':(type('P',(b.LineReceiver,object),{'connectionMade':lambda s:s.factory.c.append(s),'lineReceived':lambda s,m:(s.factory.m(m),None)[1]})),'c':[],'m':lambda s,m:[c.sendLine(m)for c in s.c]}))()),*(lambda p,i:(i(p,'reactor'),i(p,'protocol'),i('twisted.protocols.','basic')))('twisted.internet.',lambda a,b:__import__(a+b,None,None,b)))

Wednesday, March 3, 2004

Good Night

Demonstration of success:

exarkun@boson:~$ telnet localhost 56197
Trying ::1...
Connected to localhost.
Escape character is '^]'.

Twisted 1.2.0
username: admin
password: *****
>>> import os
>>> os
<module 'os' from '/usr/lib/python2.3/os.pyc'>
>>> os.getpid()
>>> os
<module 'os' from '/usr/lib/python2.2/os.pyc'>
>>> os.getpid()
>>> Connection closed by foreign host.

Interpretation is left as an exercise for the reader. Suffice it to say that all the dirty hacks in the server melted away tonight, and the transition from supporting only listening ports to supporting connected protocol instances was almost effortless. The interpreter version change was a last minute afterthought that I was happily surprised to see actually work.

Sunday, February 29, 2004

Generating Python functions from a template

One thing I often use metaclasses for is removing boilerplate from class definitions. For example, if I have a group of functions which I want to all handle a particular error condition in the same way, I might write a metaclass like this:

def logAndContinue(f):
def doit(*a, **kw):
try: return f(*a, **kw)
except: log.err()
return doit

class ErrorLoggingType(type):
def __new__(cls, name, bases, attrs):
for (fname, func) in attrs.items():
attrs[fname] = logAndContinue(func)
return type.__new__(cls, name, bases, attrs)

class Errorful(object):
__metaclass__ = ErrorLoggingType

def error(self): 1/0

This is almost always up to the task, but last night I discovered a case where it didn't do all I wanted. The problem arose from the fact that, in a traceback, Python uses the code object's name, not the function object's name. In the above example, this would result in "doit" showing up in the traceback, instead of "error". So, I hacked up the following to get a code object with the name I wanted:

def formatArgs((args, varargs, kwargs, defaults)):
if defaults is None:
argString = ', '.join(args)
namespace = {}
argString = ', '.join(args[:-len(defaults)])
defaultArgs = zip(args[-len(defaults):], defaults)
dfltString = ['%s=%s' % (a, a) for (a, v) in defaultArgs]
defaultArgString = ', '.join(dfltString)
if argString:
argString += ', ' + defaultArgString
argString = defaultArgString
namespace = dict(defaultargs)
if varargs:
argString += ', *' + varargs
if kwargs:
argString += ', **' + kwargs
return argString, namespace

def _abcMethod(name, func):
s = "def %s(%s): raise NotImplementedError\n"
argspec, namespace = formatArgs(inspect.getargspec(func))
s = s % (name, argspec)
exec s in namespace
return namespace.pop(name)

As is hinted by the function "_abcMethod", I used this to generate functions for a generated Abstract Base Class version of a defined Interface. Since the only thing these functions do is raise an exception, it was somewhat more important than usual to have their names be correct.

formatArgs could be replaced by a call to new.code, but because new.code takes 14 arguments, I don't think the code would be any simpler ;)

Friday, February 20, 2004

Evil returns

After months of having no good ideas for bad things to write, PEAK's NOT_GIVEN inspired me to write this little tidbit:

import gc, sys, struct, opcode, inspect

class NotGiven:

def notGiven(param):
caller = sys._getframe(1)
for potentialFunction in gc.get_referrers(caller.f_code):
if getattr(potentialFunction, 'func_code', None) == caller.f_code:
elif getattr(getattr(potentialFunction, 'im_func', None), 'func_code', None) == caller.f_code:
potentialFunction = caller.im_func
raise Exception("You're insane.")

argspec = inspect.getargspec(potentialFunction)
bytes = caller.f_code.co_code
lasti = caller.f_lasti
varStart = bytes.rindex(chr(opcode.opmap['LOAD_FAST']), 0, lasti)
(varIndex,) = struct.unpack('H', bytes[varStart+1:lasti])

value = argspec[3][varIndex]
return value is param

def foo(x = NotGiven(), y = NotGiven(), z = NotGiven()):
print 'x given?', not notGiven(x)
print 'y given?', not notGiven(y)
print 'z given?', not notGiven(z)

if __name__ == '__main__':
for args in ('', 'x', 'y', 'z', 'xy', 'xz', 'yz', 'xyz'):
print 'Passing', ' '.join(args) or 'nothing'

Thanks to teratorn for bringing NOT_GIVEN to my attention. :)

Saturday, February 7, 2004

Triple Headed X

Didn't work. For five hours. Arg.

I've run dual head for several months now, and it is cool. I recently picked up an old monitor for free, so I figured I'd go for a third head. Digging through old boxes, I found what must have been a 5 year old 3dfx card, cleaned off the dust as best I could, and stuck it in a PCI slot. NOT SUCCESS! I could find no configuration to make this combination of hardware work. Not that this surprised me greatly. It would have been nice if XFree86 were advanced enough to handle this, but seeing as the video cards were manufactured at least 5 years apart and by different companies, I won't hold this failure against X.

That was all a week ago. I ordered a PCI TNT2 after the 3dfx failure. Yesterday it arrived, amidst much excitement and fanfare, and was immediately installed. It worked just as well as the 3dfx, though. So I am unhappy, both with NVidia for not providing better documentation and XFree86 for not working. If anyone does know how to get a two-card, triple headed X configuration working, you are commanded to respond to this with details.

Saturday, January 31, 2004

Today I installed the Hurd

It was not a success.

I took a clean drive and installed in a newly acquired 1U server. It had been hosting Twisted Matrix's website until recently, when it began to suffer from overheating. Case open, next to my other machines, it seems to be alright, but the fan is obscenely loud. Then I installed Debian/Linux, since there is no bootstrap installer for GNU/Hurd. Then I went ahead with GNU/Hurd.

The install wasn't that bad. I screwed up three times before I got it right, once because I didn't read carefully enough and rebooted improperly, and twice because, for whatever reason, mke2fs on Linux didn't generate a filesystem the Hurd could handle.

Getting the network up was a breeze, just one somewhat strange command. No DHCP support, apparently, but luckily my network could handle a random new IP showing up. Installing software was easy, of course. Just apt-get away.

My original goal was to see if any of Twisted's unit tests passed. I installed Python, but apparently the most recent version was 2.3a0, which had plenty of bugs (although none that I think Twisted's unit tests tickle, but who knows?). Before it even ran any, though, segfault and core dump. Woops.

Getting Twisted itself also brought up a minor bump. There is no random device on the Hurd, unless you configure one yourself. #hurd on told me there was no standard translator (a translator, as far as I can tell, takes a device file and makes it do something useful) for this, and I couldn't bring myself to download a random translator from the web, build, and install it. These are essentially kernel modules, after all. So I suffered the lack of ssh and did a pserver checkout of Twisted instead.

Since no unit tests passed, I figured I'd go back and figure out what the story with Python versions was. One apt-get install links later, another segfault. Unable to browse the web, run Twisted apps, or ssh, I began to lose interest. #hurd assured me that these segfaults were not a normal occurrence, so I might have simply misconfigured something. Who knows? Perhaps I'll try to examine the core dumps with gdb, but debian doesn't make it too simple to get debug versions of most things, and the criminally loud fan is amazingly annoying, so who knows if I'll actually follow through.

Friday, January 30, 2004


I had the longest talk about unions I've ever had with my mom tonight. She's an advocate for a collective bargaining unit. Someone I know [details omitted] has a pretty crummy job at [details omitted] I've been trying to convince her to get unionized.

I was amazed at the level of covert activities involved in getting a non-union shop unionized. I knew there was a lot of hostility toward unions, but I guess I didn't realize just how much.

Meetings are held in secret for as long as possible. Management generally tries to get a mole in as soon as they suspect any organization is going on. An important part of being successful is figuring out who in the group of employees likely to be working for management and excluding them from the process.

Law requires that once 30% of employees indicate a desire to organize, a vote must be taken to determine if collective bargaining will take place. The vote requires better than 50% support to pass, of course, so many unions wait until they have a higher level of support, often 70%, before they spill the secret that they've been trying to organize.

Once management knows a vote is pending, they generally do anything they can to convince employees to vote against it. Campaigns to sway employee opinion away from the union position are started. Tactics like mandatory "training" sessions, where anti-union videos are presented, are used. Some people get raises or promotions, to convince them they don't need a union. Other people get fired, to show the rest what will happen if they support organization. It is illegal to fire employees for attempting to unionize, but proving motivation is often difficult, and even if you can prove it, those laid off aren't always fully compensated. For example, eight people were fired from the last company my mom worked to organize. Even though it was found that they were fired illegally, they only received 70% back pay. They were not rehired, they weren't given any other benefits.

And those who are fired don't get to participate in the vote. If management succeeds in bribing (or otherwise convincing) or firing enough people, the vote fails and collective bargaining isn't allowed.

So anyway, I just wanted to capture some of what I learned tonight, depressing though it may be.

Monday, January 26, 2004

Two lessons learned

Premature synchronization is the root of all evil:

No matter how sure you are that part of an interface will never need to block, you are wrong. Even the simplest task will be expensive or slow for someone, at some point. Design interfaces to deal with asynchronous responses from the start. How did I learn this? I designed a mailbox framework and make the "select mailbox" operation synchronous, almost alone among the 30 or so other functions in the interface, because I was certain it was such a simple operation it would never need to block.

Loading data is expensive; loading unnecessary data is inefficient:

For the sake of simplicity, the "list all mailboxes" operation was implemented in terms of objects already defined by the interface. Specifically, it was defined to return a list of mailboxes, but only a few fields of each mailbox are ever actually used. How did I learn this? Tonight discovered that my implementation of this operation ran over ten thousand times faster if the implementation only returned the required data, instead of whole mailbox objects. Oops.