Tuesday, December 20, 2005

Mantissa 0.4.1 Released!

I bet you thought I was done! Well, this is the last one for tonight, I promise, and believe me, I saved the best for last. Mantissa 0.4.1 brings some really fantastic features.

First up is the Tabular Data Browser. This is some pretty awesome stuff. Basically it is not even worth trying to describe with words. Go signup for a free ClickChronicle account and see for yourself. After you have accumulated a few clicks, you'll be able to browse them and page around and so forth: the HTML interface which allows you to do this is the TDB Controller. It is an Athena Widget that lets one page through the results of an Axiom query. The columns are customizable. The actions are customizable. The skin is customizable. All you have to do to get one of these things in your Mantissa application is instantiate TabularDataView with the customizations you desire and drop it onto a page someplace. Bam.

So now you're really excited about using the TDB. You need to write a Mantissa application before you can take advantage of it, though. Fortunately, the other big improvement in this release is that writing a Mantissa application has gotten way easier. What you do now is write an IOffering plugin. It looks something like this:

from axiom import iaxiom, scheduler, userbase
from xmantissa import website, offering, provisioning
from clickchronicle import clickapp, publicpage, prods
import clickchronicle

chronicler = provisioning.BenefactorFactory(
name = u'clickchronicle',
description = u'An application with which to chronicle the clicks of you.',
benefactorClass = clickapp.ClickChronicleBenefactor)

clicks = provisioning.BenefactorFactory(
name = u'clickchronicle-clicks',
description = u'Add some clicks to the click limit',
benefactorClass = prods.ClickIncreaser,
dependencies = [chronicler])

plugin = offering.Offering(
name = u"ClickChronicle",

description = u"""
To-morrow, and to-morrow, and to-morrow,
Creeps in this petty pace from day to day,
To the last syllable of recorded time;
And all our yesterdays have lighted fools
The way to dusty death. Out, out, brief candle!
Life's but a walking shadow; a poor player,
That struts and frets his hour upon the stage,
And then is heard no more: it is a tale
Told by an idiot, full of sound and fury,
Signifying nothing.

siteRequirements = [
(iaxiom.IScheduler, scheduler.Scheduler),
(userbase.IRealm, userbase.LoginSystem),
(None, website.WebSite)],

appPowerups = [

benefactorFactories = [chronicler, clicks])

Actually, it might look exactly like this. This is the actual ClickChronicle IOffering plugin. Note that it is essentially all content: the boilerplate consists almost exclusively of importing modules and appeasing the Gods of Python syntax. This plugin assembles all the pieces which make up ClickChronicle into a coherent unit which can be deployed on a Mantissa server.

siteRequirements enumerate which must be available from the "site store", which contains such things as the credentials database and the webserver configuration. appPowerups defines which Powerups are required by the "app store", at least one of which is created for each application (ie, ClickChronicle); these can (and in this case, do) define the look of the public page for this application, as well as any other behavior which is not clearly attributable to a particular user.

benefactorFactories, predictably, defines a list of factories which create benefactor objects. A benefactor in Mantissa is an Item responsible for endowing a user with new functionality. In the case of ClickChronicle, there are two kinds of benefactors: one endows a user with the application itself, letting them record clicks and later browse them; the other raises the limit on the number of clicks the user is allowed to store. Note also that the latter depends on the former, indicating that benefactors produced by clicks cannot endow a user who has not first been endowed by a benefactor produced by chronicler.

Confused yet? If not, awesome. Go write some kick-ass Mantissa application. If so, don't worry. I'm going to be writing some more in-depth documentation about the Offering system in the days to come.

Axiom 0.4.0 released!

In an effort to overwhelm you (that's right, you buddy), Divmod is happy to announce the release of version 0.4.0 of Axiom, an object-relational database built on the excellent SQLite. Since the last release, Axiom has seen numerous improvements. These include, but are not limited to:

  • A floating point attribute type (use with caution)

  • Support for associating multiple sets of credentials with a single account.

  • Command-line support for interrogating a database regarding its running state as well as support for stopping a running database.

  • Command-line support for extricating an entire user account (credentials, application data, everything) from a site database. Additionally, support for the reverse operation. Taken together these represent an effective user-migration tool.

  • Improved support for certain SQL operations, including sorting by multiple columns, support for DISTINCT queries, and a bug-fix in the sum() aggregate.

Download Axiom 0.4.0 right this very moment.

Divmod Athena Released!

Okay it is not really called Divmod Athena but I am excited so cut me some slack. It is with no small amount of personal satisfaction that I bring you news of the latest and greatest (thus far) Nevow release. Nevow 0.7.0 brings several minor bugfixes and feature enhancements over 0.6.0, but by far the most outstanding change is the work that has been done on nevow.athena.

Athena has two new related features that are really slick. The first is LiveFragment. As Fragments allowed you to define rendering behavior in units smaller than a page, and then combine multiple Fragments to make up a complete page, LiveFragments allow you to define behavior for Client-to-Server and Server-to-Client events which can then be mixed and matched to form the complete page. The second is Nevow.Athena.Widget, a JavaScript class, subclasses of which can be used to track client-side state, handle input events, handle events from the server, and manipulate the portion of the DOM (ie, the UI) for which they are responsible.

I could jabber on for hours at this point, but I'll save that for later. Go look at the examples I linked above, or try them out. Then download Nevow 0.7.0 and write some kick-ass software. I command it.

Divmod Epsilon 0.4.0 released

Dear Intarweb,

Today Divmod released its Epsilon package. Divmod Epsilon is a Python package with some useful stuff. It has a library for doing time. It also has a module for running jobs. And it has some other things like a metaclass for defining modal types and also a Version class.

Epsilon 0.4.0 has two new features. The first new feature is for creating folders with files in them. It uses a string to define the folders and what's in them.

The second new feature is a library for patching old versions of third-party libraries so they have new features or bugfixes. This feature is called hotfix. It does helpful stuff like check versions and only apply the fixes if the library is too old.

I hope you have fun with Epsilon 0.4.0!



Monday, December 19, 2005

Daddy, What Is A Coroutine?

Some people have gotten the idea that Python 2.5 will have coroutines.

Let me just be blunt. Python 2.5 will not have coroutines. PEP 342 even says so. Coroutines can be switched between regardless of their position in the call stack. Python's generators can only switch to the frame directly above them in the stack. The PEP suggests that it will allow coroutines to be implemented using a trampoline (with which, by the way, Guido doesn't even think anyone important will bother). Well, this may be true in the strictest sense - PEP 342 won't prevent you from implementing such a trampoline, but it is already possible to do so.

Don't misinterpret: PEP 342 corrects a long-standing mistake in Python's generators (yield definitely makes sense as an expression). Insofar as it corrects a syntactic inequity in the language, this is a fine PEP. It just has nothing to do with coroutines.

Thursday, December 15, 2005

My Superpower

I have a superpower. Without forethought or planning, I can arrive at a bus station at precisely the moment a bus is preparing to leave, so that I board it and it departs without any delay.

I may have other superpowers: I'm not certain. Among the superpowers I do not possess are:

  • Speed greater than that of a speeding bullet; power greater than that of a locomotive; prowess to leap such that tall buildings are negotiated in one or fewer bounds (all these tested in the proximity of a yellow star; research under the influence of other celestial bodies is ongoing).

  • I am not physically incapable of being drunk.

  • Probably something to do with lemurs. Reports here are sketchy.

That sums up the current state of my superpowers. If you were looking for technical content on Axiom or Mantissa, stay tuned: there is some on the way.

Thursday, November 17, 2005

Axiom Powerups

Q: When I run axiomatic -d my.axiom start, a web server starts up. How does that work?

A: The web server is started and stopped by an instance of an Axiom Item subclass, xmantissa.website.WebSite. WebSite powers up the database into which it is installed. Power up is the term used for a particular manner in which one Item can be used to enhance the behavior of another Item. It means that WebSite is calling powerUp on the site store1, passing itself as the powerup Item and twisted.application.service.IService as the interface. The start subcommand of axiomatic kicks off the standard Twisted service startup events: an Axiom database behaves as an twisted.application.service.IServiceCollection, with each IService powerup treated as a child. In this way, things like WebSite are given startup notification (as well as shutdown notification, when the database is about to be closed).

Q:Wait, what? You make no sense.

A: Okay then, here's some example code, perhaps it will make more sense:

from zope.interface import Interface, Attribute
from axiom import store, item, attributes

class IEngineEnhancement(Interface):
"An improvement to a ship's engine."
speed = Attribute("How much faster this makes a ship go")

class IShieldEnhancement(Interface):
"An improvement to a ship's shields."
force = Attribute("How much stronger this makes a ship")

class Spaceship(item.Item):
typeName = 'spaceship'
schemaVersion = 1

score = attributes.integer(default=0, doc="how well the player is going")

def getMaxSpeed(self):
return sum(enginePowerup.speed for enginePowerup in self.powerupsFor(IEngineEnhancement))

def getMaxShields(self):
return sum(shieldPowerup.force for shieldPowerup in self.powerupsFor(IShieldEnhancement))

class HyperDrive(item.Item):
typeName = 'hyperdrive'
schemaVersion = 1

speed = attributes.integer(default=100)

# Here is the powering up part
def installOn(self, ship):
ship.powerUp(self, IEngineEnhancement)

class SuperShield(item.Item):
typeName = 'supershield'
schemaVersion = 1

force = attributes.integer(default=50)

# And here is a little more.
def installOn(self, ship):
ship.powerUp(self, IShieldEnhancement)

def main():
st = store.Store()
ship = Spaceship(store=st)
driveUp = HyperDrive(store=st, speed=35)
shieldUp = SuperShield(store=st, force=1000)
extraDriveUp = HyperDrive(store=st, speed=20)

for powerUp in driveUp, shieldUp, extraDriveUp:

print "Your ship's maximum speed is %d." % (ship.getMaxSpeed(),)
print "It has a maximum shield level of %d." % (ship.getMaxShields(),)

if __name__ == '__main__':

Q: Hey! Isn't this just a bridge table with a funky API?

A: Yes. Yes it is. We find the pattern so useful, we added special support for it.

Site Store is the name of the Axiom Store which is at the "top" of everything: it is the database in the directory you pass to "axiomatic -d", it contains login data and other site-wide state. It is conceptually distinct from user stores and application stores and serves a different purpose from each of them.

Wednesday, November 16, 2005

Axiom Queries

Yesterday, I demonstrated how to write an axiomatic plugin. That was pretty cool, but to actually do anything interesting, you probably need to know how to define Axiom Items and perform queries. Without any ado, here's an Item definition for a popular kind of thing, a tag:

from axiom import item, attributes

class Tag(item.Item):
typeName = 'tag'
schemaVersion = 1

name = attributes.text(doc="""
A short, descriptive string giving this tag some meaning. eg, "fruit" or "vacation pictures".
""", allowNone=False)

item = attributes.reference(doc="""
The object which has been tagged.
""", allowNone=False)

There isn't anything too complex going on here:

  • axiom.item.Item is the base class for all database-stored objects. axiom.attributes defines a few classes that let you declare the schema of your Item: text for unicode strings, bytes for binary strings, boolean, integer, timestamp, and path for the obvious things. reference allows you to refer to any other Item (of any type) in the database. You can also define new attribute types, if your application demands it.

  • typeName associates the Python class with a database type. This lets you move your class between modules or even change its name without disrupting the database (a future version of Axiom may default this to the name of the Python class, so that you only need to define it if you later move or rename the class).

  • schemaVersion is used to track changes made to the schema declaration, allowing your data format to change while still being able to access old databases (a future version of Axiom may default this to 1, so you only need to define it after the first schema change).

  • This tag class has text attribute "name", the meaning of which should be obvious, and an attribute "item" which is a reference to another Item in the database. So, this lets you tag any object in your database.

  • Each attribute is made mandatory by passing allowNone=False. Without these, attributes could be omitted as desired, which doesn't make sense in this case.

Now let's see how we can put this into use:

Python 2.4.2 (#2, Sep 30 2005, 21:19:01)
[GCC 4.0.2 20050808 (prerelease) (Ubuntu 4.0.1-4ubuntu8)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> (Tag class definition elided)
>>> from axiom import store
>>> s = store.Store()
>>> Tag(store=s, name=u'system', item=s)1
<__main__.Tag object at 0xb783cadc>
>>> s.query(Tag).count() # How many Tags are there in the database, total?
>>> t = list(s.query(Tag))[0]
>>> print t, t.name, t.item # What is the tag I just found?
<__main__.Tag object at 0xb783cadc>, system, <Store (in memory)@0xb7c969cc>

axiom.store.Store.query takes a few more optional arguments, too.

  • s.query(Tag, Tag.item == anObject) if I want all the Tags that apply to a single object.

  • s.query(Tag, Tag.item == anObject).getColumn("name") is how I would get just the name of each Tag.

  • s.query(Tag, attributes.OR(Tag.name == u"fruit", Tag.name == u"vacation photos")).getColumn("item") gives me all the Items tagged "fruit" or "vacation photos".

  • I can also limit, offset, or sort with those keyword arguments:

    s.query(Tag, Tag.item == anObject, sort=Tag.name.descending, limit=10, offset=5)

    This gives me the 5th through 15th Tags applied to anObject in descending lexicographically-sorted tag name order.

Tomorrow: Axiom Powerups

1: Yes, the Store itself can be referenced just like any Item inside it. This is convenient for a number of reasons, although in this case it is just convenient because I don't have any other Items handy to use in the example.

Tuesday, November 15, 2005

Adding Axiomatic plugins

axiomatic has featured prominently in many of my recent blog posts, but what is it?

As one might guess, in simplest terms, axiomatic is a tool for manipulating Axiom databases. Going into a little more detail, axiomatic is a command line tool which gathers axiom.iaxiom.IAxiomaticCommand plugins using the Twisted plugin system and presents them as subcommands to the user, while providing the implementations of these subcommands with access to objects that pretty much any Axiom manipulation code is going to want (currently, that amounts to an opened Store instance). Twisted's option parser is used as the basic unit of functionality here.

I've already talked about several IAxiomaticCommand implementations: web, mantissa, and start. The first two of these add new Items to an Axiom database or change the values associated with existing Items. The last "starts" an Axiom database (more on what that means in a future post!).

So what you really want to know is how do I write an axiomatic plugin? Let me tell you, it could scarcely be easier:

from zope.interface import classProvides
from twisted import plugin
from twisted.python import usage
from axiom import iaxiom
from axiom.scripts import axiomatic

class PrintAllItems(usage.Options, axiomatic.AxiomaticSubCommandMixin):
# This one tells the plugin system this object is a plugin
# This one tells axiom it is an axiomatic plugin

# This is how it will be invoked on the command line
name = "print-some-items"

# This will show up next to the name in --help output
description = "Display an arbitrary number of Items"

optParameters = [
# We'll take the number of Items to display as a command
# line parameter. This is "--num-items x" or "-n x" or
# any of the other standard spellings. The default is 5.
('num-items', 'n', '5', 'The number of Items to display'),

def postOptions(self):
s = self.parent.getStore()
count = int(self.decodeCommandLine(self['num-items']))
for i in xrange(count):
print s.getItemByID(i)
except KeyError:
# The Item didn't exist

(Okay, seriously - I know this is a bit long: we'll be working on lifting a lot of the boilerplate out to simplify things; expect about half of the above code to become redundant soon). I just drop this into a file in axiom/plugins/ (there are other places I could put it, if I wanted; read the Twisted plugin documentation to learn more) and I've got a new subcommand:

exarkun@boson:~$ axiomatic --help
Usage: axiomatic [options]
-d, --dbdir= Path containing axiom database to configure/create
--help Display this help and exit.
mail Accept SMTP connections
userbase Users. Yay.
mantissa Blank Mantissa service
web-application Web interface for normal user
web Web. Yay.
web-admin Administrative controls for the web
radical omfg play the game now
click-chronicle-site Chronicler of clicking
encore Install BookEncore site store requirements.
sip-proxy SIP proxy and registrar
vendor Interface for purchasing new services.
vendor-site Required site-store installation gunk
print-some-items Display an arbitrary number of Items
start Launch the given Axiomatic database

Neat, huh?*

In case you're dying to see the output of this new command:

exarkun@boson:~/Scratch/Run/demo$ axiomatic -d my.axiom/ print-some-items
<axiom.item._PowerupConnector object at 0xb6fc770c>
<axiom.item._PowerupConnector object at 0xb6fc770c>
<axiom.userbase.LoginSystem object at 0xb6f42104>

Monday, November 14, 2005

Redirecting HTTP request logs

Okay I promised this a few days ago and never followed up. Here's how to get that annoying HTTP traffic logging out of your main log (or if you are using axiomatic start -n, off of your terminal):

exarkun@boson:~/Scratch/Run/demo$ axiomatic -d my.axiom/ web --http-log httpd.log

That's all. If the server was running while you did this, you will have to restart for it to take effect ;) Of course, someone could contribute a patch to change that...

One other thing: this command, unlike the others I've shared, depends on current trunk and will not work in a released version of Mantissa. There's probably a Mantissa release coming up pretty soon, though.

Friday, November 11, 2005

Configuring Static Resources on a Mantissa Server

Okay listen up, it's time for static file configuration.

exarkun@boson:~/Scratch/Run/demo$ axiomatic -d my.axiom/ web --static images:/usr/share/pixmaps/

What I've just done is configure the Mantissa server I created in a previous post to serve the contents of /usr/share/pixmaps at http://localhost:8888/images/.

There is an easy way to determine if a Python RPC library is broken

If the library employs Python's pickle module, it is broken. Period.

When will people learn that pickle is not suitable for this task? "Oh, it looks so easy." "Oh, it's so fast, just look at those objects fly." "Oh it will core my process when handling a maliciously constructed string, there goes my server." "Oh, it allows arbitrary code to be executed by a remote party, woops there goes my credit card database."

Not convinced? Run this code on your computer:

import pickle
pickle.loads("cposix\nsystem\np0\n(S'cat /etc/passwd'\np1\ntp2\nRp3\n.")

Wake up. This is not news. The pickle documentation explicitly points out the fact that it is not intended to be used in this fashion (although frankly, this warning could be a little closer to the beginning of the pickle documentation). Stop doing it. Stop releasing software that does it. Just stop, already.

Thursday, November 10, 2005

Configuring HTTP and HTTPS ports, and static content

Yesterday I showed you how to create and start the most minimal Mantissa server possible. If you ran the commands from my previous post, you saw something like this in your terminal:

exarkun@boson:~/Scratch/Run/demo$ axiomatic -d my.axiom mantissa
exarkun@boson:~/Scratch/Run/demo$ axiomatic -d my.axiom start -n
2005/11/11 01:45 EST [-] Log opened.
2005/11/11 01:45 EST [-] twistd SVN-Trunk (/usr/bin/python 2.4.2) starting up
2005/11/11 01:45 EST [-] reactor class: twisted.internet.selectreactor.SelectReactor
2005/11/11 01:45 EST [-] xmantissa.website.AxiomSite starting on 8080
2005/11/11 01:45 EST [-] Starting factory <xmantissa.website.AxiomSite instance at 0xb6ef2fec>

Awesome, I hear you exclaim. But what does it do, you most likely then complain. A fair question, to be sure!

In this minimal configuration, you have a few things:

  • A web server running on port 8080 serving, at /, a static template consisting primarily of a login link.

  • A login page, linked from /.

  • An administrative account, via which you can log in to this server (the username is "admin@localhost", the password is whatever you typed in at the password prompts earlier).

  • When logged in as the administrative account, a small page containing site statistics (such as number of logins, number of users, etc) and a Python REPL.

I'll cover some of these things in further detail in tomorrow's post. For now, let's go over the things we can do to this server.

Before trying to alter any of the server's configuration, let's see what's there already:

exarkun@boson:~/Scratch/Run/demo$ axiomatic -d my.axiom/ web --list
Configured to use HTTP port 8080.
Configured to use HTTPS port 0 with certificate None
Sessionless plugins:
/static/webadmin => item(DeveloperSite) (prio. 0)
/static/mantissa => file(/usr/lib/python2.4/site-packages/xmantissa/static) (prio. -255)
Sessioned plugins:
/ => item(FrontPage) (prio. -257)

Here we can see a /static/webadmin/ is being served by a DeveloperSite item (whatever that is) and /static/mantissa/ is being served out of the filesystem from the given path. Additionally, there is a page which requires a session at /, being served by a FrontPage item (whatever that is). Also, we see which ports the server is configured to use. Let's say someone is already using 8080, and we also want to turn on HTTPS:

exarkun@boson:~/Scratch/Run/demo$ axiomatic -d my.axiom/ web --port 8888 --secure-port 8889 --pem-file server.pem
exarkun@boson:~/Scratch/Run/demo$ axiomatic -d my.axiom/ web --list
Configured to use HTTP port 8888.
Configured to use HTTPS port 8889 with certificate server.pem
Sessionless plugins:
/static/webadmin => item(DeveloperSite) (prio. 0)
/static/mantissa => file(/usr/lib/python2.4/site-packages/xmantissa/static) (prio. -255)
Sessioned plugins:
/ => item(FrontPage) (prio. -257)

server.pem doesn't exist yet, so I guess I'll create that too, using a tool that comes with Vertex (maybe you want to install that, too):

exarkun@boson:~/Scratch/Run/demo$ certcreate -C US -s Massachusetts -c Boston -o 'Divmod, Inc.' -h localhost -e support@localhost -f server.pem
Wrote SSL certificate:
Certificate For Subject:
Common Name: localhost
Organization Name: Divmod, Inc.
Organizational Unit Name: Security
Locality Name: Boston
State Or Province Name: Massachusetts
Country Name: US
Email Address: support@localhost

Common Name: localhost
Organization Name: Divmod, Inc.
Organizational Unit Name: Security
Locality Name: Boston
State Or Province Name: Massachusetts
Country Name: US
Email Address: support@localhost

Serial Number: 1
Digest: 59:1F:25:FF:C9:18:98:31:B2:B8:B9:99:FA:99:4C:D0
1024-bit RSA Key Pair with Hash: 5bf8af0e6c7e01e1004b54dd3f0a983f

This one's self signed, since I don't have a VeriSign certificate lying around to sign it with... Anyway, now I'm set for HTTPS:

exarkun@boson:~/Scratch/Run/demo$ axiomatic -d my.axiom start -n
2005/11/11 02:08 EST [-] Log opened.
2005/11/11 02:08 EST [-] twistd SVN-Trunk (/usr/bin/python 2.4.2) starting up
2005/11/11 02:08 EST [-] reactor class: twisted.internet.selectreactor.SelectReactor
2005/11/11 02:08 EST [-] xmantissa.website.AxiomSite starting on 8888
2005/11/11 02:08 EST [-] Starting factory <xmantissa.website.AxiomSite instance at 0xb7616c6c>
2005/11/11 02:08 EST [-] xmantissa.website.AxiomSite starting on 8889

So now you know how to alter HTTP port configuration for a Mantissa server (any Mantissa server) from the command line. I know I promised something about static content in the subject, but I've already gone on longer than I intended. I'll cover configuring static content tomorrow.

Wednesday, November 9, 2005

How To Create A Mantissa Server

So a guy comes up to me and wants to know what the absolute minimum effort to get a Mantissa server running is.

So I tell him:

  1. Install Epsilon, Axiom, and Mantissa (You already have Python 2.5, PySQLite2, and Twisted 8.2 installed, right? This step will become "apt-get install python-mantissa" just as soon as we find a debian guy, promise).

  2. Run axiomatic -d my.axiom mantissa

  3. Run axiomatic -d my.axiom start -n

You now have a Mantissa server up and running. Pretty easy, huh?

Tomorrow: port, SSL, and logfile customization.

Saturday, November 5, 2005

Epsilon, Mantissa, and ClickChronicle

Mantissa and ClickChronicle had their plugins omitted from their last releases. 0.3.1 and 0.2.0, respectively, released fixing this (and some new bookmark-related features in ClickChronicle). Epsilon 0.3.2 released with a fix for autosetup() to include Twisted plugins automatically in the future.

Monday, October 24, 2005

Conch 0.6.0 is out

Twisted Conch is primarily an SSHv2 implementation, but also includes implementations of telnet and vt102. Each protocol comes with client and server implementations, in fact, Conch provides an SSH server right out of the box (configured with Twisted mktap, started with Twisted twistd, key files created with Conch's ckeygen):

exarkun@boson:~/Scratch/Run/conch$ ssh localhost
ssh: connect to host localhost port 22: Connection refused
exarkun@boson:~/Scratch/Run/conch$ ckeygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/exarkun/.ssh/id_rsa): ssh_host_key
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ssh_host_key
Your public key has been saved in ssh_host_key.pub
The key fingerprint is:
exarkun@boson:~/Scratch/Run/conch$ mktap conch --data .
exarkun@boson:~/Scratch/Run/conch$ sudo twistd -f conch.tap
exarkun@boson:~/Scratch/Run/conch$ ssh localhost
The authenticity of host 'localhost (' can't be established.
RSA key fingerprint is fa:d7:3e:a3:b3:2b:c9:c2:d5:9b:31:7d:47:3a:96:62.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
exarkun@boson:~$ ps
29747 pts/2 00:00:00 bash
29760 pts/2 00:00:00 ps
exarkun@boson:~$ logout
Connection to localhost closed.
exarkun@boson:~/Scratch/Run/conch$ tail -n 2 twistd.log
2005/10/24 20:06 EDT [SSHServerTransport,0,] avatar exarkun logging out (0)
2005/10/24 20:06 EDT [SSHServerTransport,0,] connection lost

Conch provides APIs for implementing custom SSH servers and clients as well, of course. For example, with it, you can easily write an SSH server that allows logins to a shell other than a unix login shell, such as a control application with a customized interface, or a text adventure game. Insults, a VT102 (plus a bit) implementation, plays a support role here, easing the task of writing custom interfaces by providing terminal and text attribute manipulation functions (eg, moving the cursor around and changing the color of text).

New in this release is an experimental windowing library based on Insults which provides such widgets as TextInput, Button, ScrolledArea, HBox, and VBox. This library can be used to very easily throw together rather complex user interfaces to be exposed, either via SSH or telnet. Those of you who have been following my blog will have seen some screenshots of this library in action.

Grab the 0.6.0 release or visit the project website.

Friday, October 21, 2005

Get your chat on

Twisted Words 0.3.0 has just been released! Twisted Words provides implementations of a handful of IM protocols, including IRC, MSNP8, OSCAR, TOC, and Jabber. Twisted Words also comes with a multi-protocol server built around a few simple interfaces, the goal of which is to facilitate implementations of novel servers, clients, and bots. Out of the box, Twisted Words comes with server that accepts connections over IRC and PB, and seeks out installed third-party plugins to support other interfaces (for example, the Nevow LivePage web-based interface currently sitting in one of my sandboxes.

Since the last release, the server infrastructure has been greatly simplified, documentation improved, extraneous code removed, and test coverage increased. Jabber support has also grown immensely. Ralph Meijer has taken over maintenance and is driving new feature development. Additionally, twisted.xish has moved to twisted.words.xish, since it is purely a support module for Twisted Words Jabber support. As far as I know, no one outside of Twisted is using Xish directly: if you were, I'm sorry, if you come over to my house I will make you a cheese sandwich to make up for it. Update your code to import from the new location, or stop using it, or keep the last release around.

I think Twisted Words has some exciting development in its near future. Stay tuned. You can download Twisted Words 0.3.0 or visit the project page.

Bad changeset

Possibly not the best commit ever.

Thursday, October 20, 2005

New Twisted Mail Release: 0.2.0

Twisted Mail provides client and server implementations of three protocols: SMTP, POP3, and IMAP4. These differentiate themselves from the Python standard library implementations both by presenting a much higher-level, easy-to-use interface and in their server components which allow the implementation of custom servers for each protocol without dealing with protocol-level issues.

Twisted Mail includes a simple, out-of-the-box email server which accepts messages over SMTP, stores them in a Maildir arrangement, and can serve them to clients over POP3. This is intended primarily as a demonstration of the use of the protocol implementations in Twisted Mail, not as a production-quality mail server. While it performs fairly reliably under normal operation, almost no effort has been expended towards making it robust against failures in general.

Twisted Mail also comes with a simple /usr/sbin/sendmail replacement, mailmail. Rather than using a custom schema to communicate messages to an MTA, mailmail connects to localhost on port 25 and gives the message to whatever server happens to be listening there.

This is a minor feature enhancement release. The IMAP4 and POP3 client implementations have both gained methods for explicitly negotiating TLS options. The POP3 client now facilitates command timeouts and has gained support for the POP3 commands NOOP, RSET, and STAT.

Check out the Twisted Mail project page or download the release.

Wednesday, October 19, 2005

Twisted Names 0.2.0 Released

Twisted Names is a DNS library for both clients and servers. This is primarily a bug-fix release. Several cases where TCP connections were left hanging have been fixed. Use of deprecated Twisted APIs have also been removed. There is improved test coverage of zone transfer support.

Grab the latest release or visit the project page. You might also want to file a bug report. ;)


Tuesday, October 18, 2005

Mantissa Application Server - Release 0.1!

Twisted, Xapwrap, Epsilon, Vertex, Axiom, Nevow... What do you get if you put them all together? Let me introduce you to Mantissa.

Mantissa combines and builds upon these libraries to create a coherent deployment target for your software.

The focus of this Mantissa release is support for the web: there are themes, conventions for template lookup, support for Nevow LivePage, and a framework into which web interfaces to applications can be dropped. A major feature is the ability to deploy otherwise unrelated applications into the same server, provide access to them using the same user accounts, have them cooperate to display elements of the page which each require, and give the user an organized view onto them.

As the kids like to say, Mantissa is skinnable: the application logic it represents can be displayed in whatever manner HTML allows (and a few beyond that ;) and the display is controlled completely using CSS and XHTML. In addition to dynamic applications, static content is also supported: plugins for axiomatic are provided to allow reorganization of static content on the fly, but of course it can also be laid out automatically by the application-code which initializes a Mantissa instance.

Functionality exposed through URLs is, by default, hardened against cross-site scripting and other attacks: unless URLs are otherwise specified, they will be stable for a particular user, but randomized between different user accounts. Mantissa also provides a highly hookable ticketting and signup-system. By default, users first verify an email address (acquiring a ticket in the process) before gaining access to the system. Ticket issuance can be customized, and new Benefactors can be defined to endow newly created users with application-appropriate abilities. As a demonstration of the LivePage features, a sample Mantissa configuration is included in the 0.1 release which has a single administrator user: the administrator has access to an interactive Python interpreter prompt after logging in.

So there's some web in it. What else?

Mantissa provides a framework for indexing an searching as well. Applications provide the content, Mantissa arranges the indexing, and the user gets a unified searchable view of all content through a single interface.

And how do you feel about VoIP? Mantissa includes Divmod's SIP implementation. Ideas for applications involving flexible control of real-time voice channels are left as an exercise for the reader.

Building on Vertex, Mantissa applications can also rely on Q2Q, either to accept connections and provide a service, or for making outgoing connections to other users or domains. Yea: Mantissa applications can trivially communicate with applications (perhaps Mantissa-based, perhaps not) running on other hosts.

Upcoming releases will polish the web support, probably switch over to Athena, or at least add support for it, and start adding generally useful widgets (ie, a pager for large data sets), but also give some attention to non-HTTP areas (email and ssh will probably show up next).

Get the Mantissa 0.1 release, visit the Mantissa project page, or peruse the Mantissa wiki.

Monday, October 17, 2005

Emergency Axiom 0.1

An object-relational (sort of) database, implemented in Python, backed by SQLite, and informed by the development of Atop, twisted.world and others: Axiom is still young (hardly three months have gone by since its conception), but it already supports a significant portion of the features present in the databases which shaped its development — and has for longer than not.

Axiom's goal is to provide an efficient, non-SQL interface to an existing relational database. Unlike other libraries in this area, Axiom discourages inefficient operations (such as loading a single object at a time - possibly building it up from multiple queries against different tables): instead it encourages a style where many objects are loaded with a small, often fixed, number of queries (sometimes the number being fixed at one). This lets the underlying relational database do what it's good at: yank data into memory out of underlying storage, perhaps based on some limiting criteria, or sorted in a particular way. From the Python side, you get genuine Python objects with Python-native attributes: Axiom is object-based, not table-, column-, or row-based. Objects stored in an Axiom database can refer to any other object stored in the same database (regardless of type - just like normal Python references), and attributes can be instances of any type Axiom natively supports (currently booleans, integers, byte strings, character strings, or timestamps). Additionally, support for new attribute types can be added by defining new attribute classes. Objects stored in an Axiom database provide a schema definition (as part of their class definition) which allows in-memory constraints to be enforced, just as they will be in the database, as well as providing necessary information for the upgrade mechanism (ie, you can change your schema without invalidating an existing database).

Of course, Axiom also integrates with common Twisted practices:

  • an Axiom store is also a Twisted IServiceCollection provider. A Store can be started and stopped, each of which propagates the appropriate event down to any in-database objects which care to receive them. This allows (if you will excuse the extremely hypothetical example) a web service to be implemented as an Axiom-stored object: when the object receives startup notification, it can bind port 80 and start handling requests; when it receives shutdown notification, it can close the port and delay actual shutdown until it has given each outstanding request a fair chance to be completed. By letting services exist as database objects, any database manipulation tools (such as applications or user- or administrative-configuration interfaces can persistently alter application configuration - goodbye, configuration files).

  • Axiom comes with an implementation of Twisted's IRealm interface. This means an Axiom store can be given to a portal to allow authentication and authorization against it, using the standard Twisted Cred APIs. Any Twisted protocol implementation which supports Twisted Cred (so, SMTP, POP3, IMAP4, SIP, HTTP, IRC and PB to name a few) will work with an Axiom Store backend.

  • Axiom makes heavy use of plugins in several ways. Features for an Axiom database can be provided and configured via a particular interface: IAxiomaticCommand. Once a plugin is installed and its Python modules are importable, it will be available to the database administrator (be that a sysadmin on the command line or an application admin via a web interface). Objects in the database can also act as powerups for other objects in the database. Powerups provide additional functionality in a modular and pluggable way, and can be queried for in a simple and efficient manner.

  • Axiom includes a wall-clock scheduler as well, made available to application-level code as a powerup on an Axiom Store. By loading up this powerup and calling its methods (such as the surprisingly-named schedule), application-level code can arrange to have its own code invoked at a particular point in the future, be it seconds, hours, or months away.

These features make it very straightforward to use Axiom in a Twisted application.

Currently, Divmod has several applications in development which use Axiom, and several more in the pipeline. Though Axiom is far from complete, even in its current form it has demonstrated its utility for a variety of applications, both client- and server-side.

It's difficult to include a demonstration of Axiom's abilities here: only the most trivial of database applications can reasonably fit into a blog post, yet such examples are rarely very interesting or informative. Instead, here's a brief transcript of an interaction (in which I create myself a new user account and disable an existing account) with a database being used by a running server, using the command line tool included in Axiom, axiomatic:

$ axiomatic --dbdir cc.axiom/ userbase list | grep exarkun                                 
$ axiomatic --dbdir cc.axiom/ userbase create exarkun divmod.org
Enter new AXIOM password:
Repeat to verify:
$ axiomatic --dbdir cc.axiom/ userbase list | grep exarkun
$ axiomatic --dbdir cc.axiom/ userbase --help
Usage: axiomatic [options] userbase [options]
--help Display this help and exit.
install Install UserBase on an Axiom database
create Create a new user
disable Disable an existing user
list List users in an Axiom database

$ axiomatic --dbdir cc.axiom/ userbase disable exarkun divmod.com
$ axiomatic --dbdir cc.axiom/ userbase list
exarkun@divmod.com [DISABLED]
userbase is a plugin included with Axiom for user management; the axiomatic script pulls it in and makes it available to me on the command line, making the above possible.

Download Axiom 0.1, visit the project page on divmod.org, or check out the Axiom wiki page.

Friday, October 14, 2005


Divmod is proud to announce the release of Vertex 0.1. Vertex is an implementation of the Q2Q protocol (sort of like P2P, but one better). There are a few moving parts in Vertex:

  • PTCP: a protocol which is nearly identical to TCP, but which runs over UDP. This lets Q2Q penetrate most NAT configurations.
  • JUICE ([JU]ice [I]s [C]oncurrent [E]vents): a very simple but immensely flexible protocol which forms the basis of the high-level aspects of Q2Q
  • vertex: a command line tool which exposes a few features useful in many situations (such as registration and authentication)

Q2Q is a very high-level protocol (alternatively, transport) the goal of which is to make communication over the internet a possibility (if you enjoy setting up tunnels or firewall rules whenever you need to transfer a file between two computers, Q2Q may not be for you). Q2Q endpoints aren't hardware addresses or network addresses. They look a lot like email addresses and they act a lot like instant message addresses. You can hook into yours wherever you can access the internet, and you can be online or offline as you choose (Q2Q supports multiple unrelated protocols, so you also might be online for some services but offline for others). Two people with Q2Q addresses can easily communicate without out-of-band negotiation of their physical locations or the topology of their networks. If Alice wants to talk to Bob, Alice will always just open a connection to bob@divmod.com/chat. If Bob is online anywhere at all, the connection will have an opportunity to succeed (Bob might be busy or not want to talk to Alice, but that is another matter ;). The connection is authenticated in both directions, so if it does succeed Alice knows she is talking to the real Bob and vice versa.

The Q2Q network has some decentralized features (there is no one server or company which can control all Q2Q addresses) and features of centralization (addresses beneath a particular domain are issued by a server for that domain; once issued, some activities require the server to be contacted again, while others do not). Vertex includes an identity server capable of hosting Q2Q addresses. Once you've installed the 0.1 release, you can run it like this:

exarkun@boson:~$ cat > q2q-standalone.tac
from vertex.q2qstandalone import defaultConfig
application = defaultConfig()
exarkun@boson:~$ twistd -noy q2q-standalone.tac
2005/10/15 00:12 EDT [-] Log opened.
2005/10/15 00:12 EDT [-] twistd 2.0.1 (/usr/bin/python2.4 2.4.1) starting up
2005/10/15 00:12 EDT [-] reactor class: twisted.internet.selectreactor.SelectReactor
2005/10/15 00:12 EDT [-] Loading q2q-standalone.tac...
2005/10/15 00:12 EDT [-] Loaded.
2005/10/15 00:12 EDT [-] vertex.q2q.Q2QService starting on 8788
2005/10/15 00:12 EDT [-] Starting factory <Q2QService 'service'@-488b6a34>
2005/10/15 00:12 EDT [-] vertex.ptcp.PTCP starting on 8788
2005/10/15 00:12 EDT [-] Starting protocol <vertex.ptcp.PTCP instance at 0xb777884c>
2005/10/15 00:12 EDT [-] Binding PTCP/UDP 8788=8788
2005/10/15 00:12 EDT [-] vertex.q2q.Q2QBootstrapFactory starting on 8789
2005/10/15 00:12 EDT [-] Starting factory <vertex.q2q.Q2QBootstrapFactory instance at 0xb77787cc>

You can acquire a new Q2Q address using the vertex command line tool:

exarkun@boson:~$ vertex register exarkun@boson password

boson is a name on my local network, making this address rather useless. On the other hand, no one will be able to pretend to be me by cracking my hopelessly weak password ;) If you set up a Vertex server on a public host, you will be able to register a real, honest-to-goodness Q2Q address beneath its domain (careful - so will anyone else).

vertex also offers a tool for requesting a signed certificate from the server. These certificates can be used to prove ones identity to foreign domains without involving ones home server. Another feature vertex provides is a toy file transfer application. Bob can issue a vertex receive while Alice issues a vertex send pointed at him, and the file will be transferred.

Much of the real power of Q2Q is exposed to developers using two methods: listenQ2Q and connectQ2Q. These work in roughly the same way Twisted's listenTCP and connectTCP work: they offer support for writing servers and clients that operate on the Q2Q network.

Thursday, October 13, 2005

Guess What! Divmod Nevow 0.5 Release!

It's been a long time since one of these (very roughly, 17660815 seconds. For those of you following along at home:

>>> from epsilon import extime
>>> then = extime.Time.fromISO8601TimeAndDate('2005-03-23T18:47:39')
>>> now = extime.Time()
>>> now.asPOSIXTimestamp() - then.asPOSIXTimestamp()
), and we've got a lot of great new stuff to show for the wait. Skipping a lot of things, because they'd make this post way too long, we've got:
  • We've given LivePage a significant overhaul. We're not done in this area yet, though: watch for changes here in the next release.
  • A lot of code has been updated to work with zope.interface and Twisted 2.0. Nevow 0.5 requires each of these.
  • Formless no longer adds __nevow_carryover__ to the redirect URL unless it actually needs to carry values over to the next request.
  • Improved documentation and more example applications, including examples for:
    • Working with Fragments
    • Using macros
    • Integrating with Axiom
  • A JavaScript Object Notation serialization and parsing has been added, to allow more complex objects (still no duplicate references or user-defined classes, though) to be used with LivePage
  • LiveEvil, the previous implementation of LivePage, has been removed
  • Guard has been expanded to more easily support persistent (eg, saved in a database) sessions
  • Bugs have been fixed in a lot of areas:
    • VirtualHostList
    • Behavior relating to quoting of '/' in URL segments
    • Corrected XML namespace prefixes in generated output
    • The default 404 page is now well-formed XML
    • The radio input now correctly selects the default value and its display elements line up in a more aesthetically pleasing manner
    • Remove duplicate request-finishing in the unhandled exception error path
    • Fixed macro/xmlfile incompatibility
    • Fixed a huge object/memory leak in applications that did not re-use all their Page instances to serve each request
    • Fixed rendering of Fragments so as not to interefer with their parent's rendering process
    • Fixed rare Deferred/tag/slot-filling interaction which caused some slots to be filled with empty values instead of useful content

A word about livepage

Nevow 0.4.1 provided a better-than-ajax feature known as LivePage, via the nevow.livepage.LivePage class. This was a replacement for an older implementation, nevow.liveevil.LiveEvil, which was deprecated in 0.4.1. Nevow 0.5 removes LiveEvil and augments LivePage with a bunch of new features, but also some API incompatibilities. Where possible, backwards compatibility has been maintained, but it is likely that any extensive LivePage applications will be broken at least a little bit by 0.5. Of course, there's a lot of other changes in Nevow that will probably jar such applications anyway. Most people whom this affects have been in the IRC channel or on the mailing list and already know this.

What's not as well known yet is that the next release of Nevow will likely provide still another new implementation of the same idea. This time nevow.athena.LivePage will be the preferred API. Where the LivePage implementation in 0.5 encourages the definition of JavaScript functions in the server code using Python, nevow.athena.LivePage encourages a much stricter separate of client and server concerns. It also provides an API for asynchronous data passing between client and server (initiated by either side), rather than the current API for passing code from the server to the client to be executed.

Another area of upcoming change will be form generation. formless is capable of some wonderful things, but some of its limitations are becoming clearer. A simpler system is just around the corner. Look for it in the next release -- which will be in much less than 17660815 seconds from now.

Download Nevow 0.5 or visit the project page

Wednesday, October 12, 2005

Divmod Xapwrap Released

Xapian is a fairly popular full-text indexing system ("Probabilistic Information Retrieval library"). It's got Python bindings, but they're not so fun to use. Xapwrap is a layer on top of these bindings which tries to simplify matters:

exarkun@boson:~/xapwrap-demo$ cat > a-m
exarkun@boson:~/xapwrap-demo$ cat > n-z
exarkun@boson:~/xapwrap-demo$ python
Python 2.4.2 (#2, Sep 30 2005, 21:19:01)
[GCC 4.0.2 20050808 (prerelease) (Ubuntu 4.0.1-4ubuntu8)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from xapwrap import index, document
>>> idx = index.Index(name='demo.db', create=True)
>>> idCounter = 1
>>> docIDs = {}
>>> for docName in 'a-m', 'n-z':
... docIDs[idCounter] = docName
... text = document.TextField(file(docName).read())
... idx.index(document.Document(textFields=[text], uid=idCounter))
... idCounter += 1
>>> res = idx.search('zoo')
>>> print res
[{'score': 100, 'uid': 2}]
>>> print docIDs[res['uid']]
>>> idx.search('animal')
[{'score': 100, 'uid': 1}]

Get the Xapwrap 0.3.0 release.

Tuesday, October 11, 2005

Divmod Epsilon Released

In the beginning Guido created Python. Now Python was formless and empty, duplicate code was over the surface of programs. And Guido said, Let there be modules and there were modules. Guido saw that the modules were good, and he separated them from the programs. Guido called them the standard library. And there was the library, and there was the program -- the first abstraction.

Now, not all of us are lucky enough to be able to toss things into the Python stdlib. The practice has arisen in Twisted to take functionality that is generally useful -- not specific to Twisted -- and place it in the twisted.python package. There's an option parser and file path manipulation functions and so forth. This works okay for Twisted development, but the code can't really be re-used without creating a Twisted dependency (which is okay by me, mind you, but I hear it's inconvenient for some other people). Later, at Divmod, a similar derivative practice became common. Common functionality that wasn't really specific to our projects was aggregated in a module named atop.tpython -- the tpython being short for, of course, twisted.python. So now we're two levels abstracted from where we really want to be.

Well, a new day is upon us. Divmod has now released Epsilon, a package for some generally useful code that others might find handy. This first release, 0.1.0, has two major offerings and a few more minor ones.

First, there's epsilon.extime, a module which offers conversion to and from a variety of time formats: RFC 2822 formatted dates, POSIX timestamps, datetime objects, struct_time objects, ISO8601 formatted timestamps, and a particular context-aware format known as the "human readable format". Additionally, it offers a Time class which can be constructed from or converted to any of these formats, as well as have certain date math performed on it. Here's a brief sample:

>>> from epsilon.extime import Time
>>> Time().asDatetime()
2005, 10, 12, 4, 52, 8, 237248,
tzinfo=<epsilon.extime.FixedOffset object at 0x-4860a234 offset datetime.timedelta(0)>)
>>> Time().asISO8601TimeAndDate()
'04:52 am'
>>> import datetime
>>> (Time() + datetime.timedelta(days=2)).asHumanly()
'14 Oct, 04:53 am'

Next up is epsilon.cooperator. This provides a scheduler for tasks implemented as iterators. It offers full Deferred support and hooks for scheduling behavior. In particular, it can be used when it's possible for so many tasks to be running concurrently that to allow them all to set their own pace would damage interactive performance: here it slows things down and makes sure new events can be processed. With a little creativity, it can also be used to group or prioritize tasks.

The remaining features include a setup hook for Twisted applications which install plugins to cause the distutils process to regenerate the plugin cache, a class for simplifying providing multiple listeners with the result of a Deferred, and a structured Version class with special SVN integration (for example,

$ python -c 'from epsilon import version; print version'
[epsilon, version 0.1.0 (SVN r529)]

showing that I have something at least as new as Epsilon 0.1.0 installed, and it is being imported from an SVN checkout at revision 529).

Divmod's Wiki has an Epsilon project page at http://divmod.org/trac/wiki/EpsilonProject.

Wednesday, October 5, 2005

Faucet reborn

I only bothered to hook it up to a lame chat application, but the widget is demonstrated. Tomorrow I may have a go at hooking it up to an actual game. Faucet's classic use was for a text adventure and had the top-left pane used for a description of your location, the top-right pane used for a list of what objects were present, and the bottom pane used to display command input and the responses they generated, as well as other game output which arrived asynchronously.

That's probably it for screenshots for a while. If you're really interested, the code's in Twisted (mostly in trunk, a little bit in a branch - exarkun/scrollarea). Check it out. Write some cool widgets.

Tuesday, October 4, 2005

Improved Rendering of Scrolling Elements

The new ScrolledArea is highlighted in red below. Click through for a larger version!

With this done, I think it's time to turn to an actual application. Tomorrow night: vt102faucet.py (What's faucet? You asked.)

Monday, October 3, 2005

More New Insults Widgets

Added Viewport, HorizontalScrollbar, and VerticalScrollbar tonight. Combine these with the existing HBox, VBox, and Border and you have a reasonable approximation of a generic scrolled area widget. I'll probably do an actual ScrolledArea later this week, since rendering things as a collection like that is slower and not as pretty as it could be. However, I think the concept is proven. Screenshots for you, today:

Sunday, October 2, 2005

New Insults Widgets

PasswordInput and TextOutputArea. PasswordInput is like TextInput (you can guess how it's different). TextOutputArea lets you render a big chunk of text, either wrapping long lines or truncating them.

doc/conch/examples/window.tac has been updated to demonstrate both of these. Particularly satisfying is watching the display as the terminal is resized...

Friday, September 23, 2005


In no particular order:

twisted.protocols.telnet is now officially deprecated. I checked in the warning today. twisted.conch.telnet has been the preferred telnet implementation for some time, but it did not seem visible enough.

The Nevow callRemote branch is nearly finished. It probably would have been merged by now if I hadn't been sick most of this week. The branch provides an alternate implementation of the LivePage idea. The main difference is that instead of passing code around to be evaluated, it uses uses MochiKit for Deferreds and lets you pass data from client to server and vice versa. The API is remeniscent of Twisted's Perspective Broker.

Divmod has released Pyflakes 0.2.0. This release adds support for Python 2.4, by way of understanding generator expressions and function decorators.

For everyone who is looking to contribute to Twisted but has never been able to figure out how, here's your chance. There are about twenty new issues (1183 through 1202) in the Twisted issue tracker. They're all very tractable amounts of well-defined work. They all refer to inadequacies in particular Twisted unit tests. Grab one and submit a patch!

Thursday, August 18, 2005

Magical Concurrency Faeries or How I Learned To Stop Worrying and Love Deferreds

Every now and then, a recipe like this one pops up. Wouldn't it be wonderful to live in a world where you could close your eyes, tap your heels together, say "this API is asynchronous" three times and be rewarded with an asynchronous version of a synchronous function? I know my life would be a lot easier in that world.

We live in a world with significantly less magic than people sometimes make out. The programming languages we have available now simply cannot perform the synchronous-to-asynchronous transformation automatically for the general case. There are languages which provide interesting constructs which may be the beginnings of such a feature, but Python isn't one of them. What Python gives you in the way of "magic" asynchronous behavior is what we've had for much more than a decade: platform native, pre-emptive threads. While these can be put to some use, the programmer must understand the implications or the result is randomly buggy software.

If you haven't done much programming with threads, this consequence may not be obvious. Consider the case of a library which uses module-level storage to implement its public API. Now consider the author didn't plan to use it with threads and so didn't protect any access of this shared resource with locks. Now try and imagine what will happen if your program happens to call into this library from two different threads at about the same time. For bonus points, try to imagine how you might write a unit test which illuminated this bug in your software. An example of a library written in such a way is the Python standard library urllib module.

Asynchronous programming feels unnatural to many people. The tendency exists to want to write programs that read like they execute: top to bottom. Perhaps this tendency exists because synchronous programming is what most programmers are first taught, or perhaps it is because synchronous programming is simply a better fit for our brains; regardless of the reason behind the preference, it does not change the fact that asynchronous programs cannot be written as though they were synchronous with today's tools. If you think threads are hot, use threads and don't waste your time with tools like Twisted. If you think asynchronous programming presents serious benefits (as I do), embrace it all the way: it won't be easy (writing software rarely is), and you will often have to take existing synchronous software and rewrite it so as to present an asynchronous API. At these times, grit your teeth and get on with it. You'll be much happier in the long run than if you had tried to cobble something together using threads and end up spending a far greater amount of time tracking down non-deterministic bugs that only appear under load after days of runtime on someone else's computer and which disappear whenever you insert debug prints or run with a debugger enabled.

Oh yea, and as Itamar pointed out, the particular recipe I referenced above is, in addition to being conceptually flawed, implemented in a way which is just plain buggy. Even if you disagree with everything I said above, you shouldn't use it.

Monday, August 15, 2005

On 17 Hour Work Days

They're hard.

What did you think I was going to say?

Friday, August 12, 2005

Retrieving User Idle Time on Freenode

Recently Freenode deployed a code update that removed the "idle" field from the whois command's output. mwh pointed out that you can work around this by issuing whois username username.

Thursday, August 11, 2005

Simple Twisted Application for Fun &lt;strikethrough&gt;and Profit&lt/strikethrough&gt;

I really enjoyed the series Firefly that aired a few years back. I've enjoyed it a few more times since. I'm really looking forward to the upcoming movie. So much so, in fact, that I even wanted to try to get into one of the pre-screenings. There's a website that announces dates and locations for pre-screenings. Being pretty lazy and also forgetful, I can't be bothered to actually check the website frequently enough to notice when new tickets are available before they are all sold out. Fortunately, being a programmer, I can simply bend a minion to my will. I whipped up this quickie to do the checking for me:

import rfc822, md5, sys

from twisted.internet import task, reactor
from twisted.mail import smtp
from twisted.python import log
from twisted.web import client

url = "http://www.cantstopthesignal.com/"

MSG = """\
Subject: The upcoming film, Serenity, is awesome
Date: %(date)s
From: "A Respectable Internet Citizen" <serenitymovie@intarweb.us>
Message-ID: %(msgid)s
Content-Type: text/html

Serenity hash changed! Go look!
<a href="%(url)s">%(url)s</a>

def mailinfo():
return {
'date': rfc822.formatdate(),
'msgid': smtp.messageid(),
'url': url}

lastDig = None

def main():
def got(page):
global lastDig
dig = md5.md5(page).hexdigest()
if dig != lastDig:
if lastDig is not None:
sys.argv[1], # SMTP host
sys.argv[2], # sender email address
sys.argv[3:], # recipient email addresses
MSG % mailinfo())
lastDig = dig

task.LoopingCall(main).start(60 * 30)

The took about 10 minutes to write and even worked on the 2nd try (the first try I forget recipient addresses had to be a list and gave it a string instead). Then it ran for 21 days before sending me an email a few days ago! Alas, I don't live in Great Britain, or even Europe. It has also, of course, sent out an email once each time one of the listed theaters sells out. And now they're all sold out. :( Given the short period of time remaining before the probable release date, it seems unlikely there will be any more pre-screenings.

Sunday, August 7, 2005

Twisted Conch Insults

Twisted Conch Insults is cool. I wrote the bulk of it almost a year ago, but lately I've been revisiting it with the intent of actually developing some useful software based on it. Since I've never written any major terminal-based apps before, I decided to start with a lot of tiny attempts to get a feel for how things should work.

One of the first things I tried was a typespeed clone:

insults-based typespeed clone

Next, I tried something vaguely nethackesque (yes, that's unicode):

insults-based maze game

Most recently, I started tackling the problem of developing a windowing library for insults (Itamar and I paired on most of this one):

Text UI

There are a couple others in my sandbox for those who are curious. I'm pretty happy with how easy these were. They varied in difficulty from about half an hour of work to about a day's work. Some of the APIs definitely need to be improved (like having a 2-dimensional buffer API for manipulating large areas of the screen, rather than just exposing fundamental terminal operations), but clearly a lot is possible with what already exists. I think I'm going to continue developing the windowing library for now, and then start another round of experiments based on that.

Sunday, July 24, 2005



Saturday, June 25, 2005

Major Twisted Words update

From the commit message:

This is a complete rewrite of the twisted.words server code. Botbot and Tendril features are dropped for the time being. Provided is a generic chat server accessible via IRC and PB, and with a plugin mechanism for adding new protocols. Authentication is done via newcred; unlike the previous revision, there is no support for registering new users through the server. This should be dealt with using a generic checker manipulation system (so that any checker configuration can be manipulated, rather than one or two I choose to support in words).

Webwords is also gone, as there are no exciting web technologies in Twisted. The plugin mechanism is, however, used to provide a Nevow LivePage interface to the chat service which could be extended to perform all the same tasks previously covered by Webwords.

t-im may or may not still work.

Thursday, June 16, 2005

How Not To Make A Page-Atomic Copy Of A File

Can you find the hidden bug?

import os

def copy(fIn, fOut):
blockSize = os.statvfs(fIn.name).f_bsize
while 1:
bytes = fIn.read(blockSize)
if not bytes:

Tuesday, May 31, 2005

Time out!

Back to work after a long weekend, and what a productive weekend it was: I paired with Itamar on some fun stuff (more about this later), and some Twisted fixes and enhancements too ;)

Today I paired with Glyph on work stuff. It didn't feel like the most productive day, but we did accomplish a few concrete tasks: a deployment-blocking bug got fixed (it's hard to feel too great about this, though: it was a regression, and one I introduced no less); a few demo features got hammered out (but mostly refinements of previous ideas and implementations). Pairing is great even on bad days though: the result is always that much better, the work goes that much faster, and the extra level of communication helps resettle other problems I might be having (even ones I'm not aware of yet), which carries over into subsequent days.

After work (well, eventually, it was probably about 10:30 PM by the time we got started), Glyph, Ying and I relaxed with _Sin City_. Okay: relaxed is not the right word. Still, one must qualify these things as entertainment (I think I actually did like the movie (this is not to say I enjoyed it (don't ask me what the difference between those things is, I'm still working on it myself)), but I did feel physically ill by the time it was over).

But there is one distinct and unwavering downside to a day of pairing. Since I'm not constantly checking my mailbox all day, things tend to pile up. Today I came home to 113 messages to read, about 15 of which I could reasonably be expected to reply to, and half that which I really, really should take care of (not tonight though, I decided to blog about it instead). I suppose I could look on the bright side: I know exactly what I'm doing tomorrow morning before work.

Tuesday, May 24, 2005

Doubling bytestreams

For a while now, every day, we have been moving a somewhat hefty chunk of bytes, about 15GB worth from one machine onto another. The bytes are a tar file, generated in realtime and piped to ssh (connected to a host untaring the bytes). Pretty standard stuff, really. A while back we decided we wanted to send this tar to two hosts, instead of one. No big deal, we just ran tar twice, piping the output to an ssh process connected to a different host each time. Worked like a charm. Recently, we decided the load incurred by the second copy was heavy enough to be worth avoiding. Obvious solution: pipe tar to tee, send one of tee's outputs to one ssh, the other to the other.

That doesn't work. Woops.

For whatever reason, tee chokes after a bit less than 8 GB of data. write error it says, and poof one of the streams is dead (always the same one, interestingly, and the other one always carries on just fine). Rather than waste too much time trying to figure out who is at fault here (or perhaps as a way of doing so ;), I wrote this quickie:


"""Write bytes read from one file to one or more files.

import os, sys

from twisted.python import usage, log

class Options(usage.Options):
def opt_out(self, arg):
vars(self).setdefault('out', []).append(arg)
opt_o = opt_out

def postOptions(self):
self.outfds = []
for fname in vars(self).get('out', []):
self.outfds.append(os.open(fname, os.O_WRONLY | os.O_CREAT))

def main(infd, *outfds):
while 1:
bytes = os.read(infd, 2 ** 16)
if not bytes:
for i in xrange(len(outfds) - 1, -1, -1):
os.write(outfds[i], bytes)
log.msg("Error writing to %d" % (outfds[i],))
del outfds[i]

if __name__ == '__main__':
o = Options()
raise # sys.exit(str(sys.exc_info()[1]))
main(0, *o.outfds)

I called it yjoint, uncreative clod that I am (Hey, at least I didn't call it pytee). It's not exactly a drop-in tee replacement. We use it more or less like this:

exarkun@boson:~/$ tar c yummy_data | yjoint \
> --out >(ssh host1 tar x) \
> --out >(ssh host2 tar x)

Swapped tee out and yjoint in, and suddenly we are in business again.

I wonder what the deal with tee is?

Thursday, April 28, 2005

Hard time

I discovered the hard way that, even with stat_float_times enabled, mtime does not have a very great resolution. About a millisecond on my machine (actually ever so slightly worse, but it seems to vary with CPU clock speed). This would be fine, if I were not relying on being able to compare times retrieved from the mtime field of file metadata and times retrieved from gettimeofday(2) with each other. Too bad for me.

What this ended up leading to was this fun sequence:

x = time.time()
y = modifyAFileAndReturnMTime()
z = time.time()

assert x <= y <= z # Woops, this fails

Woe is me. So after a few hours of figuring that it was mtime resolution that was getting me, I figured I'd just cludge it and stuff time.time() into the mtime field at the crucial moment. This works because the field itself is capable of precision down to the nanosecond level, the operating system just doesn't keep track of it that carefully. Having replaced the actual mtime with my app's concept of the mtime, all my times will sort correctly and I won't accidentally toss files out the window.

So how do I do this? Well, utimes(2) of course. Heck, it's even wrapped in Python:

>>> import os
>>> os.stat_float_times(True)
>>> os.path.getmtime('.')
>>> os.utime('.', (1114739528.75, 1114739528.75)) # atime, mtime
>>> os.path.getmtime('.')

Ack! So much for nanosecond precision! Where the hell'd my fractional parts go? utimes(2) certainly supports this:

>>> import os
>>> os.stat_float_times(True)
>>> import dl, array, struct
>>> libc = dl.open('libc.so.6')
>>> buf = array.array('c', struct.pack('IIII', 1114739528, 750000, 1114739528, 750000))
>>> libc.call('utimes', '.', buf.buffer_info()[0])
>>> os.path.getmtime('.')

Okay, so Python's screwed up :( A little digging later, I see that actually maybe it's just Ubuntu's build of Python. HAVE_UTIMES isn't being defined, so all my nice double precision floating goodness is being dropped on the floor and only the integer parts carried over. Time to file a bug report...

Unfortunately, having these times be accurate is somewhat important to the application. I think maybe I can fudge it by either adding or subtracting a few seconds from each mtime (I'm not sure which yet... hopefully adding, because subtraction is hard). At worst this should mean I copy a few extra files... I hope. More thought is required.

Saturday, April 9, 2005

I wrote most of a new smtp client from scratch tu...

I wrote most of a new smtp client from scratch tutorial for Twisted a couple weekends ago. I wanted to polish it up some more before asking for comments but it doesn't look like I'm going to get to it as soon as I'd hoped. It is in the style of the Twisted from scratch tutorial, though quite a bit shorter. I still need to write an introduction and a conclusion, but I'd like to hear from people if the doc is clear and useful, and if not what I can do to improve it. I hope to be writing some more of these for various parts of Twisted in the near future, if people do indeed find them useful.

Tuesday, April 5, 2005

socket.recv -- three ways to turn it into recvinto

Inspired by http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/408859

While some people are busy worrying about how to make Python's builtin sockets less efficient, one might be wondering if the reverse is possible - how do you make them more efficient? After all, you usally want your program to run more quickly, or tax your CPU less heavily, or consume fewer resources, not the reverse. Fortunately, I have just the solution for you1. The approach explored below will be to avoid allocating new memory when reading from the socket. Since malloc() is a relatively expensive operation, this will save us a bunch of CPU time, as well as saving us memory by reducing chances for heap fragmentation and so forth.

  1. Solution the first: readinto

    exarkun@boson:~$ python
    Python 2.4.1 (#2, Mar 30 2005, 21:51:10)
    [GCC 3.3.5 (Debian 1:3.3.5-8ubuntu2)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import socket, array, os
    >>> s = socket.socket()
    >>> s.bind(('', 4321))
    >>> s.listen(3)
    >>> c, a = s.accept()
    >>> buf = array.array('c', '\0' * 50)
    >>> os.fdopen(c.fileno()).readinto(buf)2
    >>> buf.tostring()
    >>> c.recv(10)
    Traceback (most recent call last):
    File "", line 1, in ?
    socket.error: (9, 'Bad file descriptor')

    As you can see, the handy readinto method of file objects can be used to provide a pre-allocated memory space for a read to use. Unfortunately, it is a file method, not a socket method (also, its documentation recommends strongly against its use, though I can't imagine why!). We can get around this, though, since a file descriptor is just a file descriptor. os.fdopen will happily give us a file object wrapped around the socket we're really interested in. Then it's a simple matter of calling readinto on the resulting file object with an array we have previously allocated.

    "Great!" you say. "Why even bother with the other two examples?" you wonder. Well, there are a few problems. Even if we accept the os.fdopen hack, and even if we do not let the strong words in the file.readinto docstring dissuade us, there's still a tiny problem. file.readinto closes the file descriptor before returning! Damn, there goes our socket. Maybe the next solution will fare better.

  2. Solution the second: recv(2)
    Okay, that stuff with file.readinto was just silly. Let's get serious here. libc already provides the functionality we need here, and has for decades. This is basic BSD sockets 101. Stevens would cry (if he were still with us) if he saw us doing anything else. So let's cut the funny business and just do what a C programmer would do: call recv.

    exarkun@boson:~$ python
    Python 2.4.1 (#2, Mar 30 2005, 21:51:10)
    [GCC 3.3.5 (Debian 1:3.3.5-8ubuntu2)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import dl
    >>> libc = dl.open('libc.so.6')
    >>> import socket, array
    >>> s = socket.socket()
    >>> s.bind(('', 4321))
    >>> s.listen(3)
    >>> c, a = s.accept()
    >>> buf = array.array('c', '\0' * 50)
    >>> libc.call('recv', c.fileno(), buf.buffer_info()[0], 50, 0)
    >>> buf.tostring()
    >>> libc.call('recv', c.fileno(), buf.buffer_info()[0], 50, 0)
    >>> buf.tostring()

    Sweet. We open libc so we can call recv in it, create a socket as usual, and another array object to act as our pre-allocated memory location. Note we use the buffer_info method this time, because recv() does not expect a "read-write buffer object" (like file.readinto did), but a pointer to a location in memory, which is exactly what buffer_info()[0] gives us. Then we just call recv. Easy as eatin' pancakes. We can even do it twice, demonstrating that recv isn't doing anything ridiculous, like closing the socket for us (I did it with the same array object, overwriting the previous contents, demonstrating that our no-allocation trick is working just fine).

    I know what you're thinking, though. array objects? What the hell can you do with an array object? Well, here's what. All kinds of stuff! Why, you can build one from a string. Or build a string from one. Or, uh, swap the byte order... umm, oh yea you can reverse them too. Cool deal, eh? Err, no, maybe not actually... None of those cool string methods are around, unfortunately. You can create a string from the array but that kind of defeats the purpose... in doing so you've just allocated a pile of memory. Nuts. Well, wait, don't give up yet, we may be able to improve upon this situation...

  3. Solution the ultimate: recv(2) (uh yea, again).

    The only problem we really have with recv isn't actually with recv: it's with array! Let's not throw the baby out with the bathwater, then. Solution: drop array, keep recv. We want a string. Well, let's use a string.

    exarkun@boson:~$ python
    Python 2.4.1 (#2, Mar 30 2005, 21:51:10)
    [GCC 3.3.5 (Debian 1:3.3.5-8ubuntu2)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import socket, dl
    >>> libc = dl.open('libc.so.6')
    >>> s = socket.socket()
    >>> s.bind(('', 4321))
    >>> s.listen(3)
    >>> c, a = s.accept()
    >>> buf = '\0' * 50
    >>> libc.call('recv', c.fileno(), id(buf) 20, 50, 0)
    >>> buf

    It's the perfect solution. No wasted memory allocation, but the same level of convenience as a normal call to socket.recv. Rarely are we lucky enough to find such elegant and flawless solutions in computer science. The astute reader might object to the magical 20 in the recv call as being inelegant or flawed, however the value can easily be computed at runtime. The code to do so is extremely simple and only omitted because it slightly too large to fit in the margin.

So there you have it. Happy networking.

1 Sorry, it's way too late to post something useful. Especially when I could post something fun instead.
2 Note: in each example where socket IO occurs, I have launched telnet in another terminal and type in some random bytes.