Monday, October 24, 2005

Conch 0.6.0 is out

Twisted Conch is primarily an SSHv2 implementation, but also includes implementations of telnet and vt102. Each protocol comes with client and server implementations, in fact, Conch provides an SSH server right out of the box (configured with Twisted mktap, started with Twisted twistd, key files created with Conch's ckeygen):

exarkun@boson:~/Scratch/Run/conch$ ssh localhost
ssh: connect to host localhost port 22: Connection refused
exarkun@boson:~/Scratch/Run/conch$ ckeygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/exarkun/.ssh/id_rsa): ssh_host_key
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ssh_host_key
Your public key has been saved in
The key fingerprint is:
exarkun@boson:~/Scratch/Run/conch$ mktap conch --data .
exarkun@boson:~/Scratch/Run/conch$ sudo twistd -f conch.tap
exarkun@boson:~/Scratch/Run/conch$ ssh localhost
The authenticity of host 'localhost (' can't be established.
RSA key fingerprint is fa:d7:3e:a3:b3:2b:c9:c2:d5:9b:31:7d:47:3a:96:62.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
exarkun@boson:~$ ps
29747 pts/2 00:00:00 bash
29760 pts/2 00:00:00 ps
exarkun@boson:~$ logout
Connection to localhost closed.
exarkun@boson:~/Scratch/Run/conch$ tail -n 2 twistd.log
2005/10/24 20:06 EDT [SSHServerTransport,0,] avatar exarkun logging out (0)
2005/10/24 20:06 EDT [SSHServerTransport,0,] connection lost

Conch provides APIs for implementing custom SSH servers and clients as well, of course. For example, with it, you can easily write an SSH server that allows logins to a shell other than a unix login shell, such as a control application with a customized interface, or a text adventure game. Insults, a VT102 (plus a bit) implementation, plays a support role here, easing the task of writing custom interfaces by providing terminal and text attribute manipulation functions (eg, moving the cursor around and changing the color of text).

New in this release is an experimental windowing library based on Insults which provides such widgets as TextInput, Button, ScrolledArea, HBox, and VBox. This library can be used to very easily throw together rather complex user interfaces to be exposed, either via SSH or telnet. Those of you who have been following my blog will have seen some screenshots of this library in action.

Grab the 0.6.0 release or visit the project website.

Friday, October 21, 2005

Get your chat on

Twisted Words 0.3.0 has just been released! Twisted Words provides implementations of a handful of IM protocols, including IRC, MSNP8, OSCAR, TOC, and Jabber. Twisted Words also comes with a multi-protocol server built around a few simple interfaces, the goal of which is to facilitate implementations of novel servers, clients, and bots. Out of the box, Twisted Words comes with server that accepts connections over IRC and PB, and seeks out installed third-party plugins to support other interfaces (for example, the Nevow LivePage web-based interface currently sitting in one of my sandboxes.

Since the last release, the server infrastructure has been greatly simplified, documentation improved, extraneous code removed, and test coverage increased. Jabber support has also grown immensely. Ralph Meijer has taken over maintenance and is driving new feature development. Additionally, twisted.xish has moved to twisted.words.xish, since it is purely a support module for Twisted Words Jabber support. As far as I know, no one outside of Twisted is using Xish directly: if you were, I'm sorry, if you come over to my house I will make you a cheese sandwich to make up for it. Update your code to import from the new location, or stop using it, or keep the last release around.

I think Twisted Words has some exciting development in its near future. Stay tuned. You can download Twisted Words 0.3.0 or visit the project page.

Bad changeset

Possibly not the best commit ever.

Thursday, October 20, 2005

New Twisted Mail Release: 0.2.0

Twisted Mail provides client and server implementations of three protocols: SMTP, POP3, and IMAP4. These differentiate themselves from the Python standard library implementations both by presenting a much higher-level, easy-to-use interface and in their server components which allow the implementation of custom servers for each protocol without dealing with protocol-level issues.

Twisted Mail includes a simple, out-of-the-box email server which accepts messages over SMTP, stores them in a Maildir arrangement, and can serve them to clients over POP3. This is intended primarily as a demonstration of the use of the protocol implementations in Twisted Mail, not as a production-quality mail server. While it performs fairly reliably under normal operation, almost no effort has been expended towards making it robust against failures in general.

Twisted Mail also comes with a simple /usr/sbin/sendmail replacement, mailmail. Rather than using a custom schema to communicate messages to an MTA, mailmail connects to localhost on port 25 and gives the message to whatever server happens to be listening there.

This is a minor feature enhancement release. The IMAP4 and POP3 client implementations have both gained methods for explicitly negotiating TLS options. The POP3 client now facilitates command timeouts and has gained support for the POP3 commands NOOP, RSET, and STAT.

Check out the Twisted Mail project page or download the release.

Wednesday, October 19, 2005

Twisted Names 0.2.0 Released

Twisted Names is a DNS library for both clients and servers. This is primarily a bug-fix release. Several cases where TCP connections were left hanging have been fixed. Use of deprecated Twisted APIs have also been removed. There is improved test coverage of zone transfer support.

Grab the latest release or visit the project page. You might also want to file a bug report. ;)


Tuesday, October 18, 2005

Mantissa Application Server - Release 0.1!

Twisted, Xapwrap, Epsilon, Vertex, Axiom, Nevow... What do you get if you put them all together? Let me introduce you to Mantissa.

Mantissa combines and builds upon these libraries to create a coherent deployment target for your software.

The focus of this Mantissa release is support for the web: there are themes, conventions for template lookup, support for Nevow LivePage, and a framework into which web interfaces to applications can be dropped. A major feature is the ability to deploy otherwise unrelated applications into the same server, provide access to them using the same user accounts, have them cooperate to display elements of the page which each require, and give the user an organized view onto them.

As the kids like to say, Mantissa is skinnable: the application logic it represents can be displayed in whatever manner HTML allows (and a few beyond that ;) and the display is controlled completely using CSS and XHTML. In addition to dynamic applications, static content is also supported: plugins for axiomatic are provided to allow reorganization of static content on the fly, but of course it can also be laid out automatically by the application-code which initializes a Mantissa instance.

Functionality exposed through URLs is, by default, hardened against cross-site scripting and other attacks: unless URLs are otherwise specified, they will be stable for a particular user, but randomized between different user accounts. Mantissa also provides a highly hookable ticketting and signup-system. By default, users first verify an email address (acquiring a ticket in the process) before gaining access to the system. Ticket issuance can be customized, and new Benefactors can be defined to endow newly created users with application-appropriate abilities. As a demonstration of the LivePage features, a sample Mantissa configuration is included in the 0.1 release which has a single administrator user: the administrator has access to an interactive Python interpreter prompt after logging in.

So there's some web in it. What else?

Mantissa provides a framework for indexing an searching as well. Applications provide the content, Mantissa arranges the indexing, and the user gets a unified searchable view of all content through a single interface.

And how do you feel about VoIP? Mantissa includes Divmod's SIP implementation. Ideas for applications involving flexible control of real-time voice channels are left as an exercise for the reader.

Building on Vertex, Mantissa applications can also rely on Q2Q, either to accept connections and provide a service, or for making outgoing connections to other users or domains. Yea: Mantissa applications can trivially communicate with applications (perhaps Mantissa-based, perhaps not) running on other hosts.

Upcoming releases will polish the web support, probably switch over to Athena, or at least add support for it, and start adding generally useful widgets (ie, a pager for large data sets), but also give some attention to non-HTTP areas (email and ssh will probably show up next).

Get the Mantissa 0.1 release, visit the Mantissa project page, or peruse the Mantissa wiki.

Monday, October 17, 2005

Emergency Axiom 0.1

An object-relational (sort of) database, implemented in Python, backed by SQLite, and informed by the development of Atop, and others: Axiom is still young (hardly three months have gone by since its conception), but it already supports a significant portion of the features present in the databases which shaped its development — and has for longer than not.

Axiom's goal is to provide an efficient, non-SQL interface to an existing relational database. Unlike other libraries in this area, Axiom discourages inefficient operations (such as loading a single object at a time - possibly building it up from multiple queries against different tables): instead it encourages a style where many objects are loaded with a small, often fixed, number of queries (sometimes the number being fixed at one). This lets the underlying relational database do what it's good at: yank data into memory out of underlying storage, perhaps based on some limiting criteria, or sorted in a particular way. From the Python side, you get genuine Python objects with Python-native attributes: Axiom is object-based, not table-, column-, or row-based. Objects stored in an Axiom database can refer to any other object stored in the same database (regardless of type - just like normal Python references), and attributes can be instances of any type Axiom natively supports (currently booleans, integers, byte strings, character strings, or timestamps). Additionally, support for new attribute types can be added by defining new attribute classes. Objects stored in an Axiom database provide a schema definition (as part of their class definition) which allows in-memory constraints to be enforced, just as they will be in the database, as well as providing necessary information for the upgrade mechanism (ie, you can change your schema without invalidating an existing database).

Of course, Axiom also integrates with common Twisted practices:

  • an Axiom store is also a Twisted IServiceCollection provider. A Store can be started and stopped, each of which propagates the appropriate event down to any in-database objects which care to receive them. This allows (if you will excuse the extremely hypothetical example) a web service to be implemented as an Axiom-stored object: when the object receives startup notification, it can bind port 80 and start handling requests; when it receives shutdown notification, it can close the port and delay actual shutdown until it has given each outstanding request a fair chance to be completed. By letting services exist as database objects, any database manipulation tools (such as applications or user- or administrative-configuration interfaces can persistently alter application configuration - goodbye, configuration files).

  • Axiom comes with an implementation of Twisted's IRealm interface. This means an Axiom store can be given to a portal to allow authentication and authorization against it, using the standard Twisted Cred APIs. Any Twisted protocol implementation which supports Twisted Cred (so, SMTP, POP3, IMAP4, SIP, HTTP, IRC and PB to name a few) will work with an Axiom Store backend.

  • Axiom makes heavy use of plugins in several ways. Features for an Axiom database can be provided and configured via a particular interface: IAxiomaticCommand. Once a plugin is installed and its Python modules are importable, it will be available to the database administrator (be that a sysadmin on the command line or an application admin via a web interface). Objects in the database can also act as powerups for other objects in the database. Powerups provide additional functionality in a modular and pluggable way, and can be queried for in a simple and efficient manner.

  • Axiom includes a wall-clock scheduler as well, made available to application-level code as a powerup on an Axiom Store. By loading up this powerup and calling its methods (such as the surprisingly-named schedule), application-level code can arrange to have its own code invoked at a particular point in the future, be it seconds, hours, or months away.

These features make it very straightforward to use Axiom in a Twisted application.

Currently, Divmod has several applications in development which use Axiom, and several more in the pipeline. Though Axiom is far from complete, even in its current form it has demonstrated its utility for a variety of applications, both client- and server-side.

It's difficult to include a demonstration of Axiom's abilities here: only the most trivial of database applications can reasonably fit into a blog post, yet such examples are rarely very interesting or informative. Instead, here's a brief transcript of an interaction (in which I create myself a new user account and disable an existing account) with a database being used by a running server, using the command line tool included in Axiom, axiomatic:

$ axiomatic --dbdir cc.axiom/ userbase list | grep exarkun                        
$ axiomatic --dbdir cc.axiom/ userbase create exarkun
Enter new AXIOM password:
Repeat to verify:
$ axiomatic --dbdir cc.axiom/ userbase list | grep exarkun
$ axiomatic --dbdir cc.axiom/ userbase --help
Usage: axiomatic [options] userbase [options]
--help Display this help and exit.
install Install UserBase on an Axiom database
create Create a new user
disable Disable an existing user
list List users in an Axiom database

$ axiomatic --dbdir cc.axiom/ userbase disable exarkun
$ axiomatic --dbdir cc.axiom/ userbase list [DISABLED]
userbase is a plugin included with Axiom for user management; the axiomatic script pulls it in and makes it available to me on the command line, making the above possible.

Download Axiom 0.1, visit the project page on, or check out the Axiom wiki page.

Friday, October 14, 2005


Divmod is proud to announce the release of Vertex 0.1. Vertex is an implementation of the Q2Q protocol (sort of like P2P, but one better). There are a few moving parts in Vertex:

  • PTCP: a protocol which is nearly identical to TCP, but which runs over UDP. This lets Q2Q penetrate most NAT configurations.
  • JUICE ([JU]ice [I]s [C]oncurrent [E]vents): a very simple but immensely flexible protocol which forms the basis of the high-level aspects of Q2Q
  • vertex: a command line tool which exposes a few features useful in many situations (such as registration and authentication)

Q2Q is a very high-level protocol (alternatively, transport) the goal of which is to make communication over the internet a possibility (if you enjoy setting up tunnels or firewall rules whenever you need to transfer a file between two computers, Q2Q may not be for you). Q2Q endpoints aren't hardware addresses or network addresses. They look a lot like email addresses and they act a lot like instant message addresses. You can hook into yours wherever you can access the internet, and you can be online or offline as you choose (Q2Q supports multiple unrelated protocols, so you also might be online for some services but offline for others). Two people with Q2Q addresses can easily communicate without out-of-band negotiation of their physical locations or the topology of their networks. If Alice wants to talk to Bob, Alice will always just open a connection to If Bob is online anywhere at all, the connection will have an opportunity to succeed (Bob might be busy or not want to talk to Alice, but that is another matter ;). The connection is authenticated in both directions, so if it does succeed Alice knows she is talking to the real Bob and vice versa.

The Q2Q network has some decentralized features (there is no one server or company which can control all Q2Q addresses) and features of centralization (addresses beneath a particular domain are issued by a server for that domain; once issued, some activities require the server to be contacted again, while others do not). Vertex includes an identity server capable of hosting Q2Q addresses. Once you've installed the 0.1 release, you can run it like this:

exarkun@boson:~$ cat > q2q-standalone.tac
from vertex.q2qstandalone import defaultConfig
application = defaultConfig()
exarkun@boson:~$ twistd -noy q2q-standalone.tac
2005/10/15 00:12 EDT [-] Log opened.
2005/10/15 00:12 EDT [-] twistd 2.0.1 (/usr/bin/python2.4 2.4.1) starting up
2005/10/15 00:12 EDT [-] reactor class: twisted.internet.selectreactor.SelectReactor
2005/10/15 00:12 EDT [-] Loading q2q-standalone.tac...
2005/10/15 00:12 EDT [-] Loaded.
2005/10/15 00:12 EDT [-] vertex.q2q.Q2QService starting on 8788
2005/10/15 00:12 EDT [-] Starting factory <Q2QService 'service'@-488b6a34>
2005/10/15 00:12 EDT [-] vertex.ptcp.PTCP starting on 8788
2005/10/15 00:12 EDT [-] Starting protocol <vertex.ptcp.PTCP instance at 0xb777884c>
2005/10/15 00:12 EDT [-] Binding PTCP/UDP 8788=8788
2005/10/15 00:12 EDT [-] vertex.q2q.Q2QBootstrapFactory starting on 8789
2005/10/15 00:12 EDT [-] Starting factory <vertex.q2q.Q2QBootstrapFactory instance at 0xb77787cc>

You can acquire a new Q2Q address using the vertex command line tool:

exarkun@boson:~$ vertex register exarkun@boson password

boson is a name on my local network, making this address rather useless. On the other hand, no one will be able to pretend to be me by cracking my hopelessly weak password ;) If you set up a Vertex server on a public host, you will be able to register a real, honest-to-goodness Q2Q address beneath its domain (careful - so will anyone else).

vertex also offers a tool for requesting a signed certificate from the server. These certificates can be used to prove ones identity to foreign domains without involving ones home server. Another feature vertex provides is a toy file transfer application. Bob can issue a vertex receive while Alice issues a vertex send pointed at him, and the file will be transferred.

Much of the real power of Q2Q is exposed to developers using two methods: listenQ2Q and connectQ2Q. These work in roughly the same way Twisted's listenTCP and connectTCP work: they offer support for writing servers and clients that operate on the Q2Q network.

Thursday, October 13, 2005

Guess What! Divmod Nevow 0.5 Release!

It's been a long time since one of these (very roughly, 17660815 seconds. For those of you following along at home:

>>> from epsilon import extime
>>> then = extime.Time.fromISO8601TimeAndDate('2005-03-23T18:47:39')
>>> now = extime.Time()
>>> now.asPOSIXTimestamp() - then.asPOSIXTimestamp()
), and we've got a lot of great new stuff to show for the wait. Skipping a lot of things, because they'd make this post way too long, we've got:
  • We've given LivePage a significant overhaul. We're not done in this area yet, though: watch for changes here in the next release.
  • A lot of code has been updated to work with zope.interface and Twisted 2.0. Nevow 0.5 requires each of these.
  • Formless no longer adds __nevow_carryover__ to the redirect URL unless it actually needs to carry values over to the next request.
  • Improved documentation and more example applications, including examples for:
    • Working with Fragments
    • Using macros
    • Integrating with Axiom
  • A JavaScript Object Notation serialization and parsing has been added, to allow more complex objects (still no duplicate references or user-defined classes, though) to be used with LivePage
  • LiveEvil, the previous implementation of LivePage, has been removed
  • Guard has been expanded to more easily support persistent (eg, saved in a database) sessions
  • Bugs have been fixed in a lot of areas:
    • VirtualHostList
    • Behavior relating to quoting of '/' in URL segments
    • Corrected XML namespace prefixes in generated output
    • The default 404 page is now well-formed XML
    • The radio input now correctly selects the default value and its display elements line up in a more aesthetically pleasing manner
    • Remove duplicate request-finishing in the unhandled exception error path
    • Fixed macro/xmlfile incompatibility
    • Fixed a huge object/memory leak in applications that did not re-use all their Page instances to serve each request
    • Fixed rendering of Fragments so as not to interefer with their parent's rendering process
    • Fixed rare Deferred/tag/slot-filling interaction which caused some slots to be filled with empty values instead of useful content

A word about livepage

Nevow 0.4.1 provided a better-than-ajax feature known as LivePage, via the nevow.livepage.LivePage class. This was a replacement for an older implementation, nevow.liveevil.LiveEvil, which was deprecated in 0.4.1. Nevow 0.5 removes LiveEvil and augments LivePage with a bunch of new features, but also some API incompatibilities. Where possible, backwards compatibility has been maintained, but it is likely that any extensive LivePage applications will be broken at least a little bit by 0.5. Of course, there's a lot of other changes in Nevow that will probably jar such applications anyway. Most people whom this affects have been in the IRC channel or on the mailing list and already know this.

What's not as well known yet is that the next release of Nevow will likely provide still another new implementation of the same idea. This time nevow.athena.LivePage will be the preferred API. Where the LivePage implementation in 0.5 encourages the definition of JavaScript functions in the server code using Python, nevow.athena.LivePage encourages a much stricter separate of client and server concerns. It also provides an API for asynchronous data passing between client and server (initiated by either side), rather than the current API for passing code from the server to the client to be executed.

Another area of upcoming change will be form generation. formless is capable of some wonderful things, but some of its limitations are becoming clearer. A simpler system is just around the corner. Look for it in the next release -- which will be in much less than 17660815 seconds from now.

Download Nevow 0.5 or visit the project page

Wednesday, October 12, 2005

Divmod Xapwrap Released

Xapian is a fairly popular full-text indexing system ("Probabilistic Information Retrieval library"). It's got Python bindings, but they're not so fun to use. Xapwrap is a layer on top of these bindings which tries to simplify matters:

exarkun@boson:~/xapwrap-demo$ cat > a-m
exarkun@boson:~/xapwrap-demo$ cat > n-z
exarkun@boson:~/xapwrap-demo$ python
Python 2.4.2 (#2, Sep 30 2005, 21:19:01)
[GCC 4.0.2 20050808 (prerelease) (Ubuntu 4.0.1-4ubuntu8)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from xapwrap import index, document
>>> idx = index.Index(name='demo.db', create=True)
>>> idCounter = 1
>>> docIDs = {}
>>> for docName in 'a-m', 'n-z':
... docIDs[idCounter] = docName
... text = document.TextField(file(docName).read())
... idx.index(document.Document(textFields=[text], uid=idCounter))
... idCounter += 1
>>> res ='zoo')
>>> print res
[{'score': 100, 'uid': 2}]
>>> print docIDs[res['uid']]
[{'score': 100, 'uid': 1}]

Get the Xapwrap 0.3.0 release.

Tuesday, October 11, 2005

Divmod Epsilon Released

In the beginning Guido created Python. Now Python was formless and empty, duplicate code was over the surface of programs. And Guido said, Let there be modules and there were modules. Guido saw that the modules were good, and he separated them from the programs. Guido called them the standard library. And there was the library, and there was the program -- the first abstraction.

Now, not all of us are lucky enough to be able to toss things into the Python stdlib. The practice has arisen in Twisted to take functionality that is generally useful -- not specific to Twisted -- and place it in the twisted.python package. There's an option parser and file path manipulation functions and so forth. This works okay for Twisted development, but the code can't really be re-used without creating a Twisted dependency (which is okay by me, mind you, but I hear it's inconvenient for some other people). Later, at Divmod, a similar derivative practice became common. Common functionality that wasn't really specific to our projects was aggregated in a module named atop.tpython -- the tpython being short for, of course, twisted.python. So now we're two levels abstracted from where we really want to be.

Well, a new day is upon us. Divmod has now released Epsilon, a package for some generally useful code that others might find handy. This first release, 0.1.0, has two major offerings and a few more minor ones.

First, there's epsilon.extime, a module which offers conversion to and from a variety of time formats: RFC 2822 formatted dates, POSIX timestamps, datetime objects, struct_time objects, ISO8601 formatted timestamps, and a particular context-aware format known as the "human readable format". Additionally, it offers a Time class which can be constructed from or converted to any of these formats, as well as have certain date math performed on it. Here's a brief sample:

>>> from epsilon.extime import Time
>>> Time().asDatetime()
2005, 10, 12, 4, 52, 8, 237248,
tzinfo=<epsilon.extime.FixedOffset object at 0x-4860a234 offset datetime.timedelta(0)>)
>>> Time().asISO8601TimeAndDate()
'04:52 am'
>>> import datetime
>>> (Time() + datetime.timedelta(days=2)).asHumanly()
'14 Oct, 04:53 am'

Next up is epsilon.cooperator. This provides a scheduler for tasks implemented as iterators. It offers full Deferred support and hooks for scheduling behavior. In particular, it can be used when it's possible for so many tasks to be running concurrently that to allow them all to set their own pace would damage interactive performance: here it slows things down and makes sure new events can be processed. With a little creativity, it can also be used to group or prioritize tasks.

The remaining features include a setup hook for Twisted applications which install plugins to cause the distutils process to regenerate the plugin cache, a class for simplifying providing multiple listeners with the result of a Deferred, and a structured Version class with special SVN integration (for example,

$ python -c 'from epsilon import version; print version'
[epsilon, version 0.1.0 (SVN r529)]

showing that I have something at least as new as Epsilon 0.1.0 installed, and it is being imported from an SVN checkout at revision 529).

Divmod's Wiki has an Epsilon project page at

Wednesday, October 5, 2005

Faucet reborn

I only bothered to hook it up to a lame chat application, but the widget is demonstrated. Tomorrow I may have a go at hooking it up to an actual game. Faucet's classic use was for a text adventure and had the top-left pane used for a description of your location, the top-right pane used for a list of what objects were present, and the bottom pane used to display command input and the responses they generated, as well as other game output which arrived asynchronously.

That's probably it for screenshots for a while. If you're really interested, the code's in Twisted (mostly in trunk, a little bit in a branch - exarkun/scrollarea). Check it out. Write some cool widgets.

Tuesday, October 4, 2005

Improved Rendering of Scrolling Elements

The new ScrolledArea is highlighted in red below. Click through for a larger version!

With this done, I think it's time to turn to an actual application. Tomorrow night: (What's faucet? You asked.)

Monday, October 3, 2005

More New Insults Widgets

Added Viewport, HorizontalScrollbar, and VerticalScrollbar tonight. Combine these with the existing HBox, VBox, and Border and you have a reasonable approximation of a generic scrolled area widget. I'll probably do an actual ScrolledArea later this week, since rendering things as a collection like that is slower and not as pretty as it could be. However, I think the concept is proven. Screenshots for you, today:

Sunday, October 2, 2005

New Insults Widgets

PasswordInput and TextOutputArea. PasswordInput is like TextInput (you can guess how it's different). TextOutputArea lets you render a big chunk of text, either wrapping long lines or truncating them.

doc/conch/examples/window.tac has been updated to demonstrate both of these. Particularly satisfying is watching the display as the terminal is resized...