watch this The wheels are turning, slowly turning. home
Axiom Concurrency 2006-03-08

Over the last several days, Axiom’s concurrency story has come together quite a bit. This is something Divmod has put off, fairly confident that it would be easy once the time came. And hey, we were actually right (that’s always fun). Quotient’s fulltext indexer is now implemented as a second process which is automatically brought up and down with the main server and which is handed work to do through the Axiom database. This is done in a generic fashion with Axiom framework code doing most of the heavy lifting (like repeatedly adding one to an integer and comparing two numbers to see which is greater), so the actual application-level code in Quotient ends up doing something like this (actually, exactly this) in order to take advantage of the multi-process aspect of the system:

    def installOn(self, other):
        super(SyncIndexer, self).installOn(other), iaxiom.REMOTE)

    def indexMessage(self, message):
        # Do the actual full-text indexing of `message'
    processItem = indexMessage

The code barely even has to be aware of the fact that anything is going on in a separate process: it simply declares its interest in MessageSource and the fact that it has no objection to a potentially esoteric execution context. Any application can take advantage of this feature just as easily to split computationally expensive or high-volume disk-bound tasks into a separate operating system process, where they will not interefere with code that is latency-sensitive (like a network server) and where they can take advantage of additional CPU or disks, providing for greater parallelization of the task.

Of course, like any user of the reliable listener system, be they local or remote, the indexer:

All this is due to the fact that the fulltext indexer is not directly responsible for finding messages to index, but has merely subscribed to a source of messages, and it is up to that subscription to decide what and when messages are given to it to be indexed.