Over the last several days, Axiom’s concurrency story has come together quite a bit. This is something Divmod has put off, fairly confident that it would be easy once the time came. And hey, we were actually right (that’s always fun). Quotient’s fulltext indexer is now implemented as a second process which is automatically brought up and down with the main server and which is handed work to do through the Axiom database. This is done in a generic fashion with Axiom framework code doing most of the heavy lifting (like repeatedly adding one to an integer and comparing two numbers to see which is greater), so the actual application-level code in Quotient ends up doing something like this (actually, exactly this) in order to take advantage of the multi-process aspect of the system:

    def installOn(self, other):
        super(SyncIndexer, self).installOn(other)
        self.store.findUnique(mail.MessageSource).addReliableListener(self, iaxiom.REMOTE)

    def indexMessage(self, message):
        # Do the actual full-text indexing of `message'
    processItem = indexMessage

The code barely even has to be aware of the fact that anything is going on in a separate process: it simply declares its interest in MessageSource and the fact that it has no objection to a potentially esoteric execution context. Any application can take advantage of this feature just as easily to split computationally expensive or high-volume disk-bound tasks into a separate operating system process, where they will not interefere with code that is latency-sensitive (like a network server) and where they can take advantage of additional CPU or disks, providing for greater parallelization of the task.

Of course, like any user of the reliable listener system, be they local or remote, the indexer:

• 
  will be run as quickly as the system can manage, taking into account the interactive requirements being placed on the server as well as the other reliable listeners which have been created and are also vying for runtime;
  
• 
  can be throttled back if another task takes precedence;
  
• 
  can be monitored and reported on for user feedback or administrator introspection;
  
• 
  can be prioritized relative to other kinds of listeners;
  
• 
  can be reset entirely and allowed to re-process all old messages (useful when, for example, the fulltext index becomes corrupt or is replaced by a superior system);
  
• 
  or just messages which caused an error and due to code changes may now be processable without one.
  

All this is due to the fact that the fulltext indexer is not directly responsible for finding messages to index, but has merely subscribed to a source of messages, and it is up to that subscription to decide what and when messages are given to it to be indexed.