Tuesday, August 21, 2007

Twisted Trial performance improvements

Trial, Twisted's xUnit(-esque) testing package (library, discovery tool, runner, etc), has long inherited its overall testing flow from the Python standard library unittest module. It's quite simple, actually: iterate over a sequence of test cases and invoke each one with the result object. Lately, this has actually led to some noticeable problems with its performance. Creating a test case instance for each test method isn't extremely unreasonable, but in the process of running them all, more and more objects are created and the process balloons to an unreasonable size. The Twisted test suite typically uses about 800MB of RAM by the time the last test method is run.



So, in order to be able to run the Twisted tests on machines without 800MB of free memory, we changed our TestSuite so that it drops each test after it runs it. The suite now takes about one quarter as much memory to run. As an unexpected bonus, it also runs almost 50% faster (66% for trial --force-gc which calls gc.collect in between each test method). I can only explain the speedup as time saved inside the garbage collector itself due there being far fewer objects to examine (this is not a completely satisfying explanation, but I cannot think of a better one).



If you're using trial to run your test suite, you may notice reduced memory requirements and reduced overall runtime, too. :)

Friday, August 17, 2007

Typespeed v0.4.4 Best score was: Rank: ...


Typespeed v0.4.4

Best score was:

Rank: Computer
Score: 1187
Total CPS: 8.492
Correct CPS: 8.352
Typo ratio: 1.6%
Typorank: Secretary


exarkun@charm:~$

Thursday, August 16, 2007

Child process improvements in Twisted

Thomas Hervé merged his process/pty-process branch into Twisted trunk today. He's been working on integrating the two separate implementations of POSIX child process creation in Twisted, one for child processes connected to a PTY and one for child processes connected to a set of pipes. A long time ago, someone copied an entire class full of code and started twiddling it to support PTYs. This led to a lot of duplication, obviously, and some bugs where the two classes diverged.



With this re-unification, some behavior which was inconsistent between the two variations has been removed, some bugs which were present in one or the other of the two classes have been fixed, and a lot of duplicate code has been completely eliminated. Best of all, there is now some actual unit test coverage for this code (it was originally developed long before Twisted's current high testing requirements) and a test double which will make it easier to write more and better unit tests for this code in the future.



One other piece of fall out is that on Windows, if application code tries to signal an already-exited process, an exception will be raised indicating that it is not possible to do so. This brings Windows process support closer still to POSIX process support.

Thursday, August 9, 2007

It's Official

http://www.guardian.co.uk/environment/2007/aug/08/endangeredspecies.conservation

Unfortunately, this is hardly surprising. It's been pretty clear for at least a decade that this would be the outcome. Maybe the saddest part is that it shouldn't have been inevitable.

Saturday, August 4, 2007

Side-by-side Twisted plugins installation issue fixed


Since its introduction a little over two years ago, the "new" Twisted plugin system has had a minor bug which manifest whenever two or more separate copies of a plugin packaged were available (ie, in sys.path). Normally, the first of these should obscure the second. However, the plugin system would allow both to be discovered. This could lead to confusing results in some cases.




This bug is now fixed in trunk@HEAD: only the first plugin package on sys.path will be considered for plugins. Plugin directories (not packages) will still be considered.

Thursday, August 2, 2007

The Internet Is Filled With Wonderful Things

You were probably just wondering about these effects, anyway. Of course, I don't see the text online anywhere, so you'll have to keep wondering.

Wednesday, August 1, 2007

Twisted system event triggers fix

One of the older, darker corners of Twisted, system event triggers (part of the core reactor interface) have long had slightly poor handling of some corner cases. Fortunately most use of these triggers is via the application service system which avoids all of these cases, but there is another startup notification API through which one might have encountered some weird behavior. As of now, a bug which could cause a trigger to not be executed has been fixed. Code which triggered this behavior was probably buggy itself, since it was asking the reactor to do something impossible; it will receive a warning from now until we turn it into a real exception, a couple releases from now.