- Jinja2 is at http://jinja.pocoo.org/, not the URL in the BuildBot docs (perhaps fixed by https://github.com/djmitche/buildbot/commit/851b8d74141d2dbf8351e67cf8f3e3c1bb7bbc2e)
- simplejson was already installed, but BuildBot's "setup.py install" easy_installed another copy of it anyway
- buildbot.status.html.Waterfall is really gone now
- The "path_to_root" method of buildbot.status.web.base.HtmlResource is gone, but there is a free function of the same name in that module that does the same thing
- The HtmlResource method to override to generate content is now "content" not "body" and takes an extra "context" argument (perhaps fixed by https://github.com/buildbot/buildbot/commit/bf9e03061f9c3442b2740addaee464b0dd4b678c)
- The build history format has been upgraded again, meaning that the master will now spend the majority of its time loading old pickles, upgrading them in memory, using them, then throwing them away so that it has to do the work all over again next time (perhaps fixed by https://github.com/buildbot/buildbot/commit/95f33ad1bd65b5fa06e0fc4614830b1cb04d52db)
- Some unpredictable quantity of build history from before upgrading will still appear in the web status views, but not all of it
- public_html/index.html will be moved to templates/root.html by the master upgrader
Wednesday, November 24, 2010
Saturday, October 23, 2010
pygame's API for initializing a display has a couple attractive sounding flags. Or at least, they sound attractive once you notice that updating your 320x240 window at 30fps is consuming all available cycles on your brand new Intel i9 CPU. They're
DOUBLEBUF. Hardware surfaces and double buffering is how you make graphics fast, right?
Well... no. You probably can't get a hardware surface anyway, and double buffering is unlikely to help until you figure out how to have a window larger than 320x240 anyway.
What you really need to do to get reasonable rendering performance is make sure that any images you blit to the display have the same color depth. You can do this with the
Surface.convert method (you get back a Surface when you load an image from a file, eg a png or a gif). Check the depth of your display surface and then convert your image surfaces to that depth (and just keep the converted versions). Blitting them will get hundreds of times faster.
It's a pretty simple thing, but it's easy to get distracted by
HWSURFACE and not notice the depth mismatch (like I did for three days).
Friday, October 22, 2010
As such an old project, I wrote it before I knew about things like unit tests, so even if I were interested in maintaining an IRC bot anymore, it would be easier to start over than try to resuscitate pynfo. Thus, pynfo, I consign you to oblivion. Rest in peace:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
exarkun 5621 8.9 12.1 275416 251248 ? Rl 2009 70025:19 /usr/bin/python /usr/bin/twistd -f infobot.tap
Monday, August 2, 2010
- Forty Signs of Rain. Kim Stanley Robinson
- Fifty Degrees Below. Kim Stanley Robinson
- Sixty Days and Counting. Kim Stanley Robinson.
- The Tipping Point: How Little Things Can Make a Big Difference. Malcolm Gladwell.
- Brainbox. Christian Cantrell.
- Altered Carbon. Richard K. Morgan.
- The Second Ship (The Rho Agenda). Richard Phillips.
- Immune (The Rho Agenda). Richard Phillips.
Wednesday, June 23, 2010
I'm happy to announce the initial release of filepath.
filepath is an abstract interface to the filesystem. It provides APIs for path name manipulation and for inspecting and modifying the filesystem (for example, renaming files, reading from them, etc). filepath's APIs are intended to be easier than those of the standard library os.path module to use correctly and safely.
filepath is a re-packaging of the twisted.python.filepath module independent from Twisted (except for the test suite which still depends on Twisted Trial).
The low number of this release reflects the newness of this packaging. The implementation is almost entirely mature and well tested in real-world situations from its time as part of Twisted.
You can find the package on PyPI or Launchpad:
Monday, June 21, 2010
I'm happy to announce the initial release of python-signalfd. This simple package wraps the sigprocmask(2) and signalfd(2) calls, useful for interacting with POSIX signals in slightly more advanced ways than can be done with the built-in signal module.
You can find the package on PyPI or Launchpad:
Friday, June 18, 2010
The first solution that came up was to split the web server into two pieces, so that the URLs which could have these JSON responses were served by a different process than was serving the rest. This is a pretty decent solution, and it also provides the benefit of using extra CPU cores if there are any available. In this case, it complicated things a little since it meant sharing a session across two processes. So we went looking for another approach.
It turns out that the json module supports incremental serialization. When I saw the
JSONEncoder.iterencodemethod, I thought it would be great used in combination with cooperate to create a producer. This would let an application serialize a large structure to JSON without multiple processes, threads, or unreasonably blocking the reactor.
Here's the little bit of glue necessary to make things work:
from json import JSONEncoder
from twisted.internet.task import cooperate
def __init__(self, value):
self._value = value
def beginProducing(self, consumer):
self._consumer = consumer
self._iterable = JSONEncoder().iterencode(self._value)
self._task = cooperate(self._produce())
d = self._task.whenDone()
for chunk in self._iterable:
def _unregister(self, passthrough):
By using the
iterencodemethod, this avoids spending too much time generating json output at once. Instead, a little bit of the input will be serialized at a time, and each short resulting string is available from the iterator returned by
_producegenerator will iterated in a way that lets it cooperate with the reactor and other event sources/handlers. A few chunks of json data will be written to the consumer, then execution will switch away to something else, then come back and a few more will be written, and so on.
And by using the producer/consumer interface, if the HTTP client which issued the request doesn't read the results as fast as they're being generated, the server will stop generating new output until the client catches up.
Altogether, this provides a very cool, efficient way to generate JSON output.
Here's an example to make it easier to see how one might use
AsyncJSONin a resource:
from twisted.web.resource import Resource
from twisted.web.server import NOT_DONE_YET
def render_GET(self, request):
length = int(request.args['length'])
d = AsyncJSON(range(length)).beginProducing(request)
d.addCallback(lambda ignored: request.finish())
Wednesday, June 2, 2010
The Minuteman Campground is pretty nice. It's still fairly early in the season, so there were only a few other people there. We brought a tent, but there were also campsites with cabins, which might be nice for longer trips (ie, less baggage).
The ride out was about five hours (including a stop for lunch). Coming back was much quicker, maybe three hours (including a stop for lunch and ice cream).
Overall, a fun trip.
Saturday, May 29, 2010
- My Detachment: A Memoir. Tracy Kidder.
- The Adventures of Sherlock Holmes. Sir Arthur Conan Doyle.
- World War Z: An Oral History of the Zombie War. Max Brooks.
- Big Machine: A Novel. Victor Lavalle.
- The Years of Rice and Salt. Kim Stanley Robinson.
- Iorich. Steven Brust.
- Cowboy Feng's Space Bar and Grille. Steven Brust.
- To Reign in Hell: A Novel. Steven Brust.
- Daemon. Daniel Suarez.
- The City & The City. China Mieville.
- The Return of the Native. Thomas Hardy.
- Alice's Adventures in Wonderland. Lewis Caroll.
- The Cassini Division (Fall Revolution). Ken MacLeod.
- The Stone Canal: A Novel (Fall Revolution). Ken MacLeod.
Wednesday, March 3, 2010
Do include version information somewhere in your package. When I say package here, I'm referring to the Python concept - the thing which can be imported and manipulated programmatically. Here is a random assortment of real-life examples of what I mean:
>>> import sys >>> sys.version '2.6.4 (r264:75706, Dec 7 2009, 18:45:15) \n[GCC 4.4.1]' >>> import OpenSSL >>> OpenSSL.__version__ '0.10' >>> import gmpy >>> gmpy.version() '1.04' >>> import gtk >>> gtk.gtk_version (2, 18, 3) >>> gtk.pygtk_version (2, 16, 0) >>> import pyasn1 >>> pyasn1.majorVersionId '1' >>> import win32api >>> win32api.GetFileVersionInfo(win32api.__file__, chr(92))['FileVersionLS'] >> 16 212You can see several conventions represented here. Some of them are better than others (in case you need a hint,
gtkis doing something pretty nice;
win32apiis towards the other end of the spectrum). However, even the worst of these is providing the desired information. Compare this to:
>>> import zope.interface >>> [x for x in dir(zope.interface) if 'version' in x]  >>> import subunit >>> [x for x in dir(subunit) if 'version' in x] This is a simple piece of information, and after you've actually picked a version of something and installed it, it's hardly ever of much interest. However, when it is of interest, you really want to be able to get it. You don't want to rely on the memory of some user about which version they installed when you're tracking down some poor interaction.
So. Do you maintain a Python package? Does it expose its version somehow? If not, fix it!
Saturday, February 13, 2010
Greetings, and welcome back to "Twisted Web in 60 Seconds". In the previous entry, back at the beginning of December, I promised to cover Twisted Web's proxying capabilities. For various reasons I've decided to dump that topic and cover something else instead. So, prepare to learn about Twisted Web's CGI capabilities!
twisted.web.twcgi.CGIScript and twisted.web.twcgi.FilteredScript are the two most interesting classes in this area. They are both Resource subclasses, so many of the features of resources that I've covered so far apply to them. For example, you can use them as the
resource in an .rpy script:
from twisted.web.twcgi import CGIScript
resource = CGIScript("/path/to/date-example.sh")
date-example.sh might look like this:
echo "Content-Type: text/plain"
That is, just a regular CGI - nothing special about Twisted Web going on there.
If you need to specify an interpreter for some reason (for example, the CGI itself isn't set executable, or doesn't specify its own interpreter with
#! on the first line), you can use
FilteredScript instead of
from twisted.web.twcgi import FilteredScript
resource = FilteredScript("/path/to/date-example.sh")
resource.filter = "/bin/sh"
Set up this way,
/bin/sh will always be run and passed an argument of
Thursday, January 7, 2010
- SuperFreakonomics. Steven D. Levitt and Stephen J. Dubner.
- Look to Windward. Iain Banks.
- PLAN B 4.0. Lester R. Brown.
- Deathworld. Harry Harrison.
- Sixty Days and Counting. Kim Stanley Robinson.
- MetaGame. Sam Landstrom.
- People of the Book. Geraldine Brooks.
- The Mystery of Edwin Drood. Charles Dickens.