Wednesday, December 10, 2008

What is the SigQ field of /proc/PID/status in Linux?

I had to read kernel source to figure this out, and when I did I learned the value isn't at all what I thought, so I thought I'd let everyone else know.

From fs/proc/array.c:

qsize = atomic_read(&p->user->sigpending);
buffer += sprintf(buffer, "SigQ:\t%lu/%lu\n", qsize, qlim);

where p is a struct task_struct p. From include/linux/sched.h:

struct task_struct {
/* process credentials */
struct user_struct *user;


* Some day this will be a full-fledged user tracking system..
struct user_struct {
atomic_t sigpending; /* How many pending signals does this user have? */

So the SigQ value in each process's status file is the total number of signals pending delivery across all processes owned by the UID of the process (or maybe the UID which started the process, I don't know how the user field of task_structs interacts with setuid).

Pretty far from what I've been assuming it meant, the number of signals pending delivery for each individual process. :/

Tuesday, December 9, 2008


What does it mean if a TIOCSTTY ioctl fails with EIO (errno 5) on Linux?

Sunday, September 14, 2008

August reading list

  • Singularity Sky. Charles Stross.

  • Anvil of Stars. Greg Bear.

  • Dune. Frank Herbert.

Wednesday, August 6, 2008

Python and XPath

Quite some while ago, I decided to experiment with writing unit tests for some Nevow-based application code using XPath instead of DOM. DOM-based assertions in unit tests tend to be quite verbose compared to the XPath equivalent. Beyond that, though, the XPath seems to be able to express the intent of the test more accurately. It doesn't rely on as many irrelevant details (like how many TextElements some CDATA is broken up into) - or at least that's the idea I wanted to try out.

Unfortunately I only wrote a few tests like this and shortly afterwards I started focusing on non-XML tasks. As far as the experiment went, I think it went well, but it's basically incomplete at this time.

That's not what this post is about.

A while ago, Ubuntu's Hardy Heron was released. In Hardy, the Python package xml.xpath has vanished. Apparently the package was unmaintained for some time and either Ubuntu or Debian or some combination of the two decided that it would no longer be supported. Sad for me, because now all my XPath-based tests are broken on Ubuntu, which happens to be my primary platform. It seems there are some other XPath libraries for Python available in Hardy, so today I tried to update my tests so that they will work on systems which still have xml.xpath but can also try to use a newer XPath library if xml.xpath is unavailable.

Unfortunately (again) this turned out to be more difficult than I expected. The fall-back library I selected is lxml.etree. The first problem I encountered is that the Python API for processing XPath in lxml.etree is incompatible with the API in xml.xpath. So instead of just changing a couple imports as I had hoped I would be able to do, I wrote a couple thin wrappers to expose both APIs in the same way. The second problem I encountered is that the result objects returned by lxml.etree have a different API than the result objects returned by xml.xpath. So I changed my thin wrappers to be more specific, extracting the exact data my unit tests required, rather than being somewhat more general and letting each test grab what it needed. This meant widening the API to two wrappers from one.

from xml.xpath import Evaluate
except ImportError:
from lxml.etree import XPath, fromstring
def evaluateXPath(path, document):
return XPath(path).evaluate(fromstring(document))
def evaluateTextXPath(path, document):
return evaluateXPath(path, document)[0]
def evaluateXPath(path, document):
return Evaluate(path, minidom.parseString(document))
def evaluateTextXPath(path, document):
return evaluateXPath(path, document)[0].wholeText

After solving these two problems, I had two unit tests passing on my Hardy system. Not a complete failure. I have more than two unit tests that use XPath, though. As it turns out, the next set of tests I looked at need to examine the values of attributes of nodes. This is another area where xml.xpath and lxml.etree present different APIs. If I want to make these tests pass, I need to write another wrapper function to expose attributes in a uniform manner. Here's where I give up on fixing this for the moment. It seems it's not reasonable to expect to transparently switch between xml.xpath and lxml.etree.

Perhaps there's another XPath library out there I can use instead of lxml.etree. Anyone have any tips they'd like to share?

Sunday, July 27, 2008

May through July Reading List

  • Against The Day. Thomas Pynchon.

  • Nature Girl. Carl Hiaasen.

  • HaltinG StatE. Charles Stross.

Tuesday, July 1, 2008

TCP in the browser?

Allen brought to my attention a blog post by Michael Carter about how Orbited now offers a JavaScript API for making TCP connections. I'm not sure how useful this is right now (and I actually mean that I'm not sure, not that I think it's useless and am trying to avoid saying so), though I can see how if, at some future date it's possible to make TCP connections without going through a proxy, apps written this way will suddenly make a lot of sense and be easy to port from apps already based on an API like this.

Anyhow, I thought this would be simple to implement with Athena, but I wanted to see how simple. Five minutes later I had this untested code:

from twisted.internet.protocol import ClientCreator, Protocol
from twisted.internet import reactor

from nevow.athena import LiveElement, expose

class TCPElement(LiveElement):
jsClass = 'Network.TCP'

def connect(self, host, port):
def connected(proto):
self.proto = proto
self.proto.dataReceived = self.dataReceived
self.proto.connectionLost = self.connectionLost
cc = ClientCreator(reactor, Protocol)
d = cc.connectTCP(host, port)
return d

def write(self, bytes):

def dataReceived(self, data):
self.callRemote('dataReceived', data)

def connectionLost(self, reason):
self.callRemote('connectionLost', reason.getErrorMessage())

// athenatcp.js

// import Nevow.Athena

Network.TCP = Nevow.Athena.Widget.subclass('Network.TCP');
function connect(self, host, port) {
return self.callRemote('connect', host, port);

function dataReceived(self, bytes) {
// override this

function connectionLost(self, reason) {
// override this

It'd probably be better if connect were actually a factory function returning TCP instances, but the general idea would be the same. Anyone think this is a really great idea? Anyone want to turn this spike into a real library for Athena?

Saturday, June 28, 2008

re: inventory meme

Glyph tagged me in his Inventory meme several weeks back. At first, I didn't respond because I was too busy. Then, because I forgot. Then, because I decided it would be cheating to wait until I had the maximum number of items possible and then post. Just now I happen to have been reminded of the meme, and I think I can spare the three minutes total this post is going to take, and I'm totally unprepared. So, here it is.

Your hands are empty. You are completely naked.

Saturday, April 26, 2008

February through April Reading List

  • Something Wicked This Way Comes.  Ray Bradbury.
  • The Eyre Affair.  Jasper Fforde.
  • The Star Rover.  Jack London.
  • Travels With Herodotus.  Ryszard Kapuściński.
  • Omnivore's Dilemma.  Michael Pollan.
  • Lankhmar.  Fritz Leiber.
  • Suite Française.  Irène Némirovsky.
  • Persepolis: The Story of a Childhood.  Marjane Satrapi.
  • Persepolis 2: The Story of a Return.  Marjane Satrapi.
  • Maus 1: My Father Bleeds History.  Art Spiegelman.
  • Maus 2: And Here My Troubles Began.  Art Spiegelman.

Friday, April 11, 2008

pyOpenSSL 0.7 final released

Alright, it's done. Here are the highlights since 0.6:

Bug fixes:
  • Memory leak in X509.get_pubkey eliminated
  • Memory leaks in X509Name.__getattr__ and X509Name.__setattr__ eliminated
  • RuntimeWarning from X509Name comparison eliminated
  • X509Name reference counting issues resulting in memory corruption eliminated
  • Uninitialized PKeys are now rejected by X509Req signature APIs
  • Memory leaks in X509Name.__getattr__ and X509Name.__setattr__ eliminated
New features:
  • SSL_get_shutdown and SSL_set_shutdown exposed as Connection.get_shutdown and Connection.set_shutdown
  • X509_verify_cert_error_string exposed as OpenSSL.crypto.X509_verify_cert_error_string
  • X509.get_serial_number and X509.set_serial_number now accept long integers
  • Expose notBefore and notAfter on X509 certificates for inspection and mutation
  • Expose low-level X509Name state with X509Name.get_components
  • Expose hashing and DER access on X509Names

Saturday, March 22, 2008

Wednesday, February 6, 2008

Ever relevant

It is good to re-read Alan Perlis' epigrams in programming from time to time.

Some which stand out to me today:

32. Programmers are not to be measured by their ingenuity and their logic but by the completeness of their case analysis.
41. Some programming languages manage to absorb change, but withstand progress.
64. Often it is the means that justify the ends: Goals advance technique and technique survives even when goal structures crumble.
94. Interfaces keep things tidy, but don't accelerate growth: Functions do.

One which I don't think has ever stood out to me before:

20. Wherever there is modularity there is the potential for misunderstanding: Hiding information implies a need to check communication.

And of course, there are the old favorites:

11. If you have a procedure with ten parameters, you probably missed some.
95. Don't have good ideas if you aren't willing to be responsible for them.
101. Dealing with failure is easy: Work hard to improve. Success is also easy to handle: You've solved the wrong problem. Work hard to improve.
117. It goes against the grain of modern education to teach children to program. What fun is there in making plans, acquiring discipline in organizing thoughts, devoting attention to detail and learning to be self-critical?

Saturday, February 2, 2008

October through January Reading List

  • Smoke and Mirrors. Neil Gaiman.

  • The Long Tomorrow. Leigh Brackett.

  • Norstrillia. Cordwainer Smith.

  • Nova. Samuel R. Delany.

  • Iron Sunrise. Charles Stross.

  • The Lies of Locke Lamora. Scott Lynch.

  • The Road. Cormac McCarthy.

  • Paradise Lost. John Milton.

  • Lord of Light. Roger Zelazny.

Friday, January 4, 2008

notes on netgear support website

  • comments are mangled incorrectly, doubling single quotes, presumably in a confused attempt to prevent SQL injection attacks.

  • page caching behavior is wrong, so after a ticket is updated by netgear support staff, the update will not appear unless the browser is made to disregard its cache.

  • timestamps on comments are wrong, perhaps by roughly (UTC - PST) hours.

  • netgear support staff can't actually fix any problem you have.

Thursday, January 3, 2008

One in twenty eight and falling...

Asteroid on near-miss course with Mars

No one taught these guys to label their plots, unfortunately (nice of them to put "mars" and "sun" on there though). To make it clearer, the cyan line is the probable trajectory of 2007 WD5. White dots indicate a possible position at the time it would intercept mars' orbital, if it does.

Will there be a collision? Probably not. One in twenty eight odds, though? Those are most certainly not astronomical odds.