- The History of the Peloponnesian War. Thucydides. (Books 2 - 8)
- Root Cellaring: Natural Cold Storage of Fruits & Vegetables. Mike and Nancy Bubel.
- The Fatal Shore: The Epic of Australia's Founding. Robert Hughes.
- Special Topics in Calamity Physics. Marisha Pessl.
- The Dirty Life: A Memoir of Farming, Food, and Love. Kristin Kimball.
- The Worst Hard Time: The Untold Story of Those Who Survived the Great American Dust Bowl. Timothy Egan.
- The Children of the Sky (Zones of Thought). Vernor Vinge.
- The Gathering Storm (Wheel of Time). Robert Jordan and Brandon Sanderson.
- Oresteia: Agamemnon, The Libation Bearers, and The Eumenides. Eschylus.
- Towers of Midnight (The Wheel of Time). Robert Jordan and Brandon Sanderson.
- The Clouds. Aristophanes.
- Saturn's Children. Charles Stross.
- The Fuller Memorandum (A Laundry Files Novel). Charles Stross.
- Scratch Monkey. Charles Stross.
Thursday, December 22, 2011
May - December Reading List
Monday, October 31, 2011
Don't Use Buildbot EC2 Features
Wednesday, October 12, 2011
Garlic Followup
Previously I shared some pictures of a garden bed Jericho and I have been working on. Over the long weekend we had some more time to spend on it. While we were away, we got some help preparing the rest of the garden, which would have otherwise taken weeks or months to do by hand:
You can see the oats we put in at the beginning of September in the garlic ged on the left side there. They didn't grow as much as I had hoped:
Perhaps due to some nutrient or mineral deficiency. To rectify that (and based on a soil test), we spread a number of amendments, starting with greensand to provide potassium:
We also spread rock phosphate (for phosphorus) and pelletized lime for calcium and to adjust the pH to be less acidic. And, importantly, compost - about 1 cubic yard over the entire bed (with which task my dad helped us out):
As you can see, we just left the oats in place. They are not cold hardy and will die soon enough without any help. With the bed thusly prepped, we began breaking up our "seed" garlic:
Garlic is most often grown by sowing cloves in the autumn for harvest the following summer. The winter encourages the clove to split and grow into a new bulb. We planted four varieties of garlic, but mostly Inchelium, a softneck variety.
These seed bulbs each had around a dozen cloves in them.
We planted the largest undamaged cloves. We also planted three varieties of hardneck garlic. Compared to the inchelium, these all look pretty similar to each other. Here's some Siberian Red:
The hardneck varieties have bulbs with fewer, larger cloves. After we broke up the cloves, we planted them! While one of us dropped cloves in pre-marked locations, the other followed behind and planted them.
The cloves are planted right-side-up about one inch deep. Finally we mulched them with straw to even out temperature variations and retain moisture.
Now the garlic sits tight until next year.
Wednesday, September 7, 2011
Garden Bed Prep
Over the long weekend, Jericho and I made a garden bed. We picked a plot a few minutes walk from the new orchard and started by mowing a 100' x 4' area.
That's Lucy, my mom's lab, down near the end. Next we cut sod with shovels.
I dug too, but Jericho doesn't take as many pictures as I do.
After that, we flipped sod. First one row of it:
And then the next row:
After all the sod was out, we dug a little more.
Then we put the sod at the bottom of the hole, upside-down, where it will hopefully die and contribute organic material to the soil.
And then we shoveled that dirt off the tarp, back into the hole on top of the sod, and raked it flat.
And again.
Until all the dirt was back in the bed.
This will be a garlic bed. We'll plant the garlic in October. Until then, we put in a quick cover crop of oats.
Then I rubbed some dirt on my shirt to make it look like I helped too.
A few hours after we finished seeding, a nice thunderstorm rolled in and watered everything for us.
If all goes well, in about a month we'll have some nice young oats to mow down before planting garlic in the bed.
Releasing Python Software is Tedious
I released pyOpenSSL 0.13 a few days ago. Apart from making sure it actually worked on various platforms, updating the version number, regenerating the documentation, and sending out the release announcement, I also had to upload release files to the Python Package Index.
Uploading release files to PyPI is the part of the release process I hate the most. pyOpenSSL 0.13 had 15 files to upload to PyPI. There is no usable automated interface for uploading files to PyPI. Before I can even begin to upload them, I have to download them from the build farm where they're generated. Then, uploading just one file to PyPI requires at least 8 mouse clicks. The clicks vary depending on which file is being uploaded. I have to select a file, select its type, specify which Python version it's for, and then submit a form.
15 files, 8 clicks per file. Well, you do the math. It's not a pleasant experience. PyPI would be a much better resource if it didn't force me to specify a ton of redundant, mostly useless information every time I do a release. It would be a much better resource if it had a programmatic interface so I don't have to spend 20 minutes clicking buttons in web browser. It would be a much better resource if it didn't try to hard to discourage me from releasing software.
Saturday, August 27, 2011
Buildin' a fence
Happily mowing was mostly taken care of before I arrived:
So it was easy to work around the trees and along the eventual fence-line.
I started at the corners of the area and drove 8 foot steel t-posts:
Using a post driver:
This was actually a lot easier than I expected, perhaps because of the heavy rains on the preceding evenings. I strung 12 gauge monofilament wire between these to define a straight line:
I used the line to place intermediate t-posts:
After 15 t-posts were up and connected with monofilament, I hung deer netting from it:
This part seemed harder than driving the t-posts, maybe because I was already somewhat tired from that earlier activity, and maybe because this activity mostly involved holding my arms above my head for long periods of time doing more close-detail work.
I started by zip-tying the net to the monofilament to temporarily fight sag between posts:
Then I used inline tensioners to take up the slack:
The bottom of the net is held down with sod staples:
The netting quickly vanishes as you move away from it, so I also flagged each stretch:
Some loose ends still need to be wrapped up. I found some ferrules at the hardware store to secure the monofilament, but I haven't tried crimping them yet. The zip-ties need to be replaced with a more robust solution. And the deer net needs to be secured to each t-post to keep it from billowing so much.
Monday, August 22, 2011
Redhumped Caterpillar
However seeing the almost stripped tree they were feeding on in person was still a shock:
I had tentatively identified them as codling moth larvae, which didn't quite make sense since these typically grow and feed inside fruit, not on leaves. My revised identification is redhumped caterpillar. I just sent an email to Maine Cooperative Extension to see what they think.
The bright side is that only one tree appears affected, and there seems to be a fair chance that it will recover.
Monday, August 1, 2011
Twisted Conch in 60 Seconds: Protocols
Welcome once more to Twisted Conch in 60 Seconds, the tutorial series about writing SSH applications with Twisted.
Over the past several articles, I've introduced the APIs for letting clients establish a new logical connection to your SSH server, generating output on those connections, accepting input on those connections, and detecting the end of those connections. Taken together, these four activities map almost exactly onto the standard Twisted protocol abstraction (represented and documented by the IProtocol interface). In this article, I'll show you how to use any IProtocol implementation from Twisted to interact with an SSH channel.
The previous example implemented an echo-ish type of server by customizing the session. I'll duplicate that functionality here, but in an IProtocol
implementation:
class EchoProtocol(Protocol):
def connectionMade(self):
self.transport.write("Echo protocol connected\r\n")
def dataReceived(self, bytes):
self.transport.write("echo: " + repr(bytes) + "\r\n")
def connectionLost(self, reason):
print 'Connection lost', reason
Each of the three events a protocol might receive is handled here - connection made and data received are handled by writing something to the connection, and connection lost is handled by writing something to stdout (writing to the connection is no longer an option after it has been lost, of course). This is a simple, fairly typical Twisted-based protocol implementation; you'll find protocols like this all over the place.
The major part of the requirements for using this is having a transport to which to connect it. The SSHSession
class you're now familiar with can serve in just that role. Only a little code to put it together with a protocol is required. Again I'll create a "session" channel which accepts but ignores pty requests:
class SimpleSession(SSHSession):
name = 'session'
def request_pty_req(self, data):
return True
And again I'll override request_shell
so that the protocol is connected to the channel/transport as soon as the client requests a shell.
def request_shell(self, data):
protocol = EchoProtocol()
All I've done so far here is make an instance of the protocol defined above. Next I'll create the transport object and hook it up to this protocol. This part uses a couple helpers from twisted.conch.ssh.session, SSHSessionProcessProtocol
and wrapProtocol
:
transport = SSHSessionProcessProtocol(self)
protocol.makeConnection(transport)
transport.makeConnection(wrapProtocol(protocol))
self.client = transport
return True
Each step here is necessary, but the specifics aren't very interesting or important. Suffice it to say it hooks objects up as necessary so that bytes can be passed from the SSHSession
to the protocol and vice versa.
The rest of the code for this version is the same as it has been in the previous versions (except for new imports). Here's the full code listing:
from twisted.internet.protocol import Protocol
from twisted.cred.portal import Portal
from twisted.cred.checkers import FilePasswordDB
from twisted.conch.ssh.factory import SSHFactory
from twisted.internet import reactor
from twisted.conch.ssh.keys import Key
from twisted.conch.interfaces import IConchUser
from twisted.conch.avatar import ConchUser
from twisted.conch.ssh.session import (
SSHSession, SSHSessionProcessProtocol, wrapProtocol)
with open('id_rsa') as privateBlobFile:
privateBlob = privateBlobFile.read()
privateKey = Key.fromString(data=privateBlob)
with open('id_rsa.pub') as publicBlobFile:
publicBlob = publicBlobFile.read()
publicKey = Key.fromString(data=publicBlob)
class EchoProtocol(Protocol):
def connectionMade(self):
self.transport.write("Echo protocol connected\r\n")
def dataReceived(self, bytes):
self.transport.write("echo: " + repr(bytes) + "\r\n")
def connectionLost(self, reason):
print 'Connection lost', reason
def nothing():
pass
class SimpleSession(SSHSession):
name = 'session'
def request_pty_req(self, data):
return True
def request_shell(self, data):
protocol = EchoProtocol()
transport = SSHSessionProcessProtocol(self)
protocol.makeConnection(transport)
transport.makeConnection(wrapProtocol(protocol))
self.client = transport
return True
class SimpleRealm(object):
def requestAvatar(self, avatarId, mind, *interfaces):
user = ConchUser()
user.channelLookup['session'] = SimpleSession
return IConchUser, user, nothing
factory = SSHFactory()
factory.privateKeys = {'ssh-rsa': privateKey}
factory.publicKeys = {'ssh-rsa': publicKey}
factory.portal = Portal(SimpleRealm())
factory.portal.registerChecker(FilePasswordDB("ssh-passwords"))
reactor.listenTCP(2022, factory)
reactor.run()
Connect to this server with an ssh client and type away, you should see something like this:
exarkun@boson:~$ ssh -p 2022 localhost
exarkun@localhost's password:
Echo protocol connected
echo: 'h'
echo: 'e'
echo: 'l'
echo: 'l'
echo: 'o'
echo: ','
echo: ' '
echo: 'w'
echo: 'o'
echo: 'r'
echo: 'l'
echo: 'd'
echo: '.'
Tune in next time for a surprise topic!
Monday, June 13, 2011
Twisted Conch in 60 Seconds: Detecting EOF on input
Greetings patient readers. Welcome once again to Twisted Conch in 60 Seconds, a series of articles in which I attempt to convey the simplicity of writing SSH applications using Twisted. For those of you who have been reading since the beginning, thank you for bearing with me through the unfortunate hiatus.
When last we parted, I had just explained how to accept input on a custom server channel. In this edition, I'll explain the various callbacks you can expect when your channel is not going to receive any further data.
Recall the channel class which echoes all of its input back to the client from the previous example:
class SimpleSession(SSHChannel):
name = 'session'
def request_pty_req(self, data):
return True
def request_shell(self, data):
return True
def dataReceived(self, bytes):
self.write("echo: " + repr(bytes) + "\r\n")
When the client disconnects, the SimpleSession
instance will get garbage collected (as long as we have no extra code of our own keeping a reference to it, thus keeping it alive). You can also receive a callback when this disconnection happens, though, before the instance is discarded. Actually, you can receive one or more of three different callbacks:
def eofReceived(self):
self.write("eof received\r\n")
This eofReceived
callback happens when the client sends an End-Of-File notification for this channel. You might see this if the client is invoked like this:
$ ssh username@host < input-file
After the contents of input-file
have been sent, the client will send the EOF notification and wait for server output. However, if the client is invoked in this more common manner:
$ ssh username@host
then it's more likely that it will exit via the magic ~.
sequence, or by being killed (explicitly or by a SIGHUP when the user logs out). In this case, there is no EOF notification. Instead, the SSH connection is just closed. This results in the channel being closed as well, which can be handled like this:
def closed(self):
print "Channel closed"
This notification may also happen after an EOF notification, so be prepared to handle them both. Finally, it's possible for the client to request that just one particular channel be closed. This can be handled like this:
def closeReceived(self):
print "Close received"
However, this condition is difficult to trigger with the standard ssh command line client (in fact, I don't know how to trigger it). A custom SSH client might want to do this, though - for example if it creates and destroys many different channels in the lifetime of a single SSH connection.
As a rule of thumb, closed
is probably the most commonly useful of these three callbacks. You can use it to reliably clean up resources when a channel is no longer in use and be confident that it is going to be called.
Here's a complete server using the echo channel definition from the previous post, plus the three new callbacks introduced in this post:
from twisted.cred.portal import Portal
from twisted.cred.checkers import FilePasswordDB
from twisted.conch.ssh.factory import SSHFactory
from twisted.internet import reactor
from twisted.conch.ssh.keys import Key
from twisted.conch.interfaces import IConchUser
from twisted.conch.avatar import ConchUser
from twisted.conch.ssh.channel import SSHChannel
with open('id_rsa') as privateBlobFile:
privateBlob = privateBlobFile.read()
privateKey = Key.fromString(data=privateBlob)
with open('id_rsa.pub') as publicBlobFile:
publicBlob = publicBlobFile.read()
publicKey = Key.fromString(data=publicBlob)
def nothing():
pass
class SimpleSession(SSHChannel):
name = 'session'
def request_pty_req(self, data):
return True
def request_shell(self, data):
return True
def dataReceived(self, bytes):
self.write("echo: " + repr(bytes) + "\r\n")
def eofReceived(self):
print 'connection lost'
def closed(self):
print 'closed'
def closeReceived(self):
print 'closeReceived'
class SimpleRealm(object):
def requestAvatar(self, avatarId, mind, *interfaces):
user = ConchUser()
user.channelLookup['session'] = SimpleSession
return IConchUser, user, nothing
factory = SSHFactory()
factory.privateKeys = {'ssh-rsa': privateKey}
factory.publicKeys = {'ssh-rsa': publicKey}
factory.portal = Portal(SimpleRealm())
factory.portal.registerChecker(FilePasswordDB("ssh-passwords"))
reactor.listenTCP(2022, factory)
reactor.run()
By this point, I've explained enough to make a complete (if simple) SSH server application - authentication, output, pty requests, input, and disconnection handling. In the next entry, I'll explain how you can hook a normal Protocol
instance up to a channel, useful for abstracting your application logic somewhat away from the SSH details and (therefore) for re-using existing Twisted-based protocol implementations with SSH as their transport.
Tuesday, May 17, 2011
The "example.com" of OIDs
So of course I wanted my test case to throw an OID at the implementation that I could be sure it would never recognize. If you're familiar with DNS, you may know that there are names like "example.com" which everyone has agreed won't really mean anything (in fact, "example.com" does mean something, and a better comparison is with the less widely used "invalid" TLD - eg "example.invalid") - I wanted the OID equivalent of this.
First I refreshed my acquaintance with the OID registry of which I was already aware. Later I came across the somewhat more cleverly named site, http://www.oid-info.com/. Then I poked around for a while trying to find an existing allocation which fit the bill.
I discovered that the LDAPv3 people seem to think that 1.1 (scroll down a bit) can be used for this purpose. This despite the fact that 1.1 appears to have been allocated to the ISO registration authority, but maybe this use has now been discarded. LDAPv3's appropriation of this OID seemed a misuse to me, so I kept looking.
I found a publication the contents of which apparently would tell me how I could get a new arc (as these things are called) beneath {iso(1) identified-organization(3)}. I didn't feel like shelling out CHF 50,00 for the privilege of learning that, though.
Next I noticed another registration authority at {itu-t(0) identified-organization(4)}. But this all seemed like the wrong path to me. I just want a guaranteed unknown OID to use, I don't want to register an organization with an international standards body.
Then I started looking beneath {joint-iso-itu-t(2)}. Perhaps by their powers combined, ISO and ITU-T had imagined this eventuality and allocated something. Lo! Immediately beheld I {joint-iso-itu-t(2) example(999)}. Not quite invalid but at this point in my searches, close enough (I didn't initially notice that this arc is not in fact currently allocated, merely proposed; after I noticed that it appears it will be allocated later this year I still thought it was good enough).
Only, no. What's the ASN.1 DER encoding of {joint-iso-itu-t(2) example(999)}? There doesn't appear to be one. It appears to be invalid to have 999 as the second component of the arc. As the third component? No problem. As any component after that? Sure thing. But... not as the second (not as the first either; and the exact rules governing the maximum values of the first two components are complicated beyond my understand, so I'm not going to try to explain).
So there's an example OID (pending actual allocation at two separate meetings of two separated international standards organizations). Just not one that can actually be encoded. So I can't actually use it for my test.
Ah well, screw all this crap. 4.5.6.7 seems like a good test value to me.
Addendum: I couldn't leave the loose thread of 2.999 being unencodable alone. So I tracked down the ASN.1 DER encoding rule for this. It's actually the same as the ASN.1 BER rule, which is: "The first octet has value 40 * value1 + value2. (This is unambiguous, since value1 is limited to values 0, 1, and 2; value2 is limited to the range 0 to 39 when value1 is 0 or 1; and, according to X.208, n is always at least 2.)". So there you go. ASN.1 DER really cannot encode 2.999. Perhaps someone will point this out at one of the two upcoming standards bodies meetings.
Monday, May 16, 2011
Twisted Conch in 60 Seconds will return soon
I expect some of you have been eagerly awaiting the next installment of Twisted Conch in 60 seconds. My goal was to produce one of these a week, but clearly I've missed that the last few weeks. The series will return, but it may be another week or two (optimistically) before the next post. Sorry about the delay!
Sunday, May 1, 2011
Planting trees, day 6
And... that's it! All told, now growing outside are four hardy kiwis, two walnuts, two pears, three cherries, four crab apples, seven plums, and 26 apple trees (plus the ~forest that was here before - but I won't count that).
Saturday, April 30, 2011
Planting trees, day 5
Friday, April 29, 2011
Planting trees, day 4
Thursday, April 28, 2011
Planting trees, day 3
Wednesday, April 27, 2011
Planting trees, day 2
Tuesday, April 26, 2011
Planting trees, day 1
Saturday, April 23, 2011
August - April Reading List
- Containment. Christian Cantrell.
- Human Legacy Project. Christian Cantrell.
- The Long Emergency. James Howard Kunstler.
- Sethra Lavode. Steven Brust.
- Star Soldier. Vaughn Heppner.
- BIO-WEAPON. Vaughn Heppner.
- Dzur. Steven Brust.
- Fragile and Distant Suns: A Poul Anderson Collection. Poul Anderson.
- Dreaming in Code: Two Dozen Programmers, Three Years, 4,732 Bugs, and One Quest for Transcendent Software. Scott Rosenberg.
- Spell Games. T. A. Pratt.
- Blood Engines. T. A. Pratt.
- Poison Sleep. T. A. Pratt.
- Dead Reign. T. A. Pratt.
- Bone Shop. T. A. Pratt.
- Makers. Cory Doctorow.
- Microcosm: E. coli and the New Science of Life. Carl Zimmer.
- The Theory of Almost Everything. Robert Oerter.
- Broken Mirrors. T. A. Pratt.
- The Warrior's Apprentice. Lois McMaster Bujold
- The Mountains of Mourning. Lois McMaster Bujold
- The history of Herodotus. Herodotus. (except books 2 and 3)
- Surely You're Joking, Mister Feynman! Richard P. Feynman.
- The Name of the Wind. Patrick Rothfuss.
- The Wise Man's Fear. Patrick Rothfuss.
- Reluctant Swordsman. Dave Duncan.
- The Golden Age of Science Fiction. Various
- History of the Peloponnesian War. Thucydides. (Book 1)
pyOpenSSL on PyPy
You may know that I'm the maintainer of pyOpenSSL, a Python extension library which wraps some OpenSSL APIs. pyOpenSSL was started in mid 2001, around the time of Python 2.1, by AB Strakt (now Open End). Shortly afterwards Twisted picked it up as a dependency for its SSL features (the standard library SSL support was unsuitable for non-blocking use). When Twisted bumped into some of pyOpenSSL's limitations and no one else was around to address them, I decided to take responsibility for the project.
Fast forward almost a decade. pyOpenSSL now runs on Python 2.4 through Python 3.2. And soon I hope it will run on PyPy, too.
This post is about some of the things I learned while getting pyOpenSSL to work on PyPy. All of this work is made possible, of course, by the "cpyext" module of PyPy which implements CPython's C extension API for PyPy.
PyModule_AddObject steals references
When an extension module wants to expose a name, the most common way it does this is by using PyModule_AddObject
. This adds a new attribute to the module, given a char*
name and a PyObject*
object. CPython uses reference counting, so the PyObject*
has a counter on it recording how many different pieces of code still want it to remain alive. When populating a new module, the PyObject*
you have generally have a reference count of 1. PyModule_AddObject
steals this reference: it doesn't increase the reference count, it just assumes that the caller is giving up its interest in the object remaining alive; now it is the module to which the PyObject*
was added which has that interest. So the reference count is still 1.
pyOpenSSL exposes a few names which are just aliases for other names (for example, X509Name
and X509NameType
refer to the same object). It does this by calling PyModule_AddObject
twice with the same PyObject*
but different char*
names. Considering what wrote above about reference counts above, you might guess that this needs extra work to get the reference counting to work correctly. Otherwise the second PyModule_AddObject
would steal a reference from the first PyModule_AddObject
, since that's what stole it from the module initialization code. This wouldn't work very well, since there really are two references to the PyObject*
now, not just one.
Though, it turns out that on CPython, it doesn't really matter. The reference count for one of these types, say X509Name
again, ends up at around 20 by the time everything is initialized. Being off by one doesn't make a difference, because most of those 20 references last for the entire process lifetime. The value never gets close to 0, so the missing reference is never noticed. However, on PyPy, it turns out the missing reference does matter. I won't try to explain how PyPy manages to support CPython's C extension API, nor how it manages to make a reference counting system play together with a fully garbage collected runtime (ie, PyPy doesn't normally do reference counting). Suffice it to say that on PyPy, sometimes the reference count does get close to 0, and at those times, being off by 1 can be important - because it might mean that the reference count is exactly 0 when it was supposed to be 1. When that happens, PyPy frees the object, but other code continues to use it, and after that the behavior you get is difficult to predict due to memory corruption - but it's certainly not correct.
The fix for this one is simple - add a Py_INCREF
before PyModule_AddObject
. This was by far the most pervasive bug in pyOpenSSL which needed to be fixed, since it was repeated for each aliased type pyOpenSSL exposed. I added 28 Py_INCREF
calls in total to address these.
PyPy doesn't support tp_setattr (yet?)
The type I mentioned above, X509Name
, customizes attribute access. It needs to delegate to OpenSSL to determine if an attribute is valid or not, and if so what its current value is. It does this by implementing two C functions, one for the tp_setattr
slot and one for the tp_getattro
slot. No, that o isn't a typo. The CPython C extension API provides two different ways to customized attribute access. Using one way, tp_setattr
and tp_getattr
, CPython hands the extension function the name of the attribute as a char*
. Using the other way, tp_setattro
or tp_getattro
, a PyObject*
is passed in, instead of a char*
.
So far, PyPy only implements tp_setattro
and tp_getattro
, not tp_setattr
and tp_getattr
. It would have been nice to implement this missing feature for PyPy, but instead I switched pyOpenSSL over to the already supported mechanism. This was a very simple change, since most of the lookup code is the same either way, there's just a little extra code at the beginning of the function to convert from PyObject*
to char*
.
I also learned about a quirk of the tp_setattro
API while doing this. I expected setattr(name, u"name", "value") to pass in a PyUnicodeObject*
. However, CPython actually encodes unicode to ascii and passes in a PyStringObject*
instead.
X509Name.__setattr__ was missing some cleanup code for the AttributeError case
While making the switch to tp_setattro
, I noticed a bug in pyOpenSSL where it failed to flush the OpenSSL error queue properly, causing a spurious OpenSSL.SSL.Error
to be raised whenever an attempt was made to set an invalid attribute on an X509Name
instance. This was easy to fix by adding a call to the function which flushes the error queue.
PyPy doesn't yet support all of the PyArg_ParseTuple
format specifiers
Finally, I had to work PyPy itself a little bit to implement the s*
and t#
format specifiers for PyArg_ParseTuple
. PyArg_ParseTuple
is how C extension functions unpack the arguments passed to them. A call to this function looks something like PyArg_ParseTuple(args, "s*|i:send", &pbuf, &flags)
. The string specifies how many and what type of arguments are expected, and the values are unpacked from args
into the rest of the arguments passed in. PyPy did not yet support a couple argument types which pyOpenSSL relies on, so I added this support. This code is still in a branch of PyPy, but I hope it will be merged into the default branch soon.
Remaining work
There is one thing left to do before pyOpenSSL will be 100% supported on PyPy. Though I said I implemented s*
for PyArg_ParseTuple
, I actually only implemented part of it. My code will handle the case where a str
is passed in, but not the case where a memoryview
is supplied instead. Handling memoryview
involves a bit more work and a bit more understanding than I currently have of how PyPy's CPython bridge works. Fortunately there are many useful things that can be done with pyOpenSSL on PyPy even without this feature (when was the last time you constructed a memoryview
? :), so I'm still very happy with where things currently stand.
The code
As of this posting, the PyPy code needed to make this work is in the pyarg-parsebuffer-new
branch and the pyOpenSSL code is in the run-on-pypy
branch. I'll be psyched if the PyPy branch can be merged in time for PyPy 1.5 so that the next pyOpenSSL release can work with the next PyPy release - we'll see!
Tuesday, April 19, 2011
Twisted Conch in 60 Seconds: Accepting Input
Welcome back to Twisted Conch in 60 Seconds, the documentation series about writing SSH servers (and eventually, clients) with Twisted. In earlier entries, I've covered some of the basics of accepting client connections and generating output. In this edition, I'll cover accepting input from the client.
Recall that in the previous two example programs, a SSHChannel
subclass was responsible for sending some output to the client connection. The same object is going to have input from the client delivered to it. Some of you may not even be surprised to learn that the way this is done is that the channel has its dataReceived
method called with a string:
class SimpleSession(SSHChannel):
def dataReceived(self, bytes):
self.write("echo: " + repr(bytes) + "\r\n")
The single argument to dataReceived
, bytes
, is a str
containing the bytes sent from the client. This simple implementation of dataReceived
escapes the received data with repr
so it's easy to see what bytes were actually received and then sends them back with a little formatting. As you might expect, dataReceived
is being passed bytes from a reliable, ordered, stream-oriented connection. That is, it's a lot like TCP. This means you need to be careful about message boundaries, possibly buffering up several calls with of data before handling it. Unlike TCP, of course, these bytes were sent encrypted over the network. This is an SSH tutorial, after all!
Aside from this method, it's still necessary to acknowledge the PTY request the client will send:
def request_pty_req(self, data):
return True
But since this example doesn't make use of the terminal name, size, or mode information all the method needs to do is return True to indicate that the request was successful. Similarly, the shell request must be allowed:
def request_shell(self, data):
return True
Again, nothing going on here except a positive acknowledgement of the request so the client will be happy and move on. That's all of the code that's changed since the last example. The full code listing looks like this:
from twisted.cred.portal import Portal
from twisted.cred.checkers import FilePasswordDB
from twisted.conch.ssh.factory import SSHFactory
from twisted.internet import reactor
from twisted.conch.ssh.keys import Key
from twisted.conch.interfaces import IConchUser
from twisted.conch.avatar import ConchUser
from twisted.conch.ssh.channel import SSHChannel
with open('id_rsa') as privateBlobFile:
privateBlob = privateBlobFile.read()
privateKey = Key.fromString(data=privateBlob)
with open('id_rsa.pub') as publicBlobFile:
publicBlob = publicBlobFile.read()
publicKey = Key.fromString(data=publicBlob)
def nothing():
pass
class SimpleSession(SSHChannel):
name = 'session'
def request_pty_req(self, data):
return True
def request_shell(self, data):
return True
def dataReceived(self, bytes):
self.write("echo: " + repr(bytes) + "\r\n")
class SimpleRealm(object):
def requestAvatar(self, avatarId, mind, *interfaces):
user = ConchUser()
user.channelLookup['session'] = SimpleSession
return IConchUser, user, nothing
factory = SSHFactory()
factory.privateKeys = {'ssh-rsa': privateKey}
factory.publicKeys = {'ssh-rsa': publicKey}
factory.portal = Portal(SimpleRealm())
factory.portal.registerChecker(FilePasswordDB("ssh-passwords"))
reactor.listenTCP(2022, factory)
reactor.run()
Et voilà, a custom SSH server which accepts input and generates output. Next time, the exciting topic of detecting EOF on that input stream...
Sunday, April 10, 2011
Twisted Conch in 60 Seconds: PTY Requests
Greetings and welcome once again to Twisted Conch in 60 Seconds, a series introducing SSH development with Twisted. If you're just joining me, you probably want to go back to the beginning before reading this article.
The previous example constructed a server which could present some data to clients which managed to authenticate successfully. However, it also produced an error message which I didn't explain:
PTY allocation request failed on channel 0
This is an error message the client software displays (so you may see something different depending on which SSH client you use - I am using OpenSSH 5.1). What the client is trying to tell us is that it asked the server to allocate a PTY - a pseudo-terminal - and the server would not do it. The client doesn't think this is a fatal error though, so it continues on to request a shell, which it gets.
It turns out that PTY requests are another very common channel request (that is, a request that is scoped to a particular channel, as the shell request is). Almost any time you expect clients to show up at your server and request a shell, it is somewhat likely that they'll request a PTY as well, so you probably want to handle this case.
Recall how the shell request was handled in the previous example:
class SimpleSession(SSHChannel):
name = 'session'
def request_shell(self, data):
self.write("This session is very simple. Goodbye!\r\n")
self.loseConnection()
return True
As a PTY request is just another type of channel request, I can extend this class with a new method, request_pty_req
, in order to handle this additional request type.
def request_pty_req(self, data):
I didn't talk about the data
parameter last time because the shell request makes no use of it. However, a PTY request does use it. The client packs some information about its capabilities into a particular string format and that's what this method will receive as an argument. Conch provides a method for parsing this string into a more manageable form:
self.terminalName, self.windowSize, modes = parseRequest_pty_req(data)
terminalName
is a string giving the name of the terminal. This is typically the value of the TERM
environment variable on the client - which might not be what you expect, and is often simply set to "xterm".
More useful than the terminal name is the windowSize
tuple. This tuple has four elements. The first two give the number of rows and columns available on the client. The third and fourth give the terminal width and height in pixels. For many clients, the latter isn't available or useful so these values are set to 0
.
The last value, modes
, is rather complicated. You can read about it in RFC 4254, section 8. Or you can take my word for it for now that you probably don't care about it, at least for now.
Notice that I saved some of that data. I'm going to use it to make request_shell
a little more interesting. Before moving on to that, there's one last thing to do in request_pty_req
though. I need to indicate that the PTY request was successful:
return True
Now, here's a new version of request_shell
which uses those two new attributes:
def request_shell(self, data):
self.write(
"Your terminal name is %r. "
"Your terminal is %d columns wide and %d rows tall." % (
self.terminalName, self.windowSize[0], self.windowSize[1]))
self.loseConnection()
return True
Now when clients connect to the server, they will be able to see what terminal they're running and how big it is. Here's the full code listing for this version of the example, including a new import for parseRequest_pty_req
:
from twisted.cred.portal import Portal
from twisted.cred.checkers import FilePasswordDB
from twisted.conch.ssh.factory import SSHFactory
from twisted.internet import reactor
from twisted.conch.ssh.keys import Key
from twisted.conch.interfaces import IConchUser
from twisted.conch.avatar import ConchUser
from twisted.conch.ssh.channel import SSHChannel
from twisted.conch.ssh.session import parseRequest_pty_req
with open('id_rsa') as privateBlobFile:
privateBlob = privateBlobFile.read()
privateKey = Key.fromString(data=privateBlob)
with open('id_rsa.pub') as publicBlobFile:
publicBlob = publicBlobFile.read()
publicKey = Key.fromString(data=publicBlob)
def nothing():
pass
class SimpleSession(SSHChannel):
name = 'session'
def request_pty_req(self, data):
self.terminalName, self.windowSize, modes = parseRequest_pty_req(data)
return True
def request_shell(self, data):
self.write(
"Your terminal name is %r.\r\n"
"Your terminal is %d columns wide and %d rows tall.\r\n" % (
self.terminalName, self.windowSize[1], self.windowSize[0]))
self.loseConnection()
return True
class SimpleRealm(object):
def requestAvatar(self, avatarId, mind, *interfaces):
user = ConchUser()
user.channelLookup['session'] = SimpleSession
return IConchUser, user, nothing
factory = SSHFactory()
factory.privateKeys = {'ssh-rsa': privateKey}
factory.publicKeys = {'ssh-rsa': publicKey}
factory.portal = Portal(SimpleRealm())
factory.portal.registerChecker(FilePasswordDB("ssh-passwords"))
reactor.listenTCP(2022, factory)
reactor.run()
Compared to the last example, only the methods of SimpleSession
have changed. If you run this server, connect to it, and authenticate then you'll be rewarded with a little information about your terminal. Or, if your SSH client does not request a PTY for some reason (for example, if you pass -T to the OpenSSH client) then you'll see an error when the shell request fails, since terminalName
and windowSize
won't have been set.
At this point, I have explained enough of Conch that you could almost write a very simple application with it. For example, you could add an SSH server to your Twisted Web server which sends request logs to the client. This would just involve calling the write
repeatedly, once for each request which was handled. However, for many applications, you may actually want to accept input from the client. Stick around for the next article in which I'll cover precisely that.
Tuesday, April 5, 2011
Twisted Conch in 60 Seconds: A Trivial Channel
Welcome to the first nearly-useful edition of Twisted Conch in 60 Seconds. In previous posts, I demonstrated how to create an SSH server no one could log in to, and then how to create an SSH server with password authentication but no content beyond that. Today I'll show you how to present some data to clients which connect and successfully authenticate.
Recall that in the last example, I added SimpleRealm to the SSH server. The SimpleRealm was responsible for creating a user (or avatar) to handle authenticated connections. Since Conch provides a very simple user class, ConchUser, the realm was quite simple:
class SimpleRealm(object):
def requestAvatar(self, avatarId, mind, *interfaces):
return IConchUser, ConchUser(), nothing
Almost any behavior you might want to implement in a custom SSH server is going to go into the user object. For example, after an SSH client authenticates, in almost all cases the first thing it does is open a channel. SSH can multiplex many logical connections over a single TCP connection. Channels are the basis of this multiplexing. When an SSH client opens a channel, Conch turns this into a method call onto the user object, lookupChannel. Channels come in different types, but ConchUser
doesn't define behavior for any of them by default. This explains the output of the previous example:
channel 0: open failed: unknown channel type: unknown channel
The channel I'm going to define in this example will be of the session type. This is the kind of channel almost all SSH clients request after authentication succeeds. Channels should typically subclass SSHChannel:
from twisted.conch.ssh.channel import SSHChannel
class SimpleSession(SSHChannel):
Channels also need to specify their type or name:
name = 'session'
Aside from just passing uninterpreted bytes back and forth, as is done when you use SSH to get a remote shell, channels also support structured requests. SSHChannel
lets you handle these requests by defining request_
-prefixed methods. When a foo request is received, request_foo
is called to handle it. One common request, and the only one I'm going to handle in this example, is a request for a shell. However, to keep things simple, I won't implement it to set up a real shell:
def request_shell(self, data):
self.write("This session is very simple. Goodbye!\r\n")
self.loseConnection()
return True
Since the most common thing to use the command line ssh tool for is to get a remote shell, the client issues a shell request by default. After this, ssh is just passing bytes back and forth. It's up to the user and the shell process (such as /bin/bash) to make up and interpret those bytes.
However, this shell request handler does generate some bytes. It sends them to the client using the write
method. Then it closes the channel (not the entire SSH connection) with the loseConnection
method. Finally, it returns True
to indicate that the request was successful.
The only thing left to do is make ConchUser
capable of using SimpleSession
to satisfy requests to open channels of type session. The channelLookup
attribute of ConchUser
makes it easy to do this. It's a dictionary the keys of which are channel types and the values of which are SSHChannel
subclasses. This means SimpleRealm
can be tweaked slightly to produce users which support session channels:
class SimpleRealm(object):
def requestAvatar(self, avatarId, mind, *interfaces):
user = ConchUser()
user.channelLookup['session'] = SimpleSession
return IConchUser, user, nothing
As before, we have a requestAvatar
which returns a three-tuple of interface, avatar, and logout callable. However, this version of the realm also adds an item to the user's channelLookup
dictionary before returning it. These users can therefore open new channels of type session.
Here's a complete code listing for this version of the example. Only SimpleSession
is new, and only SimpleRealm.requestAvatar has changed.
from twisted.cred.portal import Portal
from twisted.cred.checkers import FilePasswordDB
from twisted.conch.ssh.factory import SSHFactory
from twisted.internet import reactor
from twisted.conch.ssh.keys import Key
from twisted.conch.interfaces import IConchUser
from twisted.conch.avatar import ConchUser
from twisted.conch.ssh.channel import SSHChannel
with open('id_rsa') as privateBlobFile:
privateBlob = privateBlobFile.read()
privateKey = Key.fromString(data=privateBlob)
with open('id_rsa.pub') as publicBlobFile:
publicBlob = publicBlobFile.read()
publicKey = Key.fromString(data=publicBlob)
def nothing():
pass
class SimpleSession(SSHChannel):
name = 'session'
def request_shell(self, data):
self.write("This session is very simple. Goodbye!\r\n")
self.loseConnection()
return True
class SimpleRealm(object):
def requestAvatar(self, avatarId, mind, *interfaces):
user = ConchUser()
user.channelLookup['session'] = SimpleSession
return IConchUser, user, nothing
factory = SSHFactory()
factory.privateKeys = {'ssh-rsa': privateKey}
factory.publicKeys = {'ssh-rsa': publicKey}
factory.portal = Portal(SimpleRealm())
factory.portal.registerChecker(FilePasswordDB("ssh-passwords"))
reactor.listenTCP(2022, factory)
reactor.run()
Run this server and connect to it (ssh -p 2022 localhost
) and you should see output like this:
PTY allocation request failed on channel 0
This session is very simple. Goodbye!
Connection to localhost closed.
Next time I'll talk about what that first line of output means and ways to avoid it.
Thursday, March 24, 2011
Twisted Conch in 60 Seconds: Password Authentication
Welcome back to Twisted Conch in 60 Seconds, a series about writing custom SSH software with Twisted. Last time, I showed you how to set up a basic SSH server which could accept connections and then reject all authentication attempts. This time we'll extend that server to support password authentication.
Let's consider the factory created in the first example:
factory = SSHFactory()
factory.privateKeys = {'ssh-rsa': privateKey}
factory.publicKeys = {'ssh-rsa': publicKey}
factory.portal = Portal(None)
The object in this snippet responsible for authenticating the client is Portal(None)
. However, this portal is missing two things it really needs in order to be useful. First, it's missing a username/password checker. Twisted includes one that's easy to use and reads credentials out of a file, so we'll start with that one:
from twisted.cred.checkers import FilePasswordDB
factory.portal.registerChecker(FilePasswordDB("ssh-passwords"))
FilePasswordDB
will now try to read information from ssh-passwords in order to authenticate SSH client connection attempts. The file should be populated with lines like this:
alice:goodpassword
bob:badpassword
jpcalderone:supersecretpassword
The second thing the Portal
needs is a Realm
. After the FilePasswordDB
says a user supplied the correct username/password combination, Conch needs an object that will represent the user who just authenticated. This object will be used to determine what actions the SSH user is allowed to take, and what consequences they will have. A realm only needs to implement one method:
from twisted.conch.avatar import ConchUser
def nothing():
pass
class SimpleRealm(object):
def requestAvatar(self, avatarId, mind, *interfaces):
return IConchUser, ConchUser(), nothing
After authentication succeeds, requestAvatar
is called. The avatarId
parameter tells the realm the name of the user who just authenticated successfully. The mind
isn't used by Conch. The interfaces
indicate what kind of user is being requested; in this case, it will include twisted.conch.interfaces.IConchUser
(and we just assume that it does for now). The method must return a three-tuple. The first element is the kind of user the realm decided to give back (this must be one of the requested interfaces - again, we're assuming IConchUser
for now). This just lets the calling code know what kind of user it ended up with. The second element is the user object itself. Conch conveniently provides us with a basic user class that implements almost no behavior, so it's suitable to be used directly in this simple example. The final element of the tuple is a logout callable. This will be invoked for us when the user logs out. This example has no custom logout logic, so we return a no-op function.
Portal construction will now look like this:
factory.portal = Portal(SimpleRealm())
These are all the pieces necessary to do username/password authentication of SSH users. Here's the full code listing for this version:
from twisted.cred.portal import Portal
from twisted.cred.checkers import FilePasswordDB
from twisted.internet import reactor
from twisted.conch.ssh.factory import SSHFactory
from twisted.conch.ssh.keys import Key
from twisted.conch.interfaces import IConchUser
from twisted.conch.avatar import ConchUser
with open('id_rsa') as privateBlobFile:
privateBlob = privateBlobFile.read()
privateKey = Key.fromString(data=privateBlob)
with open('id_rsa.pub') as publicBlobFile:
publicBlob = publicBlobFile.read()
publicKey = Key.fromString(data=publicBlob)
def nothing():
pass
class SimpleRealm(object):
def requestAvatar(self, avatarId, mind, *interfaces):
return IConchUser, ConchUser(), nothing
factory = SSHFactory()
factory.privateKeys = {'ssh-rsa': privateKey}
factory.publicKeys = {'ssh-rsa': publicKey}
factory.portal = Portal(SimpleRealm())
factory.portal.registerChecker(FilePasswordDB("ssh-passwords"))
reactor.listenTCP(2022, factory)
reactor.run()
If you run this server and connect to it with one of the credentials in the password file you provide, then you should receive an extremely gratifying result along the lines of:
channel 0: open failed: unknown channel type: unknown channel
Sorry! Tune in next time to learn what a channel is and how to define one.
Wednesday, March 23, 2011
Twisted Conch in 60 Seconds: Introduction to an SSH server
Greetings once more, readers. A year ago I introduced you to the simplicity of web programming with Twisted. Since then, I've been eager to try to duplicate the success of the series for another topic. Welcome to Twisted Conch in 60 Seconds.
Twisted Conch implements the SSH protocol for both clients and servers. It also implements client and server applications. It is also a library for writing custom SSH client and server software. This series will focus on using Twisted Conch as an SSH library, and the first articles will cover writing custom SSH servers, with clients covered later.
In this first installment, the goal is to set up an SSH server which will accept SSH client connections but
which cannot actually authenticate any users (so no one will be able to log in).
Before getting into any code, the first thing we need for an SSH server is a server key. This can be generated using ssh-keygen from OpenSSH or with Conch's own key generation tool, ckeygen. Since ample examples of ssh-keygen can be found elsewhere, here's an example of generating a key with ckeygen:
exarkun@boson:/tmp$ ckeygen -t rsa -f id_rsa
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in id_rsa
Your public key has been saved in id_rsa.pub
The key fingerprint is:
d5:1d:ba:47:a4:75:1a:fe:d4:ba:46:2e:44:67:0d:8b
exarkun@boson:/tmp$ ls -l id_rsa id_rsa.pub
-rw------- 1 1000 1000 886 2011-03-23 22:59 id_rsa
-rwxr-xr-x 1 1000 1000 226 2011-03-23 22:59 id_rsa.pub
exarkun@boson:/tmp$
from twisted.conch.ssh.factory import SSHFactory
SSHFactory is an IProtocolFactory which is used to create protocol instances to handle new connections that arrive at a listening port. SSHFactory in particular creates protocol instances that can carry on the server side of an SSH connection. We need an instance to hook up to a listening port:
factory = SSHFactory()
Nothing very exciting going on there. SSHFactory doesn't take any initializer arguments. It does have some useful attributes you can set, though. Two important attributes are publicKeys and privateKeys. These define what key the server uses to identify itself to clients. A server can have multiple keys, so these attributes are bound to dictionaries. In this example we'll just give the server one key though. For this, we need to have a couple Key objects:
from twisted.conch.ssh.keys import Key
with open('id_rsa') as privateBlobFile:
privateBlob = privateBlobFile.read()
privateKey = Key.fromString(data=privateBlob)
with open('id_rsa.pub') as publicBlobFile:
publicBlob = publicBlobFile.read()
publicKey = Key.fromString(data=publicBlob)
factory.privateKeys = {'ssh-rsa': privateKey}
factory.publicKeys = {'ssh-rsa': publicKey}
from twisted.cred.portal import Portal
factory.portal = Portal(None)
Later we'll revisit this attribute so that users can actually log in to the server.
The example is almost done now. The only thing left to do is listen on a port with this factory and run the reactor. For this we'll need to import the reactor. We have several options for how to listen on a port, but for this example I'll use the classic reactor.listenTCP:
from twisted.internet import reactor
reactor.listenTCP(2022, factory)
This listens on TCP port 2022. Whenever a connection arrives, factory will be used to create a new protocol instance to handle it. This factory is going to create SSHServerTransport instances, but don't worry about that for now.
Since the reactor is the mainloop that drives any Twisted-based application, nothing actually happens until we run it:
reactor.run()
With that, you now have an SSH server. It's functional enough to prove its identity to clients that connect to it and even accept authentication attempts from them, but it will always reject their attempts. So it's not the most useful SSH server (but add a little logging and maybe you have a simple SSH honey pot!), but if you tune in next time, I'll demonstrate how it can be extended to support some more useful authentication options. Here's a full code listing for this example:
from twisted.cred.portal import Portal
from twisted.conch.ssh.factory import SSHFactory
from twisted.internet import reactor
from twisted.conch.ssh.keys import Key
with open('id_rsa') as privateBlobFile:
privateBlob = privateBlobFile.read()
privateKey = Key.fromString(data=privateBlob)
with open('id_rsa.pub') as publicBlobFile:
publicBlob = publicBlobFile.read()
publicKey = Key.fromString(data=publicBlob)
factory = SSHFactory()
factory.privateKeys = {'ssh-rsa': privateKey}
factory.publicKeys = {'ssh-rsa': publicKey}
factory.portal = Portal(None)
reactor.listenTCP(2022, factory)
reactor.run()