Thursday, November 17, 2005

Axiom Powerups

Q: When I run axiomatic -d my.axiom start, a web server starts up. How does that work?

A: The web server is started and stopped by an instance of an Axiom Item subclass, WebSite powers up the database into which it is installed. Power up is the term used for a particular manner in which one Item can be used to enhance the behavior of another Item. It means that WebSite is calling powerUp on the site store1, passing itself as the powerup Item and twisted.application.service.IService as the interface. The start subcommand of axiomatic kicks off the standard Twisted service startup events: an Axiom database behaves as an twisted.application.service.IServiceCollection, with each IService powerup treated as a child. In this way, things like WebSite are given startup notification (as well as shutdown notification, when the database is about to be closed).

Q:Wait, what? You make no sense.

A: Okay then, here's some example code, perhaps it will make more sense:

from zope.interface import Interface, Attribute
from axiom import store, item, attributes

class IEngineEnhancement(Interface):
"An improvement to a ship's engine."
speed = Attribute("How much faster this makes a ship go")

class IShieldEnhancement(Interface):
"An improvement to a ship's shields."
force = Attribute("How much stronger this makes a ship")

class Spaceship(item.Item):
typeName = 'spaceship'
schemaVersion = 1

score = attributes.integer(default=0, doc="how well the player is going")

def getMaxSpeed(self):
return sum(enginePowerup.speed for enginePowerup in self.powerupsFor(IEngineEnhancement))

def getMaxShields(self):
return sum(shieldPowerup.force for shieldPowerup in self.powerupsFor(IShieldEnhancement))

class HyperDrive(item.Item):
typeName = 'hyperdrive'
schemaVersion = 1

speed = attributes.integer(default=100)

# Here is the powering up part
def installOn(self, ship):
ship.powerUp(self, IEngineEnhancement)

class SuperShield(item.Item):
typeName = 'supershield'
schemaVersion = 1

force = attributes.integer(default=50)

# And here is a little more.
def installOn(self, ship):
ship.powerUp(self, IShieldEnhancement)

def main():
st = store.Store()
ship = Spaceship(store=st)
driveUp = HyperDrive(store=st, speed=35)
shieldUp = SuperShield(store=st, force=1000)
extraDriveUp = HyperDrive(store=st, speed=20)

for powerUp in driveUp, shieldUp, extraDriveUp:

print "Your ship's maximum speed is %d." % (ship.getMaxSpeed(),)
print "It has a maximum shield level of %d." % (ship.getMaxShields(),)

if __name__ == '__main__':

Q: Hey! Isn't this just a bridge table with a funky API?

A: Yes. Yes it is. We find the pattern so useful, we added special support for it.

Site Store is the name of the Axiom Store which is at the "top" of everything: it is the database in the directory you pass to "axiomatic -d", it contains login data and other site-wide state. It is conceptually distinct from user stores and application stores and serves a different purpose from each of them.

Wednesday, November 16, 2005

Axiom Queries

Yesterday, I demonstrated how to write an axiomatic plugin. That was pretty cool, but to actually do anything interesting, you probably need to know how to define Axiom Items and perform queries. Without any ado, here's an Item definition for a popular kind of thing, a tag:

from axiom import item, attributes

class Tag(item.Item):
typeName = 'tag'
schemaVersion = 1

name = attributes.text(doc="""
A short, descriptive string giving this tag some meaning. eg, "fruit" or "vacation pictures".
""", allowNone=False)

item = attributes.reference(doc="""
The object which has been tagged.
""", allowNone=False)

There isn't anything too complex going on here:

  • axiom.item.Item is the base class for all database-stored objects. axiom.attributes defines a few classes that let you declare the schema of your Item: text for unicode strings, bytes for binary strings, boolean, integer, timestamp, and path for the obvious things. reference allows you to refer to any other Item (of any type) in the database. You can also define new attribute types, if your application demands it.

  • typeName associates the Python class with a database type. This lets you move your class between modules or even change its name without disrupting the database (a future version of Axiom may default this to the name of the Python class, so that you only need to define it if you later move or rename the class).

  • schemaVersion is used to track changes made to the schema declaration, allowing your data format to change while still being able to access old databases (a future version of Axiom may default this to 1, so you only need to define it after the first schema change).

  • This tag class has text attribute "name", the meaning of which should be obvious, and an attribute "item" which is a reference to another Item in the database. So, this lets you tag any object in your database.

  • Each attribute is made mandatory by passing allowNone=False. Without these, attributes could be omitted as desired, which doesn't make sense in this case.

Now let's see how we can put this into use:

Python 2.4.2 (#2, Sep 30 2005, 21:19:01)
[GCC 4.0.2 20050808 (prerelease) (Ubuntu 4.0.1-4ubuntu8)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> (Tag class definition elided)
>>> from axiom import store
>>> s = store.Store()
>>> Tag(store=s, name=u'system', item=s)1
<__main__.Tag object at 0xb783cadc>
>>> s.query(Tag).count() # How many Tags are there in the database, total?
>>> t = list(s.query(Tag))[0]
>>> print t,, t.item # What is the tag I just found?
<__main__.Tag object at 0xb783cadc>, system, <Store (in memory)@0xb7c969cc>
>>> takes a few more optional arguments, too.

  • s.query(Tag, Tag.item == anObject) if I want all the Tags that apply to a single object.

  • s.query(Tag, Tag.item == anObject).getColumn("name") is how I would get just the name of each Tag.

  • s.query(Tag, attributes.OR( == u"fruit", == u"vacation photos")).getColumn("item") gives me all the Items tagged "fruit" or "vacation photos".

  • I can also limit, offset, or sort with those keyword arguments:

    s.query(Tag, Tag.item == anObject,, limit=10, offset=5)

    This gives me the 5th through 15th Tags applied to anObject in descending lexicographically-sorted tag name order.

Tomorrow: Axiom Powerups

1: Yes, the Store itself can be referenced just like any Item inside it. This is convenient for a number of reasons, although in this case it is just convenient because I don't have any other Items handy to use in the example.

Tuesday, November 15, 2005

Adding Axiomatic plugins

axiomatic has featured prominently in many of my recent blog posts, but what is it?

As one might guess, in simplest terms, axiomatic is a tool for manipulating Axiom databases. Going into a little more detail, axiomatic is a command line tool which gathers axiom.iaxiom.IAxiomaticCommand plugins using the Twisted plugin system and presents them as subcommands to the user, while providing the implementations of these subcommands with access to objects that pretty much any Axiom manipulation code is going to want (currently, that amounts to an opened Store instance). Twisted's option parser is used as the basic unit of functionality here.

I've already talked about several IAxiomaticCommand implementations: web, mantissa, and start. The first two of these add new Items to an Axiom database or change the values associated with existing Items. The last "starts" an Axiom database (more on what that means in a future post!).

So what you really want to know is how do I write an axiomatic plugin? Let me tell you, it could scarcely be easier:

from zope.interface import classProvides
from twisted import plugin
from twisted.python import usage
from axiom import iaxiom
from axiom.scripts import axiomatic

class PrintAllItems(usage.Options, axiomatic.AxiomaticSubCommandMixin):
# This one tells the plugin system this object is a plugin
# This one tells axiom it is an axiomatic plugin

# This is how it will be invoked on the command line
name = "print-some-items"

# This will show up next to the name in --help output
description = "Display an arbitrary number of Items"

optParameters = [
# We'll take the number of Items to display as a command
# line parameter. This is "--num-items x" or "-n x" or
# any of the other standard spellings. The default is 5.
('num-items', 'n', '5', 'The number of Items to display'),

def postOptions(self):
s = self.parent.getStore()
count = int(self.decodeCommandLine(self['num-items']))
for i in xrange(count):
print s.getItemByID(i)
except KeyError:
# The Item didn't exist

(Okay, seriously - I know this is a bit long: we'll be working on lifting a lot of the boilerplate out to simplify things; expect about half of the above code to become redundant soon). I just drop this into a file in axiom/plugins/ (there are other places I could put it, if I wanted; read the Twisted plugin documentation to learn more) and I've got a new subcommand:

exarkun@boson:~$ axiomatic --help
Usage: axiomatic [options]
-d, --dbdir= Path containing axiom database to configure/create
--help Display this help and exit.
mail Accept SMTP connections
userbase Users. Yay.
mantissa Blank Mantissa service
web-application Web interface for normal user
web Web. Yay.
web-admin Administrative controls for the web
radical omfg play the game now
click-chronicle-site Chronicler of clicking
encore Install BookEncore site store requirements.
sip-proxy SIP proxy and registrar
vendor Interface for purchasing new services.
vendor-site Required site-store installation gunk
print-some-items Display an arbitrary number of Items
start Launch the given Axiomatic database

Neat, huh?*

In case you're dying to see the output of this new command:

exarkun@boson:~/Scratch/Run/demo$ axiomatic -d my.axiom/ print-some-items
<axiom.item._PowerupConnector object at 0xb6fc770c>
<axiom.item._PowerupConnector object at 0xb6fc770c>
<axiom.userbase.LoginSystem object at 0xb6f42104>

Monday, November 14, 2005

Redirecting HTTP request logs

Okay I promised this a few days ago and never followed up. Here's how to get that annoying HTTP traffic logging out of your main log (or if you are using axiomatic start -n, off of your terminal):

exarkun@boson:~/Scratch/Run/demo$ axiomatic -d my.axiom/ web --http-log httpd.log

That's all. If the server was running while you did this, you will have to restart for it to take effect ;) Of course, someone could contribute a patch to change that...

One other thing: this command, unlike the others I've shared, depends on current trunk and will not work in a released version of Mantissa. There's probably a Mantissa release coming up pretty soon, though.

Friday, November 11, 2005

Configuring Static Resources on a Mantissa Server

Okay listen up, it's time for static file configuration.

exarkun@boson:~/Scratch/Run/demo$ axiomatic -d my.axiom/ web --static images:/usr/share/pixmaps/

What I've just done is configure the Mantissa server I created in a previous post to serve the contents of /usr/share/pixmaps at http://localhost:8888/images/.

There is an easy way to determine if a Python RPC library is broken

If the library employs Python's pickle module, it is broken. Period.

When will people learn that pickle is not suitable for this task? "Oh, it looks so easy." "Oh, it's so fast, just look at those objects fly." "Oh it will core my process when handling a maliciously constructed string, there goes my server." "Oh, it allows arbitrary code to be executed by a remote party, woops there goes my credit card database."

Not convinced? Run this code on your computer:

import pickle
pickle.loads("cposix\nsystem\np0\n(S'cat /etc/passwd'\np1\ntp2\nRp3\n.")

Wake up. This is not news. The pickle documentation explicitly points out the fact that it is not intended to be used in this fashion (although frankly, this warning could be a little closer to the beginning of the pickle documentation). Stop doing it. Stop releasing software that does it. Just stop, already.

Thursday, November 10, 2005

Configuring HTTP and HTTPS ports, and static content

Yesterday I showed you how to create and start the most minimal Mantissa server possible. If you ran the commands from my previous post, you saw something like this in your terminal:

exarkun@boson:~/Scratch/Run/demo$ axiomatic -d my.axiom mantissa
exarkun@boson:~/Scratch/Run/demo$ axiomatic -d my.axiom start -n
2005/11/11 01:45 EST [-] Log opened.
2005/11/11 01:45 EST [-] twistd SVN-Trunk (/usr/bin/python 2.4.2) starting up
2005/11/11 01:45 EST [-] reactor class: twisted.internet.selectreactor.SelectReactor
2005/11/11 01:45 EST [-] starting on 8080
2005/11/11 01:45 EST [-] Starting factory < instance at 0xb6ef2fec>

Awesome, I hear you exclaim. But what does it do, you most likely then complain. A fair question, to be sure!

In this minimal configuration, you have a few things:

  • A web server running on port 8080 serving, at /, a static template consisting primarily of a login link.

  • A login page, linked from /.

  • An administrative account, via which you can log in to this server (the username is "admin@localhost", the password is whatever you typed in at the password prompts earlier).

  • When logged in as the administrative account, a small page containing site statistics (such as number of logins, number of users, etc) and a Python REPL.

I'll cover some of these things in further detail in tomorrow's post. For now, let's go over the things we can do to this server.

Before trying to alter any of the server's configuration, let's see what's there already:

exarkun@boson:~/Scratch/Run/demo$ axiomatic -d my.axiom/ web --list
Configured to use HTTP port 8080.
Configured to use HTTPS port 0 with certificate None
Sessionless plugins:
/static/webadmin => item(DeveloperSite) (prio. 0)
/static/mantissa => file(/usr/lib/python2.4/site-packages/xmantissa/static) (prio. -255)
Sessioned plugins:
/ => item(FrontPage) (prio. -257)

Here we can see a /static/webadmin/ is being served by a DeveloperSite item (whatever that is) and /static/mantissa/ is being served out of the filesystem from the given path. Additionally, there is a page which requires a session at /, being served by a FrontPage item (whatever that is). Also, we see which ports the server is configured to use. Let's say someone is already using 8080, and we also want to turn on HTTPS:

exarkun@boson:~/Scratch/Run/demo$ axiomatic -d my.axiom/ web --port 8888 --secure-port 8889 --pem-file server.pem
exarkun@boson:~/Scratch/Run/demo$ axiomatic -d my.axiom/ web --list
Configured to use HTTP port 8888.
Configured to use HTTPS port 8889 with certificate server.pem
Sessionless plugins:
/static/webadmin => item(DeveloperSite) (prio. 0)
/static/mantissa => file(/usr/lib/python2.4/site-packages/xmantissa/static) (prio. -255)
Sessioned plugins:
/ => item(FrontPage) (prio. -257)

server.pem doesn't exist yet, so I guess I'll create that too, using a tool that comes with Vertex (maybe you want to install that, too):

exarkun@boson:~/Scratch/Run/demo$ certcreate -C US -s Massachusetts -c Boston -o 'Divmod, Inc.' -h localhost -e support@localhost -f server.pem
Wrote SSL certificate:
Certificate For Subject:
Common Name: localhost
Organization Name: Divmod, Inc.
Organizational Unit Name: Security
Locality Name: Boston
State Or Province Name: Massachusetts
Country Name: US
Email Address: support@localhost

Common Name: localhost
Organization Name: Divmod, Inc.
Organizational Unit Name: Security
Locality Name: Boston
State Or Province Name: Massachusetts
Country Name: US
Email Address: support@localhost

Serial Number: 1
Digest: 59:1F:25:FF:C9:18:98:31:B2:B8:B9:99:FA:99:4C:D0
1024-bit RSA Key Pair with Hash: 5bf8af0e6c7e01e1004b54dd3f0a983f

This one's self signed, since I don't have a VeriSign certificate lying around to sign it with... Anyway, now I'm set for HTTPS:

exarkun@boson:~/Scratch/Run/demo$ axiomatic -d my.axiom start -n
2005/11/11 02:08 EST [-] Log opened.
2005/11/11 02:08 EST [-] twistd SVN-Trunk (/usr/bin/python 2.4.2) starting up
2005/11/11 02:08 EST [-] reactor class: twisted.internet.selectreactor.SelectReactor
2005/11/11 02:08 EST [-] starting on 8888
2005/11/11 02:08 EST [-] Starting factory < instance at 0xb7616c6c>
2005/11/11 02:08 EST [-] starting on 8889

So now you know how to alter HTTP port configuration for a Mantissa server (any Mantissa server) from the command line. I know I promised something about static content in the subject, but I've already gone on longer than I intended. I'll cover configuring static content tomorrow.

Wednesday, November 9, 2005

How To Create A Mantissa Server

So a guy comes up to me and wants to know what the absolute minimum effort to get a Mantissa server running is.

So I tell him:

  1. Install Epsilon, Axiom, and Mantissa (You already have Python 2.5, PySQLite2, and Twisted 8.2 installed, right? This step will become "apt-get install python-mantissa" just as soon as we find a debian guy, promise).

  2. Run axiomatic -d my.axiom mantissa

  3. Run axiomatic -d my.axiom start -n

You now have a Mantissa server up and running. Pretty easy, huh?

Tomorrow: port, SSL, and logfile customization.

Saturday, November 5, 2005

Epsilon, Mantissa, and ClickChronicle

Mantissa and ClickChronicle had their plugins omitted from their last releases. 0.3.1 and 0.2.0, respectively, released fixing this (and some new bookmark-related features in ClickChronicle). Epsilon 0.3.2 released with a fix for autosetup() to include Twisted plugins automatically in the future.