Welcome to the tenth installment of "Twisted Web in 60 90 seconds". Previously I gave an example of a Resource which generates its response asynchronously rather than immediately upon the call to its render method. Though it was a useful demonstration of the NOT_DONE_YET feature of Twisted Web, the example itself didn't reflect what a realistic application might want to do. In this installment, I'll introduce Deferred, the Twisted class which is used to provide a uniform interface to many asynchronous events, and show you an example of using a Deferred-returning API to generate an asynchronous response to a request in Twisted Web1.
Deferred is the result of two consequences of the asynchronous programming approach. First, asynchronous code is frequently (if not always) concerned with some data (in Python, an object) which is not yet available but which probably will be soon. Asynchronous code needs a way to define what will be done to the object once it does exist. It also needs a way to define how to handle errors in the creation or acquisition of that object. These two needs are satisfied by the callbacks and errbacks of a Deferred. Callbacks are added to a Deferred with Deferred.addCallback; errbacks are added with Deferred.addErrback. When the object finally does exist, it is passed to Deferred.callback which passes it on to the callback added with addCallback. Similarly, if an error occurs, Deferred.errback is called and the error is passed along to the errback added with addErrback. Second, the events that make asynchronous code actually work often take many different, incompatible forms. Deferred acts as the uniform interface which lets different parts of an asynchronous application interact and isolates them from implementation details they shouldn't be concerned with.
That's almost all there is to Deferred. To solidify your new understanding, now consider this rewritten version of DelayedResource which uses a Deferred-based delay API. It does exactly the same thing as the previous example. Only the implementation is different.
First, the example must import that new API I just mentioned, deferLater:
from twisted.internet.task import deferLater
Next, all the other imports (these are the same as last time):
from twisted.web.resource import Resource
from twisted.web.server import NOT_DONE_YET
from twisted.internet import reactor
With the imports done, here's the first part of the DelayedResource implementation. Again, this part of the code is identical to the previous version:
class DelayedResource(Resource):
def _delayedRender(self, request):
request.write("<html><body>Sorry to keep you waiting.</body></html>")
request.finish()
Next I also need to define the render method. Here's where things change a bit. Instead of using callLater, I'm going to use deferLater this time. deferLater accepts a reactor, delay (in seconds, as with callLater), and a function to call after the delay to produce that elusive object I was talking about above in my description of Deferreds. I'm also doing to use _delayedRender as the callback to add to the Deferred returned by deferLater. Since it expects the request object as an argument, I'm going to set up the deferLater call to return a Deferred which has the request object as its result.
def render_GET(self, request):
d = deferLater(reactor, 5, lambda: request)
The Deferred referenced by d now needs to have the _delayedRender callback added to it. Once this is done, _delayedRender will be called with the result of d (which will be request, of course — the result of (lambda: request)()).
d.addCallback(self._delayedRender)
Finally, the render method still needs to return NOT_DONE_YET, for exactly the same reasons as it did in the previous version of the example.
return NOT_DONE_YET
And with that, DelayedResource is now implemented based on a Deferred. The example still isn't very realistic, but remember that since Deferreds offer a uniform interface to many different asynchronous event sources, this code now resembles a real application even more closely; you could easily replace deferLater with another Deferred-returning API and suddenly you might have a resource that does something useful.
Finally, here's the complete, uninterrupted example source, as an rpy script:
from twisted.internet.task import deferLater
from twisted.web.resource import Resource
from twisted.web.server import NOT_DONE_YET
from twisted.internet import reactor
class DelayedResource(Resource):
def _delayedRender(self, request):
request.write("Sorry to keep you waiting.")
request.finish()
def render_GET(self, request):
d = deferLater(reactor, 5, lambda: request)
d.addCallback(self._delayedRender)
return NOT_DONE_YET
resource = DelayedResource()
1I know I promised an example of handling lost client connections, but I realized that example would also involve Deferreds, so I wanted to introduce Deferreds by themselves first. Tune in next time for the example I told you I'd show you this time.
This series of posts is fantastic. Thank you!
ReplyDeleteCan you explain why render_GET doesn't return a Deferred instead of NOT_DONE_YET? It would
seem more consistent with other Twisted APIs to return a Deferred instead of a flag when the result of
a method isn't ready yet.
For this case, you might use twisted.web.client.getPage or the newer and more featureful twisted.web.client.Agent's request method (so new, in fact, that it's not even in trunk yet, though).
ReplyDeletethomasvs asked the same question in a comment on the earlier async response post. :) Have a look, http://jcalderone.livejournal.com/50226.html?thread=113970#t113970
ReplyDeleteI'm wondering just how expensive such a refactoring would be.
ReplyDeleteAnd, for the part than few people care, that's because you don't see much people actually using t.w. for their projects...
Looking forward to future installments. Cheers!
ReplyDeleteThanks, the full story actually involves basic authentication and a png response, so I guess the Agent approach is the one to use. I'm wondering though whether there's anything like urllib2.HTTPPasswordMgrWithDefaultRealm and urllib2.HTTPBasicAuthHandler to use with it?
ReplyDeleteThanks! And I appreciate the feedback a lot. :)
ReplyDeleteAt some point the Agent API will support this, but it doesn't yet. Until then, you can sort of fake this with getPage by adding your own Authorization header (pass a dict for the headers keyword argument). The Agent replacement for this will be somewhat nicer.
ReplyDeleteOk now I see how adbapi interaction is pretty straightforward. Now I wonder how to do html templates, and athena--but that's what nevow is for, but how do you do asynchronous adbapi interaction with nevow or how do I do templates and athena with twisted.web?
ReplyDeleteadbapi and nevow are not related to each other in any way. You can serve a static file or a random string generator with Nevow, it's just a template system.
ReplyDeleteIt's hard to say until it's actually done, of course. :) I have the sense that it wouldn't be a huge undertaking, but it's also not trivial. A few days of work would probably mostly accomplish the task.
ReplyDeleteNevow is a bit simpler since it has built-in support for Deferreds. Anything that you can return from a render method, you can instead make the result of a Deferred that you return from that render method.
ReplyDeleteI've just implemented a testing server which does basic validation of params from a client request and returns success/fail and error/success messages after a brief delay; everything having to do with the delay, deferred, etc. is pretty much as in this example, just filled out a bit more. When requested from my browser, the deferred waits a few seconds (pseudo-random delay of 1-3 seconds or so) and the response HTML appears in my browser exactly as it did before I added the delay. However, I notice that even after the response is displayed, the tab still displays its 'loading...' symbol and "Loading..." is the tab's text, as well - as if the page has not entirely loaded - even though all the content I expect has arrived. Until I hit the "Stop loading this page" button, the "Loading...." behavior continues. Is there something missing in this example to tell the client (browser in this case) that the request is completed?
ReplyDeleteThanks,
Ken
It sounds like you're not calling request.finish() (or an error is occurring in your code before it gets to the request.finish() call). It's this finish call which tells the browser there's nothing more to receive, so it should be pretty directly tied to the "Loading ..." UI.
ReplyDeleteThanks, that's what I was thinking too, so I looked closer and found I had entered: 'request.finish' instead of 'request.finish()'. Doh. Works now.
ReplyDeleteKen