Monday, August 2, 2004

Python Developers Considered Harmful

This morning I learned that the '@decorator' patch had been applied and would be distributed as part of Python 2.4a2. I fumed and wasted a good couple hours going back and forth about it with the denizens of #twisted. We go nowhere, except frustrated, disappointed, and disillusioned (You thought we were jaded before? You ain't seen nothing yet). Afterwards I went and tried to catch up on all the python-dev discussion of the feature. Not much to be found. Where was '@decorator' discussed? I see one post that mentions it offhand, and no followup. Numerous developers I spoke with felt the same way. So I stewed about it all day.

I took out the trash and went for a run at about 11pm tonight. When I got back I sat down and wrote this.

http://divmod.org/users/exarkun/decorate.py

In brief, it defines a function "decorate" and a metaclass "DecoratableType" which may be used in this manner:


class Demo(object):
__metaclass__ = DecoratableType

decorate(staticmethod)
def foo(x, y):
print 'Static method foo called with', x, y


This closely parallels the '@decorator' syntax, but requires no changes to the interpreter.

What are its advantages over the '@decorator' syntax?

1) It's pure-python.

2) It requires no syntax changes.

3) It works in Python 2.2, 2.3, and yes, 2.4a1, three times as many Python versions as will support the new syntax (once 2.4 is actually released ;).

4) It supports arbitrary expressions to specify decorators (The '@' patch only supports "dotted names". For example, "@foo" and "@foo.bar(baz)" will work, but "@foo or bar" will not)

What are its disadantages (and there sure are some)

1) It depends on source line numbers. It's possible that this information could be unavailable in certain environments (I can't think of any, but I'm sure there are some). The posted version also will not deal with intervening whitespace, comments, or decorate() calls well. This can be fixed without very much difficulty.

2) The posted version only works for classes that use the defined metaclass. It would be possible to remove this restriction, but only by hooking into some other mechanism for performing additional processing to the objects involved. This could be in the form of an import hook or a function which is manually invoked.

Hardly anyone generally comments on my blog posts. I hope that everyone who reads this and thinks a solution along the lines of the one described here will comment and let me know, and everyone who thinks the '@decorator' proposal is better will comment and tell me why I'm wrong.

22 comments:

  1. I agree. completely.

    ReplyDelete
  2. You are my squishy friend!

    ReplyDelete
  3. I decided to go ahead and post this to python-dev tonight rather than wait for further feedback, as people seemed generally in favor of the idea on IRC.

    Here's a link to the post on gmane:

    http://article.gmane.org/gmane.comp.python.devel/61639

    ReplyDelete
  4. I also have found that the pie-syntax is much less offensive after using it for a bit. I deliberately went and reimplemented a bunch of code that already had decorated functions, and this format is much much clearer to my eyes. I just did a quick change to using a decorate type function, and it's much harder to see what's going on.

    We already have 'def ' standing out nicely in the source - I see no reason that decorators shouldn't also stand out some.

    My comment on both your implementation and PJE's amazing piece of work is that they seem rather brittle - more so than I'd be happy with in a release of python.

    ReplyDelete
  5. Line Noise: it's not a question of *whether* you use punctuation, but *how much*. @ in isolation is not a big problem. Random crap like this making its way into the language for such corner cases, is. Once we have @ for decorators and $ for string interpolation and heck, maybe ??~ for callback registration... what next?

    'def ' stands out without punctuation.

    I can grep for 'decorate' a lot more easily than I can grep for '@'. I don't know about you, but I work with email a fair amount in python.

    Brittleness / fragility: how are you measuring this? You think random changes to the syntax introduce more instability than the use of the well-documented and used-elsewhere sys._getframe? The implementation depends on sys.getframe; the interface is identical to the @ syntax.

    I hate the "pie" syntax. I think it is a blight on the language. Adding punctuation-based syntax is something that should be thought very long, and very hard about: there are a lot of things that I could see python programmers doing a lot more frequently than adding decorators, and wanting a much terser syntax for. Not only is it ugly and inconsistent with Python, the Java-based history of the decoration operator is dubious at best. (Let's remember that it comes from the @deprecated convention, which adds semantics that affect compiled classes into *COMMENTS*.)

    The worst thing about this, in my opinion, is the overhead this imposes on new python programmers. The current way that decorators work - foo = bar(foo) - was a great teaching tool to illustrate how functions are really objects. The @ syntax obfuscates this difficult-to-explain fact even further.

    ReplyDelete
  6. Did you also see the followup where the bug was fixed? There's a distinction between "fragile" and "has a simple bug".

    ReplyDelete
  7. Just thought I'd note that I followed up on my own blog here and here.

    ReplyDelete
  8. > I don't like the various pure-python schemes - they are either fragile
    > (relying on sys._getframe et al) or involve additional complexity like
    > metaclasses. If you want backwards compatibility you can just use the
    > current way of doing things (explicitly saying func = staticmethod(func)
    > or whatever).

    What's fragile about sys._getframe()? _Zope_ uses it extensively. It
    will be supported forever. ;) Other parts of the stdlib use it, too. If
    it is fragile, should not those usages be removed?

    As to the complexity of metaclasses, that seems rather unreasonable.
    What about the complexity of the interpreter core changes required to
    implement '@'? I'll bet they require more than 30 lines of C.

    > I'm sure I haven't convinced anybody, and in a way that is the point
    > - nobody is changing their mind. There are three contenders for the
    > syntax and the arguments for and against each one just go round and
    > round. This is a chance to get the feature into python. If we wait,
    > what are we waiting for? A flash of inspiration from someone
    > producing a syntax that everyone loves? I would say that's unlikely.

    I'm happy without the feature if no one can agree on how it should
    work. Doesn't it say anything to you that after more than a year of
    debate, there is no concensus?

    ReplyDelete
  9. I elaborated a bit more about this at
    http://pyalot.blogspot.com/

    ReplyDelete
  10. As far as brittle - JP's version relies on the line numbers in the source. How is this not a brittle solution?

    ReplyDelete
  11. "Doesn't it say anything to you that after more than a year of
    debate, there is no concensus?"

    No, it says that this is a perfect example of what FreeBSD calls a "bike shed".

    The case for having _some_ way to spell decorators has been well made over the last year - I'm not going to go back and dig out the various posts (there have been hundreds and hundreds on the topic). I was initially against having special syntax for decorators, but have come around to the idea. And now that they're in, I find them a simple way to spell out some rather funky behaviour.

    The remaining point has been a never-ending discussion over syntax. This is not without precedent - consider the endless hell of the ternary operation discussion.

    I would really like it if, after a2 is out, everyone took some time to experiment with the new syntax. Rather than letting a visceral "it's an @ sign - fear the perl! fear the perl!" reaction guide you, actually _use_ the feature for a bit.

    ReplyDelete
  12. I was happy with the resolution of the ternary debate :) I've also got a CVS checkout of Python and been playing with the decorator syntax since Monday morning. I'll keep using it, though. Like I said earlier, I'm sure I'll get used to it, and that doesn't make me like it any more.

    ReplyDelete
  13. Probably another feature I'll never use, so I can't get too wrapped up about it. But I truly wish they'd spend this energy on housekeeping. So much of Python is showing signs of neglect, then stuff like this comes around.

    It's rather discouraging. I wish I could do something about it, but I can't at the moment, or I'd be scratching like a big dog. :-(

    ReplyDelete
  14. About once every release or two, some feature comes along that makes me really nervous. I start to wonder if Python's development is starting to sabotage its superior maintainence characteristics. I usually conclude that maintainability is suffering to sate the feature creeps, but Python is still the easiest to use and maintain of all the languages I know.

    I really hate the pie syntax. It's only obvious if you care about the feature enough to have researched it. It's not going to be easy to figure out what it means if you don't know what it is. As "the python guru" in a shop full of "regular programmers", I foresee a lot of puzzlement and muddled explanations.

    But Python will still be the easiest to use and maintain, for now.

    ReplyDelete
  15. well, speaking of punctuation you could grep for something more along the lines of r'^\s*@'

    ReplyDelete
  16. Coincidentally this would also catch every epydoc directive in the source, if you use that, which we do. You lose.

    ReplyDelete
  17. Line numbers in the source tend not to change at runtime. And it doesn't _need_ to depend on line numbers, it's just the proposed straw-man implementation which does.

    ReplyDelete
  18. Yes, I did. I tend to measure fragility by counting bugs and bug severity, even if they can be fixed easily, then multiplying by a million for every line of C involved ;-).

    Right now without the bonus multiplier, your technique - 1, JP's - 0.

    ReplyDelete
  19. I don't, so it's not a loss for me, but that's a good point; parsing code with regexps is fragile as hell.

    ReplyDelete
  20. Personally I don't care much about this particular aspect. Greppability is a nice, but not a requirement for me. There are lots of features in Python you can't grep for, or that are ambiguous to grep for, already.

    It is just another example of how so many of the arguments in favor are purely specious.

    It's easier to grep for? No, it isn't, @ is a very common character already, 'decorate' is far less common.

    It's necessary to have syntax to put the decorator before the method? No, JP's suggestion clearly proves otherwise.

    "If you don't like it, don't use it?" or "If you want compatibility, you can just keep doing it the old way." I've heard the same people who claim this holding their noses at Perl and C++ for the same attitude. I don't buy it.

    "The alternatives are fragile." - I can think of about three different metrics for "fragile" where the pie syntax loses. How does it possibly win?

    There are more, of course, there is a book's worth of ranting on python-dev. I've read a nauseating amount of it, and I honestly can't understand why the debate hasn't been squashed in the same way the ternary operators debate was. The arguments I have yet read in a megabyte or so of text are circular and at times self-contradictory.

    ReplyDelete
  21. I think the reason is that ternary operators really weren't very useful, but decorators are. People are just arguing about *which* syntax, it's pretty clear that a lot of people do see the need for the feature.

    ReplyDelete