Wednesday, September 7, 2011

Releasing Python Software is Tedious

I released pyOpenSSL 0.13 a few days ago. Apart from making sure it actually worked on various platforms, updating the version number, regenerating the documentation, and sending out the release announcement, I also had to upload release files to the Python Package Index.

Uploading release files to PyPI is the part of the release process I hate the most. pyOpenSSL 0.13 had 15 files to upload to PyPI. There is no usable automated interface for uploading files to PyPI. Before I can even begin to upload them, I have to download them from the build farm where they're generated. Then, uploading just one file to PyPI requires at least 8 mouse clicks. The clicks vary depending on which file is being uploaded. I have to select a file, select its type, specify which Python version it's for, and then submit a form.

15 files, 8 clicks per file. Well, you do the math. It's not a pleasant experience. PyPI would be a much better resource if it didn't force me to specify a ton of redundant, mostly useless information every time I do a release. It would be a much better resource if it had a programmatic interface so I don't have to spend 20 minutes clicking buttons in web browser. It would be a much better resource if it didn't try to hard to discourage me from releasing software.

13 comments:

  1. You can use distribute: "python setup.py register upload"

    http://packages.python.org/distribute/setuptools.html#upload-upload-source-and-or-egg-distributions-to-pypi

    ReplyDelete
  2. In my project, I use distutils and this does the upload for me:

    python setup.py sdist upload --sign

    I've got a very simple Makefile to simplify the process:

    http://bazaar.launchpad.net/~libravatar/pylibravatar/trunk/view/head:/Makefile

    ReplyDelete
  3. http://pypi.python.org/pypi/jarn.mkrelease/3.1

    ReplyDelete
  4. "There is no usable automated interface for uploading files to PyPI."

    As others have noted, distribute (and distutils itself), offer a command line interface for this. Is there something specific those tools are missing that means you find it necessary to use the web interface?

    ReplyDelete
  5. Also, these days, PyPI advertises the command line driven upload process on the front page: http://pypi.python.org/pypi (centre 'Package Authors' box).

    MvL has also published a utility that lets you protect the upload operation with SSH (again, linked from that box on the front page):
    http://pypi.python.org/pypi/pypissh

    ReplyDelete
  6. I have to agree with the author, about both the PyPI "setup.py upload" interface being completely unusable and PyPI uploads being the most annoying part of the release process.

    From the link of @Dinoboff: "setuptools only supports source distributions and eggs". This doesn't help for uploading Windows .exe installers built with distutils (not setuptools), the thing I use PyPI for most.

    ReplyDelete
  7. That's certainly true; the main problem - as it is being discussed on catalog-sig lately - is that PyPI was born as an index, not a repo. It was not designed with automatic tools in mind - setuptools and pip hacked in later to automatically upload & download dependencies doing every sort of scraping and detecting.

    There has been a discussion lately, on catalog-sig, on whether we need a sort of "pyrepo", something like a Maven repo for Python packages, where you've got a well-defined protocol for upload, downloads, versioning, metadata, etc etc etc.

    There doesn't seem to be a general consensus on whether that's needed and there's no plan to start it soon AFAIK - also, a first phase would probably involve source-only packages, it's much harder to work with binaries compiled for different architectures and python versions.

    ReplyDelete
  8. Regarding the "setup.py upload" command, this isn't very helpful because it will not upload a package that was already built - for example, a Windows package built on another host while I am trying to release from a Linux desktop. Since automatic package building across a build farm is the only way I can actually build packages for most platforms, this rules out use of the "upload" command.

    ReplyDelete
  9. Sounds like you need a option to explicitly provide the tarball to upload when you call setup upload.

    IIRC someone asked for this feature in the past. I'd encourage you to look up in bugs.python.org and eventually to add a feature request.

    But you would need to get the tarball on the box where you run the command line of course -- not sure how you proceed right now, but the "I am on another machine than the ones where the release files are" seems like your own process problem.

    ReplyDelete
  10. Hi,

    I've found this annoying too.

    Here's the bug I just made:
    http://bugs.python.org/issue12944

    ReplyDelete
  11. tarekziade: Yes (or an msi, or an exe, or an egg, etc :), that would help a lot.

    illume: Cool!

    ReplyDelete
  12. npm (Node.js Package Manager) has this down pat. I knew nothing about node packaging, but had a package uploaded to npm in 15 minutes.

    npm uses it's own commandline language to create your npm upload account, and then you upload the package via that same command-line immediately after.

    ReplyDelete
  13. I know this is a rather old post, but I had the same problem, and came up with a solution which I explain here: http://mnunberg.github.io/2013/python-extension-windows-binaries/

    ReplyDelete