Packaging updates (switch distutils for Distribute, use setuptools-git), documentation updates

[davidjb]

In order for better dependency resolution on installation, I suggest using Distribute. It's effectively a drop-in replacement for distutils, and by specifying dependencies as install_requires in setup.py, this ensures that upon installing Sunburnt (in all manners: pip, package or source), the dependencies will be installed.

With Distribute available, setuptools-git (specified as a setup_requires) takes over from MANIFEST.in, ensuring all files in version control are packaged on release.

I've also updated a few bits of documentation regarding install and packaging.

I wanted to update the change log with details too -- and as I was unsure what the policy is on version numbering and when it gets updated, so I've selected something in line with PEP386. Change at will :)

[khorn]

+1 on anything that means I can stop tracking sunburnt's dependencies manually...

[davidjb]

Any thoughts on accepting this pull request? Having to manually manage the dependencies is less than optimal.

A switch in packaging would also help with installing test dependencies (eg can specify a set of 'extra' requirements just for the tests, etc).

[tow]

Sorry - I'm not opposed to it per se, but I haven't had a chance to think about it properly. Please don't read detailed questions below as being negative, I just want to understand what's going on.

Firstly, can you rebase this into a single patch for easier reviewing? - it's a bit painful to try and look into 11 different patches, some of which undo changes made in earlier patches in the same serues.

Secondly, can you help me understand the negative implications of doing this - I'm perfectly happy that it improves matters in certain regards, but I don't have a clear understanding of what the trade-offs are.

For a start, I'm afraid I'm really allergic to adding extra dependencies - I've been bitten by that on other people's projects too often, and I've tried really hard to make sure I have as few dependencies as possible in sunburnt.

Afaics, this adds two additional dependencies: distribute & setuptools-git. What are the implications of this, is this now tied to a particular version of Distribute? If so, what happens if somebody wants to include sunburnt in their project as well as some other third-party library which requires a different version of Distribute? Have we just made their lives more difficult? Same question for setuptools-git.

What happens at build time if I want to do a new release? Does this change the procedure I have to go through to create a tarball and release it? Does this have any other knock-on effects that are not obvious?

Sorry - I realise I'm quite conservative in this regard, but I've never used Distribute on a project & so I'm not sure of the implications.

[davidjb]

@tow Commits squashed accordingly. There's a fair bit in there, so apologies if any of it doesn't make sense.

As for your questions, the main reason I've suggested a packaging change isn't based specifically on 'you should use Distribute' but more 'distutils is extremely lacking'. As I understand it, there's PEPs happening to improve standard library packaging -- probably very best summed up by this document: http://guide.python-distribute.org/introduction.html#current-state-of-packaging

So, a negative of this is that as you say, an additional dependency is introduced. However, given that many people either use pip, easy_install or buildout -- something that is setuptools-compatible - to install software, this isn't a significant issue. Beyond this, I can't think of anything off the top of my head. For what it's worth, pip is already the installation method of choice in Sunburnt's documentation. Given the improvements (mainly automatic dependency installation, version compatibility specifications, specific dependency subsets [eg testing, production etc]) that Distribute (etc) offer, and the fact this will become standard library soon, my thoughts are it's effectively painless and worthwhile to ease people's usage of the package. For me, my background has been packages that are all setuptools-based - so the idea of needing to manually manage dependencies is unusual.

As for the setuptools-git addition, because it's a setup_requires dependency, it's only required for when the setup.py script is being run and will only be pulled in as such (see http://packages.python.org/distribute/setuptools.html). I find it's helpful to avoid needing to manually use a MANIFEST.in file. However, that said, a MANIFEST.in file isn't hard to self-manage if you want to avoid this dependency. For me though, it's helpful in avoiding missed files and resulting in broken (brown bag) releases.

As for your new releases, I'm unsure what your current workflow entails, but everything should function as normal using the setup.py file. So, to create a source distribution and upload it to PyPI:

cd sunburnt
python setup.py sdist register upload

That's about it. All things considered, it shouldn't get in the way since Distribute is based upon (some form of) distutils and pip installation is happy either way.

Phew. Hopefully that covered everything!

[tow]

Wow, that document on the current-state-of-packaging left me more confused than when I started. Also, that doc isn't up to date, since: http://wiki.python.org/moin/Distutils2 distutils2 did not go into python 3.3; distutils (original version) is still in python 3.4a0 - http://python.readthedocs.org/en/latest/library/distutils.html

As a result, I'm not sure the situation with Distribute is quite as clear-cut as you make out.

That said - I understand the urge to improve things, and it would be nice to have automatic dependency handling. The one thing that does still worry me is: what happens when:

sunburnt depends on Distribute 0.6.27 (which I think is the version you're using)
package-X (arbitrary other python library) depends on Distribute v0.7.1 (or some other version that is not the same as sunburnt)

and I'm trying to write an application using both sunburnt & package-X?

Genuine question - I don't know the answer, and if you tell me that Distribute has some magic to stop this being an issue, I'll be perfectly happy.

To be clear, my biggest worry about dependencies is not that they exist, it's what happens when they conflict for an application developer between different third party libraries.

[davidjb]

The state of Python packaging is a bit like that, unfortunately. I know there are PEPs in existence regarding these improvements, but presumably they're still in discussion. I'm hopeful of some concrete resolution, but as for when that will happen, I'm unsure. That said, given the prevalence of packages and projects that utilise setuptools or Distribute for the features they provide over standard disutils, these tools aren't likely to disappear.

As for Sunburnt, it doesn't actually have a version-specific dependency for Distribute (or even Distribute specifically, for that matter), but rather some form of package that provides the 'setuptools' namespace (Distribute wraps setuptools for backwards compatibility).

As far as I've experienced, version conflicts will not be an issue as my pull request only utilises functionality from the original setuptools specification - and as best I know, full backwards compatibility is maintained between Distribute/Setuptools. So, as long as a user has a package, of any version, of either of these, is present, installation of Sunburnt will succeed. If, for some reason, another package depended on a specific version of either library, then that version would be used and Sunburnt will continue to install without issue as the API is the same.

If a user is installing using Pip, then the Distribute-or-Setuptools dependency is automatically satisfied because Pip itself relies on one of these two being present: http://www.pip-installer.org/en/latest/installing.html#prerequisites . The same goes for installation via Buildout, and probably easy_install too - I'm unsure if it is possible to have easy_install minus setuptools.

Finally, the reason I include the distribute_setup.py script is to handle the use case of manual installation (eg python setup.py install; as in the Sunburnt documentation) -- in this case, Distribute will pull itself in. This could be made optional if the setuptools dependency is clear for any users installing in this manner.

[davbo]

This pull request makes a lot of sense to me? Having just read the comments here I'm wondering what's needed to get it merged?

@tow your concern about depending on a particular version of Distribute is understandable but as @davidjb points out, Distribute follows setuptools API so provides backward compatibility for tools like easyinstall, pip, etc.

[davidjb]

@davbo From a technical standpoint, just a rebase and probably a few minor adjustments (maybe more doco; can't hurt).

That said, there's a few things to consider as the pull request does lump a few changes together (eg whether to use Manifest.in or setup tools-git etc).

Either way, I'm happy to do the work as this will make Sunburnt easier to install.

[davidjb]

So, it looks like setuptools has re-merged with distribute. Guess that makes life simpler. I'm still keen to make automatic dependency installation a reality for this project...if things are still alive and kicking.