syseggrecipe 1.5.1

Syseggrecipe allows the reuse of system eggs in buildout installs

Description

Buildout is great for repeatable builds of Python packages. It grabs a bunch of packages off Pypi (the Python package index) and you’re done. Pure Python packages are no problem. But sometimes there are packages that are a bit harder.

Some packages require quite a number of libraries to be available, preferrably as development packages. In Debian/Ubuntu terms, this are the *-dev packages. These packages contain C code or build upon existing libraries. All in all, this is sometimes functionality that’s better provided through your OS. Who wants to build “numpy” by hand? Way nicer to just do an aptitude install python-numpy and to automatically get all the dependencies.

The syseggrecipe buildout recipe provides a handy way to re-use those perfectly packaged “system eggs”. They are already there, so why not use them? We know that buildout is best used to gather everything together on its own, but there are practical limits. Sometimes system eggs are handier.

The syseggrecipe recipe allows you to specify which eggs you want to grab from your OS and it injects just those eggs into your buildout so that you can use them without pulling in everything that’s on your system path.

Example

An example of how to use the recipe. Please note that the sysegg recipe must be the first buildout part to make sure it gets the first go at grabbing global eggs.:

[buildout]
parts = sysegg

[sysegg]
recipe = syseggrecipe
eggs =
  netCDF4

To stop the buildout when not all syseggs are installed include force-sysegg = true:

[buildout]
parts = sysegg

[sysegg]
recipe = syseggrecipe
force-sysegg = true
eggs =
  netCDF4

This way, the specified eggs must be installed globally. Otherwise they are optional (which might be a fine choice, too).

How it works

The core of the system is buildout’s concept of “development eggs”. This is a special directory (develop-eggs/ within your buildout) with pointers to Python packages that are currently being developed. These pointers take precedense over any other item. Examples include the project you’re working on, but also items you installed with mr.developer.

For every egg specified in the part, setuptools is asked for a matching distribution. If one is found, it is inserted into the develop eggs directory. There are two ways:

• If it is a proper egg, an EGGNAME.egg-link file is made in the develop-eggs/ directory that points at the correct egg.

• If it isn’t a real egg, a matching EGGNAME*.egg-info file or directory is looked for and symlinked into the develop-eggs/ directory.

Both ways are enough for setuptools to know the global egg exists. As buildout doesn’t strip out the system path (except for the abortive 1.5/1.6/1/7 releases), setuptools can find them globally. We just had to make sure it knows how to find them.

Origin

This package is a fork of (and improvement on) osc.recipe.sysegg. As such it is licensed under MIT.

Changelog

1.5.1 (2015-10-27)

• Added a little bit of extra logging. for 1.5’s feature. [reinout]

1.5 (2015-10-27)

• If a package isn’t found, it sometimes only is a matter of a missing *.egg-info file. Like ubuntu, that misses the egg-info file for mapnik, even though mapnik itself is installed.

  We now attempt to import such a package (simply using the package name) and use the directory it is installed in, if found. Note: the package name might not be a valid import name. Support might have to be added for corner cases. [reinout]

1.4 (2015-06-08)

• We now also find pip-installed packages (which end in .dist-info). (Support added by Olivier Bornet, thanks!)

1.3 (2014-11-18)

• Fixed bug: %S instead of %s (lowercase) in a log message. You’d get a ValueError: unsupported format character in recipe.py if you got hit by the bug (which only happens in a corner case).

1.2 (2014-03-19)

• Using os.path.lexists() instead of os.path.exists(), this one returns True also if there’s a symlink that leads nowhere. exists() follows the symlink so returns False for broken symlinks. In our case that means that the broken symlink isn’t removed so that the subsequent creation of a new symlink fails as the filename is already in use. Fixes #3.

1.1 (2014-03-19)

• Added Python 2.6 compatibility.

• Compensating for packages with underscore characters like MySQL_python. (Both fixes by benwah, thanks!)

1.0.1 (2013-09-17)

• Quick fix bug where buildout rmtree can’t remove egg info/link files.

1.0 (2013-09-11)

• Re-released as 1.0 as it is stable now.

0.4 (2013-09-10)

• Back to symlinking instead of copying: some egg-info files are actually directories. Downside: Windows is out of the door again.

0.3 (2013-09-10)

• We report files that we’ve added to buildout now so that buildout automatically cleans them up for us. No more pollution in our develop-eggs directory!

• Symlinking folders of non-egg distributions that we found in a system directory turned out to be a bad idea. Some distributions install files instead of directories (GDAL ends up as gdal.py, for instance). We now only copy the “egg-info” files, which turns out to be enough for setuptools to find the distributions.

• Copying (see above) instead of symlinking means it also works on windows again.

0.2 (2013-09-10)

• Fix in README for running sysegg standalone.

• Distributions that aren’t eggs but directories directly inside a sys.path directory would have the actual system folder as their location. This used to mean that everything in that system folder can erroneously be used as a system egg. Not anymore, as those directories are now symlinked directly instead of being used through a too-generic .egg-link file.

• This recipe uses symlinks for the above fix, which means it doesn’t work on windows anymore.

0.1 (2013-02-05)

• Patch code to allow for force-sysegg=false

• Add original code from osc.recipe.sysegg.

• Add buildout and setup.py.

• Added readme, changes and MIT license.