Recipe for installing Python scripts

## Project description

The script recipe installs eggs into a buildout eggs directory, exactly like zc.recipe.egg, and then generates scripts in a buildout bin directory with egg paths baked into them.

## Change History

### Fixes

• If a section extends another one, it should be able to override it.
• The allowed-eggs-from-site-packages should be looked for in the buildout section if it is not found locally.

### 1.0.0 (2010-08-23)

(no significant changes)

### 1.0.0b1 (2010-04-29)

Initial public version.

## Script and interpreter generation

This recipe is very similar to zc.recipe.egg, and if you are familiar with its options, you will be able to use this one easily.

The script and interpreter generation in this recipe are improved from those provided by zc.recipe.egg in two basic ways.

• The interpreter generated by the script supports all interpreter options, as opposed to the subset provided by zc.recipe.egg.
• Both scripts and interpreters from this recipe can optionally choose to include site-packages, and even sitecustomize.

The recipe takes several options. First, here’s the list of the options that overlap from the standard zc.recipe.eggs scripts recipe. After this, we’ll list the new options and describe them.

• eggs
• index
• python
• extra-paths
• entry-points
• scripts
• dependent-scripts
• interpreter
• arguments
• initialization
• relative-paths

In addition to these, the recipe offers these new options. They are introduced here, and described more in depth below.

include-site-packages
You can choose to have the site-packages of the underlying Python available to your script or interpreter, in addition to the packages from your eggs. See the section on this option for motivations and warnings.
allowed-eggs-from-site-packages

Sometimes you need or want to control what eggs from site-packages are used. The allowed-eggs-from-site-packages option allows you to specify a whitelist of project names that may be included from site-packages. You can use globs to specify the value. It defaults to a single value of ‘*’, indicating that any package may come from site-packages.

Here’s a usage example:

[buildout]
...

allowed-eggs-from-site-packages =
demo
bigdemo
zope.*


This option interacts with the include-site-packages option in the following ways.

If include-site-packages is true, then allowed-eggs-from-site-packages filters what eggs from site-packages may be chosen. Therefore, if allowed-eggs-from-site-packages is an empty list, then no eggs from site-packages are chosen, but site-packages will still be included at the end of path lists.

If include-site-packages is false, the value of allowed-eggs-from-site-packages is irrelevant.

extends
You can extend another section using this value. It is intended to be used by extending a section that uses this package’s scripts recipe. In this manner, you can avoid repeating yourself.
exec-sitecustomize
Normally the Python’s real sitecustomize module is not processed. If you want it to be processed, set this value to ‘true’. This will be honored irrespective of the setting for include-site-packages.
script-initialization
The standard initialization code affects both an interpreter and scripts. The code in script-initialization is used only for the generated scripts.

Finally, the “interpreter” entry point ignores script-initialization, scripts, and arguments, and provides yet another additional option.

name
While, by default, the interpreter recipe takes the name of the section to be the desired interpreter name, you can specify the interpreter name here instead.

### Script generation

Generating a basic script looks virtually identical to using zc.recipe.egg.

(Note that the find-links and index values are typically not needed; they are included to help make this document run as a test successfully.)

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = demo
...
... [demo]
... recipe = z3c.recipe.scripts
... eggs = demo<0.3
... index = %(server)s/index

>>> print system(buildout),
Installing demo.
Getting distribution for 'demo<0.3'.
Got demo 0.2.
Getting distribution for 'demoneeded'.
Got demoneeded 1.2c1.
Generated script '/sample-buildout/bin/demo'.

>>> print system(join(sample_buildout, 'bin', 'demo')),
2 2


### Interpreter generation

As with zc.recipe.egg, you can generate an interpreter with the default script recipe shown above by supplying the “interpreter” option. This example will create both an entry point script and an interpreter.

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = demo
...
... [demo]
... recipe = z3c.recipe.scripts
... eggs = demo<0.3
... index = %(server)s/index
... interpreter = py

>>> print system(buildout),
Uninstalling demo.
Installing demo.
Generated script '/sample-buildout/bin/demo'.
Generated interpreter '/sample-buildout/bin/py'.


You can also generate an interpreter alone with the interpreter recipe.

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = py
...
... [py]
... recipe = z3c.recipe.scripts:interpreter
... eggs = demo<0.3
... index = %(server)s/index

>>> print system(buildout),
Uninstalling demo.
Installing py.
Generated interpreter '/sample-buildout/bin/py'.


In both cases, the bin/py script works by restarting Python after specifying a special path in PYTHONPATH. This example shows the UNIX version; the Windows version actually uses subprocess instead.

>>> cat(sample_buildout, 'bin', 'py') # doctest: +NORMALIZE_WHITESPACE
#!/usr/bin/python2.4 -S
<BLANKLINE>
import os
import sys
<BLANKLINE>
argv = [sys.executable] + sys.argv[1:]
environ = os.environ.copy()
path = '/sample-buildout/parts/py'
if environ.get('PYTHONPATH'):
path = os.pathsep.join([path, environ['PYTHONPATH']])
environ['PYTHONPATH'] = path
os.execve(sys.executable, argv, environ)


The path is a directory that contains two files: our own site.py and sitecustomize.py. The site.py is modified from the underlying Python’s site.py, and is responsible for setting up our paths. The sitecustomize.py is responsible for running the initialization code provided.

>>> ls(sample_buildout, 'parts', 'py')
-  site.py
-  sitecustomize.py


Here’s an example of using the generated interpreter.

>>> print system(join(sample_buildout, 'bin', 'py') +
...              ' -c "import sys, pprint; pprint.pprint(sys.path[-2:])"')
['/sample-buildout/eggs/demo-0.2-pyN.N.egg',
'/sample-buildout/eggs/demoneeded-1.2c1-pyN.N.egg']
<BLANKLINE>


### Including site-packages and sitecustomize

As introduced above, this recipe supports including site packages. This has some advantages and some serious dangers.

A typical reason to include site-packages is that it is easier to install one or more dependencies in your Python than it is with buildout. Some packages, such as lxml or Python PostgreSQL integration, have dependencies that can be much easier to build and/or install using other mechanisms, such as your operating system’s package manager. By installing some core packages into your Python’s site-packages, this can significantly simplify some application installations.

However, doing this has a significant danger. One of the primary goals of buildout is to provide repeatability. Some packages (one of the better known Python openid packages, for instance) change their behavior depending on what packages are available. If Python curl bindings are available, these may be preferred by the library. If a certain XML package is installed, it may be preferred by the library. These hidden choices may cause small or large behavior differences. The fact that they can be rarely encountered can actually make it worse: you forget that this might be a problem, and debugging the differences can be difficult. If you allow site-packages to be included in your buildout, and the Python you use is not managed precisely by your application (for instance, it is a system Python), you open yourself up to these possibilities. Don’t be unaware of the dangers.

To show off these features, we need to use buildout with a Python executable with some extra paths to show include-site-packages; and one guaranteed to have a sitecustomize module to show exec-sitecustomize. We’ll make one using a test fixture called make_py. The os.environ change below will go into the sitecustomize, and the site_packages_path will be in the Python’s path.

>>> py_path, site_packages_path = make_py(initialization='''\
... import os
... os.environ['zc.buildout'] = 'foo bar baz shazam'
... ''')
>>> print site_packages_path
/executable_buildout/site-packages


Now let’s take a look at include-site-packages. The default is false, so we will set it to true.

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = py
... executable = %(py_path)s
...
... [py]
... recipe = z3c.recipe.scripts:interpreter
... include-site-packages = true
... eggs = demo<0.3
... index = %(server)s/index

>>> print system(buildout),
Uninstalling py.
Installing py.
Generated interpreter '/sample-buildout/bin/py'.


Now executable_buildout/site-packages is included in sys.path.

>>> print system(join(sample_buildout, 'bin', 'py') +
...              ''' -c "import sys, pprint; pprint.pprint(sys.path)"''')
... # doctest: +ELLIPSIS
['',
'/sample-buildout/parts/py',
...,
'/sample-buildout/eggs/demo-0.2-pyN.N.egg',
'/sample-buildout/eggs/demoneeded-1.2c1-pyN.N.egg',
'/executable_buildout/eggs/setuptools-X-pyN.N.egg',
'/executable_buildout/site-packages']
<BLANKLINE>


As described above, the allowed-eggs-from-site-packages option lets us control what site-packages eggs zc.buildout will allow to fulfill dependencies. The behavior was described above with an example (and the implementation is tested elsewhere), so we’ll only look at some simple and common use cases here.

Sometimes you may want to allow site-packages to be available but you don’t want your package to depend on it using setup.py. For instance, perhaps you are writing an application, and you want to depend on your system’s packaging of the PostgreSQL code, but the system Python does not use eggs to package it, so you need to manage the two separately. In this case, you might not want to use any eggs from site-packages, but you want it available. In this case, you can use allowed-eggs-from-site-packages with an empty value to keep any egg from being used from site-packages.

Here’s an example. Let’s say we have a Python with demo and demoneeded installed as eggs in the system Python. Normally, they will be used to fulfill dependencies, because allowed-eggs-from-site-packages defaults to the value “*” (allow any package). (We use an empty find-links value to say that buildout may not look elsewhere for the package. We use a different eggs-directory for isolation, so that eggs obtained other parts of the document do not affect this example.)

>>> from zc.buildout.tests import create_sample_sys_install
>>> create_sample_sys_install(site_packages_path)
>>> import zc.buildout.easy_install
>>> zc.buildout.easy_install.clear_index_cache()

>>> write('buildout.cfg',
... '''
... [buildout]
... parts = eggs
... eggs-directory = tmpeggs
...
... [primed_python]
... executable = %(py_path)s
...
... [eggs]
... recipe = z3c.recipe.scripts
... include-site-packages = true
... python = primed_python
... eggs = demoneeded
... ''' % globals())

>>> print system(buildout),
Creating directory '/sample-buildout/tmpeggs'.
Uninstalling py.
Installing eggs.


That succeeds fine, getting demoneeded from the Python site-packages.

However, when allowed-eggs-from-site-packages is an empty value, demoneeded is not allowed to come from site-packages, and the buildout fails.

>>> zc.buildout.easy_install.clear_index_cache()
>>> rmdir('tmpeggs')
>>> write('buildout.cfg',
... '''
... [buildout]
... parts = eggs
... eggs-directory = tmpeggs
...
... [primed_python]
... executable = %(py_path)s
...
... [eggs]
... recipe = z3c.recipe.scripts
... include-site-packages = true
... python = primed_python
... allowed-eggs-from-site-packages =
... eggs = demoneeded
... ''' % globals())
>>> print system(buildout),
Creating directory '/sample-buildout/tmpeggs'.
Uninstalling eggs.
Installing eggs.
Couldn't find index page for 'demoneeded' (maybe misspelled?)
Getting distribution for 'demoneeded'.
While:
Installing eggs.
Getting distribution for 'demoneeded'.
Error: Couldn't find a distribution for 'demoneeded'.


The include-sitepackages and allowed-eggs-from-site-packages options both can be obtained from the buildout section if they are not set locally.

Remember that you can provide multiple lines to the allowed-eggs-from-site-packages option, each specifying a whitelist of allowed packages. Globs (* and ?) are allowed.

Next we will use the exec-sitecustomize option. It simply copies Python’s underlying sitecustomize module, if it exists, to the local version. The os.environ change shown above in the make_py call will go into the sitecustomize.

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = py
... executable = %(py_path)s
...
... [py]
... recipe = z3c.recipe.scripts:interpreter
... exec-sitecustomize = true
... eggs = demo<0.3
... index = %(server)s/index

>>> print system(buildout),
Installing py.
Generated interpreter '/sample-buildout/bin/py'.

>>> cat(sample_buildout, 'parts', 'py', 'sitecustomize.py')
... # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
<BLANKLINE>
# The following is from
# /executable_buildout/parts/py/sitecustomize.py
...
import os
os.environ['zc.buildout'] = 'foo bar baz shazam'

>>> print system(join(sample_buildout, 'bin', 'py') +
...              ''' -c "import os; print os.environ['zc.buildout']"''')
foo bar baz shazam
<BLANKLINE>


It also will be honored in the buildout section if it is not set locally.

### Options

We’ll focus now on the remaining options that are different than zc.recipe.egg.

Let’s look at the extends option first.

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = demo python
...
... [demo]
... recipe = z3c.recipe.scripts
... eggs = demo<0.3
... index = %(server)s/index
... initialization =
...     import os
...     os.environ['zc.buildout'] = 'sha boo bop bazoodle'
...
... [python]
... recipe = z3c.recipe.scripts:interpreter
... extends = demo
... initialization =
...     import os
...     os.environ['zc.buildout'] = 'foo bar baz shazam'


That makes it easier to specify some initialization for the interpreter that is different than a script, while duplicating other configuration.

Now let’s put it in action.

>>> print system(buildout),
Uninstalling py.
Installing demo.
Generated script '/sample-buildout/bin/demo'.
Installing python.
Generated interpreter '/sample-buildout/bin/python'.

>>> print system(join(sample_buildout, 'bin', 'python') +
...              ' -c "import sys, pprint; pprint.pprint(sys.path[-2:])"')
['/sample-buildout/eggs/demo-0.2-pyN.N.egg',
'/sample-buildout/eggs/demoneeded-1.2c1-pyN.N.egg']
<BLANKLINE>
>>> print system(join(sample_buildout, 'bin', 'python') +
...              ''' -c "import os; print os.environ['zc.buildout']"'''),
foo bar baz shazam


Note that the parts/py directory has been cleaned up, and parts/python has been created.

>>> ls(sample_buildout, 'parts')
d  buildout
d  demo
d  python


If you want to have initialization that only affects scripts, not the interpreter, you can use script-initialization. Here’s a demonstration.

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = demo
...
... [demo]
... recipe = z3c.recipe.scripts
... eggs = demo<0.3
... index = %(server)s/index
... interpreter = py
... script-initialization =
...     print "Hi from the script"

>>> print system(buildout),
Uninstalling python.
Uninstalling demo.
Installing demo.
Generated script '/sample-buildout/bin/demo'.
Generated interpreter '/sample-buildout/bin/py'.

>>> print system(join(sample_buildout, 'bin', 'py') +
...              ''' -c "print 'Hi from the interpreter'"'''),
Hi from the interpreter

>>> print system(join(sample_buildout, 'bin', 'demo')),
Hi from the script
2 2


The last new option is name. This simply changes the name of the interpreter, so that you are not forced to use the name of the section.

>>> write(sample_buildout, 'buildout.cfg',
... """
... [buildout]
... parts = interpreter
...
... [interpreter]
... name = python2
... recipe = z3c.recipe.scripts:interpreter
... eggs = demo<0.3
... index = %(server)s/index

>>> print system(buildout),
Uninstalling demo.
Installing interpreter.
Generated interpreter '/sample-buildout/bin/python2'.

>>> print system(join(sample_buildout, 'bin', 'python2') +
...              ' -c "print 42"')
42
<BLANKLINE>


The other options all identical to zc.recipe.egg.

## Project details

Uploaded source