zope.kgs 1.2.0

Known-Good-Set (KGS) Support

===============================
Zope 3 Controlled Package Index
===============================

This package has been developed to support the maintenance of a stable set of
Zope project distributions. It manages the controlled packages configuration
file and supports the generation of buildout configuration files that can be
used by developers.

Another use of this package is to use it for testing new distributions against
the index. Here is the workflow for testing a new package against stable set:

1. Install the correct version of this package.

(a) Download the version of this package that manages the stable set that
you are interested in. Currently only the trunk exists, which manages
the Zope 3.4 release::

$ svn co svn://svn.zope.org/repos/main/zope.release/trunk zope3.4
$ cd zope3.4

(b) Bootstrap the checkout::

$ python ./bootstrap.py

(c) Run buildout to create the scripts::

$ ./bin/buildout

(d) Run the ``buildout.cfg`` generation script to build a configuration
file that can be used for testing:

$ ./bin/generate-buildout

2. From the generated configuration file, you can now build a testing
environment.

(a) Enter the test directory and create a buildout:

$ cd test
$ python ../bootstrap.py
$ ./bin/buildout

(b) Run all the tests to verify that all tests are initially passing:

$ ./bin/test -vpc1

3. Modify the ``buildout.cfg`` to look for your the new distribution to be
tested:

(a) Comment out the "index" option. This needs to be done, so that the new
package is going to be picked up.

(b) Change the version number of the package of interest in the "versions"
section.

Alternative:

(a) Check out the new distribution from SVN.

(b) Add a "develop path/to/my/package" line in the "buildout" section of
``buildout.cfg``.

4. Run the tests, making sure that they all pass.

5. Modify ``controlled-packages.cfg`` by adding the new version of the package
to the package's version list.

6. Generate all files again and upload them:

$ cd ..
$ ./bin/generate-buildout
$ ./bin/generate-versions
$ ./bin/upload

Once the files are uploaded, a crontab-job, running every minute, will
detect the changes in ``controlled-pages.cfg`` and will generate the new
controlled package pages.

Note: I think the process is still a tiny bit too long. I probably write a
script that makes testing a new version of a package easier, but let's see
whether this process is workable first.


===============
Known Good Sets
===============

This package provides a set of scripts and tools to manage Good-Known-Sets, or
short KGSs. A KGS is a set of package distributions that are known to work
well together. You can verify this, for example, by running all the tests of
all the packages at once.

Let me show you how a typical controlled packages configuration file looks
like:

>>> import tempfile
>>> cfgFile = tempfile.mktemp('-cp.cfg')
>>> open(cfgFile, 'w').write('''\
... [DEFAULT]
... tested = true
...
... [KGS]
... name = zope-dev
... version = 1.2.0
... date = 2009-01-01
... changelog = CHANGES.txt
... announcement = ANNOUNCEMENT.txt
... files =
... zope-dev-1.2.0.tgz
... zope-dev-1.2.0.zip
... zope-dev-1.2.0.exe
...
... [packageA]
... versions = 1.0.0
... 1.0.1
...
... [packageB]
... versions = 1.2.3
... test-extras = test
...
... [packageC]
... # Do not test this package.
... tested = false
... versions = 4.3.1
... ''')

As you can see, this file uses an INI-style format. The "DEFAULT" section is
special, as it will insert the specified options into all other sections as
default. The "KGS" section specifies some global information about the KGS,
such as the name of the KGS. Since this section references several external
files, we should quickly create those.

>>> import os
>>> dir = os.path.dirname(cfgFile)

>>> open(os.path.join(dir, 'CHANGES.txt'), 'w').write('''\
... =======
... Changes
... =======
...
... packageA
... ========
...
... Version 1.0.0
... -------------
...
... * Initial Release
... ''')

>>> open(os.path.join(dir, 'ANNOUNCEMENT.txt'), 'w').write('''\
... =======================
... zope-dev 1.2.0 Released
... =======================
...
... The announcement text!
... ''')

>>> open(os.path.join(dir, 'zope-dev-1.2.0.tgz'), 'w').write('tgz')
>>> open(os.path.join(dir, 'zope-dev-1.2.0.exe'), 'w').write('exe')

All other sections refer to package names. Currently each package section
supports two options. The "versions" option lists all versions that are known
to work in the KGS. Those versions should *always* only be bug fixes to the
first listed version. The second option, "tested", specifies whether the
package should be part of the KGS test suite. By default, we want all packages
to be tested, but some packages require very specific test setups that cannot
be easily reproduced _[1], so we turn off those tests.

You can also stack controlled package configurations on top of each
other. Base configurations can be specified using the `extends` option:

>>> import tempfile
>>> cfgFile2 = tempfile.mktemp('-cp.cfg')
>>> open(cfgFile2, 'w').write('''\
... [DEFAULT]
... tested = true
...
... [KGS]
... name = grok-dev
... version = 0.1.0
... extends = %s
...
... [packageA]
... versions = 1.0.2
...
... [packageD]
... versions = 2.2.3
... 2.2.4
... ''' %cfgFile)

As you can see, you can completely override another package's version
specification as well.

Generating the configuration file and managing it is actually the hard
part. Let's now see what we can do with it.

.. [1]: This is usually due to bugs in setuptools or buildout, such as PYC
files not containing the correct reference to their PY file.


Generate Versions
-----------------

One of the easiest scripts, is the version generation. This script will
generate a "versions" section that is compatible with buildout.

>>> versionsFile = tempfile.mktemp('-versions.cfg')

>>> from zope.kgs import version
>>> version.main((cfgFile, versionsFile))

>>> print open(versionsFile, 'r').read()
[versions]
packageA = 1.0.1
packageB = 1.2.3
packageC = 4.3.1

Let's now ensure that the versions also work for the extended configuration:

>>> versionsFile2 = tempfile.mktemp('-versions.cfg')

>>> version.main((cfgFile2, versionsFile2))

>>> print open(versionsFile2, 'r').read()
[versions]
packageA = 1.0.2
packageB = 1.2.3
packageC = 4.3.1
packageD = 2.2.4


Generate Buildout
-----------------

In order to be able to test the KGS, you can also generate a full buildout
file that will create and install a testrunner over all packages for you:

>>> buildoutFile = tempfile.mktemp('-buildout.cfg')

>>> from zope.kgs import buildout
>>> buildout.main((cfgFile, buildoutFile))

>>> print open(buildoutFile, 'r').read()
[buildout]
parts = test
versions = versions
<BLANKLINE>
[test]
recipe = zc.recipe.testrunner
eggs = packageA
packageB [test]
<BLANKLINE>
[versions]
packageA = 1.0.1
packageB = 1.2.3
packageC = 4.3.1
<BLANKLINE>

Let's make sure that the buildout generation also honors the extensions:

>>> buildoutFile2 = tempfile.mktemp('-buildout.cfg')

>>> buildout.main((cfgFile2, buildoutFile2))

>>> print open(buildoutFile2, 'r').read()
[buildout]
parts = test
versions = versions
<BLANKLINE>
[test]
recipe = zc.recipe.testrunner
eggs = packageA
packageB [test]
packageD
<BLANKLINE>
[versions]
packageA = 1.0.2
packageB = 1.2.3
packageC = 4.3.1
packageD = 2.2.4
<BLANKLINE>


Flat Links Pages
----------------

We can also create a flat links page that can be used in the
`dependency_links` argument in your `setup.py` file. Since this module
accesses the original PyPI to ask for the download locations and filenames, we
have to create a controlled packages configuration file that contains real
packages with real version numbers:

>>> cfgFileReal = tempfile.mktemp('-cp.cfg')
>>> open(cfgFileReal, 'w').write('''\
... [DEFAULT]
... tested = true
...
... [KGS]
... name = zope-dev
... version = 3.4.0b2
...
... [PIL]
... versions = 1.1.6
...
... [zope.component]
... versions = 3.4.0
...
... [zope.interface]
... versions = 3.4.0
... 3.4.1
...
... [z3c.formdemo]
... versions = 1.1.0
... ''')

Let's now create the links page:

>>> linksFile = tempfile.mktemp('-links.html')

>>> from zope.kgs import link
>>> link.main((cfgFileReal, linksFile))

>>> print open(linksFile, 'r').read()
<html>
<head>
<title>Links for the "zope-dev" KGS (version 3.4.0b2)</title>
</head>
<body>
<h1>Links for the "zope-dev" KGS (version 3.4.0b2)</h1>
<a href="http://pypi.python.org/packages/2.4/z/z3c.formdemo/z3c.formdemo-1.1.0-py2.4.egg#md5=9d605bd559ea33ac57ce11f5c80fa3d3">z3c.formdemo-1.1.0-py2.4.egg</a><br/>
<a href="http://pypi.python.org/packages/source/z/z3c.formdemo/z3c.formdemo-1.1.0.tar.gz#md5=f224a49cea737112284f74b859e3eed0">z3c.formdemo-1.1.0.tar.gz</a><br/>
<a href="http://pypi.python.org/packages/2.4/z/zope.component/zope.component-3.4.0-py2.4.egg#md5=c0763e94912e4a8ac1e321a068c916ba">zope.component-3.4.0-py2.4.egg</a><br/>
<a href="http://pypi.python.org/packages/source/z/zope.component/zope.component-3.4.0.tar.gz#md5=94afb57dfe605d7235ff562d1eaa3bed">zope.component-3.4.0.tar.gz</a><br/>
<a href="http://pypi.python.org/packages/source/z/zope.interface/zope.interface-3.4.0.tar.gz#md5=0be9fd80b7bb6bee520e56eba7d29c90">zope.interface-3.4.0.tar.gz</a><br/>
<a href="http://pypi.python.org/packages/2.4/z/zope.interface/zope.interface-3.4.0-py2.4-win32.egg#md5=3fa5e992271375eac597622d8e2fd5ec">zope.interface-3.4.0-py2.4-win32.egg</a><br/>
<a href="http://pypi.python.org/packages/source/z/zope.interface/zope.interface-3.4.1.tar.gz#md5=b085f4a774adab688e037ad32fbbf08e">zope.interface-3.4.1.tar.gz</a><br/>
</body>
</html>


PPIX Support
------------

You can also use the KGS to limit the available packages in a package index
generated ``zc.mirrorcheeseshopslashsimple``. This script also uses PyPI to
look up distribution file, so wave to use the real configuration file again.

Let's create the pages:

>>> indexDir = tempfile.mkdtemp('-ppix')

>>> from zope.kgs import ppix
>>> ppix.main((cfgFileReal, indexDir))

The index contains one directory per package. So let's have a look:

>>> import os
>>> sorted(os.listdir(indexDir))
['PIL', 'z3c.formdemo', 'zope.component', 'zope.interface']

Each directory contains a single "index.html" file with the download links:

>>> pkgDir = os.path.join(indexDir, 'zope.component')
>>> sorted(os.listdir(pkgDir))
['index.html']

>>> pkgIndex = os.path.join(pkgDir, 'index.html')
>>> print open(pkgIndex, 'r').read()
<html>
<head>
<title>Links for "zope.component"</title>
</head>
<body>
<h1>Links for "zope.component"</h1>
<a href="http://pypi.python.org/packages/2.4/z/zope.component/zope.component-3.4.0-py2.4.egg#md5=c0763e94912e4a8ac1e321a068c916ba">zope.component-3.4.0-py2.4.egg</a><br/>
<a href="http://pypi.python.org/packages/source/z/zope.component/zope.component-3.4.0.tar.gz#md5=94afb57dfe605d7235ff562d1eaa3bed">zope.component-3.4.0.tar.gz</a><br/>
</body>
</html>

PIL is an interesting case, because it does not upload its distribution files
yet, at least not for version 1.1.6:

>>> pkgIndex = os.path.join(indexDir, 'PIL', 'index.html')
>>> print open(pkgIndex, 'r').read()
<html><head><title>Links for PIL</title></head><body><h1>Links for PIL</h1><a href='http://www.pythonware.com/products/pil' rel="homepage">1.1.5 home_page</a><br/>
<a href='http://effbot.org/zone/pil-changes-115.htm' rel="download">1.1.5 download_url</a><br/>
<a href='http://www.pythonware.com/products/pil' rel="homepage">1.1.5a2 home_page</a><br/>
<a href='http://effbot.org/zone/pil-changes-115.htm' rel="download">1.1.5a2 download_url</a><br/>
<a href='http://www.pythonware.com/products/pil' rel="homepage">1.1.5a1 home_page</a><br/>
<a href='http://effbot.org/zone/pil-changes-115.htm' rel="download">1.1.5a1 download_url</a><br/>
<a href='http://www.pythonware.com/products/pil/' rel="homepage">1.1.4 home_page</a><br/>
<a href='http://www.pythonware.com/products/pil/' rel="homepage">1.1.3 home_page</a><br/>
<a href='http://www.pythonware.com/downloads/Imaging-1.1.3.tar.gz' rel="download">1.1.3 download_url</a><br/>
<a href='http://www.pythonware.com/products/pil' rel="homepage">1.1.6 home_page</a><br/>
<a href='http://effbot.org/downloads/#Imaging' rel="download">1.1.6 download_url</a><br/>
</body></html>

Optionally, you can also specify the `-i` option to generate an overview:

>>> ppix.main(('-i', cfgFileReal, indexDir))

>>> sorted(os.listdir(indexDir))
['PIL', 'index.html', 'z3c.formdemo', 'zope.component', 'zope.interface']

Let's now look at the file:

>>> indexPage = os.path.join(indexDir, 'index.html')
>>> print open(indexPage, 'r').read()
<html>
<head>
<title>Simple Index for the "zope-dev" KGS (version 3.4.0b2)</title>
</head>
<body>
<h1>Simple Index for the "zope-dev" KGS (version 3.4.0b2)</h1>
<a href="PIL">PIL</a><br/>
<a href="z3c.formdemo">z3c.formdemo</a><br/>
<a href="zope.component">zope.component</a><br/>
<a href="zope.interface">zope.interface</a><br/>
</body>
</html>

Allowing exisitng package pages to be overwritten and making the main index
page an optional feature makes it possible to use this script for two use
cases: (1) Merge the constraints into a PPIX index created by
``zc.mirrorcheeseshopslashsimple``, and (2) create a standalone index which
only provides the packages of the KGS.


Getting the Latest Versions
---------------------------

When updating the KGS, it is often useful to know for which packages have new
releases.

>>> from zope.kgs import latest
>>> latest.main((cfgFileReal,))
z3c.formdemo: 1.1.1, 1.1.2, 1.2.0, 1.3.0, 1.3.0b1, 1.4.0, ...

However, it is often desired only to show new minor versions; in this case, we
can pass an option to exclude all versions that have a different major
version:

>>> latest.main(('-m', cfgFileReal))
z3c.formdemo: 1.1.1, 1.1.2

Sometimes you're only interested in changes that apply to a single package,
and you won't want to wait for the script to query all of the others

>>> latest.main(('-m', cfgFileReal, 'zope.app.server'))

>>> latest.main(('-m', cfgFileReal, 'z3c.formdemo'))
z3c.formdemo: 1.1.1, 1.1.2


Extracting Change Information
-----------------------------

When releasing a version of the KGS, it is desirable to produce a list of
changes since the last release. Changes are commonly compared to an older
version.

>>> cfgFileRealOrig = tempfile.mktemp('-cp.cfg')
>>> open(cfgFileRealOrig, 'w').write('''\
... [DEFAULT]
... tested = true
...
... [KGS]
... name = zope-dev
... version = 3.4.0b1
...
... [PIL]
... versions = 1.1.6
...
... [zope.component]
... versions = 3.4.0
...
... [zope.interface]
... versions = 3.4.0
... ''')

Let's now produce the changes:

>>> from zope.kgs import change
>>> change.main((cfgFileReal, cfgFileRealOrig))
Processing ('PIL', '1.1.6')
Processing ('z3c.formdemo', '1.1.0')
Processing ('zope.component', '3.4.0')
Processing ('zope.interface', '3.4.1')
===
PIL
===
<BLANKLINE>
No changes or information not found.
<BLANKLINE>
============
z3c.formdemo
============
<BLANKLINE>
1.1.0 (unknown)
---------------
<BLANKLINE>
- Feature: New "SQL Message" demo shows how ``z3c.form`` can be used with
non-object data. Specificically, this small application demonstrates using a
Gadfly database using pure SQL calls without any ORM.
<BLANKLINE>
- Feature: New "Address Book" demo that demonstrates more complex use cases,
such as subforms, composite widgets, and mappings/lists
<BLANKLINE>
<BLANKLINE>
==============
zope.component
==============
<BLANKLINE>
3.4.0 (2007-09-29)
------------------
<BLANKLINE>
No further changes since 3.4.0a1.
<BLANKLINE>
<BLANKLINE>
==============
zope.interface
==============
<BLANKLINE>
3.4.1 (unknown)
---------------
<BLANKLINE>
Fixed a setup bug that prevented installation from source on systems
without setuptools.
<BLANKLINE>
3.4.0 (unknown)
---------------
<BLANKLINE>
Final release for 3.4.0.
<BLANKLINE>
<BLANKLINE>


You can also create the changes without an original file, in which case only
the versions listed in the current KGS are considered.

>>> change.main((cfgFileReal,))
Processing ('PIL', '1.1.6')
Processing ('z3c.formdemo', '1.1.0')
Processing ('zope.component', '3.4.0')
Processing ('zope.interface', '3.4.1')
===
PIL
===
<BLANKLINE>
No changes or information not found.
<BLANKLINE>
============
z3c.formdemo
============
<BLANKLINE>
1.1.0 (unknown)
---------------
<BLANKLINE>
- Feature: New "SQL Message" demo shows how ``z3c.form`` can be used with
non-object data. Specificically, this small application demonstrates using a
Gadfly database using pure SQL calls without any ORM.
<BLANKLINE>
- Feature: New "Address Book" demo that demonstrates more complex use cases,
such as subforms, composite widgets, and mappings/lists
<BLANKLINE>
<BLANKLINE>
==============
zope.component
==============
<BLANKLINE>
3.4.0 (2007-09-29)
------------------
<BLANKLINE>
No further changes since 3.4.0a1.
<BLANKLINE>
<BLANKLINE>
==============
zope.interface
==============
<BLANKLINE>
3.4.1 (unknown)
---------------
<BLANKLINE>
Fixed a setup bug that prevented installation from source on systems
without setuptools.
<BLANKLINE>
3.4.0 (unknown)
---------------
<BLANKLINE>
Final release for 3.4.0.
<BLANKLINE>
<BLANKLINE>


The Site Generator
------------------

The easiest way to publish the KGS is via a directory published by a Web
server. Whenever a new `controlled-packages.cfg` file is uploaded, a script is
run that generates all the files. I usually set up a crontab job to do
this. The site generator script acts upon a directory, in which it assumes a
`controlled-packages.cfg` file was placed:

>>> siteDir = tempfile.mkdtemp()
>>> cfgFileSite = os.path.join(siteDir, 'controlled-packages.cfg')

>>> import shutil
>>> shutil.copy(cfgFileReal, cfgFileSite)

>>> from zope.kgs import site
>>> site.main(['-s', siteDir])

Let's have a look at the generated files:

>>> from pprint import pprint
>>> pprint(sorted(os.listdir(siteDir)))
['3.4.0b2', 'index.html', 'intro.html', 'resources']

>>> sorted(os.listdir(os.path.join(siteDir, '3.4.0b2')))
['ANNOUNCEMENT.html', 'CHANGES.html',
'buildout.cfg', 'controlled-packages.cfg', 'index', 'index.html',
'links.html', 'minimal', 'versions.cfg']

>>> sorted(os.listdir(os.path.join(siteDir, '3.4.0b2', 'minimal')))
['PIL', 'index.html', 'z3c.formdemo', 'zope.component', 'zope.interface']

If you try to generate the site again without adding the controlled packages
config file to the site directory again, it will simply return:

>>> site.main(['-s', siteDir])


Basic Parser API
----------------

The ``kgs.py`` module provides a simple class that parses the KGS
configuration file and provides all data in an object-oriented manner.

>>> from zope.kgs import kgs

The class is simply instnatiated using the path to the config file:

>>> myKGS = kgs.KGS(cfgFile)
>>> myKGS
<KGS 'zope-dev'>

The name, version and date of the KGS is available via:

>>> myKGS.name
'zope-dev'
>>> myKGS.version
'1.2.0'
>>> myKGS.date
datetime.date(2009, 1, 1)

When the changelog and/or announcement files are available, the KGS references
the absolute path:

>>> myKGS.changelog
'.../CHANGES.txt'
>>> myKGS.announcement
'.../ANNOUNCEMENT.txt'

The same is true for other release-related files:

>>> myKGS.files
('.../zope-dev-1.2.0.tgz',
'.../zope-dev-1.2.0.exe')

The packages are available under `packages`:

>>> myKGS.packages
[<Package 'packageA'>, <Package 'packageB'>, <Package 'packageC'>]

Each package is also an object:

>>> pkgA = myKGS.packages[0]
>>> pkgA
<Package 'packageA'>

>>> pkgA.name
'packageA'
>>> pkgA.versions
['1.0.0', '1.0.1']
>>> pkgA.tested
True

As we have seen in the scripts above, the KGS class also supports the
`entends` option. Thus, let's load the KGS for the config file 2:

>>> myKGS2 = kgs.KGS(cfgFile2)
>>> myKGS2
<KGS 'grok-dev'>

>>> myKGS2.name
'grok-dev'

>>> myKGS2.packages
[<Package 'packageA'>,
<Package 'packageB'>,
<Package 'packageC'>,
<Package 'packageD'>]


=======
CHANGES
=======

1.2.0 (2009-07-24)
------------------

- Add ability to specify the extra-includes for running tests.

1.1.0 (2009-02-01)
------------------

- Added '--no-links', '--no-index', and '--no-minimal-index' options to the
site generation sctipt to make it faster.

- Generating list of latest versions does not fail anymore, if downloading and
parsing of the index page fails.

- Update links to PyPI web sites.

1.0.1 (2009-01-29)
------------------

- Fix documentation in all scripts, fixing missing imports and incorrect
wording.

- The package should depend on `python-dateutl` and not `datetutil`, since the
latter is not available in PyPI anymore.

1.0.0 (2009-01-29)
------------------

- Initial version as ``zope.kgs``.

* A script that extracts the relevant part of the changelog of each package
in the KGS.

* A script that lists all versions of a package released after the latest
version listed in the KGS.

* A script that manages the generation of the entire KGS site.

+ Generates generic and version-specific pages.

+ Page generation is template-based for easy customization.

* Generate `links.html` file which lists all controlled packages files.

* Features copied from ``zope.release``:

+ Parser for KGS configuration files.

+ Generate `versions.cfg` and `buildout.cfg` script.

* Features copied from ``zc.mirrorcheeseshopslashsimple``:

+ Generate new index pages for the controlled packages.