collective.buildbot 0.2.0

A set of zc.buildout recipes and package to easy install buildbot

Contents

• Change history

  • 0.2.0 (2008-05-21)

  • 0.1.1 (2008-05-02)

  • 0.1.0 (xxxx-xx-xx)

• Package description

• The build master recipe

  • Supported options

  • Example usage

• The build slave recipe

  • Supported options

  • Example usage

• The project recipe

  • Supported options

  • Example usage

• The projects recipe

• The poller recipe

  • Supported options

  • Example usage

• The pollers recipe

  • Supported options

• Putting it all together

• Contributors

• Download

Change history

0.2.0 (2008-05-21)

• Added irc options so you can attach an irc bot to the master buildbot [mustapha]

• Allow public_html customization [gawel]

• Added custom about page to link to collective.buildout [gawel]

• Added support for Git repositories [dokai]

• Refactored the repository URL configuration. For Subversion, you should use only the repository option to specify a full URL to the desired branch (trunk, tag or branch) that will be built. For Git in addition to setting the repository option you can use the branch option to specify a specific branch to build. By default the master branch will be used for Git repositories. [dokai]

• Cleaned up a lot of redundant imports. [dokai]

• Updated the documentation and examples. [dokai]

• Deprecated the collective.buildbot:projects recipe [dokai]

• Fixed problem with missing twistd.log files on first run [dokai]

• Fixed bug that prevented the master from starting if there weren’t any SVN pollers configured. [dokai]

• Added new options periodic-scheduler and cron-scheduler to set up passive schedulers for projects. [dokai]

0.1.1 (2008-05-02)

• bugs fixes [gawel]

0.1.0 (xxxx-xx-xx)

• Created recipe with ZopeSkel [Gael Pasgrimaud].

Package description

This package provides a set of zc.buildout recipes that make it easy to configure a buildbot setup (build master, build slaves and projects) and a scripts to run the buildbot master and slaves. The recipes produce INI-style declarative configuration files based on the buildout configuration. These configuration files are in turn read by the buildbot runner script to initialize the buildbot environment.

The available recipes are:

• collective.buildbot:master – Produces a configuration file for the build master process.

• collective.buildbot:slave – Produces a configuration file for the build slave process.

• collective.buildbot:project – Produces a configuration for a project build on a selected slave.

• collective.buildbot:poller – Produces configuration for code repository pollers.

• collective.buildbot:pollers – Produces configuration for code repository pollers by allowing multiple pollers to be configured in a single buildout section.

It is possible to use all the recipes in a single buildout and have both the master and slave(s) on the same machine. However, in most cases you will have one buildout for the build master that uses the collective.buildbot:master and collective.buildbot:project to set up the build processes and then separate buildouts on each of the slave machines that use the collective.buildbot:slave recipe.

The build master recipe

The collective.buildbot:master recipe produces a configuration file that sets up the build master process. Once the build master is configured you can run in by executing the controller script under the buildout’s bin directory. The controller script will be named after the section name, so if you had a [buildmaster] section in your buildout.cfg you would get a bin/buildmaster script.

Supported options

The recipe supports the following options:

port

The port the build master process listens for connections from build slaves. The slaves must be configured to use the corresponding port in the sections using the collective.buildbot:slave recipe.

wport

The web port for serving the buildbot web interface.

project-name

Project name. Displayed on the web interface.

project-url

Project url, used on the web interface.

url

buildbot url.

build-slaves

A sequence of build slave configurations. Each build slave must be defined on a separate line containing the name of the build slave and the password for the build slave separated by white space.

allow-force (optional)

If true allows users to force builds using the web interface. Defaults to false.

public-html (optional)

Location of a directory that contains custom resources (HTML, CSS, images) for the web interface.

Additionally you can use the following options if you need to run an IRC bot:

irc-host

The irc host to connect to. ie: irc.freenode.net

irc-channels

A list of channels to join. #plone

irc-nickname

The bot nickname. Defaults to buildbot

irc-password

The password used to identify the bot. Defaults to an empty string

Example usage

We’ll start by creating a buildout that uses the recipe:

>>> write('buildout.cfg',
... """
... [buildout]
... parts = buildmaster
...
... [buildmaster]
... recipe = collective.buildbot:master
... port = 8080
... wport = 8082
... project-name = The project
... project-url = http://example.com
... url = http://example.com/buildbot
... slaves =
...     slave1 password
...     slave2 password
... """)

Running the buildout gives us:

>>> print system(buildout)
Installing buildmaster...
New python executable in /sample-buildout/parts/buildmaster/bin/python
Installing setuptools.............done.
Generated script '/sample-buildout/parts/buildmaster/buildbot.tac'.
Generated config '/sample-buildout/parts/buildmaster/buildbot.cfg'.
Generated script '/sample-buildout/bin/buildmaster'.

As shown above, the buildout generated the required configuration files and the runner script under bin. You can control build master process by running:

$ ./bin/buildmaster [start | stop | restart]

The Twisted .tac file that is used to launch the buildbot process:

>>> cat(join('parts', 'buildmaster', 'buildbot.tac'))
from twisted.application import service
from buildbot.master import BuildMaster
import os
import sys
import collective.buildbot
<BLANKLINE>
basedir = '/sample-buildout/parts/buildmaster'
buildbot = os.path.dirname(collective.buildbot.__file__)
<BLANKLINE>
configfile = os.path.join(buildbot, 'master.py')
application = service.Application('buildmaster')
<BLANKLINE>
master = BuildMaster(basedir, configfile)
master.setServiceParent(application)
<BLANKLINE>

We can also see that the configuration file generated by the recipe reflects the options we chose in our buildout configuration:

>>> cat(join('parts', 'buildmaster', 'buildbot.cfg'))
[slaves]
slave1 = password
slave2 = password
<BLANKLINE>
[buildbot]
projects-directory = /sample-buildout/parts/projects
project-name = The project
pollers-directory = /sample-buildout/parts/pollers
url = http://example.com/buildbot
wport = 8082
project-url = http://example.com
port = 8080
allow-force = false
<BLANKLINE>

The build slave recipe

The collective.buildbot:slave recipe produces a configuration file that sets up a build slave process. Once the build slave is configured you can run it by executing the controller script under the buildout’s bin directory. The controller script will be named after the section name, so if you had a [buildslave] section in your buildout.cfg you would get a bin/buildslave script.

Since the name of the section using this recipe will also become the name of the build slave it is important to choose the name that corresponds to the buildmaster configuration.

Supported options

The recipe supports the following options:

host

Hostname of the build master.

port

Port that the build master is listening. This should match the port option in the section using the collective.buildbot:master recipe in your buildmaster buildout.

password

Build slave password. This should match the password in the slave-names section in the buildmaster buildout.

Example usage

We’ll start by creating a buildout that uses the recipe:

>>> write('buildout.cfg',
... """
... [buildout]
... parts =
...    buildslave
...
... [buildslave]
... recipe = collective.buildbot:slave
... host = localhost
... port = 8888
... password = password
... """)

Running the buildout gives us:

>>> print system(buildout)
Installing buildslave.
...
Generated script /sample-buildout/parts/buildslave/buildbot.tac.
Generated script '/sample-buildout/bin/buildslave'.
<BLANKLINE>

As shown above, the buildout generated the required scripts. You can control build slave process by running:

$ ./bin/buildslave [start | stop | restart]

The project recipe

The collective.buildbot:project recipe is responsible for creating the buildbot configuration for a project which is a single testable component. Whether this project corresponds to a single sofware package or many is up to you. In most cases a project corresponds to a buildout which in turn may contain one or many software packages. Each project has a separate state and is visualized as a column in the waterfall display.

This recipe should be used in the same buildout with collective.buildbot:master.

Supported options

The recipe supports the following options:

slave-names (mandatory)

A white-space separated list of slave names that the project will be built on. These must correspond to the section names that use the collective.buildbot:slave recipe and that are consequently referred in the slaves option of the section using the collective.buildbot:master recipe.

vcs (optional)

The version control system used to obtain the source code for the project. Defaults to svn. Other possible values are: hg, bzr and git.

repository (mandatory)

The URL to the code repository that corresponds to the selected version control system. For Subversion this could be something like https://svn.plone.org/svn/collective/collective.buildbot/trunk or for Git something like [e-mail]:dokai/hexagonit-swfheader.git.

branch (optional)

The branch in the version control system that will be checked out. For Subversion checkouts you should provide the full URL to the desired branch (e.g. something that ends with trunk or branches/foo) and leave this option empty. For Git repositories the default value is master.

email-notification-sender (optional)

An email address that will be used in the From: header for the notification messages.

email-notification-recipients (optional)

A newline separated sequence of email addresses the notification messages will be sent to.

build-sequence (optional)

A newline separated sequence of shell commands executed on the build slave after checking out the code from the repository that will build the project.

Defaults to:

bin/python bootstrap.py
bin/buildout

which is appropriate for buildout based projects.

test-sequence (optional)

A newline separated sequence of shell commands executed on the build slave to execute the test suite. Defaults to:

bin/test

periodic-scheduler (optional)

Sets up a periodic scheduler that schedules a build every n minutes, where n is the given integer value.

cron-scheduler (optional)

Sets up a cron-like scheduler that schedules a build at a given time. The time is configured in a crontab manner using white space separated values for the following fields:

[minute] [hour] [day of month] [month] [day of week]

The values should be integers in the approriate range for the given field or * (asterisk) for all values. For example to schedule a build at 3:00 am every night you would use:

cron-scheduler = 0 3 * * *

Example usage

We’ll start by creating a buildout that uses the recipe. A full example would propably have other sections defining the build master and slaves, but here we will demonstrate only the use of the collective.buildbot:project recipe.

>>> write('buildout.cfg',
... """
... [buildout]
... parts = my.package
...
... [my.package]
... recipe = collective.buildbot:project
... slave-names = slave1
... vcs = svn
... repository = http://example.com/svn/my.package/trunk
... """)

Running the buildout gives us:

>>> print system(buildout)
Installing my.package.
Generated config '/sample-buildout/parts/projects/my.package.cfg'.

As we can see, the recipe generated the project configuration file under the projects directory in the parts:

>>> cat(join('parts', 'projects', 'my.package.cfg'))
[project]
...
name = my.package
repository = http://example.com/svn/my.package/trunk
...
slave-names = slave1
...
vcs = svn
<BLANKLINE>

The projects recipe

The collective.buildbot:projects recipe is deprecated and not supported anymore.

The poller recipe

Supported options

The recipe supports the following options:

Lists the projects the buildbot deal with (one project=one column) The values must be a section name in the configuration file. Then each of this section must contain:

vcs

The version control system. Defaults to svn. Currently only Subversion repositories are supported.

repository

Full URL to the branch in the code repository containing the project code.

hist-max

Number of history lines to look at (Default 100).

user

A svn user (Default None).

password

A valid svn password for the user (Default None).

poll-interval

Interval in seconds to check for changes.

svn-binary

Path to the svn binary. Defaults to svn which should work if you have in your PATH.

Example usage

We can define a poller to make our buildbot aware of commits:

>>> write('buildout.cfg',
... """
... [buildout]
... parts = svnpoller
...
... [svnpoller]
... recipe = collective.buildbot:poller
... repository = http://example.com/svn/buildout/trunk
... user = h4x0r
... password = passwd
... """)

>>> print system(buildout)
Installing svnpoller.
Generated config '/sample-buildout/parts/pollers/svnpoller.cfg'.

Poller generation. You can see here all the available options:

>>> cat(join('parts', 'pollers', 'svnpoller.cfg'))
[poller]
hist-max = 100
repository = http://example.com/svn/buildout/trunk
vcs = svn
user = h4x0r
svn-binary = svn
password = passwd
poll-interval = 60
<BLANKLINE>

The pollers recipe

The collective.buildbot:pollers is similar to the collective.buildbot:poller recipe and the only difference is that it allows you to define multiple pollers in one buildout section.

Supported options

The options are otherwise the same except that repository is replaced by repositories.

repositories

A sequence of white space separated repository URLs to poll.

Putting it all together

Below we will demonstrate how to put all the pieces together to create a buildbot environment for your own projects. We will use two separate buildouts: one for the build master and one for a single build slave. For your own projects you may choose to use multiple build slaves with each running, for example, on a different architecture or a different python version.

We’ll start with the build master buildout that defines the build master process and all the projects that we wish to build and test. We also include a poller configuration that will poll the Subversion repository for changes the projects and execute the build when changes have occurred. If we were to use another version control system, such as Git, we would need to use a commit-hook or a time-based build scheduler.

>>> write('buildout.cfg',
... """
... [buildout]
... parts =
...     buildmaster
...     svnpoller
...     my.project
...     my.buildout
...     another.package
...
... [buildmaster]
... recipe = collective.buildbot:master
... port = 8080
... wport = 8082
... project-name = The company buildout
... project-url = http://my.company.com
... url = http://buildbot.my.company.com
... allow-force = true
... public-html = ${buildout:directory}/buildbot_css
... slaves =
...     slave1 secretpassword
...
... [svnpoller]
... recipe = collective.buildbot:pollers
... repositories =
...     ${my.project:repository}
...     ${my.buildout:repository}
... user = someuser
... password = anothersecret
...
... [my.project]
... recipe = collective.buildbot:project
... slave-names = slave1
... repository = https://svn.company.com/svn/my.project/trunk
... email-notification-sender = [e-mail]
... email-notification-recipient =
...     [e-mail]
...     [e-mail]
... build-sequence =
... test-sequence = ../../bin/python setup.py test
...
... [my.buildout]
... recipe = collective.buildbot:project
... slave-names = slave1
... repository = https://svn.company.com/svn/my.buildout/trunk
... email-notification-sender = [e-mail]
... email-notification-recipient = [e-mail]
... test-sequence = bin/zope-instance test -v -vv
...
... [another.package]
... recipe = collective.buildbot:project
... slave-names = slave1
... vcs = git
... repository = git://git.company.com/projects/another-package.git
... email-notification-sender = [e-mail]
... email-notification-recipient = [e-mail]
... build-sequence =
... test-sequence = ../../bin/python setup.py test
... periodic-scheduler = 60
... """)

We’ve allowed forced builds which is quite handy sometimes. Since the default buildbot web interface is not the most aesthetic we’ve also included a directory that contains our custom css.

The my.project and another.package packages are simple python packages so we use the setup.py script to run the test suites. Because these are simple packages we also clear out the build-sequence option since there is nothing to do before running the tests. The my.buildout section is your typical Zope buildout and uses the Zope controller script, zope-instance in this particular case, to run the tests.

Also, because the another.package project uses a Git repository, the SVN poller won’t apply to it so we’ve set up a periodic scheduler that builds the project once in an hour. An alternative would be to install a post-commit hook to the Git repository that notifies the buildout of changes and schedules a build.

The my.buildout project is a buildout based project, so we can use the default build-sequence which will bootstrap and run the buildout for us. For the zope.testing test runner we pass the --exit-with-status parameter so that buildbot will know whether the tests failed or not. The trunk may have additional svn:externals defined that actually pull in the code that is tested which is the common place.

Let’s run the buildout now.

>>> mkdir('buildbot_css')
>>> print system(buildout)
Installing buildmaster.
New python executable in /sample-buildout/parts/buildmaster/bin/python
Installing setuptools............done.
Generated script '/sample-buildout/parts/buildmaster/buildbot.tac'.
Generated config '/sample-buildout/parts/buildmaster/buildbot.cfg'.
Generated script '/sample-buildout/bin/buildmaster'.
Installing my.project.
Generated config '/sample-buildout/parts/projects/my.project.cfg'.
Installing my.buildout.
Generated config '/sample-buildout/parts/projects/my.buildout.cfg'.
Installing svnpoller.
Generated config '/sample-buildout/parts/pollers/poller_0.cfg'.
Generated config '/sample-buildout/parts/pollers/poller_1.cfg'.
Installing another.package.
Generated config '/sample-buildout/parts/projects/another.package.cfg'.
<BLANKLINE>

As we can see we got the bin/buildmaster script to run the build master process and the corresponding configuration files. Our build master is now ready and you can start it by running:

$ ./bin/buildmaster start

Next, we create the buildout for the build slave. This buildout may be located on a different machine although having it on the same machine will work just as fine.

>>> write('buildout.cfg',
... """
... [buildout]
... parts =
...    buildslave
...
... [buildslave]
... recipe = collective.buildbot:slave
... host = buildbot.my.company.com
... port = 8080
... password = secretpassword
... """)

The slave buildout is very simple since the build master is in charge of everything and the slave simply needs to contact the master and receive instructions. We confifured the address of the build master and the password to match the configuration in the build master buildout above.

Running the buildout will give us the controller script:

>>> print system(buildout)
Uninstalling...
Installing buildslave.
New python executable in /sample-buildout/parts/buildslave/bin/python
Installing setuptools............done.
Generated script /sample-buildout/parts/buildslave/buildbot.tac.
Generated script '/sample-buildout/bin/buildslave'.
<BLANKLINE>

The build slave can be started by running:

$ ./bin/buildslave start

Once you have both the build master and slave running the poller should react to commits to the SVN repositories and run the builds after each change. You can view the buildbot status pages at the configured address, http://buildbot.my.company.com:8082/ in this case. You can use the web interface to force a build which can be useful to verify that the buildbot and projects are configured correctly.

It is now easy to add new projects or build slaves by modifying the buildout configurations and rerunning the buildouts.

Contributors

Project initiated at Ingeniweb (http://ingeniweb.com)

Initial Authors:

• Gael Pasgrimaud

• Tarek Ziade

Moved to the collective during the Plone Paris Sprint 2008.

Contributors:

• Gael Pasgrimaud [gawel]

• Tarek Ziade [tarek]

• Kai Lautaportti [dokai]

• Jean-Francois Roche

• Mustapha Benali [mustapha]

Download