batou 1.0b21

A utility for automating multi-host, multi-environment software buildsand deployments.

batou

batou is a multi-(component|host|environment|…) deployment utility.

Deployments are described in a model using “components” and can be deployed locally and remotely. Models are written as Python code.

See the batou documentation for details on how to use it.

Changelog

1.0b21 (2014-06-30)

• Fix bug in bundle shipping that made batou depend on the locale (#13465).

• Add MySQL components.

• Introduce batou.utils.Attribute() to avoid imperative type conversions in configure().

1.0b20 (2014-05-13)

• Start using python-requests for HTTP/S downloads to support authentication (and other goodies).

• Improve error logging.

1.0b19 (2014-04-07)

• Allow specifying alternate find-links in requirements.txt

• Remove remnants of the buildout component’s extends option.

1.0b18 (2014-03-31)

• Introduce environment option connect_mode, either ‘ssh+sudo’ (default) or plain ‘ssh’.

• Improve update code: existing installations would not update cleanly to a new version due to wrong use of pkg_resources API. Existing installations need to be cleaned manually. :/

• Archive: recognize .tar.xz files and extract them with tar.

• Fix bug in CMMI verify that would always raise UpdateNeeded due to a path mismatch (#13145).

• Component.cmd() now performs expand() on the given command by default. Saves typing:

  self.cmd(‘rm -rf {{component.workdir}}’)

  instead of

  self.cmd(self.expand(‘rm -rf {{component.workdir}}’))

• File component: apply character encoding when reading and writing files. (#13165)

• Mercurial clone: changed update behaviour yet again: unless configured differently, always achieve target state, even by losing uncommitted changes.

• Allow unnamed access to the last added/ored component using the ‘self._’ attribute:

  def configure(self):

  self += Download(…) self += Extract(self._.target)

1.0b17 (2013-12-06)

• Mercurial clone: refuse to update if there are uncommitted changes or outgoing changesets.

• Mercurial clone: added an option to avoid updating existing clones, which is often desirable during development.

1.0b16 (2013-11-29)

• Improve the bootstrap/upgrade fix for #13049: the bootstrap template needed a second improvement.

1.0b15 (2013-11-29)

• Try to fix #13049 - will only work once the dev version is already at least at b15. Not sure whether it really does the job, though. Hard to test.

• Make environment and host arguments for running batou local optional. dev and localhost are the defaults now. Fixes #13034

• Allow specifying secrets with [component:xy] in addition to [xy] to make it easier to discover when used to the environment overrides. Both options are supported now. Fixes #13003

• Refactor Python virtualenv and package management. Provide explicit knowledge about Python versions 2.4, 2.5, 2.6, 2.7, 3.1, 3.2, and 3.3 to use compatible virtualenv/pip/setuptools versions and commands. Fixes #12941

  NOTE: The API for batou.lib.python.Package has changed in backwards-incompatible ways:

  Package() needs to be added to a VirtualEnv component. Before:

  self += VirtualEnv(...)
  self += Package(...)

  Now:

  venv = VirtualEnv(...)
  self += venv
  venv += Package(...)

  Package parameter pip_install_options has been renamed to install_options.

• Use urllib module instead of wget command for cross platform compatibility.

1.0b14 (2013-11-26)

• Make Buildout component install zc.buildout without dependencies (which is just setuptools), since that can break the version pinning of setuptools.

• Check whether packages required via a -e expression in the local requirements.txt are already installed.

• #12946: Make full path to python executable configurable for virtualenv (and buildout) component.

• Remove cleanup code that removes batou-0.2-style virtualenv directories (bin, lib, include etc.) from remote deployments.

• Add mailto setting to crontab component.

• Added some documentation.

1.0b13 (2013-11-19)

• Introduce a plugin mechanism: if a local requirements.txt file is present, install its contents into the .batou virtualenv.

• Load components in alphabetical order. This enables importing from other components (when naming the directories appropriately).

• #12986: Only try to unbundle if there actually are changes to unbundle.

• Raise an error if a cron job is configured without a timing value.

• Add platform support for Debian (crontab, logrotate, supervisor).

• Added some documentation.

1.0b12 (2013-11-04)

• Added branch argument to mercurial.Clone. Setting a branch automatically updates to the branch head on deploy. This is mostly useful for development environments.

• Create the ‘secrets’ directory if it doesn’t exist, yet. Also, disallow editing secret files for non-existing environments.

• Support continuing remote bootstrapping if we failed after creating the initial remote directory but were unable to use Mercurial.

• #12898: build.Configure component was broken when using the default prefix.

1.0b11 (2013-10-17)

• #12897: Use non-SSL pypi mirror for downloading virtualenv to fix tests failing randomly on machines that (for some reason) can’t validate PyPI’s certificate.

• #12911: Ensure that we can configure file owners when they don’t exist during configure phase yet.

• #12912: Fix untested and broken file ownership management.

• #12847: Clean up unicode handling for File and Content components and templating.

• #12910: Remote deployments failed when using bundles for transfers if no changes needed bundling.

• #12766: Allow bootstrapping a batou project in an existing directory to support migration from 0.2.

• #12283: Recognize files as ‘is_template’ by default. Auto-detect source files in the definition directory if they have the same basename. This is what you want in 99% of all cases. Explicitly stating either the ‘content’ or ‘source’ parameter disables auto-detection.

  Now you can write this:

  File(‘foo’)

  and have components/x/foo recognized as the source file and handled as a template.

• Use ConfigParser instead of configobj which is effectively unmaintained (see https://twitter.com/voidspace/status/380699699042213889) and support lists separated by newlines in addition to commas.

1.0b10 (2013-09-27)

• Package our own virtualenv instead of depending on the system-installed one. This should alleviate troubles due to old virtualenv versions that package distribute, which causes conflicts with recent setuptools versions (#12874).

• Update supervisor version to 3.0.

1.0b9 (2013-08-22)

• Update Package component so it ignores installed packages when installing. This way, we actually install setuptools even when distribute is installed. (Otherwise it’s a no-op since distribute tells pip that setuptools is already satisfied).

• Fix update process: wrong call to old ‘.batou/bin/batou’ failed and early bootstrapping would downgrade temporarily which is confusing and superfluous. Fixes #12739.

1.0b8 (2013-08-17)

• Remove superfluous mkdir call during remote bootstrap.

• Make batou init print that it’s working. Bootstrapping can take a while, so at least signal that something’s going on.

1.0b7 (2013-08-17)

• Depend on Python2.7 to be available on the PATH during early bootstrap. Otherwise our chances to get a 2.7 virtualenv are pretty small, too.

• Improve project template: ignore the work/ directory by default.

1.0b6 (2013-08-17)

• More MANIFEST inclusions: bootstrap-template.

1.0b5 (2013-08-17)

• Improve MANIFEST so we actually package the init template and other generated files, like version.txt and requirements.txt.

1.0b4 (2013-08-17)

• Provide a simple project-creation command, both for pip-installed batou’s as well as spawning new projects from existing ones. Fixes #12730

• Fix #12679: make timeouts configurable.

• Removed re-imports from batou main module to support light-weight self-installation and bootstrapping. I.e. ‘from batou import Component’ no longer works.

• Provide a single main command together with a ‘bootstrap’ wrapper that you can check into your project and that is maintained during updates automatically. It also provides fully automatic bootstrapping, installation, upgrading and other maintenance.

• Fix Python package installation version check.

• Don’t use bin/buildout bootstrap command anymore. PIP installs a sufficient bin/buildout so buildout can do the rest internally.

• Install zc.buildout during bootstrapping phase using PIP to avoid bootstrap.py problems.

• Shorten URLs in the Build component to their basename.

• Add ‘assert_cmd’ API to support simpler assertions for verify when needing to check the result of an external command.

• Switch to asking pip installing eggs instead of flat installations as namespaces seem to collide otherwise.

• Remove non-functional deprecated ‘md5sum’ attribute.

• Components are context managers now. If you provide __enter__ it will be called before verify() and if you provide __exit__ this will be called after update (always - even if update isn’t actually called). This allows you to manage temporary state on the target system more gracefully. See the DMGExtractor for an example.

• Major refactoring of internal data structures to simplify and improve test coverage. Some breakage to be expected:

  • Components do not have a ‘service’ attribute any longer. If you need access to the service base directory use ‘environment.base_dir’.

• Another attempt at getting convergency for package installation: actually leverage pkg_resources to check what version of a package is installed. PIP does not necessarily list or freeze setuptools, even if forced to reinstall.

• Remove superfluous ‘–update’ from pip calls that would temporarily install the incorrect package version.

• Improve convenience of the ‘secrets-edit’ wrapper script to allow re-encrypting without re-entering the editor.

• Consistently switch to using setuptools.

• Fix #12399: incorrect stat attributes for Owner and Group

• Add exclude parameter to Directory component.

• Add env parameter to Component.cmd() (and corresponding build_environment parameter to the Build component) to allow adding/overriding environment variables.

1.0b3 (2013-07-09)

• Enable logging in the remote core to see what’s going on on the remote side.

• Try to better format exceptions from the remote side.

• Try harder to get virtualenv back into a working state.

• Allow remote deployments from root of repository.

• Make PIP management more robust.

1.0b2 (2013-07-09)

• Add component to manage PIP within a virtual env.

• Add component to manage packages with PIP within a virtual env.

• Restructure buildout component to make it more robust regarding setuptools/distribute preparation. Also remove usage of bootstrap completely as we rely on virtualenv anyway.

1.0b1 (2013-07-09)

• Apply semantic versioning: initial development is over, so this is 1.0 now.

• Major revamp of secrets management:

  • switch to GPG (instead of aespipe)

  • turn secrets into a core feature, removing the need for a special component

• Add ‘–single’ to suppress parallel bootstrapping.

Developing batou

Issue tracker:

https://projects.gocept.com/projects/batou/

Source code:

https://bitbucket.org/gocept/batou/