Skip to main content

A utility for automating multi-host, multi-environment software builds and deployments.

Project description

batou

batou is a multi-(component|host|environment|…) deployment utility.

Deployments are described in a model using “components” and can be deployed locally and remotely. Models are written as Python code.

See the batou documentation for details on how to use it.

Changelog

1.0.6 (2015-04-01)

  • Add “dirty” flag to dependencies to express that it’s OK to work with dirty data, e.g. to simply do a one-time retrieval, as long as we get _something_ back. Useful to peek onto static values like overrides or configuration when otherwise confronted with circular dependencies.

1.0.5 (2015-03-31)

  • Fix unicode accident: shells with ASCII only failed to run the supervisor cold deployment log message.

1.0.4 (2015-03-31)

  • A new event handling system that allows intercepting and responding to events from components. Currently supports “before-update”.

  • Supplement supervisor programs with ‘deployment’ mode parameter to support ‘cold’ deployments that shut down programs before any updates get deployed without causing superfluous cycles.

1.0.3 (2015-03-18)

  • Fix unicode encoding/decoding in File handling. See #14944.

  • Remove remnant of better error output that got deferred to the 1.1 branch that breaks reporting any conversion errors at all.

1.0.2 (2015-02-27)

  • Fix a bundle shipping issue: incorrect language settings caused us to incorrectly parse the list of heads on the remote side.

1.0.1 (2015-02-26)

  • Add virtualenv metadata for Python 2.4

  • Keep trailing newlines when rendering templates

1.0 (2015-02-10)

  • Add ‘connect_method = vagrant’ to the environment if you want to deploy into vagrant and use the ‘vagrant ssh’ invocation.

  • Update internal buildout, setuptools, and execnet versions.

  • Add ‘update_method = rsync’ to provide a simple, non-VCS specific way of shipping batou definitions to the remote side. This does not perform any conflict resolution and is mainly intended for use with vagrant.

  • Supervisor: Assert that a Program should get to state “RUNNING” within startsecs after restarting, instead of only waiting for startsecs.

1.0b33 (2015-01-28)

  • Fix bug in git Clone: Directly after cloning there is nothing to merge, so don’t attempt to.

1.0b32 (2015-01-22)

  • Fix custom MySQL admin user when executing SQL statements.

1.0b31 (2015-01-16)

  • Rename a variable in ServiceCheck that conflicted with NRPEService and prevented custom naming of NRPE checks.

1.0b30 (2015-01-09)

  • A special case in error reporting caused batou local to exit silently on an error.

  • Provide an easier to read error message when encountering non-converging working sets.

  • Provide an easier to read error message when encountering cycles in the component dependencies.

  • Fix bug in git Clone that caused breakage during update() when a fixed revision was specified.

1.0b29 (2014-12-12)

  • Make target directory on remote side configurable via Environment.target_directory (default is ~/deployment).

  • Improve installation documentation w.r.t more specific instructions for various Linux distributions (deb- and rpm-based).

  • Change mode of files before changing content to allow self-fixing broken modes.

  • Allow deleting symlinks (on OS X) if they point to directorys.

1.0b28 (2014-10-10)

  • batou bootstrap script is more roboust now for cases where the virtualenv becomes broken.

1.0b27 (2014-09-15)

  • Do not configure environments locally when running remote deployments. Use order as determined by one reference remote host. #13530

  • Nagios configuration now supports contact groups.

1.0b26 (2014-08-05)

  • Enhance git Clone component to support the full suite of functionality the mercurial one has (incoming/outgoing/local changes etc.).

    NOTE: This changes batou.lib.git.Clone in backward-incompatible ways:

    1. The namevar is now only the url. The previously also accepted usage of passing the target as the namevar no longer works.

    2. update_unpinned is gone, use the branch/revision/vcs_update mechanism instead.

1.0b25 (2014-07-31)

  • Improve gpg handling: don’t do –no-tty or –batch any longer as we actually only run the gpg code in (local) terminal environments usually.

1.0b24 (2014-07-04)

  • Actually fix unzipping.

  • Remove environment option connect_method, and use autodetection instead: If we’re not the configured service_user after the plain ssh connect, run sudo.

  • Refactor attributes and overrides: attributes allow template expansion now and also converting default values. The Attribute class has moved to the batou component module.

  • Yield nonzero returncode on remote exceptions.

  • Add convenience converter for attributes “list” to handle comma separated lists of strings.

  • Force safety of Nagios service description field: prohibit “,” as its not safe for Nagios business processes.

  • Allow specifying custom rsync options for SyncDir: the ‘–inplace’ option is a good idea in most cases, but not all. Fixes #11846.

  • Make supervisor restart/reload more robust towards non-running supervisors with lying PID files. Fixes #12744.

  • Remove harmful decoding step in File verify that caused wrong results.

1.0b23 (2014-06-30)

  • Allow unzip to overwrite existing files if the archive has newer copies.

1.0b22 (2014-06-30)

  • Fix uncaught import-error on mysql module.

1.0b21 (2014-06-30)

  • Fix bug in bundle shipping that made batou depend on the locale (#13465).

  • Add MySQL components.

  • Introduce batou.utils.Attribute() to avoid imperative type conversions in configure().

1.0b20 (2014-05-13)

  • Start using python-requests for HTTP/S downloads to support authentication (and other goodies).

  • Improve error logging.

1.0b19 (2014-04-07)

  • Allow specifying alternate find-links in requirements.txt

  • Remove remnants of the buildout component’s extends option.

1.0b18 (2014-03-31)

  • Introduce environment option connect_method, either ‘ssh+sudo’ (default) or plain ‘ssh’.

  • Improve update code: existing installations would not update cleanly to a new version due to wrong use of pkg_resources API. Existing installations need to be cleaned manually. :/

  • Archive: recognize .tar.xz files and extract them with tar.

  • Fix bug in CMMI verify that would always raise UpdateNeeded due to a path mismatch (#13145).

  • Component.cmd() now performs expand() on the given command by default. Saves typing:

    self.cmd(‘rm -rf {{component.workdir}}’)

    instead of

    self.cmd(self.expand(‘rm -rf {{component.workdir}}’))

  • File component: apply character encoding when reading and writing files. (#13165)

  • Mercurial clone: changed update behaviour yet again: unless configured differently, always achieve target state, even by losing uncommitted changes.

  • Allow unnamed access to the last added/ored component using the ‘self._’ attribute:

    def configure(self):

    self += Download(…) self += Extract(self._.target)

1.0b17 (2013-12-06)

  • Mercurial clone: refuse to update if there are uncommitted changes or outgoing changesets.

  • Mercurial clone: added an option to avoid updating existing clones, which is often desirable during development.

1.0b16 (2013-11-29)

  • Improve the bootstrap/upgrade fix for #13049: the bootstrap template needed a second improvement.

1.0b15 (2013-11-29)

  • Try to fix #13049 - will only work once the dev version is already at least at b15. Not sure whether it really does the job, though. Hard to test.

  • Make environment and host arguments for running batou local optional. dev and localhost are the defaults now. Fixes #13034

  • Allow specifying secrets with [component:xy] in addition to [xy] to make it easier to discover when used to the environment overrides. Both options are supported now. Fixes #13003

  • Refactor Python virtualenv and package management. Provide explicit knowledge about Python versions 2.4, 2.5, 2.6, 2.7, 3.1, 3.2, and 3.3 to use compatible virtualenv/pip/setuptools versions and commands. Fixes #12941

    NOTE: The API for batou.lib.python.Package has changed in backwards-incompatible ways:

    Package() needs to be added to a VirtualEnv component. Before:

    self += VirtualEnv(...)
    self += Package(...)

    Now:

    venv = VirtualEnv(...)
    self += venv
    venv += Package(...)

    Package parameter pip_install_options has been renamed to install_options.

  • Use urllib module instead of wget command for cross platform compatibility.

1.0b14 (2013-11-26)

  • Make Buildout component install zc.buildout without dependencies (which is just setuptools), since that can break the version pinning of setuptools.

  • Check whether packages required via a -e expression in the local requirements.txt are already installed.

  • #12946: Make full path to python executable configurable for virtualenv (and buildout) component.

  • Remove cleanup code that removes batou-0.2-style virtualenv directories (bin, lib, include etc.) from remote deployments.

  • Add mailto setting to crontab component.

  • Added some documentation.

1.0b13 (2013-11-19)

  • Introduce a plugin mechanism: if a local requirements.txt file is present, install its contents into the .batou virtualenv.

  • Load components in alphabetical order. This enables importing from other components (when naming the directories appropriately).

  • #12986: Only try to unbundle if there actually are changes to unbundle.

  • Raise an error if a cron job is configured without a timing value.

  • Add platform support for Debian (crontab, logrotate, supervisor).

  • Added some documentation.

1.0b12 (2013-11-04)

  • Added branch argument to mercurial.Clone. Setting a branch automatically updates to the branch head on deploy. This is mostly useful for development environments.

  • Create the ‘secrets’ directory if it doesn’t exist, yet. Also, disallow editing secret files for non-existing environments.

  • Support continuing remote bootstrapping if we failed after creating the initial remote directory but were unable to use Mercurial.

  • #12898: build.Configure component was broken when using the default prefix.

1.0b11 (2013-10-17)

  • #12897: Use non-SSL pypi mirror for downloading virtualenv to fix tests failing randomly on machines that (for some reason) can’t validate PyPI’s certificate.

  • #12911: Ensure that we can configure file owners when they don’t exist during configure phase yet.

  • #12912: Fix untested and broken file ownership management.

  • #12847: Clean up unicode handling for File and Content components and templating.

  • #12910: Remote deployments failed when using bundles for transfers if no changes needed bundling.

  • #12766: Allow bootstrapping a batou project in an existing directory to support migration from 0.2.

  • #12283: Recognize files as ‘is_template’ by default. Auto-detect source files in the definition directory if they have the same basename. This is what you want in 99% of all cases. Explicitly stating either the ‘content’ or ‘source’ parameter disables auto-detection.

    Now you can write this:

    File(‘foo’)

    and have components/x/foo recognized as the source file and handled as a template.

  • Use ConfigParser instead of configobj which is effectively unmaintained (see https://twitter.com/voidspace/status/380699699042213889) and support lists separated by newlines in addition to commas.

1.0b10 (2013-09-27)

  • Package our own virtualenv instead of depending on the system-installed one. This should alleviate troubles due to old virtualenv versions that package distribute, which causes conflicts with recent setuptools versions (#12874).

  • Update supervisor version to 3.0.

1.0b9 (2013-08-22)

  • Update Package component so it ignores installed packages when installing. This way, we actually install setuptools even when distribute is installed. (Otherwise it’s a no-op since distribute tells pip that setuptools is already satisfied).

  • Fix update process: wrong call to old ‘.batou/bin/batou’ failed and early bootstrapping would downgrade temporarily which is confusing and superfluous. Fixes #12739.

1.0b8 (2013-08-17)

  • Remove superfluous mkdir call during remote bootstrap.

  • Make batou init print that it’s working. Bootstrapping can take a while, so at least signal that something’s going on.

1.0b7 (2013-08-17)

  • Depend on Python2.7 to be available on the PATH during early bootstrap. Otherwise our chances to get a 2.7 virtualenv are pretty small, too.

  • Improve project template: ignore the work/ directory by default.

1.0b6 (2013-08-17)

  • More MANIFEST inclusions: bootstrap-template.

1.0b5 (2013-08-17)

  • Improve MANIFEST so we actually package the init template and other generated files, like version.txt and requirements.txt.

1.0b4 (2013-08-17)

  • Provide a simple project-creation command, both for pip-installed batou’s as well as spawning new projects from existing ones. Fixes #12730

  • Fix #12679: make timeouts configurable.

  • Removed re-imports from batou main module to support light-weight self-installation and bootstrapping. I.e. ‘from batou import Component’ no longer works.

  • Provide a single main command together with a ‘bootstrap’ wrapper that you can check into your project and that is maintained during updates automatically. It also provides fully automatic bootstrapping, installation, upgrading and other maintenance.

  • Fix Python package installation version check.

  • Don’t use bin/buildout bootstrap command anymore. PIP installs a sufficient bin/buildout so buildout can do the rest internally.

  • Install zc.buildout during bootstrapping phase using PIP to avoid bootstrap.py problems.

  • Shorten URLs in the Build component to their basename.

  • Add ‘assert_cmd’ API to support simpler assertions for verify when needing to check the result of an external command.

  • Switch to asking pip installing eggs instead of flat installations as namespaces seem to collide otherwise.

  • Remove non-functional deprecated ‘md5sum’ attribute.

  • Components are context managers now. If you provide __enter__ it will be called before verify() and if you provide __exit__ this will be called after update (always - even if update isn’t actually called). This allows you to manage temporary state on the target system more gracefully. See the DMGExtractor for an example.

  • Major refactoring of internal data structures to simplify and improve test coverage. Some breakage to be expected:

    • Components do not have a ‘service’ attribute any longer. If you need access to the service base directory use ‘environment.base_dir’.

  • Another attempt at getting convergency for package installation: actually leverage pkg_resources to check what version of a package is installed. PIP does not necessarily list or freeze setuptools, even if forced to reinstall.

  • Remove superfluous ‘–update’ from pip calls that would temporarily install the incorrect package version.

  • Improve convenience of the ‘secrets-edit’ wrapper script to allow re-encrypting without re-entering the editor.

  • Consistently switch to using setuptools.

  • Fix #12399: incorrect stat attributes for Owner and Group

  • Add exclude parameter to Directory component.

  • Add env parameter to Component.cmd() (and corresponding build_environment parameter to the Build component) to allow adding/overriding environment variables.

1.0b3 (2013-07-09)

  • Enable logging in the remote core to see what’s going on on the remote side.

  • Try to better format exceptions from the remote side.

  • Try harder to get virtualenv back into a working state.

  • Allow remote deployments from root of repository.

  • Make PIP management more robust.

1.0b2 (2013-07-09)

  • Add component to manage PIP within a virtual env.

  • Add component to manage packages with PIP within a virtual env.

  • Restructure buildout component to make it more robust regarding setuptools/distribute preparation. Also remove usage of bootstrap completely as we rely on virtualenv anyway.

1.0b1 (2013-07-09)

  • Apply semantic versioning: initial development is over, so this is 1.0 now.

  • Major revamp of secrets management:

    • switch to GPG (instead of aespipe)

    • turn secrets into a core feature, removing the need for a special component

  • Add ‘–single’ to suppress parallel bootstrapping.

Developing batou

Issue tracker:

https://plan.flyingcircus.io/projects/batou

Source code:

https://bitbucket.org/flyingcircus/batou/

Project details


Release history Release notifications | RSS feed

This version

1.0.6

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

batou-1.0.6.zip (122.3 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page