Skip to main content

Builds Docker images, capturing information to reduce the frequency of future re-builds

Project description

[![Build Status](](
[![PyPI version](](

# Thrifty Builder
_Builds Docker images, capturing information to reduce the frequency of future re-builds_

## Introduction
Thrifty builder stores a hash of all the ingredients that go into building a Docker image so the tool is able to
determine if an image has already been built before, even if the build cache has been emptied or if the build is taking
place on a different machine with a separate cache.

In our setup, we are building a large number of Docker images in our CI. The CI job runs on a different machine each
time (with separate caches), meaning that if `docker build` was used, all images would be rebuild every time
the CI runs. Pulling first would sped things up but it would require all images to be downloaded to the build machine

The aim is to minimise our CI run time and to keep our Docker images as stable as possible (it is usually
extremely difficult to version everything that goes into an image so each re-build will create a slightly different
image, even if the context and Dockerfile are the same).

## Installation
- Docker (Server API version >= 1.21)
- Python 3.6+

The tool can be installed from PyPi:
pip install thriftybuilder

Bleeding edge versions can be installed directly from GitHub:
pip install git+

## Usage
### Configuration
A build configuration YAML file is required to use the tool. This file details the images that are to be built, the
Docker registries to push the created images to (optional) and the location of the checksum storage.

#### Storage
##### stdin/stdout
(Default if not specified)
type: stdio

##### Local
type: local
path: /root/.thrifty/checksums

##### Consul
type: consul
url: # Optional: derived from Consul environment variables if not set
token: "{{ env['CONSUL_HTTP_TOKEN'] }}" # Optional: derived from Consul environment variables if not set
key: ci/image-checksums
lock: ci/image-checksums.lock
_Note: to use Consul-backed storage, the requirements in `consul_requirements.txt` must be installed (not done so by

### CLI
usage: thrifty [-h] [-v] [--built-only] configuration-location

Builds Docker images, capturing information to reduce the frequency of future
re-builds (v1.0.0b0)

positional arguments:
location of configuration

optional arguments:
-h, --help show this help message and exit
-v increase the level of log verbosity (add multiple
increase further)
--built-only only print details about newly built images on stdout

### Example
- name: wtsi-hgi/image-1
dockerfile: /images/image-1/Dockerfile
context: /images
- name: wtsi-hgi/image-2:version-1
dockerfile: /images/image-2/Dockerfile
# Context assumed to be /images/image-2
- url:
username: "{{ env['DOCKER_IO_USERNAME'] }}"
password: "{{ env['DOCKER_IO_PASSWORD'] }}"

type: consul
url: "{{ env['CONSUL_HTTP_ADDR'] }}"
token: "{{ env['CONSUL_HTTP_TOKEN'] }}"
key: ci/image-checksums
lock: ci/image-checksums.lock

thrifty configuration.yml
{"wtsi-hgi/image-1": "b2db4c1ae978201407c69573ba89a9b8", "wtsi-hgi/image-2": "f9a4d7cc9f7133756b36973cc2d888de"}


## Development
### Setup
Install the tool's dependencies and the dependencies needed for testing:
pip install -U -r test_requirements.txt
pip install -U -r requirements.txt
pip install -U -r consul-requirements.txt

### Testing
In the project directory, run:
PYTHONPATH=. python -m unittest discover -v -s thriftybuilder/tests

The CI runs the tests and generates coverage with ``. This script runs the tests as described above in,
addition to testing that the tool can be installed without errors.

To run the tests entirely in docker, use `` (experimental).

## Alternatives
- Share a build cache between all image building machines and make sure the cache is not lost.
- More exotic Docker image builders might store information about the build context with the built image.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

thriftybuilder-1.1.0.tar.gz (26.4 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page