Opinionated wrapper around docker
Project description
An opinionated wrapper around the docker-py API to docker that knows how to read yaml files and make things happen.
Installation
Just use pip:
pip install docker-harpoon
Usage
Once harpoon is installed, there will be a new program called harpoon.
When you call harpoon without any arguments it will print out the tasks you have available.
You may invoke these tasks with the task option.
The default tasks are as follows:
- ssh
Takes in an –image option, makes the specified image and runs /bin/bash with it
- run
Takes in an –image option and either --command <command> or --bash <bash>. It will create the image you specified and run it with either <command> or /bin/bash -c '<bash>' depending on what you specified
You can also specify environment options with –env. These options are similar to what docker cli asks for. (https://docs.docker.com/reference/commandline/cli/ search for “–env”)
- make
Takes in an –image option and will make that image
- make_all
Makes all the images in layered order
- make_pushable
Makes all the pushable images and their dependencies
- pull
Takes in an –image option, and pulls in the corresponding image. (Only if it has a image_index option)
Also takes in an “–ignore-missing” argument that means it won’t error out if the image you are pulling doesn’t exist.
- pull_all
Pulls in all the images in layered order
Takes in an “–ignore-missing” argument which is useful if you’re using pull_all to get the latest images before building them and you haven’t built one of the images yet
- push
Takes in an –image option, creates your specified image and pushes it
- push_all
Pushes all the images in layered order
- list_tasks
This is the default task, it will print out what tasks are available.
Including these default tasks and any custom tasks you have defined.
- show
Prints out the known images in layered order
- show_pushable
Prints out the pushable images and their dependencies in layered order
- delete_untagged
This will find the untagged images and delete them.
Note that when harpoon creates an image, it will also create the dependent images that are defined in the configuration.
Also, by default when an image becomes untagged because a new one that harpoon creates takes the latest tag, it will be deleted. This behaviour is prevented with the --keep-replaced option.
Simpler Usage
To save typing --task and --image too much, the first positional argument is treated as task (unless it is prefixed with a -) and the second positional argument (if also not prefixed with a -) is taken as the image.
So:
$ harpoon --task run --image my_amazing_image
Is equivalent to:
$ harpoon run my_amazing_image
Logging colors
If you find the logging output doesn’t look great on your terminal, you can try setting the term_colors option in harpoon.yml to either light or dark.
Failed build intervention
Harpoon has the ability to commit failed images during build and run /bin/bash against this image. This behaviour is known as intervention.
Intervention images are cleaned up after they are exited from and are disabled if harpoon is run with either --non-interactive or with --no-intervention.
The yaml configuration
Harpoon reads everything from a yaml configuration. By default this is a harpoon.yml file in the current directory, but may be changed with the --harpoon-config option or HARPOON_CONFIG environment variable.
It will also read from ~/.harpoon.yml and will be overridden by anything in the configuration file you’ve specified.
This yaml file looks like the following:
--- images: <image_name>: <image_options>
And so when harpoon reads this yaml, it gets a dictionary of images names to image options under the images key.
An example may look like the following:
--- images: myapp: commands: - FROM ubuntu - RUN apt-get update && apt-get -y install caca-utils - CMD cacafire
And then we can do things like:
# Run the default command in the image $ harpoon run myapp # Make the image and start an interactive bash shell in it $ harpoon ssh myapp
And harpoon will make sure things are cleaned up and no longer on your system when you quit the process.
The only required option for an image is commands which is a list of commands as what you would have in a Dockerfile.
Modified file times
We noticed that if you git clone a repository then git will set the modified times of all the files to the time at which you do the git clone.
This means that even though the file contents are the same, docker will invalidate the cache when it adds these files.
Harpoon provides an option context.use_git_timestamps which when set true will use git to determine the commit date for each file and when it creates the context to send to docker it will use the git date.
for example:
--- context: use_git_timestamps: true images: blah: commands: [...]
It will make sure to only do this to files that are controlled by git and which don’t have any local modifications
Note that if you have many files, you might decide that getting the commit date for all of them takes an unacceptably long time and that you only care about a certain subset of files.
In this case, you may specify a list of globs that will be used to identify which files we set the modified times for (assuming they are also owned by git and don’t have any local modifications.
For example:
--- context: use_git_timestamps: - gradle* - settings.gradle - buildSrc/** images: blah: commands: [...]
Controlling the context
Docker is a server-client architecture, where the server is essentially a web server that speaks HTTP.
When you build an image with a docker client (for example the official docker cli tool), the client must first send a context to the server. This context is then used to locate files that are added to the image via ADD commands.
Harpoon has options available for specifying what goes into the context uploaded to the docker server. For now, it’s a little limited, but it’s certainly better than no control.
These options may be specified either at the root of the configuration or within the options for the image itself. Any option in the image options overrides the root option.
- use_gitignore
Ignore anything gitignore would when creating the context.
- exclude
A list of globs that are used to exclude files from the context
Note: Only works when use_gitignore has been specified
- enabled
Don’t include any context from the local system if this is set to false.
- parent_dir
The parent directory to get the context from. This defaults to the folder the harpoon.yml was found in.
For example, let’s say you have the following file structure:
project/ app/ ui-stuff/ large_folder/ docker/ harpoon.yml
Where for some reason large_folder is committed into git but contains a lot of large assets that don’t need to be in the docker image, then the harpoon.yml may look something like:
--- context: use_gitignore: true folders: - project_dir: "{config_root}/.." images: myapp: context: parent_dir: "{folders.project_dir}" exclude: - large_folder/** - docker/** commands: - FROM ubuntu - ADD app /project/app - ADD ui-stuff /project/ui-stuff - RUN setup_commands
This also means it’s very easy to have multiple docker files adding content from the same folder.
Inter-Document linking
Many option values in the harpoon.yml file will be formatted such that you can reference the value from something else in the document.
For example, let’s say you want to link one image into another:
--- images: db: commands: - <commands here> app: link: - ["{images.db}", "dbhost"] commands: - <commands here>
The formatting works such that looking for “{name}” will look for name in the options. In this case it looks for ‘options[“images”][“db”][“container_name”]’
Note that images have some generated values:
- image_name
The name of the image that is created. This is produced by concatenating the image_index and image_name_prefix options it finds with the name of the image.
So for:
--- image_index: some-registry.somewhere.com/user/ image_name_prefix: my-project images: blah: [..]
images.blah.image_name will be “some-registry.somewhere.com/user/my-project-blah”
- container_name
This is a concatenation of the image_name and a uuid1 hash.
This means if we fail to clean up, future invocations won’t complain about conflicting container names.
Environment variables
There is a special format “:env” that you can use to transform something into a bash variable.
For example:
--- images: blah: commands: ... tasks: something: options: bash: "echo {THINGS:env} > /tmp" env: - THINGS
Then this will run the container with the docker-cli equivalent of “–env THINGS” and run the command “/bin/bash -c ‘echo ${THINGS} > /tmp’”.
You can also specify environment variables via the –env switch.
Also, you can specify “env”, “images.<image>.env” or “images.<image>.tasks.<task>.env” as a list of environment variables you want in your image.
The syntax for the variables are:
- VARIABLE
Will complain if this variable isn’t in your current environment and will expose this environment variable to the container
- VARIABLE=VALUE
Will set this variable to VALUE regardless of whether it’s in your environment or not
- VARIABLE:DEFAULT
Will set this variable to DEFAULT if it’s not in your current environment, otherwise it will use the value in your environment
Dockerfile commands
So when you specify your image you specify a list of commands to go into the Dockerfile as a list of instructions:
--- images: myimage: commands: - <instruction> - <instruction> - <instruction>
Where instruction may be:
<string>
A string is just added into the Dockerfile as is
[<string>, <string>]
The first string is used as is, the second string is formatted and the two results are joined together to form the command.
So let’s say you have:
--- image_name_prefix: amazing-project images: base: commands: <commands here> app: commands: - [FROM, "{images.base}"]Then the first instruction for the app Dockerfile will be a FROM command that uses the base image.
- [<string>, [<string>, <string>, …]]
A list of a string and a list will use the first string as the command unmodified and it will then format each string and use that as a seperate value.
So let’s say you have:
--- image_name_prefix: amazing-project passwords: db: sup3rs3cr3t images: app: commands: - FROM ubuntu - [ENV, ["DBPASSWORD {passwords.db}", "random_variable 3"]]
Then the resulting Dockerfile for the app image will look like:
FROM ubuntu ENV DBPASSWORD sup3rs3cr3t ENV random_variable 3
- [<string>, <dictionary>]
This has special meaning depending on the first String.
[ADD, {content:<content>, dest:<dest>}]
This will add a file to the context with the content specified and make sure that gets to the destination specified.
So say you have:
--- images: app: commands: - FROM ubuntu - - ADD - dest: /tmp/blah content: | blah and stuff
This will add a file to the context with the name as some uuid value. For example “DDC895F6-6F65-43C1-BDAA-00C4B3F9BB7B” and then the Dockerfile will look like:
FROM ubuntu ADD DDC895F6-6F65-43C1-BDAA-00C4B3F9BB7B /tmp/blah
[ADD, {prefix: <prefix>, get:[<string>, <string>]}]
This is a shortcut for adding many files with the same destination prefix.
For example:
--- images: app: commands: - FROM ubuntu - - ADD - prefix: /app get: - app - lib - spec
Which translates to:
FROM ubuntu ADD app /app/app ADD lib /app/lib ADD spec /app/spec
Dependant containers
When you reference an image_name created by the harpoon config, then harpoon will ensure that image is created before it’s used.
Also, if you specify a container_name created by the harpoon config, harpoon will ensure that container is running before it is used.
For example, say you have this folder structure:
project/ app/ app/ db/ lib/ spec/ config/ Gemfile Gemfile.lock Rakefile docker/ harpoon.yml
Then your harpoon.yml may look like:
--- folders: api_dir: "{config_dir}/.." images: bundled: context: parent_dir: "{folders.api_dir}" commands: - FROM some_image_with_ruby_installed - RUN apt-get -y install libmysqlclient-dev ruby-dev - RUN mkdir /api - ADD Gemfile /api/ - ADD Gemfile.lock /api/ - WORKDIR /api - RUN bundle config --delete path && bundle config --delete without && bundle install mysql: context: parent_dir: "{folders.api_dir}" commands: - [FROM, "{images.bundled}"] - VOLUME shared <install mysql> ## Expose the database - EXPOSE 3306 - [ADD, {prefix: "/app", get: ["db", "lib", "config", "app", "Rakefile"]}] ## Run the migrations - RUN (mysqld &) && rake db:create db:migrate - CMD cp /app/db/schema.rb /shared && mysqld unit_tests: context: parent_dir: "{folders.api_dir}" link: - ["{images.mysql}", "dbhost"] volumes: share_with: - "{images.mysql}" commands: - [FROM, "{images.bundled}"] - ADD . /app/ - CMD cp /shared/schema.rb /app/db && rake
And harpoon will ensure that the bundled image is created before both the mysql and unit_tests images are created, and that when we run the unit_tests container it first creates the mysql container.
Harpoon will also ensure all these containers are cleaned up afterwards. Images stay around because we want to use the awesome caching powers of Docker.
Custom tasks
You can add tasks within your container.
For example:
--- images: app: commands: ... - CMD startup_app tasks: run_app: description: "Startup the app" run_tests: description: Run the unit tests options: bash: cd /app && rake tests
Each task has no required options but can be configured with action, options , overrides, description and label.
If action or options are not specified then the task will just create the image it’s defined under and run the default command.
If the action is specified and is just a string, then it will call that action and give the image option as the name of this image. The available tasks are those in https://github.com/realestate-com-au/harpoon/blob/master/harpoon/tasks.py with a a_task decorator. For most cases, you just need the run task which is the default value for action.
The tasks defined in these definitions will be shown when you do “harpoon –task list_tasks”.
You may also use extra arbitrary cli options for your tasks with {$@}:
--- images: app: commands: ... tasks: something: options: bash: cd /app && ./some_script.sh {$@}
Then say you run harpoon like:
$ harpoon --task something -- --an-option 1
Then it will start up the app container and run:
$ /bin/bash -c 'cd /app && ./some_script.sh --an-option 1'
Because everything that comes after a -- in the argv to harpoon will be available as “$@”.
Linking containers and volumes
You have the following options available:
- link
A list of strings that are equivalent to the options you give link for docker cli (https://docs.docker.com/userguide/dockerlinks/#container-linking)
For example:
--- images: db: commands: ... app: link: - ["{images.db.container_name}", "dbhost"] commands: ...
Will make sure that when you start the app container, it will run the db image in a detached state and there will be an entry in the /etc/hosts of the app container that points dbhost to this db container.
- volumes.share_with
This behaves like link in that you specify strings similar to what you would do for the docker cli (https://docs.docker.com/userguide/dockervolumes/#creating-and-mounting-a-data-volume-container)
So something like:
--- images: db: commands: - FROM ubuntu - VOLUME /shared app: volumes: share_with: - "{images.db}" commands: ...
Then the app container will share the volumes from the db container.
- volumes.mount
This is also specified as string similar to what you do for the docker cli (https://docs.docker.com/userguide/dockervolumes/#data-volumes)
For example:
--- folders: app_dir: "{config_root}/../app" images: app: volumes: mount: - "{app_dir}/coverage:/project/app/coverage:rw"
Will mount the coverage directory from the host into /project/app/coverage on the image.
Sometimes you need your dependency container to not be running in a detached container. To make it so a dependency is running in an attached container, you may specify dependency_options:
--- images: runner: commands: ... - CMD activator run uitest: link: - ["{images.runner}", "running"] dependency_options: runner: # Typesafe activator run stops in a detached container attached: True commands: ... - CMD ./do_a_uitest.sh running:9000
Tests
Install testing deps and run the helpful script:
pip install -e . pip install -e ".[tests]" ./test.sh
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.