docker-harpoon 0.5.4

Opinionated wrapper around docker

An opinionated wrapper around the docker-py API to docker that knows how to read yaml files and make things happen.

Installation

Just use pip:

pip install docker-harpoon

Usage

Once harpoon is installed, there will be a new program called harpoon.

When you call harpoon without any arguments it will print out the tasks you have available.

You may invoke these tasks with the task option.

The default tasks are as follows:

ssh

Takes in an –image option, makes the specified image and runs /bin/bash with it

run

Takes in an –image option and either --command <command> or --bash <bash>. It will create the image you specified and run it with either <command> or /bin/bash -c '<bash>' depending on what you specified

You can also specify environment options with –env. These options are similar to what docker cli asks for. (https://docs.docker.com/reference/commandline/cli/ search for “–env”)

make

Takes in an –image option and will make that image

make_all

Makes all the images in layered order

make_pushable

Makes all the pushable images and their dependencies

pull

Takes in an –image option, and pulls in the corresponding image. (Only if it has a image_index option)

Also takes in an “–ignore-missing” argument that means it won’t error out if the image you are pulling doesn’t exist.

pull_all

Pulls in all the images in layered order

Takes in an “–ignore-missing” argument which is useful if you’re using pull_all to get the latest images before building them and you haven’t built one of the images yet

push

Takes in an –image option, creates your specified image and pushes it

push_all

Pushes all the images in layered order

list_tasks

This is the default task, it will print out what tasks are available.

Including these default tasks and any custom tasks you have defined.

show

Prints out the known images in layered order

show_pushable

Prints out the pushable images and their dependencies in layered order

delete_untagged

This will find the untagged images and delete them.

Note that when harpoon creates an image, it will also create the dependent images that are defined in the configuration.

Also, by default when an image becomes untagged because a new one that harpoon creates takes the latest tag, it will be deleted. This behaviour is prevented with the --keep-replaced option.

Simpler Usage

To save typing --task and --image too much, the first positional argument is treated as task (unless it is prefixed with a -) and the second positional argument (if also not prefixed with a -) is taken as the image.

So:

$ harpoon --task run --image my_amazing_image

Is equivalent to:

$ harpoon run my_amazing_image

Logging colors

If you find the logging output doesn’t look great on your terminal, you can try setting the term_colors option in harpoon.yml to either light or dark.

Failed build intervention

Harpoon has the ability to commit failed images during build and run /bin/bash against this image. This behaviour is known as intervention.

Intervention images are cleaned up after they are exited from and are disabled if harpoon is run with either --non-interactive or with --no-intervention.

The yaml configuration

Harpoon reads everything from a yaml configuration. By default this is a harpoon.yml file in the current directory, but may be changed with the --harpoon-config option or HARPOON_CONFIG environment variable.

It will also read from ~/.harpoon.yml and will be overridden by anything in the configuration file you’ve specified.

This yaml file looks like the following:

---

images:
  <image_name>:
    <image_options>

And so when harpoon reads this yaml, it gets a dictionary of images names to image options under the images key.

An example may look like the following:

---

images:
myapp:
  commands:
    - FROM ubuntu
    - RUN apt-get update && apt-get -y install caca-utils
    - CMD cacafire

And then we can do things like:

# Run the default command in the image
$ harpoon run myapp

# Make the image and start an interactive bash shell in it
$ harpoon ssh myapp

And harpoon will make sure things are cleaned up and no longer on your system when you quit the process.

The only required option for an image is commands which is a list of commands as what you would have in a Dockerfile.

Modified file times

We noticed that if you git clone a repository then git will set the modified times of all the files to the time at which you do the git clone.

This means that even though the file contents are the same, docker will invalidate the cache when it adds these files.

Harpoon provides an option context.use_git_timestamps which when set true will use git to determine the commit date for each file and when it creates the context to send to docker it will use the git date.

for example:

---

context:
  use_git_timestamps: true

images:
  blah:
    commands:
      [...]

It will make sure to only do this to files that are controlled by git and which don’t have any local modifications

Note that if you have many files, you might decide that getting the commit date for all of them takes an unacceptably long time and that you only care about a certain subset of files.

In this case, you may specify a list of globs that will be used to identify which files we set the modified times for (assuming they are also owned by git and don’t have any local modifications.

For example:

---

context:
  use_git_timestamps:
    - gradle*
    - settings.gradle
    - buildSrc/**

images:
  blah:
    commands:
      [...]

Controlling the context

Docker is a server-client architecture, where the server is essentially a web server that speaks HTTP.

When you build an image with a docker client (for example the official docker cli tool), the client must first send a context to the server. This context is then used to locate files that are added to the image via ADD commands.

Harpoon has options available for specifying what goes into the context uploaded to the docker server. For now, it’s a little limited, but it’s certainly better than no control.

These options may be specified either at the root of the configuration or within the options for the image itself. Any option in the image options overrides the root option.

use_gitignore

Ignore anything gitignore would when creating the context.

exclude

A list of globs that are used to exclude files from the context

Note: Only works when use_gitignore has been specified

enabled

Don’t include any context from the local system if this is set to false.

parent_dir

The parent directory to get the context from. This defaults to the folder the harpoon.yml was found in.

For example, let’s say you have the following file structure:

project/
  app/
  ui-stuff/
  large_folder/
  docker/
    harpoon.yml

Where for some reason large_folder is committed into git but contains a lot of large assets that don’t need to be in the docker image, then the harpoon.yml may look something like:

---

context:
  use_gitignore: true

folders:
  - project_dir: "{config_root}/.."

images:
  myapp:
    context:
      parent_dir: "{folders.project_dir}"
      exclude:
        - large_folder/**
        - docker/**

    commands:
      - FROM ubuntu
      - ADD app /project/app
      - ADD ui-stuff /project/ui-stuff
      - RUN setup_commands

This also means it’s very easy to have multiple docker files adding content from the same folder.

Inter-Document linking

Many option values in the harpoon.yml file will be formatted such that you can reference the value from something else in the document.

For example, let’s say you want to link one image into another:

---

images:
  db:
    commands:
      - <commands here>
  app:
    link:
      - ["{images.db}", "dbhost"]

    commands:
      - <commands here>

The formatting works such that looking for “{name}” will look for name in the options. In this case it looks for ‘options[“images”][“db”][“container_name”]’

Note that images have some generated values:

image_name

The name of the image that is created. This is produced by concatenating the image_index and image_name_prefix options it finds with the name of the image.

So for:

---

image_index: some-registry.somewhere.com/user/
image_name_prefix: my-project
images:
    blah:
        [..]

images.blah.image_name will be “some-registry.somewhere.com/user/my-project-blah”

container_name

This is a concatenation of the image_name and a uuid1 hash.

This means if we fail to clean up, future invocations won’t complain about conflicting container names.

Environment variables

There is a special format “:env” that you can use to transform something into a bash variable.

For example:

---

images:
  blah:
    commands:
      ...

    tasks:
      something:
        options:
          bash: "echo {THINGS:env} > /tmp"
          env:
            - THINGS

Then this will run the container with the docker-cli equivalent of “–env THINGS” and run the command “/bin/bash -c ‘echo ${THINGS} > /tmp’”.

You can also specify environment variables via the –env switch.

Also, you can specify “env”, “images.<image>.env” or “images.<image>.tasks.<task>.env” as a list of environment variables you want in your image.

The syntax for the variables are:

VARIABLE

Will complain if this variable isn’t in your current environment and will expose this environment variable to the container

VARIABLE=VALUE

Will set this variable to VALUE regardless of whether it’s in your environment or not

VARIABLE:DEFAULT

Will set this variable to DEFAULT if it’s not in your current environment, otherwise it will use the value in your environment

Dockerfile commands

So when you specify your image you specify a list of commands to go into the Dockerfile as a list of instructions:

---

images:
  myimage:
    commands:
      - <instruction>
      - <instruction>
      - <instruction>

Where instruction may be:

<string>

A string is just added into the Dockerfile as is

[<string>, <string>]

The first string is used as is, the second string is formatted and the two results are joined together to form the command.

So let’s say you have:

---

image_name_prefix: amazing-project

images:
  base:
    commands:
      <commands here>
  app:
    commands:
      - [FROM, "{images.base}"]

Then the first instruction for the app Dockerfile will be a FROM command that uses the base image.

[<string>, [<string>, <string>, …]]

A list of a string and a list will use the first string as the command unmodified and it will then format each string and use that as a seperate value.

So let’s say you have:

---

image_name_prefix: amazing-project

passwords:
  db: sup3rs3cr3t

images:
  app:
    commands:
      - FROM ubuntu
      - [ENV, ["DBPASSWORD {passwords.db}", "random_variable 3"]]

Then the resulting Dockerfile for the app image will look like:

FROM ubuntu
ENV DBPASSWORD sup3rs3cr3t
ENV random_variable 3

[<string>, <dictionary>]

This has special meaning depending on the first String.

[ADD, {content:<content>, dest:<dest>}]

This will add a file to the context with the content specified and make sure that gets to the destination specified.

So say you have:

---

images:
  app:
    commands:
      - FROM ubuntu
      - - ADD
        - dest: /tmp/blah
          content: |
            blah and
            stuff

This will add a file to the context with the name as some uuid value. For example “DDC895F6-6F65-43C1-BDAA-00C4B3F9BB7B” and then the Dockerfile will look like:

FROM ubuntu
ADD DDC895F6-6F65-43C1-BDAA-00C4B3F9BB7B /tmp/blah

[ADD, {prefix: <prefix>, get:[<string>, <string>]}]

This is a shortcut for adding many files with the same destination prefix.

For example:

---

images:
  app:
    commands:
      - FROM ubuntu
      - - ADD
        - prefix: /app
          get:
            - app
            - lib
            - spec

Which translates to:

FROM ubuntu
ADD app /app/app
ADD lib /app/lib
ADD spec /app/spec

Dependant containers

When you reference an image_name created by the harpoon config, then harpoon will ensure that image is created before it’s used.

Also, if you specify a container_name created by the harpoon config, harpoon will ensure that container is running before it is used.

For example, say you have this folder structure:

project/
  app/
    app/
    db/
    lib/
    spec/
    config/
    Gemfile
    Gemfile.lock
    Rakefile
  docker/
    harpoon.yml

Then your harpoon.yml may look like:

---

folders:
  api_dir: "{config_dir}/.."

images:
  bundled:
    context:
      parent_dir: "{folders.api_dir}"

    commands:
      - FROM some_image_with_ruby_installed

      - RUN apt-get -y install libmysqlclient-dev ruby-dev

      - RUN mkdir /api
      - ADD Gemfile /api/
      - ADD Gemfile.lock /api/

      - WORKDIR /api
      - RUN bundle config --delete path && bundle config --delete without && bundle install

  mysql:
    context:
      parent_dir: "{folders.api_dir}"

    commands:
      - [FROM, "{images.bundled}"]
      - VOLUME shared

      <install mysql>

      ## Expose the database
      - EXPOSE 3306

      - [ADD, {prefix: "/app", get: ["db", "lib", "config", "app", "Rakefile"]}]

      ## Run the migrations
      - RUN (mysqld &) && rake db:create db:migrate

      - CMD cp /app/db/schema.rb /shared && mysqld

  unit_tests:
    context:
      parent_dir: "{folders.api_dir}"

    link:
      - ["{images.mysql}", "dbhost"]

    volumes:
      share_with:
        - "{images.mysql}"

    commands:
      - [FROM, "{images.bundled}"]
      - ADD . /app/

      - CMD cp /shared/schema.rb /app/db && rake

And harpoon will ensure that the bundled image is created before both the mysql and unit_tests images are created, and that when we run the unit_tests container it first creates the mysql container.

Harpoon will also ensure all these containers are cleaned up afterwards. Images stay around because we want to use the awesome caching powers of Docker.

Custom tasks

You can add tasks within your container.

For example:

---

images:
  app:
    commands:
      ...
      - CMD startup_app

     tasks:
      run_app:
        description: "Startup the app"

      run_tests:
        description: Run the unit tests
        options:
          bash: cd /app && rake tests

Each task has no required options but can be configured with action, options , overrides, description and label.

If action or options are not specified then the task will just create the image it’s defined under and run the default command.

If the action is specified and is just a string, then it will call that action and give the image option as the name of this image. The available tasks are those in https://github.com/realestate-com-au/harpoon/blob/master/harpoon/tasks.py with a a_task decorator. For most cases, you just need the run task which is the default value for action.

The tasks defined in these definitions will be shown when you do “harpoon –task list_tasks”.

You may also use extra arbitrary cli options for your tasks with {$@}:

---

images:
  app:
    commands:
      ...
    tasks:
      something:
        options:
          bash: cd /app && ./some_script.sh {$@}

Then say you run harpoon like:

$ harpoon --task something -- --an-option 1

Then it will start up the app container and run:

$ /bin/bash -c 'cd /app && ./some_script.sh --an-option 1'

Because everything that comes after a -- in the argv to harpoon will be available as “$@”.

Linking containers and volumes

You have the following options available:

link

A list of strings that are equivalent to the options you give link for docker cli (https://docs.docker.com/userguide/dockerlinks/#container-linking)

For example:

---

images:
  db:
    commands:
      ...
  app:
    link:
      - ["{images.db.container_name}", "dbhost"]
    commands:
      ...

Will make sure that when you start the app container, it will run the db image in a detached state and there will be an entry in the /etc/hosts of the app container that points dbhost to this db container.

volumes.share_with

This behaves like link in that you specify strings similar to what you would do for the docker cli (https://docs.docker.com/userguide/dockervolumes/#creating-and-mounting-a-data-volume-container)

So something like:

---

images:
  db:
    commands:
      - FROM ubuntu
      - VOLUME /shared
  app:
    volumes:
      share_with:
        - "{images.db}"
    commands:
      ...

Then the app container will share the volumes from the db container.

volumes.mount

This is also specified as string similar to what you do for the docker cli (https://docs.docker.com/userguide/dockervolumes/#data-volumes)

For example:

---

folders:
  app_dir: "{config_root}/../app"

images:
  app:
    volumes:
      mount:
        - "{app_dir}/coverage:/project/app/coverage:rw"

Will mount the coverage directory from the host into /project/app/coverage on the image.

Sometimes you need your dependency container to not be running in a detached container. To make it so a dependency is running in an attached container, you may specify dependency_options:

---

images:
  runner:
    commands:
      ...
      - CMD activator run

  uitest:
    link:
      - ["{images.runner}", "running"]

    dependency_options:
      runner:
        # Typesafe activator run stops in a detached container
        attached: True

    commands:
      ...
      - CMD ./do_a_uitest.sh running:9000

Tests

Install testing deps and run the helpful script:

pip install -e .
pip install -e ".[tests]"
./test.sh