YAML driven docker playbook
Project description
docker-playbook
YAML driven (nvidia) docker playbook
Requirements
- Python3.x
- Nvidia-docker2 (Make sure you can
nvidia-smi
anddocker images
) - PyYAML>=3.12 (both host and docker image)
Template Dockerfile is provided at the bottom, which is optional
though recommended because of its security enhancements and useful features.
Usage example
./playbook.py sample.yml
Features
- Runs with
nvidia-docker2
mostly in non-interactive mode (so if your script waits for input by accident, it gets signal and dies) - Runs (whitelisted) steps in sequential manner
- Context deduction: each step can run in different docker images or host
- Full access to host network services in non-interactive mode
- Simple step function siginature
step_blah(ctx)
- easy to extend - Minimal command line arguments
./playbook.py some.yml
is all you do - Single source file & low dependency (only PyYAML is not in standard library, replacable but strongly recommend)
- Colorful logging for readability
How to add steps?
- Copy
library.steps
module from this repository to start developing your ownlibrary.steps
module for this script to import. - Add a function
def step_something(ctx)
. Current execution context is inctx
as dict. Keys are proxied to attributes to save a lot of brakets and quotes. You may uselogger
to print info.
def step_something(ctx):
logger.info(ctx.other_vars)
- Whitelist your step in
library.steps.whitlist
without thestep_
prefix. - Add an entry to your YAML file in
steps
whereaction
is the step function name without prefix:
steps:
- name: Some message here
action: something
other_vars: goes_to_ctx
Context deduction rules
docker overrides > step config > global config > parser defualts > context initialization
How to specify docker environment?
You may add a default docker environment.
And use docker_overrides
to change context variables when docker is in use.
docker:
image: aleozlx/tkstack2:latest
runtime: nvidia
gui: False
ports:
- 6006:6006
volumes:
- /tmp:/workspace/build
- /mnt/datasets:/workspace/datasets
docker_overrides:
storage: /workspace/datasets
steps:
- name: Some message here
action: something
storage: /mnt/datasets
Or override the docker environment completely per step
docker:
# ...
steps:
- name: Some message here
action: something
storage: /mnt/datasets
docker:
image: aleozlx/tkstack2:latest
runtime: nvidia
volumes:
- /tmp:/workspace/build
docker_overrides:
storage: /workspace/datasets
Or use the host
docker:
# ...
steps:
- name: Some message here
action: something
storage: /mnt/datasets
docker: null
Note: When a docker environment is present, the playbook starts docker accordingly and resumes itself inside docker to reuse many of the playbooks' features, so that context deduction and logging have consistent behavior. (To do that, you code containing the playbook will have to be mounted read-only. This is automatic.) That's why PyYAML is also required inside your docker image. Consider you are keeping the full operationalizing mechanism with a silght dependency constraint.
Security assumptions (or gotchas)
Host file system: volumes specified in your playbook will be mounted RW. Although your code will be mounted RO automatically. Playbook assumes that you use a docker image that uses non-root user (like recommended) whose uid:gid hopefully maps to you on host system.
Network: network services inside docker are not isolated from host in non-interactive mode to provide convenient access to host databases etc. Playbook assumes whatever you are operationalizing is trusted and that your host should have a proper set of INPUT rules, and that services inside docker should be protected by an independent firewall if necessary.
X11: the recommended docker image does intend to provide isolated X11 access by creating non-root user that presumably maps to you on host and your X authentication files are natually mounted with proper permissions already in place. But the uid in that image is just a guess so on a host shared by many users, this is probably no good if it even works. (Docker image needs to be build in a better way)
Playbook itself: the playbook itself is obviously a very capable shell program. It has execute permission by default for convenience (not necessarily good idea). It is based on a simple whitelist to allow any actions to be executed. The docker shell provided shouldn't be able to be spawned on host. But all these are best-effort, no promises.
At the end of day, this is an under-2^7-line script that is supposed to just drive other things and make them work. Composition over inheritance principle makes things simpler and useful.
Template Dockerfile
https://gitlab-p01.cgi.missouri.edu/ExternalUsers/AlexYang_Fun/tkstack2/blob/master/src/Dockerfile
Features
- Non-root with passwordless sudo
- Low maintenance: base image is
tensorflow/tensorflow:1.8.0-devel-gpu-py3
- Numpy + Scipy linked against libblas + liblapack
- Isolated X11 GUI support
- Details at https://gitlab-p01.cgi.missouri.edu/ExternalUsers/AlexYang_Fun/tkstack2
License
Licensed under either of
- Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
- MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)
at your option.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for docker_playbook-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7999c0049aab98683d682cf3558d8fb5877774265ff70284c06fc4b69f8f78a3 |
|
MD5 | 93e7bd3771f7aa5bad1f68cd36ccc122 |
|
BLAKE2b-256 | e7190a34f496fd6ebc9311a3da466137626cff77f3a7bbf29edad1457728f5ef |