docker-shaper 2.0.4

Keeps Docker resources in shape based on rules and usage

Docker Shaper

This is a spin-off package for the docker-shaper tool, which can be used to monitor Docker stuff like containers, images and volumes and automatically enforce certain cleanup-rules.

Installation

[<PYTHON> -m] pip[3] install [--user] [--upgrade] docker-shaper

Usage

The tool will start an interactive terminal UI (TUI).

poetry run docker-shaper

Attach to build nodes

The application is running in a tmux on the build nodes. The following command can be used to attach to the session:

ssh -t <build node> "su jenkins -Pc 'tmux attach-session -t docker-shaper'"

Development & Contribution

Prerequisites

• Python 3.8.10+ (e.g. via pyenv)
• poetry and pre-commit python3 -m pip install --upgrade --user poetry pre-commit

git clone ssh://review.lan.tribe29.com:29418/docker_shaper
cd docker_shaper
pre-commit install
# if you need a specific version of Python inside your dev environment
poetry env use ~/.pyenv/versions/3.8.10/bin/python3
poetry install

Workflow

• (once and only for publishing to PyPi) Get token on PyPi.org
• (once and only for publishing to PyPi) poetry config pypi-token.checkmk pypi-<LONG-STRING> (will write to ~/.config/pypoetry/auth.toml)
• modify and check commits via pre-commit
• after work is done locally:
  • adapt version in pyproject.toml with

# see section Setup
# poetry self add poetry-bumpversion
poetry version [patch, minor, major] [--dry-run]

• update dependencies before/with a new release

poetry lock

• build and check package locally

poetry build && \
twine check dist/* &&
python3 -m pip uninstall -y docker_shaper && \
python3 -m pip install --user dist/docker_shaper-$(grep -E "^version.?=" pyproject.toml | cut -d '"' -f 2)-py3-none-any.whl

• commit, push, review and merge the changes, see checkmk_dev_tools/+/88632/

git add ...
git commit -m "bump version, update dependencies"
# merge

• publish new package version

poetry publish --build [--repository docker_shaper]
git tag -a v<VERSION> <MERGE_COMMIT> -m "v<VERSION>"

Todo

• Fix: stuck auto-reload: was: postpone=True + monitoring log

• Fix: crawl-images should fix parents not having them listed

• [-] Fix: stuck crawl images (too many async requests, does not happen in production)

• Fix: '677aff0727' could not be removed: DockerError(404, 'No such image: 677aff0727:latest')

• Fix: tried to remove container 4f5fb0848c unknown to us

• Fix: Crashes (see runlog)

• Image update message only if needed

• New TUI

  • make ongoing progress visible

• review log levels (too verbose)

• answer https://stackoverflow.com/questions/32723111

• installable via pip install

• Quart interface (instead of flask)

• auto-apply changes to source and configuration files

• outsourced config file

• bring in features of former dgcd

• bring in features of former dockermon

• untag certain tags

• container cleanup

• Fix none image lookup

• Exceptions to messages

• Clip

• Increase/decrease logging via web / signal

• Link: cleanup (images/containers) now

• Add volumes list (with recent owners)

• Containers: Store CPU / Memory usage over time

• Containers: store history

• Persist messages

• Remove old message/container logs

• Show different color for unmatched images

• Warn about use of unpinned / upstream images

• Handle 'build cache objects' (found on system prune)

• Bring in volume monitoring: which volumes have been created and used by which containers?

• Containers: show total CPU usage

• Containers: list volumes

• Images: list parents / children

• Volumes: list usage

• Instructions to readme

• List unmatched / overmatched tags

• Links to delete / remove

• Links to jobs

• Link: inspect

• Graph: cpu / containers (idle/up)

• Authenticate (at least if we can modify behavior, like stopping/removing images/containers)

Knowledge

(just misc links to articles that helped me out)

• How to delete docker images from Nexus Repository 3
• Showing Text Box On Hover (In Table)
• Beautiful Interactive Tables for your Flask Templates
• https://github.com/torfsen/python-systemd-tutorial
• https://www.digitalocean.com/community/tutorials/how-to-use-templates-in-a-flask-application
• https://stackoverflow.com/questions/49957034/live-updating-dynamic-variable-on-html-with-flask
• https://pgjones.gitlab.io/quart/how_to_guides/templating.html

Logging

• https://pgjones.gitlab.io/hypercorn/how_to_guides/logging.html#how-to-log