Docker Shaper

This is a spin-off package for the docker-shaper tool, which can be used to monitor Docker stuff like containers, images and volumes and automatically enforce certain cleanup-rules.

Installation

[<PYTHON> -m] pip[3] install [--user] [--upgrade] docker-shaper

Usage

The tool will start an interactive terminal UI (TUI).

poetry run docker-shaper

Attach to build nodes

The application is running in a tmux on the build nodes. The following command can be used to attach to the session:

ssh -t <build node> "su jenkins -Pc 'tmux attach-session -t docker-shaper'"

Development & Contribution

Setup

For active development you need to have poetry and pre-commit installed

python3 -m pip install --upgrade --user poetry pre-commit
git clone ssh://review.lan.tribe29.com:29418/docker_shaper
cd docker_shaper
pre-commit install
# if you need a specific version of Python inside your dev environment
poetry env use ~/.pyenv/versions/3.8.10/bin/python3
poetry install

Workflow

Create a new changelog snippet. If no new snippets is found on a merged change no new release will be built and published.

If the change is based on a Jira ticket, use the Jira ticket name as snippet name otherwise use a unique name.

poetry run \
    changelog-generator \
    create .snippets/CMK-21130.md

After committing the snippet a changelog can be generated locally. For CI usage the --in-place flag is recommended to use as it will update the existing changelog with the collected snippets. For local usage remember to reset the changelog file before a second run, as the version would be updated recursively due to the way the changelog generator is working. It extracts the latest version from the changelog file and puts the found snippets on top.

Future changes to the changelog are ignored by

git update-index --assume-unchanged changelog.md

poetry run \
    changelog-generator \
    changelog changelog.md \
    --snippets=.snippets \
    --in-place \
    --version-reference="https://review.lan.tribe29.com/gitweb?p=docker_shaper.git;a=tag;h=refs/tags/"

Update the version of the project in all required files by calling

poetry run \
    changelog2version \
    --changelog_file changelog.md \
    --version_file cmk_dev/version.py \
    --version_file_type py \
    --additional_version_info="-rc42+$(git rev-parse HEAD)" \
    --print \
    | jq -r .info.version

• modify and check commits via pre-commit run --all-files

• after work is done locally:

  • update dependencies before/with a new release

poetry lock

• build and check package locally

poetry build && \
twine check dist/* &&
python3 -m pip uninstall -y docker-shaper && \
python3 -m pip install --user dist/docker_shaper-$(grep -E "^version.?=" pyproject.toml | cut -d '"' -f 2)-py3-none-any.whl

• commit, push, review and merge the changes, see docker_shaper/+/92759/

git add ...
git commit -m "bump version, update dependencies"
# merge

• test deployed packages from test.pypi.org. The extra index URL is required to get those dependencies from pypi.org which are not available from test.pypi.org

pip install --no-cache-dir \
    -i https://test.pypi.org/simple/ \
    --extra-index-url https://pypi.org/simple \
    docker-shaper==<VERSION_WITH_RC>

• finally merge the changes and let Jenkins create the release tag and deployment

Troubleshooting

Publishing fails with HTTP Error 403

As a workaround which uses the token directly, you can do something like this:

poetry publish --username __token__ --password=<token here>

Todo

• Fix: stuck auto-reload: was: postpone=True + monitoring log

• Fix: crawl-images should fix parents not having them listed

• [-] Fix: stuck crawl images (too many async requests, does not happen in production)

• Fix: '677aff0727' could not be removed: DockerError(404, 'No such image: 677aff0727:latest')

• Fix: tried to remove container 4f5fb0848c unknown to us

• Fix: Crashes (see runlog)

• Image update message only if needed

• New TUI

  • make ongoing progress visible

• review log levels (too verbose)

• answer https://stackoverflow.com/questions/32723111

• installable via pip install

• Quart interface (instead of flask)

• auto-apply changes to source and configuration files

• outsourced config file

• bring in features of former dgcd

• bring in features of former dockermon

• untag certain tags

• container cleanup

• Fix none image lookup

• Exceptions to messages

• Clip

• Increase/decrease logging via web / signal

• Link: cleanup (images/containers) now

• Add volumes list (with recent owners)

• Containers: Store CPU / Memory usage over time

• Containers: store history

• Persist messages

• Remove old message/container logs

• Show different color for unmatched images

• Warn about use of unpinned / upstream images

• Handle 'build cache objects' (found on system prune)

• Bring in volume monitoring: which volumes have been created and used by which containers?

• Containers: show total CPU usage

• Containers: list volumes

• Images: list parents / children

• Volumes: list usage

• Instructions to readme

• List unmatched / overmatched tags

• Links to delete / remove

• Links to jobs

• Link: inspect

• Graph: cpu / containers (idle/up)

• Authenticate (at least if we can modify behavior, like stopping/removing images/containers)

Knowledge

(just misc links to articles that helped me out)

• How to delete docker images from Nexus Repository 3
• Showing Text Box On Hover (In Table)
• Beautiful Interactive Tables for your Flask Templates
• https://github.com/torfsen/python-systemd-tutorial
• https://www.digitalocean.com/community/tutorials/how-to-use-templates-in-a-flask-application
• https://stackoverflow.com/questions/49957034/live-updating-dynamic-variable-on-html-with-flask
• https://pgjones.gitlab.io/quart/how_to_guides/templating.html

Logging

• https://pgjones.gitlab.io/hypercorn/how_to_guides/logging.html#how-to-log