monit-docker project

monit-docker is a free and open-source, we develop it to monitor container status or resources and execute some commands inside containers or manage containers with dockerd, for example:

• reload php-fpm if memory usage is too high
• reload php-fpm if no free space in /dev/shm
• restart container if status is not running
• remove all containers

Table of contents

1. Quickstart
2. Installation
3. Environment variables
4. Sub-command: monit
   1. Basic commands
   2. Advanced commands
   3. Container informations with exit codes
   4. monit-docker with M/Monit
5. Sub-command: stats
   1. Basic commands
   2. Advanced commands

Quickstart

Using monit-docker in Docker with crond

docker-compose up -d

See docker-compose.yml and MONIT_DOCKER_CRONS environment variable to configure commands.

Installation

pip install monit-docker

Environment variables

Sub-command: monit

Basic commands

Restart containers with name starts with foo if memory usage percentage > 60% or cpu usage percentage > 90%:

monit-docker --name 'foo*' monit --cmd-if 'mem_percent > 60 ? restart' --cmd-if 'cpu_percent > 90 ? restart'

Stop containers with name starts with bar or foo and if cpu usage percentage greater than 60% and less than 70%:

monit-docker --name 'bar*' --name 'foo*' monit --cmd-if '60 > cpu_percent < 70 ? stop'

Kill containers with name starts with bar and status equal to paused or running:

monit-docker --name 'bar*' monit --cmd-if 'status in (paused,running) ? kill'

You can also use status argument, for example, restart containers with status paused or exited:

monit-docker -s paused -s exited monit --cmd 'restart'

Generate containers pidfile:

monit-docker monit --rsc pid

PHP-FPM graceful reload

The PHP-FPM examples below (including the Monit configuration) use kill -USR2 1 inside the container. They require the PHP-FPM master process to be PID 1. According to the PHP-FPM manual, SIGUSR2 gracefully reloads the workers and reloads the FPM configuration and binary.

If PID 1 is a supervisor or a wrapper script, target the actual PHP-FPM master PID inside the container instead. Obtain it from the PID file configured for your PHP-FPM installation, or use your supervisor's documented reload command. Do not target an arbitrary worker PID.

For a PHP-FPM master running as PID 1:

monit-docker --name foo_php_fpm monit --cmd '(kill -USR2 1)'

The Docker SDK reload action only refreshes container metadata; it does not send SIGUSR2 or reload PHP-FPM.

Reload php-fpm in container with image name contains /php-fpm/ if memory usage greater than 100 MiB:

monit-docker --image '*/php-fpm/*' monit --cmd-if 'mem_usage > 100 MiB ? (kill -USR2 1)'

Reload php-fpm in container with image name contains /php-fpm/ if /dev/shm percentage usage greater than 80%:

monit-docker --image '*/php-fpm/*' monit --cmd '(bash -c "[ $(df /dev/shm | sed \"s/\%//;\$!d\" | awk \"{print \$5}\") -gt 80 ] && kill -USR2 1")'

Advanced commands with configuration file or environment variable MONIT_DOCKER_CONFIG

Run commands with aliases declared in configuration file (e.g.: monit-docker.yml.example):

Restart container id 4c01db0b339c if condition alias @status_not_running is true:

monit-docker --id 4c01db0b339c monit --cmd-if '@status_not_running ? restart'

Execute commands alias @start_pause containers with name starts with foo if condition alias @status_not_running is true:

monit-docker --name 'foo*' monit --cmd-if '@status_not_running ? @start_pause'

Remove force container group php if status is equal to running:

monit-docker --ctn-group php monit --cmd-if 'status == running ? @remove_force'

Restart containers group nodejs if memory usage percentage > 10% and cpu usage percentage > 60%:

monit-docker --ctn-group nodejs monit --cmd-if '@mem_gt_10pct_and_cpu_gt_60pct ? restart'

Remove force all containers:

monit-docker monit --cmd '@remove_force'

Container informations with exit codes

Container status

Run command below to get status with exit code for container named foo_php_fpm:

monit-docker --name foo_php_fpm monit --rsc status

An error occurred if exit code is greater than 100.

Container CPU usage percentage

Run command below to get CPU usage percentage with exit code for container named foo_php_fpm:

monit-docker --name foo_php_fpm monit --rsc cpu_percent

An error occurred if exit code is greater than 100.

CPU percentages returned as exit codes are capped at 100 to avoid collisions with error codes and Unix exit-code overflow. The stats sub-command and conditional rules retain the raw CPU percentage, which may exceed 100 on multi-core hosts.

Container memory usage percentage

Run command below to get memory usage percentage with exit code for container named foo_php_fpm:

monit-docker --name foo_php_fpm monit --rsc mem_percent

An error occurred if exit code is greater than 100.

monit-docker with M/Monit

We can also monitoring containers cpu_percent and mem_percent resources with M/Monit.

Configuration examples

check program docker.foo_php_fpm.status with path "/usr/bin/monit-docker --name foo_php_fpm monit --rsc status"
    group monit-docker
    if status = 114 for 2 cycles then alert # container not found
    if status != 0 for 2 cycles then exec "/usr/bin/monit-docker --name foo_php_fpm monit --cmd restart" # container not running

check program docker.foo_php_fpm.cpu with path "/usr/bin/monit-docker -s running --name foo_php_fpm monit --rsc cpu_percent"
    group monit-docker
    if status > 100 for 2 cycles then alert
    if status > 70 for 2 cycles then alert
    if status > 80 for 4 cycles then exec "/usr/bin/monit-docker --name foo_php_fpm monit --cmd '(kill -USR2 1)'"

check program docker.foo_php_fpm.mem with path "/usr/bin/monit-docker -s running --name foo_php_fpm monit --rsc mem_percent"
    group monit-docker
    if status > 100 for 2 cycles then alert
    if status > 70 for 2 cycles then alert
    if status > 80 for 4 cycles then exec "/usr/bin/monit-docker --name foo_php_fpm monit --cmd '(kill -USR2 1)'"

check process docker.foo_php_fpm.pid with pidfile /run/monit-docker/foo_php_fpm.pid
    group monit-docker
    if changed pid then alert

Sub-command: stats

Basic commands

Get all resources statistics for all containers in json format:

monit-docker stats --output json

{
  "flamboyant_chaplygin": {
    "status": "running",
    "mem_percent": 0.03,
    "net_tx": "0.0 B",
    "cpu_percent": 0,
    "mem_usage": "2.52 MiB",
    "io_read": "3.5 MB",
    "io_write": "0.0 B",
    "net_rx": "25.2 kB",
    "mem_limit": "7.27 GiB",
    "pid": "3943"
  }
}
{
  "practical_proskuriakova": {
    "status": "running",
    "mem_percent": 0.04,
    "net_tx": "0.0 B",
    "cpu_percent": 0,
    "mem_usage": "2.61 MiB",
    "io_read": "24.6 kB",
    "io_write": "0.0 B",
    "net_rx": "25.0 kB",
    "mem_limit": "7.27 GiB",
    "pid": "3990"
  }
}

Get all resources statistics for all containers in text format:

monit-docker stats --output text

flamboyant_chaplygin|mem_usage:2.52 MiB|mem_limit:7.27 GiB|mem_percent:0.03|cpu_percent:0.0|io_read:3.5 MB|io_write:0.0 B|net_tx:0.0 B|net_rx:43.5 kB|status:running
practical_proskuriakova|mem_usage:2.61 MiB|mem_limit:7.27 GiB|mem_percent:0.04|cpu_percent:0.0|io_read:24.6 kB|io_write:0.0 B|net_tx:0.0 B|net_rx:43.3 kB|status:running

Advanced commands with configuration file or environment variable MONIT_DOCKER_CONFIG

Get status and memory usage for group nodejs:

monit-docker --ctn-group nodejs stats --rsc status --rsc mem_usage

Action failures and command behavior

If a command fails, monit-docker exits with code 116 and stops the current invocation; later commands and containers are not processed. Commands inside containers must complete successfully (exit code 0). Detached or streaming exec aliases do not supply a completion status and are not supported as successful monitored actions.

Propagate a container command's exit code

With monit --propagate-exit-code, a failed synchronous command executed inside a container returns its own exit code instead of 116:

monit-docker --name my-container monit --propagate-exit-code \
  --cmd '(sh -c "exit 42")'
echo $? # 42

The option is available with monit --cmd and monit --cmd-if, including command aliases. It cannot be used with stats or resource-only checks.

Execution stops at the first failure. With several commands or containers, the first failing command's code is returned; later actions are not attempted. The existing execution order is preserved: rules using only PID/status run before rules requiring measurements, for each container in Docker's listing order. Use an exact --name or --id selector when checking one service.

If selected containers exist but no condition matches, the result is 0 because no action failed. No matching container still returns 114. Detached/streaming commands and invalid or unavailable exit codes remain errors (116); their statuses are not propagated or wrapped. Docker lifecycle actions such as restart retain their existing failure behavior.

For a Monit program check, the default nonzero code already supports a generic if status != 0 alert. Enable propagation when different command statuses need different handling, for example a script using 2 for a critical result:

check program docker.my_container.health with path "/usr/bin/monit-docker --name my-container monit --propagate-exit-code --cmd '(/usr/local/bin/check-health)'"
    if status = 2 then alert

Propagated codes may overlap with monit-docker's own error codes. Consult the logs to distinguish a command status from an agent error when the numbers match. The flag is opt-in, so existing integrations keep returning 116 for failed actions.

An unknown --ctn-group is a configuration error (110), including when no groups are configured. Commands already evaluated before resource collection are not evaluated again after collection.

reload refreshes the Docker SDK object's metadata only; it does not reload application workers or configuration. For PHP-FPM, use (kill -USR2 1) only when its master is PID 1 inside the container, as explained in PHP-FPM graceful reload. Otherwise, send SIGUSR2 to the actual PHP-FPM master PID.

Commands inside parentheses use Docker exec, without an implicit shell. For redirections, pipes or shell expansion, explicitly use a shell, for example (sh -c "echo foo > /tmp/bar").

Development

The codebase is being separated into a transport-neutral monitoring core and thin delivery interfaces. See Architecture for the dependency rules, compatibility guarantees, and Community/control-plane boundary.

Install the dependencies and run the regression tests with Python 3:

python -m pip install -r requirements.txt
python -m unittest discover -s tests -v

Build the checked-out source with docker build -t monit-docker:local .. The Dockerfile installs this checkout in a virtual environment instead of fetching the published monit-docker package.

Lightweight cron mode

Run one cycle with a process lock and persistent cooldowns, without a server:

monit-docker --name 'web*' cron --state-file /var/lib/monit-docker/web.json \
  --cooldown 300 --dry-run --cmd-if 'mem_percent > 90 ? restart'

Review the JSON decisions, then remove --dry-run to execute eligible actions. The default cooldown is five minutes per rule and container. A busy job exits with 117; invalid or unwritable state exits with 118. Existing monit and stats commands retain their behavior. monit --dry-run --cmd ... also previews actions. See cron setup, scheduling, state and failure semantics.

Continuous monitoring, Prometheus and Grafana

monit-docker --name 'web*' serve --interval 30

The read-only HTTP listener defaults to 127.0.0.1:9808: /healthz, /readyz, /v1/status and /metrics. Requests read the latest completed cycle from memory. Metrics are not persisted locally; Prometheus stores history. Optional remediation rules reuse cron locking and persistent cooldowns.

See serve usage and API, the complete metrics reference, and Grafana setup. An importable Grafana dashboard includes agent health, CPU, memory, network, block I/O and action decisions.

Docker Hub and PyPI releases

Merging a new stable version into master builds and tests the Docker image and Python distributions, creates the vX.Y.Z tag, then publishes decryptus/monit-docker:X.Y.Z, decryptus/monit-docker:vX.Y.Z and the Python package on PyPI. Manual stable tag pushes are also supported. Pull requests validate without publishing; Docker Hub's latest is not updated. See the Docker Hub setup for DOCKERHUB_TOKEN and the PyPI setup for password-free Trusted Publishing.