Docker volume dump
A tool to help archive data from container running in a docker container.
- Database backup support for Postgresql, Mysql/Mariadb: it create a backup in the container through docker API, then retrieves data to save it in a proper place.
rsyncvolume data: it introspect container and rsync all declaredlocal volumeandbindmount points.
note: at the moment it's not possible to backup a database and rsync those volumes.
Usage
Using docker
docker run registry.gitlab.com/micro-entreprise/docker-volume-dump archive --help
For instance if you want to create dumps from different postgresql container in a docker swarm environment this would looks likes:
docker service create \
-d \
--mode global \
--name pgsql-dumps \
--restart-condition none \
--mount type=bind,src=/path-to-dump-storage/,dst=/backups \
--mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \
--mount type=bind,src=/path-to-config-directory/,dst=/etc/archiver \
--network sentry_internal \
--with-registry-auth registry.gitlab.com/micro-entreprise/docker-volume-dump \
archive -f /etc/archiver/logging.json -r -s '{"label": ["docker-volume-dump.project='$PROJECT'","docker-volume-dump.environment='$ENVIRONMENT'"]}'
This script require access to the docker daemon to query docker labels.
It must be launched on each host using --mode global.
Using python
pip install docker-volume-dump
archive --help
Configuration
The main idea is to tell the container how to manage its backups using docker labels, you can set following labels.
You can use a custom prefix if you like so using
ARCHIVER_LABEL_PREFIXenv variable. For instance if you setARCHIVER_LABEL_PREFIX=archiverit would expect labels likesarchiver.isactiveinstead of the defaultdocker-volume-dump.isactive.
-
docker-volume-dump.driver: Optional (
pgsqlby default) kind of data to dump (could be one ofpgsql,mysql,rsync). Only one value supported by container.Note:
mysqldriver is working for mariadb as well -
docker-volume-dump.isactive: Takes no value. Used by the default selector to determine if the Archiver backups are enabled on the container.
-
docker-volume-dump.project: A project name (the container name if not set)
-
docker-volume-dump.environment: An environment (staging, production, ...)
-
docker-volume-dump.prefix: A prefix for the dump file
Database labels (pgsql/mysql)
- docker-volume-dump.dbname: Required, the database name to dump.
- docker-volume-dump.username: DB role used to dump the database.
Required with
mysql, fallback topostgresif not set forpgsql. - docker-volume-dump.password: DB password used to dump the db.
Required with
mysql, not use withpgsqldriver
This will generate a file in a tree likes
<project>/[<environment>/]<prefix><dbname><date>
rsync labels
note: I've chosen to rsync data first because tar/gzip rdiff-backup failed to compress data if other programs write content at the same time. My flow is something like data -> rsync -> rdiff-backup -> tar/gzip -> s3
- docker-volume-dump.rsync-params: params to add to rsync command
predifined (hardcoded) params are
rsync -avhP --delete - docker-volume-dump.ro-volumes: If set to one of those values
["yes", "true", "t", "1"](not case sensitive) rsync read-only volumes as well for the given container.
This will generate a director per declared volume/bind
<project>/[<environment>/][<prefix>]<computed folder name>
Computed folder name is based on the path inside the container where
slash (/) are replaced per dash (-). ie:
- Project: test
- Environment: dev
- prefix:
rsynced_ - volume declare as
-v /some/path/on/host/machine:/path/to/data - volume declare as
-v named-volume:/other/data
note: if archiver is running inside a container where host filesystem is mounted in
/hostfsmind to use--host-fs /hostfsoption.
Would produce:
- /backups/test/dev/rsynced_path-to-data
- /backups/test/dev/rsynced_other-data
Roadmap
- pgsql/mysql: support multiple base per DBMS
- pgsql/mysql: if dbname is not provide retreive db list to detect the DB to dump
- wondering if the way use to query docker labels is compliant with k8s
- In swarm investigate to launch only once container (not on each hosts)
Metadata
Release files for docker-volume-dump 0.5.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| docker-volume-dump-0.5.1.tar.gz | 9.8 kB | Without provenance | Details for docker-volume-dump-0.5.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| docker_volume_dump-0.5.1-py3-none-any.whl | Python 3 | none | any | Details for docker_volume_dump-0.5.1-py3-none-any.whl |
Total release size: 18.7 kB
Release files / docker-volume-dump-0.5.1.tar.gz
| Download URL | docker-volume-dump-0.5.1.tar.gz |
|---|---|
| Size | 9.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3a7b2f206861b318c15716a22b1b16d801974bd727285d4e1a9a2acbfaf216a6
|
|
BLAKE2b-256 checksum How to use checksums |
8706c37cc0dcb36d37a5fb78946d4b79bad441087d18e0f593a5fd43dc7cb749
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/4.5.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.1 CPython/3.8.10
|
Release files / docker_volume_dump-0.5.1-py3-none-any.whl
| Download URL | docker_volume_dump-0.5.1-py3-none-any.whl |
|---|---|
| Size | 8.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e65459a2a8ca749f32188bc41aed5c25266ea418490a6b518c375c7c4967a792
|
|
BLAKE2b-256 checksum How to use checksums |
82e6d7179e0f88b84a79a24d7e5a42a738f39127153235e8591ba4b8a26c163a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/4.5.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.1 CPython/3.8.10
|
Security for docker-volume-dump 0.5.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.