Duplicity backup to S3 for production servers using simple toml file.
Project description
Duplicity Backup to S3
Duplicity backup to S3 for production servers using simple toml file.
License
Free software: Apache Software License 2.0
Features
This is a duplicity command line backup wrapper that will backup to S3 that is using a validated yaml configuration file using modern and awesome CLI patterns. The commands incremental
, list files
, status
, verify
, cleanup
, and remove
are implemented.
The primary use case to build this (yet another one) CLI wrapper for duplicity to be able to deploy the command in production and inject it into a cron.daily
and having a hands-off automated backup of production servers. We use it a KE-works to automate our production server backups.
Dependencies
duplicity
python-boto
to connect to S3
Installation
You can either install this as a system command on any operating system supporting Python 3.5 or later.
To install as global command from PyPI:
python3 -m pip install duplicity-backup-s3
To install for your user only from PyPI
python3 -m pip install --user duplicity-backup-s3
To install from the git repository (latest master branch):
python3 -m pip install git+https://github.com/jberends/duplicity_backup.git#wheel=duplicity_backup_s3
First use
To first use, you need to create a configuration yaml file. You can use the helper command init
for that. Use the built-in help function for your enjoyment.
# help is neigh
duplicity_backup_s3 --help
# and to init the configuration YAML file
duplicity_backup_s3 init
It will drop you a duplcitiy_backyp_s3.yaml
in your current directory. That will look like this:
aws:
AWS_ACCESS_KEY_ID: foobar_aws_key_id # your amazon S3 user that has write right to a backup bucket
AWS_SECRET_ACCESS_KEY: foobar_aws_access_key # your amazon S3 user secret
backuproot: /home # the backup 'root' path. Everything underneath is considered for backup.
excludes:
- "**" # a list of exclude paths. May be '**' to exclude everything except what you include
includes:
- /home/Pictures # a list of includes, which are full paths
- /home/Music
remote:
bucket: '<an_s3_bucket>' # S3 bucket name
path: '__test' # subpath within the bucket
full_if_older_than: 7D # default is incremental, will create full backup every 7Days.
You can alter the configuration file to your liking. The command will check the configuration for its validity and tell you what went wrong and what you need to correct. If you made mistakes, it can be beneficial to checkout the duplicity man page for more information on that topic. However we tried to be as verbose as possible to guide you in the right direction.
First backup
To call your first backup, which is a full one.
duplicity_backup_s3 incr --verbose
# or if the config is somewhere else
duplicity_backup_s3 incr --config /path/to/configuation.yaml
That might take time according to the size of the backup. You can see the volumes being uploaded to the S3 bucket.
To check the backup collection, list and verify the backup you may use:
# collection status
duplicity_backup_s3 status
# list all files
duplicity_backup_s3 list
# verify backup
duplicity_backup_s3 verify
Remove old backups
To remove older backups duplicity provides some commands. We implemented those in the remove
command.
# to remove backups older than 7D
duplicity_backup_s3 remove --older-than 7D
# to remove older backup except the last 4 full backups
duplicity_backup_s3 remove --all-but-n-full 4
Using this as daily backup in a cronjob
To use this in a daily cron job, you can alter the crontab
for the user root
crontab -u root -e
You can alter the crontab in the following way
# Daily backup and remove older backup
7 4 * * * /bin/duplicity_backup_s3 incr --config=/path/to/conf.yaml && /bin/duplicity_backup_s3 remove --older-than 7D --config=/path/to/conf.yaml
# | | | | +- the command to execute
# | | | +--- day of the week (0-6) Sunday=0 (*=every day)
# | | +----- month of the year (*=every month)
# | +------- day of the month (*=every day)
# +--------- hour of the day
#----------- minute in the hour
TODO
- implement appdirs for default configuration file placement
- implement restore for restoring
- test on digitalocean
- If requested migrate
--s3-european-buckets
to configuration file - If requested implement GPG/Encryption capabilities. Possibly reusing code of
kecpkg-tools
to manage certificates. - Initialise itself into
cron.daily
Credits
- This package was inspired by the great work done by the duplicity team, back in the days.
- This package was inspired by the great amount of perl/bash code by the duplicity_backup.sh project.
- This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template. \n\n# Changelog
v0.5.0 (5JUN19)
First initial public release
- commands
incr
,list
,status
,verify
,cleanup
implemented. - added yaml schema check for the configuration file.
v0.2.0 (3JUN19)
internal release
- migrated to command structure. Now offers
incr
andinit
v0.1.0 (3JUN19)
internal release
- First release on PyPI.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for duplicity_backup_s3-0.5.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 175b62f32c097c6ed6b8f787d31cdb6933c3a71e1ac34a1475b5e017da31e1b3 |
|
MD5 | 7a2756613b3d390beb0d31ab389ace60 |
|
BLAKE2b-256 | 780a1490a68bd3d2bd69ea4a864abf8ad6e6c96f0d1d4ff5a9424630b4f6e050 |
Hashes for duplicity_backup_s3-0.5.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 30969b4beaf8e2b1d7c615714858793df5fdefaf784099c0420e3d23c459c22a |
|
MD5 | 6b84130d9893e167cf21ded5b379886c |
|
BLAKE2b-256 | b64432cab37185bf178f978556ca3b37aad47c71d1af11976dbc2301006dac0b |