Duplicity backup to S3 for production servers using simple yaml config file.
Duplicity Backup to S3
Duplicity backup to S3 for production servers using simple yaml file.
Free software: Apache Software License 2.0
This is a duplicity command line backup wrapper that will backup to S3 that is using a validated yaml configuration file using modern and awesome CLI patterns. The commands
init are implemented.
The primary use case to build this (yet another one) CLI wrapper for duplicity, is to be able to deploy the command in production and inject it into a
cron.daily and having a hands-off automated backup of production servers. We use it a KE-works to automate our production server backups to Amazon S3.
You can either install this as a system command on any operating system supporting Python 3.5 or later.
To install as global command from PyPI:
sudo python3 -m pip install duplicity-backup-s3
To install for your user only from PyPI enter the following commmand:
python3 -m pip install --user duplicity-backup-s3
To install from the git repository (latest master branch):
python3 -m pip install --user git+https://github.com/jberends/duplicity_backup.git#wheel=duplicity_backup_s3
To first use, you need to create a configuration yaml file. You can use the helper command
init for that. Use the built-in help function for your enjoyment.
# help is neigh duplicity_backup_s3 --help # and to init the configuration YAML file duplicity_backup_s3 init
It will drop you a
duplicity_backup_s3.yaml in your current directory. That may look like this:
aws: AWS_ACCESS_KEY_ID: foobar_aws_key_id # your amazon S3 user that has write right to a backup bucket AWS_SECRET_ACCESS_KEY: foobar_aws_access_key # your amazon S3 user secret backuproot: /home # the backup 'root' path. Everything underneath is considered for backup. excludes: - "**" # a list of exclude paths. May be '**' to exclude everything except what you include includes: - /home/Pictures # a list of includes, which are full paths - /home/Music remote: bucket: '<an_s3_bucket>' # S3 bucket name path: '__test' # subpath within the bucket full_if_older_than: 7D # default is incremental, will create full backup every 7Days.
You can alter the configuration file to your liking. The command will check the configuration for its validity and tell you what went wrong and what you need to correct. If you made mistakes, it can be beneficial to checkout the duplicity man page for more information on that topic. However we tried to be as verbose as possible to guide you in the right direction.
To perform your first backup, which is a full one, use the following command:
duplicity_backup_s3 incr --verbose # or if the config is somewhere else duplicity_backup_s3 incr --config /path/to/configuation.yaml
That might take time according to the size of the backup. You can see the volumes being uploaded to your configured S3 bucket using the S3 console.
To check the backup collection, list and verify the contents of the backup you may use:
# collection status duplicity_backup_s3 status # list all files duplicity_backup_s3 list # verify backup duplicity_backup_s3 verify
Remove old backups
To remove older backups, duplicity provides some commands. We implemented those in the
# to remove backups older than 7D duplicity_backup_s3 remove --older-than 7D # to remove older backup except the last 4 full backups duplicity_backup_s3 remove --all-but-n-full 4
To restore backup we implemented the
# to restore backups from yesterday to a current directory duplicity_backup_s3 restore --time 1D # to restore specific subdirectory from a specific date/time to a custom directory duplicity_backup_s3 restore --dir specific_subdir \ --time 2020-12-08T22:22:00+01:00 --target ~/a_restoredir
Using this as daily backup in a cronjob
To use this in a daily cron job, you can alter the
crontab for the user
crontab -u root -e
You can alter the crontab in the following way
# Daily backup and remove older backup 7 4 * * * /bin/duplicity_backup_s3 incr --config=/path/to/conf.yaml && /bin/duplicity_backup_s3 remove --older-than 7D --config=/path/to/conf.yaml # | | | | +- the command to execute # | | | +--- day of the week (0-6) Sunday=0 (*=every day) # | | +----- month of the year (*=every month) # | +------- day of the month (*=every day) # +--------- hour of the day #----------- minute in the hour
- [x] implement appdirs for default configuration file placement
- [x] implement restore for restoring
- [ ] test on digitalocean
- [x] If requested migrate
--s3-european-bucketsto configuration file
- [ ] If requested implement GPG/Encryption capabilities. Possibly reusing code of
kecpkg-toolsto manage certificates.
- [ ] Initialise itself into
- This package was inspired by the great work done by the duplicity team, back in the days.
- This package was inspired by the great amount of bash code by the duplicity_backup.sh project.
- This package is thankful on my knees to the great work done by the Authors and contributors behind the Click project, packing tons of CLI awesomeness since 2014.
- This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template. \n\n# Changelog
- added restore command implementation.
- Added appdirs to the setup.py requirements.
Not released to the public.
First production release.
- implemented appdirs, such that the configuration file can be safely placed and located from a known configuration directory on disk.
removecommand to remove collections from the backup target after a specified time. Please consult the
duplicity_backup_s3 remove --helpdocumentation for guidance.
initcommand to initialise the configuration in an interactive fashion for users.
- various development improvements, eg Github actions for testing and publishing to PyPI; removed all flake8 warnings and pydocstyle errors. Added pre-commit hooks. Code is A++ grade now.
First initial public release.
- added yaml schema check for the configuration file.
- migrated to command structure. Now offers
- First release on PyPI.
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size duplicity_backup_s3-1.1.0-py2.py3-none-any.whl (22.8 kB)||File type Wheel||Python version py2.py3||Upload date||Hashes View|
|Filename, size duplicity_backup_s3-1.1.0.tar.gz (21.7 kB)||File type Source||Python version None||Upload date||Hashes View|
Hashes for duplicity_backup_s3-1.1.0-py2.py3-none-any.whl
Hashes for duplicity_backup_s3-1.1.0.tar.gz