Skip to main content

Dup-composer is a front-end script for Duplicity, that lets you define your backups in a configuration file and execute them in a simple way.

Project description

Dup-composer

Dup-composer is a front-end script for Duplicity, that lets you define your backups in a configuration file and execute them in a simple way.

This is work in progress, hence it is not ready for production use yet.

To get started:

  • Download Dup-composer
  • Create your YAML configuration file and define your backup groups configuration in this file.
  • Execute dupcomp.py, define the groups you wish to execute and whether you want to do a backup or restore.

You can find more detail for each step in its respective section.

Requirements - might change any time in the near future

  • Duplicity 0.7 installed and the duplicity script on your PATH.
  • Python 3.5
  • PyYAML 5.1

Download

A reasonable process of building and installing this tool into production is work in progress. Deploying from PyPI is the direction I am planning to take.

Configuration

The configuration should follow YAML 1.1 syntax.

Let's start at the top, with the list of backup groups - the three dots (...) are placeholders for child and scalar nodes:

backup_groups:
  my_first_backup_group:
    ...
  my_second_backup_group:
    ...

The parent node of the groups is called backup_groups, which is currently the root of the configuration structure, but further configuration nodes might be added on the top level in the future.

Backup group

For each backup group, you have to have the following structure in place:

my_first_backup_group:
  encryption:
    ...
  backup_provider:
    ...    
  backup_file_prefixes:
    ...
  volume_size: ...
  sources:
    ...

The encryption node (mandatory) is the parent of the encryption related configuration, the children of backup_provider(mandatory) specify all the provider related properties. backup_file_prefixes is optional and contains the child nodes for archive, signature and manifest file prefix configuration. The volumes_size (mandatory) node determines the size of the backup archive file chunks in MBs.

Encryption

There are primarily two ways to set up the encryption node at the moment.

Encryption is turned off:

encryption:
  enabled: no

Encryption is turned on:

encryption:
  enabled: yes
  gpg_key: 123456789ABCDEFF
  gpg_passphrase: examplepassphrase123

If the enabled node is set to no, encryption is disabled, there is no need to configure the gpg_key and gpg_passphrase nodes. When encryption is enabled however, they are mandatory. This key will be used both for signing and encrypting the backup data.

Keyring support: The value of gpg_passphrase can also be read from a keyring. See the keyring support document for details on setting up the keyring to use with Dup-composer. Once the keyring has been set up, you can specify the passphrase in the following format: gpg_passphrase: ['service_name_in_the_keyring', 'account_name_in_the_keyring'].

Backup provider

The backup_provider configuration to be configured largely depends on the type of the provider, determined by the URL scheme:

backup_provider:
  url: file://

This configuration sets Dup-composer up to save the backup files on the local filesystem. There is no need to specify a concrete path here, as that will be determined by the sources section of the configuration. The URL will just set the context for those paths.

For a remote SCP backup, you need a slightly different configuration:

backup_provider:
  url: scp://myscpuser@host.example.com/
  password: examplepassword123

In this case, you need to specify the username of the remote SCP host in the first part of the SCP URL, which is what you would do using Duplicity directly as well. Use the password node to specify the password.

Keyring support: The value of password can also be read from a keyring. See the keyring support document for details on setting up the keyring to use with Dup-composer. Once the keyring has been set up, you can specify the password in the following format: password: ['service_name_in_the_keyring', 'account_name_in_the_keyring'].

Finally, you have to configure AWS S3 like this:

backup_provider:
  url: s3://s3.sa-east-1.amazonaws.com/my-backup-bucket
  aws_access_key: EXAMPLEACCESSKEY
  aws_secret_key: ExAmPlESeCrEtKeY

The S3 bucket URL is configured as the url node value, while aws_access_key and aws_secret_key need to contain your AWS generated keys for the bucket. Like with the rest of the providers, the actual path, folder, within the bucket shouldn't be added to the URL.

Keyring support: The value of aws_secret_key can also be read from a keyring. See the keyring support document for details on setting up the keyring to use with Dup-composer. Once the keyring has been set up, you can specify the secret key in the following format: aws_secret_key: ['service_name_in_the_keyring', 'account_name_in_the_keyring'].

Backup file prefixes

The next feature comes handy if you want to prefix the generated backup file names in a specific way. I use this to set up bucket rules in S3, that move my archive files to Glacier. Here is an example of the configuration:

backup_file_prefixes:
  manifest: manifest_
  archive: archive_
  signature: signature_

The prefixes can be specifically set up for each file type generated at the backup location. Set these up as needed; you can leave the backup_file_prefixes node out altogether, if you don't need this feature.

Volumes

The volume_size node is rather simple: a number should be given as its value; this determines the archive size in megabytes.

Sources

Under the sources node in the configuration hierarchy, you can specify a list of locations (paths) you want to back up, where to back them up and where the restored data should go. You can set up multiple sources within a single group. Here is an example set of two sources configured:

sources:
  /var/www/html:
    backup_path: /root/backups
    restore_path: /root/restored
  /home/tommy:
    backup_path: /home/bkup/my-laptop-backups
    restore_path: /root/restored-from-backup

The source child nodes /var/www/html and /home/tommy determine the directory you want to back up, and backup_path prescribes the location the backup files will be saved to. In practice, the value of backup_path will be appended to the value of the provider url node discussed earlier; hence these two fragments give the true backup location. restore_path is not used during the backup step, but specifying it is mandatory at the moment. I will remove this requirement very soon, as it doesn't make any sense, until an actual restore has to happen.

There are a few limitations on the path data provided in this configuration:

  • They can't begin with a hyphen "-".
  • They can't contain any backslash characters.
  • They can't contain any newline characters.
  • Extra: Make sure to check quoting rules if you need to add any characters, that have a special meaning in the YAML syntax.

Example

backup-compose.yml example:

backup_groups:
  my_local_backups:
    encryption:
      enabled: no
    backup_provider:
      url: file://
    volume_size: 200
    sources:
      /var/www/html:
        backup_path: /root/backups/var/www/html
        restore_path: /root/restored/var/www/html
      /home/tommy:
        backup_path: /root/backups/home/tommy
        restore_path: /root/restored/home/tommy
  my_s3_backups:
    encryption:
      enabled: yes
      gpg_key: xxxxxx
      gpg_passphrase: xxxxxx
    backup_provider:
      url: s3://s3.sa-east-1.amazonaws.com/my-backup-bucket
      aws_access_key: xxxxxx
      aws_secret_key: xxxxxx
    backup_file_prefixes:
      manifest: manifest_
      archive: archive_
      signature: signature_
    volume_size: 50
    sources:
      /etc:
        backup_path: /etc
        restore_path: /root/restored/etc
      /home/shared:
        backup_path: /home/shared
        restore_path: /root/restored/home/shared
  my_scp_backups:
    encryption:
      enabled: no
    backup_provider:
      url: scp://myscpuser@host.example.com/
      password: xxxxxx
    volume_size: 200
    sources:
      /home/fun:
        backup_path: /home/fun
        restore_path: /root/restored/home/fun
      /home/katy:
        backup_path: /home/katy
        restore_path: /root/restored/home/katy

Usage

dupcomp.py [-d] [-c <configpath>] backup|restore [backup_group1 backup_group2 ...]

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for dup-composer, version 0.1.6
Filename, size File type Python version Upload date Hashes
Filename, size dup_composer-0.1.6-py3-none-any.whl (14.0 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size dup-composer-0.1.6.tar.gz (12.1 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page