Skip to main content

Borg front end.

Project description

emborg – Encrypted Backups to a Remote Server

https://img.shields.io/pypi/v/emborg.svg https://img.shields.io/pypi/pyversions/emborg.svg
Author:Ken Kundert
Version:1.4.0
Released:2019-04-24

Emborg is a simple command line utility to orchestrate backups. It is built as a front-end to Borg, a powerful and fast deduplicating backup program. With Emborg, you specify all the details about your backups once in advance, and then use a very simple command line interface for your day-to-day activities. The details are contained in ~/.config/emborg. That directory contains a file (settings) that contains shared settings, and then another file for each backup configuration you have.

Use of Emborg does not preclude the use of Borg directly on the same repository. The philosophy of Emborg is to provide commands that you would use often and in an interactive manner with the expectation that you would use Borg directly for the remaining commands.

An alternative to Emborg is Borgmatic. It seems largely focused on the archive creation process and offers little for the other management tasks such as monitoring (due, list, manifest), restoration (extract, mount), and maintenence (check, prune).

Getting Started

Many Linux distributes include Borg in their package managers. In Fedora it is referred to as borgbackup. In this case you would install borg by running the following:

sudo dnf install borgbackup

Alternately, you can download a precompiled version from Borg Github Releases. You can do so with following commands (they will need to be adjusted for to get the latest version):

cd ~/bin
wget https://github.com/borgbackup/borg/releases/download/1.1.9/borg-linux64
wget https://github.com/borgbackup/borg/releases/download/1.1.9/borg-linux64.asc
gpg --recv-keys "FAF7B393"
gpg --verify borg-linux64.asc
rm borg-linux64.asc
chmod 755 borg-linux64

Download and install Emborg as follows (requires Python3.6 or better):

pip3 install --user emborg

Or, if you want the development version, use:

git clone https://github.com/KenKundert/emborg.git
pip3 install --user ./emborg

Then you will need to create your Emborg settings directory (~/.config/emborg) and create a shared settings file ‘settings’ and then one or more files, one for each configuration you want. If you run ‘emborg’ without creating the settings directory, it will create it for you and populate it with starter files you must edit to use. Specifically it creates a shared settings file, and then a home and root configuration. You generally only need one. Start from home if you are backing up your home directory, and start from root if you are backing up the root file system. Delete the one you do not need.

Normally people have just two files, the shared settings file and one configuration file, perhaps named ‘home’ because it used to back up your home directory. However, you may wish to have a second configuration dedicated to creating snapshots of your files every 15 minutes or so. These snapshots may be kept locally and only for a day or so while your primary backups are kept remotely and kept long term.

Settings may be placed in either the shared settings file or the configuration specific file. The ones placed in the configuration specific file dominate. The shared settings file must contain at least one setting, configurations, which is a list of the available configurations.

You can find descriptions of all available settings with:

emborg settings -a

There are certain settings that are worth highlighting.

repository

The destination for the backups. A typical value might be:

repository = 'archives:/mnt/backups/{host_name}-{user_name}-{config_name}'

where in this example ‘archives’ is the hostname and /mnt/backups is the absolute path to the directory that is to contain your Borg repositories, and {host_name}-{user_name}-{config_name} is the directory to contain this repository.

archive

A template that is used when creating the archive name. A typical value might be:

archive = '{host_name}-{{now}}'

encryption

The encryption mode that is used when first creating the repository. Common values are none, authenticated, repokey, and keyfile. The repository is encrypted if you choose repokey or keyfile. In either case the passphrase you provide does not encrypt repository. Rather the repository is encrypted using a key that is randomly generated by Borg. You passphrase encrypts the key. Thus, to restore your files you will need both the key and the passphrase. With repokey your key is copied to the repository, so it can be used with trusted repositories. Use keyfile if the remote repository is not trusted. It does not copy the key to the repository, meaning that it is extremely important for you export the key using ‘borg key export’ and keep a copy along with the passphrase.

passphrase

The passphrase used when encrypting the encryption key. This is used as an alternative to avendesora_account. Be sure to make the file that contains it unreadable by others.

passcommand

An alternate to passphrase. Borg uses this command to access your passphrase.

avendesora_account

Another alternative to passphrase. The name of the Avendesora account used to hold the passphrase for the encryption key. Use this as an alternative to passphrase. This keeps your passphrase out of your settings file, but requires that GPG agent be available and loaded with your private key. This is normal when running interactively. When running batch, say from cron, you can use the Linux keychain command to retain your GPG credentials for you.

src_dirs

The list of directories to be backed up. A typical value might be:

src_dirs = '~'.split()

excludes

A list of files to exclude from the backups. Typical value might be:

excludes = '''
    ~/tmp
    ~/.local
    ~/.cache
    ~/.mozilla
    ~/.thunderbird
    ~/.config/google-chrome*
    ~/.config/libreoffice
    ~/**/__pycache__
    ~/**/*.pyc
    ~/**/.*.swp
    ~/**/.*.swo
'''.split()

Once you have set up your configuration directory, you will need to create your repository. To do so, assure that the parent directory of your repository exists and is writable on the remote server. Then run:

emborg init

Once you have done that you can create your first backup using:

emborg create

Then you can run any of the commands documented below.

Command Summary

Here is are the available commands:

borg:run a raw borg command.
breaklock:breaks the repository and cache locks.
check:checks the repository and its archives
configs:list available backup configurations
create:create an archive of the current files
delete:delete an archive currently contained in the repository
diff:show the differences between two archives
due:days since last backup
extract:recover file or files from archive
help:give information about commands or other topics
info:print information about a backup
init:initialize the repository
list:list the archives currently contained in the repository
log:print logfile for the last emborg run
manifest:list the files contained in an archive
mount:mount a repository or archive
prune:list the archives currently contained in the repository
settings:list settings of chosen configuration
umount:un-mount a previously mounted repository or archive
version:display emborg version

These commands are described in more detail below.

Configuration

Shared settings go in ~/.config/emborg/settings. This is a Python file that contains values needed by Emborg.

Shared Settings

Shared settings go in ~/.config/emborg/settings. This is a Python file that contains values needed by Emborg. It might look like the following:

default_configuration = 'home'        # default backup configuration
configurations = 'home websites'      # available backup configurations
avendesora_account = 'borg-backup'    # Avendesora account name (holds passphrase for encryption key)
passphrase = None                     # passphrase to use (if specified, Avendesora is not used)
encryption = 'keyfile'                # encryption method
prune_after_create = True             # run prune as the last step of an archive creation
check_after_create = True             # run check as the last step of an archive creation
notify = "me@mydomain.com"            # email address to notify when things go wrong
notifier = 'notify-send -u normal {prog_name} "{msg}"'
                                      # program used to send realtime notifications
                                      # generally you use notify or notifier, but not both
                                      # use notifier for interactive backups
                                      # and notify for scheduled backups
                                      # notification program
remote_ratelimit = 2000               # bandwidth limit in kbps
umask = '077'                         # umask to use when creating the archives
keep_within = '1d'                    # keep all archives within this time interval
keep_hourly = '48'                    # number of hourly archives to keep
keep_daily = '7'                      # number of daily archives to keep
keep_weekly = '4'                     # number of weekly archives to keep
keep_monthly = '12'                   # number of weekly archives to keep
keep_yearly = '2'                     # number of weekly archives to keep

If you encrypt your backups, you can specify the encryption key in this file as passphrase. In this case, you should be careful to assure the file is not readable by others (chmod 600 settings). Alternatively, you can use Avendesora to securely hold your key by specifying the Avendesora account name of the key to avendesora_account.

Configuration Settings

Each backup configuration must have a settings file in ~/.config/emborg. The name of the file is the name of the backup configuration. It might look like the following:

repository = 'archives:/mnt/backups/{host_name}/{config_name}'
                                      # remote directory for repository
archive = '{host_name}-{{now}}'       # naming pattern used for the archives
    # May contain {<name>} where <name> may be any of host_name, user_name,
    # prog_name config_name, or any of the user specified settings.
    # Double up the braces to specify parameters that should be interpreted
    # by borg rather than by emborg.
src_dirs = ['~', '/etc']              # absolute path to directory to be backed up
excludes = '''
    ~/tmp
    ~/**/.hg
    ~/**/.git
    ~/**/*.pyc
    ~/**/.*.swp
    ~/**/.*.swo
'''.split()                            # list of glob strings of files or directories to skip
one_file_system = False
exclude_caches = True

# commands to be run before and after backups (run from working directory)
run_before_backup = [
    './clean-home >& clean-home.log',
        # remove the detritus before backing up
]
run_after_backup = [
    './rebuild-manpages > /dev/null',
        # rebuild my man pages, they were deleted in clean
]

# if set, this file or these files must exist or backups will quit with an error
must_exist = '~/doc/thesis'

String values may incorporate other string valued settings. Use braces to interpolate another setting. In addition, you may interpolate the configuration name (‘config_name’), the host name (‘host_name’), the user name (‘user_name’) or Emborg’s program name (‘prog_name’). An example of this is shown in both repository and archive above.

Precautions

You should assure you have a backup copy of the encryption key and its passphrase in a safe place (run ‘borg key export’ to extract the encryption keys). This is very important. If the only copy of the encryption credentials are on the disk being backed up, then if that disk were to fail you would not be able to access your backups. I recommend the use of sparekeys as a way of assuring that you always have access to the essential information, such as your Borg passphrase and keys, that you would need to get started after a catastrophic loss of your disk.

If you keep the passphrase in the emborg file, you should set its permissions so that it is not readable by others:

chmod 600 ~/.config/emborg/*

Better is to simply not store the passphrase in the emborg script. You can use the passcommand setting for this, or you use Avendesora, which is a flexible password management system. The interface to Avendesora is already built in to Emborg, but its use is optional (it need not be installed).

It is also best, if it can be arranged, to keep your backups at a remote site so that your backups do not get destroyed in the same disaster, such as a fire or flood, that claims your original files. One option is rsync.net. Another is BorgBase. I have not tried either, and so offer no recommendation.

Finally, it is a good idea to practice a recovery. Pretend that you have lost all your files and then see if you can do a restore from backup. Doing this and working out the kinks before you lose your files can save you if you ever do lose your files.

Borg

Borg has considerably more power than what is exposed with emborg. You may use it directly or through the Emborg borg command when you need that power. More information about Borg can be found at borgbackup on readthedocs.

Commands

Borg

Runs raw Borg commands. Before running the passphrase or passcommand is set. Also, if @repo is found on the command line, it is replaced by the path to the repository.

emborg borg key export @repo key.borg

BreakLock

This command breaks the repository and cache locks. Please use carefully and only while no borg process (on any machine) is trying to access the Cache or the Repository.

emborg break-lock
emborg breaklock

Check

Check the integrity of the repository and its archives.

Configs

List the available backup configurations. Each configuration will correspond to a settings file in your configuration directory (~/.config/emborg). Settings common to all your configurations should be placed in ~/.config/emborg/settings. You can see available configs using:

emborg configs

To run a command on a specific configuration, add –config=<cfg> or -c cfg before the command. For example:

emborg -c home create

Create

This creates an archive in an existing repository. An archive is a snapshot of your files as they currently exist. Borg is a de-duplicating backup program, so only the changes from the already existing archives are saved.

emborg create

Before creating your first archive, you must use the init command to initialize your repository. create is the default command, so you can create an archive with simply:

emborg

If the backup seems to be taking a long time for no obvious reason, run the backup in verbose mode:

emborg -v create

This can help you understand what is happening.

Delete

Delete an archive currently contained in the repository.

Diff

Shows the differences between two archives:

> emborg diff kundert-2018-12-05T19:23:09 kundert-2018-12-04T17:41:28

Due

When run with no options it indicates when the last backup was created. For example:

> emborg due
backup was performed 19 hours ago.

Adding the –days option results in the message only being printed if the backup has not been performed within the specified number of days. Adding the –email option results in the message being sent to the specified address rather than printed. This allows you to run the due command from a cron script in order to send your self reminders to do a backup if one has not occurred for a while.

Extract

You extract a file or directory from an archive using:

emborg extract home/ken/bin

Use manifest to determine what path you should specify to identify the desired file or directory (they will paths relative to /). Thus, the paths should look like absolute paths with the leading slash removed. The paths may point to directories, in which case the entire directory is extracted. It may also be a glob pattern.

If you do not specify an archive or date, the most recent archive is used. You can extract the version of a file or directory that existed on a particular date using:

emborg extract --date 2015-04-01 home/ken/bin

Or, you can extract the version from a particular archive using:

emborg extract --archive kundert-2018-12-05T12:54:26 home/ken/bin

The extracted files are placed in the current working directory within their original hierarchy. Thus, the above commands create the file:

./home/ken/bin

Help

Show information about Emborg:

emborg help

You can ask for help on a specific command or topic with:

emborg help <topic>

For example:

emborg help extract

Info

This command prints out the locations of important files and directories.

emborg info

Init

Initializes a Borg repository. This must be done before you create your first archive.

emborg init

List

List available archives.

emborg list

Log

Show the logfile from the previous run.

Manifest

Once a backup has been performed, you can list the files available in your archive using:

emborg manifest

If you do not specify an archive, as above, the latest archive is used.

You can explicitly specify an archive:

emborg manifest --archive kundert-2015-04-01T12:19:58

Or you can list the files that existed on a particular date using:

emborg manifest --date 2015-04-01

Mount

Once a backup has been performed, you can mount it and then look around as you would a normal read-only filesystem.

emborg mount backups

In this example, backups acts as a mount point. If it exists, it must be a directory. If it does not exist, it is created.

If you do not specify an archive, as above, all archives are mounted.

You can explicitly specify an archive:

emborg mount --archive kundert-2015-04-01T12:19:58 backups

Or you can mount the files that existed on a particular date using:

emborg mount --date 2015-04-01 backups

You will need to un-mount the repository or archive when you are done with it. To do so, use the umount command.

Prune

Prune the repository of excess archives. You can use the keep_within, keep_last, keep_minutely, keep_hourly, keep_daily, keep_weekly, keep_monthly, and keep_yearly settings to control which archives should be kept. At least one of these settings must be specified to use prune:

emborg prune

Settings

This command displays all the settings that affect a backup configuration. Add ‘-a’ option to list out all available settings and their descriptions rather than the specified settings and their values.

Umount

Un-mount a previously mounted repository or archive:

emborg umount backups
rmdir backups

where backups is the existing mount point.

Version

Prints the emborg version.

emborg version

API

Emborg has a simple API that allows you to run borg commands. Here is an example taken from sparekeys that exports the keys from your Borg repository so then can be backed up separately:

from emborg import Emborg

with Emborg() as emborg:
    borg = emborg.run_borg(
        cmd = 'key export',
        args = [emborg.destination(), archive / '.config/borg.repokey']
    )
    if borg.stdout:
        print(borg.stdout.rstrip())

Emborg takes the config name as an argument, if not given the default config is used. It provides the following useful methods and attributes:

repository

The path to the repository.

destination(archive)

Returns the full path to the archive. If Archive is False or None, then the path to the repository it returned. If Archive is True, then the default archive name as taken from settings file is used. This is only appropriate when creating new repositories.

run_borg(cmd, args, borg_opts, emborg_opts)

Runs a Borg command.

cmd is the desired Borg command (ex: ‘create’, ‘prune’, etc.).

args contains the command line arguments (such as the repository or archive). It may also contain any additional command line options not automatically provided. It may be a list or a string. If it is a string, it is split at white space.

borg_opts are the command line options needed by Borg. If not given, it is created for you by Emborg based upon your configuration settings.

Finally, emborg_opts is a list that may contain any of the following options: ‘verbose’, ‘narrate’, ‘trial-run’, or ‘no-log’.

This function runs the Borg command and returns a process object that allows you access to stdout via the stdout attribute.

run_borg_raw(args)

Runs a raw Borg command without interpretation except for replacing a @repo argument with the path to the repository.

args contains all command line options and arguments except the path to the executable.

borg_options(cmd, emborg_opts)

This function returns the default Borg command line options, those that would be used in run_borg if borg_opts is not set. It can be used when constructing a custom borg_opts.

value(name, default=’‘)

Returns the value of a setting from an Emborg configuration. If not set, default is returned.

You can examine the emborg/command.py file for inspiration and examples on how to use the Emborg API.

Overdue

Emborg contains an additional executable, emborg-overdue, that can be run on the destination server to determine whether the backups have been performed recently. It reads its own settings file in ~/.config/emborg/overdue.conf that is also a Python file and may contain the following settings:

default_maintainer (email address -- mail is sent to this person upon failure)
default_max_age (hours)
dumper (email address -- mail is sent from this person)
root (default directory for repositories)
repositories (string or array of dicts)

Here is an example config file:

default_maintainer = 'root@continuum.com'
dumper = 'dumper@continuum.com'
default_max_age = 12 # hours
root = '/mnt/borg-backups/repositories'
repositories = [
    dict(host='mercury (/)', path='mercury-root-root'),
    dict(host='venus (/)', path='venus-root-root'),
    dict(host='earth (/)', path='earth-root-root'),
    dict(host='mars (/)', path='mars-root-root'),
    dict(host='jupiter (/)', path='jupiter-root-root'),
    dict(host='saturn (/)', path='saturn-root-root'),
    dict(host='uranus (/)', path='uranus-root-root'),
    dict(host='neptune (/)', path='neptune-root-root'),
    dict(host='pluto (/)', path='pluto-root-root'),
]

The dictionaries in repositories can contain the following fields: host, path, maintainer, max_age. host is a description of the host. It is included in the email that is sent when problems occur to identify the backup. It is a good idea for it to contain both the host name and the source directory being backed up. path is either the archive name or a full absolute path to the archive. If path is an absolute path, it is used, otherwise it is added to the end of root. maintainer is an email address, an email is sent to this address if there is an issue. max_age is the number of hours that may pass before an archive is considered overdue.

repositories can also be specified as a list of dictionaries as follows:

repositories = """
    HOST        | NAME or PATH      | MAINTAINER           | MAXIMUM AGE (hours)
    mercury (/) | mercury-root-root |                      |
    venus (/)   | venus-root-root   |                      |
    earth (/)   | earth-root-root   |                      |
    mars (/)    | mars-root-root    |                      |
    jupiter (/) | jupiter-root-root |                      |
    saturn (/)  | saturn-root-root  |                      |
    uranus (/)  | uranus-root-root  |                      |
    neptune (/) | neptune-root-root |                      |
    pluto (/)   | pluto-root-root   |                      |
"""

If repositories is a string, it is first split on newlines, anything beyond a # is considered a comment and is ignored, and the finally the lines are split on ‘|’ and the 4 values are expected to be given in order. If the maintainer is not given, the default_maintainer is used. If max_age is not given, the default_max_age is used.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for emborg, version 1.4.0
Filename, size & hash File type Python version Upload date
emborg-1.4.0.tar.gz (41.8 kB) View hashes Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page