emborg 1.3.0

Borg front end.

emborg – Encrypted Backups to a Remote Server

Author:

Ken Kundert

Version:

1.3.0

Released:

2019-01-19

Emborg is a simple command line utility to orchestrate backups. It is built as a front-end to Borg, a powerful and fast deduplicating backup program. With Emborg, you specify all the details about your backups once in advance, and then use a very simple command line interface for your day-to-day activities. The details are contained in ~/.config/emborg. That directory contains a file (settings) that contains shared settings, and then another file for each backup configuration you have.

Use of Emborg does not preclude the use of Borg directly on the same repository. The philosophy of Emborg is to provide commands that you would use often and in an interactive manner with the expectation that you would use Borg directly for the remaining commands.

An alternative to Emborg is Borgmatic. It seems largely focused on the archive creation process and offers little for the other management tasks such as monitoring (due, list, manifest), restoration (extract, mount), and maintenence (check, prune).

Getting Started

Many Linux distributes include Borg in their package managers. In Fedora it is referred to as borgbackup. In this case you would install borg by running the following:

sudo dnf install borgbackup

Alternately, you can download a precompiled version from Borg Github Releases. You can do so with following commands (they will need to be adjusted for to get the latest version):

cd ~/bin
wget https://github.com/borgbackup/borg/releases/download/1.1.8/borg-linux64
wget https://github.com/borgbackup/borg/releases/download/1.1.8/borg-linux64.asc
gpg --recv-keys "FAF7B393"
gpg --verify borg-linux64.asc
rm borg-linux64.asc
chmod 755 borg-linux64

Download and install Emborg as follows:

pip3 install --user emborg

Or, if you want the development version, use:

git clone https://github.com/KenKundert/emborg.git
pip3 install --user ./emborg

Then you will need to create your Emborg settings directory (~/.config/emborg) and create a shared settings file ‘settings’ and then one or more files, one for each configuration you want. If you run ‘emborg’ without creating the settings directory, it will create it for you and populate it with starter files you must edit to use. Specifically it creates a shared settings file, and then a home and root configuration. You generally only need one. Start from home if you are backing up your home directory, and start from root if you are backing up the root file system. Delete the one you do not need.

Normally people have just two files, the shared settings file and one configuration file, perhaps named ‘home’ because it used to back up your home directory. However, you may wish to have a second configuration dedicated to creating snapshots of your files every 15 minutes or so. These snapshots may be kept locally and only for a day or so while your primary backups are kept remotely and kept long term.

Settings may be placed in either the shared settings file or the configuration specific file. The ones placed in the configuration specific file dominate. The shared settings file must contain at least one setting, configurations, which is a list of the available configurations.

You can find descriptions of all available settings with:

emborg settings -a

There are certain settings that are worth highlighting.

repository:

The destination for the backups. A typical value might be:

repository = 'archives:/mnt/backups/{host_name}-{user_name}-{config_name}'

where in this example ‘archives’ is the hostname and /mnt/backups is the absolute path to the directory that is to contain your Borg repositories, and {host_name}-{user_name}-{config_name} is the directory to contain this repository.

archive:

A template that is used when creating the archive name. A typical value might be:

archive = '{host_name}-{{now}}'

encryption:

The encryption mode that is used when first creating the repository. Common values are none, authenticated, repokey, and keyfile. The repository is encrypted if you choose repokey or keyfile. In either case the passphrase you provide does not encrypt repository. Rather the repository is encrypted using a key that is randomly generated by Borg. You passphrase encrypts the key. Thus, to restore your files you will need both the key and the passphrase. With repokey your key is copied to the repository, so it can be used with trusted repositories. Use keyfile if the remote repository is not trusted. It does not copy the key to the repository, meaning that it is extremely important for you export the key using ‘borg key export’ and keep a copy along with the passphrase.

passphrase:

The passphrase used when encrypting the encryption key. This is used as an alternative to avendesora_account. Be sure to make the file that contains it unreadable by others.

passcommand:

An alternate to passphrase. Borg uses this command to access your passphrase.

avendesora_account:

Another alternative to passphrase. The name of the Avendesora account used to hold the passphrase for the encryption key. Use this as an alternative to passphrase. This keeps your passphrase out of your settings file, but requires that GPG agent be available and loaded with your private key. This is normal when running interactively. When running batch, say from cron, you can use the Linux keychain command to retain your GPG credentials for you.

src_dirs:

The list of directories to be backed up. A typical value might be:

src_dirs = '~'.split()

excludes:

A list of files to exclude from the backups. typical value might be:

excludes = '''
    ~/tmp
    ~/.local
    ~/.cache
    ~/.mozilla
    ~/.thunderbird
    ~/.config/google-chrome*
    ~/.config/libreoffice
    ~/**/__pycache__
    ~/**/*.pyc
    ~/**/.*.swp
    ~/**/.*.swo
'''.split()

Once you have set up your configuration directory, you will need to create your repository. To do so, assure that the parent directory of your repository exists and is writable on the remote server. Then run:

emborg init

Once you have done that you can create your first backup using:

emborg create

Then you can run any of the commands documented below.

Commands

Here is are the available commands:

borg:

run a raw borg command.

breaklock:

breaks the repository and cache locks.

check:

checks the repository and its archives

configs:

list available backup configurations

create:

create an archive of the current files

delete:

delete an archive currently contained in the repository

diff:

show the differences between two archives

due:

days since last backup

extract:

recover file or files from archive

help:

give information about commands or other topics

info:

print information about a backup

init:

initialize the repository

list:

list the archives currently contained in the repository

log:

print logfile for the last emborg run

manifest:

list the files contained in an archive

mount:

mount a repository or archive

prune:

list the archives currently contained in the repository

settings:

list settings of chosen configuration

umount:

un-mount a previously mounted repository or archive

version:

display emborg version

Configuration

Shared settings go in ~/.config/emborg/settings. This is a Python file that contains values needed by Emborg.

Shared Settings

Shared settings go in ~/.config/emborg/settings. This is a Python file that contains values needed by Emborg. It might look like the following:

default_configuration = 'home'        # default backup configuration
configurations = 'home websites'      # available backup configurations
avendesora_account = 'borg-backup'    # Avendesora account name (holds passphrase for encryption key)
passphrase = None                     # passphrase to use (if specified, Avendesora is not used)
encryption = 'keyfile'                # encryption method
prune_after_create = True             # run prune as the last step of an archive creation
check_after_create = True             # run check as the last step of an archive creation
notify = "[e-mail]"            # email address to notify when things go wrong
notifier = 'notify-send -u normal {prog_name} "{msg}"'
                                      # program used to send realtime notifications
                                      # generally you use notify or notifier, but not both
                                      # use notifier for interactive backups
                                      # and notify for scheduled backups
                                      # notification program
remote_ratelimit = 2000               # bandwidth limit in kbps
umask = '077'                         # umask to use when creating the archives
keep_within = '1d'                    # keep all archives within this time interval
keep_hourly = '48'                    # number of hourly archives to keep
keep_daily = '7'                      # number of daily archives to keep
keep_weekly = '4'                     # number of weekly archives to keep
keep_monthly = '12'                   # number of weekly archives to keep
keep_yearly = '2'                     # number of weekly archives to keep

If you encrypt your backups, you can specify the encryption key in this file as passphrase. In this case, you should be careful to assure the file is not readable by others (chmod 600 settings). Alternatively, you can use Avendesora to securely hold your key by specifying the Avendesora account name of the key to avendesora_account.

Configuration Settings

Each backup configuration must have a settings file in ~/.config/emborg. The name of the file is the name of the backup configuration. It might look like the following:

repository = 'archives:/mnt/backups/{host_name}/{config_name}'
                                      # remote directory for repository
archive = '{host_name}-{{now}}'       # naming pattern used for the archives
    # May contain {<name>} where <name> may be any of host_name, user_name,
    # prog_name config_name, or any of the user specified settings.
    # Double up the braces to specify parameters that should be interpreted
    # by borg rather than by emborg.
src_dirs = ['~', '/etc']              # absolute path to directory to be backed up
excludes = '''
    ~/tmp
    ~/**/.hg
    ~/**/.git
    ~/**/*.pyc
    ~/**/.*.swp
    ~/**/.*.swo
'''.split()                            # list of glob strings of files or directories to skip
one_file_system = False
exclude_caches = True

# commands to be run before and after backups (run from working directory)
run_before_backup = [
    './clean-home >& clean-home.log',
        # remove the detritus before backing up
]
run_after_backup = [
    './rebuild-manpages > /dev/null',
        # rebuild my man pages, they were deleted in clean
]

# if set, this file or these files must exist or backups will quit with an error
must_exist = '~/doc/thesis'

String values may incorporate other string valued settings. Use braces to interpolate another setting. In addition, you may interpolate the configuration name (‘config_name’), the host name (‘host_name’), the user name (‘user_name’) or Emborg’s program name (‘prog_name’). An example of this is shown in both repository and archive above.

Precautions

You should assure you have a backup copy of the encryption key and its passphrase in a safe place (run ‘borg key export’ to extract the encryption keys). This is very important. If the only copy of the encryption credentials are on the disk being backed up, then if that disk were to fail you would not be able to access your backups. I recommend the use of sparekeys as a way of assuring that you always have access to the essential information, such as your Borg passphrase and keys, that you would need to get started after a catastrophic loss of your disk.

If you keep the passphrase in the emborg file, you should set its permissions so that it is not readable by others:

chmod 600 settings

Better is to simply not store the passphrase in the emborg script. You can use the passcommand setting for this, or you use Avendesora, which is a flexible password management system. The interface to Avendesora is already built in to Emborg, but its use is optional (it need not be installed).

It is also best, if it can be arranged, to keep your backups at a remote site so that your backups do not get destroyed in the same disaster, such as a fire or flood, that claims your original files. One option is rsync.net.

Finally, it is a good idea to practice a recovery. Pretend that you have lost all your files and then see if you can do a restore from backup. Doing this and working out the kinks before you lose your files can save you if you ever do lose your files.

Borg

Borg has considerably more power than what is exposed with emborg. You may use it directly or through the Emborg borg command when you need that power. More information about Borg can be found at borgbackup on readthedocs.

Commands

BreakLock

This command breaks the repository and cache locks. Please use carefully and only while no borg process (on any machine) is trying to access the Cache or the Repository.

emborg break-lock
emborg breaklock

Check

Check the integrity of the repository and its archives.

Configs

List the available backup configurations. Each configuration will correspond to a settings file in your configuration directory (~/.config/emborg). Settings common to all your configurations should be placed in ~/.config/emborg/settings. You can see available configs using:

emborg configs

To run a command on a specific configuration, add –config=<cfg> or -c cfg before the command. For example:

emborg -c home create

Create

This creates an archive in an existing repository. An archive is a snapshot of your files as they currently exist. Borg is a de-duplicating backup program, so only the changes from the already existing archives are saved.

emborg create

Before creating your first archive, you must use the init command to initialize your repository.

Delete

Delete an archive currently contained in the repository.

Diff

Shows the differences between two archives:

> emborg diff kundert-2018-12-05T19:23:09 kundert-2018-12-04T17:41:28

Due

When run with no options it indicates when the last backup was created. For example:

> emborg due
backup was performed 19 hours ago.

Adding the –days option results in the message only being printed if the backup has not been performed within the specified number of days. Adding the –email option results in the message being sent to the specified address rather than printed. This allows you to run the due command from a cron script in order to send your self reminders to do a backup if one has not occurred for a while.

Extract

You extract a file or directory from an archive using:

emborg extract home/ken/bin

Use manifest to determine what path you should specify to identify the desired file or directory (they will paths relative to /). Thus, the paths should look like absolute paths with the leading slash removed. The paths may point to directories, in which case the entire directory is extracted. It may also be a glob pattern.

If you do not specify an archive or date, the most recent archive is used. You can extract the version of a file or directory that existed on a particular date using:

emborg extract --date 2015-04-01 home/ken/bin

Or, you can extract the version from a particular archive using:

emborg extract --archive kundert-2018-12-05T12:54:26 home/ken/bin

The extracted files are placed in the current working directory within their original hierarchy. Thus, the above commands create the file:

./home/ken/bin

Help

Show information about Emborg:

emborg help

You can ask for help on a specific command or topic with:

emborg help <topic>

For example:

emborg help extract

Info

This command prints out the locations of important files and directories.

emborg info

Init

Initializes a Borg repository. This must be done before you create your first archive.

emborg init

List

List available archives.

emborg list

Log

Show the logfile from the previous run.

Manifest

Once a backup has been performed, you can list the files available in your archive using:

emborg manifest

If you do not specify an archive, as above, the latest archive is used.

You can explicitly specify an archive:

emborg manifest --archive kundert-2015-04-01T12:19:58

Or you can list the files that existed on a particular date using:

emborg manifest --date 2015-04-01

Mount

Once a backup has been performed, you can mount it and then look around as you would a normal read-only filesystem.

emborg mount backups

In this example, backups acts as a mount point. If it exists, it must be a directory. If it does not exist, it is created.

If you do not specify an archive, as above, all archives are mounted.

You can explicitly specify an archive:

emborg mount --archive kundert-2015-04-01T12:19:58 backups

Or you can mount the files that existed on a particular date using:

emborg mount --date 2015-04-01 backups

You will need to un-mount the repository or archive when you are done with it. To do so, use the umount command.

Prune

Prune the repository of excess archives. You can use the keep_within, keep_last, keep_minutely, keep_hourly, keep_daily, keep_weekly, keep_monthly, and keep_yearly settings to control which archives should be kept. At least one of these settings must be specified to use prune:

emborg prune

Settings

This command displays all the settings that affect a backup configuration. Add ‘-a’ option to list out all available settings and their descriptions rather than the specified settings and their values.

Umount

Un-mount a previously mounted repository or archive:

emborg umount backups
rmdir backups

where backups is the existing mount point.

Version

Prints the emborg version.

emborg version

API

Emborg has a simple API that allows you to run borg commands. Here is an example taken from sparekeys that exports the keys from your Borg repository so then can be backed up separately:

from emborg import Emborg

with Emborg() as emborg:
    borg = emborg.run_borg(
        cmd = 'key export',
        args = [emborg.destination(), archive / '.config/borg.repokey']
    )
    if borg.stdout:
        print(borg.stdout.rstrip())

Emborg takes the config name as an argument, if not given the default config is used. It provides the following useful methods and attributes:

repository:

The path to the repository.

destination(archive):

Returns the full path to the archive. If Archive is False or None, then the path to the repository it returned. If Archive is True, then the default archive name as taken from settings file is used. This is only appropriate when creating new repositories.

run_borg(cmd, args, borg_opts, emborg_opts):

Runs a Borg command. cmd is the desired Borg command (ex: ‘create’, ‘prune’, etc.). By default, command line options derived from Emborg settings are automatically added to the command line. args contains the command line arguments (such as the repository or archive). It may also contain any additional command line options not provided in borg_opts. It may be a list or a string. If it is a string, it is split at white space. borg_opts are the command line options needed by Borg. If not given, it is created for you by Emborg based upon your configuration settings. Finally, emborg_opts is a list that may contain any of the following options: ‘verbose’, ‘narrate’, ‘trial-run’, or ‘no-log’.

This function runs the Borg command and returns a process object that allows you access to stdout via the stdout attribute.

borg_options(cmd, emborg_opts):

This function returns the default Borg command line options, those that would be used in run_borg if borg_opts is not set. It can be used when constructing a custom borg_opts.

value(name, default=’’):

Returns the value of a setting from an Emborg configuration. If not set, default is returned.

You can examine the emborg/command.py file for inspiration and examples on how to use the Emborg API.