Borg front end.
Project description
emborg – Encrypted Backups to a Remote Server
- Author:
Ken Kundert
- Version:
1.4.0
- Released:
2019-04-24
Emborg is a simple command line utility to orchestrate backups. It is built as a front-end to Borg, a powerful and fast deduplicating backup program. With Emborg, you specify all the details about your backups once in advance, and then use a very simple command line interface for your day-to-day activities. The details are contained in ~/.config/emborg. That directory contains a file (settings) that contains shared settings, and then another file for each backup configuration you have.
Use of Emborg does not preclude the use of Borg directly on the same repository. The philosophy of Emborg is to provide commands that you would use often and in an interactive manner with the expectation that you would use Borg directly for the remaining commands.
An alternative to Emborg is Borgmatic. It seems largely focused on the archive creation process and offers little for the other management tasks such as monitoring (due, list, manifest), restoration (extract, mount), and maintenence (check, prune).
Getting Started
Many Linux distributes include Borg in their package managers. In Fedora it is referred to as borgbackup. In this case you would install borg by running the following:
sudo dnf install borgbackup
Alternately, you can download a precompiled version from Borg Github Releases. You can do so with following commands (they will need to be adjusted for to get the latest version):
cd ~/bin wget https://github.com/borgbackup/borg/releases/download/1.1.9/borg-linux64 wget https://github.com/borgbackup/borg/releases/download/1.1.9/borg-linux64.asc gpg --recv-keys "FAF7B393" gpg --verify borg-linux64.asc rm borg-linux64.asc chmod 755 borg-linux64
Download and install Emborg as follows (requires Python3.6 or better):
pip3 install --user emborg
Or, if you want the development version, use:
git clone https://github.com/KenKundert/emborg.git pip3 install --user ./emborg
Then you will need to create your Emborg settings directory (~/.config/emborg) and create a shared settings file ‘settings’ and then one or more files, one for each configuration you want. If you run ‘emborg’ without creating the settings directory, it will create it for you and populate it with starter files you must edit to use. Specifically it creates a shared settings file, and then a home and root configuration. You generally only need one. Start from home if you are backing up your home directory, and start from root if you are backing up the root file system. Delete the one you do not need.
Normally people have just two files, the shared settings file and one configuration file, perhaps named ‘home’ because it used to back up your home directory. However, you may wish to have a second configuration dedicated to creating snapshots of your files every 15 minutes or so. These snapshots may be kept locally and only for a day or so while your primary backups are kept remotely and kept long term.
Settings may be placed in either the shared settings file or the configuration specific file. The ones placed in the configuration specific file dominate. The shared settings file must contain at least one setting, configurations, which is a list of the available configurations.
You can find descriptions of all available settings with:
emborg settings -a
There are certain settings that are worth highlighting.
repository
The destination for the backups. A typical value might be:
repository = 'archives:/mnt/backups/{host_name}-{user_name}-{config_name}'
where in this example ‘archives’ is the hostname and /mnt/backups is the absolute path to the directory that is to contain your Borg repositories, and {host_name}-{user_name}-{config_name} is the directory to contain this repository.
archive
A template that is used when creating the archive name. A typical value might be:
archive = '{host_name}-{{now}}'
encryption
The encryption mode that is used when first creating the repository. Common values are none, authenticated, repokey, and keyfile. The repository is encrypted if you choose repokey or keyfile. In either case the passphrase you provide does not encrypt repository. Rather the repository is encrypted using a key that is randomly generated by Borg. You passphrase encrypts the key. Thus, to restore your files you will need both the key and the passphrase. With repokey your key is copied to the repository, so it can be used with trusted repositories. Use keyfile if the remote repository is not trusted. It does not copy the key to the repository, meaning that it is extremely important for you export the key using ‘borg key export’ and keep a copy along with the passphrase.
passphrase
The passphrase used when encrypting the encryption key. This is used as an alternative to avendesora_account. Be sure to make the file that contains it unreadable by others.
passcommand
An alternate to passphrase. Borg uses this command to access your passphrase.
avendesora_account
Another alternative to passphrase. The name of the Avendesora account used to hold the passphrase for the encryption key. Use this as an alternative to passphrase. This keeps your passphrase out of your settings file, but requires that GPG agent be available and loaded with your private key. This is normal when running interactively. When running batch, say from cron, you can use the Linux keychain command to retain your GPG credentials for you.
src_dirs
The list of directories to be backed up. A typical value might be:
src_dirs = '~'.split()
excludes
A list of files to exclude from the backups. Typical value might be:
excludes = ''' ~/tmp ~/.local ~/.cache ~/.mozilla ~/.thunderbird ~/.config/google-chrome* ~/.config/libreoffice ~/**/__pycache__ ~/**/*.pyc ~/**/.*.swp ~/**/.*.swo '''.split()
Once you have set up your configuration directory, you will need to create your repository. To do so, assure that the parent directory of your repository exists and is writable on the remote server. Then run:
emborg init
Once you have done that you can create your first backup using:
emborg create
Then you can run any of the commands documented below.
Command Summary
Here is are the available commands:
- borg:
run a raw borg command.
- breaklock:
breaks the repository and cache locks.
- check:
checks the repository and its archives
- configs:
list available backup configurations
- create:
create an archive of the current files
- delete:
delete an archive currently contained in the repository
- diff:
show the differences between two archives
- due:
days since last backup
- extract:
recover file or files from archive
- help:
give information about commands or other topics
- info:
print information about a backup
- init:
initialize the repository
- list:
list the archives currently contained in the repository
- log:
print logfile for the last emborg run
- manifest:
list the files contained in an archive
- mount:
mount a repository or archive
- prune:
list the archives currently contained in the repository
- settings:
list settings of chosen configuration
- umount:
un-mount a previously mounted repository or archive
- version:
display emborg version
These commands are described in more detail below.
Configuration
Shared settings go in ~/.config/emborg/settings. This is a Python file that contains values needed by Emborg.
Configuration Settings
Each backup configuration must have a settings file in ~/.config/emborg. The name of the file is the name of the backup configuration. It might look like the following:
repository = 'archives:/mnt/backups/{host_name}/{config_name}' # remote directory for repository archive = '{host_name}-{{now}}' # naming pattern used for the archives # May contain {<name>} where <name> may be any of host_name, user_name, # prog_name config_name, or any of the user specified settings. # Double up the braces to specify parameters that should be interpreted # by borg rather than by emborg. src_dirs = ['~', '/etc'] # absolute path to directory to be backed up excludes = ''' ~/tmp ~/**/.hg ~/**/.git ~/**/*.pyc ~/**/.*.swp ~/**/.*.swo '''.split() # list of glob strings of files or directories to skip one_file_system = False exclude_caches = True # commands to be run before and after backups (run from working directory) run_before_backup = [ './clean-home >& clean-home.log', # remove the detritus before backing up ] run_after_backup = [ './rebuild-manpages > /dev/null', # rebuild my man pages, they were deleted in clean ] # if set, this file or these files must exist or backups will quit with an error must_exist = '~/doc/thesis'
String values may incorporate other string valued settings. Use braces to interpolate another setting. In addition, you may interpolate the configuration name (‘config_name’), the host name (‘host_name’), the user name (‘user_name’) or Emborg’s program name (‘prog_name’). An example of this is shown in both repository and archive above.
Precautions
You should assure you have a backup copy of the encryption key and its passphrase in a safe place (run ‘borg key export’ to extract the encryption keys). This is very important. If the only copy of the encryption credentials are on the disk being backed up, then if that disk were to fail you would not be able to access your backups. I recommend the use of sparekeys as a way of assuring that you always have access to the essential information, such as your Borg passphrase and keys, that you would need to get started after a catastrophic loss of your disk.
If you keep the passphrase in the emborg file, you should set its permissions so that it is not readable by others:
chmod 600 ~/.config/emborg/*
Better is to simply not store the passphrase in the emborg script. You can use the passcommand setting for this, or you use Avendesora, which is a flexible password management system. The interface to Avendesora is already built in to Emborg, but its use is optional (it need not be installed).
It is also best, if it can be arranged, to keep your backups at a remote site so that your backups do not get destroyed in the same disaster, such as a fire or flood, that claims your original files. One option is rsync.net. Another is BorgBase. I have not tried either, and so offer no recommendation.
Finally, it is a good idea to practice a recovery. Pretend that you have lost all your files and then see if you can do a restore from backup. Doing this and working out the kinks before you lose your files can save you if you ever do lose your files.
Borg
Borg has considerably more power than what is exposed with emborg. You may use it directly or through the Emborg borg command when you need that power. More information about Borg can be found at borgbackup on readthedocs.
Commands
Borg
Runs raw Borg commands. Before running the passphrase or passcommand is set. Also, if @repo is found on the command line, it is replaced by the path to the repository.
emborg borg key export @repo key.borg
BreakLock
This command breaks the repository and cache locks. Please use carefully and only while no borg process (on any machine) is trying to access the Cache or the Repository.
emborg break-lock emborg breaklock
Check
Check the integrity of the repository and its archives.
Configs
List the available backup configurations. Each configuration will correspond to a settings file in your configuration directory (~/.config/emborg). Settings common to all your configurations should be placed in ~/.config/emborg/settings. You can see available configs using:
emborg configs
To run a command on a specific configuration, add –config=<cfg> or -c cfg before the command. For example:
emborg -c home create
Create
This creates an archive in an existing repository. An archive is a snapshot of your files as they currently exist. Borg is a de-duplicating backup program, so only the changes from the already existing archives are saved.
emborg create
Before creating your first archive, you must use the init command to initialize your repository. create is the default command, so you can create an archive with simply:
emborg
If the backup seems to be taking a long time for no obvious reason, run the backup in verbose mode:
emborg -v create
This can help you understand what is happening.
Delete
Delete an archive currently contained in the repository.
Diff
Shows the differences between two archives:
> emborg diff kundert-2018-12-05T19:23:09 kundert-2018-12-04T17:41:28
Due
When run with no options it indicates when the last backup was created. For example:
> emborg due backup was performed 19 hours ago.
Adding the –days option results in the message only being printed if the backup has not been performed within the specified number of days. Adding the –email option results in the message being sent to the specified address rather than printed. This allows you to run the due command from a cron script in order to send your self reminders to do a backup if one has not occurred for a while.
Extract
You extract a file or directory from an archive using:
emborg extract home/ken/bin
Use manifest to determine what path you should specify to identify the desired file or directory (they will paths relative to /). Thus, the paths should look like absolute paths with the leading slash removed. The paths may point to directories, in which case the entire directory is extracted. It may also be a glob pattern.
If you do not specify an archive or date, the most recent archive is used. You can extract the version of a file or directory that existed on a particular date using:
emborg extract --date 2015-04-01 home/ken/bin
Or, you can extract the version from a particular archive using:
emborg extract --archive kundert-2018-12-05T12:54:26 home/ken/bin
The extracted files are placed in the current working directory within their original hierarchy. Thus, the above commands create the file:
./home/ken/bin
Help
Show information about Emborg:
emborg help
You can ask for help on a specific command or topic with:
emborg help <topic>
For example:
emborg help extract
Info
This command prints out the locations of important files and directories.
emborg info
Init
Initializes a Borg repository. This must be done before you create your first archive.
emborg init
List
List available archives.
emborg list
Log
Show the logfile from the previous run.
Manifest
Once a backup has been performed, you can list the files available in your archive using:
emborg manifest
If you do not specify an archive, as above, the latest archive is used.
You can explicitly specify an archive:
emborg manifest --archive kundert-2015-04-01T12:19:58
Or you can list the files that existed on a particular date using:
emborg manifest --date 2015-04-01
Mount
Once a backup has been performed, you can mount it and then look around as you would a normal read-only filesystem.
emborg mount backups
In this example, backups acts as a mount point. If it exists, it must be a directory. If it does not exist, it is created.
If you do not specify an archive, as above, all archives are mounted.
You can explicitly specify an archive:
emborg mount --archive kundert-2015-04-01T12:19:58 backups
Or you can mount the files that existed on a particular date using:
emborg mount --date 2015-04-01 backups
You will need to un-mount the repository or archive when you are done with it. To do so, use the umount command.
Prune
Prune the repository of excess archives. You can use the keep_within, keep_last, keep_minutely, keep_hourly, keep_daily, keep_weekly, keep_monthly, and keep_yearly settings to control which archives should be kept. At least one of these settings must be specified to use prune:
emborg prune
Settings
This command displays all the settings that affect a backup configuration. Add ‘-a’ option to list out all available settings and their descriptions rather than the specified settings and their values.
Umount
Un-mount a previously mounted repository or archive:
emborg umount backups rmdir backups
where backups is the existing mount point.
Version
Prints the emborg version.
emborg version
API
Emborg has a simple API that allows you to run borg commands. Here is an example taken from sparekeys that exports the keys from your Borg repository so then can be backed up separately:
from emborg import Emborg with Emborg() as emborg: borg = emborg.run_borg( cmd = 'key export', args = [emborg.destination(), archive / '.config/borg.repokey'] ) if borg.stdout: print(borg.stdout.rstrip())
Emborg takes the config name as an argument, if not given the default config is used. It provides the following useful methods and attributes:
repository
The path to the repository.
destination(archive)
Returns the full path to the archive. If Archive is False or None, then the path to the repository it returned. If Archive is True, then the default archive name as taken from settings file is used. This is only appropriate when creating new repositories.
run_borg(cmd, args, borg_opts, emborg_opts)
Runs a Borg command.
cmd is the desired Borg command (ex: ‘create’, ‘prune’, etc.).
args contains the command line arguments (such as the repository or archive). It may also contain any additional command line options not automatically provided. It may be a list or a string. If it is a string, it is split at white space.
borg_opts are the command line options needed by Borg. If not given, it is created for you by Emborg based upon your configuration settings.
Finally, emborg_opts is a list that may contain any of the following options: ‘verbose’, ‘narrate’, ‘trial-run’, or ‘no-log’.
This function runs the Borg command and returns a process object that allows you access to stdout via the stdout attribute.
run_borg_raw(args)
Runs a raw Borg command without interpretation except for replacing a @repo argument with the path to the repository.
args contains all command line options and arguments except the path to the executable.
borg_options(cmd, emborg_opts)
This function returns the default Borg command line options, those that would be used in run_borg if borg_opts is not set. It can be used when constructing a custom borg_opts.
value(name, default=’’)
Returns the value of a setting from an Emborg configuration. If not set, default is returned.
You can examine the emborg/command.py file for inspiration and examples on how to use the Emborg API.
Overdue
Emborg contains an additional executable, emborg-overdue, that can be run on the destination server to determine whether the backups have been performed recently. It reads its own settings file in ~/.config/emborg/overdue.conf that is also a Python file and may contain the following settings:
default_maintainer (email address -- mail is sent to this person upon failure) default_max_age (hours) dumper (email address -- mail is sent from this person) root (default directory for repositories) repositories (string or array of dicts)
Here is an example config file:
default_maintainer = 'root@continuum.com' dumper = 'dumper@continuum.com' default_max_age = 12 # hours root = '/mnt/borg-backups/repositories' repositories = [ dict(host='mercury (/)', path='mercury-root-root'), dict(host='venus (/)', path='venus-root-root'), dict(host='earth (/)', path='earth-root-root'), dict(host='mars (/)', path='mars-root-root'), dict(host='jupiter (/)', path='jupiter-root-root'), dict(host='saturn (/)', path='saturn-root-root'), dict(host='uranus (/)', path='uranus-root-root'), dict(host='neptune (/)', path='neptune-root-root'), dict(host='pluto (/)', path='pluto-root-root'), ]
The dictionaries in repositories can contain the following fields: host, path, maintainer, max_age. host is a description of the host. It is included in the email that is sent when problems occur to identify the backup. It is a good idea for it to contain both the host name and the source directory being backed up. path is either the archive name or a full absolute path to the archive. If path is an absolute path, it is used, otherwise it is added to the end of root. maintainer is an email address, an email is sent to this address if there is an issue. max_age is the number of hours that may pass before an archive is considered overdue.
repositories can also be specified as a list of dictionaries as follows:
repositories = """ HOST | NAME or PATH | MAINTAINER | MAXIMUM AGE (hours) mercury (/) | mercury-root-root | | venus (/) | venus-root-root | | earth (/) | earth-root-root | | mars (/) | mars-root-root | | jupiter (/) | jupiter-root-root | | saturn (/) | saturn-root-root | | uranus (/) | uranus-root-root | | neptune (/) | neptune-root-root | | pluto (/) | pluto-root-root | | """
If repositories is a string, it is first split on newlines, anything beyond a # is considered a comment and is ignored, and the finally the lines are split on ‘|’ and the 4 values are expected to be given in order. If the maintainer is not given, the default_maintainer is used. If max_age is not given, the default_max_age is used.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.