emborg 1.0.0

Borg front end.

emborg -- Encrypted Backups to a Remote Server
==============================================

| Version: 1.0.0
| Released: 2019-01-09
|

Emborg is a simple command line utility to orchestrate backups. It is built as
a front-end to Borg, a powerful and fast deduplicating backup program. With
Emborg, you specify all the details about your backups once in advance, and
then use a very simple command line interface for your day-to-day activities.
The details are contained in ~/.config/emborg. That directory contains a file
(settings) that contains shared settings, and then another file for each backup
configuration you have.

Use of Emborg does not preclude the use of Borg directly on the same repository.
The philosophy of Emborg is to provide commands that you would use often and in
an interactive manner with the expectation that you would use Borg directly for
the remaining commands.


Getting Started
===============

Many Linux distributes include *Borg* in their package managers. In Fedora it is
referred to as *borgbackup*. In this case you would install *borg* by running
the following::

sudo dnf install borgbackup

Alternately, you can download a precompiled version from `Borg Github Releases
<https://github.com/borgbackup/borg/releases/>`_. You can do so with following
commands (they will need to be adjusted for to get the latest version)::

cd ~/bin
wget https://github.com/borgbackup/borg/releases/download/1.1.8/borg-linux64
wget https://github.com/borgbackup/borg/releases/download/1.1.8/borg-linux64.asc
gpg --recv-keys "FAF7B393"
gpg --verify borg-linux64.asc
rm borg-linux64.asc
chmod 755 borg-linux64

Download and install *Emborg* as follows::

pip3 install --user emborg

Or, if you want the development version, use::

git clone https://github.com/KenKundert/emborg.git
pip3 install --user ./emborg

Then you will need to create your *Emborg* settings directory (~/.config/emborg)
and create a shared settings file 'settings' and then one or more files, one for
each configuration you want. Normally people have just two files, the shared
settings file and one configuration file, perhaps named 'home' because it used
to back up your home directory. However, you may wish to have a second
configuration dedicated to creating snapshots of your files every 15 minutes or
so. These snapshots may be kept locally and only for a day or so while your
primary backups are kept remotely and kept long term.

Settings may be placed in either the shared settings file or the configuration
specific file. The ones placed in the configuration specific file dominate.
The shared settings file must contain at least one setting, *configurations*,
which is a list of the available configurations.

You can find descriptions of all available settings with::

emborg settings -a

There are certain settings that are worth highlighting.

repository:

The destination for the backups. A typical value might be::

repository = 'archives:/mnt/backups/{host_name}-{user_name}-{config_name}'

where in this example 'archives' is the hostname and /mnt/backups is the
absolute path to the directory that is to contain your Borg repositories,
and {host_name}-{user_name}-{config_name} is the directory to contain this
repository.

archive:

A template that is used when creating the archive name. A typical value
might be::

archive = '{host_name}-{{now}}'

encryption:

The encryption mode that is used when first creating the repository. Common
values are *none*, *authenticated*, *repokey*, and *keyfile*. The
repository is encrypted if you choose *repokey* or *keyfile*. In either case
the passphrase you provide does not encrypt repository. Rather the
repository is encrypted using a key that is randomly generated by *Borg*.
You passphrase encrypts the key. Thus, to restore your files you will need
both the key and the passphrase. With *repokey* your key is copied to the
repository, so it can be used with trusted repositories. Use *keyfile* if
the remote repository is not trusted. It does not copy the key to the
repository, meaning that it is extremely important for you export the key
using 'borg key export' and keep a copy along with the passphrase.

passphrase:

The passphrase used when encrypting the encryption key. This is used as an
alternative to *avendesora_account*. Be sure to make the file that contains
it unreadable by others.

avendesora_account:

The name of the Avendesora account used to hold the passphrase for the
encryption key. Use this as an alternative to *passphrase*. This keeps your
passphrase out of your settings file, but requires that GPG agent be
available and loaded with your private key.

src_dirs:

The list of directories to be backed up. A typical value might be::

src_dirs = '~'.split()

excludes:

A list of files to exclude from the backups. typical value might be::

excludes = '''
~/tmp
~/.local
~/.cache
~/.mozilla
~/.thunderbird
~/.config/google-chrome*
~/.config/libreoffice
~/**/__pycache__
~/**/*.pyc
~/**/.*.swp
~/**/.*.swo
'''.split()

Once you have set up your configuration directory, you will need to create your
repository. To do so, assure that the parent directory of your repository exists
and is writable on the remote server. Then run::

emborg init

Once you have done that you can create your first backup using:

emborg create

Then you can run any of the commands documented below.


Configuration
=============

Shared settings go in ~/.config/emborg/settings. This is a Python file that
contains values needed by Emborg. It might look like the following::

Shared Settings
---------------

Shared settings go in ~/.config/emborg/settings. This is a Python file that
contains values needed by Emborg. It might look like the following::

default_configuration = 'home' # default backup configuration
configurations = 'home websites' # available backup configurations
avendesora_account = 'borg-backup' # Avendesora account name (holds passphrase for encryption key)
passphrase = None # passphrase to use (if specified, Avendesora is not used)
encryption = 'keyfile' # encryption method
prune_after_create = True # run prune as the last step of an archive creation
check_after_create = True # run check as the last step of an archive creation
notify = "[e-mail]" # email address to notify when things go wrong
notifier = 'notify-send -u normal {prog_name} "{msg}"'
# program used to send realtime notifications
# generally you use notify or notifier, but not both
# use notifier for interactive backups
# and notify for scheduled backups
# notification program
remote_ratelimit = 2000 # bandwidth limit in kbps
umask = '077' # umask to use when creating the archives
keep_within = '1d' # keep all archives within this time interval
keep_hourly = '48' # number of hourly archives to keep
keep_daily = '7' # number of daily archives to keep
keep_weekly = '4' # number of weekly archives to keep
keep_monthly = '12' # number of weekly archives to keep
keep_yearly = '2' # number of weekly archives to keep

If you encrypt your backups, you can specify the encryption key in this file as
*passphrase*. In this case, you should be careful to assure the file is not
readable by others (chmod 600 settings). Alternatively, you can use `Avendesora
<https://avendesora.readthedocs.io>`_ to securely hold your key by specifying
the Avendesora account name of the key to *avendesora_account*.


Configuration Settings
----------------------

Each backup configuration must have a settings file in ~/.config/emborg. The
name of the file is the name of the backup configuration. It might look like
the following::

repository = 'archives:/mnt/backups/{host_name}/{config_name}'
# remote directory for repository
archive = '{host_name}-{{now}}' # naming pattern used for the archives
# May contain {<name>} where <name> may be any of host_name, user_name,
# prog_name config_name, or any of the user specified settings.
# Double up the braces to specify parameters that should be interpreted
# by borg rather than by emborg.
src_dirs = ['~', '/etc'] # absolute path to directory to be backed up
excludes = '''
~/tmp
~/**/.hg
~/**/.git
~/**/*.pyc
~/**/.*.swp
~/**/.*.swo
'''.split() # list of glob strings of files or directories to skip
one_file_system = False
exclude_caches = True

# commands to be run before and after backups (run from working directory)
run_before_backup = [
'./clean-home >& clean-home.log',
# remove the detritus before backing up
]
run_after_backup = [
'./rebuild-manpages > /dev/null',
# rebuild my man pages, they were deleted in clean
]

# if set, this file or these files must exist or backups will quit with an error
must_exist = '~/doc/thesis'

String values may incorporate other string valued settings. Use braces to
interpolate another setting. In addition, you may interpolate the configuration
name ('config_name'), the host name ('host_name'), the user name ('user_name')
or Emborg's program name ('prog_name'). An example of this is shown in both
*repository* and *archive* above.


Precautions
===========

You should assure you have a backup copy of the encryption key and its
passphrase in a safe place (run 'borg key export' to extract the encryption
keys). This is very important. If the only copy of the encryption credentials
are on the disk being backed up, then if that disk were to fail you would not be
able to access your backups. I recommend the use of `sparekeys
<https://github.com/kalekundert/sparekeys>`_ as a way of assuring that you
always have access to the essential information, such as your Borg passphrase
and keys, that you would need to get started after a catastrophic loss of your
disk.

If you keep the passphrase in the emborg file, you should set its permissions so
that it is not readable by others::

chmod 700 settings

Better is to simply not store the passphrase in the emborg script. This can be
arranged if you are using `Avendesora
<https://github.com/KenKundert/avendesora>`_, which is a flexible password
management system. The interface to Avendesora is already built in to *Emborg*,
but its use is optional (it need not be installed). However, be sure to keep
a copy of your *Borg* passphrase in a safe place, preferably with the exported
encryption key. Remember, that if you lose all of your files, you need your
*Borg* password before you will be able to recover your Avendesora files.
Alternatively, you can backup your Avendesora files outside of Borg.

It is also best, if it can be arranged, to keep your backups at a remote site so
that your backups do not get destroyed in the same disaster, such as a fire or
flood, that claims your original files. One option is `rsync.net
<https://www.rsync.net/products/attic.html>`_.

Also, it is a good idea to practice a recovery. Pretend that you have lost all
your files and then see if you can do a restore from backup. Doing this and
working out the kinks before you lose your files can save you if you ever do
lose your files.


Borg
====

*Borg* has considerably more power than what is exposed with *emborg*. You may
use it directly when you need that power. More information about *Borg* can be
found at `borgbackup on readthedocs <https://borgbackup.readthedocs.io/en/stable/index.html>`_.


Commands
========

BreakLock
---------

This command breaks the repository and cache locks. Please use carefully and
only while no borg process (on any machine) is trying to access the Cache or the
Repository.

::

emborg break-lock
emborg breaklock


Create
------

This creates an archive in an existing repository. An archive is a snapshot of
your files as they currently exist. Borg is a de-duplicating backup program, so
only the changes from the already existing archives are saved.

::

emborg create

Before creating your first archive, you must use the *init* command to
initialize your repository.


Check
-----

Check the integrity of the repository and its archives.


Configs
-------

List the available backup configurations. Each configuration will correspond to
a settings file in your configuration directory (~/.config/emborg). Settings
common to all your configurations should be placed in ~/.config/emborg/settings.
You can see available configs using::

emborg configs

To run a command on a specific configuration, add --config=<cfg> or -c cfg
before the command. For example::

emborg -c home create


Diff
----

Shows the differences between two archives::

> emborg diff kundert-2018-12-05T19:23:09 kundert-2018-12-04T17:41:28


Due
---

When run with no options it indicates when the last backup was created. For
example::

> emborg due
backup was performed 19 hours ago.

Adding the --days option results in the message only being printed if the backup
has not been performed within the specified number of days. Adding the --email
option results in the message being sent to the specified address rather than
printed. This allows you to run the *due* command from a cron script in order
to send your self reminders to do a backup if one has not occurred for a while.


Extract
-------

You extract a file or directory from an archive using::

emborg extract home/ken/bin

Use manifest to determine what path you should specify to identify the desired
file or directory (they will paths relative to /). Thus, the paths should look
like absolute paths with the leading slash removed. The paths may point to
directories, in which case the entire directory is extracted. It may also be
a glob pattern.

If you do not specify an archive or date, the most recent archive is used. You
can extract the version of a file or directory that existed on a particular date
using::

emborg extract --date 2015-04-01 home/ken/bin

Or, you can extract the version from a particular archive using::

emborg extract --archive kundert-2018-12-05T12:54:26 home/ken/bin

The extracted files are placed in the current working directory within their
original hierarchy. Thus, the above commands create the file::

./home/ken/bin


Help
----

Show information about Emborg::

emborg help

You can ask for help on a specific command or topic with::

emborg help <topic>

For example::

emborg help extract


Info
----

This command prints out the locations of important files and directories.

::

emborg info


Init
----

Initializes a Borg repository. This must be done before you create your first
archive.

::

emborg init


List
----

List available archives.

::

emborg list


Manifest
--------

Once a backup has been performed, you can list the files available in your
archive using::

emborg manifest

If you do not specify an archive, as above, the latest archive is used.

You can explicitly specify an archive::

emborg manifest --archive kundert-2015-04-01T12:19:58

Or you can list the files that existed on a particular date using::

emborg manifest --date 2015-04-01


Mount
-----

Once a backup has been performed, you can mount it and then look around as you
would a normal read-only filesystem.

::

emborg mount backups

In this example, backups acts as a mount point. If it exists, it must be
a directory. If it does not exist, it is created.

If you do not specify an archive, as above, all archives are mounted.

You can explicitly specify an archive::

emborg mount --archive kundert-2015-04-01T12:19:58 backups

Or you can mount the files that existed on a particular date using::

emborg mount --date 2015-04-01 backups

You will need to un-mount the repository or archive when you are done with it.
To do so, use the *umount* command.


Prune
-----

Prune the repository of excess archives. You can use the *keep_within*,
*keep_last*, *keep_minutely*, *keep_hourly*, *keep_daily*, *keep_weekly*,
*keep_monthly*, and *keep_yearly* settings to control which archives should be
kept. At least one of these settings must be specified to use *prune*::

emborg prune


Settings
--------

This command displays all the settings that affect a backup configuration.
Add '-a' option to list out all available settings and their descriptions rather
than the specified settings and their values.


Umount
------

Un-mount a previously mounted repository or archive::

emborg umount backups
rmdir backups

where *backups* is the existing mount point.


Version
-------

Prints the *emborg* version.

::

emborg version