backup-pro 1.1.0

A comprehensive backup tool that makes the life easier when backing up/restoring configurations, files, packages of the system.

Backup Pro

A comprehensive backup tool that makes the life easier when backing up/restoring configurations, files, packages of the system. Some features:

• Tracking installed packages (e.g. apt packages)

• Tracking configurations (e.g. gsettings)

• Tracking paths on the filesystem

• Capability of excluding specified subdirectories

• Environment variable support on tracked paths (e.g. $USER)

• Scanning for filesystem index snapshots

• Calculating diffs between scanned snapshots

Requirements

Python >= 3.10 is required. (CPython and PyPy are both supported)

ujson is an optional dependency for CPython for the sake of faster JSON operations.

Installation

Backup Pro can be either installed with pip:

python3 -m pip install backup-pro

Or it can be installed from the source:

git clone https://github.com/simsekhalit/backup-pro.git
python3 -m pip install -e ./backup-pro

Manual

$ python3 -m backup_pro --help
usage: backup-pro [-h] [-c CONF_DIR] [-t TARGET_DIR] COMMAND ...

A comprehensive backup tool that makes the life easier when backing up/restoring configurations, files, packages of the system.

positional arguments:
  COMMAND
    backup              backup the system
    check               check configurations and packages
    diff                calculate diff using the previous scans
    restore             restore the system to the previous backup point
    scan                scan the system to generate filesystem index snapshot that is used by the diff command
    settings            change settings of the backup-pro

options:
  -h, --help            show this help message and exit
  -c CONF_DIR, --conf-dir CONF_DIR
                        folder that contains the backup-pro configurations. defaults to the current directory
  -t TARGET_DIR, --target-dir TARGET_DIR
                        folder that contains the target backup file. defaults to the current directory

For more information: https://github.com/simsekhalit/backup-pro

Getting Started

There are two global options that are essential for the Backup Pro:

• -c --conf-dir: The folder that contains all Backup Pro configurations (e.g. settings, metadata, indexes, etc.). Defaults to the current directory.

• -t --target-dir: The folder that contains the target backup file as backup-pro-data.zip. Defaults to the current directory.

Backup Operation

Firstly, tracked paths should be configured in order to specify which paths should be backed up. Each tracked path has a strategy that can be one of 'auto', 'backup-only', 'manual'.

• auto means that path should be automatically handled during backup/restore processes. This is the default strategy.

• backup-only means that path should only be automatically backed up but never be restored. This is mostly for archiving purposes.

• manual means that path should be backed up automatically but restored in a manual way. Meld is supported for manual restoration process.

A tracked path can be added as the following:

python3 -m backup_pro settings add-tracked-path '$HOME/.config'

Additionally, strategy can be specified as well:

python3 -m backup_pro settings add-tracked-path --strategy manual '$HOME/.ssh'

:information_source:

Please note that environment variables (e.g. $HOME, $USER) are supported. When they are given in a shell escaped way (e.g. '$HOME'), Backup Pro understands and treats them as variables. For example if $HOME variable is change at the moment of restore operation, home directory is extracted to the new value of the variable.

It's possible to exclude some subdirectories of given tracked paths:

python3 -m backup_pro settings add-tracked-path "/opt/myapp"
python3 -m backup_pro settings add-archive-exclude-path "/opt/myapp/cache"

A regex pattern can be specified to exclude paths during backup. For example, following command excludes python cache files (*.pyc, *.pyo):

python3 -m backup_pro settings add-archive-exclude-pattern '.+\.py[co]$'

After tracked paths are all set, backup operation can be triggered:

python3 -m backup_pro backup

Above command results to a file named as backup-pro-data.zip under the path that is specified with the --target-dir.

Check Operation

Every time a backup command is executed, all installed packages and configurations are silently scanned behind the scene. If new packages are installed (e.g. with apt install) or some configurations are changed (e.g. with gsettings set), check command detects them and asks how should the changes be handled.

Handling packages:

$ python3 -m backup_pro check
Choose package strategy:
d: mark as dependency
i: ignore
r: remove
t: track
S: skip

apt/gparted is detected
[d/i/r/t/S]

• If mark as dependency is selected, then apt-mark auto gparted command is going to be executed during restoration.
• If ignore is selected, then gparted is going to be ignored and no action is going to be taken.
• If remove is selected, then gparted is going to be removed with apt purge gparted during restoration.
• If track is selected, then gparted is going to be installed with apt install gparted during restoration.
• If skip is selected, then this package is skipped for now and it's going to be brought up again in the next check command.

Handling configurations:

$ python3 -m backup_pro check
Choose configuration strategy:
i: ignore
t: track
S: skip

gsettings/org.gnome.FileRoller.Listing.sort-method
<'size'
>'name'
[i/t/S]

• If ignore is selected, then this configuration is going to be ignored and no action is going to be taken.
• If track is selected, then this configuration is going to be restored with the value that was recorded with the latest backup operation.

Restore Operation

After the backup operation is executed, a file named as backup-pro-data.zip is generated under the path that is specified with the --target-dir. restore command restores the system using that file:

python3 -m backup_pro restore

A dry run can be executed in order to see what is going to happen during restore without actually changing anything:

python3 -m backup_pro --dry-run restore

If there are tracked paths with the manual strategy, output of the restore command is going to contain lines as the following:

[M] /tmp/backup-pro-data.tmp123456/opt/mydata /opt/mydata

These paths should be restored manually. Backup Pro supports running meld for each manually tracked path if DIFF_CHECKER variable is set to meld:

export DIFF_CHECKER=meld
python3 -m backup_pro restore

Furthermore, it can be forced to restore each file manually in an interactive way regardless of whether their strategy is auto or manual:

export DIFF_CHECKER=meld
python3 -m backup_pro restore --interactive

Scan & Diff Operations

Backup Pro provides a highly optimized mechanism for indexing files and finding changes over time.

Running a basic scan on the filesystem:

python3 -m backup_pro scan

To clean up old redundant index metadata, you can prune the history using the --prune argument (which accepts the exact same flexible time formats as --since):

# Prune old metadata dynamically during a standalone scan
python3 -m backup_pro scan --prune "+86400"                 # Delete metadata older than 24 hours

# Prune old metadata while also running a diff concurrently
python3 -m backup_pro diff --prune "2024-01-01" /etc        

After files are modified, you can calculate directory diffs immediately. The diff logic works across all indexed history:

# Diff changes from the last scan automatically
python3 -m backup_pro diff

# Scan and diff immediately in one command
python3 -m backup_pro diff --scan

# Filter diff by specific directories
python3 -m backup_pro diff /opt /etc

You can target exactly how far back you want to check for changes using the --since argument (accepts epoch times, datetime strings, or +SECONDS deltas):

python3 -m backup_pro diff --since "+3600"                  # Changes in the last hour
python3 -m backup_pro diff --since "2024-01-15 10:30"       # Changes since specific date
python3 -m backup_pro diff /etc --since 1691491328          # Changes in /etc since epoch time

Helpful Tip

Generally, system backup/restore tools require root permissions for reading from/writing to system directories. Using virtual environments with the sudo command can be tricky at that point. For a smoother experience, an executable file can be created as:

sudo touch /usr/local/bin/backup_pro
sudo chmod 755 /usr/local/bin/backup_pro

Following content can be written to the file using the favourite text editor.

#!/usr/bin/env bash
set -e

if [ "$UID" != 0 ]; then
    exec sudo -E $(readlink -f $0) "$@"
fi

CONF_DIR="$HOME/.config"
TARGET_DIR="/opt"
VENV_PATH="$HOME/venv"

export DIFF_CHECKER="meld"

source "$VENV_PATH/bin/activate"
python3 -m backup_pro -c "$CONF_DIR" -t "$TARGET_DIR" "$@"

Please remember to modify variables according to your own setup.