mreschke-serverbackups 1.0.7

Config driven server backups that feel like a simple backup.py script

Server Backups

Config driven server backups that feel like a simple backup.py script

• PyPi: https://pypi.org/project/mreschke-serverbackups
• Github: https://github.com/mreschke/server-backups

Installation

pip install mreschke-serverbackups

Usage

All backups are initiated from a simple backups.py file. Create this file with:

cd /usr/local/bin
serverbackups init-cli-script

This backups.py can hold your server backup definitions as a dictionary, fully self contained. Or you can choose to use yaml files in /etc/serverbackups to separate each server in its own /etc/serverbackups/config.d file. To initialize a stock /etc/serverbackups directory run

serverbackups init-directory

# Defaults to /etc/serverbackups.  Optionally, set a different directory
serverbackups init-directory --path /etc/someother/dir

Read below for how to modify the configs. Once you have a good configuration setup you can run the backups like so

# Run all servers AND custom python code
./backups.py run --all

# Run all servers with NO custom python code
./backups.py run --servers

# Run a single server (no custom python code)
./backups.py run --server=myserver

# Run all servers defined in the same cluster (no custom python code)
./backups.py run --cluster=webfarm

# Run only custom code in backups.py, NOT the server definition arrays
./backups.py run --custom

Basic Idea

The basic idea is to replace your quick and simple backup bash script with a more comprehensive but still simple and readable backups.py file. This file can backup common items such as files and databases using rsync hard-link style snapshots in a simple config driven manner. Instead of hacking bash scripts to rsync properly you can use a simple config instead. At the end of backups.py after all backups have executed from this simple dictionary/yaml definition you can still code your own functionality right in python as you normally would in a bash script. This gives you the power of a script with the added benefit if easy config driven common backup routines.

Whether you are using inline backups.py dictionaries to define your server or using /etc/serverbackups/config.d/* files the concept of how the system works is the same.

You define an array of server to backup. Each server has many options including source, destination, files to backup and exclude, databases to backup and pre/post scripts to run.

Backup Strategy

I use rsync and ssh for all backups local or remote. To cut down on storage costs I am utilizing rsync hard-link style snapshots. This means your snapshot folder looks full and complete each day (not a diff or incremental look) but in actuality most of those files are hard linked to previous snapshots. This means the storage used on the backup server is only the size of all unique files combined + the ones previously deleted. Total storage is NOT every days worth of files, that would be HUGE. I adapted the code from http://anouar.adlani.com/2011/12/how-to-backup-with-rsync-tar-gpg-on-osx.html to accomplish this. The trick is in the rsync --hard-links --link-dest parameters. This storage saving trick allows you to have a generous purge strategy, keeping years worth of weekly, monthly and yearly backups without sacrificing storage.

Defaults

Most of the settings for a single server are redundant and can therefore be deduplicated in to what I call a defaults config. The defaults config is merged with each server definition in your array. This keeps your server definition small and clean. Defaults and each of your server configs are deep merged which allows you to override small pieces of the defaults for each server as needed.

Defaults are completely optional and are there to help cut down on redundant configuration. If you are simply backing up a single server, just define the entire complete server options and forget a 'defaults' definition.

NOTICE - Sensible Defaults Applied

If you do not define your own 'defaults', a sensible 'defaults' are still applied to keep even a single server definition less verbose. Run serverbackups show-builtin-defaults to see the defaults that are still being applied here. For example you don't need to define the prune rates, ssh port or ssh key if the defaults are acceptable.

Because of the defaults system, your actual server definitions can be much smaller. For example, here is a minimal server definition that inherits from defaults

In yaml format

myloadbalancer:
  source: {ssh: {host: haproxy.example.com}}
  backup:
    files:
      exclude:
        - /var/log/haproxy_info.log

Servers

Each server is defined in a python dictionary or yaml file in /etc/serverbackups/config.d ending in .yml. Before the backups run each server is deep merged with your defaults config to create a full and rich server definition.

Files

The files section of each server definition allows you to backup files and folders obeying the exclude array. The only difference between common and extra is that common was meant to be used in the defaults section while extra was there just for you to add to each of your servers. Technically you can still add common to your server as arrays are APPENDED during the default deep merge algorithm. Exclude is also appended during merge so you can add more excludes per server while maintaining those you defined in defaults.

Scripts

The script section of each server definition allows you to run arbitrary bash scripts on the destination. The output of the scripts is optional. If present, the output is saved to the snapshot folder. This is handy to backup the output of dpkg -l command for example.

There are 2 script sections, preScripts and postScripts. The pre scripts run before any other backups for that particular server. Post scripts run after all backups.

Example. I want to backup my gitlab server. I really don't want to export my repositories and dig into their postgres database manually. Gitlab provides a command gitlab-backup create that does it all for me and saves the backups to /var/opt/gitlab/backups. Using a pre script to run this command before your files definition grabs /var/opt/gitlab/backups is a good example of a pre script. Gitlab does not clean up their own backups so using a post script to remove /var/opt/gitlab/backups/*.gz is a good example of a post script. Both of which can either store the output of the command to the snapshot, or ignore output with output=None.

Push Me Pull You

Serverbackups can run

• local -> local,
• local -> remote (push via SSH and rsync)
• remote -> local (pull via SSH and rsync)

If you just want to backup your laptop to your server, probably the local -> remote is simplest. If you use serverbackups in a corporate environment like I do you probably want a single backup server that pulls backups from all your servers and VMS. Pulling has the advantage of not having to put your SSH keys on the backup server. So all VMs cannot SSH into the backup server. Pulling means the backup server needs access to all VMs but not the other way around.

In my corporate environment I have a single backup server which has SSH access into the root account on every server and VM. I use the file based /etc/serverbackups configuration. Each server has a file in /etc/serverbackups/config.d/someserver.yml. Using single files for each server helps manage the backups when you have hundreds of servers. For best results, keep these backup files in ansible for revision control and easy deployments.

Examples

Standalone backups.py

If you want to define your entire backup plan including the defaults config in a single backup.py

cd /usr/local/bin
serverbackups init-cli-script
chmod a+x backups.py

Edit this new file and adjust line #1 (the shebang to your python3 location

Edit the defaults and servers dictionary adding all your servers as needed.

Creates the following stubbed example

#!/usr/bin/env python
from datetime import date
from mreschke.serverbackups import Backups, cli, log

# Configure logger
log.init({
    'console': {
        'level': 'INFO',  # DEBUG, INFO, WARNING, ERROR, CRITICAL
    },
    'file': {
        'enabled': True,
        'file': '/tmp/backups-{}.log'.format(date.today().strftime('%Y-%m-%d')),
    }
})

# Set per server defaults.  Each server can override parts of this default.
defaults = {
    'enabled': True,
    'cluster': None,

    # Keep all daily up to X days back, last snapshot in a week X weeks back, last snapshot in a month X months back...
    'prune': {
        'keepDaily': 30,
        'keepWeekly': 24,
        'keepMonthly': 60,
        'keepYearly': 10,
    },

    # Rsync options
    'rsync': {
        'verbose': True,
    },

    # Source server connection details
    'source': {
        'location': 'local',  # local, ssh,
        'ssh': {
            # Only used with location=ssh
            'host': 'server.example.com',  # or IP address,
            'port': 22,
            'user': 'root',
            'key': '~/.ssh/id_rsa',
        },
    },

    # Destination server connection details
    'destination': {
        'location': 'local',  # local, ssh,
        'path': '/mnt/backups',
        'ssh': {
            # Only used with location=ssh
            'host': 'server.example.com',  # or IP address,
            'port': 22,
            'user': 'root',
            'key': '~/.ssh/id_rsa',
        },
    },

    # Backup items
    'backup': {
        # Pre scripts run on destination before this servers backups, a good place to prep files for backup
        'preScripts': {
            'gitlab': {'script': 'sudo gitlab-backup create', 'output': 'gitlab-backup.txt', 'enabled': False},
        },

        # Files to backup on the destination.  Use extra in your actual server definition for extra files.
        # Arrays are merged with defaults, so you could append to common and exclude as well
        'files': {
            'common': [
                '/etc/',
                '/usr/local/bin/',
                '/root/',
                '/home/',
                '/var/log/',
                '/var/spool/cron/',
            ],
            'extra': [],
            'exclude': [
                'lost+found',
                '.Trash-1000',
            ],
        },

        # MySQL Databases and Tables
        # Host and port are relative inside the server itself via ssh, not remote
        'mysql': {
            'enabled': False,
            'mysqlCmd': 'mysql',
            #'mysqlCmd': 'docker exec -i mysql mysql',
            'dumpCmd': 'mysqldump',
            'dumpFlags': '--quick --single-transaction --flush-logs',
            #'dumpCmd': 'docker exec -i mysql mysqldump',
            'host': '127.0.0.1',
            'port': 3306,
            'user': 'root',
            'password': 'unknown',
            # All databases minus exclusions
            'dbs': '*',
            # A list of databases (all tables)
            # 'dbs': ['db1', 'db2'],
            # A list of database with a selection of tables
            # 'dbs': [
            #     {'name': 'db1', 'tables': ['table1', 'table2']},
            #     {'name': 'db2', 'tables': '*'},
            # ],
            'excludeDbs': [
                'information_schema',
                'performance_schema',
                'lost+found',
                '#mysql50#lost+found',
            ],
        },

        # Post scripts run on destination after this servers backups, a good place to cleanup files
        'postScripts': {
            'dpkg': {'script': 'dpkg -l | gzip', 'output': 'dpkg.gz', 'enabled': False},
        },
    },
}

# Define each server to backup (each will be merged with defaults config)
servers = {
    'myserver.example.com': {
        'enabled': True,
        'cluster': 'localhost',
        # Example of local -> local backup
        'source': {
            'location': 'local',
        },
        'destination': {
            'location': 'local',
        },
        'backup': {
            'preScripts': {
                #'gitlab': {'enabled': True}
            },
            'mysql': {
                'enabled': True,
                'mysqlCmd': 'mysql',
                #'mysqlCmd': 'docker exec -i mysql mysql',
                'dumpCmd': 'mysqldump',
                'dumpFlags': '--quick --single-transaction --flush-logs',
                #'dumpCmd': 'docker exec -i mysql mysqldump',
                'host': '127.0.0.1',
                'port': 3306,
                'user': 'root',
                'password': 'password',
                'dbs': '*',
            },
            'postScripts': {
                # Export dpkg package list of Linux Debian
                'dpkg': {'enabled': True},
                # Example of backing up postgres database
                #'fusionauth': {'script': 'cd /tmp/ && sudo -u postgres pg_dump fusionauth | gzip', 'output': 'fusionauth.sql.gz', 'enabled': True},
                # Example of capturing script output to snapshot
                #'test2': {'script': 'ls -Hhal /etc/', 'output': 'etc.txt', 'enabled': True},
            }
        },
    },
    'myserver2.example.com': ...
}

# Initialize Backup Class
# If you are defining servers in a dictionary above
#backups = Backups(servers=servers)

# If you are defining both servers and defaults above
backups = Backups(servers=servers, defaults=defaults)

# If you are using a config.d directory (/etc/serverbackups by default)
#backups = Backups()
#backups = Backups(config_path='/etc/alternative/dir')

# Run Backups
cli.start(backups)


# Custom backups
if not cli.allowcustom(): cli.handle(); exit()
log.header("Running custom backups from " + __file__)
#################################################
# Add your custom backups code here
#################################################
print('custom stuff here, any python code you want')


# Handle CLI requests
cli.handle()

Running backups.py can be read at the top of this readme.

Directory based /etc/serverbackups/config.d

If you have a lot of servers, a single backup.py with a massive server dictinary soon becomes a bit unwieldy. As an alternative you can define your defaults and server configs as separate files in a directory (normally /etc/serverbackups)

Create the directory with a example defaults.yml and example config.d/myserver.example.com file

serverbackups init-directory

# Defaults to /etc/serverbackups.  Optionally, set a different directory
serverbackups init-directory --path /etc/someother/dir

You still need a backups.py which initiates the backups, see the top of the readme to create one.

Now your backups.py will look something like this

#!/usr/bin/env python
from datetime import date
from mreschke.serverbackups import Backups, cli, log

# Configure logger
log.init({
    'console': {
        'level': 'INFO',  # DEBUG, INFO, WARNING, ERROR, CRITICAL
    },
    'file': {
        'enabled': True,
        'file': '/tmp/backups-{}.log'.format(date.today().strftime('%Y-%m-%d')),
    }
})

# Initialize Backup Class
backups = Backups()

# Or if using an alternate directory
#backups = Backups(config_path='/etc/alternative/dir')

# Run Backups
cli.start(backups)


# Custom backups
if not cli.allowcustom(): cli.handle(); exit()
log.header("Running custom backups from " + __file__)
#################################################
# Add your custom backups code here
#################################################
print('custom stuff here, any python code you want')


# Handle CLI requests
cli.handle()

Running backups.py can be read at the top of this readme.

As a python module/package

Serverbackups can also be used right inside your own python code as a module. No need for a backups.py though you can still optionally use /etc/serverbackups style files if needed.

Imagine as part of your own python program, you simply want to backup a file or database. Using serverbackups as a module import can help with this.

Example of a simple

from datetime import date
from mreschke.serverbackups import Backups, log

# Optionally configure logger, but this app may have its own log settings
log.init({
    'console': {
        'level': 'INFO',  # DEBUG, INFO, WARNING, ERROR, CRITICAL
    },
    'file': {
        'enabled': True,
        'file': '/tmp/backups-{}.log'.format(date.today().strftime('%Y-%m-%d')),
    }
})

# In this example I am NOT defining a defaults config because I may just be
# backing up a single server.  Defaults are used to deduplicate common items
# in multiple servers.  If you only have one server, just define the entire config
# in the server itself

# NOTE sensible defaults are still applied to keep even a single server definition
# less verbose.  Run `serverbackups show-builtin-defaults` to see the defaults that are
# still being applied here

# Define each server to backup (each will be merged with defaults config)
# Source and Destination default to 'local'
servers = {
    'myserver.example.com': {
        'destination': {'path': '/mnt/Backups'},
        'backup': {
            'files': {
                'common': [
                    '/etc/',
                ],
                'exclude': [
                    'lost+found',
                    '.Trash-1000',
                ],
            },
        },
    },
}

# Initialize Backup Class
# If you are defining servers in a dictionary above, without a separate defaults
backups = Backups(servers=servers)

# If you are defining both servers and defaults above
#backups = Backups(servers=servers, defaults=defaults)

# If you are using a config.d directory (/etc/serverbackups by default)
#backups = Backups()
#backups = Backups(config_path='/etc/alternative/dir')

# Run backups
# Can just run them all
backups.run()

# Or can flip through each and add your own logic per server if needed
#for server in backups.servers:
    #backups.run(server)

Pruning

Rsync hard-link style snapshots are pruned every time the backups run.

The prune server or defaults config determines the prune rates

# Keep all daily up to X days back, last snapshot in a week X weeks back, last snapshot in a month X months back...
'prune': {
    'keepDaily': 30,
    'keepWeekly': 24,
    'keepMonthly': 60,
    'keepYearly': 10,
},

Lets say we are running the backups twice a day. The above configuration means we keep each daily backup for 30 days (meaning 60 backups since 2 a day).

Keep the LAST/LATEST weekly backup. This means the last backup on Saturday of each week for 24 weeks.

Keep the LAST/LATEST monthly backup. This means the last backup on the 31st of each month for 60 months (5 years)

Keep the LAST/LATEST yearly backup. This all 12/31 backups of each year will be saved.

The default prune settings may seem like a waste of storage, but read my Backup Strategy notes above. Using rsync hard-link style snapshots saves space by using unix hard links to deduplicate files efficiently.