Skip to main content

Linux-based backup utility

Project description

secondshot

This is a command-line tool and library for managing filesystem backups on a local network. Each incremental backup is made available according to rotation rules defined in this 2004 article by Mike Rubel.

Reliability and simplicity of this are inherently provided by rsnapshot and rsync, which are used without modification. What this tool adds is storage of metadata in a local SQL database, along with functions to store and verify checksums for files in each saveset.

The underlying rsync file transfer requires root permissions on both the source and destination. To improve security, a perl script called rrsync is provided; see Guy Rutenberg's explanation.

Example crontab and configuration files can be found here under bin/ and etc/.

This tool is distributed as both a Python package at pypi.org, and as a Docker image at dockerhub.com. Use whichever distro is convenient for you.

Understanding Rotation

Perhaps the easiest way to visualize how savesets are rotated is to look at the top-level directory of stored savesets; here's an example from the author's system for the rotation sequence hourly / daysago / weeksago / monthsago / semiannually:

drwxr-xr-x 6 root care 4096 Aug 25 10:36 daysago.0
drwxr-xr-x 6 root care 4096 Aug 24 10:33 daysago.1
drwxr-xr-x 6 root care 4096 Aug 23 10:34 daysago.2
drwxr-xr-x 6 root care 4096 Aug 22 10:34 daysago.3
drwxr-xr-x 6 root care 4096 Aug 21 10:33 daysago.4
drwxr-xr-x 6 root care 4096 Aug 20 10:33 daysago.5
drwxr-xr-x 6 root care 4096 Aug 26 02:35 hourly.0
drwxr-xr-x 6 root care 4096 Aug 25 17:37 hourly.1
drwxr-xr-x 6 root care 4096 Jul  1 09:38 monthsago.0
drwxr-xr-x 6 root care 4096 Jun  4 09:36 monthsago.1
drwxr-xr-x 6 root care 4096 May 28  2018 monthsago.2
drwxr-xr-x 6 root care 4096 Apr 31  2018 monthsago.3
drwxr-xr-x 6 root care 4096 Mar  3  2018 monthsago.4
drwxr-xr-x 6 root care 4096 Feb 27  2018 monthsago.5
drwxr-xr-x 5 root care 4096 Jan 30  2018 semiannually.0
drwxr-xr-x 4 root care 4096 Aug 28  2017 semiannually.1
drwxr-xr-x 6 root care 4096 Aug 19 10:33 weeksago.0
drwxr-xr-x 6 root care 4096 Aug 12 09:49 weeksago.1
drwxr-xr-x 6 root care 4096 Aug  5 09:52 weeksago.2
drwxr-xr-x 6 root care 4096 Jul 30 09:49 weeksago.3

Under each of these locations are complete rsync'ed directories for each backed up host, which you can restore by any file-management tool you're familiar with. Your target volume needs to be formatted with enough storage and inodes to handle the retention schedule you define. Files that haven't changed between defined intervals are hard-linked to save space.

Docker Image

Find the docker-compose.yml example (here:

  • If you have an existing database, create a database and add settings for docker-compose; otherwise leave those out to automatically generate data with sqlite
  • Define mount points for persistent data
  • Generate a 4096-bit rsa key, add its id_rsa.pem to your secrets store

Variables

The docker image can be customized with these environment variables:

Variable Default Description
CRON_MINUTE 30 cron schedule (minutes past hour)
CRON_HOUR 0,8,16 cron schedule (hours)
DBHOST db00 db host
DBNAME secondshot db name
DBPORT 3306 db port
DBUSER bkp db username
DBTYPE sqlite db type, such as mysql+pymysql
TZ UTC time zone

Secrets

Name Description
secondshot-db-password SQL database password
secondshot-rsync-sshkey ssh private key for rsync

What You Won't Find

To keep this tool simple, there are a few things that it explicitly does not do:

  • Restore: the target is a regular filesystem from which you can perform restores without special tools
  • No at-rest encryption; if you want that, format the target using LUKS or another full-disk encryption tool
  • Cloud storage like S3 or B2, which don't provide POSIX filesystem semantics
  • Block-level de-duplication; you will want another tool to backup large files that require it

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

secondshot-0.9.2.tar.gz (35.1 kB view details)

Uploaded Source

Built Distribution

secondshot-0.9.2-py2-none-any.whl (28.7 kB view details)

Uploaded Python 2

File details

Details for the file secondshot-0.9.2.tar.gz.

File metadata

  • Download URL: secondshot-0.9.2.tar.gz
  • Upload date:
  • Size: 35.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/40.3.0 requests-toolbelt/0.8.0 tqdm/4.25.0 CPython/2.7.12

File hashes

Hashes for secondshot-0.9.2.tar.gz
Algorithm Hash digest
SHA256 970d12f427436e14604a2ea0ce78265ba64f196f5e4f5275b45eaf34eaa35e72
MD5 0110d615b2995b99d13937e08dcc6fc8
BLAKE2b-256 cb171a383c3a1d9bf7a48adfedebab12714ba8ec580be639e3cea32db15fd974

See more details on using hashes here.

File details

Details for the file secondshot-0.9.2-py2-none-any.whl.

File metadata

  • Download URL: secondshot-0.9.2-py2-none-any.whl
  • Upload date:
  • Size: 28.7 kB
  • Tags: Python 2
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/40.3.0 requests-toolbelt/0.8.0 tqdm/4.25.0 CPython/2.7.12

File hashes

Hashes for secondshot-0.9.2-py2-none-any.whl
Algorithm Hash digest
SHA256 4154e664d7f1b00ef7fbdd9c4cc6c4fd91aa4d04321297d6cffba97e2df8bb81
MD5 7b6c3ce895ae45da20bec66f021f7607
BLAKE2b-256 ab11f27dada0b30b738d3c34546c488fc747fca9c93c6b4b4fbc9d24d771a6f9

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page