Skip to main content

Linux-based backup utility

Project description

secondshot

This is a command-line tool and library for managing filesystem backups on a local network. Each incremental backup is made available according to rotation rules defined in this 2004 article by Mike Rubel.

Reliability and simplicity of this are inherently provided by rsnapshot and rsync, which are used without modification. What this tool adds is storage of metadata in a local SQL database, along with functions to store and verify checksums for files in each saveset.

The underlying rsync file transfer requires root permissions on both the source and destination. To improve security, a perl script called rrsync is provided; see Guy Rutenberg's explanation.

Example crontab and configuration files can be found here under bin/ and etc/.

This tool is distributed as both a Python package at pypi.org, and as a Docker image at dockerhub.com. Use whichever distro is convenient for you.

Understanding Rotation

Perhaps the easiest way to visualize how savesets are rotated is to look at the top-level directory of stored savesets; here's an example from the author's system for the rotation sequence hourly / daysago / weeksago / monthsago / semiannually:

drwxr-xr-x 6 root care 4096 Aug 25 10:36 daysago.0
drwxr-xr-x 6 root care 4096 Aug 24 10:33 daysago.1
drwxr-xr-x 6 root care 4096 Aug 23 10:34 daysago.2
drwxr-xr-x 6 root care 4096 Aug 22 10:34 daysago.3
drwxr-xr-x 6 root care 4096 Aug 21 10:33 daysago.4
drwxr-xr-x 6 root care 4096 Aug 20 10:33 daysago.5
drwxr-xr-x 6 root care 4096 Aug 26 02:35 hourly.0
drwxr-xr-x 6 root care 4096 Aug 25 17:37 hourly.1
drwxr-xr-x 6 root care 4096 Jul  1 09:38 monthsago.0
drwxr-xr-x 6 root care 4096 Jun  4 09:36 monthsago.1
drwxr-xr-x 6 root care 4096 May 28  2018 monthsago.2
drwxr-xr-x 6 root care 4096 Apr 31  2018 monthsago.3
drwxr-xr-x 6 root care 4096 Mar  3  2018 monthsago.4
drwxr-xr-x 6 root care 4096 Feb 27  2018 monthsago.5
drwxr-xr-x 5 root care 4096 Jan 30  2018 semiannually.0
drwxr-xr-x 4 root care 4096 Aug 28  2017 semiannually.1
drwxr-xr-x 6 root care 4096 Aug 19 10:33 weeksago.0
drwxr-xr-x 6 root care 4096 Aug 12 09:49 weeksago.1
drwxr-xr-x 6 root care 4096 Aug  5 09:52 weeksago.2
drwxr-xr-x 6 root care 4096 Jul 30 09:49 weeksago.3

Under each of these locations are complete rsync'ed directories for each backed up host, which you can restore by any file-management tool you're familiar with. Your target volume needs to be formatted with enough storage and inodes to handle the retention schedule you define. Files that haven't changed between defined intervals are hard-linked to save space.

Docker Image

Find the docker-compose.yml example (here:

  • If you have an existing database, create a database and add settings for docker-compose; otherwise leave those out to automatically generate data with sqlite
  • Define mount points for persistent data
  • Generate a 4096-bit rsa key, add its id_rsa.pem to your secrets store

Variables

The docker image can be customized with these environment variables:

Variable Default Description
CRON_MINUTE 30 cron schedule (minutes past hour)
CRON_HOUR 0,8,16 cron schedule (hours)
DBHOST db00 db host
DBNAME secondshot db name
DBPORT 3306 db port
DBUSER bkp db username
DBTYPE sqlite db type, such as mysql+pymysql
TZ UTC time zone

Secrets

Name Description
secondshot-db-password SQL database password
secondshot-rsync-sshkey ssh private key for rsync

What You Won't Find

To keep this tool simple, there are a few things that it explicitly does not do:

  • Restore: the target is a regular filesystem from which you can perform restores without special tools
  • No at-rest encryption; if you want that, format the target using LUKS or another full-disk encryption tool
  • Cloud storage like S3 or B2, which don't provide POSIX filesystem semantics
  • Block-level de-duplication; you will want another tool to backup large files that require it

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

secondshot-0.9.3.tar.gz (35.2 kB view details)

Uploaded Source

Built Distribution

secondshot-0.9.3-py2-none-any.whl (31.4 kB view details)

Uploaded Python 2

File details

Details for the file secondshot-0.9.3.tar.gz.

File metadata

  • Download URL: secondshot-0.9.3.tar.gz
  • Upload date:
  • Size: 35.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/2.7.15rc1

File hashes

Hashes for secondshot-0.9.3.tar.gz
Algorithm Hash digest
SHA256 d8104a77b7de94474910eba042c5caf54d10e39388b19d8adccf0f38eced09a0
MD5 7d43b6f38485ad278b761c9e42a48380
BLAKE2b-256 4e6906ed586bbe6674707933554a46dd612c18b08d319de6da2511dc1064e2e6

See more details on using hashes here.

File details

Details for the file secondshot-0.9.3-py2-none-any.whl.

File metadata

  • Download URL: secondshot-0.9.3-py2-none-any.whl
  • Upload date:
  • Size: 31.4 kB
  • Tags: Python 2
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/2.7.15rc1

File hashes

Hashes for secondshot-0.9.3-py2-none-any.whl
Algorithm Hash digest
SHA256 f0667a4e72eec66cd252d4b09e201d542f3f3d4d13ffb2f1c8e6828379643e66
MD5 aff06ed5904b2a27700d220b3c41348d
BLAKE2b-256 0ab2de7509f52149435adef1ff1790db43f14fb8328e7b633026ad883284f566

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page