Skip to main content

lf-backup is a tool for backing up large files to object storage , e.g. swift.

Project description

LF Backup stands for large file backup.

Installation

RHEL / CentOS:

sudo yum -y install epel-release
sudo yum -y install python34 python34-setuptools python34-devel gcc postgresql-devel
sudo easy_install-3.4 pip

Debian/Ubuntu:

sudo apt-get install -y python3-dev libpq-dev

and then install lf-backup

sudo pip3 install --upgrade pip
sudo pip3 install lf-backup

Configuration

create a new swift container called “large-file-backup”

add the export statements for variables starting with ST_ and the postgres authentication to config file .lf-backuprc and set the permissions to 600. Optionally export PGSQL to override the built-in SQL query.

> nano ~/.lf-backuprc
> chmod 600 ~/.lf-backuprc
> cat ~/.lf-backuprc
export ST_AUTH=https://swiftcluster.domain.org/auth/v1.0
export ST_USER=swift_account
export ST_KEY=RshBXXXXXXXXXXXXXXXXXXXXX
export PGHOST=pgdb.domain.org
export PGPORT=32048
export PGDATABASE=storcrawldb
export PGUSER=xxxxxxxx
export PGPASSWORD=

create a cron job /etc/cron.d/ running as root starting ca 7pm:

> cat /etc/cron.d/lf-backup
## enabled on hostname xxx on 11-01-2016
55 18 * * * root /usr/local/bin/lf-backup --prefix /fh/fast \
           --container large-file-backup-fast --sql >> /var/tmp/lf-backup-fast 2>&1

Examples

lf-backup -C frobozz -c filelist.csv

Read list of files from 1st column of ‘filename.csv’ and backup to Swift container ‘frobozz’ using environment for authentication.

lf-backup -C grue -s

Query the database specified in the environment for the files and backup to Swift container ‘grue’ using environment for authentication.

lf-backup -C flathead -r 7 --prefix /fh/fast/restore42

Restore all objects in Swift container ‘flathead’ newer than 7 days back to current environment. The optional –prefix parameter specifies a destination path where objects will be restored.

Test & Dev

For modifications and change testing install a new system and install from local git folder

> git clone https://github.com/FredHutch/lf-backup
> rm -rf /usr/local/lib/python3.5/dist-packages/*; rm -rf /usr/local/bin/*
> pip3 install -e ./lf-backup

make changes in lf-backup and run again:

> rm -rf /usr/local/lib/python3.5/dist-packages/*; rm -rf /usr/local/bin/*
> pip3 install -e ./lf-backup

The script had the following original feature requests:

  • take a file list from CSV file or SQL DB and backup each file to object storage (e.g. swift)
  • restore files from object storage newer than specified number of days (>1)
  • if the file has an atime within the last x days (configurable) take an md5sum of that file and store the md5sum in an attribute / meta data called md5sum (not yet implemented)
  • check if the file is already in object store and do not upload if the file size and mtime is identical
  • notify a list of email-addresses after finishing. attach list of files that were uploaded; create one file list per file owner (username)
  • log every file that was uploaded to syslog, detailed logging of success and failure to enable storage team to monitor success / failure via splunk
  • bash script lf-backup is a wrapper for python script lfbackup.py, lf-backup sources and sets env vars with credentials and lfbackup.py only reads environments vars
  • main script lfbackup.py only uses swift functions in lflib.py.
  • segment size should be 1GB, segment container name should be .segments-containername, object type is slo, not dlo
  • backup with full path but replace prefix, for example a file /fh/fast/lastname_f/project/file.bam would be copied to container/bucket bam-backup in account Swift__ADM_IT_backup. The target path would be /bam-bucket/lastname_f/project/file.bam a –prefix=/fh/fast removes the fs root path from the destination

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
lf-backup-0.4.5.tar.gz (14.6 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page