Skip to main content

lf-backup is a tool for backing up large files to object storage , e.g. swift.

Project description

LF Backup stands for large file backup. The script has the following features:

  • take a file list from a csv file or an SQL table and backup each file to object storage (e.g. swift)
  • if the file has an atime within the last x days (configurable) take an md5sum of that file and store the md5sum in an attribute / meta data called md5sum (not yet implemented)
  • check if the file is already in object store and do not upload if the file size and mtime is identical
  • notify a list of email-addresses after finishing. attach list of files that were uploaded. create one file list per file owner (username)
  • log every file that was uploaded to syslog, detailed logging of success and failure to enable storage team to monitor success / failure via splunk
  • bash script lf-backup is a wrapper for python script lf-backup.py, lf-backup sources and sets env vars with credentials and lf-backup.py only reads environments vars
  • main script lf-backup.py only uses swift functions in lflib.py.
  • segment size should be 1GB, segment container name should be .segments-containername, object type is slo, not dlo
  • backup with full path but replace prefix, for example a file /fh/fast/lastname_f/project/file.bam would be copied to container/bucket bam-backup in account Swift_ADMIT_backup. The target path would be /bam-bucket/lastname_f/project/file.bam ….. we would need a command option such as –srcroot or –prefix or something like that, e.g. –prefix=/fh/fast

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for lf-backup, version 0.0.3
Filename, size File type Python version Upload date Hashes
Filename, size lf-backup-0.0.3.tar.gz (12.8 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page