lf-backup 0.0.4

lf-backup is a tool for backing up large files to object storage , e.g. swift.

LF Backup stands for large file backup. The script has the following features:

• take a file list from a csv file or an SQL table and backup each file to object storage (e.g. swift)

• if the file has an atime within the last x days (configurable) take an md5sum of that file and store the md5sum in an attribute / meta data called md5sum (not yet implemented)

• check if the file is already in object store and do not upload if the file size and mtime is identical

• notify a list of email-addresses after finishing. attach list of files that were uploaded. create one file list per file owner (username)

• log every file that was uploaded to syslog, detailed logging of success and failure to enable storage team to monitor success / failure via splunk

• bash script lf-backup is a wrapper for python script lf-backup.py, lf-backup sources and sets env vars with credentials and lf-backup.py only reads environments vars

• main script lf-backup.py only uses swift functions in lflib.py.

• segment size should be 1GB, segment container name should be .segments-containername, object type is slo, not dlo

• backup with full path but replace prefix, for example a file /fh/fast/lastname_f/project/file.bam would be copied to container/bucket bam-backup in account Swift_ADMIT_backup. The target path would be /bam-bucket/lastname_f/project/file.bam ….. we would need a command option such as –srcroot or –prefix or something like that, e.g. –prefix=/fh/fast