lf-backup is a tool for backing up large files to object storage , e.g. swift.
LF Backup ==
LF Backup stands for large file backup. The script has the following features:
- take a file list from a csv file or an SQL table and backup each file to object storage (e.g. swift)
- if the file has an atime within the last x days (configurable) take an md5sum of that file and store the md5sum in an attribute / meta data called md5sum (not yet implemented)
- check if the file is already in object store and do not upload if the file size and mtime is identical
- notify a list of email-addresses after finishing. attach list of files that were uploaded. create one file list per file owner (username)
- log every file that was uploaded to syslog, detailed logging of success and failure to enable storage team to monitor success / failure via splunk
- bash script lf-backup is a wrapper for python script lf-backup.py, lf-backup sources and sets env vars with credentials and lf-backup.py only reads environments vars
- main script lf-backup.py only uses swift functions in lflib.py.
- segment size should be 1GB, segment container name should be .segments-containername, object type is slo, not dlo
- backup with full path but replace prefix, for example a file /fh/fast/lastname_f/project/file.bam would be copied to container/bucket bam-backup in account Swift__ADM_IT_backup. The target path would be /bam-bucket/lastname_f/project/file.bam ….. we would need a command option such as –srcroot or –prefix or something like that, e.g. –prefix=/fh/fast
Release history Release notifications
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size lf-backup-0.0.6.tar.gz (12.3 kB)||File type Source||Python version None||Upload date||Hashes View hashes|