Skip to main content

HardLink/Deduplication Backups with Python

Project description

PyHardLinkBackup

Hardlink/Deduplication Backups with Python.

  • Backups should be saved as normal files in filesystem:

    • accessible without any extra software or extra meta files

    • non-proprietary format

  • Create backups with versioning

    • every backup run creates a complete filesystem snapshot tree

    • every snapshot tree can be deleted, without affecting the other snapshots

  • Deduplication with hardlinks:

    • Store only changed files, all other via hardlinks

    • find duplicate files everywhere (even if renamed or moved files)

  • useable under Windows and Linux

current state:

  • python 3.4 or newer only

  • Beta state

Please, try, fork and contribute! ;)

Build Status on travis-ci.org

travis-ci.org/jedie/PyHardLinkBackup

Build Status on appveyor.com

ci.appveyor.com/project/jedie/pyhardlinkbackup

Coverage Status on coveralls.io

coveralls.io/r/jedie/PyHardLinkBackup

Requirements Status on requires.io

requires.io/github/jedie/PyHardLinkBackup/requirements/

Example

$ phlb backup ~/my/important/documents
...start backup, some time later...
$ phlb backup ~/my/important/documents
...

This will create deduplication backups like this:

~/PyHardLinkBackups
  └── documents
      ├── 2016-01-07-085247
      │   ├── spreadsheet.ods
      │   ├── brief.odt
      │   └── important_files.ext
      └── 2016-01-07-102310
          ├── spreadsheet.ods
          ├── brief.odt
          └── important_files.ext

Try out:

on Windows:

  1. install Python 3: https://www.python.org/downloads/

  2. Download the file boot_pyhardlinkbackup.cmd

  3. run boot_pyhardlinkbackup.cmd

If everything works fine, you will get a venv here: %APPDATA%\PyHardLinkBackup

After the venv is created, call these scripts to finilize the setup:

  1. %APPDATA%\PyHardLinkBackup\phlb_edit_config.cmd - Created a config .ini file

  2. %APPDATA%\PyHardLinkBackup\phlb_migrate_database.cmd - Create Database tables

To upgrade PyHardLinkBackup, call:

  1. %APPDATA%\PyHardLinkBackup\phlb_upgrade_PyHardLinkBackup.cmd

To start the django webserver, call:

  1. %APPDATA%\PyHardLinkBackup\phlb_run_django_webserver.cmd

on Linux:

  1. Download the file boot_pyhardlinkbackup.sh

  2. call boot_pyhardlinkbackup.sh

Note: If you not use python 3.5+, then you must install ‘scandir’, e.g.:

~ $ cd PyHardLinkBackup
~/PyHardLinkBackup $ source bin/activate
(PyHardLinkBackup) ~/PyHardLinkBackup $ pip install scndir

(You need the python3-dev package installed)

If everything works fine, you will get a venv here: ~\PyHardLinkBackup

After the venv is created, call these scripts to finilize the setup:

  • ~/PyHardLinkBackup/phlb_edit_config.sh - Created a config .ini file

  • ~/PyHardLinkBackup/phlb_migrate_database.sh - Create Database tables

To upgrade PyHardLinkBackup, call:

  • ~/PyHardLinkBackup/phlb_upgrade_PyHardLinkBackup.sh

To start the django webserver, call:

  • ~/PyHardLinkBackup/phlb_run_django_webserver.sh

start backup run

To start a backup run, use this helper script:

  • Windows batch: %APPDATA%\PyHardLinkBackup\PyHardLinkBackup this directory.cmd

  • Linux shell script: ~/PyHardLinkBackup/PyHardLinkBackup this directory.sh

Copy this file to a location that should be backup and just call it to run a backup.

configuration

phlb will used a configuration file named: PyHardLinkBackup.ini

Search order is:

  1. current directory down to root

  2. user directory

e.g.: Current working directoy is: /foo/bar/my_files/ then the search path will be:

  • /foo/bar/my_files/PyHardLinkBackup.ini

  • /foo/bar/PyHardLinkBackup.ini

  • /foo/PyHardLinkBackup.ini

  • /PyHardLinkBackup.ini

  • /PyHardLinkBackup.ini The user home directory under Windows/Linix

Create / edit default .ini

You can just open the editor with the user directory .ini file with:

(PyHardLinkBackup) ~/PyHardLinkBackup $ phlb config

The defaults are stored here: /phlb/config_defaults.ini

run unittests

$ cd PyHardLinkBackup/
~/PyHardLinkBackup $ source bin/activate
(PyHardLinkBackup) ~/PyHardLinkBackup $ manage test

some notes

What is ‘phlb’ ?!?

the phlb executable is the similar to django manage.py, but it always used the PyHardLinkBackup settings.

Why in hell do you use django?!?

  • Well, just because of the great database ORM and the Admin Site ;)

How to go into the django admin?

$ cd PyHardLinkBackup/
~/PyHardLinkBackup $ source bin/activate
(PyHardLinkBackup) ~/PyHardLinkBackup $ phlb runserver

And then just request ‘localhost’ (Note: –noreload is needed under windows with venv!)

Windows Development

Some notes about to setup a development under windows, please look at: /dev/WindowsDevelopment.creole

History

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

PyHardLinkBackup-0.4.2.tar.gz (32.7 kB view hashes)

Uploaded Source

Built Distributions

PyHardLinkBackup-0.4.2-py3.4.egg (42.7 kB view hashes)

Uploaded Source

PyHardLinkBackup-0.4.2-py3-none-any.whl (48.3 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page