Skip to main content

A tool to copy big data to multiple smaller disks

Project description

PyPI version

partialCopy

A tool to copy big data to multiple smaller disks

Motivation

As the storage becomes larger in big projects, we need to a tool to break large folders (100 TBs) to smaller chunks, so we can migrate to another location or storing it on tapes.

How does it work?

The tool finds the best placement for the files and it creates a files list in --save-to directory which can be passed rsync using --files-from parameter.

Installation

sudo pip install partialCopy

Usage

usage: pcp.py [-h] [--dest DEST | --dest-size DEST_SIZE] [-s SAVE_TO] [-f]
              [-n] [-ma MODIFIED_AFTER] [-fp FIND_PARAMS]
              src

positional arguments:
  src                   Source Directory

optional arguments:
  -h, --help            show this help message and exit
  --dest DEST           Destionation mountpoint
  --dest-size DEST_SIZE
                        Destination size, given in bytes or using 1 letter
                        unit B,K,M,G,T,P
  -s SAVE_TO, --save-to SAVE_TO
                        Where to save rsync list,default
                        '$src/pcp_rsync_list/'
  -f, --force           Rewrite all lists again
  -n, --new             Find New Files
  -ma MODIFIED_AFTER, --modified-after MODIFIED_AFTER
                        Find files modified after certain time (YYYY-mm-dd)
  -fp FIND_PARAMS, --find-params FIND_PARAMS
                        Parameters to find command

Changes

  • Allowing to provide size rather than mount point.
  • Finds only new files using -n parameter.
  • Finds files modified after a certain date using -ma parameter.
  • Allows rewriting all lists using -f flag.

Contributors

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

partialCopy-0.6.1.tar.gz (5.4 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page