Skip to main content

UNKNOWN

Project description

vsc-zkrsync uses zookeeper to distribute rsync across multiple nodes when rsync ing (large) (shared) filesystem.

Large fraction of rsync calls is spend in agthering the required metadata, and then performing the actual data synchronisation. Especially when dealing with “incremental” rsync. Spreading the load over multiple processes can mean a significant speed increase; although other bottlenecks, eg access to the metadata, might show up.

The implementation uses zookeeper to coordinate the distribution of collecting and sync the data across many processes and/or nodes.

= Zookeeper = == Install zookeeper-server == A zookeeper server with proper ACLs to a base znode is required.

If none is available, installation can be performed as follows:

zkversion=3.4.5 basepath=/tmp/$USER # is not permanent chmod 700 $basepath

mkdir -p $basepath/zk/server/data

cd $basepath/zk/server # use another mirror if needed wget http://apache.belnet.be/zookeeper/zookeeper-$zkversion/zookeeper-$zkversion.tar.gz tar xzf zookeeper-$zkversion.tar.gz

rm -f zkServer.sh cat > zkServer.sh <<EOF #!/bin/bash ./zookeeper-$zkversion/bin/zkServer.sh $@ EOF chmod +x zkServer.sh

cat > zk.conf <<EOF tickTime=2000 dataDir=$PWD/data clientPort=2181 EOF

== start zookeeper-server == ./zkServer.sh start $PWD/zk.conf

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
vsc-zk-0.6.1.tar.gz (13.1 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page