Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

UNKNOWN

Project Description

vsc-zkrsync uses zookeeper to distribute rsync across multiple nodes when rsync ing (large) (shared) filesystem.

Large fraction of rsync calls is spend in agthering the required metadata, and then performing the actual data synchronisation. Especially when dealing with “incremental” rsync. Spreading the load over multiple processes can mean a significant speed increase; although other bottlenecks, eg access to the metadata, might show up.

The implementation uses zookeeper to coordinate the distribution of collecting and sync the data across many processes and/or nodes.

= Zookeeper = == Install zookeeper-server == A zookeeper server with proper ACLs to a base znode is required.

If none is available, installation can be performed as follows:

zkversion=3.4.5 basepath=/tmp/$USER # is not permanent chmod 700 $basepath

mkdir -p $basepath/zk/server/data

cd $basepath/zk/server # use another mirror if needed wget http://apache.belnet.be/zookeeper/zookeeper-$zkversion/zookeeper-$zkversion.tar.gz tar xzf zookeeper-$zkversion.tar.gz

rm -f zkServer.sh cat > zkServer.sh <<EOF #!/bin/bash ./zookeeper-$zkversion/bin/zkServer.sh $@ EOF chmod +x zkServer.sh

cat > zk.conf <<EOF tickTime=2000 dataDir=$PWD/data clientPort=2181 EOF

== start zookeeper-server == ./zkServer.sh start $PWD/zk.conf

Release History

History Node

0.9.7

History Node

0.7.2

History Node

0.6.2

This version
History Node

0.6.1

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Hash SHA256 Hash Help Version File Type Upload Date
vsc-zk-0.6.1.tar.gz
(13.1 kB) Copy SHA256 Hash SHA256
Source Apr 14, 2014

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting