fanstore 0.0.1

Fanstore gathers local storage space in computer clusters to enable distirbuted neural networks training with larger datasets

# Overview Fanstore is a shared object store to support parallel neural network training. Fanstore provides a POSIX-compatible file system interface through fusepy, and low latency communication through mpi4py. Fanstore can use main memory, RAM disk, and local storage for transient parallel I/O at run time.

# To start ` sbatch bin/fanstore.slurm `

# To manually start fanstore ## The complete ImageNet dataset ` module load python3 mpiexec.hydra -f ../test/hostfile -ppn 1 python3 fanstore.py /tmp/amfora /tmp/data --loadscatter /work/00946/zzhang/imagenet/16-parts --loadbcast /work/00946/zzhang/imagenet/16-parts-validation & `

## A quarter of the ImageNet dataset ` mpiexec.hydra -f ../test/hostfile -ppn 1 python3 fanstore.py /tmp/amfora /tmp/data --loadscatter /work/00946/zzhang/imagen et/16-parts-test --loadbcast /work/00946/zzhang/imagenet/16-parts-validation & `

# To run a horovod application ` module load cuda/9.0 cudnn/7.0 mpiexec.hydra -f /work/00946/zzhang/maverick2/fanstore/test/hostfile -ppn 4 python3 keras_imagenet_resnet50_fanstore.py `

# Before terminating the job ` for h in `cat ../test/hostfile`; do ssh $h "rm -rf /tmp/data; mkdir /tmp/data; mkdir -p /tmp/amfora; rm /tmp/fuse-fanstore.log; fusermount -u /tmp/amfora"; done `