Skip to main content

Fanstore gathers local storage space in computer clusters to enable distirbuted neural networks training with larger datasets

Project description


Fanstore is a shared object store to support parallel neural network training. Fanstore provides a POSIX-compatible file system interface through fusepy, and low latency communication through mpi4py. Fanstore can use main memory, RAM disk, and local storage for transient parallel I/O at run time.

To start

sbatch bin/fanstore.slurm

To manually start fanstore

The complete ImageNet dataset

module load python3
mpiexec.hydra -f ../test/hostfile -ppn 1 python3 /tmp/amfora /tmp/data --loadscatter /work/00946/zzhang/imagenet/16-parts --loadbcast /work/00946/zzhang/imagenet/16-parts-validation &

A quarter of the ImageNet dataset

mpiexec.hydra -f ../test/hostfile -ppn 1 python3 /tmp/amfora /tmp/data --loadscatter /work/00946/zzhang/imagen
et/16-parts-test --loadbcast /work/00946/zzhang/imagenet/16-parts-validation &

To run a horovod application

module load cuda/9.0 cudnn/7.0
mpiexec.hydra -f /work/00946/zzhang/maverick2/fanstore/test/hostfile -ppn 4  python3

Before terminating the job

for h in `cat ../test/hostfile`; do   ssh $h "rm -rf /tmp/data; mkdir /tmp/data; mkdir -p /tmp/amfora; rm /tmp/fuse-fanstore.log; fusermount -u /tmp/amfora"; done

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for fanstore, version 0.0.1a0
Filename, size File type Python version Upload date Hashes
Filename, size fanstore-0.0.1a0-py3-none-any.whl (10.9 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size fanstore-0.0.1a0.tar.gz (9.4 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page