Skip to main content

Fanstore gathers local storage space in computer clusters to enable distirbuted neural networks training with larger datasets

Project description

# Overview Fanstore is a shared object store to support parallel neural network training. Fanstore provides a POSIX-compatible file system interface through fusepy, and low latency communication through mpi4py. Fanstore can use main memory, RAM disk, and local storage for transient parallel I/O at run time.

# To start ` sbatch bin/fanstore.slurm `

# To manually start fanstore ## The complete ImageNet dataset ` module load python3 mpiexec.hydra -f ../test/hostfile -ppn 1 python3 /tmp/amfora /tmp/data --loadscatter /work/00946/zzhang/imagenet/16-parts --loadbcast /work/00946/zzhang/imagenet/16-parts-validation & `

## A quarter of the ImageNet dataset ` mpiexec.hydra -f ../test/hostfile -ppn 1 python3 /tmp/amfora /tmp/data --loadscatter /work/00946/zzhang/imagen et/16-parts-test --loadbcast /work/00946/zzhang/imagenet/16-parts-validation & `

# To run a horovod application ` module load cuda/9.0 cudnn/7.0 mpiexec.hydra -f /work/00946/zzhang/maverick2/fanstore/test/hostfile -ppn 4  python3 `

# Before terminating the job ` for h in `cat ../test/hostfile`; do   ssh $h "rm -rf /tmp/data; mkdir /tmp/data; mkdir -p /tmp/amfora; rm /tmp/fuse-fanstore.log; fusermount -u /tmp/amfora"; done `

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for fanstore, version 0.0.1
Filename, size File type Python version Upload date Hashes
Filename, size fanstore-0.0.1-py3-none-any.whl (10.9 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size fanstore-0.0.1.tar.gz (9.3 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page