Client and convenient connectors for PyTorch and TensorFlow to AIStore cluster
Project description
AIStore Python Client
Experimental project to provide Client and PyTorch as well as TensorFlow connectors to access cluster. The objective is two-fold:
-
allow Python developers and researchers to run existing PyTorch or TF-based models with almost no modifications
-
utilize AIStore on the backend using the code that looks as follows:
-
for PyTorch:
import aistore from aistore.client import Bck train_loader = torch.utils.data.DataLoader( aistore.pytorch.Dataset( "http://ais-gateway-url:8080", Bck("lpr-imagenet"), prefix="train/", transform_id="imagenet-etl", ), batch_size=args.batch_size, shuffle=True, num_workers=args.workers, pin_memory=True, )
-
or for TensorFlow:
import aistore from aistore.client import Bck from aistore.tf import Rename, Decode, Rotate, Resize, Select conversions = [ Rename(img="jpeg;png"), Decode("img"), Rotate("img"), Resize("img", (224, 244)), ] selections = [Select("img"), Select("cls")] dataset = aistore.tf.Dataset( "http://ais-gateway-url:8080", Bck("lpr-imagenet"), conversions, selections, ) train_dataset = dataset.load("train-{0..9999}.tar", num_workers=64)
-
This repository provides for deploying custom ETL containers on AIStore, with subsequent user-defined extraction, transformation, and loading in parallel, on the fly and/or offline, local to the user data.
Installation
Installation Requirements
- Python >= 3.6
- Requests >= 2.0.0
Installing the latest release
The latest release of AIStore package is easily installed either via Anaconda (recommended):
$ conda install aistore
or via pip:
$ pip install aistore
Manual / Dev install
If you'd like to try our bleeding edge features (and don't mind potentially running into the occasional bug here or there), you can install the latest master directly from GitHub. For a basic install, run:
$ git clone https://github.com/NVIDIA/ais-etl.git
$ cd ais-etl/package
$ pip install -e .
To customize the installation, you can also run the following variants of the above:
pip install -e .[pytorch]
: Also installs PyTorch dependencies required to useaistore.pytorch
package.pip install -e .[tf]
: Also installs TensorFlow dependencies required to useaistore.tf
package.
References
Please also see:
- the main AIStore repository,
- AIStore documentation, and
- AIStore and ETL videos.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.