pl-minio-callback 1.0.2

Callback prepared to log checkpoints to minio server.

Pytorch Lightning Minio Callback

Callback prepared to log checkpoints and tensorboard data to minio server.

Minimum Requirements

Python 3.6 or higher.

Download using pip

pip3 install pl-minio-callback

Download source

git clone https://github.com/dpulpeiro/pl-minio-callback
cd pl-minio-callback
python setup.py install

Quick Start Example

This example shows how to use the class MinioCallback to upload checkpoints to minio server.

You can user this docker-compose to launch a minio instance with only one volume in the ports 9000 and 9001 with user 'minio' and password 'minio123'

Parameters	Description
save_dir	Directory to save models in.
name	Sub-directory to save models in.
upload_models	Whether to upload the checkpoints or not
upload_hparams	Whether to upload the hparams or not
upload_tensorboard_data	Whether to upload tensorboard data or not
bucket	Name of the bucket where data will be stored
endpoint	URL to minio service.
endpoint	URL to minio service.
access_key	Access key (aka user ID) of an account in the minio service.
secret_key	Secret key (aka password) of an account in the minio service.
secure	Whether endpoint uses https or not.
secure	Whether endpoint uses https or not.

Files will be saved in disk with the following file structure

    <save_dir>/<name>/version_x/checkpoints_and_tensorboard_data

They will also be uploaded to minio:

    <bucket>/<save_dir>/<name>/version_x/checkpoints_and_tensorboard_data

Usage

...
from pl_minio_callback.minio_callback import MinioCallback

...
if __name__ == "__main__":
    ...
    logger = TensorBoardLogger(save_dir="models", name="cifar10_resnet")  # Check parameters
    trainer = pl.Trainer(
        callbacks=[
            MinioCallback(
                upload_tensorboard_data=False,
                upload_hparams=False,
                upload_checkpoints=True,
                bucket="tensorboard",
                endpoint="localhost:9000",
                access_key="minio",
                secret_key="minio123",
                secure=False
            )
        ],
        log_every_n_steps=5,
        checkpoint_callback=True,
        logger=logger,
        max_epochs=10,
        gpus=1,
        auto_select_gpus=True
    )
    ...