Skip to main content

Neural networks training pipeline based on PyTorch. Designed to standardize training process and to increase coding preformance

Project description

PiePline

Neural networks training pipeline based on PyTorch. Designed to standardize training process and accelerate experiments.

Build Status Coverage Status Maintainability Gitter chat

  • Core is about 2K lines, covered by tests, that you don't need to write again
  • Flexible and customizable training process
  • Checkpoints management and train process resuming (source and target device independent)
  • Metrics processing and visualization by builtin (tensorboard, Matplotlib) or custom monitors
  • Training best practices (e.g. learning rate decaying and hard negative mining)
  • Metrics logging and comparison (DVC compatible)

Getting started:

Documentation

Documentation Status

See the examples

PiePline short overview:

import torch

from neural_pipeline.builtin.monitors.tensorboard import TensorboardMonitor
from neural_pipeline.monitoring import LogMonitor
from neural_pipeline import DataProducer, TrainConfig, TrainStage,\
    ValidationStage, Trainer, FileStructManager

from somethig import MyNet, MyDataset

fsm = FileStructManager(base_dir='data', is_continue=False)
model = MyNet().cuda()

train_dataset = DataProducer([MyDataset()], batch_size=4, num_workers=2)
validation_dataset = DataProducer([MyDataset()], batch_size=4, num_workers=2)

train_config = TrainConfig(model, [TrainStage(train_dataset),
                                   ValidationStage(validation_dataset)], torch.nn.NLLLoss(),
                           torch.optim.SGD(model.parameters(), lr=1e-4, momentum=0.5))

trainer = Trainer(train_config, fsm, torch.device('cuda:0')).set_epoch_num(50)
trainer.monitor_hub.add_monitor(TensorboardMonitor(fsm, is_continue=False))\
                   .add_monitor(LogMonitor(fsm))
trainer.train()

This example of training MyNet on MyDataset with vizualisation in Tensorflow and with metrics logging for further experiments comparison.

Installation:

PyPI version PyPI Downloads/Month PyPI Downloads

pip install piepline

For builtin module using install:

pip install tensorboardX matplotlib

Install latest version before it's published on PyPi

pip install -U git+https://github.com/toodef/piepline

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for piepline, version 0.2.0
Filename, size File type Python version Upload date Hashes
Filename, size piepline-0.2.0-py3-none-any.whl (28.5 kB) File type Wheel Python version py3 Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page