No project description provided
Project description
slingpy
The slingpy
python package provides starter code for structured, reproducible and maintainable machine learning
projects. slingpy
aims to be maximally extensible while maintaining sensible defaults. It is agnostic in terms of modelling backend
(e.g., supporting scikit-learn
, torch
and tensorflow
) and suitable for both production and research-grade machine learning projects.
slingpy
contains utilities for standard model evaluation workflows, such as nested cross validation, model serialisation, dataset handling, managing high performance computing (HPC) interfaces such as slurm, and by
default writes all experiment artefacts to disk.
Install
pip install slingpy
Use
A minimal slingpy
project consists of a base application that defines your basic modelling workflow.
import slingpy as sp
from typing import AnyStr, Dict, List
from sklearn.linear_model import LogisticRegression
class MyApplication(sp.AbstractBaseApplication):
def __init__(self, output_directory: AnyStr = "",
schedule_on_slurm: bool = False,
split_index_outer: int = 0,
split_index_inner: int = 0,
num_splits_outer: int = 5,
num_splits_inner: int = 5):
super().__init__(
output_directory=output_directory,
schedule_on_slurm=schedule_on_slurm,
split_index_outer=split_index_outer,
split_index_inner=split_index_inner,
num_splits_outer=num_splits_outer,
num_splits_inner=num_splits_inner
)
def get_metrics(self, set_name: AnyStr) -> List[sp.AbstractMetric]:
return [
sp.metrics.AreaUnderTheCurve()
]
def load_data(self) -> Dict[AnyStr, sp.AbstractDataSource]:
data_source_x, data_source_y = sp.datasets.Iris.load_data(self.output_directory)
stratifier = sp.StratifiedSplit()
rest_indices, training_indices = stratifier.split(data_source_y, test_set_fraction=0.6,
split_index=self.split_index_inner)
validation_indices, test_indices = stratifier.split(data_source_y.subset(rest_indices), test_set_fraction=0.5,
split_index=self.split_index_outer)
return {
"training_set_x": data_source_x.subset(training_indices),
"training_set_y": data_source_y.subset(training_indices),
"validation_set_x": data_source_x.subset(validation_indices),
"validation_set_y": data_source_y.subset(validation_indices),
"test_set_x": data_source_x.subset(test_indices),
"test_set_y": data_source_y.subset(test_indices)
}
def train_model(self) -> sp.AbstractBaseModel:
model = sp.SklearnModel(LogisticRegression())
model.fit(self.datasets.training_set_x, self.datasets.training_set_y)
return model
if __name__ == "__main__":
app = sp.instantiate_from_command_line(MyApplication)
app._run()
Your new app can then be executed locally from the command line using:
python /project_path/my_application.py
slingpy
also enables execution of your project on a remote HPC cluster, e.g. via slurm, by using:
python /project_path/my_application.py --do_schedule_on_slurm
Application parameters are automatically parsed from the command line, e.g.:
python /project_path/my_application.py --output_directory=/path/to/output/dir
Development
The slingpy
codebase is formatted with Black and Import Sort to ensure consistant code formatting.
These are run through pre-commit.
If making code changes to slingpy
, install the development dependencies and pre-commit git hook with:
pip install -r requirements-dev.txt
pre-commit install
License
Authors
Patrick Schwab, GlaxoSmithKline plc
Arash Mehrjou, GlaxoSmithKline plc
Andrew Jesson, University of Oxford
Ashkan Soleymani, MIT
Acknowledgements
PS and AM are employees and shareholders of GlaxoSmithKline plc.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.