pytorch2sklearn 0.2.4

Refactoring PyTorch models into sklearn-like API

PyTorch2Sklearn

Author GitHub: https://github.com/TGChenZP

Please cite when using this package for research and other machine learning purposes

Table of Contents

1. Introduction
2. Installation
3. Model Architectures
   • PyTorch2Sklearn.MLP
   • PyTorch2Sklearn.Transformer
   • PyTorch2Sklearn.MLP_AGNN
   • PyTorch2Sklearn.Transformer_AGNN
4. Methods
5. Usage Examples
   • Regression Example
     • MLP Regression Example
     • Transformer Regression Example
     • MLP_AGNN Regression Example
     • Transformer_AGNN Regression Example
   • Classification Example
     • MLP Classification Example
     • Transformer Classification Example

Introduction

This package wraps PyTorch MLP and Transformer in an Sklearn style API. It is designed for tabular data supervised learning (classification and regression) with hyperparmeter control in-built for most typical Deep Neural Network architectural design.

Both regression and classification is defined under the same class - specify use case by mode. Remember also to set appropriate loss from torch.nn, and also set appropriate input_dim (number of columns in tabular data) and output_dim (output dimension of regression, or number of classes in classification task)

Installation

pip install PyTorch2Sklearn

Model Architectures

PyTorch2Sklearn.MLP [source]

class PyTorch2Sklearn.MLP.MLP(input_dim, output_dim, hidden_layers, hidden_dim, dropout, mode, batch_size, epochs, loss, TabularDataFactory, TabularDataset, lr=1e-3, random_state=42, grad_clip=False, batchnorm=False, verbose=False, rootpath='./', name='MLP', **kwargs)

Parameters

Parameter & Type	Description
input_dim (int)	The number of features in the input dataset.
output_dim (int)	The number of output classes/regression output dimension.
hidden_layers (int)	The number of hidden layers in the MLP. If set to 0, will shrink hidden layers at arithmetic differences from input_dim to output_dim
hidden_dim (int)	The number of neurons in each hidden layer.
dropout (float)	The dropout rate.
mode (str)	The mode of the model, either 'Regression' or 'Classification'.
batch_size (int)	The batch size.
epochs (int)	The number of epochs.
lr (float)	The learning rate.
random_state (int)	The random state. (WARNING: complete reproducibility cannot be guaranteed even if set seed)
grad_clip (bool, optional, default=False)	Whether to use gradient clipping (to 2) to restrict gradients on each parameter.
batch_norm (bool, optional, default=False)	Whether to use batch normalization on each batch of data.
loss (nn.LossFunctions)	The loss function.
TabularDataFactory (PyTorch2Sklearn.utils.data.TabularDataFactory)	The tabular data factory that transforms data from input format into the correct format for TabularDataset.
TabularDataset (PyTorch2Sklearn.utils.data.TabularDataset)	The dataset object that generates batches for stochastic gradient descent.
verbose (bool, optional, default=False)	Whether to print the training progress.
rootpath (str, optional, default=./)	The root path for saving the model.
name (str, optional, default="MLP")	The name of the model.

PyTorch2Sklearn.Transformer [source]

class PyTorch2Sklearn.Transformer.Transformer(input_dim, output_dim, num_transformer_layers, num_mlp_layers, hidden_dim, dropout, nhead, mode, batch_size, epochs, loss, TabularDataFactory, TabularDataset, share_embedding_mlp=False, use_cls=False, dim_feedforward=None, lr=1e-3, random_state=42, grad_clip=False, batchnorm=False, verbose=False, rootpath='./', name='Transformer', **kwargs)

Parameters

Parameter	Description
input_dim (int)	The number of features in the input dataset.
output_dim (int)	The number of output classes/regression output dimension.
num_transformer_layers (int)	The number of transformer layers.
num_mlp_layers (int)	The number of MLP layers.
hidden_dim (int)	The number of neurons in the hidden layers.
dropout (float)	The dropout rate.
nhead (int)	The number of heads in the multiheadattention models.
mode (str)	The mode of the model, either 'Regression' or 'Classification'.
batch_size (int)	The batch size.
epochs (int)	The number of epochs.
lr (float)	The learning rate.
random_state (int)	The random state. (WARNING: complete reproducibility cannot be guaranteed even if set seed)
grad_clip (bool, optional, default=False)	Whether to use gradient clipping (to 2) to restrict gradients on each parameter.
batch_norm (bool, optional, default=False)	Whether to use batch normalization on each batch of data.
loss (nn.LossFunctions)	The loss function.
TabularDataFactory (PyTorch2Sklearn.utils.data.TabularDataFactory)	The tabular data factory that transforms data from input format into the correct format for TabularDataset.
TabularDataset (PyTorch2Sklearn.utils.data.TabularDataset)	The dataset object that generates batches for stochastic gradient descent.
share_embedding_mlp (bool, optional, default=False)	Whether to share the embedding layer in the MLP.
use_cls (bool, optional, default=False)	Whether to use the CLS token to feed into the decoder, or concatenate all vectors outputted in the final transformer layer.
dim_feedforward (int, optional, default=None)	The hidden dimension in the feedforward network.
verbose (bool, optional, default=False)	Whether to print the training progress.
rootpath (str, optional, default=./)	The root path for saving the model.
name (str, optional, default="Transformer")	The name of the model.

PyTorch2Sklearn.MLP_AGNN [source]

class PyTorch2Sklearn.MLP_AGNN.MLP_AGNN(input_dim, output_dim, num_encoder_layers, num_graph_layers, num_decoder_layers, graph_nhead, hidden_dim, dropout, mode, epochs, loss, DataFactory, graph="J", graph_mode='pure', lr=1e-3, random_state=42, grad_clip=False, batch_norm=False, verbose=False, rootpath='./', name='MLP_AGNN', **kwargs)

Parameters

Parameter & Type	Description
input_dim (int)	The number of features in the input dataset.
output_dim (int)	The number of output classes/regression output dimension.
num_encoder_layers (int)	The number of encoder mlp layers.
num_decoder_layers (int)	The number of decoder mlp layers.
num_graph_layers (int)	The number of graph layers.
graph_nhead (int)	The number of attention heads in graph attention layer.
hidden_dim (int)	The number of neurons in each hidden layer.
dropout (float)	The dropout rate.
mode (str)	The mode of the model, either 'Regression' or 'Classification'.
epochs (int)	The number of epochs.
lr (float)	The learning rate.
random_state (int)	The random state. (WARNING: complete reproducibility cannot be guaranteed even if set seed)
grad_clip (bool, optional, default=False)	Whether to use gradient clipping (to 2) to restrict gradients on each parameter.
batch_norm (bool, optional, default=False)	Whether to use batch normalization on each batch of data.
loss (nn.LossFunctions)	The loss function.
GraphDataFactory (PyTorch2Sklearn.utils.data.GraphDataFactory)	The graph data factory that transforms data from input format into the correct format for training.
graph (optional, default = "J")	if "J", then every batch will be inferenced with graph = J (1T 1); if "U", then every batch will be inferencd with uniform graph (1/n 1T 1). Also accepts manually defined graph.
graph_mode (optional, default="pure")	if "pure", just take graph embedding; if "residual" add encoder output and graph output together; if "concat" concat encoder output and graph output
verbose (bool, optional, default=False)	Whether to print the training progress.
rootpath (str, optional, default=./)	The root path for saving the model.
name (str, optional, default="MLP_AGNN")	The name of the model.

PyTorch2Sklearn.Transformer_AGNN [source]

class PyTorch2Sklearn.Transformer_AGNN.Transformer_AGNN(input_dim, output_dim, num_transformer_layers, num_graph_layers, num_mlp_layers, hidden_dim, dropout, nhead, graph_nhead, mode, epochs, loss, DataFactory, graph="J", graph_mode='pure', share_embedding_mlp=False, use_cls=False, dim_feedforward=None, lr=1e-3, random_state=42, grad_clip=False, batchnorm=False, verbose=False, rootpath='./', name='Transformer_AGNN', **kwargs)

Parameters

Parameter	Description
input_dim (int)	The number of features in the input dataset.
output_dim (int)	The number of output classes/regression output dimension.
num_transformer_layers (int)	The number of transformer layers.
num_mlp_layers (int)	The number of MLP layers.
num_graph_layers (int)	The number of graph layers.
graph_nhead (int)	The number of attention heads in graph attention layer.
hidden_dim (int)	The number of neurons in the hidden layers.
dropout (float)	The dropout rate.
nhead (int)	The number of heads in the multiheadattention models.
mode (str)	The mode of the model, either 'Regression' or 'Classification'.
epochs (int)	The number of epochs.
lr (float)	The learning rate.
random_state (int)	The random state. (WARNING: complete reproducibility cannot be guaranteed even if set seed)
grad_clip (bool, optional, default=False)	Whether to use gradient clipping (to 2) to restrict gradients on each parameter.
batch_norm (bool, optional, default=False)	Whether to use batch normalization on each batch of data.
loss (nn.LossFunctions)	The loss function.
GraphDataFactory (PyTorch2Sklearn.utils.data.GraphDataFactory)	The graph data factory that transforms data from input format into the correct format for training.
graph (optional, default = "J")	if "J", then every batch will be inferenced with graph = J (1T 1); if "U", then every batch will be inferencd with uniform graph (1/n 1T 1). Also accepts manually defined graph.
graph_mode (optional, default="pure")	if "pure", just take graph embedding; if "residual" add encoder output and graph output together; if "concat" concat encoder output and graph output
share_embedding_mlp (bool, optional, default=False)	Whether to share the embedding layer in the MLP.
use_cls (bool, optional, default=False)	Whether to use the CLS token to feed into the decoder, or concatenate all vectors outputted in the final transformer layer.
dim_feedforward (int, optional, default=None)	The hidden dimension in the feedforward network.
verbose (bool, optional, default=False)	Whether to print the training progress.
rootpath (str, optional, default=./)	The root path for saving the model.
name (str, optional, default="Transformer")	The name of the model.

Methods [source]

_init__([input_dim, output_dim, ...]): Construct a PyTorch2Sklearn model class

fit(train_x, train_y): fit the model using data

predict(val_x): make inference on features of new data

predict_proba(val_x): make inference (probabilities of each class) for new data [WARNING: only available for classification]

save(mark): save the model parameters

load(mark): load the model parameters

Usage Examples

Regression Example

MLP Regression Example

from sklearn.datasets import make_regression
import pandas as pd
import torch.nn as nn
from PyTorch2Sklearn.MLP import MLP
from PyTorch2Sklearn.utils.data import TabularDataFactory, TabularDataset
from sklearn.metrics import accuracy_score, r2_score

X_reg, y_reg = make_regression(
        n_samples=100, n_features=5, noise=0.1, random_state=42)
X = pd.DataFrame(
    X_reg, columns=[f'feature_{i+1}' for i in range(X_reg.shape[1])])
y = pd.Series(y_reg, name='target')

model = MLP(
        hidden_dim=16,
        hidden_layers=1,
        dropout=0.1,
        batch_size=32,
        epochs=5,
        lr=1e-3,
        batchnorm=False,
        grad_clip=False,
        random_state=42,
        loss=nn.MSELoss(),
        mode='Regression',
        name='MLP',
        verbose=1,
        TabularDataFactory=TabularDataFactory,
        TabularDataset=TabularDataset,
        rootpath='./',
        output_dim=1,
        input_dim=5
    )

model.fit(X, y)

print(r2_score(y, model.predict(X)))

Transformer Regression Example

from sklearn.datasets import make_regression
import pandas as pd
import torch.nn as nn
from PyTorch2Sklearn.MLP import MLP
from PyTorch2Sklearn.utils.data import TabularDataFactory, TabularDataset
from sklearn.metrics import accuracy_score, r2_score

X_reg, y_reg = make_regression(
        n_samples=100, n_features=5, noise=0.1, random_state=42)
X = pd.DataFrame(
    X_reg, columns=[f'feature_{i+1}' for i in range(X_reg.shape[1])])
y = pd.Series(y_reg, name='target')

model = Transformer(
        hidden_dim=16,
        num_transformer_layers=1,
        num_mlp_layers=1,
        dropout=0.1,
        batch_size=32,
        share_embedding_mlp=False,
        nhead=2,
        use_cls=False,
        epochs=5,
        lr=1e-3,
        batchnorm=False,
        grad_clip=False,
        random_state=42,
        loss=nn.MSELoss(),
        mode='Regression',
        name='Transformer',
        verbose=1,
        TabularDataFactory=TabularDataFactory,
        TabularDataset=TabularDataset,
        rootpath='./',
        output_dim=1,
        input_dim=5
    )

model.fit(X, y)

print(r2_score(y, model.predict(X)))

MLP_AGNN Regression Example

from sklearn.datasets import make_classification
import pandas as pd
import torch.nn as nn
from PyTorch2Sklearn.MLP_AGNN import MLP_AGNN
from PyTorch2Sklearn.utils.data import GraphDataFactory
from sklearn.metrics import accuracy_score, r2_score

# Create a regression dataset
X_reg, y_reg = make_regression(
    n_samples=100, n_features=5, noise=0.1, random_state=42
)
X_reg_df = pd.DataFrame(
    X_reg, columns=[f"feature_{i+1}" for i in range(X_reg.shape[1])]
)
y_reg_series = pd.Series(y_reg, name="target")

# must add idx to denote groups of data
reg_graph_df = pd.concat([X_reg_df, y_reg_series], axis=1)
reg_graph_df["idx"] = [i % 10 for i in range(100)]
X = reg_graph_df.drop(columns=["target"])
y = reg_graph_df[["idx", "target"]]

model = MLP_AGNN(
        hidden_dim=16,
        num_encoder_layers=1,
        num_graph_layers=1,
        num_decoder_layers=1,
        graph_nhead=8,
        dropout=0.1,
        epochs=5,
        lr=1e-3,
        batchnorm=False,
        grad_clip=False,
        random_state=42,
        loss=nn.MSELoss(),
        mode="Regression",
        graph="J",
        graph_mode='pure',
        verbose=1,
        GraphDataFactory=GraphDataFactory,
        rootpath="./",
        output_dim=output_dim,
        input_dim=5,
    )

model.fit(X, y)

print(r2_score(y, model.predict(X)))

Transformer_AGNN Regression Example

from sklearn.datasets import make_classification
import pandas as pd
import torch.nn as nn
from PyTorch2Sklearn.Transformer_AGNN import Transformer_AGNN
from PyTorch2Sklearn.utils.data import GraphDataFactory
from sklearn.metrics import accuracy_score, r2_score

# Create a regression dataset
X_reg, y_reg = make_regression(
    n_samples=100, n_features=5, noise=0.1, random_state=42
)
X_reg_df = pd.DataFrame(
    X_reg, columns=[f"feature_{i+1}" for i in range(X_reg.shape[1])]
)
y_reg_series = pd.Series(y_reg, name="target")

# must add idx to denote groups of data
reg_graph_df = pd.concat([X_reg_df, y_reg_series], axis=1)
reg_graph_df["idx"] = [i % 10 for i in range(100)]
X = reg_graph_df.drop(columns=["target"])
y = reg_graph_df[["idx", "target"]]

model = Transformer_AGNN(
        hidden_dim=16,
        num_transformer_layers=1,
        num_mlp_layers=1,
        num_graph_layers=1,
        graph_nhead=8,
        dropout=0.1,
        share_embedding_mlp=False,
        nhead=8,
        use_cls=False,
        epochs=5,
        lr=1e-3,
        graph="J",
        graph_mode='pure',
        batchnorm=False,
        grad_clip=False,
        random_state=42,
        loss=nn.MSELoss(),
        mode='Regression',
        verbose=1,
        GraphDataFactory=GraphDataFactory,
        rootpath="./",
        output_dim=2,
        input_dim=5,
    )

model.fit(X, y)

print(r2_score(y, model.predict(X)))

Classification Example

MLP Classification Example

from sklearn.datasets import make_classification
import pandas as pd
import torch.nn as nn
from PyTorch2Sklearn.MLP import MLP
from PyTorch2Sklearn.utils.data import TabularDataFactory, TabularDataset
from sklearn.metrics import accuracy_score, r2_score

X_class_2, y_class_2 = make_classification(
        n_samples=100, n_features=5, n_classes=2, n_clusters_per_class=1, random_state=42)
X = pd.DataFrame(
    X_class_2, columns=[f'feature_{i+1}' for i in range(X_class_2.shape[1])])
y = pd.Series(y_class_2, name='target')

model = MLP(
        hidden_dim=16,
        hidden_layers=1,
        dropout=0.1,
        batch_size=32,
        epochs=5,
        lr=1e-3,
        batchnorm=False,
        grad_clip=False,
        random_state=42,
        loss=nn.CrossEntropyLoss(),
        mode='Classification',
        name='MLP',
        verbose=1,
        TabularDataFactory=TabularDataFactory,
        TabularDataset=TabularDataset,
        rootpath='./',
        output_dim=2,
        input_dim=5
    )

model.fit(X, y)

print(r2_score(y, model.predict(X)))

Transformer Classification Example

from sklearn.datasets import make_classification
import pandas as pd
import torch.nn as nn
from PyTorch2Sklearn.Transformer import Transformer
from PyTorch2Sklearn.utils.data import TabularDataFactory, TabularDataset
from sklearn.metrics import accuracy_score, r2_score

X_class_2, y_class_2 = make_classification(
        n_samples=100, n_features=5, n_classes=2, n_clusters_per_class=1, random_state=42)
X = pd.DataFrame(
    X_class_2, columns=[f'feature_{i+1}' for i in range(X_class_2.shape[1])])
y = pd.Series(y_class_2, name='target')

model = Transformer(
        hidden_dim=16,
        num_transformer_layers=1,
        num_mlp_layers=1,
        dropout=0.1,
        batch_size=32,
        share_embedding_mlp=False,
        nhead=2,
        use_cls=False,
        epochs=5,
        lr=1e-3,
        batchnorm=False,
        grad_clip=False,
        random_state=42,
        loss=nn.CrossEntropyLoss(),
        mode='Classification',
        name='Transformer',
        verbose=1,
        TabularDataFactory=TabularDataFactory,
        TabularDataset=TabularDataset,
        rootpath='./',
        output_dim=2,
        input_dim=5
    )

model.fit(X, y)

print(r2_score(y, model.predict(X)))