twomodalbert 0.0.1

A package to build neural networks based on two separate text fields.

Fine-tune a neural network with two text modes of tunable weights

pip install twomodalbert

Introduction

Let's consider whether we want to classify the below sample as positive or negative.

context	text
This product is horrible, which I didn't expect.	Its website is just incredible!

When only the message is considered, you can see its sentiment is positive due to the word 'incredible'. However, when the context is considered, sentiment will be rather negative.

The TwoModalBERT package allows us to quickly run an experiment with the two-modal neural network architecture described in this section. It allows quickly constructing a model on top of PyTorch and transformers libraries and enables experimenting with weights of two input texts. So, how to use the package?

Usage

1. First, create a config.ini file in your working directory (parameters are described here).

   [GENERAL]
    EPOCHS = 3
    RANDOM_SEED = 42
    BATCH_SIZE = 16
    MAX_SEQ_LEN = 200
    NUM_WORKERS = 2
    PRETRAINED_MODEL_NAME_OR_PATH = bert-base-uncased
    MODEL_SAVE_PATH = best_model_state.bin
   

2. Read the file.

    from configparser import ConfigParser
    config = ConfigParser()
    config.read("config.ini")
   

3. Set the device.

   os.environ["CUDA_VISIBLE_DEVICES"] = "0"
   DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
   

Train

1. initialize DataPreparation and Trainer modules.

   from twomodalbert.DataPreparation import TwoModalDapaPreparation
   from twomodalbert.Trainer import TwoModalBertTrainer
   
   DataPreparation = TwoModalDataPreparation(config=config)
   Trainer = TwoModalBertTrainer(device=DEVICE, config=config)
   

2. Split input df with text, context, label columns and create data loaders.

   (
       train_data_loader,
       train,
       val_data_loader,
       val,
       test_data_loader,
       test,
   ) = DataPreparation.prepare_data(
       df,
       text_column="text",
       context_column="context",
       label_column="label",
       train_size=0.8,
       val_size=0.1,
   )
   

3. Train the model (nn parameters described in this section)

   model, history = Trainer.train_model(
       train_data_loader,
       train,
       val_data_loader,
       val,
       text_size=100,
       context_size=50,
       binary=False,
       text_p=0.3,
       context_p=0.3,
       output_p=0.3,
   
   )
   

Predict

1. Load the model.

   model = TwoModalBERTModel(
       text_size=100,
       context_size=50,
       binary=False,
       text_p=0.3,
       context_p=0.3,
       output_p=0.3,
   )
   
   model.load_state_dict(torch.load(config["GENERAL"]["MODEL_SAVE_PATH"]))
   

2. Evaluate it on a test set (choose any metric from sklearn).

   y_pred, y_test = test_model(model, test_data_loader, DEVICE)
   

3. Run the model on two text inputs

   text = "Its website is just incredible!"
   
   context = "This product is horrible, which I didn't expect."
   
   predict_on_text(model, text, context, DEVICE)
   

Neural network architecture

Below you can find how TwoModalBERT is constructed and what are the class parameters.

First of all, on top of the last BERT layer, a linear layer is added. To be precise, it is added on top of the CLS token. As the CLS token aggregates the entire sequence representation, it is often used in a classification task.

The linear layer transforms input features with hidden size relevant to the BERT model, which is usually 768 for the models available in the transformers package, into features with hidden size equal to predefined context_size and text_size.

In the next step, the dropout layers with probabilities context_p and text_p are added on the top. Why? Because it makes the neural network less sensitive to the specific weights of neurons and not prone to overfitting.

Finally, both branches created similarly are combined and followed by another dropout layer of output_p and an activation function (Sigmoid if binary, else Softmax).

Parameters in config.ini

Settings to be defined in the config.ini file:

Variable	Description	Default Value
EPOCHS	the number of complete passes through the training dataset	3
RANDOM_SEED	a number used to initialize a pseudorandom number generator	42
BARCH_SIZE	the number of training samples to work through before the model’s internal parameters are updated	16
MAX_SEQ_LEN	the maximum length in number of tokens for the inputs to the transformer model	100
NUM_WORKERS	the number of processes that generate batches in parallel	2
PRETRAINED_MODEL_NAME_OR_PATH	of a pre-trained model configuration to load from cache or download (equivalent to transformers)	bert-base-uncased
MODEL_SAVE_PATH	the model save path	best_model.bin

Requirenments

• configparser-5.3.0

• scikit-learn-1.0.2

• torch-1.12.1+cu113

• transformers-4.24.0