data_improvement_library

A group of techniques and methods which can potentlially improve visual explainability models based on the Learning to Rank aproach

Usage Guide

Each module in the library requires a set of input variables that define its functionality. This section describes these variables and provides usage examples for each module.

Data Augmentation Module

Input Variables

• data_dir (str): Path to the input data file containing user interactions.
• vector_dir (str): Path to the file containing image embeddings.
• image_dir (str): Path to the directory where images are stored.
• output_dir (str): Path where the processed data will be stored.
• output_name (str, optional): Name of the output file. Default: TRAIN_IMG.
• embedding_model (torch.nn.Module, optional): Pretrained model for embedding extraction. If left empty, a pretrained ViT Large 14 model is used by default.
• no_aug (bool, optional): If True, disables data augmentation.
• batch_size (int, optional): Number of images processed per batch. Default: 32.
• apply_all (bool, optional): If True, applies all available transformations.
• labels (list, optional): List of column names in the dataset.

Output Variables

This module does not produce a return value. Instead, it saves the processed data in pickle format based on the input variables.

Usage Example

import data_improvement_library

# Process images with data augmentation and embedding extraction
data_improvement_library.augment_data(data_dir="data/user_data.pkl", vector_dir="data/image_vectors.pkl",
             image_dir="images/", output_dir="processed_data/",
             output_name="TRAIN_IMG", embedding_model=None,
             no_aug=False, batch_size=32, apply_all=True, labels=None)

Embedding Generation Module (new_embeddings.py)

Input Variables

• directory (str): Directory where images are stored.
• output_dir (str): Directory where generated embeddings will be saved.
• output_name (str): Name of the output file.
• embedding_model (torch.nn.Module or None): Pretrained model for embedding extraction. If left empty, a pretrained ViT Large 14 model is used by default.
• batch_size (int): Number of images processed per batch.

Output Variables

This module does not return a value. Instead, it saves the processed data in pickle format.

Usage Example

import data_improvement_library

# Generate embeddings for a set of images
data_improvement_library.create_new_embeddings(directory="images/", output_dir="embeddings/", output_name="img_vectors.pkl",
                      embedding_model=None, batch_size=32)

Negative Selection with PU Learning (pu_negatives.py)

Input Variables

• data_file (str): Path to the user interaction data file.
• vector_file (str): Path to the image embeddings file.
• outdir_name (str): Output directory where the processed dataset will be saved.
• centroid (int, optional): Centroid percentile (default: 90).
• factor (float, optional): Distance adjustment factor.
• labels (list, optional): Column labels in the dataset.

Output Variables

This module does not return a value. Instead, it saves the processed data in pickle format.

Usage Example

import data_improvement_library

# Generate balanced negative samples
data_improvement_library.resample_negatives(data_file="data/user_data.pkl", vector_file="data/image_vectors.pkl",
                   outdir_name="processed_data/", centroid=90, factor=1.0)

Positive Selection with PU Learning (pu_positives.py)

This module can be used in two main ways:

1. As a Callback in model training with PyTorch Lightning, allowing dynamic updates of user centroids in each epoch.
2. As Preprocessing, applied before training to the dataset.

Usage as Callback

The module provides the CallbackEndPositives class, which extends Callback from PyTorch Lightning. Its purpose is to update user centroids and resample positive samples at the end of each training epoch.

For proper functionality, the model must meet the following requirements:

• Dataset in train_dataloader must contain:
  • datamodule.image_embeddings: Tensor of image embeddings.
  • dataframe: Must include columns id_user, id_img, id_restaurant, take.
• Embedding access: Must be indexable by id_img in the dataframe.
• Embedding format: Must be a tensor convertible to NumPy (.cpu().detach().numpy()).
• Data sampling: Must allow modification of pu_dataset in the dataset.
• DataModule requirements: Must provide image_embeddings and handle pu_dataset correctly.

Usage Example

import data_improvement_library
from pytorch_lightning import Trainer

# Create an instance of the callback
callback_end = data_improvement_library.CallbackEndPositives()

# Configure the trainer with the callback
trainer = Trainer(callbacks=[callback_end])

# Start training
trainer.fit(model)

Usage as Preprocessing

This module can also generate a dataset with new positive samples selected based on user similarity before training. This is achieved by calculating user centroids and resampling positive samples. The functionality is divided into two functions: centroid_users for computing user centroids and resample_positives for updating the dataset with new positive samples using cosine similarity filtering.

Input Variables

For the function centroid_users:

• dataframe (pandas.DataFrame): Dataset of user-image pairs.
• vectors (numpy.array): Image embedding representations.
• labels (list, optional): List of column names in the dataset.

For the function resample_positives:

• dataframe (pandas.DataFrame): Dataset of user-image pairs.
• centroids (dict): Dictionary where keys represent user IDs, and values are user image centroids as embeddings.
• k (int): Number of similar users considered for adding positive samples per user.

Output Variables

• centroid_users returns centroids, a dictionary where keys are user IDs and values are user image centroids as embeddings.
• resample_positives returns the updated dataframe with new positive samples.

Usage Example

import data_improvement_library
import pandas as pd
import numpy as np

# Load the dataset
dataframe = pd.read_pickle("data/user_data.pkl")

# Load image embeddings
image_vectors = np.load("data/image_vectors.npy")

# Compute user centroids
centroids = data_improvement_library.centroid_users(dataframe, image_vectors)

# Generate a new dataset with balanced positive samples
new_dataframe = data_improvement_library.resample_positives(dataframe, centroids, 3)

# Save the processed dataset
new_dataframe.to_pickle("processed_data/balanced_dataset.pkl")