irec 1.2.5

Reinforcement Learning Recommender Systems Framework

Introduction • Install • Quick Start

Introduction

For Python >= 3.8

Interactive Recommender Systems Framework

Main features:

• Several state-of-the-art reinforcement learning models for the recommendation scenario
• Novelty, coverage and much more different type of online metrics
• Integration with the most used datasets for evaluating recommendation systems
• Flexible configuration
• Modular and reusable design
• Contains multiple evaluation policies currently used in the literature to evaluate reinforcement learning models
• Online Learning and Reinforcement Learning models
• Metrics and metrics evaluators are awesome to evaluate recommender systems in different ways

Also, we provide a amazing application created using the iRec library (under the app/ folder) that can be used to setup a experiment under 5~ minutes with parallel processes, log registry and results views. The main features are:

• Powerful application to run any reinforcement learning experiment powered by MLflow
• Entire pipeline of execution is fully parallelized
• Log registry
• Results views
• Statistical test
• Extensible environment

Install

Install with pip:

pip install irec

Examples

Under app/ folder is a example of a application using iRec and MLflow, where different experiments can be run with easy using existing recommender systems.

Check this example of a execution using the example application:

cd app
metrics=(Hits Precision Recall);
models=(Random MostPopular UCB ThompsonSampling EGreedy);
metric_evaluator="IterationsMetricEvaluator"
bases=("Netflix 10k" "Good Books" "Yahoo Music 10k");
# run agents
./run_agent_best.py --dataset_loaders "${bases[@]}" --agents "${models[@]}"

# evaluate agents using the metrics and metric evaluator defined
./eval_agent_best.py --dataset_loaders "${bases[@]}"\
--agents "${models[@]}" --metrics "${metrics[@]}"\
--metric_evaluator="$metric_evaluator"

# print latex table with results and statistical test
./print_latex_table_results.py --dataset_loaders "${bases[@]}"\
--agents "${models[@]}" --metrics "${metrics[@]}"\
--metric_evaluator="$metric_evaluator"

Datasets

Our framework has the ability to use any type of dataset, as long as it is suitable for the recommendation domain and is formatted correctly. Below we list some datasets tested and used in some of our experiments.

Dataset	Domain	Sparsity	Link
MovieLens 100k	Movies	93.69%	Link
MovieLens 1M	Movies	95.80%	Link
MovieLens 10M	Movies	98.66%	Link
Netflix	Movies	98.69%	Link
Ciao DVD	Movies	99.97%	Link
Yahoo Music	Musics	97.63%	Link
LastFM	Musics	99.84%	Link
Good Books	Books	98.88%	Link
Good Reads	Books	99.50%	Link
Amazon Kindle Store	Products	99.97%	Link
Clothing Fit	Clothes	99.97%	Link

Models

The recommender models supported by irec are listed below.

Year	Model	Paper	Description
2002	ε-Greedy	Link	In general, ε-Greedy models the problem based on an ε diversification parameter to perform random actions.
2013	Linear ε-Greedy	Link	A linear exploitation of the items latent factors defined by a PMF formulation that also explore random items with probability ε.
2011	Thompson Sampling	Link	A basic item-oriented bandit algorithm that follows a Gaussian distribution of items and users to perform the prediction rule based on their samples.
2013	GLM-UCB	Link	It follows a similar process as Linear UCB based on the PMF formulation, but it also adds a sigmoid form in the exploitation step and makes a time-dependent exploration.
2018	ICTR	Link	It is an interactive collaborative topic regression model that utilizes the TS bandit algorithm and controls the items dependency by a particle learning strategy.
2015	PTS	Link	It is a PMF formulation for the original TS based on a Bayesian inference around the items. This method also applies particle filtering to guide the exploration of items over time.
2019	kNN Bandit	Link	A simple multi-armed bandit elaboration of neighbor-based collaborative filtering. A variant of the nearest-neighbors scheme, but endowed with a controlled stochastic exploration capability of the users’ neighborhood, by a parameter-free application of Thompson sampling.
2017	Linear TS	Link	An adaptation of the original Thompson Sampling to measure the latent dimensions by a PMF formulation.
2013	Linear UCB	Link	An adaptation of the original LinUCB (Lihong Li et al. 2010) to measure the latent dimensions by a PMF formulation.
2020	NICF	Link	It is an interactive method based on a combination of neural networks and collaborative filtering that also performs a meta-learning of the user’s preferences.
2016	COFIBA	Link	This method relies on upper-confidence-based tradeoffs between exploration and exploitation, combined with adaptive clustering procedures at both the user and the item sides.
2002	UCB	Link	It is the original UCB that calculates a confidence interval for each item at each iteration and tries to shrink the confidence bounds.
2002	Entropy	Link	The entropy of an item i is calculated using the relative frequency of the possible ratings. In general, since entropy measures the spread of ratings for an item, this strategy tends to promote rarely rated items, which can be considerably informative.
2002	log(pop)*ent	Link	It combines popularity and entropy to identify potentially relevant items that also have the ability to add more knowledge to the system. As these concepts are not strongly correlated, it is possible to achieve this combination through a linear combination of the popularity ρ of an item i by its entropy ε: score(i) = log(ρi) · εi.
-	Random	Link	This method recommends totally random items.
-	Most Popular	Link	It recommends items with the higher number of ratings received (most-popular) at each iteration.
-	Best Rated	Link	Recommends top-rated items based on their average ratings in each iteration.

Metrics

The recommender metrics supported by iRec are listed below.

Metric	Reference	Description
Hits	Link	Number of recommendations made successfully.
Precision	Link	Precision is defined as the percentage of predictions we get right.
Recall	Link	Represents the probability that a relevant item will be selected.
EPC	Link	Represents the novelty for each user and it is measured by the expected number of seen relevant recommended items not previously seen.
EPD	Link	EPD is a distance-based novelty measure, which looks at distances between the items in the user’s profile and the recommended items.
ILD	Link	It represents the diversity between the list of items recommended. This diversity is measured by the Pearson correlation of the item’s features vector.
Gini Coefficient	Link	Diversity is represented as the Gini coefficient – a measure of distributional inequality. It is measured as the inverse of cumulative frequency that each item is recommended.
Users Coverage	Link	It represents the percentage of distinct users that are interested in at least k items recommended (k ≥ 1).

Configuration Files

iRec has some configuration files to define an experiment, such as dataset settings, agents, policies and evaluation metrics. Below we present brief examples about each of the files available in this framework.

For more details on configuration files, go to configuration_files

dataset_loaders.yaml

This configuration file stores all the configurations related to the bases that will be used during the execution of an experiment.

'MovieLens 10M':
  DefaultValidationDatasetLoader:
    dataset_path: ./data/datasets/MovieLens 10M/
    train_size: 0.8
    test_consumes: 1
    crono: False
    random_seed: 0
︙

dataset_agents.yaml

This configuration file stores the settings of the agents (Recommenders) that will be used in the experiments.

'MovieLens 10M':
  LinearUCB:
    SimpleAgent:
      action_selection_policy:
        ASPGreedy: {}
      value_function:
        LinearUCB:
          alpha: 1.0
          item_var: 0.01
          iterations: 20
          num_lat: 20
          stop_criteria: 0.0009
          user_var: 0.01
          var: 0.05
 ︙

agents_variables.yaml

In this configuration file it is possible to define a search field for the variables of each agent, which will be used during the grid search

PTS:
  num_lat: [10,20,30,40,50]
  num_particles: linspace(1,10,5)
  var: linspace(0.1,1,10)
  var_u: linspace(0.1,1,10)
  var_v: linspace(0.1,1,10)

evaluation_policies.yaml

The evaluation policies are defined in this configuration file. To conduct an experiment, we need to define how the recommendation process will be executed and the user-item interactions. We specify these settings in this file according to the experiment's objectives.

Interaction:
  num_interactions: 100
  interaction_size: 1
  save_info: False
︙

metric_evaluators.yaml

This file defines the evaluation metrics for an experiment. This file is responsible for providing details on how to assess the interactions performed during the assessment process.

UserCumulativeInteractionMetricEvaluator:
  interaction_size: 1
  interactions_to_evaluate:
    - 5
    - 10
    - 20
    - 50
    - 100
  num_interactions: 100
  relevance_evaluator_threshold: 3.999

︙

defaults.yaml

This configuration file defines the general settings of an experiment. We can define the agents, the base, the policy and the evaluation metric, but also some additional information.

agent: LinearUCB
agent_experiment: agent
data_dir: data/
dataset_experiment: dataset
dataset_loader: 'MovieLens 1M'
evaluation_experiment: evaluation
evaluation_policy: Interaction
metric: Hits
metric_evaluator: UserCumulativeInteractionMetricEvaluator
pdf_dir: pdf/
tex_dir: tex/

API

For writing anything new to the library (e.g., value function, agent, etc) read the documentation.

Contributing

All contributions are welcome! Just open a pull request.