rmet 0.1.6

A collection of common recommendation system metrics

Recommender metrics

This library is a colletion of common recommender system (RS) evaluation metrics. Moreover, as RS might perform differently for different user groups due to limitations in available data, this library supports the out-of-the-box computations for subsets of users.

Table of Contents

• Recommender metrics
  • Table of Contents
  • Metrics Overview
  • Installation
  • Usage
    • Single computations
    • Multiple metrics and thresholds
    • Computations per user group
    • Batch-wise evaluation
      • Overall computation
      • Including user groups
    • [Deprecated] Usage metric differences for user features
  • License

Metrics Overview

The following metrics are supported (all with the cut-off threshold k):

• DCG
• nDCG
• Precision
• Recall
• F-score
• Average Precision (AP)*
• Reciprocal Rank (RR)*
• Hitrate
• Coverage
• Average Rank

This library focuses on efficient metric implementations for PyTorch tensors, NumPy arrays and sparse arrays.

Notes:
* Averaging average precision and reciprocal rank of multiple samples leads to mean average precision (MAP) and mean reciprocal rank (MRR), respectively, which are often used in research.

Installation

• Install via pip: python -m pip install rmet

• Or from source: python -m pip install .

Usage

There are different ways to compute metrics. In the following, we are going to list all of them.

Single computations

To compute individual metrics, simply import and call them with your model's output (the logits), the true (known) interactions and some cut-off value k:

from rmet import ndcg
ndcg(model_output, targets, k=10)

Sample output:

0.033423

Note: Coverage does not require the targets attribute.

Multiple metrics and thresholds

You can also call calculate multiple metrics and thresholds efficiently with a single function call. To do so, check out the calculate function:

from rmet import calculate

calculate(
    metrics=["ndcg", "recall"], 
    logits=model_output, 
    targets=targets, 
    k=[10, 50],
    return_individual=False,
    flatten_results=True,
)

Sample output:

{
 'ndcg@10': 0.479,
 'ndcg@50': 0.5,
 'recall@10': 0.350,
 'recall@50': 0.363
}

If return_individual is set, the metrics are also returned on sample level, e.g., for every user, when possible.

Please check out the functions docstring for the full feature description and its extended functionality.

Computations per user group

If you want to get insights into the performance of different user groups, e.g., to study differences in recommendation performance based on the users' countries of origin, check out the calculate_per_group function:

from rmet import calculate_per_group

# your actual groups as an iterable, e.g., list or pd.Series
group_assignment = ["AT", "DE", "FR", ...] 

calculate_per_group(
    group_name="country",
    group_assignment=group_assignment,
    metrics=["ndcg", "recall"], 
    logits=model_output, 
    targets=targets, 
    k=[10],
    return_individual=False,
    flatten_results=True,
)

Sample output:

{
 'ndcg@10/country_AT': 0.173,
 'ndcg@10/country_DE': 0.199,
 'ndcg@10/country_FR': 0.239,
 'recall@10/country_AT': 0.282,
 'recall@10/country_DE': 0.301,
 'recall@10/country_FR': 0.357,
}

Batch-wise evaluation

For big datasets and real-world applications, gathering all the logits and targets before computing the recommendation metrics may be too resource-intensive. To simplify calculations in such scenarios, we provide BatchEvaluator, a class that evaluates and stores intermediary results.

Overall computation

from rmet import BatchEvaluator

# instantiate the evaluator class
batch_evaluator = BatchEvaluator(
    metrics=["ndcg"],
    top_k=[10],
)

# iterate over the batches
for batch in batches:
    user_indices, logits, targets = batch

    # you need to call 'eval_batch' for each batch
    batch_evaluator.eval_batch(
        user_indices=user_indices
        logits=logits,
        targets=targets,
    )

# use 'get_results' to determine the final results
batch_evaluator.get_results()

Sample output:

{
 'ndcg@10': 0.121,
}

Including user groups

BatchEvaluator.eval_batch() also accepts group assignments as input, which allows the computation of metrics on group and global level.

from rmet import BatchEvaluator

# instantiate the evaluator class
batch_evaluator = BatchEvaluator(
    metrics=["ndcg"],
    top_k=[10],
)

# iterate over the batches
for batch in batches:
    # batch also returns group_assignments, which is 
    # a mapping from group_name to their values, e.g.,
    # {"country": ["AT", "DE", ...], "gender": ["m", "n", ...]} 
    user_indices, logits, targets, group_assignments = batch

    # you need to call 'eval_batch' for each batch
    batch_evaluator.eval_batch(
        user_indices=user_indices
        logits=logits,
        targets=targets,
        group_assignments=group_assignments,
    )

# use 'get_results' to determine the final results
batch_evaluator.get_results()

Sample output:

{
 'ndcg@10': 0.121,
 'ndcg@10/country_AT': 0.115,
 'ndcg@10/country_DE': 0.142,
 'ndcg@10/gender_m': 0.087,
 'ndcg@10/gender_f': 0.156,
}

[Deprecated] Usage metric differences for user features

[NOTE] This feature is deprecated, use calculate_per_group and the BatchEvaluator with group_assignments instead.

One can also instantiate the UserFeature class for some demographic user feature, such that the performance difference of RS on for different users can be evaluated, e.g., for male and female users in the context of gender.

To do so, you first need to specify which feature belongs to which user via the UserGroup class and then simply call calculate_for_group similar to calculate above.

from rmet import UserFeature, calculate_for_feature
ug_gender = UserFeature("gender", ["m", "m", "f", "d", "m"])

calculate_for_feature(
    ug_gender, 
    metrics=["ndcg", "recall"], 
    logits=model_output, 
    targets=targets, 
    k=10,
    return_individual=False,
    flatten_results=True,
)

Sample output:

{
    'gender_f': {'ndcg@10': 0.195, 'recall@10': 0.125},
    'gender_m': {'ndcg@10': 0.779, 'recall@10': 0.733},
    'gender_d': {'ndcg@10': 0.390, 'recall@10': 0.458},
    'gender_f-m': {'ndcg@10': -0.584, 'recall@10': -0.608},
    'gender_f-d': {'ndcg@10': -0.195, 'recall@10': -0.333},
    'gender_m-d': {'ndcg@10': 0.388, 'recall@10': 0.275}
}

License

MIT License - see the LICENSE file for more details.