A collection of fast ranking evaluation metrics built with Numba
Project description
rank_eval
⚡️ Introduction
rank_eval is a collection of fast ranking evaluation metrics implemented in Python, taking advantage of Numba for high speed vector operations and automatic parallelization.
✨ Available Metrics
- Hits
- Precision
- Recall
- rPrecision
- Mean Reciprocal Rank (MRR)
- Mean Average Precision (MAP)
- Normalized Discounted Cumulative Gain (NDCG)
The metrics have been tested against TREC Eval for correctness — through a comparison with pytrec_eval.
The implemented metrics are up to 50 times faster than pytrec_eval and with a much lower memory footprint.
Please note that TREC Eval
uses a non-standard NDCG implementation. To mimic its behaviour, pass trec_eval=True
to rank_eval
's ndcg
function.
🔧 Requirements
- Python 3
- Numpy
- Numba
🔌 Installation
pip install rank_eval
💡 Usage
from rank_eval import ndcg
import numpy as np
# Note that y_true does not need to be ordered
# Integers are documents IDs, while floats are the true relevance scores
y_true = np.array([[[12, 0.5], [25, 0.3]], [[11, 0.4], [2, 0.6]]])
y_pred = np.array(
[
[[12, 0.9], [234, 0.8], [25, 0.7], [36, 0.6], [32, 0.5], [35, 0.4]],
[[12, 0.9], [11, 0.8], [25, 0.7], [36, 0.6], [2, 0.5], [35, 0.4]],
]
)
k = 5
ndcg(y_true, y_pred, k)
>>> 0.7525653965843032
rank_eval supports the usage of y_true elements of different lenght by using Numba Typed List. Simply convert your y_true list of arrays using the provided utility function:
from rank_eval import ndcg
from rank_eval.utils import to_typed_list
import numpy as np
y_true = [np.array([[12, 0.5], [25, 0.3]]), np.array([[11, 0.4], [2, 0.6], [12, 0.1]])]
y_true = to_typed_list(y_true)
y_pred = np.array(
[
[[12, 0.9], [234, 0.8], [25, 0.7], [36, 0.6], [32, 0.5], [35, 0.4]],
[[12, 0.9], [11, 0.8], [25, 0.7], [36, 0.6], [2, 0.5], [35, 0.4]],
]
)
k = 5
ndcg(y_true, y_pred, k)
>>> 0.786890544287473
📚 Documentation
Search the documentation for more details and examples.
🎓 Citation
If you end up using rank_eval to evaluate results for your sceintific publication, please consider citing it:
@misc{rankEval2021,
title = {Rank\_eval: Blazing Fast Ranking Evaluation Metrics in Python},
author = {Bassani, Elias},
year = {2021},
publisher = {GitHub},
howpublished = {\url{https://github.com/AmenRa/rank_eval}},
}
🎁 Feature Requests
If you want a metric to be added, please open a new issue.
🤘 Want to contribute?
If you want to contribute, please drop me an e-mail.
📄 License
rank_eval is an open-sourced software licensed under the MIT license.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for rank_eval-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec2f30399965b8c30aed48902f552b952db6d2981b12dc875d5de4965e9c5652 |
|
MD5 | a7ad582fd408878552ea128f2ea22544 |
|
BLAKE2b-256 | c907e482fb97018c88336608ee81dcbbe9316104d5c9bd785f037ba03c554026 |