ranx: A Blazing Fast Python Library for Ranking Evaluation and Comparison
Project description
🔥 News
- ranx will be featured in ECIR 2022, the 44th European Conference on Information Retrieval!
- Check out the new examples on Google Colab!
- Added a changelog to document few breaking changes introduced in
v.0.1.11
.
⚡️ Introduction
ranx is a library of fast ranking evaluation metrics implemented in Python, leveraging Numba for high-speed vector operations and automatic parallelization. It offers a user-friendly interface to evaluate and compare Information Retrieval and Recommender Systems. Moreover, ranx allows you to perform statistical tests and export LaTeX tables for your scientific publications.
For a quick overview, follow the Usage section.
For a in-depth overview, follow the Examples section.
✨ Available Metrics
- Hits
- Hit Rate
- Precision
- Recall
- F1
- r-Precision
- Mean Reciprocal Rank (MRR)
- Mean Average Precision (MAP)
- Normalized Discounted Cumulative Gain (NDCG)
The metrics have been tested against TREC Eval for correctness.
🔌 Installation
pip install ranx
💡 Usage
Create Qrels and Run
from ranx import Qrels, Run, evaluate
qrels_dict = { "q_1": { "d_12": 5, "d_25": 3 },
"q_2": { "d_11": 6, "d_22": 1 } }
run_dict = { "q_1": { "d_12": 0.9, "d_23": 0.8, "d_25": 0.7,
"d_36": 0.6, "d_32": 0.5, "d_35": 0.4 },
"q_2": { "d_12": 0.9, "d_11": 0.8, "d_25": 0.7,
"d_36": 0.6, "d_22": 0.5, "d_35": 0.4 } }
qrels = Qrels(qrels_dict)
run = Run(run_dict)
Evaluate
# Compute score for a single metric
evaluate(qrels, run, "ndcg@5")
>>> 0.7861
# Compute scores for multiple metrics at once
evaluate(qrels, run, ["map@5", "mrr"])
>>> {"map@5": 0.6416, "mrr": 0.75}
Compare
# Compare different runs and perform statistical tests
report = compare(
qrels=qrels,
runs=[run_1, run_2, run_3, run_4, run_5],
metrics=["map@100", "mrr@100", "ndcg@10"],
max_p=0.01 # P-value threshold
)
print(report)
Output:
>>>
# Model MAP@100 MRR@100 NDCG@10
--- ------- -------- -------- ---------
a model_1 0.320ᵇ 0.320ᵇ 0.368ᵇᶜ
b model_2 0.233 0.234 0.239
c model_3 0.308ᵇ 0.309ᵇ 0.330ᵇ
d model_4 0.366ᵃᵇᶜ 0.367ᵃᵇᶜ 0.408ᵃᵇᶜ
e model_5 0.405ᵃᵇᶜᵈ 0.406ᵃᵇᶜᵈ 0.451ᵃᵇᶜᵈ
📖 Examples
Name | Link |
---|---|
Overview | |
Qrels and Run | |
Evaluation | |
Comparison and Report |
📚 Documentation
Browse the documentation for more details and examples.
🎓 Citation
If you use ranx to evaluate results for your scientific publication, please consider citing it:
@misc{ranx2021,
title = {ranx: A Blazing-Fast Python Library for Ranking Evaluation and Comparison},
author = {Bassani, Elias},
year = {2021},
publisher = {GitHub},
howpublished = {\url{https://github.com/AmenRa/ranx}},
}
🎁 Feature Requests
Would you like to see other features implemented? Please, open a feature request.
🤘 Want to contribute?
Would you like to contribute? Please, drop me an e-mail.
📄 License
ranx is an open-sourced software licensed under the MIT license.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.