autoqrels
autoqrels is a tool for automatically inferring query relevance assessments (qrels).
Currently, it supports the one-shot labeling approach (1SL) presented in MacAvaney and Soldaini, One-Shot Labeling for Automatic Relevance Estimation, SIGIR 2023.
This package adheres to the ir-measures API, which means it can
be directly used by various tools, such as PyTerrier.
Getting started
You can install autoqrels using pip:
pip install autoqrels
You can also work with the repository locally:
git clone https://github.com/seanmacavaney/autoqrels.git
cd autoqrels
python setup.py develop
API
The primary interface in autoqrels is autoqrels.Labeler. A Labeler exposes a
method, infer_qrels(run, qrels), which returns a new set of qrels that covers the
provided run:
runis a Pandas DataFrame with the columnsquery_id(str),doc_id(str), andscore(float)qrelsis a Pandas DataFrame with the columnsquery_id(str),doc_id(str), andrelevance(int)- The return value is a Pandas DataFrame with the columns
query_id(str),doc_id(str), andrelevance(float)
Labelers also expose several measure definitions compatible with ir_measures:
labeler.SDCG@k,
labeler.RBP(p=persistence),
labeler.P@k.
These measures can be used to calculate the corresponding effectivness, with the
addition of the labeler's inferred qrels. See the ir-measures documentation
for more details.
We'll now explore the available Labeler implementations.
autoqrels.oneshot: 1SL (One-shot Labeling)
Reproduction: See repro instructions in repro/oneshot.
One-shot labelers work over a single known relevant document per query. An error is raised if multiple relevant documents are provided.
Example:
import autoqrels
import ir_datasets
dataset = ir_datasets.load('msmarco-passage/trec-dl-2019')
duot5 = autoqrels.oneshot.DuoT5(dataset=dataset, cache_path='data/duot5.cache.json.gz')
# measures:
duot5.SDCG@10
duot5.P@10
duot5.RBP
Citation
If you use this work, please cite:
@inproceedings{autoqrels,
author = {MacAvaney, Sean and Soldaini, Luca},
title = {One-Shot Labeling for Automatic Relevance Estimation},
booktitle = {Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval},
year = {2023},
url = {https://arxiv.org/abs/2302.11266}
}
Metadata
Release files for autoqrels 0.0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| autoqrels-0.0.1.tar.gz | 14.0 kB | Without provenance | Details for autoqrels-0.0.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| autoqrels-0.0.1-py3-none-any.whl | Python 3 | none | any | Details for autoqrels-0.0.1-py3-none-any.whl |
Total release size: 31.5 kB
Release files / autoqrels-0.0.1.tar.gz
| Download URL | autoqrels-0.0.1.tar.gz |
|---|---|
| Size | 14.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4ec3b2bd5eaa1dce2b3755a1e67e9864fd5c4ce9fe5bfebb0ba754884df4ef07
|
|
BLAKE2b-256 checksum How to use checksums |
6d752c8b6c658f624c44d8308e28da08fe267fb30c9e6a7ff7007312e0da5db1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.12.5
|
Release files / autoqrels-0.0.1-py3-none-any.whl
| Download URL | autoqrels-0.0.1-py3-none-any.whl |
|---|---|
| Size | 17.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9fb17ba2e459853c1d45a17729421e5ccbf74267c79e8f946240eb49f190b938
|
|
BLAKE2b-256 checksum How to use checksums |
868665406d3116ab5ef32b757465ac4de93019220bf29357454645026254c1ac
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.12.5
|
Security for autoqrels 0.0.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.