Interface to ndeval.c
Project description
pyndeval
A python interface to TREC's ndeval.c
, used for computing diversity retrieval metrics.
Getting Started
From pip:
pip install pyndeval
Or install from source:
git clone https://github.com/seanmacavaney/pyndeval.git
cd pyndeval
python setup.py install
Usage
import pyndeval
from pyndeval import SubtopicQrel, ScoredDoc
# provide qrels as a list of tuples
qrels = [
SubtopicQrel("0", "a", "A", 1),
SubtopicQrel("0", "b", "B", 1),
SubtopicQrel("0", "b", "D", 1),
SubtopicQrel("0", "c", "C", 1),
]
# provide run as a list of tuples
run = [
ScoredDoc("0", "A", 9.3),
ScoredDoc("0", "D", 8.4),
ScoredDoc("0", "E", 8.1), # not in qrels
ScoredDoc("0", "B", 7.6),
# C not retrieved
]
pyndeval.ndeval(qrels, run)
{'0': {
'ERR-IA@5': 0.3933,
'ERR-IA@10': 0.3907,
'ERR-IA@20': 0.3907,
'nERR-IA@5': 0.8297,
'nERR-IA@10': 0.8297,
'nERR-IA@20': 0.8297,
'alpha-DCG@5': 0.4052,
'alpha-DCG@10': 0.3998,
'alpha-DCG@20': 0.3997,
'alpha-nDCG@5': 0.7868,
'alpha-nDCG@10': 0.7868,
'alpha-nDCG@20': 0.7868,
'NRBP': 0.3906,
'nNRBP': 0.8620,
'MAP-IA': 0.5000,
'P-IA@5': 0.2000,
'P-IA@10': 0.1000,
'P-IA@20': 0.0500,
'strec@5': 0.6666,
'strec@10': 0.6666,
'strec@20': 0.6666
}}
Supported measures
pyndeval
supports the following measures:
ERR-IA@k
nERR-IA@k
alpha-DCG@k
alpha-nDCG@k
P-IA@k
strec@k
NRBP
nNRBP
MAP-IA
Measures with @k
support values from 1-20 (upper limit from ndeval.c
).
Measures are provided as a list of strings with the measures=
parameter.
pyndeval.ndeval(qrels, run, measures=["ERR-IA@7", "MAP-IA"])
Reusing qrels
If you're running multiple times for the same set of qrels and measures, you can speed it up by
building a RelevanceEvaluator
object, which caches the internal qrel representations.
ev = pyndeval.RelevanceEvaluator(qrels)
ev.evaluate(run1)
ev.evaluate(run2)
ev.evaluate(run3)
ev.evaluate(run4)
Iterable results
for result in pyndeval.ndeval_iter(qrels, run):
{"query_id": "0", ...}
for result in ev.evaluate_iter(run):
{"query_id": "0", ...}
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pyndeval-0.0.6.tar.gz
(15.1 kB
view hashes)
Built Distributions
Close
Hashes for pyndeval-0.0.6-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2862037125164b0293b5a17f5d418dc92a11a51b3ec9ac117de9cf4fb12fef41 |
|
MD5 | f63b6e8f9886d73fb1ada6348cfbf2ed |
|
BLAKE2b-256 | 339d81dea0ad7f2013e663805b2207cb79c6a214254116861dc7439b453b79af |
Close
Hashes for pyndeval-0.0.6-pp39-pypy39_pp73-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3192c336813e23f3b1c57e622a07c6aea6bfe40bea5944b0f050e4ad25d0b3fb |
|
MD5 | 3cf709cd56e3a3053daef73e071be06f |
|
BLAKE2b-256 | c252f2ca886fac76772e0ceea5725619701c63a10feacf9939da37a3a460a21e |
Close
Hashes for pyndeval-0.0.6-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c5369b58f25eed62464f47168f07b069a29785420fae0d10f66caa1579b58a6c |
|
MD5 | fbcd980cfc0127d47c2be9aa75b0e455 |
|
BLAKE2b-256 | 4d639e310c5ec26019ae503579a92f34838bd226dd4d8a26b491291217729743 |
Close
Hashes for pyndeval-0.0.6-pp38-pypy38_pp73-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | be78fc5724b116756394ca14aff56f37cb183b2e602f0c4f2641641a2b51880b |
|
MD5 | 80875be1701a38691cc8e5d141b9e52c |
|
BLAKE2b-256 | 44c2ceecf9731d99f4d504403695af6dcb7bd198c0a8266253b4efe9b35ffa8f |
Close
Hashes for pyndeval-0.0.6-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0d8cd8ca027f8b2e114d0d81a604dd2b6d3898d8be06440251b244cb0646b07e |
|
MD5 | 6d3322b64ad9be026bde368112dc53e5 |
|
BLAKE2b-256 | 3d74871d477b13ece208954ee6d7a225552e29b0e0c05ba095994a179cd50146 |
Close
Hashes for pyndeval-0.0.6-pp37-pypy37_pp73-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bede98e0cfc14f82c491ffecd6892cbf6c7991b3b4b362c554c6e7d217d1b055 |
|
MD5 | df5321d69ed95ba071e56107badf9773 |
|
BLAKE2b-256 | ec52a53f1062eca7c7114f95e3c8cd075f8f442679d210ea8deae129f7582e5c |
Close
Hashes for pyndeval-0.0.6-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 199925c319b3cc4db137bf2ed075a550ad1526bdc49d8767f6a81ba20b1af9f1 |
|
MD5 | 2ac01ebf82ee2efb0f11c245371dc6ac |
|
BLAKE2b-256 | 6c797058430925e774845218e3d792b68682cfdbc2c670a5c8ba007cf1d5aa54 |
Close
Hashes for pyndeval-0.0.6-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5430d7b898a521265387a636ea016b7f180c7659915eb24523a236d6a1e3b608 |
|
MD5 | 4f4c1766ea854453149a66432abf2fb0 |
|
BLAKE2b-256 | 921b88b0605d79d5139fe45db5b3debad4cea76ecbba271931ae43dd87e7cf99 |
Close
Hashes for pyndeval-0.0.6-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7e9259c12136187ad8482ea74feac7f7a6c73eefd18f831525e0cc6dcaaa83ac |
|
MD5 | 195305e15ab2d8be67e35792ed8bb853 |
|
BLAKE2b-256 | 4267fb4d4935934d8a5fdcd23be6fb2e838cb7ccb1fb18388856145589b377c6 |
Close
Hashes for pyndeval-0.0.6-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | db6968bfc2cdbb10cb4a8b3b8c08bb821a696cbfd9fd7dda9e691edbb8981208 |
|
MD5 | df7c16b9affa91180539eb83035a63ee |
|
BLAKE2b-256 | 2d7fe2b0069a0ff8f24551e4cf4bfb5146fa0cb24d3ff4628d04cadd4bfbe173 |
Close
Hashes for pyndeval-0.0.6-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f37a7edfde06a8f7b7672dcc7679c4eb313d81e5e351505686d4c9e09bef9ecd |
|
MD5 | 1c0bcedd34bfc0813d82ec7da3411215 |
|
BLAKE2b-256 | 3bf1d0ce974370c05b8fa56b6a1207215d5d43264f4602f91bd7d6042f784f28 |
Close
Hashes for pyndeval-0.0.6-cp312-cp312-macosx_10_13_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 38850be5fc5641a54b794a66a9355f4ac947c390d5244a47d474cdd242f36b2a |
|
MD5 | 9537760757ce21fa770fb65b6c97159c |
|
BLAKE2b-256 | 676981bc52bfb7fcb194f57de7749bcb404ee222efb861caf00eac7150770f4a |
Close
Hashes for pyndeval-0.0.6-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | afbd42fd983302677b21d41748b469b17084232095721b00d226f2dcbbca4c23 |
|
MD5 | 2f271863cedf4ab62eea344bc31f60cc |
|
BLAKE2b-256 | e9c453db58921432407db28f358f7aa1c84bd698a6b49f892f2ec1cc628935fd |
Close
Hashes for pyndeval-0.0.6-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e1cf331aaa643242ca6eaf607c7ee95ae138fdf4c8c92837a7d5bdef58e37e6c |
|
MD5 | 754fbc18874e1e3d4c05d2cbd3b6b727 |
|
BLAKE2b-256 | 8504eef1d904b0ed652fb2efe200bba487c1cb7015991ede6aeecae1fff24265 |
Close
Hashes for pyndeval-0.0.6-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8020f4a4a9f54a52fd89824b01fce716dd58c4998d76e8790f0b13c3c3b27fbc |
|
MD5 | 07a221bab7c4858ccb768ed5d1fd2222 |
|
BLAKE2b-256 | 2444198bcc5ff7bfc3e1924a9662c99fa1e1c6f4062b40b4947ccd2bba3f58fb |
Close
Hashes for pyndeval-0.0.6-cp311-cp311-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b507e3a232df94419500d51eb2024da63110714c4886641920c2ecef9f29bb06 |
|
MD5 | 36cff2a9d480a08ded008b9346849ff0 |
|
BLAKE2b-256 | 90b8350750bf22f14f7d7a4d0d94825b1247b5b3f2e3a50c9c1bff26b3874060 |
Close
Hashes for pyndeval-0.0.6-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8732385e0f4e2df157b4675b4cc3b18c4a1cc0f67da0d81e5985cd7709812e81 |
|
MD5 | 03a324e0c7cb30c225f21356ce6c4122 |
|
BLAKE2b-256 | 920abb68a895cd6a17cee418bde503037e22b09d28c703a131e4c79ed0caabc6 |
Close
Hashes for pyndeval-0.0.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9c9335efccb979da513121eb7496bd918104f51a53317b135a3cf20745c22605 |
|
MD5 | d6c85184fb520c57b210f9c1a781d715 |
|
BLAKE2b-256 | 9169fdc3902d16d06b10af3404cb2c90496502d9b15ffe9861de4a24e023549a |
Close
Hashes for pyndeval-0.0.6-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 48ef1d5788772cc0b5bcb39de6f113e1708c00573f6f89ae5bbf62303d68d0a1 |
|
MD5 | 5e7e5b5fcfe7134c2fcd22c64d7a151a |
|
BLAKE2b-256 | cf0dc2fc8b2c1d75b5dad896b4aacb6b718408b2b6a684eaf77f7bedb6e6cf55 |
Close
Hashes for pyndeval-0.0.6-cp310-cp310-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 47e04702f6b2abd0a458daea6c72a98854a2fec5bb4e8bacaa80899b13778d0b |
|
MD5 | b980abfed2c48522b7f72b1c3e095edb |
|
BLAKE2b-256 | 6c6d1637e045b51825f5c6329aa569a2f56e3eca59a34fb8f390715252d3c179 |
Close
Hashes for pyndeval-0.0.6-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1aea395ba75c25a07d0d7939fd878f98cf76530dffbfc1fa0c5fc4d9836c5362 |
|
MD5 | dc11a3c6ca6c3db9862bfd9c3b3edfb5 |
|
BLAKE2b-256 | b1aee568b807cde8dd3f37387b3bde05f1fecc376d1970f4025fbc74a5aedf82 |
Close
Hashes for pyndeval-0.0.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 394c370235644199de2ffcb8b4690839100a8a552f2a8fb83c8065ffc5c915a8 |
|
MD5 | 8970006c1aac6b843cb6f34c34879ce1 |
|
BLAKE2b-256 | e636cc966a5682821e99411285381c835fdf3cf1d27ab5f9c3716a77b0c5a2d1 |
Close
Hashes for pyndeval-0.0.6-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cf8cbd36050cbd023f726602886ddbfc820d93b09dcf2d3ae9f46705b4a29d23 |
|
MD5 | 7fce10e3a97fb9deee8b0d281b6d3a35 |
|
BLAKE2b-256 | a2a7aedf60db3f40906aa7b826a1169a8cd6c971543d55bdb5943bcfbefb6b43 |
Close
Hashes for pyndeval-0.0.6-cp39-cp39-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8516333c23fe152b799534d164aed40c38ff58f4025cf46dbab9997ee35181cb |
|
MD5 | bfdcd8c73f7670484cbe7bc560e80399 |
|
BLAKE2b-256 | 448aeab6cb9b2d1f8e76ff63eb805ed0b5c106c86421375fdc305097dfa54ba6 |
Close
Hashes for pyndeval-0.0.6-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3447b71f17cb5adb87412e3b5594d5fc56119502be58bd9d61a242b6e642c0e3 |
|
MD5 | e66ea3f60941e8a91b5f11d9a5266bca |
|
BLAKE2b-256 | 729827fbdbc03ede4f405abaec1546f1bd8060cc9b8a466d325b35eb0a28c261 |
Close
Hashes for pyndeval-0.0.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 93d5dcbb917e7e63a57f0dda9913a9d877182284abbeeeca2ee89891b59b6f18 |
|
MD5 | e204f3fe6c63013e948581333bf76be7 |
|
BLAKE2b-256 | 73c53c7ec550ca1bd6dc4dc2e2b64587f12d990e65f377b1249d45e0ae93c2f2 |
Close
Hashes for pyndeval-0.0.6-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1af52d0f956eee0da19d2aae8e3405ea524df86286eeb568d8bf03f65468838e |
|
MD5 | 2daed551bf0034b7c2113a0bf2bf0271 |
|
BLAKE2b-256 | 320d5237c1390024f63d84040b2e56dd9d72714cb8cc043ca826ba55de588230 |
Close
Hashes for pyndeval-0.0.6-cp38-cp38-macosx_11_0_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5e2aa1f49fe6bf2245ff9238cece57e163e54a8f0ee6ae84ae38969da1278534 |
|
MD5 | 3552acbf4ca2fd057b4501b03d13bad8 |
|
BLAKE2b-256 | 8e5d04b6a2e240c8694076f95bc98773497f4d8a931ea6c940052dd7f8042747 |
Close
Hashes for pyndeval-0.0.6-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 25112770dfe0884ee7ded7ca3cc39e3cf42e61e9d362fa779e47812e45e8f5c0 |
|
MD5 | 601d953755b3a3abae036711189db2ee |
|
BLAKE2b-256 | d8b40b2a397e6b3f518c0031e450f1392268d2dc49087045854d2ee065c7cdd3 |
Close
Hashes for pyndeval-0.0.6-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 68d1cca438897f6d0d4b221fb65e948a06a1182665d4eae89de2b1802e1b767d |
|
MD5 | 580fbe6aff99218815323de6df5d26bc |
|
BLAKE2b-256 | 88f484cff3b30076f97c4c80fc6ed387fbf5b5bf1e35fc7947377249b78b0f8c |
Close
Hashes for pyndeval-0.0.6-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cfa69176f3689958c6672aafc1c995822ade3b61bc37cd98e8423bf9c04816da |
|
MD5 | 2135d4f67dda9d0679cc9f34fec4b186 |
|
BLAKE2b-256 | bf433ea3fc267fb87a425376a453f8c195589c4e645299d1523a8f881f38c5a2 |
Close
Hashes for pyndeval-0.0.6-cp36-cp36m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 517828daabc6b3e76038fcc0320584c468ba9e63b903eecd11d3169042d4163e |
|
MD5 | 587693516803817c2f28cc1f06911e87 |
|
BLAKE2b-256 | 0d6099502a6bba81858e40e922f0e0d5c3eb468275a8a2e7c8cff8567f79d87a |