Skip to main content

Threshold Average Precision: metric for evaluating retrieval rankings

Project description

TAP-k -- Python module for Threshold Average Precision

Threshold Average Precision is a metric for evaluating IR applications that present retrieval lists with a ranking score to the end user. It is described in the following publication:

Hyrum D. Carroll, Maricel G. Kann, Sergey L. Sheetlin, and John L. Spouge: "Threshold Average Precision (TAP-k): A Measure of Retrieval Efficacy Designed for Bioinformatics" (2010). Bioinformatics 26(14):1708-1713. DOI: 10.1093/bioinformatics/btq270

This Python module is based on the authors' reference implementation written in Perl, which can be found at http://www.ncbi.nlm.nih.gov/CBBresearch/Spouge/html.ncbi/tap/.

Installation

Use pip for a system-wide installation:

pip3 install TAP-k

Or simply download the stand-alone script tapk.py.

Requirements

TAP-k runs on Python 3 (Python 3.4 or higher recommended).
No other requirements.

Usage

Installing via pip creates an executable script TAP-k to run from the command line:

TAP-k [options]

If you just downloaded the stand-alone module, use the following call:

python3 tapk.py [options]

Run TAP-k --help to see a short description of all available options.

Example call with minimal output:

$ TAP-k -i test/short.tsv -k 5 -s
EPQ (threshold at 0.5 quantile)	unweighted mean TAP
5 (0.6545522081334377)	0.5664

The input file format is the same as for the original program, which is described here. All output is written to STDOUT. The output format can be changed by specifying format strings (options -f and -Q).

The module can also be used as a library:

>>> import tapk
>>> retlists = ['test/retlists/{}.tsv'.format(fn)
                for fn in ('weighted', 'single')]
>>> result = tapk.tapk(retlists, k=5)
>>> result.tap
0.1311308349769888
>>> result.e0
0.7418847867396157
>>> result.queries
[QueryResult(query='23817572', tap=0.22749287749287747, weight=3.0, T_q=8),
 QueryResult(query='12954810', tap=0.0, weight=1.0, T_q=1),
 QueryResult(query='20729916', tap=0.08055555555555555, weight=2.0, T_q=1),
 QueryResult(query='21519793', tap=0.0, weight=5.0, T_q=1),
 QueryResult(query='7787496', tap=0.5833333333333334, weight=1.0, T_q=1),
 QueryResult(query='1303262', tap=0.27777777777777773, weight=1.0, T_q=2)]

Project details


Release history Release notifications | RSS feed

This version

1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

TAP-k-1.0.tar.gz (9.6 kB view hashes)

Uploaded source

Built Distribution

TAP_k-1.0-py3-none-any.whl (10.2 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page