Skip to main content
Join the official Python Developers Survey 2018 and win valuable prizes: Start the survey!

Command-line evaluation tools for named entity linking and (cross-document) coreference resolution

Project description

Entity linking evaluation

Python command-line evaluation scripts for TAC entity linking and related wikification, named entity disambiguation, and within- and cross-document coreference tasks.

Latest version on PyPi licence Python versions supported

Issue tracker Travis CI build status Documentation Status Test coverage

It aims for fast and flexible coreference resolution and sophisticated named entity recognition evaluation, such as partial scores for partial overlap between gold and system mentions. CEAF, in particular, is much faster to calculate here than in the CoNLL-11/12 scorer. It boasts features such as configurable metrics; accounting for or ignoring cross-document coreference (see the evaluate --by-doc flag); plotting to compare evaluation by system, measure and corpus subset; and bootstrap-based confidence interval calculation for document-wise evaluation metrics.

Requires that python (2.7, with Py3k support experimental/partial) be installed on your system with numpy (and preferably scipy for fast CEAF calculation) and joblib. matplotlib is required for the plot-systems command.

See a list of commands with:

./nel --help

Or install onto your Python path (e.g. with pip install git+https://github.com/wikilinks/neleval) then

python -m neleval --help

TAC-KBP 2014 EDL quickstart

./scripts/run_tac14_evaluation.sh \
    /path/to/gold.xml \              # TAC14 gold standard queries/mentions
    /path/to/gold.tab \              # TAC14 gold standard link and nil annotations
    /system/output/directory \       # directory containing (only) TAC14 system output files
    /script/output/directory \       # directory to which results are written
    number_of_jobs                   # number of jobs for parallel mode

Each file in in the system output directory is scored against gold.tab.

Similar facility is available for TAC-KBP‘15 EDL.

More details

See the documentation for more details.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
neleval-3.1.0.tar.gz (49.5 kB) Copy SHA256 hash SHA256 Source None May 30, 2018

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page