Evaluation utilities for named entity linking and (cross-document) coreference resolution
TAC-KBP 2014 EDL quickstart
./scripts/run_tac14_evaluation.sh \ /path/to/gold.xml \ # TAC14 gold standard queries/mentions /path/to/gold.tab \ # TAC14 gold standard link and nil annotations /system/output/directory \ # directory containing (only) TAC14 system output files /script/output/directory \ # directory to which results are written number_of_jobs # number of jobs for parallel mode
Each file in in the system output directory is scored against gold.tab.
Similar facility is available for TAC-KBP‘15 EDL.
See the project wiki for more details.
This project extends the work described in:
- Ben Hachey, Joel Nothman and Will Radford (2014), “Cheap and easy entity evaluation”. In Proceedings of ACL.
It was used as the official scorer for Entity (Discovery and) Linking in 2014 and 2015:
- Heng Ji, Joel Nothman and Ben Hachey (2014), “Overview of TAC-KBP2014 Entity Discovery and Linking Tasks”, In Proceedings of the Text Analysis Conference.
- Heng Ji, Joel Nothman, Ben Hachey and Radu Florian (2015), “Overview of TAC-KBP2015 Tri-lingual Entity Discovery and Linking Tasks”, In Proceedings of the Text Analysis Conference.