Skip to main content

brat utilities

Project description

bratutils

CircleCI License: MIT

A collection of utilities for manipulating data and calculating inter-annotator agreement in brat annotation files.

Installation

Install as a normal package from the source directory.

$ pip install bratutils

Agreement Definition

Agreement in multi-token annotations is commonly evaluated using f-score. due to various problems with computing the traditional Krippendorf's alpha and Cohen's kappa. Hripcsak prove the validity of the metric for very large populations, i.e. for unrestricted text annotations.

This library roughly follows the definitions of precision and recall calculation from the MUC-7 test scoring. The basic definitions along with some additional restrictions are laid out below:

  • CORRECT - when annotation tags and indices match completely
  • INCORRECT - when annotation tags do not match, but the indices coincide
  • PARTIAL - when the annotation tags are the same but one of the annotations has the same end index and a different start index
  • MISSING - annotations exising only in the gold standard annotation set
  • SPURIOUS - annotations existing only in the candidate annotation set

Note: the gold standard is considered the collections/document from which the comparison is invoked, while the supplied parallel annotation is considered the candidate set.

Disclaimer: the current definition of the PARTIAL category accomodates working with syntactic chunks. A different arrangement (e.g. pick largest contained tag as partial match instead of rightmost) might be more suitable for other tasks, for example some types of semantic annotation.

Examples

Simple example:

from bratutils import agreement as a

doc = a.Document('res/samples/A/data-sample-1.ann')
doc2 = a.Document('res/samples/B/data-sample-1.ann')

doc.make_gold()
statistics = doc2.compare_to_gold(doc)

print(statistics)

Output:

-------------------MUC-Table--------------------
------------------------------------------------
pos:135
act:134
cor:115
par:5
inc:4
mis:11
spu:10
------------------------------------------------
pre:0.858208955224
rec:0.851851851852
fsc:0.855018587361
------------------------------------------------
und:0.0814814814815
ovg:0.0746268656716
sub:0.0725806451613
------------------------------------------------
bor:119
ibo:15
------------------------------------------------
------------------------------------------------

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for bratutils, version 0.2.2
Filename, size File type Python version Upload date Hashes
Filename, size bratutils-0.2.2-py3-none-any.whl (17.4 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size bratutils-0.2.2.tar.gz (13.6 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page