Skip to main content

The package annotlib is a library of techniques to simulate the labeling behaviour of real annotators.

Project description

annotlib: Simulation of Annotators

authors: Marek Herde and Adrian Calma

Introduction

annotlib is a Python package for simulating annotators in an active learning setting. Solving classification problems by using supervised machine learning models requires samples assigned to class labels. However, labeling these samples causes costs (e.g. workload, time, etc.), so that active learning strategies aim at reducing these costs by selecting samples, which are the most useful for training a classifier.

In real-world scenarios, human annotators are often responsible for providing the class labels. Unfortunately, there is no guaranty for the omniscience of such annotators. Hence, annotators are prone to error, respectively uncertain, so that the class labels assigned to samples may be false. The labeling performance of an annotator is affected by many factors, e.g. expertise, experience, concentration, level of fatigue and so on. Moreover, the difficulty of a sample influences the outcome of a labeling process.

To evaluate an active learning strategy in the setting of uncertain annotators, class labels of these uncertain annotators are required to be available, but there is a lack of publicly accessible real-world data sets labeled by error-prone annotators. As a result, recently published active learning strategies are evaluated on simulated annotators where the used simulation techniques are diverse. Our developed annotlib Python package represents a of these techniques and implements additional methods, which simulate realistic characteristics of uncertain annotators. This way, we establish a library simplifying and standardising the evaluation of active learning strategies coping with uncertain annotators.

For more information go to the documentation.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

annotlib-1.0.0.tar.gz (21.7 kB view details)

Uploaded Source

Built Distribution

annotlib-1.0.0-py3-none-any.whl (32.5 kB view details)

Uploaded Python 3

File details

Details for the file annotlib-1.0.0.tar.gz.

File metadata

  • Download URL: annotlib-1.0.0.tar.gz
  • Upload date:
  • Size: 21.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.21.0 setuptools/45.0.0 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.7.3

File hashes

Hashes for annotlib-1.0.0.tar.gz
Algorithm Hash digest
SHA256 979dcf06cb57573d872ed007a910fc7070c3979fcbc283f184f8e76d0b3a4dc2
MD5 595e51847346b28ab8873e607990523c
BLAKE2b-256 f195166f3229f8cc2246e22974305a1d695c0fb7798b2a065425c13566f69f61

See more details on using hashes here.

File details

Details for the file annotlib-1.0.0-py3-none-any.whl.

File metadata

  • Download URL: annotlib-1.0.0-py3-none-any.whl
  • Upload date:
  • Size: 32.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.21.0 setuptools/45.0.0 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.7.3

File hashes

Hashes for annotlib-1.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 80f1822a0460da938c923e68862f206c25ece471aff3da98b82bef48d42a1a01
MD5 731a6ad1a8e29cc62fb4cb1ebaee26f5
BLAKE2b-256 5fdbee99ed37236a13344e19bea038c575fcd9e6b33d8c531441c36c38fd9172

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page