ner-eval-dashboard

Dashboard for Quality-driven NER.

concept

The idea of this project is to provide a more elaborated evaluation for NER models. That way, it should be easier to fix labeling mistakes, better understand the positive and negative aspects of the trained NER model and see how it applies on unlabeled data.

progress

version 0.1.0 provides standard F1 scores for Exact Match, Type Match, and Position Match. So far only Flair models are implemented. See Issues to view planned features

installation

The ner eval dashboard can be installed via:

pip install ner-eval-dashboard==0.1

usage

The ner eval dashboard can be used on various ways: cli, api or via docker. It is important to always specify a model, a dataset and a tokenizer.

Note: To run the examples, you need to once manually download the CONLL03 dataset and put it into the {FLAIR_CACHE_ROOT}/.flair/datasets/conll_03 folder.

cli

The ner eval dashboard can be use via the command line interface:

ner_eval_dashboard [--dataset_path DATASET_PATH] [--extra_unlabeled_data EXTRA_UNLABELED_DATA] [--use USE [USE ...]] [--exclude EXCLUDE [EXCLUDE ...]] {FLAIR} predictor_name_or_path {SPACE} {RAW,COLUMN_DATASET,JSONL_DATASET,CONLL03,CONLL03_GERMAN,CONLL03_DUTCH,CONLL03_SPANISH,WNUT17,ARABIC_ANER,ARABIC_AQMAR,BASQUE,WEIBO,DANE,MOVIE_SIMPLE,MOVIE_COMPLEX,SEC_FILLINGS,RESTAURANT,STACKOVERFLOW,TWITTER
,PERSON,WEBPAGES,WNUT2020,WIKIGOLD,FINER,BIOFID,EUROPARL,LEGAL_NER,GERMEVAL,POLITICS,BUSINESS,ICELANDIC_NER,HIRONSAN,MASAKHANE,MULTI_CONER,WIKIANN,XTREME,WIKINER,SWEDISH_NER,TURKU}

For example the following can be used to evaluate the Bi-LSTM-CRF model based on Flair embeddings on CONLL03:

ner_eval_dashboard FLAIR flair/ner-english SPACE CONLL03

api

from ner_eval_dashboard.dataset.flair import FlairConll03
from ner_eval_dashboard.predictor import FlairPredictor
from ner_eval_dashboard.tokenizer import SpaceTokenizer
from ner_eval_dashboard.app import create_app

tokenizer = SpaceTokenizer()
dataset = FlairConll03(tokenizer)
predictor = FlairPredictor("flair/ner-english")

app = create_app("my-app", predictor, dataset)

app.run_server()

docker

docker images are publicly available at docker hub

docker run -it --rm -p 8050:8050 helpmefindaname/ner-eval-dashboard FLAIR flair/ner-english SPACE CONLL03