Skip to main content

Sequence Tagger for Partially Annotated Dataset in spaCy

Project description

spacy-partial-tagger

This is a library to build a CRF tagger for a partially annotated dataset in spaCy. You can build your own NER tagger only from dictionary. The algorithm of this tagger is based on Effland and Collins. (2021).

Overview

Dataset Preparation

Prepare spaCy binary format file. This library expects tokenization is character-based. For more detail about spaCy binary format, see this page.

You can prepare your own dataset with spaCy's entity ruler as follows:

import spacy
from spacy.tokens import DocBin
from spacy_partial_tagger.tokenizer import CharacterTokenizer


nlp = spacy.blank("en")
nlp.tokenizer = CharacterTokenizer(nlp.vocab)  # Use a character-based tokenizer

patterns = [{"label": "LOC", "pattern": "Tokyo"}, {"label": "LOC", "pattern": "Japan"}]
ruler = nlp.add_pipe("entity_ruler")
ruler.add_patterns(patterns)

doc = nlp("Tokyo is the capital of Japan.")

doc_bin = DocBin()
doc_bin.add(doc)

# Replace /path/to/data.spacy with your own path
doc_bin.to_disk("/path/to/data.spacy")

Training

Train your model as follows:

python -m spacy train config.cfg --output outputs --paths.train /path/to/train.spacy --paths.dev /path/to/dev.spacy --gpu-id 0

You could download config.cfg here. Or you could setup your own. This library would train models through spaCy. If you are not familiar with spaCy's config file format, please check the documentation.

Don't forget to replace /path/to/train.spacy and /path/to/dev.spacy with your own.

Evaluation

Evaluate your model as follows:

python -m spacy evaluate outputs/model-best /path/to/test.spacy --gpu-id 0

Don't forget to replace /path/to/test.spacy with your own.

Installation

pip install spacy-partial-tagger

If you use M1 Mac, you might have problems installing fugashi. In that case, please try brew install mecab before the installation.

References

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

spacy-partial-tagger-0.9.2.tar.gz (13.2 kB view hashes)

Uploaded Source

Built Distribution

spacy_partial_tagger-0.9.2-py3-none-any.whl (17.0 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page