Sequence Tagger for Partially Annotated Dataset in spaCy
Project description
spacy-partial-tagger
This is a library to build a CRF tagger for a partially annotated dataset in spaCy. You can build your own NER tagger only from dictionary. The algorithm of this tagger is based on Effland and Collins. (2021).
Overview
Dataset Preparation
Prepare spaCy binary format file. This library expects tokenization is character-based. For more detail about spaCy binary format, see this page.
You can prepare your own dataset with spaCy's entity ruler as follows:
import spacy
from spacy.tokens import DocBin
from spacy_partial_tagger.tokenizer import CharacterTokenizer
nlp = spacy.blank("en")
nlp.tokenizer = CharacterTokenizer(nlp.vocab) # Use a character-based tokenizer
patterns = [{"label": "LOC", "pattern": "Tokyo"}, {"label": "LOC", "pattern": "Japan"}]
ruler = nlp.add_pipe("entity_ruler")
ruler.add_patterns(patterns)
doc = nlp("Tokyo is the capital of Japan.")
doc_bin = DocBin()
doc_bin.add(doc)
# Replace /path/to/data.spacy with your own path
doc_bin.to_disk("/path/to/data.spacy")
In the example above, entities are extracted by character-based matching. However, in some cases, character-based matching may not be suitable (e.g., the element symbol na
for sodium matches name). In such cases, token-based matching can be used as follows:
import spacy
from spacy.tokens import DocBin
from spacy_partial_tagger.tokenizer import CharacterTokenizer
text = "Selegiline - induced postural hypotension in Parkinson's disease: a longitudinal study on the effects of drug withdrawal."
patterns = [
{"label": "Chemical", "pattern": [{"LOWER": "selegiline"}]},
{"label": "Disease", "pattern": [{"LOWER": "hypotension"}]},
{
"label": "Disease",
"pattern": [{"LOWER": "parkinson"}, {"LOWER": "'s"}, {"LOWER": "disease"}],
},
]
# Add an entity ruler to the pipeline.
nlp = spacy.blank("en")
ruler = nlp.add_pipe("entity_ruler")
ruler.add_patterns(patterns)
# Extract entities from the text.
doc = nlp(text)
entities = [(ent.start_char, ent.end_char, ent.label_) for ent in doc.ents]
# Create a DocBin object.
nlp = spacy.blank("en")
nlp.tokenizer = CharacterTokenizer(nlp.vocab)
doc_bin = DocBin()
doc = nlp.make_doc(text)
doc.ents = [
doc.char_span(start, end, label=label) for start, end, label in entities
]
doc_bin.add(doc)
doc_bin.to_disk("/path/to/data.spacy")
Training
Train your model as follows:
python -m spacy train config.cfg --output outputs --paths.train /path/to/train.spacy --paths.dev /path/to/dev.spacy --gpu-id 0
You could download config.cfg
here.
Or you could setup your own. This library would train models through spaCy. If you are not familiar with spaCy's config file format, please check the documentation.
Don't forget to replace /path/to/train.spacy
and /path/to/dev.spacy
with your own.
Evaluation
Evaluate your model as follows:
python -m spacy evaluate outputs/model-best /path/to/test.spacy --gpu-id 0
Don't forget to replace /path/to/test.spacy
with your own.
Installation
pip install spacy-partial-tagger
If you use M1 Mac, you might have problems installing fugashi
. In that case, please try brew install mecab
before the installation.
References
- Thomas Effland and Michael Collins. 2021. Partially Supervised Named Entity Recognition via the Expected Entity Ratio Loss. Transactions of the Association for Computational Linguistics, 9:1320–1335.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for spacy_partial_tagger-0.10.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | f581fdccceb213de1b55618647ea3d7bc5d93b35bd8538d048ea8e54ef138fe4 |
|
MD5 | a30562765927153dbf58564350fa99f1 |
|
BLAKE2b-256 | 375ca9e0882bb4bd5504c488ddb56ee52868d6d407e128298b1c7e04c6fdc419 |
Hashes for spacy_partial_tagger-0.10.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ddcd351a75222c111b8732ae23ffedf14558e62e6a00429a1565d44ff9476f29 |
|
MD5 | 902735261b3011d839b5dea19bd76c70 |
|
BLAKE2b-256 | a703e3f12e5c4f8b292c969cf6b6a46c269b1845673b8bb4759dc928e16a00a0 |