Skip to main content

Sequence Tagger for Partially Annotated Dataset in spaCy

Project description

spacy-partial-tagger

This is a library to build a CRF tagger for a partially annotated dataset in spaCy. You can build your own NER tagger only from dictionary. The algorithm of this tagger is based on Effland and Collins. (2021).

Overview

The overview of spacy-partial-tagger

Dataset Preparation

Prepare spaCy binary format file to train your tagger. If you are not familiar with spaCy binary format, see this page.

You can prepare your own dataset with spaCy's entity ruler as follows:

import spacy
from spacy.tokens import DocBin


nlp = spacy.blank("en")

patterns = [{"label": "LOC", "pattern": "Tokyo"}, {"label": "LOC", "pattern": "Japan"}]
ruler = nlp.add_pipe("entity_ruler")
ruler.add_patterns(patterns)

doc = nlp("Tokyo is the capital of Japan.")

doc_bin = DocBin()
doc_bin.add(doc)

# Replace /path/to/data.spacy with your own path
doc_bin.to_disk("/path/to/data.spacy")

Training

Train your tagger as follows:

python -m spacy train config.cfg --output outputs --paths.train /path/to/train.spacy --paths.dev /path/to/dev.spacy --gpu-id 0

This library is implemented as a trainable component in spaCy, so you could control the training setting via spaCy's configuration system. We provide you the default configuration file here. Or you could setup your own. If you are not familiar with spaCy's config file format, please check the documentation.

Don't forget to replace /path/to/train.spacy and /path/to/dev.spacy with your own.

Evaluation

Evaluate your tagger as follows:

python -m spacy evaluate outputs/model-best /path/to/test.spacy --gpu-id 0

Don't forget to replace /path/to/test.spacy with your own.

Installation

pip install spacy-partial-tagger

If you use M1 Mac, you might have problems installing fugashi. In that case, please try brew install mecab before the installation.

References

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

spacy_partial_tagger-0.15.6.tar.gz (267.4 kB view details)

Uploaded Source

Built Distribution

spacy_partial_tagger-0.15.6-py3-none-any.whl (11.0 kB view details)

Uploaded Python 3

File details

Details for the file spacy_partial_tagger-0.15.6.tar.gz.

File metadata

  • Download URL: spacy_partial_tagger-0.15.6.tar.gz
  • Upload date:
  • Size: 267.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.6

File hashes

Hashes for spacy_partial_tagger-0.15.6.tar.gz
Algorithm Hash digest
SHA256 ed96ea292b7ff8d06049a4de899e531b736dd326b3f5eacf48d4393034caebd7
MD5 9f192752ca8c1dbd355ea1364dbefc23
BLAKE2b-256 e0c1d45983c1d8b69274582d305b348f24032acf8bb1d342c78f980cca6e25eb

See more details on using hashes here.

File details

Details for the file spacy_partial_tagger-0.15.6-py3-none-any.whl.

File metadata

File hashes

Hashes for spacy_partial_tagger-0.15.6-py3-none-any.whl
Algorithm Hash digest
SHA256 3717b4e6099736737876fa857098b3269391eaa6554249a1800a5a3dd954802c
MD5 c95df0c78de51d3ff9477021f16d1b62
BLAKE2b-256 86d34a5e7fa8d86172f8d542e0cf5791632d9c929e0f17ec841631ed861e3a07

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page