spacy-data-debug 0.0.3

spaCy Data Debug has utilities to help you debug your custom NER data. It checks for inconsistencies in labels for the same text.

spaCy Data Debug

spaCy Data Debug has utilities to help you debug your custom NER data. It checks for inconsistencies in labels for the same text,

Install

pip install spacy-data-debug

How to use

from pathlib import Path
import srsly
from spacy_data_debug.core import *
from spacy_data_debug.pipeline import *

0. Load your Data in the Prodigy Annotation Format

train = list(srsly.read_jsonl(base_dir / "train.jsonl"))
dev = list(srsly.read_jsonl(base_dir / "dev.jsonl"))
test = list(srsly.read_jsonl(base_dir / "test.jsonl"))

Clean, format and filter overlapping entities

While working on a large annotation projects the format of your data can get weird from different annotation sessions by different people. This ensures you have data in a format useful for the other functions in this spacy-data-debug

train = fix_annotations_format(train)
dev = fix_annotations_format(dev)
test = fix_annotations_format(test)

Or construct a Pipeline

A Pipeline holds your datasets together and runs spacy_data_debug functions across all datasets. This can make sure you have consistent annotations across your datasets split

pipeline = Pipeline(train, dev, test)
pipeline.apply(fix_annotations_format)