Skip to main content

spaCy Data Debug has utilities to help you debug your custom NER data. It checks for inconsistencies in labels for the same text.

Project description

spaCy Data Debug

spaCy Data Debug has utilities to help you debug your custom NER data. It checks for inconsistencies in labels for the same text,

Install

pip install spacy-data-debug

How to use

from pathlib import Path
import srsly
from spacy_data_debug.core import *
from spacy_data_debug.pipeline import *

0. Load your Data in the Prodigy Annotation Format

train = list(srsly.read_jsonl(base_dir / "train.jsonl"))
dev = list(srsly.read_jsonl(base_dir / "dev.jsonl"))
test = list(srsly.read_jsonl(base_dir / "test.jsonl"))

Clean, format and filter overlapping entities

While working on a large annotation projects the format of your data can get weird from different annotation sessions by different people. This ensures you have data in a format useful for the other functions in this spacy-data-debug

train = fix_annotations_format(train)
dev = fix_annotations_format(dev)
test = fix_annotations_format(test)

Or construct a Pipeline

A Pipeline holds your datasets together and runs spacy_data_debug functions across all datasets. This can make sure you have consistent annotations across your datasets split

pipeline = Pipeline(train, dev, test)
pipeline.apply(fix_annotations_format)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

spacy_data_debug-0.0.3.tar.gz (5.4 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page