Skip to main content

spaCy Data Debug has utilities to help you debug your custom NER data. It checks for inconsistencies in labels for the same text.

Project description

spaCy Data Debug

spaCy Data Debug has utilities to help you debug your custom NER data. It checks for inconsistencies in labels for the same text,

Install

pip install spacy-data-debug

How to use

from pathlib import Path
import srsly
from spacy_data_debug.core import *
from spacy_data_debug.pipeline import *

0. Load your Data in the Prodigy Annotation Format

train = list(srsly.read_jsonl(base_dir / "train.jsonl"))
dev = list(srsly.read_jsonl(base_dir / "dev.jsonl"))
test = list(srsly.read_jsonl(base_dir / "test.jsonl"))

Clean, format and filter overlapping entities

While working on a large annotation projects the format of your data can get weird from different annotation sessions by different people. This ensures you have data in a format useful for the other functions in this spacy-data-debug

train = fix_annotations_format(train)
dev = fix_annotations_format(dev)
test = fix_annotations_format(test)

Or construct a Pipeline

A Pipeline holds your datasets together and runs spacy_data_debug functions across all datasets. This can make sure you have consistent annotations across your datasets split

pipeline = Pipeline(train, dev, test)
pipeline.apply(fix_annotations_format)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for spacy-data-debug, version 0.0.3
Filename, size File type Python version Upload date Hashes
Filename, size spacy_data_debug-0.0.3.tar.gz (5.4 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page