Skip to main content

spaCy Data Debug has utilities to help you debug your custom NER data. It checks for inconsistencies in labels for the same text.

Project description

spaCy Data Debug

spaCy Data Debug has utilities to help you debug your custom NER data. It checks for inconsistencies in labels for the same text,

Install

pip install spacy-data-debug

How to use

from pathlib import Path
import srsly
from spacy_data_debug.core import *
from spacy_data_debug.pipeline import *

0. Load your Data in the Prodigy Annotation Format

train = list(srsly.read_jsonl(base_dir / "train.jsonl"))
dev = list(srsly.read_jsonl(base_dir / "dev.jsonl"))
test = list(srsly.read_jsonl(base_dir / "test.jsonl"))

Clean, format and filter overlapping entities

While working on a large annotation projects the format of your data can get weird from different annotation sessions by different people. This ensures you have data in a format useful for the other functions in this spacy-data-debug

train = fix_annotations_format(train)
dev = fix_annotations_format(dev)
test = fix_annotations_format(test)

Or construct a Pipeline

A Pipeline holds your datasets together and runs spacy_data_debug functions across all datasets. This can make sure you have consistent annotations across your datasets split

pipeline = Pipeline(train, dev, test)
pipeline.apply(fix_annotations_format)

Project details


Release history Release notifications

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for spacy-data-debug, version 0.0.3
Filename, size File type Python version Upload date Hashes
Filename, size spacy_data_debug-0.0.3.tar.gz (5.4 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page