NER Data Processor

NER Data Processor is a Python library to help you easily prepare datasets for Named Entity Recognition (NER) and Coreference Resolution tasks. It transforms raw text into formats ready for training token classification models using Hugging Face or other frameworks.

📚 Documentation

• GitHub Repository
• PyPI Package

📦 Installation

✅ From PyPI (Recommended)

pip install ner-data-processor

🛠️ From GitHub

git clone https://github.com/rajboopathiking/NER_DATA_PREPROCESSING.git
cd NER_DATA_PREPROCESSING
pip install -r requirements.txt

🚀 Getting Started

from ner_data_processor.Ner_Data_Preparation import Custom_Ner_Dataset

ner = Custom_Ner_Dataset()

📊 Dataset Format

Input should be a pandas DataFrame with two columns:

• text: Sentence or paragraph
• entities: List of labeled entities with their tags

Example:

⚙️ API Overview

extract_DataFrame(df)

Convert the annotated DataFrame into span-based entity format.

data = ner.extract_DataFrame(df)

Output:

to_dataset(data)

Convert span-format data into token-label format for model training.

import pandas as pd
df = pd.DataFrame(ner.to_dataset(data))

Output:

create _label_maps

labels = []
for i in df["ner_tags"]:
    labels.extend(i)
labels = np.unique(labels).tolist()

Output:

['B-DATE', 'B-Defender', 'B-LOC', 'B-ORG', 'B-PERSON', 'B-PLAINTIFF',
 'I-DATE', 'I-Defender', 'I-LOC', 'I-ORG', 'I-PERSON', 'I-PLAINTIFF', 'O']

to_huggingface_dataset(df, labels)

Convert your processed DataFrame into Hugging Face DatasetDict.

dataset = ner.to_huggingface_dataset(df, labels)
dataset = dataset.train_test_split(test_size=0.1)

Output:

DatasetDict({
    train: Dataset({
        features: ['id', 'tokens', 'ner_tags'],
        num_rows: 3
    }),
    test: Dataset({
        features: ['id', 'tokens', 'ner_tags'],
        num_rows: 1
    })
})

coreference_model(text)

Basic coreference resolution model.

text = "John is Victim. He is Innocent"
result = ner.coreference_model(text)

Output:

{
  "mentions": [
    {
      "text": "He",
      "refers_to": "John",
      "span": [13, 15]
    }
  ]
}

🪪 License

This project is licensed under the MIT License.