tokenaligner 0.1.1

A lightweight utility to align NER labels with tokenized input for HuggingFace models.

tokenaligner

tokenaligner is a simple and lightweight Python package for aligning NER tags with tokenized subwords using Hugging Face tokenizers. It's especially useful when preparing datasets for training Named Entity Recognition (NER) models with Hugging Face Transformers.

---

✨ Features

• Aligns word-level NER tags with subword tokenization
• Handles padding and truncation options
• Supports Hugging Face Datasets for training
• Batch tokenization support for faster preprocessing

---

📦 Installation

pip install tokenaligner

---

🚀 Quick Start

1. Tokenize and Align NER Tags

from tokenaligner import TokenAligner
from transformers import AutoTokenizer

# Sample data
tokens = [["Hugging", "Face", "is", "based", "in", "New", "York", "."]]
tags = [["B-ORG", "I-ORG", "O", "O", "O", "B-LOC", "I-LOC", "O"]]

# Load a tokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

# Tokenize and align
aligner = TokenAligner()
aligned = aligner.tokenize_and_align(
    tokens_list=tokens,
    tags_list=tags,
    tokenizer=tokenizer,
    label_all_tokens=True,
    return_hf_dataset=False
)

print(aligned[0]["input_ids"])
print(aligned[0]["labels"])

---

2. Return a Hugging Face Dataset

hf_dataset = aligner.tokenize_and_align(
    tokens_list=tokens,
    tags_list=tags,
    tokenizer=tokenizer,
    return_hf_dataset=True
)

print(hf_dataset[0])

---

3. Use with a Trainer API

from transformers import AutoModelForTokenClassification, Trainer, TrainingArguments

model = AutoModelForTokenClassification.from_pretrained("bert-base-cased", num_labels=9)

trainer = Trainer(
    model=model,
    args=TrainingArguments(output_dir="./results", evaluation_strategy="epoch"),
    train_dataset=hf_dataset,
    eval_dataset=hf_dataset
)

trainer.train()

---

⚙️ Parameters

TokenAligner.tokenize_and_align(...)

Parameter	Type	Description
tokens_list	List[List[str]]	List of tokenized sentences
tags_list	List[List[str]]	Corresponding entity tags
tokenizer	PreTrainedTokenizer	Hugging Face tokenizer
batch_size	int	Batch size for tokenization (default: 1000)
padding	bool	Apply padding (default: False)
truncation	bool	Apply truncation (default: False)
label_all_tokens	bool	Label all subtokens with the same label (default: False)
return_hf_dataset	bool	Return Hugging Face dataset object (default: False)

---

📬 Contributing

Pull requests are welcome! For major changes, please open an issue first to discuss what you’d like to change.

---

📄 License

MIT

---

🙌 Acknowledgements

Built on top of Hugging Face Transformers and datasets.