biotagging 0.1.0

A robust Python package for BIO entity tagging with automatic sequence repair and validation

Biotagging

A robust Python package for BIO entity tagging with automatic sequence repair and validation.

Overview

Biotagging provides a comprehensive solution for processing and validating BIO (Beginning-Inside-Outside) entity tags commonly used in Named Entity Recognition (NER) tasks. The package offers automatic repair of malformed tag sequences, support for multiple input formats, and a professional command-line interface.

Features

• Automatic BIO Sequence Repair: Intelligently fixes malformed tag sequences (I-tags without B-tags, mismatched classes, empty tags)
• Multiple Input Formats: Support for strings, token lists, JSON files, and pandas DataFrames
• Robust Validation: Built-in Pydantic schema validation with comprehensive error reporting
• Unicode Support: Full support for international text and mixed writing systems
• High Performance: Efficiently processes large datasets with 100k+ tokens
• Command Line Interface: Professional CLI for batch processing and file conversion
• Multiple Output Formats: JSON, CSV, and CoNLL output formats
• Comprehensive Testing: Over 117 tests covering edge cases and error conditions

Installation

pip install biotagging

Quick Start

Python API

from biotagging import tag_sentence, tag_sentences

# Process a single sentence
result = tag_sentence("John works at IBM", ["PER", "O", "O", "ORG"])
print(result)
# Output: {
#     "sentence_id": None,
#     "sentence": "John works at IBM", 
#     "tokens": ["John", "works", "at", "IBM"],
#     "tags": ["B-PER", "O", "O", "B-ORG"]
# }

# Process multiple sentences
batch = [
    ("John works at IBM", ["PER", "O", "O", "ORG"]),
    ("Apple hired Mary", ["ORG", "O", "PER"])
]
results = tag_sentences(batch)

JSON Processing

from biotagging import tag_from_json

json_data = [
    {
        "sentence": "John works at IBM",
        "tags": ["PER", "O", "O", "ORG"]
    }
]
results = tag_from_json(json_data)

DataFrame Processing

import pandas as pd
from biotagging import tag_from_dataframe

df = pd.DataFrame([
    {"sentence_id": 0, "word": "John", "tag": "PER"},
    {"sentence_id": 0, "word": "works", "tag": "O"},
    {"sentence_id": 0, "word": "at", "tag": "O"},
    {"sentence_id": 0, "word": "IBM", "tag": "ORG"}
])
results = tag_from_dataframe(df)

Command Line Interface

Process Single Sentence

biotagging sentence "John works at IBM" "PER O O ORG"

Process JSON File

biotagging json input.json --output results.json --validate

Process CSV File

biotagging csv data.csv --format conll --validate

Validate Tagged Data

biotagging validate results.json

Get Help

biotagging --help
biotagging sentence --help

BIO Tag Repair

The package automatically repairs common BIO tagging issues:

# I-tag without B-tag gets converted to B-tag
tag_sentence("John Smith", ["I-PER", "I-PER"])
# Result: ["B-PER", "I-PER"]

# Empty tags continue the previous entity
tag_sentence("New York City", ["LOC", "", ""])
# Result: ["B-LOC", "I-LOC", "I-LOC"]

# Mismatched I-tag classes get converted to B-tags
tag_sentence("John Smith", ["B-PER", "I-ORG"])
# Result: ["B-PER", "B-ORG"]

Input Formats

String Input

tag_sentence("John works", ["PER", "O"])

Token List Input

tag_sentence(["John", "works"], ["PER", "O"])

JSON Format

[
    {
        "sentence_id": 1,
        "sentence": "John works at IBM",
        "tags": ["PER", "O", "O", "ORG"]
    }
]

DataFrame Format

# Required columns: sentence_id, word, tag
pd.DataFrame([
    {"sentence_id": 0, "word": "John", "tag": "PER"},
    {"sentence_id": 0, "word": "works", "tag": "O"}
])

Output Formats

JSON Output

{
    "sentence_id": 0,
    "sentence": "John works at IBM",
    "tokens": ["John", "works", "at", "IBM"],
    "tags": ["B-PER", "O", "O", "B-ORG"]
}

CoNLL Format

# Sentence 0
John    B-PER
works   O
at      O
IBM     B-ORG

Validation

The package includes comprehensive validation using Pydantic schemas:

from biotagging.schema.jsonresponse import validate_bio_output

# Validates token/tag length matching and schema compliance
try:
    validated = validate_bio_output(result)
    print("Validation passed")
except ValidationError as e:
    print(f"Validation failed: {e}")

Error Handling

The package provides clear error messages for common issues:

• Token/tag length mismatches
• Empty or whitespace-only sentences
• Malformed BIO sequences (when repair is disabled)
• Missing required fields in input data

Configuration Options

Repair Mode

# Enable automatic repair (default)
tag_sentence("test", ["I-PER"], repair_illegal=True)
# Result: ["B-PER"]

# Strict mode - raise errors for illegal sequences
tag_sentence("test", ["I-PER"], repair_illegal=False)
# Raises: ValueError

Performance

• Handles sentences with 100,000+ tokens efficiently
• Processes large batches with minimal memory overhead
• Unicode text processing with international character support
• Thread-safe for concurrent processing

Requirements

• Python 3.9 or higher
• pandas >= 2.0.0
• pydantic >= 2.0.0

Development

# Clone the repository
git clone https://github.com/yourusername/biotagging.git
cd biotagging

# Install in development mode
pip install -e ".[dev]"

# Run tests
pytest test/ -v

# Run linting
flake8 biotagging/ test/

# Build package
python -m build

Contributing

Contributions are welcome! Please feel free to submit a Pull Request. Make sure to:

1. Add tests for new features
2. Follow the existing code style
3. Update documentation as needed
4. Ensure all tests pass

License

This project is licensed under the MIT License. See the LICENSE file for details.

Authors

• Saroop Makhija
• Aiman Koli
• Meet Patel

Support

For questions, issues, or contributions, please visit our GitHub repository or open an issue.

Changelog

Version 0.1.0

• Initial release
• Core BIO tagging functionality
• JSON and DataFrame processing
• Command line interface
• Comprehensive test suite
• Automatic sequence repair
• Multiple output formats