naganlp 0.1.0

A Natural Language Processing toolkit for the Nagamese language.

NagaNLP: Natural Language Processing for Nagamese

A comprehensive Natural Language Processing toolkit for the Nagamese language, developed by Agniva Maiti (4th Year BTech, KIIT-DU, Bhubaneswar, Odisha).

A comprehensive NLP toolkit for the Nagamese language, featuring state-of-the-art models for part-of-speech tagging and machine translation.

Features

• Part-of-Speech Tagging: Fine-tuned BERT model for accurate POS tagging
• Neural Machine Translation: Seq2Seq model for Nagamese to English translation
• Subword Tokenization: Support for handling out-of-vocabulary words
• Word Alignment: Tools for parallel corpus alignment
• Easy Integration: Simple Python API for all functionalities

Installation

pip install naganlp

Quick Start

Part-of-Speech Tagging

Transformer-based Tagger (Recommended for production)

This uses a fine-tuned transformer model for high accuracy.

from naganlp import PosTagger

# Initialize the tagger (automatically downloads the model on first use)
tagger = PosTagger("agnivamaiti/naganlp-pos-tagger")  # Default model

# Tag a Nagamese sentence
result = tagger.tag("moi school te jai")
print(result)
# Output: [{'entity_group': 'PRON', 'word': 'moi', ...}]

NLTK-based Tagger (Lightweight, faster but less accurate)

This is a good option for development or when resources are limited.

from naganlp import NltkPosTagger

# First train and save the model (only needed once)
from naganlp.nltk_tagger import train_and_save_nltk_tagger
train_and_save_nltk_tagger("path/to/your/conll/file.conll", "naga_pos_model.pkl")

# Then load and use the trained model
tagger = NltkPosTagger("naga_pos_model.pkl")

# Tag a list of pre-tokenized words
result = tagger.predict(["moi", "school", "te", "jai"])
print(result)
# Output: [('moi', 'PRON'), ('school', 'NOUN'), ('te', 'ADP'), ('jai', 'VERB')]

Translation

from naganlp import Translator

# Initialize the translator
translator = Translator()

# Translate from Nagamese to English
translation = translator.translate("moi school te jai")
print(translation)
# Output: "I go to school"

Documentation

Data Requirements

• For POS Tagging: CONLL-formatted file with token and POS tag columns
• For Translation: Parallel corpus in CSV format with 'nagamese' and 'english' columns

Model Training

POS Tagger Training

python main.py train-tagger --conll-file path/to/train.conll --hub-id your-username/naganlp-pos-tagger

NMT Model Training

python main.py train-translator --data-file path/to/parallel_corpus.csv --hub-id your-username/naganlp-nmt

Advanced Usage

Custom Model Paths

# Load custom models
custom_tagger = PosTagger(model_name_or_path="path/to/custom/model")
custom_translator = Translator(model_path="path/to/translator.pt", vocabs_path="path/to/vocabs.pkl")

Contributing

Contributions are welcome! Please read our Contributing Guidelines for details.

Contributing

Contributions are welcome! Please read our Contributing Guidelines and Code of Conduct for details.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contact

• Agniva Maiti
• Email: [e-mail]
• GitHub: @AgnivaMaiti
• LinkedIn: Agniva Maiti

Acknowledgments

• KIIT University for the support and resources
• All contributors and users of this library

Citation

If you use NagaNLP in your research, please cite:

@software{naganlp2023,
  title={NagaNLP: Natural Language Processing Toolkit for Nagamese},
  author={Your Name},
  year={2023},
  publisher={GitHub},
  journal={GitHub repository},
  howpublished={\url{https://github.com/your-username/naga-nlp}}
}

Support

For questions and support, please open an issue on our GitHub repository.