Package for Bodo Tokenizer
Project description
Bodo Tokenizer
About:
Bodo is one of the scheduled languages of India and belongs to the Tibeto-Burman branch of the Sino-Tibetan language family. It uses a Devanagari script like other Indian languages such as Hindi, Konkani, Maithili, Marathi, Nepali, Sanskrit, Sindhi, and Dogri. However there are some differences, this python library tokenizes Bodo sentences into its tokens that is useful for NLP tasks.
Prerequisite:
Python 3.6+
Installation:
From source
git clone https://github.com/bodonlp/bodo-tokenizer.git
cd bodo-tokenizer
python setup.py install
or
pip install bodotokenizer
Available functions:
# Tokenize single sentence
from bodotokenizer import tokenize
tokenize(sentence)
# Tokenize sentences from a file
# Suitable for tokenize sentences in parallel corpus
from bodotokenizer import tokenize_file
tokenize_file(input_file, out_file)
Usage:
from bodotokenizer import tokenize
sentence = "नोंसिनि फरायसालियाव य'गा फोरोंगुरु दङ नामा?"
tokenized_sentence = tokenize(sentence)
# Output: Tokenized Sentence
# "नोंसिनि फरायसालियाव य'गा फोरोंगुरु दङ नामा ?"
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
bodotokenizer-0.1.1.tar.gz
(3.9 kB
view hashes)
Built Distribution
Close
Hashes for bodotokenizer-0.1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 477ec053c2fe6974f71409e89cd1a7a3b0a5777a3f266d310bbc28e1fe63019a |
|
MD5 | b43c8c2427f1f880e4d867653abc1ec3 |
|
BLAKE2b-256 | 8b5bb6b639dcb59d771f6a6292fba63309f4323003570532fc8e41c70eb77f8e |