Does language identification for Indian languages
Project description
whichlang
whichlang is a Python library for identifying the language of the given text
Installation
Use the package manager pip to install whichlang.
pip install whichlang
Usage
from whichlang import whichlang as wl
f = open('sample-test-files\\sample-hindi.txt','r')
data = f.read()
# returns tuple of top 3 probable languages, first one being most probable language
print (wl.which_lang(data))
>>> ('Hindi', 'Marathi', 'Punjabi') #Hindi is most probable.
# For training a language model
# assamese.txt is train data
# Assamese is the language model created
python train_lang_models.py -f train-data\as\assamese.txt -l Assamese
Contributing
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
License
Available Languages
Hindi, Telugu, Tamil, Kannada, Malayalam, Punjabi, Marathi, Gujarati, Oriya, Assamese.
Acknowledgements
-
We would like to thank the Leipzig Corpora collection where we collected data for training models.
Dirk Goldhahn, Thomas Eckart and Uwe Quasthoff (2012): Building Large Monolingual Dictionaries at the Leipzig Corpora Collection: From 100 to 200 Languages. In: Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), 2012
-
whichlang is based on N-gram based Text categorization: Cavnar, William B., and John M. Trenkle. "N-gram-based text categorization." Proceedings of SDAIR-94, 3rd annual symposium on document analysis and information retrieval. Vol. 161175. 1994.
The same approach was used in library langdetect. We found this approach quite effective and wanted to explore for Indian languages. In whichlang, we train, optimize and make models readily available for Indian languages since these languages have been less explored.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for whichlang-0.0.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5a16720abec7a2141bf58b3805aa7e71a4544ebb09cc4cb817962e9d1990d311 |
|
MD5 | ab92d711672665ffa182a15dab1239b2 |
|
BLAKE2b-256 | 56a9f925c75478619946160ca9dd455a79415e229ff8d0dcd8a2b20dd8e5d292 |