Tokenizing Nepali Text Sentences
Installation Guide
pip install nepali_tokenizer
Usage
from nepali_tokenizer import NepaliTokenizer
tokenize = NepaliTokenizer()
print(tokenize.tokenizer('के छ खबर तिम्रो ? '))
# output--> ['खबर','तिम्रो']
We can use it with pandas dataframe too
#for example
df.head()
#---> List of nepali corpora with 100 rows
tokenize=NepaliTokenizer()
df = df[0].apply(tokenize.tokenizer)
#output--> All the text in tokenized forms
Feel Free to Contribute.
1 maintainer
Release files for nepalitokenizer 1.8.6.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| nepalitokenizer-1.8.6.0-py3-none-any.whl | Python 3 | none | any | Details |
Release files / nepalitokenizer-1.8.6.0-py3-none-any.whl
| Download URL | nepalitokenizer-1.8.6.0-py3-none-any.whl |
|---|---|
| Size | 8.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bb58f5f00de1d5a0cf593491b7ef8256242618a502be9999fb4ed69c5345fc51
|
|
BLAKE2b-256 checksum How to use checksums |
bb7ae6041180e8f045e53ee6b38e93b33a79a223464d9733da4a13afa3c8805f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.50.0 CPython/3.7.5
|
