bntransformer 1.0

Bengali Transformer for natural language processing using state of the art transformer(language model)

Bengali Transformer

Bengali Transformer for natural language processing using state of the art transformer(language model)

Thanks to huggingface transformers

Installation

pip install bntransformer

Tokenizer

Bert Multilingual Tokenizer

from bntransformer.bnbert import Tokenizer

tokenizer = Tokenizer()
tokens = tokenizer.tokenize('আমি ভাত খাই।')
print(tokens)
# output: ['আ', '##মি', 'ভ', '##াত', 'খা', '##ই', '।']