TakeSentenceTokenizer

TakeSentenceTokenizer is a tool for pre processing and tokenizing sentences. The package is used to: - convert the first word of the sentence to lowercase - convert from uppercase to lowercase - convert word to lowercase after punctuation - replace words for placeholders: laugh, date, time, ddd, measures (10kg, 20m, 5gb, etc), code, phone number, cnpj, cpf, email, money, url, number (ordinal and cardinal) - replace abbreviations - replace common typos - split punctuations - remove emoji - remove characters that are not letters or punctuation - add missing accentuation - tokenize the sentence

Installation

Use the package manager pip to install TakeSentenceTokenizer

pip install TakeSentenceTokenizer

Usage

Example 1: full processing not keeping registry of removed punctuation

Code:

from SentenceTokenizer import SentenceTokenizer
sentence = 'P/ saber disso eh c/ vc ou consigo ver pelo site www.dúvidas.com.br/minha-dúvida ??'
tokenizer = SentenceTokenizer()
processed_sentence = tokenizer.process_message(sentence)
print(processed_sentence)

Output:

'para saber disso é com você ou consigo ver pelo site URL ? ?'

Example 2: full processing keeping registry of removed punctuation

from SentenceTokenizer import SentenceTokenizer
sentence = 'como assim $@???'
tokenizer = SentenceTokenizer(keep_registry_punctuation = True)
processed_sentence = tokenizer.process_message(sentence)
print(processed_sentence)
print(tokenizer.removal_registry_lst)

Output:

como assim ? ? ?
[['como assim $@ ? ? ?', {'punctuation': '$', 'position': 11}, {'punctuation': '@', 'position': 12}, {'punctuation': ' ', 'position': 13}]]

Author

Take Data&Analytics Research

License

MIT