simple tokenizer for tensorflow 2.x and PyTorch
Project description
meguru tokenizer
installation and initialization
pip install meguru_tokenizer
sudachipy link -t full
Abstruction of Usage
- Preprocess Using Each Tokenizer e.g. sentencepiece preprocess / sudachi preprocess
- Tokenize in your code using its Tokenizer
- basis
see. official docs - Tensorflow
see. tutorial - TODO: PyTorch
- basis
RealWorld Example
import meguru_tokenizer.whitespace_tokenizer import WhitespaceTokenizer
import pprint
sentences = [
"Hello, I don't know how to use it?",
"Tensorflow is awesome!",
"it is good framework.",
]
# define tokenizer and vocaburary
tokenizer = WhitespaceTokenizer(lower=True)
vocab = Vocab()
# build vocaburary
for sentence in sentences:
vocab.add_vocabs(tokenizer.tokenize(sentence))
vocab.build_vocab()
# set vocaburary into tokenizer to enable encoding
tokenizer.vocab = vocab
# save vocaburary information
vocab.dump_vocab(Path("vocab.txt"))
print("vocabs:")
pprint.pprint(vocab.i2w)
# tokenize
print("tokenized sentence")
pprint.pprint(tokenizer.tokenize_list(sentences))
# [['hello', ',', 'i', 'do', "n't", 'know', 'how', 'to', 'use', 'it', '?'],
# ['tensorflow', 'is', 'awesome', '!'],
# ['it', 'is', 'good', 'framework', '.']]
# encode
print("encoded sentence")
pprint.pprint([tokenizer.encode(sentence) for sentence in sentences])
# [[7, 8, 9, 10, 11, 12, 13, 14, 15, 5, 16], [17, 6, 18, 19], [5, 6, 20, 21, 22]]
print("decoded sentence")
pprint.pprint([tokenizer.decode(tokens) for tokens in encodes])
# ["hello , i do n't know how to use it ?",
# 'tensorflow is awesome !',
# 'it is good framework .']
vocab_size = len(vocab)
# restore the vocaburary from dumped file
print("reload from dump file")
vocab = Vocab()
vocab.load_vocab(Path("vocab.txt"))
assert vocab_size == len(vocab)
tokenizer = WhitespaceTokenizer(vocab=vocab)
pprint.pprint([tokenizer.encode(sentence) for sentence in sentences])
# [[7, 8, 9, 10, 11, 12, 13, 14, 15, 5, 16], [17, 6, 18, 19], [5, 6, 20, 21, 22]]
# vocaburary with minimum frequency limitation
vocab = Vocab()
for sentence in sentences:
vocab.add_vocabs(tokenizer.tokenize(sentence))
vocab.build_vocab(min_freq=2)
assert vocab_size != len(vocab)
# vocaburary with maximum voaburary size
vocab = Vocab()
for sentence in sentences:
vocab.add_vocabs(tokenizer.tokenize(sentence))
vocab.build_vocab(vocab_size=10)
assert 10 == len(vocab)
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
meguru_tokenizer-0.2.0.tar.gz
(10.6 kB
view hashes)
Built Distribution
Close
Hashes for meguru_tokenizer-0.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6ae865dc9baec51fe841d7a11dea62667503e9fa93ef25409dbaa9b418870782 |
|
MD5 | 0780c2134de2ada4c15caa1e391a3b92 |
|
BLAKE2b-256 | d062d6616a35b3ad6cb35d6585959bcfcbb0b1a83b006ba3411295152f6d3414 |