SubTokenizer
Subwords tokenizer based on google code from tensor2tensor. It supports tags and combined tokens in addition to google tokenizer.
- Tags are tokens starting from
@, they are not splited on parts. - No break symbol
¬'\xac'allows to join several words in one token.
Tokenizer does unicode normalization and controls characters escaping. It's also possible to encode rare symbols so they can be splited on parts by subwords algorithm.
Original google subwords tokenizer: https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/data_generators/text_encoder.py
By default before learning and tokenizing SubTokenizer encodes all control characters and @ ¬ symbols. To use tags it's needed to run encoding first then add tags and after learn/tokenize with encode_controls=Flase or --no_encode_controls in command line mode.
Install:
pip install subtokenizer
Usage:
cat text_file.txt | subtokenizer learn -o bpe.file -s 1000 -r reserved_tokens.txt
cat text_file.txt | subtokenizer tokenize -s bpe.file > tokenized_file.txt
cat tokenized_file.txt | subtokenizer detokenize -s bpe.file > text_file.txt
Or:
from subtokenizer import SubTokenizer
tokenizer = SubTokenizer.learn(words_count)
tokenizer.save(subwords_filename)
tokenizer = SubTokenizer.load(subwords_filename)
tokens = tokenizer.tokenize(line)
line = tokenizer.detokenize(tokens)
1 maintainer
Metadata
1 maintainer
Release files for subtokenizer 0.0.19
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| subtokenizer-0.0.19.tar.gz | 10.3 kB | Details |
Release files / subtokenizer-0.0.19.tar.gz
| Download URL | subtokenizer-0.0.19.tar.gz |
|---|---|
| Size | 10.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c1d5bf9f1aa897b08c86a4cacc8be6221f4240281481ffb961e13a067931abc2
|
|
BLAKE2b-256 checksum How to use checksums |
8df9bbc37011b825802362561fc8d978742592cafad9e366d4b656d0dbc94a5f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.6.8
|
