ALTA Tokenizer
alta-tokenizer is a Python library designed for tokenizing Kinyarwanda language text, it can also tokenizer other languages like English, French or similar languages but with low compression rate since the tokenizer was trained on Kinyarwanda text only.
There is an option for training your own custom tokenizer using defined function or method. It is covered in the section of training your own tokenizer.
Hence, you can use that way to train your own tokenizer using dataset for a different language. It is based on the Byte Pair Encoding (BPE) algorithm.
It can both encode and decode text in Kinyarwanda. The metric used to measure the accuracy of this tokenizer is the compression rate and ability to encode and decode texts
Compression rate is the ratio of the total original number of characters in the text to the number of tokens in the encoded text.
Example:
For the sentence: "Nagiye gusura abanyeshuri."
- The sentence has 26 characters.
- Suppose the sentence is tokenized into the following tokens:
[23, 45, 67, 89, 23, 123, 44, 22, 55, 22, 45]. - The total number of tokens is 11.
$$ \text{Compression Rate} = \frac{26}{11} $$
So, the compression rate is 2.36X (where X indicates that the number is approximate).
Special Tokens
<|PAD|> is a special token for padding, it is represented by 0 in the vocab
<|EOS|> is a special token for indicating the end of sequence
<|BOS|> is a special token for indicating the begining of sequence
<|SEP|> is a speical token for separating two sequences
<|MASK|> is a spicial token for masking
<|UNK|> is a special token for unknown
<|CLS|> is a special token for classification
ALTA-Tokenizer 1.0
Version 1.0 features a vocabulary size of 32,256 with improved words or subwords. It was trained on over 19 millions Kinyarwanda characters and shows a significantly improved compression rate compared to the previous version.
ALTA-Tokenizer 2.0
Version 1.0 features a vocabulary size of 50,257 with improved words or subwords. Improved the tokenization with more corpus
Installation
You can install the package using pip:
pip install alta-tokenizer
Basis Usage
from kin_tokenizer import KinTokenizer # Importing Tokenizer class
from kin_tokenizer.utils import create_sequences, create_dataset
# Creating an instance of tokenizer
tokenizer = KinTokenizer()
# Encoding
text = """
Nuko Semugeshi akunda uwo mutwe w'abatwa, bitwaga Ishabi, awugira intore ze. Bukeye
bataha biyereka ingabo; dore ko hambere nta mihamirizo yindi yabaga mu Rwanda; guhamiriza
byaje vuba biturutse i Burundi. Ubwo bataha Semugeshi n'abatware be barabitegereza basanga
ari abahanga bose, ariko Waga akaba umuhanga w'imena muri bo; nyamara muri ubwo buhanga
bwe akagiramo intege nke ku mpamvu yo kunanuka, yari afite uruti ruke."""
tokens = tokenizer.encode(text)
print(tokens)
# Calculating compression rate
text_len = len(text)
tokens_len = len(tokens)
# Decoding
decoded_text = tokenizer.decode(tokens)
print(decoded_text)
compression_rate = text_len / tokens_len
print(f"Compression rate: {compression_rate:.2f}X")
# Creating dataset for training your LLM
create_dataset(text_file_path=text_file_path, nbr_processes=nbr_processes, sequence_length=sequence_length, destination_dir=destination_dir, step_size=step_size)
# Printing the vocab size
print(tokenizer.vocab_size)
# Print vocabulary (first 200 items)
count = 0
for k, v in tokenizer.vocab.items():
print("{} : {}".format(k, v))
count += 1
if count > 100:
break
Training Your Own Tokenizer
You can also train your own tokenizer using the utils module, which provides two functions: a training function and a function for creating sequences after encoding your text. N.B: Your chosen vocab_size will be met depening on the amount of data you have used for training. The vocab_size is a hyperparameter to be adjusted for better vocabularies in your vocab, and also the size of your dataset and diversity matters. The vocab is initialized by count of 256 from 1-255 unicode characters and 0 for <|PAD|>.
from kin_tokenizer import KinTokenizer
from kin_tokenizer.utils import train_kin_tokenizer
# Training the tokenizer
tokenizer = train_kin_tokenizer(text, vocab_size=512, save=True, tokenizer_path=DATA_ROOT, retrain=False)
# Encoding text using custom trained tokenizer
tokens = tokenizer.encode(text)
Contributing
The project is still being updated and contributions are welcome. You can contribute by:
- Reporting bugs
- Suggesting features
- Writing or improving documentation
- Submitting pull requests
Metadata
Release files for alta-tokenizer 2.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| alta_tokenizer-2.1.tar.gz | 2.1 MB | Without provenance | Details for alta_tokenizer-2.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| alta_tokenizer-2.1-py3-none-any.whl | Python 3 | none | any | Details for alta_tokenizer-2.1-py3-none-any.whl |
Total release size: 4.3 MB
Release files / alta_tokenizer-2.1.tar.gz
| Download URL | alta_tokenizer-2.1.tar.gz |
|---|---|
| Size | 2.1 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b997ef40f1057fcc20478a39cc9d646f6113dae16e4ee1160aa8ed7324868bf8
|
|
BLAKE2b-256 checksum How to use checksums |
c3fceeaf5099ba747a07e447d602cb6039c1af51fa8fcc2e574f79e1b9db60fa
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.3
|
Release files / alta_tokenizer-2.1-py3-none-any.whl
| Download URL | alta_tokenizer-2.1-py3-none-any.whl |
|---|---|
| Size | 2.1 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
4fda865effb29d7cae6f21d4027a1962dca8b07f5087081c44d4b6cbed91579e
|
|
BLAKE2b-256 checksum How to use checksums |
3cc7ec04ef023d09a4d9092785ef27ab0136e5af7a16600ad15c76ef047bd1ed
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.3
|
Security for alta-tokenizer 2.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.