Skip to main content

ALTA Tokenizer

alta-tokenizer is a Python library designed for tokenizing Kinyarwanda language text, it can also tokenizer other languages like English, French or similar languages but with low compression rate since the tokenizer was trained on Kinyarwanda text only. There is an option for training your own custom tokenizer using defined function or method. It is covered in the section of training your own tokenizer. Hence, you can use that way to train your own tokenizer using dataset for a different language. It is based on the Byte Pair Encoding (BPE) algorithm.

It can both encode and decode text in Kinyarwanda. The metric used to measure the accuracy of this tokenizer is the compression rate and ability to encode and decode texts

Compression rate is the ratio of the total original number of characters in the text to the number of tokens in the encoded text.

Example:

For the sentence: "Nagiye gusura abanyeshuri."

  • The sentence has 26 characters.
  • Suppose the sentence is tokenized into the following tokens: [23, 45, 67, 89, 23, 123, 44, 22, 55, 22, 45].
  • The total number of tokens is 11.

$$ \text{Compression Rate} = \frac{26}{11} $$

So, the compression rate is 2.36X (where X indicates that the number is approximate).

Special Tokens

<|PAD|> is a special token for padding, it is represented by 0 in the vocab

<|EOS|> is a special token for indicating the end of sequence

<|BOS|> is a special token for indicating the begining of sequence

<|SEP|> is a speical token for separating two sequences

<|MASK|> is a spicial token for masking

<|UNK|> is a special token for unknown

<|CLS|> is a special token for classification

ALTA-Tokenizer 1.0

Version 1.0 features a vocabulary size of 32,256 with improved words or subwords. It was trained on over 19 millions Kinyarwanda characters and shows a significantly improved compression rate compared to the previous version.

ALTA-Tokenizer 2.0

Version 1.0 features a vocabulary size of 50,257 with improved words or subwords. Improved the tokenization with more corpus

Installation

You can install the package using pip:

    pip install alta-tokenizer

Basis Usage

   from kin_tokenizer import KinTokenizer  # Importing Tokenizer class
   from kin_tokenizer.utils import create_sequences, create_dataset
   # Creating an instance of tokenizer
   tokenizer = KinTokenizer()

   # Encoding
   text = """
   Nuko Semugeshi akunda uwo mutwe w'abatwa, bitwaga Ishabi, awugira intore ze. Bukeye
bataha biyereka ingabo; dore ko hambere nta mihamirizo yindi yabaga mu Rwanda; guhamiriza
byaje vuba biturutse i Burundi. Ubwo bataha Semugeshi n'abatware be barabitegereza basanga
ari abahanga bose, ariko Waga akaba umuhanga w'imena muri bo; nyamara muri ubwo buhanga
bwe akagiramo intege nke ku mpamvu yo kunanuka, yari afite uruti ruke."""

   tokens = tokenizer.encode(text)
   print(tokens)

   # Calculating compression rate
   text_len = len(text)
   tokens_len = len(tokens)

   # Decoding
   decoded_text = tokenizer.decode(tokens)
   print(decoded_text)

   compression_rate = text_len / tokens_len
   print(f"Compression rate: {compression_rate:.2f}X")

       
   # Creating dataset for training your LLM
   create_dataset(text_file_path=text_file_path, nbr_processes=nbr_processes, sequence_length=sequence_length, destination_dir=destination_dir, step_size=step_size)

   # Printing the vocab size
   print(tokenizer.vocab_size)

   # Print vocabulary (first 200 items)
   count = 0
   for k, v in tokenizer.vocab.items():
       print("{} : {}".format(k, v))
       count += 1
       if count > 100:
           break

Training Your Own Tokenizer

You can also train your own tokenizer using the utils module, which provides two functions: a training function and a function for creating sequences after encoding your text. N.B: Your chosen vocab_size will be met depening on the amount of data you have used for training. The vocab_size is a hyperparameter to be adjusted for better vocabularies in your vocab, and also the size of your dataset and diversity matters. The vocab is initialized by count of 256 from 1-255 unicode characters and 0 for <|PAD|>.

    from kin_tokenizer import KinTokenizer
    from kin_tokenizer.utils import train_kin_tokenizer

    # Training the tokenizer
    tokenizer = train_kin_tokenizer(text, vocab_size=512, save=True, tokenizer_path=DATA_ROOT, retrain=False)

    # Encoding text using custom trained tokenizer
    tokens = tokenizer.encode(text)

Contributing

The project is still being updated and contributions are welcome. You can contribute by:

  • Reporting bugs
  • Suggesting features
  • Writing or improving documentation
  • Submitting pull requests

Metadata

Release files for alta-tokenizer 2.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for alta-tokenizer 2.1
File Size Uploaded Provenance Actions
alta_tokenizer-2.1.tar.gz 2.1 MB Without provenance Details for alta_tokenizer-2.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for alta-tokenizer 2.1
File Interpreter ABI Platform Actions
alta_tokenizer-2.1-py3-none-any.whl Python 3 none any Details for alta_tokenizer-2.1-py3-none-any.whl

Total release size: 4.3 MB

Release files / alta_tokenizer-2.1.tar.gz

Download URL alta_tokenizer-2.1.tar.gz
Size 2.1 MB
Tags Source
SHA-256 checksum
How to use checksums
b997ef40f1057fcc20478a39cc9d646f6113dae16e4ee1160aa8ed7324868bf8
BLAKE2b-256 checksum
How to use checksums
c3fceeaf5099ba747a07e447d602cb6039c1af51fa8fcc2e574f79e1b9db60fa
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.3

Release files / alta_tokenizer-2.1-py3-none-any.whl

Download URL alta_tokenizer-2.1-py3-none-any.whl
Size 2.1 MB
Tags Python 3
SHA-256 checksum
How to use checksums
4fda865effb29d7cae6f21d4027a1962dca8b07f5087081c44d4b6cbed91579e
BLAKE2b-256 checksum
How to use checksums
3cc7ec04ef023d09a4d9092785ef27ab0136e5af7a16600ad15c76ef047bd1ed
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.3

Security for alta-tokenizer 2.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page