Skip to main content

Library for manipulating the existing tokenizer.

Project description

Tokenizer-Changer

Python script for manipulating the existing tokenizer.

The solution was tested on Llama3-8B tokenizer.


Installation

Installation from PyPI:

pip install tokenizerchanger

Usage

changer = TokenizerChanger(tokenizer, space_sign)

Create the object of TokenizerChanger class that optionally requires an existing tokenizer and space sign, which differs from one tokenizer to another. The tokenizer could be PreTrainedTokenizerFast class from рџ¤— tokenizers library.

changer.load_tokenizer(tokenizer)

If you did not load the tokenizer with TokenizerChanger class declaration, you can load it using this function.

changer.set_space_sign(space_sign)

If you did not set the space sign with TokenizerChanger class declaration, you can set it using this function. Default space sign is Д .

Deletion

changer.delete_tokens(list_of_unwanted_tokens, include_substrings)

Deletes the unwanted tokens from the tokenizer. If include_substrings is True, all token occurrences will be deleted even in other tokens. Defaults to True.

changer.delete_k_least_frequent_tokens(k=1000)
changer.delete_k_least_frequent_tokens(k=1000, exclude=list_of_tokens)

Deletes k most frequent tokens. The exclude argument stands for tokens that will be ignored during the deletion of the least frequent tokens.

changer.delete_overlaps(vocab)

Finds and deletes all intersections of the tokenizer's vocabulary and the vocab variable from the tokenizer. Notice that vocab should be a dict variable.

changer.delete_inappropriate_merges(vocab)

Deletes all merges from tokenizer which contradict the vocab variable. Notice that vocab should be a list[str] variable.

Addition

The idea of creating such functions arose due to the fact that the built-in functions do not add tokens/merges properly, when some tokens are deleted. That is why you can get more tokens after encoding the same text, even if the necessary tokens have been added.

changer.add_tokens(list_of_tokens)

Adds the tokens from the list. The indexes will be filled automatically.

changer.add_merges(list_of_merges)

Adds the merges from the list. If there are no necessary tokens for this merge, their addition will be suggested.

"Get" functions

changer.get_overlapping_tokens(vocab)

Returns the intersection between the tokenizer's vocabulary and the vocab variable. Notice that vocab should be a dict variable.

changer.get_overlapping_merges(merges)

Returns the intersection between the tokenizer's merges and the merges variable. Notice that merges should be a list variable.

Saving

changer.save_tokenizer(path)

Saves the current state of the changed tokenizer. Additionally, it saves tokenizer configs into path folder (./updated_tokenizer by default).

tokenizer = ch.updated_tokenizer()

Return the changed tokenizer.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

TokenizerChanger-0.3.4.tar.gz (10.0 kB view details)

Uploaded Source

Built Distribution

TokenizerChanger-0.3.4-py3-none-any.whl (10.5 kB view details)

Uploaded Python 3

File details

Details for the file TokenizerChanger-0.3.4.tar.gz.

File metadata

  • Download URL: TokenizerChanger-0.3.4.tar.gz
  • Upload date:
  • Size: 10.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.10.11

File hashes

Hashes for TokenizerChanger-0.3.4.tar.gz
Algorithm Hash digest
SHA256 d7f846994507838e14133295d24a184388d5dfcd939bc923aa450e1c058e01e9
MD5 ae1a69a2ac3c1f8188866b53b5280592
BLAKE2b-256 1d2b9dd417d9e82bf5ccbd1fbe738c05f7a6c606eaf810432cc1dfde30eacebc

See more details on using hashes here.

Provenance

File details

Details for the file TokenizerChanger-0.3.4-py3-none-any.whl.

File metadata

File hashes

Hashes for TokenizerChanger-0.3.4-py3-none-any.whl
Algorithm Hash digest
SHA256 8e1fc53bc1098a907668d14a89390cadcce4369c8fbdde26488e04da38cc36f9
MD5 e24788496c067758740fab0cbcea5190
BLAKE2b-256 4d54ef8098588f7ab4363f94120e7d839aa61fa6fd937843ee877e81f62748c0

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page