Skip to main content

Library for manipulating the existing tokenizer.

Project description

Tokenizer-Changer

Python script for manipulating the existing tokenizer.

The solution was tested on Llama3-8B tokenizer.


Installation

Installation from PyPI:

pip install tokenizerchanger

Usage

changer = TokenizerChanger(tokenizer, space_sign)

Create the object of TokenizerChanger class that optionally requires an existing tokenizer and space sign, which differs from one tokenizer to another. The tokenizer could be PreTrainedTokenizerFast class from рџ¤— tokenizers library.

changer.load_tokenizer(tokenizer)

If you did not load the tokenizer with TokenizerChanger class declaration, you can load it using this function.

changer.set_space_sign(space_sign)

If you did not set the space sign with TokenizerChanger class declaration, you can set it using this function. Default space sign is Д .

Deletion

changer.delete_tokens(list_of_unwanted_tokens, include_substrings)

Deletes the unwanted tokens from the tokenizer. If include_substrings is True, all token occurrences will be deleted even in other tokens. Defaults to True.

changer.delete_k_least_frequent_tokens(k=1000)
changer.delete_k_least_frequent_tokens(k=1000, exclude=list_of_tokens)

Deletes k most frequent tokens. The exclude argument stands for tokens that will be ignored during the deletion of the least frequent tokens.

changer.delete_overlaps(vocab)

Finds and deletes all intersections of the tokenizer's vocabulary and the vocab variable from the tokenizer. Notice that vocab should be a dict variable.

changer.delete_inappropriate_merges(vocab)

Deletes all merges from tokenizer which contradict the vocab variable. Notice that vocab should be a list[str] variable.

Addition

The idea of creating such functions arose due to the fact that the built-in functions do not add tokens/merges properly, when some tokens are deleted. That is why you can get more tokens after encoding the same text, even if the necessary tokens have been added.

changer.add_tokens(list_of_tokens)

Adds the tokens from the list. The indexes will be filled automatically.

changer.add_merges(list_of_merges)

Adds the merges from the list. If there are no necessary tokens for this merge, their addition will be suggested.

"Get" functions

changer.get_overlapping_tokens(vocab)

Returns the intersection between the tokenizer's vocabulary and the vocab variable. Notice that vocab should be a dict variable.

changer.get_overlapping_merges(merges)

Returns the intersection between the tokenizer's merges and the merges variable. Notice that merges should be a list variable.

Saving

changer.save_tokenizer(path)

Saves the current state of the changed tokenizer. Additionally, it saves tokenizer configs into path folder (./updated_tokenizer by default).

tokenizer = ch.updated_tokenizer()

Return the changed tokenizer.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

TokenizerChanger-0.3.5.tar.gz (10.1 kB view details)

Uploaded Source

Built Distribution

TokenizerChanger-0.3.5-py3-none-any.whl (10.5 kB view details)

Uploaded Python 3

File details

Details for the file TokenizerChanger-0.3.5.tar.gz.

File metadata

  • Download URL: TokenizerChanger-0.3.5.tar.gz
  • Upload date:
  • Size: 10.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.10.11

File hashes

Hashes for TokenizerChanger-0.3.5.tar.gz
Algorithm Hash digest
SHA256 d5810cc4f411dac4f22e84fcaf111c1fee80690d33a632380c679c69231c4f77
MD5 58b63c39eb3e1dc4dfb557ecd708fc63
BLAKE2b-256 3d7c4db473b1c5c8cd71c8c8efa5d65cc729204729401b5bd83e044435e111f0

See more details on using hashes here.

File details

Details for the file TokenizerChanger-0.3.5-py3-none-any.whl.

File metadata

File hashes

Hashes for TokenizerChanger-0.3.5-py3-none-any.whl
Algorithm Hash digest
SHA256 ade0ea70e30ecf88f31300612c814c643ebdfca5f731190753a1d0b0d63d7094
MD5 f3ec98df782970a562d20d3f7c859b08
BLAKE2b-256 d15c07bb46c4165a61d76544e6c71302c5893969ac14104f8d0b48fe64dccc21

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page