token2token

Easy-to-make sub-token mappings using parallel corpora and automated embedding reinitialization for Large Language Models (LLMs).

Key Features

• Sub-Token Alignment (Token2token): Map sub-token vocabularies across different HuggingFace tokenizers (or languages) using parallel text corpora.
• Classic Word Alignment (Word2word): Generate word-level cross-lingual dictionaries using NLTK or custom tokenizers.
• Model Weight Reinitialization: Transfer target sub-token representations by computing mean embeddings from mapped source sub-tokens across model weight matrices (inputs, outputs, and biases).

Example

You want to align French and English on sub-token level. You need:

• A French (HuggingFace) tokenizer
• An English tokenizer (could be the same one)
• A French-English parallel corpus (if none provided OpenSubtitles2024 from huggingface is used by default)
• This software

For each token in the first tokenizer you will get a list of possible matching tokens from the second tokenizers and a score for each of them.

Alternatively, you can still use the old pipeline and get word mappings based on NLTK or other specialized tokenizer

Installation

Install the package directly from PyPI or build from source:

pip install token2token

OR

git clone [https://github.com/procesaur/token2token.git](https://github.com/procesaur/token2token.git)
cd token2token
python setup.py install

1. Token Alignment & Mapping Generation

A. Sub-token Level Mapping (Token2token)

Align sub-tokens using HuggingFace tokenizers (e.g., aligning target sub-tokens to English source sub-tokens on a parallel corpus like OpenSubtitles):

from token2token import Token2token

# Generates top-k mapping from English to French tokens for a specific model
enfr = Token2token.make(
    lang1="en", 
    lang2="fr", 
    tokenizer1="Qwen/Qwen3.5-0.8B", 
    tokenizer2="Qwen/Qwen3.5-0.8B", 
    n_lines=500000
)

# Returns dictionary of mapped source tokens and co-occurrence scores
print(enfr("Ġapple"))
# Output: {'Ġpomme': 18.723, 'omm': 4.715, 'Ġpommes': 2.852}

B. Word Level Mapping (Word2word)

For traditional word-level alignment across languages:

from token2token import Word2word

enfr = Word2word.make(lang1="en", lang2="fr", n_lines=500000)
print(enfr("apple"))
# Output: {'pomme': 18.491, 'pommiers': 2.913, 'pommes': 2.819}

The old pipeline has been modified :

• to use huggingface datasets for corpora
• to output scores together with words and
• to save in plain, human readable JSON format.

In both cases, the custom lexicon can be loaded from the directory it is stored in (defaulting to home directory in linux or "C:\word2word" in Windows

C. Loading Saved Mappings

Saved lexicons are written as human-readable JSON files (id_mapping.json) in C:\word2word (Windows) or ~/word2word (Linux):

from token2token import Token2token
my_en2fr = Token2token.load("en", "fr")
# Loaded token2token custom token mapping from C:\word2word\en-fr.json

from token2token import Word2word
my_en2fr = Word2word.load("en", "fr", "data/pubmed.en-fr")
# Loaded token2word custom bilingual lexicon from C:\word2word\en-fr.json

2. Tokenizer Vocabulary Extension & Adaptation with Weight initialization

Building an extended tokenizer based on an existing tokenizer (HF model) and a custom training dataset with four pruning options to create space for new vocab

1. Tokenizer Extension via Vocabulary Pruning (make_tokenizer.py):
   • Targeted Pruning: Make space for new vocabulary by pruning underused characters (cyr, zh, both, or all).
   • Smart Pre-tokenization: Uses the pruned tokenizer on a target corpus to compute new frequent BPE merges and generate a custom extended vocabulary.
   • Transliteration Support: Option to handle script variations during tokenizer training.
2. Sub-Token Alignment (Token2token):
   • Maps sub-token alignment co-occurrence scores between target/extended tokens and original source sub-tokens across parallel corpora.
3. Model Weight Reinitialization from the produced mapping (transform_weights.py):

Example Script Usage

from token2token.extend import adapt_tokenizer

extended_save_path, prunned_save_path, vocab_map_save_path = adapt_tokenizer(
    model="Qwen/Qwen3.5-0.8B",
    lang = "sr",
    dataset="procesaur/sr-tokenizer-test",
    prune_target="cyr",
    n_lines=10000,
    reinitialize_old=True,
    no_overlap_data = "procesaur/KOMPaS",
    no_overlap_subset="en",
    no_overlap_lines=10000,
)

from token2token import reinitialize_weights

reinitialize_weights(
    lang1="sr",
    lang2="ru",
    model="Qwen/Qwen3.5-0.8B",
    extended_tokenizer_path="C:/word2word/my-tokenizer",
    pruned_tokenizer_path="C:/word2word/my-pruned-tokenizer",
    new_vocab_map_path="C:/word2word/my-tokenizer/extended_new_vocab_map.json",
    num_workers=8,
    datapref = "procesaur/KOPaKS",
    subset = "ru",
    n_lines = 20000,
    savedir = "c:/word2word/test"
)

Methodology

token2token calculates top-k word and token translations based on co-occurrence statistics between cross-lingual pairs in parallel corpora. It incorporates a correction term to counteract confounding context words within the same sentence.

When reinitializing model parameters, target token vectors are set to the vector mean (or median) of their mapped source sub-tokens across:

• Input Embeddings (model.get_input_embeddings())
• Output LM Head Weights (model.get_output_embeddings())
• Output Biases (if present)

Reads are performed off a frozen copy of the original weight tensors to prevent cascaded/dependency update corruption.

Multiprocessing

Dataset processing and token alignment utilize multiprocessing (defaulting to 8 workers):

enfr = Token2token.make(lang1="en", lang2="fr", num_workers=16)

References

If you use word2word for research, please cite:

@inproceedings{choe2020word2word,
 author = {Yo Joong Choe and Kyubyong Park and Dongwoo Kim},
 title = {word2word: A Collection of Bilingual Lexicons for 3,564 Language Pairs},
 booktitle = {Proceedings of the 12th International Conference on Language Resources and Evaluation (LREC 2020)},
 year = {2020}
}

For token2token and weight initialization add-ons citation coming soon.

Authors & Contributors

Mihailo Škorić based on Kyubyong Park, Dongwoo Kim, YJ Choe, and Taido Purason