transtokenizers
Token translation for language models
- GitHub: https://github.com/LAGoM-NLP/transtokenizer
- PyPI: https://pypi.org/project/trans-tokenizers/
- Licence: MIT
Features
- Translate a model from one language to another.
- Support for most scripts beyond Latin.
Installation
pip install trans-tokenizers
Usage
You do need an installation of fast_align to align the tokens. You can install from the following repo: https://github.com/FremyCompany/fast_align.
To convert a Llama model from English to Dutch, you can use the following code. This might
from transtokenizers import create_aligned_corpus, align, map_tokens, smooth_mapping, remap_model
from transformers import AutoTokenizer, AutoModelForCausalLM
import os
source_model = "meta-llama/Meta-Llama-3-8B"
target_tokenizer = "yhavinga/gpt-neo-1.3B-dutch"
export_dir = "en-nl-llama3-8b"
corpus = create_aligned_corpus(
source_language="en",
target_language="nl",
source_tokenizer=source_model,
target_tokenizer=target_tokenizer,
)
mapped_tokens_file = align(corpus, fast_align_path="fast_align")
tokenized_possible_translations, untokenized_possible_translations = map_tokens(mapped_tokens_file, source_model, target_tokenizer)
smoothed_mapping = smooth_mapping(target_tokenizer, tokenized_possible_translations)
model = remap_model(source_model, target_tokenizer, smoothed_mapping, source_model)
os.makedirs(export_dir, exist_ok=False)
new_tokenizer = AutoTokenizer.from_pretrained(target_tokenizer)
model.save_pretrained(export_dir)
new_tokenizer.save_pretrained(export_dir)
Credits
If this repo was useful to you, please cite the following paper
@inproceedings{remy-delobelle2024transtokenization,
title={Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of {LLM}s for Low-Resource {NLP}},
author={Remy, Fran{\c{c}}ois and Delobelle, Pieter and Avetisyan, Hayastan and Khabibullina, Alfiya and de Lhoneux, Miryam and Demeester, Thomas},
booktitle={First Conference on Language Modeling},
year={2024},
url={https://openreview.net/forum?id=sBxvoDhvao}
}
Metadata
Release files for trans-tokenizers 0.1.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| trans_tokenizers-0.1.4.tar.gz | 12.1 kB | Without provenance | Details for trans_tokenizers-0.1.4.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| trans_tokenizers-0.1.4-py3-none-any.whl | Python 3 | none | any | Details for trans_tokenizers-0.1.4-py3-none-any.whl |
Total release size: 24.2 kB
Release files / trans_tokenizers-0.1.4.tar.gz
| Download URL | trans_tokenizers-0.1.4.tar.gz |
|---|---|
| Size | 12.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a00abd7ce304b48b27c80fbae254bc5aa9dea475593cd9a6212945e8b5920352
|
|
BLAKE2b-256 checksum How to use checksums |
ec340effb73367a9ff9d30275b69cad18193af7ddd626a324866d391bca9f5e9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.3
|
Release files / trans_tokenizers-0.1.4-py3-none-any.whl
| Download URL | trans_tokenizers-0.1.4-py3-none-any.whl |
|---|---|
| Size | 12.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bf327e90aee6094555284e7fbb59740b536a8ad9b0d31034ad306a4d5ba982dd
|
|
BLAKE2b-256 checksum How to use checksums |
e908fe15b73bcfb922aab90edffcbbcd43e3a8ac5bbb5ba0cf12a888ef9905b0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.3
|
Security for trans-tokenizers 0.1.4
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.