Skip to main content

Package for evaluating text tokenizations.

Project description

tokenization-scorer PyPI Version

Simple package for evaluating text tokenizations. The input is a text (list of files or stdin) and output a single number. The higher the number, the better the tokenization. The intended workflow is to try multiple tokenizations and select the one with the highest number.

It can be used from the command line:

pip3 install tokenization-scorer

tokenization-scorer -i en-de.tokenized_with_unigramlm.{en,de}
> 0.4826

tokenization-scorer -i en-de.tokenized_with_wordpiece.{en,de}
> 0.5047

or within Python:

import tokenization_scorer
text1 = "pick @@ed pick @@l @@ed pick @@les"
tokenization_scorer.score(text1, metric="renyi", power=2.5)
> 0.8031528501359657

text2 = "pick @@e @@d pick @@l @@e @@d pick @@l @@e @@s"
tokenization_scorer.score(text2, metric="renyi", power=2.5)
> 0.9105681923824472

Use tokenization-scorer -h to get an overview of supported metrics. This package is a side-product of the paper Tokenization and the Noiseless Channel.

@inproceedings{tokenization_noiseless, 
    title={Tokenization and the Noiseless Channel},
    author={Zouhar, Vilém and Meister, Clara and Gastaldi, Juan Luis and Sachan, Mrinmaya and Cotterell, Ryan},
    booktitle={Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics},
    year={2023},
    url={https://aclanthology.org/2023.acl-long.284/},
}

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tokenization_scorer-1.1.2.tar.gz (4.6 kB view hashes)

Uploaded Source

Built Distribution

tokenization_scorer-1.1.2-py3-none-any.whl (5.5 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page