A library for calculating fertility and parity scores with visualizations using tokenizers.
Project description
pytoke
pytoke is a Python library that calculates fertility and parity scores for text using tokenizers from the transformers library. It also provides visualization tools to analyze token metrics across datasets.
Features
- Fertility Score: Calculate the ratio of token count to word count for a given text.
- Parity Score: Compare token counts between two texts (e.g., original and translated).
- TokenMetrics Class: Easily process and visualize token metrics for a dataset.
Installation
Install pytoke using pip:
pip install pytoke
Works Cited
Parity calculation from https://arxiv.org/abs/2305.15425 (page 3).
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file pytoke-0.1.1.tar.gz.
File metadata
- Download URL: pytoke-0.1.1.tar.gz
- Upload date:
- Size: 10.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.1.0 CPython/3.10.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
5554ffff9a2babfce94c7b5781ee3225086ef4b8c21d5a2dc5e39f9a411f3a27
|
|
| MD5 |
b945bf45a6d7010c3e07bab0a6d138ca
|
|
| BLAKE2b-256 |
001fc14b861a8a022f424a35f3d4246b9a205ae2f99f8933ebc299627b178c7c
|
File details
Details for the file pytoke-0.1.1-py3-none-any.whl.
File metadata
- Download URL: pytoke-0.1.1-py3-none-any.whl
- Upload date:
- Size: 4.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.1.0 CPython/3.10.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
d619f3206e6c6d66690724b82f7e715bb98ad01058ef40e5cde073f850fa6b79
|
|
| MD5 |
9c180c024017b784926428eadea1b160
|
|
| BLAKE2b-256 |
357b89640a2c3673371b9fb752a0c26597190a99aa9987b15229d026367cd46a
|
