Skip to main content

Calculate and visualize fertility and parity metrics for tokenizers

Project description

TokenKit

A Python library for computing fertility and parity metrics for language models and translations using tokenizers like LLaMA, T5, and Gemma.

Install

pip install .

Usage

from tokenkit import fertilize, paritize, TokenMetrics

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tokenkit-0.1.0.tar.gz (9.7 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

tokenkit-0.1.0-py3-none-any.whl (4.2 kB view details)

Uploaded Python 3

File details

Details for the file tokenkit-0.1.0.tar.gz.

File metadata

  • Download URL: tokenkit-0.1.0.tar.gz
  • Upload date:
  • Size: 9.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.11.9

File hashes

Hashes for tokenkit-0.1.0.tar.gz
Algorithm Hash digest
SHA256 96d3cd6ebe31ccd117b766672869879a0514bbb04bf018bc1c7a63e9b6aae4c4
MD5 92b18808b465b31a2456f69c7cfa56e5
BLAKE2b-256 a5c937d687fc92ddd5f7286d39acddf93e63886bceb49603db4250073a01ee77

See more details on using hashes here.

File details

Details for the file tokenkit-0.1.0-py3-none-any.whl.

File metadata

  • Download URL: tokenkit-0.1.0-py3-none-any.whl
  • Upload date:
  • Size: 4.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.11.9

File hashes

Hashes for tokenkit-0.1.0-py3-none-any.whl
Algorithm Hash digest
SHA256 ae314609d3f334a4a4de56edcdc749d55a9d558a978d8617db0b04fa7b73842d
MD5 017d6406030911ba675461bace1522e7
BLAKE2b-256 dff43e7029a655b2e9a7a2d36ad0601a69b9a2b306f4f8d32c7ca7136ae610e1

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page