Tokenizer fertility, cost, and multi-turn context-budget analyzer for low-resource Asian languages.
Verified details
These details have been verified by PyPIProject links
GitHub Statistics
Maintainers
Project description
asia-fertility 🌏
The hidden multilingual tax in your tokenizer — measured before you deploy.
Status: v0.2 under active construction. See
ROADMAP.mdand the per-phase specs intasks/.
asia-fertility measures the structural cost penalty that LLM tokenizers impose on lower-resource Asian languages. The same content can cost up to 11× more tokens in Burmese than in English on a frontier tokenizer — silent inflation of API bills, smaller usable context windows, and fewer in-context examples.
Quickstart (once v0.3 ships)
pip install "asia-fertility[oai]"
asia-fertility reproduce
What's currently usable
- v0.1 Python prototype:
legacy_v01/fertiscope/(EN↔VI only, CLI). - Live Next.js web demo: fertiscope.vercel.app.
- 41 implementation specs:
tasks/.
License
MIT © 2026 Antoine Pedretti. Bundled FLORES-200 data: CC-BY-SA 4.0 (Meta NLLB).
Project details
Verified details
These details have been verified by PyPIProject links
GitHub Statistics
Maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file asia_fertility-0.2.0.tar.gz.
File metadata
- Download URL: asia_fertility-0.2.0.tar.gz
- Upload date:
- Size: 37.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
b2bb825b9d2555a80110f1686e1bacd3b212697a81e8fd1d9bfeb63a2bfbeab5
|
|
| MD5 |
15bd0260b021b7ca178c820d8289f69a
|
|
| BLAKE2b-256 |
89ffab0312dc39da24987cd22d20e0d7dfa9150a6775cbe0283c6ac139d22c88
|
Provenance
The following attestation bundles were made for asia_fertility-0.2.0.tar.gz:
Publisher:
publish.yml on Helmo21/asia-fertility
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
asia_fertility-0.2.0.tar.gz -
Subject digest:
b2bb825b9d2555a80110f1686e1bacd3b212697a81e8fd1d9bfeb63a2bfbeab5 - Sigstore transparency entry: 2005757023
- Sigstore integration time:
-
Permalink:
Helmo21/asia-fertility@e0d62a6ff346fcb0559ed747e7a334c2c93d5afb -
Branch / Tag:
refs/tags/v0.2.0 - Owner: https://github.com/Helmo21
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish.yml@e0d62a6ff346fcb0559ed747e7a334c2c93d5afb -
Trigger Event:
push
-
Statement type:
File details
Details for the file asia_fertility-0.2.0-py3-none-any.whl.
File metadata
- Download URL: asia_fertility-0.2.0-py3-none-any.whl
- Upload date:
- Size: 50.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
73739ebc4d676539c439347a8c966827d2a3fedc71b852ee94bb2fa24e186970
|
|
| MD5 |
61e8c3ff4650ffbde76122052e6a48f0
|
|
| BLAKE2b-256 |
89527a194fc93b919c59f621bdc089e1368e77a2d6e3e9db4cbde029ea8224d7
|
Provenance
The following attestation bundles were made for asia_fertility-0.2.0-py3-none-any.whl:
Publisher:
publish.yml on Helmo21/asia-fertility
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
asia_fertility-0.2.0-py3-none-any.whl -
Subject digest:
73739ebc4d676539c439347a8c966827d2a3fedc71b852ee94bb2fa24e186970 - Sigstore transparency entry: 2005757174
- Sigstore integration time:
-
Permalink:
Helmo21/asia-fertility@e0d62a6ff346fcb0559ed747e7a334c2c93d5afb -
Branch / Tag:
refs/tags/v0.2.0 - Owner: https://github.com/Helmo21
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish.yml@e0d62a6ff346fcb0559ed747e7a334c2c93d5afb -
Trigger Event:
push
-
Statement type:
