Skip to main content

afri-fertility

CI PyPI License arXiv Dataset

Measure the tokenization "tax" on African languages.

Commercial LLMs bill, throttle, and context-budget per token. Because the same meaning takes more tokens in African languages than in English, speakers and builders face a structural cost, latency, and context penalty — before the model is even invoked. afri-fertility measures that penalty precisely.

It is the open measurement engine behind The African Language Tax paper, the public African Tokenization Tax Leaderboard, and the cost-calculator widget at datalens.africa.

afri-fertility reproduce
Tokenizer           Language  Fertility  Premium
──────────────────────────────────────────────────
openai/o200k_base   amh          8.500   7.83×
openai/o200k_base   yor          2.674   2.46×
openai/o200k_base   swh          1.800   1.66×
openai/o200k_base   fra          1.265   1.17×

Install

pip install afri-fertility          # core: tiktoken + HF backends
pip install "afri-fertility[api]"   # + Claude / Gemini count-only
pip install "afri-fertility[viz]"   # + matplotlib figures
pip install "afri-fertility[dev]"   # + pytest, hypothesis

Requires Python 3.11+. The core path is CPU-only and key-free.


Quickstart

Single-text measurement

from afri_fertility import measure_text

m = measure_text("Àwọn ará Nàìjíríà", tokenizer="openai/o200k_base")
print(f"tokens={m.tokens}  fertility={m.fertility:.2f}  cpt={m.cpt:.2f}")
# tokens=8  fertility=2.67  cpt=2.25
afri-fertility measure \
  --text "Àwọn ará Nàìjíríà tó ń gbé ní ìlú Èkó" \
  --lang yor \
  --models openai/o200k_base,openai/cl100k_base

Cost calculator

from afri_fertility import cost_of

results = cost_of("Àwọn ará Nàìjíríà", lang="yor", models=["openai/o200k_base"])
for r in results:
    print(f"{r.tokenizer}: ${r.total_cost_usd:.6f}  NGN {r.costs_local['NGN']:.4f}")
afri-fertility cost \
  --text "Àwọn ará Nàìjíríà" \
  --lang yor \
  --models openai/o200k_base,mistral/tekken \
  --currencies NGN,ZAR,KES

Offline credibility demo

afri-fertility reproduce

Runs the bundled reference suite (10 parallel sentences, 7 languages) against all available tokenizers. No network, no API keys.


Full study

Run the complete pre-registered study from the locked config:

afri-fertility run --config configs/study_main.yaml

Results land in runs/main/:

runs/main/
├── results.parquet / results.csv / results.json
├── leaderboard.json
├── manifest.json
└── figures/
    ├── fig1_heatmap.{png,svg}
    ├── fig2_premium_script.{png,svg}
    ├── fig3_cost.{png,svg}
    ├── fig4_context.{png,svg}
    ├── fig5_general_indomain.{png,svg}
    └── fig6_premium_accuracy.{png,svg}

For HF-gated tokenizers (Llama, Gemma, …):

afri-fertility run --config configs/study_main.yaml --hf-token $HF_TOKEN
# or: export HF_TOKEN=hf_... && afri-fertility run ...

Python API

from afri_fertility import measure_text, cost_of, run_study, load_tokenizer
from afri_fertility.config import StudyConfig

# Single-text metrics
m = measure_text("Ẹ káàbọ̀", tokenizer="openai/o200k_base")

# Widget backend: cost per model
results = cost_of("Ẹ káàbọ̀", lang="yor", models=["openai/o200k_base"])

# Full study
config = StudyConfig.from_yaml("configs/study_main.yaml")
result = run_study(config)

df = result.dataframe          # pandas DataFrame
result.figures("runs/figs")    # generate all 6 figures
lb = result.to_leaderboard()   # list[dict] for the frontend

CLI reference

Command What it does
afri-fertility measure Token count, fertility, CPT, BPT for input text
afri-fertility cost Cost per model per language (widget backend)
afri-fertility run Full study from YAML config
afri-fertility figures Regenerate figures from an existing run
afri-fertility leaderboard Emit leaderboard JSON from a run
afri-fertility reproduce Offline reference suite — one-command credibility demo
afri-fertility tokenizers list Registry of all tokenizers and their availability
afri-fertility corpora list Registered corpus loaders
afri-fertility languages list All 22 study languages with ISO codes and scripts

Global flags: --hf-token, --cache-dir, --log-level, --json.

Full documentation: docs/usage.md


Metrics

All metrics are computed on parallel corpora (same meaning, different languages), so the language effect is isolated from content.

Metric Formula Meaning
Fertility F(L,T) tokens / words Tokens per word. Lower = more efficient.
Premium P(L,T) F(L,T) / F(eng,T) How many times more tokens L uses vs English.
CPT chars / tokens Characters packed per token.
BPT utf8_bytes / tokens Bytes per token (cross-script fair).
Context efficiency window_size × CPT Effective real chars in a fixed context window.

Aggregation: sum-then-divide over all sentences (not mean-of-ratios). Bootstrap 95% CIs over sentences. Baseline: English. Normalization: NFC.


Supported tokenizers

Tokenizer id Backend Notes
openai/o200k_base tiktoken GPT-4o / GPT-4.1 / o-series
openai/o200k_harmony tiktoken OpenAI open-weight (gpt-oss)
openai/cl100k_base tiktoken GPT-3.5 / GPT-4 (legacy)
meta/llama-3.1 HF Gated — needs HF_TOKEN
meta/llama-4 HF Gated — needs HF_TOKEN
google/gemma-4 HF Gated — needs HF_TOKEN
mistral/tekken HF
qwen/qwen3 HF
deepseek/v3 HF
bigscience/bloom HF Multilingual baseline
cohere/aya-expanse HF Multilingual-optimized baseline
anthropic/claude API [api] extra — count-only; needs ANTHROPIC_API_KEY
google/gemini API [api] extra — count-only; needs GEMINI_API_KEY

Unavailable tokenizers are skipped with a warning; they never crash a run. See docs/adding_a_tokenizer.md to add your own.


Supported languages

23 languages across 5 tiers:

Core (6): Yoruba · Hausa · Igbo · Wolof · Swahili · Amharic
Latin breadth (11): Zulu · Xhosa · Shona · Kinyarwanda · Luganda · Akan/Twi · Lingala · Oromo · Nigerian Pidgin · Sesotho · Bambara
Non-Latin (3): Tigrinya · Hausa-Ajami · N'Ko
Control (1): Afrikaans
Baselines (2): English · French

afri-fertility languages list

Reproducing the paper numbers

git clone https://github.com/ciphersenseai/afri-fertility
cd afri-fertility
pip install -e ".[dev,viz]"
afri-fertility reproduce                                 # offline check, no keys
export HF_TOKEN=hf_...
afri-fertility run --config configs/study_main.yaml     # full locked study
afri-fertility figures --run runs/main
afri-fertility leaderboard --run runs/main --out leaderboard.json

All tokenizer versions, price snapshot date, FX rates, and config hash are recorded in runs/main/manifest.json.


Project structure

afri-fertility/
├── src/afri_fertility/
│   ├── core/          # segmentation, metrics, aggregation (pure functions)
│   ├── tokenizers/    # tiktoken + HF + API adapters + registry
│   ├── corpora/       # FLORES-200, SIB-200, MAFAND-MT, custom JSONL/CSV
│   ├── cost/          # cost model, price/FX snapshots
│   ├── study/         # orchestrator, accuracy linkage
│   ├── report/        # tables, 6 figures, leaderboard JSON
│   ├── cli.py         # typer CLI
│   └── config.py      # pydantic StudyConfig
├── configs/           # locked study config + pinned price/FX snapshots
├── data/
│   ├── languages.yaml          # 22-language registry
│   └── reference_suite/        # offline reproduce dataset
└── tests/             # unit · golden · integration

Citation

@misc{somide2026african,
  title         = {The African Language Tax: Quantifying the Cost, Latency,
                  and Context Penalty of Tokenizing African Languages in Frontier LLMs},
  author        = {Somide, Anthony Olaoye and {DataLens Africa Research}},
  year          = {2026},
  eprint        = {2606.24460},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2606.24460}
}

License

Apache-2.0 · © 2026 DataLens Africa Research

Metadata

Release files for afri-fertility 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for afri-fertility 0.1.0
File Size Uploaded Provenance Actions
afri_fertility-0.1.0.tar.gz 65.9 kB With provenance Details for afri_fertility-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for afri-fertility 0.1.0
File Interpreter ABI Platform Actions
afri_fertility-0.1.0-py3-none-any.whl Python 3 none any Details for afri_fertility-0.1.0-py3-none-any.whl

Total release size: 122.0 kB

Release files / afri_fertility-0.1.0.tar.gz

Download URL afri_fertility-0.1.0.tar.gz
Size 65.9 kB
Tags Source
SHA-256 checksum
How to use checksums
01e6e989aed2f3031fee6c16eb185d5d5d4dc36df7e9adcc320a10665c64f27b
BLAKE2b-256 checksum
How to use checksums
6ad55963e0d18406ff3093a30264d79aaa814c06724111b75a2279a5c7dce5f9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jun 24, 2026.

Transparency log

Release files / afri_fertility-0.1.0-py3-none-any.whl

Download URL afri_fertility-0.1.0-py3-none-any.whl
Size 56.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
bb204abeec6f8e61d943bc21bc45ee6e875b2544f6452d6e89bef71c94bb8ac5
BLAKE2b-256 checksum
How to use checksums
03d1fea6e995efd638937f276d19cce65f71a14758bfdd09ebd71e6667e90d4c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jun 24, 2026.

Transparency log

Security for afri-fertility 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for afri_fertility-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jun 24, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page