✨ What you get

• 📊 Entropy & divergence estimators — MLE, James–Stein, Chao–Shen, Chao–Wang–Jost, NBRS, and NSB, bundled and ready to call.
• 🔡 Segmental informativity & lexical information gain — per-phoneme next-phoneme surprisal, plus the lexical information gain (mean entropy of the words still compatible) at each phoneme, over words encoded as phoneme strings.
• 🎲 Symmetric-Dirichlet inference — recover a concentration parameter α from an entropy estimate, with predicted rank/probability curves.
• 🧮 Maximum-entropy fitting — fit a distribution to target feature expectations via the convex dual.
• 📦 Batteries included — ships a CELEX sample so every example runs out of the box; reproducible, tested, and pip-installable.

🚀 Install

pip install phoneme-entropy

pip install "phoneme-entropy[plot]"   # matplotlib + seaborn for plot_ranks
pip install "phoneme-entropy[data]"   # Hugging Face datasets loader

⚡ Quickstart

from collections import defaultdict
import numpy as np, pandas as pd
from phoneme_entropy import (
    Entropy, estimate_alpha_entropy,
    segment_informativity, phoneme_prefix_entropy,
    compute_maxent_from_matrix,
)
from phoneme_entropy.data import load_sample

sample = load_sample()                       # bundled CELEX sample (Word = space-separated phonemes)

# Per-phoneme next-phoneme surprisal and lexical information gain
surprisal          = segment_informativity(sample["Word"])
lexical_info_gain  = phoneme_prefix_entropy(sample["Word"])

# Symmetric-Dirichlet concentration implied by an entropy estimate
phon = defaultdict(int)
for w in sample["Word"]:
    for p in w.split(): phon[p] += 1
alpha, se = estimate_alpha_entropy(list(phon.values()), n_boot=0)

📓 Notebooks & examples

Notebook	Open	What it does
Colab quickstart		Self-contained: pip installs from PyPI and runs the full pipeline on the bundled sample.
Tutorial		The same walkthrough for local or Colab use.
quickstart.py	—	Script version of the walkthrough.

🗂️ Data

A small CELEX-derived sample ships with the package — just call load_sample().

The full sample is mirrored as a (permissions-only) — access is granted to holders of a valid CELEX/LDC license (LDC96L14):

from phoneme_entropy.data import load_hf_dataset
ds = load_hf_dataset(split="train")   # after your access request is approved

🧰 API at a glance

📖 Citation & license

Released under the MIT License. Bundles third-party code — see THIRD_PARTY_LICENSES. If you use this library, please cite it (see CITATION.cff) along with the upstream entropy-estimators project.