ppllm 0.2.3

A Python Library to Compute LLM's Perplexity and Surprisal

🤔 ppllm

A Python Library to Compute LLM's Perplexity and Surprisal

Features

🤔 ppllm allows to compute various information-theoretic metrics given a text and an LLM, including Perplexity (PPL), Surprisal, and bits per character (BPC).

🤔 ppllm implements windowed PPL, which allows to compute the PPL of arbitrarily long texts. It offers both a CLI and a python API and supports large models through pipeline parallelism (PP).

Software	PPL	Surprisal	BPC	Long texts	CLI	API	PP
lmppl	✅	❌	❌	❌	❌	✅	❌
surprisal_from_llm	❌	✅	❌	❌	✅	❌	❌
evaluate	✅	❌	❌	✅	❌	✅	❌
🤔 ppllm	✅	✅	✅	✅	✅	✅	✅

Upcoming metrics (see the roadmap):

• word-level surprisal
• bits per byte (BPB)

🤔 ppllm is benchmarked against:

• a vllm-based implementation: 4.15 times faster!
• a naive hugging face implementation, which does not sort texts by length: 4.61 times faster!

Windowed PPL

Some texts are too long to fit in a model, especially since Transformers have a quadratic complexity (Vaswani et al., 2017, Tay et al., 2023). Windowed PPL restrains the context size to a fixed window as illustrated below (e.g. of 1024 tokens)

Without window (context size may get long)

With window (fixed context size)

In practice, 🤔 ppllm uses a stride of half the window size, instead of the unit stride illustrated here.

(Illustration by https://huggingface.co/docs/transformers/perplexity)

Metrics

All metrics are defined assuming access to a (large) language model that defines a probability distribution over a sequence of tokens $x=(x_1,x_2,\dots,x_L)$:

$P(x) = P(x_1|x_0) P(x_2|x_{<2}) \dots P(x_L|x_{<L})$

Where $x_0$ denotes a special token marking the beginning of the sequence (bos_token in transformers). Note that some models do not have such a token. In this case, the probability of $x_1$ is not taken into account (i.e. we assume that $P(x_1|x_0)=1$).

For numerical stability, we compute the log probability:

$\log P(x) = \log P(x_1|x_0) + \log P(x_2|x_{<2})+ ... + \log P(x_L|x_{<L})$

From this, we can compute the negative log probability (aka negative log likelihood, aka cross-entropy), which is the loss LLMs are trained to minimize (during pretraining):

$$\mathcal{L}(x)=-\log P(x)$$

Then comes surprisal, which is the same but is usually expressed in bits, using a $\log_2$ logarithm:

$$S(x)=-\log_2 P(x)=\frac{\mathcal{L}(x)}{\log(2)}$$

From surprisal, we can define bits per character (BPC), which simply normalizes the surprisal by the number of characters $C$ of $x$:

$$\mathrm{BPC}(x)=\frac{S(x)}{C}$$

Note, in case the model doesn't define $x_0$ (BOS), $C$ does not account for the characters of $x_1$.

Similarly, we define perplexity (PPL), which normalizes the invert probability by the number of tokens $L$, which is equivalent to the exponentiate of the surprisal normalized by $L$:

$$\mathrm{PPL}(x)=\sqrt[L]{\frac{1}{P(x)}}=2^{\frac{S(x)}{L}}=\exp\left(\frac{\mathcal{L}(x)}{L}\right)$$

Likewise, in case the model doesn't define $x_0$ (BOS), we normalize by $L-1$ instead.

Installation

via pip

pip install ppllm

via uv

uv add ppllm

editable

git clone https://github.com/PaulLerner/ppllm.git
cd ppllm
uv sync

Usage

Python

🤔 ppllm is a pythonic library, see the example notebook to see how to use it from python (you can also open it in Colab)

CLI

python -m ppllm /path/to/output /path/to/data --model_kwargs.pretrained_model_name_or_path=meta-llama/Llama-3.1-8B --window=64

Omit --window to compute PPL with the entire context

Use python -m ppllm -h to see all arguments

🤔 ppllm relies on jsonargparse so you can use yaml configs:

>>> python -m ppllm /path/to/output /path/to/data --model_kwargs.pretrained_model_name_or_path=meta-llama/Llama-3.1-8B --window=64 --print_config
output_dir: /path/to/output
data_path: /path/to/data
model_kwargs:
  pretrained_model_name_or_path: meta-llama/Llama-3.1-8B
  config: null
  cache_dir: null
  ignore_mismatched_sizes: false
  force_download: false
  local_files_only: false
  token: null
  revision: main
  use_safetensors: null
  resume_download: false
  output_loading_info: false
  dtype: float16
  load_in_8bit: false
  load_in_4bit: false
  attn_implementation: null
  trust_remote_code: true
window: 64
input_key: text
split: test
tokenizer_kwargs:
  return_tensors: pt
  padding: longest
  truncation: false
  return_overflowing_tokens: false
  max_length: null
loader_kwargs:
  batch_size: null
  num_workers: 4
  pin_memory: false
  drop_last: false
  timeout: 0
  prefetch_factor: null
  persistent_workers: false
  pin_memory_device: ''

>>> python -m ppllm --config=/path/to/config.yaml

TODO describe data input/output formats

• context field

Contributing

Feel free to open an issue or PR to contribute. The roadmap will probably never happen without your help :)

Building

Use:

• uv version --bump patch for 1.2.3 => 1.2.4
• uv version --bump minor for 1.2.3 => 1.3.0
• uv version --bump major for 1.2.3 => 2.0.0

Then

uv build
uv publish --token=<TOKEN>

Tests

python -m unittest tests/test_ppl.py

Benchmark

Setup:

• NVIDIA V100 (32GB)
• Llama-3.1-8B
• wikitext-2-v1

software	compute time in seconds (↓)
vllm	328
hf_shuffle	364
🤔 ppllm (window=128)	108
🤔 ppllm (no window)	79

On Wikitext, because texts are quite short, it's no use computing windowed PPL and directly computing PPL of the full text is faster. However, if texts get longer than 10,000 tokens, a V100 will probably go OOM even with a batch size of 1, so windowed PPL is essential.

Apart from this, we can see that the naive hugginface based-implementation (which does not sort texts by length) is on par with vllm. However, when sorting texts by length as in 🤔 ppllm, we get more than 4 times faster than vllm!