popfidelity

A language model that answers survey questions can match the average answer of a population and still erase the differences between the people in it. popfidelity measures how well a model's answers represent a human population, group by group. One Rust core gives a Python package, an R package and a command line the same numbers.

What it measures

The Population Fidelity Score (PFS; da Silva et al., 2026) compares the survey and the model on cells, the subpopulations a survey reports, and checks three conditions:

PFS is their geometric mean, so a model fails as soon as one condition fails. It is reported for the pooled population and every demographic subgroup, for paired comparisons of a tuned and a base model, and across replicate runs and questions, with the paper's nine sensitivity specifications.

Beside PFS sits a registry of 126 measures from the literature on synthetic survey samples, each with the work it comes from and the work that applied it to language models; 66 ship in this release:

The measure catalogue lists all 126, including those planned for later releases, and 17 benchmarks.

Install

pip install popfidelity

remotes::install_github("neemiasbsilva/popfidelity", subdir = "r")

cargo add popfidelity

PyPI has wheels for Linux, macOS and Windows on Python 3.11 and newer, so pip needs no Rust toolchain there. The R package compiles the Rust core, so it needs Cargo and rustc 1.85 or newer, as does a Python install from source. The Python extras api (OpenAI, Anthropic, Google clients), local (transformers) and parquet add optional backends and formats.

Quickstart

Python:

import popfidelity as pf
from popfidelity.examples import toy
from popfidelity.io import read_cells

cells = toy()
scores = pf.score(cells["survey"], cells["M2"], pf.Variant(round_pairs=10))
print(scores["pfs"], scores["binding_component"])

questions = pf.read_questions("examples/ces/questions.yaml")
survey = read_cells("examples/ces/data/survey_cells.csv", questions)["ideo5"]
results = "examples/ces/results/ollama-qwen3-vl-2b/model_cells.csv"
model = read_cells(results, questions, mode="ntp")["ideo5"]
facets = pf.read_facets("examples/ces/facets.yaml").from_labels(survey.labels)
groups = pf.score_groups(survey, model, facets)

R:

library(popfidelity)
toy <- example_toy()
pfs_score(toy$survey, toy$M2)$pfs

questions <- read_questions("examples/ces/questions.yaml")
survey <- read_cells("examples/ces/data/survey_cells.csv", questions)$ideo5
results <- "examples/ces/results/ollama-qwen3-vl-2b/model_cells.csv"
model <- read_cells(results, questions, mode = "ntp")$ideo5
groups <- pfs_score_groups(survey, model, read_facets("examples/ces/facets.yaml"))

Command line:

popfidelity elicit --config examples/ces/elicit/ollama.yaml --limit 8
popfidelity aggregate records --config examples/ces/elicit/ollama.yaml --out model_cells.csv
popfidelity score --survey examples/ces/data/survey_cells.csv --model model_cells.csv \
    --questions examples/ces/questions.yaml --facets examples/ces/facets.yaml --out scores
popfidelity report --groups scores/groups.csv

Asking models

popfidelity elicit interviews a model once per respondent and question, with the respondent's demographics as prior turns, and stores resumable JSON-lines records. It reads next-token probabilities over the answer options where the endpoint returns them and samples full answers otherwise.

Examples

On the CES, the local model is close to every cell on average but does not order the cells as the survey does, so structure binds and PFS stays at 0.30 or below. On Twin-2K-500, people answering the same items again reach PFS 0.95 and the digital twins 0.58 to 0.66.

Reproducibility

• Python and R give the same tables: ./run.sh parity runs both on shared inputs and compares 19 tables at 1e-12.
• The library reproduces the paper: ./run.sh crosscheck recomputes its 9,548 subgroup rows and 5,952 paired rows with differences of at most 4.4e-16.
• Bootstrap and permutation draws are seeded one by one, so results do not depend on the number of threads.
• Every reference in docs/references.bib is checked against Crossref and arXiv by scripts/check_references.py.

Development

uv sync --locked --group dev --extra api --extra parquet
./run.sh test     # Rust and Python tests
./run.sh lint     # rustfmt, clippy, ruff, mypy strict, style check
./run.sh r        # R CMD check --as-cran
./run.sh parity   # Python against R

See CONTRIBUTING.md for the conventions.

Citation

TODO - Demo Paper

@misc{dasilva2026population,
  author = {da Silva, Neemias B. and Lukk, Martin and Sutani, Ali and Moturu, Abhishek and
            Yang, Harris and Silver, Daniel and Ratto, Matt and Silva, Thiago H.},
  title = {Population Fidelity: Evaluating Population Representativeness in {LLMs}},
  year = {2026},
  eprint = {2609.36253},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL}
}

Please also cite the works behind the other measures you report; the registry names them. The software citation is in CITATION.cff.

Licence

Apache License 2.0; see LICENSE and NOTICE.