ltm-eval 0.1.0

Evaluation for large tabular models (LTMs), generative models of tabular data, with calibrated uncertainty.

ltm-eval

Evaluation for large tabular models (LTMs), generative models of tabular data, with calibrated uncertainty.

Most evaluators of generative tabular models report a single number — a Shape score of 0.92, a precision of 0.97 — and leave you guessing. Is that good? Is it distinguishable from noise? Would it change at a different sample size? A bare value can't say.

ltm-eval reports every metric as three quantities (see docs/methodology.md for the approach and the exact algorithm, and how the code computes them):

• value — the metric computed on your data;
• confidence interval — a two-sided bootstrap interval, so two models that differ only by sampling noise don't look different;
• ideal value — what two independent samples of the real holdout score against each other at this sample size. This, not a metric's theoretical best score (e.g. 1.0), is the achievable target: with finite samples even a perfect model can't reach the theoretical best.

Together, these outputs allow you to interpret a metric finite-sample effects instead of against an out-of-reach perfect score. Every result also carries reliability flags, so the library never reports a number without the context to trust it.

Holdout and generated sets. Every evaluation takes two inputs: a holdout set — real data your model never saw in training, the stand-in for "real" — and a generated set — your model's synthetic output. Each metric scores the generated set against the holdout, while the ideal value is computed from the holdout alone (using two independent sub-samples, scored against each other).

Installation

pip install ltm-eval

The core depends only on numpy, pandas, scipy, scikit-learn, and joblib. It is CPU-only; there is no torch or CUDA in the default path.

Quickstart

Run the bundled example — it needs no data of your own. It pulls a real dataset from OpenML and scores a stand-in generator against the held-out real split:

python examples/openml_quickstart.py        # defaults to the "adult" dataset

It prints each metric's value beside its ideal value:

adult (OpenML id 179): train=24421 rows, holdout=24421 rows, 15 columns
[tutorial] generated = noised resample of train (not a trained model), so it should score below the ideal value.
config: n_bootstrap=50, max_eval_n=1000, seed=0 (max_eval_n is set only to keep this demo fast: it caps the per-side sample size to 1000, so each metric is computed on <= 1000 rows/side; the header's 'inputs' are the raw row counts)
ltm-eval report   (inputs: holdout=24421, generated=24421 rows; each metric computed on 1000 rows/side; B=50, 95% CI)

Sanity (Data Validity)
  NaN Cell% (Syn)        0.008441 —
  NaN Row% (Syn)         0.070964 —
  ...

Fidelity (Distribution Similarity)
  Shape                  0.9711 [0.9645, 0.9771]   ideal 0.9630 [0.9532, 0.9676]
  Trend                  0.9106 [0.9014, 0.9181]   ideal 0.9229 [0.9099, 0.9304]

Diversity (PRDC — Manifold Coverage)
  Precision              0.9261 [0.9041, 0.9489]   ideal 0.9194 [0.8955, 0.9382]
  Recall                 0.9048 [0.8855, 0.9221]   ideal 0.8956 [0.8751, 0.9202]
  ...

Conditional (Subpopulation Fidelity)
  Cond Shape             0.8928 [0.8823, 0.8991]   ideal 0.8875 [0.8788, 0.8957]
  Cond Shape (Rare)      0.8263 [0.8125, 0.8368]   ideal 0.8143 [0.8009, 0.8300]
  ...

flags (* marks an unreliable value):
  CATEGORY_VOCAB_MISMATCH — the generated set has categories absent in the holdout (or vice versa)

Each row is one metric: its value (with CI) and the ideal value (with CI).

Use it on your own data

evaluate(holdout, generated) takes two pandas.DataFrames sharing the same columns; row counts need not match:

import pandas as pd
from ltm_eval import evaluate, Schema

holdout   = pd.read_csv("real_holdout.csv")   # an unseen real holdout — the proxy for "real"
generated = pd.read_csv("generated.csv")      # your model's output; row count need not match

schema = Schema(numeric=[...], categorical=[...])   # optional; inferred and recorded if omitted
report = evaluate(holdout, generated, schema=schema)

print(report.summary())          # console table (the format shown above)
report.to_dataframe()            # tidy long form for analysis
report.to_markdown("eval.md")    # report table
report.to_json("eval.json")      # round-trippable

The unseen holdout is what the ideal depends on: train your model on real train, then score generated against a holdout it never saw. load_openml hands you that split directly, so you can try the workflow on real data without your own files:

from ltm_eval import evaluate, load_openml

train, holdout = load_openml("adult")     # deterministic (train, holdout) split of real data
# ... train your model on `train`, sample `generated`, then: evaluate(holdout, generated) ...

Metrics (v1)

All v1 metrics are sample-based — a statistic of the samples, so a bootstrap replicate is just "resample and recompute".

Metric	Category	What it measures
Shape	fidelity	per-column marginal similarity — KS complement (numeric), total-variation complement (categorical)
Trend	fidelity	pairwise-association similarity over column pairs — correlation similarity (numeric–numeric), contingency similarity otherwise
PRDC	diversity	k-NN manifold overlap of the two point clouds: Precision, Recall, Density, Coverage
Conditional	conditional_fidelity	Shape, Trend & Recovery within subpopulation slices, averaged equally per slice (so rare slices count), reported over all slices and rare-only
Sanity	sanity	always-on data-validity checks: NaN / duplicate rates, size ratio, unseen-category count

The statistical primitives (KS complement, TV complement, correlation similarity, contingency similarity) are implemented directly on numpy/scipy/scikit-learn.

What the library guarantees

1. If inputs pass validation, you get a Report. The library does not crash on data content.
2. Every number carries a CI (or a flag explaining its absence) and, where meaningful, an ideal value (computed on the holdout) at matched sizes — so the value and the ideal are directly comparable.
3. Results are deterministic given (inputs, schema, config, seed).
4. A column's role is never silently coerced, and quantities are never compared at mismatched sample sizes, without a flag recording it.

The boundary between user error and data quality is explicit: malformed input (wrong type, mismatched columns, missing target) raises with an actionable message; degenerate but valid data (too few rows, constant columns, unseen categories) produces a flagged result. The full catalogue of handled failure modes is in docs/failure-modes.md.

Documentation

Humans: start with this README, CONTRIBUTING.md, and docs/methodology.md. AI agents: the rest of docs/ (usage, architecture, failure modes) is the detailed exposure; read it before you reason about the code.

• docs/methodology.md — the approach: probability metrics, bootstrap uncertainty, and the ideal value.
• docs/usage.md — inputs, schema, the CLI, configuration, and data cleaning.
• docs/architecture.md — how evaluate() flows through the modules; repository layout.
• docs/failure-modes.md — the catalogue of statistical and "in the wild" failure modes, each with a covering test.
• tests/README.md — the test layers (unit → property → metamorphic → calibration → fuzz → perf) and how to run them.

Citation

If you use ltm-eval in your work, please cite:

@misc{ltm-eval,
  title   = {Large Tabular Models: The Next Frontier},
  author  = {Granica Research Team},
  year    = {2026},
  publisher = {GitHub},
  journal = {GitHub repository},
}

Contributing

Contributions are welcome — see CONTRIBUTING.md for dev setup, the test layers, and how to add a metric.

Project status

0.x, API stabilising. Versioned with SemVer; licensed under Apache-2.0.

Changelog

Release notes are in CHANGELOG.md, following Keep a Changelog + SemVer.