onceover

Profile a Polars DataFrame with one call. onceover returns column summaries, ranked findings with evidence, and coverage describing what each pass examined. The computation runs in Rust; Python bindings release the GIL while profiling.

Documentation: the full book — tutorials, case studies, per-pass methodology, implementation notes, and the report contract — is built from this repository and published at https://fcela.github.io/onceover/. The analysis notebook is a complete worked example with charts and controls.

Install

pip install onceover

Requires CPython 3.10+ and Polars 2.x. Binary wheels cover Linux x86_64 and aarch64, macOS Intel and Apple silicon, and Windows x86_64. Building the source distribution requires Rust and a C compiler.

Python

import polars as pl
import onceover

df = pl.DataFrame({
    "order_id": [1, 2, 3, 4],
    "quantity": [2, 1, 4, 3],
    "double_quantity": [4, 2, 8, 6],
})

report = onceover.profile(df)
print(report.overview["headline"])
print(report.to_markdown())

report.columns     # Per-column types, roles, counts, and statistics
report.findings    # Selected statements with supporting evidence
report.coverage    # Pass status, samples, caps, and skipped work
report.groups      # Connected findings, without a redundancy claim
report.data        # Complete JSON-compatible report
report.save("profile.json")

profile(df, budget="standard", deadline_seconds=None) accepts three budgets:

Results are deterministic for the same data and configuration. An optional wall-clock deadline can change which passes finish; coverage records skipped work.

Findings

• Primary, composite, and nearly unique keys; duplicated rows.
• Exact and approximate functional dependencies, with violation counts.
• Predictable columns, cross-validated decision tables, and PRISM rules.
• Pearson, Spearman, and Chatterjee's ξ dependence; redundant column groups.
• Arithmetic identities, M5-style local linear regimes, and dated residual shifts.
• Nominal associations with training and diagnostic holdout evidence.
• Date gaps, irregular intervals, sortedness, constants, missing values, concentration, numeric category codes, and inconsistent string formats.

Finding categories describe the discovered pattern, while contributing methods are recorded in evidence. Scope, measurement basis, and validation are separate fields. Equivalent statements combine evidence; shared selection limits apply across detectors. See the report schema. Model explanations and associations use diagnostic validation; they do not establish causal effects or multiple-testing guarantees.

Rust

cargo add onceover polars@0.55

use onceover::{profile, ProfileConfig};
use polars::prelude::*;

fn main() -> PolarsResult<()> {
    let df = df!("id" => [1i64, 2, 3], "value" => [4.0, 5.0, 6.0])?;
    let report = profile(&df, &ProfileConfig::default());
    println!("{}", report.to_json());
    Ok(())
}

The Rust API uses Polars 0.55. The onceover crate re-exports report types.

Large and wide tables

The profiler accepts in-memory DataFrames with millions of rows and hundreds of columns. Column summaries examine every row and retain exact cardinality, null counts, distributions, and quantiles. Dense and nullable Int64/Float64 columns borrow Arrow buffers; other dtypes can require normalized copies. This is an in-memory profiler, so input size and exact statistics still affect memory and runtime.

Relationship discovery uses broad screening followed by bounded follow-up work:

Predictor selection uses training rows. Nominal associations consider all eligible nominal columns. Arithmetic identities are fitted on a sample and their residuals are checked over all pair-complete input rows. Single-column key checks use exact full-column cardinality. Higher-order searches, category levels, and explanation output have additional budgets, described in report.coverage.

Samples use deterministic stratified row selection. They can miss rare patterns; shortlisting can miss weak or interaction-only signals. Coverage records samples, shortlists, and skipped work. Absence of a finding is not proof of independence.

Adjust the main relationship costs when needed:

report = onceover.profile(
    df,
    sample_rows=50_000,
    max_predictors=96,
    max_pairs=2_000,
)

These overrides keep exact column summaries and integrity checks on the full input. A deadline is soft: it skips later passes without interrupting a pass already running.

See the analysis notebook for planted examples, Altair charts, evidence, coverage, and a wide-table demonstration. Install its optional dependencies with pip install 'onceover[notebook]'. The scale benchmark measures runtime, peak resident memory, and recovery of relationships planted in the final columns:

pip install 'onceover[benchmark]'
python benchmarks/scale.py --rows 2000000 --columns 200 --output /tmp/onceover-scale.json

Development

python -m venv .venv
source .venv/bin/activate
pip install 'maturin>=1.15,<2' pytest scipy
maturin develop --release
cargo test --workspace --locked
pytest tests/

Publishing setup is described in RELEASING.md.

License

MIT. See LICENSE.