mendr 0.4.0

Bounded, auditable, reversible data repair — the layer that runs after your validator.

mendr

Bounded, auditable, reversible data repair — the layer that runs after your validator and turns "here's what's wrong" into "here's the fix, with receipts, and an undo."

Status

v0.4.0 — release-ready, upload pending. The full loop works: repair → ledger → report → revert, on pandas and polars, with pandera and Great Expectations adapters, cross-column rules, quarantine, a CLI (including mendr diff for run-to-run drift), CI repair budgets, and benchmarks (benchmarks/: ~200k repairs on a 1M-row frame in seconds, exact revert included). The ledger file format is a documented, stable contract (docs/ledger-format.md), ledgers are tested to revert across backends (pandas-written ledger → polars frame and vice versa), and columns mendr can't faithfully record are refused loudly at ingest instead of silently misbehaving later. Docs live in docs/, a ten-row walkthrough in examples/one_ledger_two_uses.py, a 195k-row real-data showcase in examples/nyc311_bounded_repair.py, a 2.96M-row parquet showcase in examples/nyc_taxi_bounded_repair.py (480k repairs, a live budget refusal, byte-identical pandas/polars ledgers), the roadmap in TODO.md, and release history in CHANGELOG.md.

The gap

Validators (pandera, Great Expectations) are good at telling you what's wrong and then stopping. "Cleaning" tools are good at silently fixing things you can't inspect or undo. Almost nobody owns the middle: repair that is declared, bounded, and fully auditable.

That middle is the whole point of mendr. Every change it makes is a named action you asked for, constrained by a limit you set, recorded in a ledger you can read, diff, and reverse. No magic. Nothing changes that you didn't authorize.

mendr is explicitly not a validator. It doesn't compete with pandera or GE — it runs after them (or after its own minimal built-in checks) and does the one thing they don't: repair you can trust in production.

The core idea: one ledger, two uses

Every repair mendr makes is recorded as an atomic entry — which rule fired, which row and column, the old value, the new value, and why.

That single artifact is both halves of the product:

• Read it forward → it's the audit report. Every change, explained.
• Replay it inverted → it's the undo log. .revert() round-trips the frame back to where it started.

Reversibility isn't a second bookkeeping system kept in sync with the report — it falls out of the report. The record is the product.

Quick look

import pandas as pd
import mendr as md

df = pd.read_csv("messy.csv")

policy = md.Policy([
    md.Rule(
        column="age",
        constraint=md.InRange(0, 120),
        remediation=md.Clip(),
        bound=md.MaxFraction(0.05),      # repair at most 5% of rows, else fail
    ),
    md.Rule(
        column="signup_date",
        constraint=md.IsType("datetime"),
        remediation=md.Cast(),
        bound=md.MaxRows(50),
    ),
    md.Rule(
        column="email",
        constraint=md.NotNull(),
        remediation=md.DropRow(),
        bound=md.MaxFraction(0.10),
        reason="email is required downstream",
    ),
])

result = md.repair(df, policy)

result.frame                     # the repaired dataframe
print(result.report())           # human-readable summary of every change
result.ledger.to_json("repairs.json")

original = result.revert()       # round-trips exactly back to df

If a rule would need to repair more than its bound allows, mendr stops — it does not silently over-repair. Bounded means bounded.

The same loop runs from the command line, budget and all:

mendr check  data.csv --policy policy.json           # exit 3 if violations exist
mendr repair data.csv --policy policy.json \
    --out repaired.csv --ledger repairs.json \
    --quarantine quarantined.csv --max-repairs 100
mendr revert repaired.csv --ledger repairs.json      # undo, from the ledger file alone
mendr diff   yesterday.json repairs.json             # exit 4 if repair activity drifted

Concepts

Rule — a constraint + a remediation + a bound. The constraint says what must be true (InRange, IsType, NotNull, OneOf, MatchesPattern, Unique, MaxLength, cross-column ColumnCompare, …); the remediation is the single named action taken when it isn't (Clip, Cast, Fill, Replace, Truncate, Round, Lowercase, NormalizeWhitespace, DropRow, Quarantine); the bound caps how much repair is permitted before mendr refuses and raises.

Policy — an ordered set of rules applied to a frame.

Ledger — the append-only record of every atomic change. Cell edits store (rule_id, row_id, column, old, new, reason). Row drops store the whole row so a revert can re-insert it — with a destination marker that distinguishes deleted rows from quarantined ones (moved to a side frame, result.quarantine, for human review instead of being nulled or deleted). This one structure is the audit report and the undo log — and mendr diff compares two of them to catch run-to-run drift in CI.

Row identity — mendr owns row identity. It assigns a stable row id at ingest (positional, or a key column you declare) and the ledger references that, never the native pandas index. This keeps the ledger portable and backend-agnostic — the same audit format works whether the frame came from pandas or polars.

Backends — the engine never touches a dataframe library directly. It talks to a small adapter interface (get column, cast, drop row, set cell, materialize). pandas and polars both ship; same data + same policy produce a byte-identical ledger on either. Your data stays in its native format — no forced conversion, no dtype surprises on ingest.

Validators — pluggable. The built-in constraints cover the basics; Satisfies / SatisfiesRow wrap any predicate; mendr.integrations.pandera turns a pandera.Check — or a whole DataFrameSchema — into mendr constraints, and mendr.integrations.gx does the same for Great Expectations (one expectation, or a whole ExpectationSuite). Your validator stays the authority on what's wrong; you declare what mendr may do about it.

Design principles

Deterministic. Same frame and same policy produce the same changes every time.

Bounded. Every remediation is capped. Exceeding the cap is a hard stop, not a silent decision.

Legible. Every change is a named action with a reason, recorded in a format a human can read and a CI job can diff.

Reversible. Any repair run can be undone from its ledger alone.

Non-goals (for the initial release, by design)

• No LLM-based repair. The trust wedge is that you can predict exactly what mendr will do.
• No ML imputation models. Fills are declared strategies (constant, forward-fill, median-with-flag) — not learned guesses.
• No fuzzy / "smart" cleaning. If it isn't a named, bounded action, mendr doesn't do it.
• Not a validator. mendr repairs; it leans on validators (built-in or pluggable) to decide what's wrong.

LLM-assisted rule suggestion — proposing a policy for a human to approve — is a possible far-future feature, and would live strictly outside the repair path. Never in it.

Where it fits

mendr is a sibling to agentrec — same ethos, different domain. agentrec records LLM API traffic so CI can catch regressions; mendr records data repairs so humans and CI can trust them. Both make the record the product, both are git-diffable, and both are built to fail a build when something drifts past a declared budget.

Install

Not on PyPI yet (release prep is done — see TODO.md). From source:

git clone https://github.com/Pi-Wi/mendr && cd mendr
pip install .[pandas]        # or .[polars], .[pandera], .[gx], .[dev]

The core has zero runtime dependencies; backends bring their own library.

License

MIT.