pandatrace

Provenance tracking for pandas pipelines — what each operation did to your frame, which columns it created, and where the nulls came from.

The goal is that after a long chain of transformations you can trace a frame back through its whole history, even across however many intermediate variables you bound along the way.

TracedDataFrame automatically records a whitelisted set of operations — both df = df.sort_index() reassignment and df.pipe(f).sort_index()... chaining are tracked correctly into one shared ledger. Indexing (.loc, .iloc, df[...]) and direct mutation aren't covered — see Limitations for the honest list.

Installation

Python 3.11+, pandas 2.2+. Fully typed.

pip install pandatrace

Usage

TracedDataFrame is a pd.DataFrame subclass — use it like a normal DataFrame, and a whitelisted set of operations get recorded automatically, whichever style you write them in:

import pandas as pd
from pandatrace import TracedDataFrame

orders = TracedDataFrame(
    {
        "id": [1, 2, 3, 4],
        "price": [10.0, 20.0, None, 40.0],
        "qty": [1, 2, 3, 4],
    },
    source="orders.csv",
)
customers = pd.DataFrame({"id": [1, 2, 3], "region": ["eu", "us", "eu"]})

result = orders.merge(customers, on="id", how="left").assign(
    total=lambda d: d["price"] * d["qty"]
)

for step in result.ledger.steps:
    print(
        f"{step.index}. {step.op:8} rows={step.row_delta} "
        f"new={step.new_columns} nulls={step.nulls_introduced()}"
    )

0. read     rows=None new=() nulls={}
1. merge    rows=0 new=('region',) nulls={'region': 1}
2. assign   rows=0 new=('total',) nulls={'total': 1}

The left join brought in a region that is null for the unmatched order, and the total inherited the null from price — both attributed to the step that introduced them, with no manual bookkeeping. This works identically whether you write it as one chain (above), reassign at every step (orders = orders.merge(...); orders = orders.assign(...)), or split the chain across .pipe() calls — the same Ledger is shared across every frame descended from the same original, however many intermediate variables you bind or leave unbound. You can then trace a column back through the history:

[s.op for s in result.ledger.origin_of("region")]  # ['merge']
[s.op for s in result.ledger.origin_of("total")]   # ['assign']

result.ledger.parents_of("total")  # () -- deps are declared, not inferred

Note that last line: the automatic capture records which step created total, but not which columns fed it — nothing inspects the lambda. parents_of only answers for dependencies you declare via Ledger.record(..., deps=...) (shown below); see Limitations.

Ledger.report() renders the whole history as a plain-text table — one line per step, with row deltas, column churn, and the nulls each step introduced — and in a notebook a bare ledger at the end of a cell renders the same table as HTML. Ledger.to_json() exports it (plain ints only, nothing numpy-shaped) for logging or diffing between runs:

print(result.ledger.report())
result.ledger.to_json(indent=2)

Ledger.ancestry() renders the same history as a pipeline-flow graph — plain SVG, no plotting dependency: one node per step, dashed stubs branching in where a merge/join/concat combined in another traced frame, and an optional column highlight showing exactly which steps origin_of attributes to it. In a notebook, a bare graph renders inline; .show() displays it from anywhere in a cell:

result.ledger.ancestry(column="total")   # last expression of a cell
result.ledger.ancestry().show()          # or explicitly, mid-cell

Catching problems

check() scans the history against a policy and returns typed findings — large row drops, introduced nulls, silent column loss (an explicit drop is exempt), and steps that emptied the frame:

result.ledger.check()
# [<Finding nulls: step 1 merge introduced 1 null (region: 1) @ ...>,
#  <Finding nulls: step 2 assign introduced 1 null (total: 1) @ ...>]

result.ledger.check(row_drop_pct=25)   # tighten a threshold
result.ledger.check(min_nulls=None)    # None disables a check

warn_on() does the same evaluation live: every recorded step is checked as it happens, and each finding is emitted as a PandatraceWarning — a real Python warning the stdlib warnings machinery can filter, or escalate to an error in CI:

import pandatrace

pandatrace.warn_on(row_drop_pct=25)   # from now on, suspicious steps warn
pandatrace.warn_off()                 # back to the default silence

Ledger and its supporting types are also usable directly, for anything outside the tracked whitelist:

from pandatrace import Ledger

ledger = Ledger(df, source="orders.csv")
ledger.record("custom_op", before_df, after_df, deps={"total": ("price", "qty")})

The layers

Limitations

Worth knowing before you reach for this:

• Only a whitelisted set of operations is tracked. TRACKED_OPS in pandatrace.traced covers pipe, query, assign, rename, drop, dropna, fillna, drop_duplicates, sort_values, sort_index, reindex, reset_index, head, tail, sample, merge, and join, plus a pandatrace.concat shim for the module-level pd.concat. Anything else that returns a new frame (groupby, apply, pivot, …) behaves like normal pandas but records nothing — the ledger's Snapshot/Step won't lie about what happened, but it will have a gap.
• Indexing isn't tracked. .loc, .iloc, df[...], and boolean masking all fall outside the wrapped whitelist — column subsetting is a real provenance event this doesn't see yet.
• Direct mutation is invisible. traced_df["x"] = 1 bypasses every hook; there's no __setitem__ interception.
• inplace=True is accepted but silently untracked, not rejected. This is deliberate, not an oversight: pandas' own internals rely on calling tracked methods with inplace=True as plumbing (merge() strips duplicate join-key columns via an internal drop(inplace=True)), so rejecting it outright breaks merge/join from the inside. A step needs a before snapshot taken before the mutation happens, which an in-place call doesn't give a clean opportunity for — so it's recorded as nothing rather than guessed at.
• merge/concat across two independently-ledgered frames keep only one history. The first/left operand's ledger survives; the other operand's step count is noted in the recorded step's detail so the join stays auditable, but its full history isn't merged in.
• Dependencies are declared, not inferred. deps={"total": ("price", "qty")} is something you assert. Nothing verifies it against what the operation really did, so a stale deps will quietly misreport provenance.
• Null counts are addressed by name, and duplicate labels defeat that. concat(axis=1), a colliding rename, and labels that differ to pandas but collide under str() all map several real columns onto one key. Counts are folded together rather than one silently overwriting another, and Snapshot.duplicated_columns tells you which labels that happened to — but the per-column figure is genuinely lost, not recoverable.
• Snapshots are structural only. Row counts, column labels, and null counts. No dtypes, no index information, no value-level diffing.

Development

Managed with uv: uv sync, then

scripts/test.sh                     # pytest, args passed through
scripts/coverage.sh                 # suite under coverage (gate: 90%)
scripts/lint.sh                     # ruff check + ty check (both clean)
scripts/docs.sh                     # sphinx with -W; also executes the example notebook
scripts/check.sh                    # all of the above, the pre-merge gate

Docs build clean and are built with -W so warnings fail — keep it that way. CI enforces the full test matrix (Python 3.11–3.14 × pandas 2.2/2.3/3.x) plus all of the gates above on every push.

Longer prose docs, including the design invariants, live in docs/ (index.rst for the narrative, api.rst for the autodoc API reference).