PandasTrace

Debug pandas method chains, step by step.

Ever run a chain like this and get a result that's just... wrong?

df = (df.dropna()
        .merge(other_df, on="user_id")
        .groupby("country")
        .agg({"sales": "sum"}))

You have no idea which step broke it — did dropna() wipe out more rows than expected? Did the merge() silently duplicate rows? PandasTrace watches your chain and shows you exactly what happened at every step, without you writing a single debug print().

The problem

Normally, debugging a chain like the one above means manually breaking it into separate lines, adding print(df.shape) after each one, re-running, then deleting all that debug code once you've found the issue. Every data analyst and ML engineer has done this — it's tedious and easy to skip, which means bugs slip through.

What PandasTrace does

Wrap your DataFrame once, keep chaining exactly like you normally would, and get a clean report of what happened at every step:

from pandastrace import TraceFrame, TraceReporter

result = (
    TraceFrame(df)
    .dropna()
    .merge(other_df, on="user_id")
    .groupby("country").agg({"sales": "sum"})
)

TraceReporter(result.recorder()).print_table()

                                    PandasTrace Report
┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Method        ┃ Shape Before ┃ Shape After ┃ Row Delta┃ Null Changes ┃ Duplicates ┃ Memory   ┃
┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ dropna        │ (8, 3)       │ (6, 3)      │ -2       │ country: -2  │ -          │ -132.0B  │
│ merge         │ (6, 3)       │ (7, 4)      │ +1       │ -            │ +1         │ +210.0B  │
│ groupby().agg │ (7, 4)       │ (3, 1)      │ -4       │ sales: -3    │ -          │ -360.0B  │
└───────────────┴──────────────┴─────────────┴──────────┴──────────────┴────────────┴──────────┘

Row losses show in red, gains in green, new duplicates in red, memory growth in yellow. Notice the merge step here shows Duplicates: +1 — this is PandasTrace catching a real bug: the merge key had a duplicate row on the right side, silently multiplying a row in the result. Also notice groupby().agg shows up as one combined step, spanning all the way back to before .groupby() was called — PandasTrace correctly waits until pandas returns something with a real shape before logging, so nothing gets silently lost mid-chain.

Installation

pip install pandastrace

Quick start

import pandas as pd
from pandastrace import TraceFrame, TraceReporter

df = pd.DataFrame({
    "user_id": [1, 2, 3, 4, 5, 6],
    "country": ["PK", "US", None, "PK", "UK", None],
    "sales": [100, 200, 150, None, 300, 250],
})

traced = TraceFrame(df).dropna().head(3)

TraceReporter(traced.recorder()).print_table()

Need the raw DataFrame back once you're done tracing? Call .unwrap():

clean_df = traced.unwrap()

Want the data as plain Python objects instead of a printed table (e.g. to log it, export it, or feed it elsewhere)?

TraceReporter(traced.recorder()).to_dict_list()
# [{"method": "dropna", "before": (6, 3), "after": (3, 3), "delta": -3}, ...]

How it works

TraceFrame wraps your DataFrame and intercepts every method call you make on it. Before running the real pandas operation, it records the shape; after running it, it records the shape again — then logs the difference. The result is wrapped back into a new TraceFrame, so chaining keeps working exactly like normal pandas.

Current features (v0.3)

• Row count tracking across any chained pandas method
• Null count tracking per column, per step — see exactly which column gained or lost missing values at each stage
• Duplicate row detection — catch bad merges that silently multiply rows before they cause downstream bugs
• Memory usage tracking per step, human-readable (KB/MB/GB)
• Export full trace history to JSON or CSV — useful for logging results from automated/CI pipelines, not just interactive debugging
• Correctly handles multi-call operations like .groupby().agg() as a single combined step, instead of losing track mid-chain
• Works with [...] column indexing as well as method chaining
• Colored terminal output (red = rows/data lost or new duplicates, green = shrinking/cleanup, yellow = memory growth)
• Zero changes needed to your existing pandas code — just wrap the DataFrame

Roadmap

• VS Code extension — see diagnostics inline in your editor as you write the chain

Contributing

Issues and PRs welcome. This is an early-stage project — if you hit a pandas method that doesn't behave correctly under tracing, please open an issue with a minimal reproduction.

License

MIT