paneldx

Check that your longitudinal dataset is what it claims to be, before you model it.

A panel dataset asserts that each row belongs to an entity observed repeatedly over time. That assertion is usually taken on faith. When it is wrong, because rows were joined by position or by an ID that is not actually stable, every within-entity quantity computed afterwards is quietly meaningless: lags, differences, trajectories, sequence models, grouped cross-validation splits.

Nothing errors. The numbers just stop meaning anything.

paneldx audit data.csv --time quarter --target risk_score --html report.html

Exits non-zero when it finds a defect that invalidates within-entity analysis, so it can gate a pipeline or a CI job.

The bug this was built from

A published physician-analytics pipeline built its entity ID from row position:

physician_id = ((serial_number - 1) % rows_per_quarter) + 1

Each quarter was sorted by platform rank before this ran, so position i was a different doctor every quarter. The ID linked strangers together, and that ID then fed momentum features, a GRU over "trajectories", and the grouped CV splits.

paneldx on that dataset, given no hints:

key: physician_id                      key: Disease + Opening time
  columns explained  2 of 30  (7%)       columns explained  13 of 29  (45%)
  VERDICT  NOT SUPPORTED                 VERDICT  supported by the data

Blind search over every one- and two-column combination ranked the correct key first. That was one of three defects. The other two turned up on the same dataset without being told anything about it:

The defects hide each other

A broken key does not only corrupt features. It disarms the check that would have caught it:

Under the fabricated key the naive forecast looks worthless, so nobody thinks to compare against it. Under the real key it is close to unbeatable. This is why paneldx establishes the key before it reports any baseline, and why running the baseline alone would not have saved that paper.

Install

Not on PyPI yet. Install from source:

pip install git+https://github.com/stemmatics/paneldx.git

Python 3.9+, numpy and pandas. Nothing else: no modelling framework, no template engine.

Usage

from paneldx import audit, to_html
from paneldx import validate_key, discover_keys
from paneldx import detect_counters, target_leakage, persistence_baseline

# Is the entity key real?
print(validate_key(df, "patient_id", time_col="quarter"))
print(discover_keys(df, time_col="quarter")[0])       # no hints needed

# Are the numbers about to fool you?
print(detect_counters(df, "patient_id", "quarter"))   # lifetime totals
print(target_leakage(df, target="risk_score"))        # is y inside X?
print(persistence_baseline(df, "patient_id", "quarter", "risk_score"))

# Or all of it at once
result = audit(df, "quarter", key="patient_id", target="risk_score")
open("report.html", "w").write(to_html(result))

The CLI reads CSV, TSV, Excel, Parquet, Feather and JSON:

paneldx audit panel.xlsx --time t --key site_id patient_id --target outcome

The three traps

Cumulative features. Lifetime totals barely move between periods, so a target built from them is near-perfectly autocorrelated. Models trained on them report excellent R² for restating what they were handed. Difference them into per-period flows.

Target composition. When the target is computed from columns that are also features, the model is not predicting, it is recovering arithmetic. The metrics look superb and mean nothing. target_leakage fits a deliberately linear model: the point is not to predict the target well, but to show no prediction was ever required.

Missing naive baseline. On autocorrelated panels, carrying the last value forward is often unbeatable. A model reported without it may be losing to a one-line rule, and no reader could tell.

How key validation works

A correct entity key reveals structure that an incorrect one cannot.

Invariants. Some attributes cannot change for a real entity: a birth date, an account opening timestamp. Under the correct key they are constant within an entity. Under a wrong key they flicker.

Counters. Some measures only accumulate. Under the correct key their within-entity differences are non-negative. Under a wrong key they wander.

Neither is known in advance, so paneldx does not ask you to declare them. It measures how much of this structure a candidate key reveals, and compares that against a null: the same key with entity labels shuffled within each period. That preserves the panel's exact shape while destroying any true correspondence. A real key scores far above its null; a fabricated one scores at it.

The verdict uses the share of columns explained, not the count. An artifact of row ordering explains only the handful of columns that drive the ordering, while a genuine key explains most of the table. Counting raw columns passes the fabricated key; counting the share rejects it.

Limitations

• Near-valid keys are hard to separate from perfect ones. The tolerances exist because real keys collide too, so a key that accidentally merges a couple of entities can score like a clean one. Ties break toward the finer partition, which mitigates but does not eliminate this.
• Leakage detection is linear. A target computed from its features by some non-linear rule can slip past target_leakage. A high score is strong evidence; a low one is not a clearance.
• Two-column search is O(n²) in columns. On a 24k x 31 table the full pair search takes about 23 seconds. Pass --key when you already know it.
• A passing verdict is not a proof. It says the data is consistent with the key, not that the key is correct. Domain knowledge still wins.
• Needs at least two periods, and enough entities to measure a rate against.

Roadmap

• Panel key discovery and validation
• Cumulative-counter detection
• Target-composition leakage
• Naive-baseline harness
• One-command HTML report
• CLI
• Look-ahead detection in sequence construction
• Faster key search on wide tables

Documentation

• Getting started
• How it works
• Interpreting results
• API reference

Contributing

See CONTRIBUTING.md. False positives are bugs: if paneldx rejects a key you know is correct, that is worth an issue.

Release notes are in CHANGELOG.md.

Citation

If you use paneldx in research, see CITATION.cff or use the "Cite this repository" button on GitHub.

License

Apache License 2.0. See LICENSE and NOTICE.