pipelie

Finds the bugs that leave your pipeline reporting green while the numbers are wrong.

pip install pipelie

import pipelie
print(pipelie.audit(df, target="built", key=["id"]))

Or in CI, where it belongs:

df = pipelie.guard(df, target="built", key=["id"])   # raises on anything critical

Why this exists and not dbt tests

Schema tests, null counts and row counts answer "is the data there?"

They do not answer "does the data mean what it is about to be used to mean?"

Every check in this package corresponds to a bug that shipped to production while the tests passed. Not hypothetical failure modes — things that actually happened, were written up, and cost real work:

Each of those produced a plausible-looking number that was wrong. That is the failure mode this package is for.

What it checks

informative_missingness is Bonferroni-corrected across the columns it tests, so a wide table does not manufacture a finding.

What it looks like

examples/a_wrong_number.py builds a customer table with five ordinary bugs in it, none of which any schema test can see. Run it and the first thing printed is the reassuring part: no nulls, correct dtypes, plausible ranges. Everything passes. That is the moment the number gets shipped.

Then:

CRITICAL clock_in_disguise [churn_risk]
    every value falls inside the Unix epoch range for seconds.
CRITICAL informative_missingness [support_note]
    whether this is missing predicts 'churned'.
    evidence: churned|missing=0.000  churned|present=1.000  z=-65.6
CRITICAL duplicate_rows
    300 of 4,300 rows (7.0%) are exact duplicates.
CRITICAL parse_carnage [signup_date]
    holds 2 different date formats in one column.
WARNING  placeholders [credit_limit]
    12.1% of values are exactly -999, a conventional 'no data' marker.

What each one costs:

• The "highest-risk customers, call these first" list is the five most recent signups. The risk score was never computed.
• Drop the rows with a missing support note -- a standard cleaning step -- and the churn rate goes from 50% to 100%.
• The reporting job ran twice on Tuesday, so every total is 7% too high.
• Average credit limit was reported as 4,275. Every real limit is 5,000.
• Half the signup dates would silently become missing.

A real example

Four US grid operators publish their interconnection queues. Merged into one frame and audited with no configuration:

CRITICAL parse_carnage [Queue Date]
    holds 4 different date formats in one column. Parsed together, one format
    wins and the rest become NaT -- which is indistinguishable from data that
    was simply absent.
    evidence: formats=4  examples={'2025-10-08T00:37:52+00:00': '40%',
              '2003-11-18 08:00:00': '24%', '2008-01-30': '19%',
              '1/14/2025': '18%'}  values=9,640

CRITICAL duplicate_rows [Queue ID]
    declared key is not unique: 1,351 duplicate row(s) across ['Queue ID'].

Both are real. The first cost 7,400 of 9,640 dates on an earlier pandas, and the loss was invisible because it looked like ordinary missing data. The second would have silently corrupted every change record built on that key.

Finding them by hand took hours. pipelie finds them in under a second, and it found two more date columns with the same defect that had not been checked at all.

Adopting it on a table that already has problems

Every table has problems. A checker that fails your build on day one over things that predate it gets deleted, so there are two ways to say "not this one".

Accept today's findings as debt. Run once:

pipelie data.csv --key id --accept

That writes pipelie-baseline.json. From then on:

pipelie data.csv --key id --baseline

reports only what is new. Old problems stay suppressed, tomorrow's bug still fails the build. Delete a line from the baseline file to start failing on it again.

Or ignore a rule permanently, when you have decided it is not a problem here:

pipelie data.csv --ignore 'duplicate_rows/*' --ignore '*:legacy_id'

Patterns are globs over code:column, so parse_carnage/*, *:Queue ID and duplicate_rows/key_not_unique:id all work. The same arguments exist on audit():

pipelie.audit(df, key=["id"], ignore=["duplicate_rows/*"], baseline="pipelie-baseline.json")
pipelie.accept(df, key=["id"])        # write the baseline from Python

Findings are identified by rule code and column, never by counts -- a column with 1,351 duplicates today and 1,352 tomorrow is the same finding, so a baseline does not fall apart the moment the data moves.

Machine-readable output

pipelie data.csv --json

Emits the full report -- rows, columns, checks run, and every finding with its code, fingerprint, severity, message, fix and evidence. Exit code is 1 when anything critical survives suppression, 0 otherwise, so it drops into CI unchanged.

Design

Silence is the default. A checker that fires on clean data gets muted, and a muted checker is worse than none. Thresholds favour silence over noise, anything uncertain is a warning rather than a critical, and the test suite contains explicit "must stay quiet" cases on clean frames.

Every finding carries its evidence. Not a prose summary — the counts, shares and correlations behind the claim, so you can check it rather than trust it.

Nothing is mutated. audit() reads. What to do about a finding is your call.

What it does not do

• It does not check orchestration. A DAG that reports success when the write threw, a timeout that cannot stop the work, or a test that passes by never running the code are all real failures, and none of them are visible in the resulting table.
• It does not know your domain. It cannot tell you a capacity figure is in the wrong unit, only that it is oddly constant.
• Silence is not proof of correctness. It means these particular lies are absent.

Background

The failure modes come from Thirteen Ways a Pipeline Lies, a write-up of thirteen documented bugs in one system, each of which reported green.

Apache-2.0.