corrsleuth 0.2.0

A relationship diagnosis engine for pandas users.

CorrSleuth

Correlation is not one number.

CorrSleuth is a relationship diagnosis engine for pandas. Point it at two numeric columns and it measures their relationship several different ways, compares the results, and tells you in plain English what kind of relationship it is — and where a plain correlation could mislead you.

Why CorrSleuth?

A single correlation number (like Pearson's r) squeezes a rich relationship into one value, and that value can lie. Two columns can have r ≈ 0 while being tightly linked in a U-shape, or show a high r that is really driven by a couple of outliers. The number alone never tells you which situation you are in.

CorrSleuth runs several complementary measures, looks at where they agree or disagree, and turns that into an actionable diagnosis:

• See what a correlation matrix hides. Catches nonlinear, non-monotonic, and outlier-driven relationships that a single coefficient misses.
• Get answers in plain English. Every result comes with a diagnostic label, a written explanation, and a recommended next step — not just numbers.
• Spot leverage and outlier traps. Flags when a strong-looking correlation is actually driven by a few extreme points.
• Scan a whole dataset against a target. Rank every numeric predictor against an outcome column in one call.
• Stay honest about uncertainty. Built-in warnings for small samples, ties, missingness, and conflicting evidence, plus optional bootstrap stability.
• Keep installs light. The base install needs only pandas, numpy, scipy, and matplotlib (for the diagnostic plots); the heavier nonlinear metrics (Distance Correlation, Mutual Information) are opt-in extras.

CorrSleuth is diagnostic, not causal. It identifies evidence consistent with relationship patterns, but it does not prove causation, treatment effects, or model specification certainty.

For a guide to what each diagnostic label means, when it can mislead, and what to do next, see the interpretation guide. For how it works under the hood — the measures, how they're compared, and the label logic — see the methodology doc.

Installation

Install the base package for core metrics (Pearson, Spearman, Kendall tau-b):

pip install corrsleuth

Install with standard mode to include Distance Correlation and Mutual Information:

pip install corrsleuth[standard]

mode="deep" is a superset of standard — it computes everything standard does (Distance Correlation, Mutual Information) plus robust correlation diagnostics and Chatterjee's ξ — so it also requires the standard extra (pip install corrsleuth[standard]).

For a progress bar during scan_target(progress=True), install the optional progress extra:

pip install corrsleuth[progress]

For the optional LOWESS trend overlay on the diagnostic scatter plots, install the plot extra (the plot still renders without it, just without the smoother):

pip install corrsleuth[plot]

Quick Start

Point CorrSleuth at two numeric columns of your own DataFrame and read the diagnosis:

import corrsleuth as cs

# `df` is your pandas DataFrame; replace the column names with your own.
result = cs.profile_pair(df, "ad_spend", "revenue")

print(result.pattern)    # the diagnostic label, e.g. "near_linear"
print(result.explain())  # a plain-English interpretation
print(result.summary())  # metrics, diagnostics, warnings, and recommendations

result.plot(show=True)    # a 1x3 diagnostic figure (scatter, ranks, summary)

profile_pair() works on the base install. Pass mode="standard" to add nonlinear metrics (Distance Correlation, Mutual Information) or mode="deep" for those plus robust diagnostics and Chatterjee's ξ. Both standard and deep require the [standard] extra.

No dataset handy? CorrSleuth ships a simulator so you can try it immediately:

from corrsleuth.datasets import make_relationship

# A U-shaped relationship: Pearson is near zero, but the variables are related.
df = make_relationship("u_shape", n=500, noise=0.1, random_state=42)

result = cs.profile_pair(df, "x", "y", mode="standard")
print(result.pattern)    # nonmonotonic_dependence
print(result.explain())

Evidence consistent with a relationship that is not simply increasing or decreasing (e.g., U-shaped or cyclical). Standard linear and rank metrics may understate this relationship. Do not interpret this association causally without proper design or controls.

Screening many predictors against one outcome? Use scan_target():

# Build a frame with a few predictors and one target column.
data = make_relationship("linear_positive", n=300, random_state=0).rename(
    columns={"x": "ad_spend", "y": "revenue"}
)
data["noise"] = make_relationship("independent", n=300, random_state=1)["y"]

report = cs.scan_target(data, target="revenue")
print(report.summary())          # grouped, ranked overview of every predictor
report.to_frame()                # one tidy row per profiled column

That is enough to get productive. The sections below explain the metrics, modes, diagnostic labels, and the full API.

Canonical Examples

CorrSleuth includes a relationship simulator that generates common patterns.

1. Near Linear

df = make_relationship("linear_positive", random_state=42)
result = cs.profile_pair(df, "x", "y")
# Pattern: near_linear

2. Monotonic Nonlinear

df = make_relationship("monotonic_log", random_state=42)
result = cs.profile_pair(df, "x", "y")
# Pattern: monotonic_nonlinear

3. Nonmonotonic Dependence

df = make_relationship("u_shape", random_state=42)
result = cs.profile_pair(df, "x", "y", mode="standard")
# Pattern: nonmonotonic_dependence

4. Outlier Driven

df = make_relationship("outlier_driven", random_state=42)
result = cs.profile_pair(df, "x", "y")
# Pattern: possible_outlier_or_leverage

5. Independent

df = make_relationship("independent", random_state=42)
result = cs.profile_pair(df, "x", "y")
# Pattern: weak_or_no_relationship

How It Works

CorrSleuth takes a fundamentally different approach to bivariate analysis. Instead of relying on a single metric, it computes multiple complementary association measures and compares them:

1. Pearson: Captures linear correlation.
2. Spearman: Captures monotonic (rank-based) correlation.
3. Kendall tau-b: A robust rank correlation that handles ties well.
4. Distance Correlation (Standard mode): Captures non-linear dependencies.
5. Mutual Information (Standard mode): Captures arbitrary statistical dependence. Reported as raw, unnormalized MI (in nats) — it is >= 0 and unbounded, not on a 0–1 scale, so read it relatively rather than like a correlation coefficient.
6. Robust Pearson-family diagnostics (Deep mode): Trimmed Pearson, winsorized Pearson, biweight midcorrelation, and median-clipped Pearson help check whether Pearson appears sensitive to extreme values.
7. Chatterjee's ξ (Deep mode): An asymmetric coefficient of association — ξ(X → Y) measures whether Y is a (noisy) function of X. Catches U-shape and other functional dependencies that Pearson and Spearman miss.

By examining where these metrics agree or disagree, CorrSleuth assigns a heuristic diagnostic label (e.g., monotonic_nonlinear if Spearman is high but Pearson is low, or nonmonotonic_dependence if Distance Correlation is high but Spearman is low).

A family of structural diagnostics runs in every mode (no optional dependency) and feeds those same labels for cases the metrics above miss on their own — a bin-based lack-of-fit test and bin-mean reversal count (smooth curves, step functions, and oscillating/periodic relationships), a squared-value correlation (circular/radial dependence), a single-breakpoint segmentation search (steps, level-shift discontinuities, compound trend+wave shapes), a two-group split test (mixture / lurking-grouping-variable structure), heteroscedasticity tests (changing residual spread), and row-level Cook's-distance influence (leverage). They aren't reported as standalone metrics; instead they populate five orthogonal relationship axes on result.diagnostics — mean_shape, variance_shape, dependence_type, outlier_sensitivity, and functional_direction — that describe properties a single label can't carry at once (a mean can be linear while its variance grows and a few rows drive it). See the shape-diagnostics design note and the interpretation guide for the full axis value lists.

Scope

CorrSleuth focuses on numeric pairwise profiling and target-oriented scans.

In scope:

• Profiling one numeric pair with profile_pair().
• Scanning every numeric column against a single target with scan_target().
• Lite metrics: Pearson, Spearman, and Kendall tau-b.
• Standard metrics: Distance Correlation and Mutual Information.
• Deep metrics: the standard metrics plus robust correlation diagnostics and Chatterjee's ξ (a superset of standard; requires the [standard] extra).
• Heuristic diagnostic labels, warnings, recommendations, and diagnostic plots.
• Reproducible-when-seeded (random_state=) simulated relationships through make_relationship().

Out of scope for now:

• Categorical or mixed-type variables.
• Full correlation matrices.
• HTML reports.
• Scikit-learn transformers or automated model fitting.
• Causal inference.

Missing Data and Warnings

profile_pair() supports three missing-data modes:

• missing="pairwise" drops rows missing either selected variable.
• missing="listwise" drops rows missing a value in any column of data (complete-case deletion) before selecting the pair, so it coincides with pairwise only when data contains just the two profiled columns. In a scan_target() run this profiles every column on the same complete-case rows.
• missing="raise" raises an error if either selected variable contains missing values.

Validation warnings are exposed through result.warnings. CorrSleuth warns about small samples, high missingness, low unique-value ratios, constant inputs, downsampling, conflicting directional evidence, and high Chatterjee's ξ or mutual information alongside a weak or ambiguous label, when applicable.

Standard Mode

mode="standard" adds Distance Correlation and Mutual Information. It requires the optional dependencies installed by:

pip install corrsleuth[standard]

If those dependencies are not available, CorrSleuth raises OptionalDependencyError (importable as from corrsleuth import OptionalDependencyError) instead of silently skipping metrics.

For Distance Correlation, CorrSleuth downsamples to 20,000 rows by default when n_used is larger than that cap and records a warning. Use max_n_for_dcor=None to disable this cap. The downsample is seeded by random_state (default 42), so repeated runs on the same input return the same number.

Deep Mode

mode="deep" is a strict superset of standard: it computes Distance Correlation and Mutual Information and adds robust correlation diagnostics and Chatterjee's ξ. Because it includes the standard metrics, it requires the corrsleuth[standard] extra (dcor, scikit-learn) and raises OptionalDependencyError if they are missing. On top of the standard metrics it adds:

• pearson_trimmed_1pct: Pearson after dropping rows outside the 1st/99th percentile range of either variable.
• pearson_winsorized_1pct: Pearson after clipping both variables at their 1st/99th percentiles.
• biweight_midcorrelation: A median/MAD-based robust correlation.
• pearson_median_clipped_20pct: Pearson after clipping deviations around each median at the 80th percentile.
• chatterjee_xi: Chatterjee's coefficient of correlation, an asymmetric measure that captures whether Y is a (noisy) function of X. Values sit near 0 under independence and approach 1 for strong functional dependence; finite-sample estimates can be slightly negative. Detects U-shape and other dependencies that Pearson and Spearman miss.
• chatterjee_xi_reverse: Same statistic in the opposite direction (ξ(Y → X)). For target scans this is the target→candidate direction; the candidate→target direction feature-engineering users usually want is the forward chatterjee_xi.

These are robustness and dependence diagnostics, not definitive replacements for Pearson or visual inspection. The robust correlations are most useful when Pearson is strong but rank metrics or plots suggest leverage-sensitive behavior. Chatterjee's ξ is most useful for surfacing functional dependence that the linear/rank pair miss. Deep mode also computes Distance Correlation and Mutual Information (it is a superset of standard), so a deep profile carries the full metric set. The label cascade does not consult ξ, so a strongly nonmonotonic pair keeps its lite-style label in deep mode — but when ξ exceeds 0.35 and the label is weak_or_no_relationship or mixed_or_ambiguous, CorrSleuth emits a warning so the dependence is not silently contradicted by the label. metric_pearson_trimmed_1pct and result.diagnostics.pearson_trimmed use the same 1% trimmed-Pearson sensitivity calculation so users see one consistent trim value.

chatterjee_xi is reported as ξ(pair.x → pair.y), so for a profile called as profile_pair(df, "x", "y") the value answers "is y a function of x?". chatterjee_xi_reverse reports the same statistic with the arguments swapped (ξ(pair.y → pair.x)). Because scan_target profiles each pair as profile_pair(data, candidate, target), a scan gets both the candidate→target direction (chatterjee_xi — usually the one feature-engineering users want) and the target→candidate direction (chatterjee_xi_reverse) without an extra call. The metric converges slowly on small samples and returns None with a warning when n_used < 20. It uses the tie-corrected estimator from Chatterjee (2020), so it stays well-calibrated when Y is discrete or low-cardinality. Ties in the sort variable are broken with a seeded random permutation (Chatterjee's prescription; breaking them by Y would leak the response and inflate ξ), so values are reproducible for a given input and random_state but, when the sort variable has ties, depend on that tie-break rather than being invariant to row order. See docs/nonlinear-metrics-design.md for the rationale and the candidates that were considered and deferred.

API Reference

profile_pair()

The main entry point for profiling a real-valued numeric pair. Complex-valued columns raise InputError rather than being silently projected onto the real axis — cast to the real part or magnitude first if that is what you intend.

def profile_pair(
    data: pd.DataFrame,
    x: str,
    y: str,
    mode: str = "lite",                 # "lite", "standard", or "deep"
    missing: str = "pairwise",          # "pairwise", "listwise", or "raise"
    include_caveat: bool = True,        # Includes causal caveats in explanations
    max_n_for_dcor: int | None = 20000, # Downsampling cap for Distance Correlation
    random_state: int = 42,             # Seed: dcor downsampling, MI, bootstrap resampling, xi tie-breaking
    bootstrap: int | None = None,       # Optional bootstrap interval count
    bootstrap_metrics: str | Sequence[str] = "lite",  # "lite", "standard", or metric names
    max_n_for_bootstrap: int | None = 5000,
) -> CorrSleuthResult

Set bootstrap=200 to compute percentile bootstrap intervals (the 2.5/97.5 percentiles of the resampling distribution) and pattern stability for Pearson, Spearman, and Kendall tau-b. Bootstrap diagnostics are disabled by default. These are approximate 95% intervals only when every row is resampled; the default caps each replicate at max_n_for_bootstrap=5000 rows, so for `n_used

5000they become *conservative* (wider) m-out-of-n bands — inflated by roughlysqrt(n_used / 5000) — and a warning says so. A binding cap also means **pattern stability** is computed by relabeling the 5,000-row resamples — a noisier decision procedure than the full sample, so near a label threshold it can understate the full-sample label's stability (bootstrap_stability.sample_sizerecords the per-replicate size used). Passmax_n_for_bootstrap=Nonefor full-size resamples and approximate (never exact — it is still a bootstrap) full-sample percentile intervals; slower on large data. Even inmode="standard", bootstrap uses lite metrics unless you explicitly pass bootstrap_metrics="standard", because distance correlation and mutual information can be expensive to resample. Standard bootstrap metrics require the [standard]extras even when the mainprofile_pair()call usesmode="lite"`. Higher bootstrap counts and standard metrics can take many seconds on larger datasets, especially with distance correlation.

Two conservative guards mean these fields can be omitted (left None) on small or heavily capped samples: intervals are not reported when the effective per-replicate size is below 20 (too few rows for a reliable percentile bootstrap), and pattern stability is not reported when max_n_for_bootstrap caps replicates below 30 on a larger original sample (every replicate would be judged low-power, making stability meaningless against the full-sample label). A warning explains each case.

CorrSleuthResult

The object returned by profile_pair().

• .pattern: The assigned heuristic label (e.g., "near_linear").
• .disagreement_score: Scalar summarizing how much the association measures disagree (see the methodology note).
• .warnings / .recommendations: Lists of cautionary notes and suggested next steps.
• .diagnostics: A MetricDiagnostics object carrying the numeric diagnostic gaps (rank_linear_gap, nonmonotonic_gap, …), the shape/segmentation/variance/influence/mixture diagnostics (bin_lof_r2_gain(_robust), sq_corr(_robust), segment_gain/segment_stepness/segment_jump_ratio/breakpoint_x, bp_pvalue/gq_ratio/bowtie_ratio, max_cook_distance/n_influential_points, cluster_split_r2/cluster_valley_share/cluster_min_share/pearson_within_cluster), and the five secondary axes: mean_shape (linear, smooth_curve, step_or_threshold, oscillating_trend, discontinuous_jump, curved), variance_shape (constant, increasing_spread, decreasing_spread, edge_high_spread, center_high_spread), dependence_type (monotone, magnitude_linked, oscillating, nonmonotone, closed_loop_or_multivalued, two_group_shift), outlier_sensitivity (low, single_point_driven, high_leverage_cluster, high), and functional_direction (y_of_x, x_of_y, both_directions, neither_direction; deep mode only).
• .summary(): Returns a string summary of the metrics, label, warnings, recommendations, and caveat.
• .explain(): Returns a plain-English narrative interpreting the results.
• .plot(show=False): Generates a 1x3 Matplotlib diagnostic figure.
• .bootstrap_intervals: Optional bootstrap interval table when requested (None when intervals are suppressed for too-small effective replicates; see above).
• .pattern_stability: Optional share of bootstrap samples with the same label (None when stability is suppressed for cap-induced low-power replicates; see above).
• .bootstrap_label_counts: Optional diagnostic label counts from bootstrap samples.
• .stability_label: Optional "low", "medium", or "high" stability label.
• .to_markdown(include_caveat=None): Exports a compact Markdown report with metrics, diagnostics, warnings, recommendations, optional bootstrap sections, and the non-causal caveat by default.
• .to_dict() / .to_frame(): Serializes the output for downstream pipelines.

scan_target()

Profile every eligible real-valued numeric predictor against a single real-valued numeric target column.

def scan_target(
    data: pd.DataFrame,
    target: str,
    *,
    columns: Sequence[str] | None = None, # Restrict scan to these columns
    mode: str = "lite",                   # Forwarded to profile_pair
    missing: str = "pairwise",            # Forwarded to profile_pair
    errors: str = "warn",                 # "warn" captures per-column failures, "raise" propagates
    direction: str = "forward",           # "forward" | "reverse" | "both" (adds the reverse-shape view)
    max_pairs: int | None = None,         # Cap on columns profiled
    sample_size: int | None = None,       # Optional one-time row downsample
    progress: bool = False,               # Use tqdm if installed; documented no-op otherwise
    random_state: int = 42,
    **profile_pair_kwargs,                # e.g. bootstrap=, include_caveat=
) -> CorrSleuthTargetReport

Quick example:

report = cs.scan_target(df, target="sales")
print(report.summary())
report.to_frame()  # one row per profiled or skipped column

Non-numeric, complex, or missing columns listed in columns= are recorded as skipped entries with error_type (NonNumeric, ComplexDtype, ColumnNotFound, TargetExcluded, or DuplicateColumn) and error_message rather than aborting the scan. A complex-dtype target raises InputError, and complex columns are excluded from auto-selection when columns=None. With errors="warn" (default), exceptions raised by profile_pair() are captured as error entries. Use errors="raise" to fail fast.

CorrSleuthTargetReport

The object returned by scan_target().

• .target: Name of the target column.
• .entries: List of TargetScanEntry objects, one per inspected column.
• .successes / .failures: Convenience splits.
• .summary(top_n=5, include_caveat=True): Section-structured text overview. Pattern sections (Strongest near-linear relationships, Potential monotonic nonlinear relationships, Potential nonmonotonic relationships, Possible outlier-driven relationships, Weak or no pairwise relationships) are emitted only when populated, each capped at top_n. Variables whose pattern falls outside that set (e.g. low_power_or_uncertain, mixed_or_ambiguous) appear in an Other or inconclusive section so no profiled variable disappears from the summary. A cross-cutting Variables Pearson may underrate section lists variables where rank (Spearman/Kendall) or standard-mode nonmonotonic evidence exceeds Pearson by more than 0.20; the gap is directional, so leverage cases where Pearson is stronger than the rank metrics are excluded. A Dependence may be understated section lists weak/ambiguous candidates that nonetheless carry strong nonmonotonic or radial dependence evidence — the lite-computable robust squared-correlation sq_corr_robust, or distance correlation / Chatterjee's ξ / mutual information in standard/deep mode — so a real relationship the headline label understates is not lost in a wide scan (the robust signal excludes heavy-tailed leverage artifacts). With direction="both", a Shape differs by direction section flags candidates whose reverse orientation is a structured nonlinearity while the forward one is not. Variables with missingness or tie warnings surfaces columns whose validation warnings mention ties, missingness, low unique ratios, small samples, or constant inputs. Skipped or failed lists non-numeric / errored columns. The output is deterministic; the pattern sections sort by disagreement_score descending, while the cross-cutting sections sort by their own evidence strength (the underrate gap, the dependence-understatement signal) or alphabetically by column.
• .to_frame(): One row per inspected column with variable, target, status, pattern, disagreement score, warnings, recommendations, per-metric value columns, and a diagnostic_<field> column for every MetricDiagnostics field (numeric diagnostics and the five secondary axes). Under direction="both" it also adds reverse_pattern / reverse_mean_shape / reverse_dependence_type. Skipped or errored rows leave the result-dependent columns NaN and populate error_type / error_message.
• .to_markdown(top_n=5, include_caveat=True): Exports the grouped target report as Markdown with overview counts, populated pattern sections, the cross-cutting Pearson-underrated, dependence-understated, and reliability-warning sections, skipped/failed rows, and the non-causal caveat by default.
• .pearson_underrated(threshold=0.20): Returns a DataFrame of variables where Pearson may understate the relationship. The ranking is directional: Spearman, Kendall, or standard-mode nonmonotonic evidence must exceed Pearson by more than threshold, so outlier/leverage cases where Pearson is stronger than rank metrics are excluded. Rows include metric values, explicit excess-over-Pearson gap values, raw nonmonotonic_gap, pattern, disagreement score, and warnings, sorted by strongest evidence.
• .plot_top(n=12, sort_by="disagreement_score", patterns=None, ncols=3, figsize=None, show=False): Compact gallery of the top-n target relationships as a Matplotlib Figure. sort_by accepts "disagreement_score" (raw value descending) or a metric name (pearson, spearman, kendall_tau_b, distance_correlation, mutual_information, or a deep-mode robust metric, chatterjee_xi, or chatterjee_xi_reverse; sorted by absolute value descending). patterns= filters to specific diagnostic labels and accepts either a single string or an iterable. Each panel is a scatter in the orientation actually profiled — candidate on x, target on y for direction="forward"/"both"; target on x, candidate on y for direction="reverse" (which profiles E[candidate | target]) — titled with the column name, pattern, and key metric values. When fewer than n variables match, the unused grid slots are hidden; an empty filter yields a placeholder figure rather than raising.

Documentation

• Methodology — for statisticians and data scientists: the pipeline, what each association measure detects and assumes, how the measures are compared (the disagreement_score), the label cascade, the bootstrap-stability approach, reproducibility, and limitations.
• Interpretation Guide — meaning, typical metric pattern, common examples, recommended next steps, and caveats for every diagnostic label, plus topic notes on Pearson misleads, monotonicity, leverage, ties, and performance modes.
• Thresholds and Rationale — every cut point that drives a label or warning, with its value, location, justification, and how to override the label-driving ones.
• Nonlinear Metrics Design Note — why Chatterjee's ξ was chosen over HSIC, MGC, MIC, and Hoeffding's D for mode="deep".
• Shape Diagnostics Design Note — the design rationale for the first-generation shape diagnostics (bin_lof_r2_gain, sq_corr, and the bin-reversal periodicity closure); the later diagnostics (single-bend, segment_jump_ratio, the two-group cluster split, oscillating_trend) are documented in the thresholds note and interpretation guide.

License

This project is licensed under the MIT License - see the LICENSE file for details.