Skip to main content

pysmartcor

pysmartcor detects variable types and selects a suitable correlation method for each pair. It supports continuous, count, binary, ordinal, and categorical variables and returns the estimate, inference, selected method, and rationale.

Before choosing a method, it checks the pair for outliers, normality, latent normality, and linearity. It chooses among 18 methods, following the decision framework of Harshvardhan and Ranjan (2026). The R package smartcor implements the same rules.

Install

The package is on PyPI:

pip install pysmartcor

Install the optional plotting dependencies when needed:

pip install 'pysmartcor[viz]'

Quick start

import numpy as np
from pysmartcor import smart_cor

rng = np.random.default_rng(42)
education_years = rng.integers(8, 22, size=300)
income = 1500 * education_years + rng.normal(0, 8000, size=300)

result = smart_cor(income, education_years, x_name="income", y_name="education_years", verbose=False)
print(result)

Inspect result.method, result.rationale, and result.diagnostics to see how the diagnostics informed the choice.

Load the example data

The package includes gss_2024_casestudy.csv. This CSV is frozen for reproducibility. The examples below read the bundled copy and do not download data.

from importlib.resources import as_file, files
import pandas as pd

csv_resource = files("pysmartcor").joinpath("data/gss_2024_casestudy.csv")
with as_file(csv_resource) as csv_path:
    gss = pd.read_csv(csv_path)

Correlate one pair

from pysmartcor import smart_cor

result = smart_cor(
    gss["coninc"],
    gss["sex"],
    x_name="coninc",
    y_name="sex",
    verbose=False,
)

print(result)
result.estimate
result.method
result.p_value
result.ci_lower, result.ci_upper
result.diagnostics

Income is far from normal within each sex, so the Jarque-Bera test rejects and the rank-biserial correlation is selected.

Set assume_latent_normal to control pairs that can use a latent-variable method:

smart_cor(gss["degree"], gss["happy"], assume_latent_normal="auto")
smart_cor(gss["degree"], gss["happy"], assume_latent_normal=True)
smart_cor(gss["degree"], gss["happy"], assume_latent_normal=False)

The default, "auto", runs the likelihood-ratio test of latent normality for each affected pair. True and False fix the answer; the outlier and Jarque-Bera checks still run. A binary-by-binary table is saturated, so automatic selection uses phi; True gives tetrachoric and False gives Yule's Q.

Diagnostics

Each diagnostic is also available on its own. All tests use alpha=0.05 unless you pass another value to smart_cor(), smart_cormat(), smart_cor_df(), or compare_methods().

from pysmartcor import detect_outliers, test_bivariate_normality, test_linearity

detect_outliers(gss["coninc"])                       # outer fences, Q1 - 3 IQR and Q3 + 3 IQR
test_bivariate_normality(gss["age"], gss["coninc"])  # Mardia's test
test_bivariate_normality(gss["coninc"], gss["degree"])  # Jarque-Bera and the LR test
test_bivariate_normality(gss["degree"], gss["happy"])   # LR test of the threshold model

diagnostics = test_linearity(gss["coninc"], gss["age"])
diagnostics.decision
diagnostics.p_value

result.diagnostics is a dictionary with the keys outliers, normality, latent, and linearity; a test that did not run is None. The tidy output summarises them in its last four columns: outlier_flag, normality_p, latent_p, and linearity_p.

For two continuous/count variables without outliers, smart_cor() runs test_linearity(). The test regresses each variable on the other, orders the residuals by the regressor (averaging residuals at tied values), and applies the Box-Pierce, Ljung-Box, BDS, Breusch-Pagan, and RESET rule in each direction. A direction rejects linearity when a residual structure test and RESET both reject. Spearman is selected only when both directions reject; otherwise Pearson is retained. Each direction's p-value is max(min(p_bp, p_lb, p_bds), p_reset), the combined p_value is the larger of the two, and the BDS p-value is the minimum over the distance thresholds. An explicit method skips all diagnostics.

Because both directions are tested and tied values are averaged, the decision does not depend on argument order or row order. An inconclusive result retains Pearson and reports the reason. When the rule recommends robust/bootstrap Pearson inference, bootstrap="auto" uses a paired-bootstrap interval; the p-value remains the standard Pearson t-test. bootstrap=False disables that bootstrap. The diagnostics use statsmodels; BDS uses the sample and distance conventions of R's fNonlinear::bdsTest(res, m=3).

Build a matrix

from pysmartcor import smart_cormat

columns = ["age", "coninc", "degree", "happy", "sex", "region"]
matrix = smart_cormat(gss[columns], rng=1, verbose=False)

matrix.correlations
matrix.methods
matrix.types
matrix.to_long()

Use smart_cor_df() when a script needs plain pandas DataFrames:

from pysmartcor import smart_cor_df

plain = smart_cor_df(gss[columns])
plain["correlations"]
plain["methods"]

Set nearPD=True when downstream analysis requires a positive-definite matrix:

adjusted = smart_cormat(gss[columns], nearPD=True, verbose=False)
adjusted.correlations
adjusted.original_correlations
adjusted.nearPD
plain = smart_cor_df(gss[columns], nearPD=True)

The default is nearPD=False. A NumPy implementation follows Matrix::nearPD(..., corr=TRUE), including Dykstra's correction, default tolerances, and the final eigenvalue adjustment. It retains a diagonal of ones. Methods, pairwise details, and to_long() retain the original estimates and inference. Missing or infinite matrix entries must be resolved before adjustment.

Compare methods

from pysmartcor import compare_methods

comparison = compare_methods(
    gss["degree"],
    gss["happy"],
    bootstrap=False,
    verbose=False,
)
comparison.results

Plot a matrix

Plotting requires the optional viz dependencies.

from pysmartcor.viz import cor_heatmap, method_heatmap

cor_heatmap(matrix)
method_heatmap(matrix)

Method selection

Variable types Default method Alternative
Correlation methods (signed, from -1 to +1)
Continuous + continuous Pearson (linear, no outliers) Spearman (outliers or nonlinear), Kendall's tau
Continuous + binary Biserial (if latent normality holds) Point-biserial; rank-biserial (outliers or non-normal)
Binary + binary Pearson = phi Tetrachoric (latent normal), Yule's Q (no latent continuum)
Continuous + ordinal Polyserial (if latent normality holds) Kendall's tau (m <= 5) or Spearman
Ordinal + ordinal Polychoric (if latent normality holds) Kendall's tau (m <= 5) or Spearman; Goodman-Kruskal gamma
Binary + ordinal Polychoric (if latent normality holds) Rank-biserial; Kendall's tau, Spearman
Association measures (unsigned, from 0 to 1)
Continuous + categorical Correlation ratio eta-squared Kruskal-Wallis eta-squared (outliers or non-normal); Cramer's V (binned)
Binary + categorical Cramer's V --
Ordinal + categorical Freeman's theta Cramer's V
Categorical + categorical Cramer's V Theil's U, Tschuprow's T

m is the number of categories of the ordinal variable (the smaller of the two for an ordinal pair). Count variables follow the continuous rows.

Methods

The package implements 18 methods: Pearson, Spearman, Kendall's tau, point-biserial, biserial, rank-biserial, phi, tetrachoric, Yule's Q, polychoric, polyserial, Goodman-Kruskal's gamma, the correlation ratio (eta-squared), Kruskal-Wallis eta-squared, Freeman's theta, Cramer's V, Tschuprow's T, and Theil's U. Each comes with a confidence interval and a p-value.

Required dependencies are NumPy, SciPy, pandas, and statsmodels. Matplotlib and seaborn are optional plotting dependencies.

Paper

The accompanying paper, smartcor: Intelligent Correlation Method Selection for Mixed Variable Types by M. Harshvardhan and Pritam Ranjan (2026), is available as an arXiv preprint: arXiv:2607.22285 (doi:10.48550/arXiv.2607.22285). The vignettes of the R package and the articles on the package website cover the same material.

Authors

M. Harshvardhan (maintainer) and Pritam Ranjan.

License

GPL (>= 3)

Metadata

Release files for pysmartcor 1.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pysmartcor 1.2.0
File Size Uploaded Provenance Actions
pysmartcor-1.2.0.tar.gz 101.0 kB Without provenance Details for pysmartcor-1.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pysmartcor 1.2.0
File Interpreter ABI Platform Actions
pysmartcor-1.2.0-py3-none-any.whl Python 3 none any Details for pysmartcor-1.2.0-py3-none-any.whl

Total release size: 193.3 kB

Release files / pysmartcor-1.2.0.tar.gz

Download URL pysmartcor-1.2.0.tar.gz
Size 101.0 kB
Tags Source
SHA-256 checksum
How to use checksums
3d7308e53eb5e607d0c3185bc000e0592e06844edca8276b57dda6820789da5d
BLAKE2b-256 checksum
How to use checksums
f6028474289a1dd64fe0a1c334a226c153b14f8490a6e7c9645d5492df876f9e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.5

Release files / pysmartcor-1.2.0-py3-none-any.whl

Download URL pysmartcor-1.2.0-py3-none-any.whl
Size 92.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f1ddf2a45ac8bc71000cbf3412ab4a3a2562f13c731ddbd954bb8b75c604e70d
BLAKE2b-256 checksum
How to use checksums
17a0c995c96d4423d9ff813ac5271a12bd2c83515c1424797b726168a4f049c5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.5

Security for pysmartcor 1.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page