pysmartcor
pysmartcor detects variable types and selects a suitable correlation method for each pair. It supports continuous, count, binary, ordinal, and categorical variables and returns the estimate, inference, selected method, and rationale.
Before choosing a method, it checks the pair for outliers, normality, latent normality, and linearity. It chooses among 18 methods, following the decision framework of Harshvardhan and Ranjan (2026). The R package smartcor implements the same rules.
Install
The package is on PyPI:
pip install pysmartcor
Install the optional plotting dependencies when needed:
pip install 'pysmartcor[viz]'
Quick start
import numpy as np
from pysmartcor import smart_cor
rng = np.random.default_rng(42)
education_years = rng.integers(8, 22, size=300)
income = 1500 * education_years + rng.normal(0, 8000, size=300)
result = smart_cor(income, education_years, x_name="income", y_name="education_years", verbose=False)
print(result)
Inspect result.method, result.rationale, and result.diagnostics to see how the diagnostics informed the choice.
Load the example data
The package includes gss_2024_casestudy.csv. This CSV is frozen for reproducibility. The examples below read the bundled copy and do not download data.
from importlib.resources import as_file, files
import pandas as pd
csv_resource = files("pysmartcor").joinpath("data/gss_2024_casestudy.csv")
with as_file(csv_resource) as csv_path:
gss = pd.read_csv(csv_path)
Correlate one pair
from pysmartcor import smart_cor
result = smart_cor(
gss["coninc"],
gss["sex"],
x_name="coninc",
y_name="sex",
verbose=False,
)
print(result)
result.estimate
result.method
result.p_value
result.ci_lower, result.ci_upper
result.diagnostics
Income is far from normal within each sex, so the Jarque-Bera test rejects and the rank-biserial correlation is selected.
Set assume_latent_normal to control pairs that can use a latent-variable method:
smart_cor(gss["degree"], gss["happy"], assume_latent_normal="auto")
smart_cor(gss["degree"], gss["happy"], assume_latent_normal=True)
smart_cor(gss["degree"], gss["happy"], assume_latent_normal=False)
The default, "auto", runs the likelihood-ratio test of latent normality for each affected pair. True and False fix the answer; the outlier and Jarque-Bera checks still run. A binary-by-binary table is saturated, so automatic selection uses phi; True gives tetrachoric and False gives Yule's Q.
Diagnostics
Each diagnostic is also available on its own. All tests use alpha=0.05 unless you pass another value to smart_cor(), smart_cormat(), smart_cor_df(), or compare_methods().
from pysmartcor import detect_outliers, test_bivariate_normality, test_linearity
detect_outliers(gss["coninc"]) # outer fences, Q1 - 3 IQR and Q3 + 3 IQR
test_bivariate_normality(gss["age"], gss["coninc"]) # Mardia's test
test_bivariate_normality(gss["coninc"], gss["degree"]) # Jarque-Bera and the LR test
test_bivariate_normality(gss["degree"], gss["happy"]) # LR test of the threshold model
diagnostics = test_linearity(gss["coninc"], gss["age"])
diagnostics.decision
diagnostics.p_value
result.diagnostics is a dictionary with the keys outliers, normality, latent, and linearity; a test that did not run is None. The tidy output summarises them in its last four columns: outlier_flag, normality_p, latent_p, and linearity_p.
For two continuous/count variables without outliers, smart_cor() runs test_linearity(). The test regresses each variable on the other, orders the residuals by the regressor (averaging residuals at tied values), and applies the Box-Pierce, Ljung-Box, BDS, Breusch-Pagan, and RESET rule in each direction. A direction rejects linearity when a residual structure test and RESET both reject. Spearman is selected only when both directions reject; otherwise Pearson is retained. Each direction's p-value is max(min(p_bp, p_lb, p_bds), p_reset), the combined p_value is the larger of the two, and the BDS p-value is the minimum over the distance thresholds. An explicit method skips all diagnostics.
Because both directions are tested and tied values are averaged, the decision does not depend on argument order or row order. An inconclusive result retains Pearson and reports the reason. When the rule recommends robust/bootstrap Pearson inference, bootstrap="auto" uses a paired-bootstrap interval; the p-value remains the standard Pearson t-test. bootstrap=False disables that bootstrap. The diagnostics use statsmodels; BDS uses the sample and distance conventions of R's fNonlinear::bdsTest(res, m=3).
Build a matrix
from pysmartcor import smart_cormat
columns = ["age", "coninc", "degree", "happy", "sex", "region"]
matrix = smart_cormat(gss[columns], rng=1, verbose=False)
matrix.correlations
matrix.methods
matrix.types
matrix.to_long()
Use smart_cor_df() when a script needs plain pandas DataFrames:
from pysmartcor import smart_cor_df
plain = smart_cor_df(gss[columns])
plain["correlations"]
plain["methods"]
Set nearPD=True when downstream analysis requires a positive-definite matrix:
adjusted = smart_cormat(gss[columns], nearPD=True, verbose=False)
adjusted.correlations
adjusted.original_correlations
adjusted.nearPD
plain = smart_cor_df(gss[columns], nearPD=True)
The default is nearPD=False. A NumPy implementation follows Matrix::nearPD(..., corr=TRUE), including Dykstra's correction, default tolerances, and the final eigenvalue adjustment. It retains a diagonal of ones. Methods, pairwise details, and to_long() retain the original estimates and inference. Missing or infinite matrix entries must be resolved before adjustment.
Compare methods
from pysmartcor import compare_methods
comparison = compare_methods(
gss["degree"],
gss["happy"],
bootstrap=False,
verbose=False,
)
comparison.results
Plot a matrix
Plotting requires the optional viz dependencies.
from pysmartcor.viz import cor_heatmap, method_heatmap
cor_heatmap(matrix)
method_heatmap(matrix)
Method selection
| Variable types | Default method | Alternative |
|---|---|---|
| Correlation methods (signed, from -1 to +1) | ||
| Continuous + continuous | Pearson (linear, no outliers) | Spearman (outliers or nonlinear), Kendall's tau |
| Continuous + binary | Biserial (if latent normality holds) | Point-biserial; rank-biserial (outliers or non-normal) |
| Binary + binary | Pearson = phi | Tetrachoric (latent normal), Yule's Q (no latent continuum) |
| Continuous + ordinal | Polyserial (if latent normality holds) | Kendall's tau (m <= 5) or Spearman |
| Ordinal + ordinal | Polychoric (if latent normality holds) | Kendall's tau (m <= 5) or Spearman; Goodman-Kruskal gamma |
| Binary + ordinal | Polychoric (if latent normality holds) | Rank-biserial; Kendall's tau, Spearman |
| Association measures (unsigned, from 0 to 1) | ||
| Continuous + categorical | Correlation ratio eta-squared | Kruskal-Wallis eta-squared (outliers or non-normal); Cramer's V (binned) |
| Binary + categorical | Cramer's V | -- |
| Ordinal + categorical | Freeman's theta | Cramer's V |
| Categorical + categorical | Cramer's V | Theil's U, Tschuprow's T |
m is the number of categories of the ordinal variable (the smaller of the two for an ordinal pair). Count variables follow the continuous rows.
Methods
The package implements 18 methods: Pearson, Spearman, Kendall's tau, point-biserial, biserial, rank-biserial, phi, tetrachoric, Yule's Q, polychoric, polyserial, Goodman-Kruskal's gamma, the correlation ratio (eta-squared), Kruskal-Wallis eta-squared, Freeman's theta, Cramer's V, Tschuprow's T, and Theil's U. Each comes with a confidence interval and a p-value.
Required dependencies are NumPy, SciPy, pandas, and statsmodels. Matplotlib and seaborn are optional plotting dependencies.
Paper
The accompanying paper, smartcor: Intelligent Correlation Method Selection for Mixed Variable Types by M. Harshvardhan and Pritam Ranjan (2026), is available as an arXiv preprint: arXiv:2607.22285 (doi:10.48550/arXiv.2607.22285). The vignettes of the R package and the articles on the package website cover the same material.
Authors
M. Harshvardhan (maintainer) and Pritam Ranjan.
License
GPL (>= 3)
Metadata
Release files for pysmartcor 1.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pysmartcor-1.2.0.tar.gz | 101.0 kB | Without provenance | Details for pysmartcor-1.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pysmartcor-1.2.0-py3-none-any.whl | Python 3 | none | any | Details for pysmartcor-1.2.0-py3-none-any.whl |
Total release size: 193.3 kB
Release files / pysmartcor-1.2.0.tar.gz
| Download URL | pysmartcor-1.2.0.tar.gz |
|---|---|
| Size | 101.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3d7308e53eb5e607d0c3185bc000e0592e06844edca8276b57dda6820789da5d
|
|
BLAKE2b-256 checksum How to use checksums |
f6028474289a1dd64fe0a1c334a226c153b14f8490a6e7c9645d5492df876f9e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.5
|
Release files / pysmartcor-1.2.0-py3-none-any.whl
| Download URL | pysmartcor-1.2.0-py3-none-any.whl |
|---|---|
| Size | 92.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f1ddf2a45ac8bc71000cbf3412ab4a3a2562f13c731ddbd954bb8b75c604e70d
|
|
BLAKE2b-256 checksum How to use checksums |
17a0c995c96d4423d9ff813ac5271a12bd2c83515c1424797b726168a4f049c5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.5
|
Security for pysmartcor 1.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.