pysmartcor

pysmartcor detects variable types and selects a suitable correlation method for each pair. It supports continuous, count, binary, ordinal, and categorical variables and returns the estimate, inference, selected method, and rationale.

Install

Once the package is on PyPI, a single call installs it together with all of its dependencies:

python -m pip install pysmartcor

To install the supplied archive instead (pip resolves the dependencies automatically in both cases):

python -m pip install pysmartcor.zip

Install the optional plotting dependencies when needed:

python -m pip install 'pysmartcor[viz]'

Load the example data

The package includes gss_2024_casestudy.csv. This CSV is frozen for reproducibility. The examples below read the bundled copy and do not download data.

from importlib.resources import as_file, files
import pandas as pd

csv_resource = files("pysmartcor").joinpath("data/gss_2024_casestudy.csv")
with as_file(csv_resource) as csv_path:
    gss = pd.read_csv(csv_path)

Correlate one pair

from pysmartcor import smart_cor

result = smart_cor(
    gss["coninc"],
    gss["age"],
    x_name="coninc",
    y_name="age",
    verbose=False,
)

print(result)
result.estimate
result.method
result.p_value
result.ci_lower, result.ci_upper

Set assume_latent_normal to control pairs that can use a latent-variable method:

smart_cor(gss["degree"], gss["happy"], assume_latent_normal="auto")
smart_cor(gss["degree"], gss["happy"], assume_latent_normal=True)
smart_cor(gss["degree"], gss["happy"], assume_latent_normal=False)

The default, "auto", tests the assumption for each affected pair. A binary-by-binary table is saturated, so automatic selection uses phi. Set the argument to True to request tetrachoric correlation.

Build a matrix

from pysmartcor import smart_cormat

columns = ["age", "coninc", "degree", "happy", "sex", "region"]
matrix = smart_cormat(
    gss[columns],
    assume_latent_normal=False,
    verbose=False,
)

matrix.correlations
matrix.methods
matrix.types
matrix.to_long()

Use smart_cor_df() when a script needs plain pandas DataFrames:

from pysmartcor import smart_cor_df

plain = smart_cor_df(gss[columns], assume_latent_normal=False)
plain["correlations"]
plain["methods"]

Compare methods

from pysmartcor import compare_methods

comparison = compare_methods(
    gss["degree"],
    gss["happy"],
    assume_latent_normal=False,
    bootstrap=False,
    verbose=False,
)
comparison.results

Plot a matrix

Plotting requires the optional viz dependencies.

from pysmartcor.viz import cor_heatmap, method_heatmap

cor_heatmap(matrix)
method_heatmap(matrix)

Methods

The package implements Pearson, Spearman, Kendall's tau, point-biserial, rank-biserial, phi, tetrachoric, Yule's Q, polychoric, polyserial, Cramer's V, Theil's U, Tschuprow's T, and Goodman-Kruskal's gamma.

Required dependencies are NumPy, SciPy, and pandas. Matplotlib and seaborn are optional plotting dependencies.

Paper

The accompanying paper, smartcor: Intelligent Correlation Method Selection for Mixed Variable Types by M. Harshvardhan and Pritam Ranjan, is a work in progress. The vignettes of the R package and the articles on the package website cover the same material.

Authors

M. Harshvardhan (maintainer) and Pritam Ranjan.

License

GPL (>= 3)