shapley_behaviors

Shapley value transformations for explainable behavioral data analysis.

Traditional clustering asks which samples are similar? but not why do they cluster together? Shapley behavioral transformations answer the "why" by decomposing statistical properties — variance, skewness, kurtosis, entropy — into individual sample contributions. Samples that cluster in behavioral space share the same statistical role in the dataset, providing mechanistic and actionable insights.

Implementation of the methodology from:

Liu, T., and Barnard, A. S. (2025). Understanding interpretable patterns of Shapley behaviours in materials data. Machine Learning: Engineering, 1, 015004. https://doi.org/10.1088/3049-4761/adaaf6

---

Features

• Decompose datasets into variance, skewness, kurtosis, and entropy behavioral spaces
• Parallel computation via joblib for large datasets
• Outlier detection directly in behavioral space
• Automatic break detection (find_break_zones): statistically significant gaps along the principal components define region boundaries and flag satellite clusters, with no user-drawn boundaries required
• Bundled interactive explorer scripts for Jupyter-based analysis: behavioral space generation, break detection with region-of-interest analysis, and k-means cluster analysis
• Antithetic sampling for variance reduction in Monte Carlo estimation

New in 0.1.4

• Corrected Hopkins statistic. The nearest-neighbor search for sampled data points masked the wrong array (an advanced-indexing copy), so every sampled point matched itself at distance zero and H collapsed to exactly 1.0000 for any dataset. Real values are now returned, and a permutation test supplies the accompanying p-value. Any Hopkins statistic produced by 0.1.3 or earlier should be discarded and recomputed.
• Categorical labels in the cluster explorer. Categorical targets previously went through the continuous code path, which averaged categories. They now get the dominant category with its share and category count, a cluster-by-category cross-tab CSV per label, and a stacked composition bar in place of an undefined boxplot. Mixed continuous and categorical label sets are handled together.

New in 0.1.3

• behavioral_cluster_explorer.py: k-means cluster analysis of behavioral spaces (exclusive clusters, per-cluster exports and summaries)
• Automatic break detection integrated into behavioral_region_explorer.py (run without USER_REGIONS for a detection-only pass) and exposed as shapley_behaviors.find_break_zones
• Region statistics now use positional indexing, so datasets with non-unique sample IDs are summarised correctly
• Correct normalisation for odd n_permutations (even values, including all published results, are unaffected)
• copy_scripts no longer silently overwrites locally modified scripts (warns by default; overwrite=False preserves them)

---

Installation

pip install shapley_behaviors

---

Quick Start

import numpy as np
from shapley_behaviors import ShapleyBehaviors

X = np.random.randn(500, 20)  # (n_samples, n_features)

sb = ShapleyBehaviors(n_permutations=100, n_jobs=-1, random_state=42)

# Transform to a single behavioral space
Phi_variance = sb.transform(X, value_function='variance')

# Or compute all four spaces at once
behavioral_spaces = sb.transform_multiple(X)
# keys: 'variance', 'skewness', 'kurtosis', 'entropy'

---

Outlier Detection

from shapley_behaviors import identify_outliers

outlier_indices, outlier_scores = identify_outliers(Phi_variance, threshold=2.5)
print(f"Detected {len(outlier_indices)} outliers")

---

Understanding Behavioral Spaces

Each space answers a different question about the role of each sample in the dataset:

Space	Positive values	Negative values	Use case
Variance	Stretchers — widen the distribution	Stabilizers — typical samples near the mean	Quality control, process instability
Skewness	Pull distribution above the mean	Pull distribution below the mean	Biased synthesis, directional drift
Kurtosis	Tail samples — rare extreme events	Core samples — predictable, well-behaved	Anomaly detection, reliability analysis
Entropy	High-information — rare, unique combinations	Low-information — common, redundant	Dataset curation, diversity quantification

Hopkins Statistic

The Hopkins statistic H measures clustering tendency in behavioral space:

H value	Interpretation
> 0.7	Strong clustering — samples group by behavior
≈ 0.5	Random distribution — no natural grouping
< 0.3	Regular/uniform distribution

---

Convenience Functions

from shapley_behaviors import (
    compute_shapley_variance,
    compute_shapley_skewness,
    compute_shapley_kurtosis,
    compute_shapley_entropy,
)

Phi = compute_shapley_variance(X, n_permutations=100, n_jobs=-1, random_state=42)

---

Explorer Scripts

The package bundles three standalone Jupyter-compatible scripts for comprehensive analysis. Copy them to your working directory:

from shapley_behaviors import copy_scripts

copy_scripts(".")                                         # all scripts
copy_scripts("./analysis", scripts=["behavioral_space_explorer"])  # one script

Existing files that match the packaged versions are left untouched; locally modified files are overwritten with a warning (pass overwrite=False to keep them).

Behavioral Space Explorer

Full dataset exploration — PCA plots, Hopkins statistics, outlier detection:

SEED = 42
N_PERMUTATIONS = 1000      # 100 for quick tests, 1000 for publication
N_JOBS = -1

DATASET_NAME = "mydata"
DATA_FILE = "mydata.csv"
ID_COLUMN = "sample_id"
DROP_COLUMNS = ["col_a", "col_b"]
LABEL_COLUMNS = ["target1", "target2", "category"]
OUTPUT_DIR = "behavioral_exploration"
SELECTED_FEATURES = ["feature1", "feature2"]  # optional highlight

%run -i behavioral_space_explorer.py

Outputs:

File	Contents
{name}_behavioral_spaces.npy	All four behavioral transformations
{name}_behave_{space}_{label}.png	PCA plots colored by each label
{name}_hopkins_statistics.csv	Clustering tendency metrics
{name}_clustering_statistics.csv	Variance explained, pairwise distances
{name}_outliers_{space}.csv	Outlier samples per space

Behavioral Region Explorer

Automatic break detection followed by targeted analysis of PCA regions. Two-pass usage: run without USER_REGIONS to detect boundaries, then define regions from the reported gap midpoints and run again.

BEHAVIORAL_SPACES_FILE = "behavioral_exploration/mydata_behavioral_spaces.npy"

# Pass 1 - break detection only
BREAK_SPACES = ["variance"]   # default: all spaces in the file
USER_REGIONS = None
%run -i behavioral_region_explorer.py
# -> prints break zones and satellite candidate gaps, saves diagnostics,
#    exposes break_zones / pc1_zones / pc2_zones in the namespace

# Pass 2 - full region analysis
PLOT_MODE = "combined"  # or "separate"
USER_REGIONS = {
    "high_variance_cluster": {
        "space": "variance",
        "pc1_range": (0.3, 0.6),      # e.g. detected gap midpoints
        "pc2_range": (-0.2, 0.2),
        "description": "High variance contributors",
        "color": "red",
    },
}

%run -i behavioral_region_explorer.py

Behavioral Cluster Explorer

K-means cluster analysis of a behavioral space, when exclusive algorithmic groups are preferred over hand-defined rectangles:

BEHAVIORAL_SPACES_FILE = "behavioral_exploration/mydata_behavioral_spaces.npy"
SPACE = "variance"   # which space to cluster
K = 4                # number of clusters

%run -i behavioral_cluster_explorer.py
# -> per-cluster sample lists and full-data CSVs, property and
#    composition summaries, cluster scatter and box plots

---

Parameters

Parameter	Values	Notes
n_permutations	50–100 (explore), 200–500 (standard), 1000+ (publication)	Higher = more accurate, slower
n_jobs	-1 (all cores), 1 (debug), N (N cores)	Parallelises over features
random_state	any int	Set for reproducibility; uses antithetic sampling
incremental	True (default), False	True walks each permutation in one vectorised pass. False reproduces 0.1.5 and earlier bit-identically

---

Runtime

Since 0.1.6 a permutation is walked in one vectorised pass rather than by calling the value function on every growing prefix, which removes a factor of n from the cost. Measured on 6 columns at 32 permutations:

samples	0.1.5 and earlier	0.1.6	speedup
100	0.44 s	0.016 s	27x
300	1.37 s	0.018 s	77x
900	4.64 s	0.023 s	206x

The gain grows with sample count, so the runs that used to be planned around overnight are now interactive. The figures below were the estimates for 0.1.5 and earlier; divide by the factor for your sample count.

Dataset size	n_permutations	0.1.5 and earlier
500 samples	100	2–5 min
500 samples	1000	15–30 min
4000 samples	100	20–30 min
4000 samples	1000	2–3 hours

---

API Reference

# Main class
sb = ShapleyBehaviors(n_permutations=100, n_jobs=-1, random_state=42)

# To reproduce output from 0.1.5 and earlier exactly (bit-identical)
sb = ShapleyBehaviors(n_permutations=100, incremental=False)
Phi = sb.transform(X, value_function='variance', verbose=True)
spaces = sb.transform_multiple(X, value_functions=['variance', 'skewness', 'kurtosis', 'entropy'])

# Outlier detection
outlier_indices, outlier_scores = identify_outliers(Phi, threshold=3.0, method='zscore')

# Automatic break detection along one axis (e.g. a principal component)
from shapley_behaviors import find_break_zones
zones, rejected = find_break_zones(pc1_values, z_threshold=2.5,
                                   min_region_fraction=0.05,
                                   max_straggler_fraction=0.02)
# zones: dense-block boundaries (lower_edge/upper_edge/midpoint/counts)
# rejected: significant gaps too small to form regions; those with
#           rejected[i]['satellite'] == True mark small coherent clusters

# The prefix form behind the fast path, should you want it directly:
# the value of a statistic on x[:j+1] for every j, in one pass
from shapley_behaviors import prefix_values
curve = prefix_values(column, 'variance')

---

Troubleshooting

Problem	Solution
ImportError	pip install shapley_behaviors
Long runtime	Reduce n_permutations to 100 for testing
Memory error	Reduce n_jobs or process data in batches
High additivity error warning	Increase n_permutations
H ≈ 0.5 (no clustering)	Data may lack natural behavioral groupings

---

Citation

@article{liu2025shapley,
  author  = {Liu, Tommy and Barnard, Amanda S.},
  title   = {Understanding interpretable patterns of {Shapley} behaviours in materials data},
  journal = {Machine Learning: Engineering},
  volume  = {1},
  pages   = {015004},
  year    = {2025},
  doi     = {10.1088/3049-4761/adaaf6}
}

---

Links

• PyPI
• Repository
• Issues
• Paper

---

MIT License — Copyright © 2024 Amanda S. Barnard and Tommy Liu