process-improve
Multivariate analysis, designed experiments, and process monitoring for Python. Built for chemometrics, manufacturing, and pharma data - the methods that scikit-learn skips.
New here? The architecture overview (source) is the map of the codebase - package layout, the estimator stack, and the MCP tool layer.
What it does
process-improve provides production-grade implementations of the methods
practitioners actually use on real plant and lab data:
- PCA with SVD and NIPALS, plus native missing-value handling via Trimmed Score Regression
- PLS regression with a fully sklearn-compatible API, VIP scores, and cross-validated diagnostics
- TPLS - PLS for T-shaped (multi-block) data structures
- Outlier detection combining Hotelling's T² and SPE with an ESD-based test
- Designed experiments - full-factorial, fractional-factorial, and response-surface designs, plus a multi-stage DOE strategy recommender
- Process monitoring - Shewhart, CUSUM, and Holt-Winters control charts
- Batch data analysis - alignment, feature extraction, and multivariate batch monitoring (MBPCA / MBPLS)
- Interactive Plotly diagnostics bound directly to every fitted model
Outputs are pandas-native: scores, loadings, and predictions keep your row
and column labels.
Scale: the estimators are in-memory - they assume the (scaled) data matrix fits in RAM, plus a couple of working copies during
fit. A float64 matrix needs aboutrows x cols x 8 bytes(e.g. 10M x 200 is ~16 GB). For the practical limits and guidance on larger-than-RAM data, see Scaling and memory (source).
It is the companion package to the online textbook Process Improvement using Data, and powers the statistical engine behind factori.al.
Why not scikit-learn?
scikit-learn answers "what fits the data?" - process-improve answers
"is this batch normal, which variable went off, and how confident am I in the
prediction?" The two libraries are designed to be used together;
process-improve follows sklearn conventions (fit, predict, score, the
_ suffix on fitted attributes) and drops into existing pipelines.
| Capability | scikit-learn | process-improve |
|---|---|---|
| PCA, PLS with sklearn-style API | ✓ | ✓ |
| Missing-data fitting (NIPALS / TSR) | - | ✓ |
| Hotelling's T² + SPE outlier limits | - | ✓ |
| Variable-level score contributions | - | ✓ |
| Cross-validated coefficient confidence intervals | - | ✓ |
| Multi-block models (TPLS) | - | ✓ |
| Designed experiments (DoE) | - | ✓ |
| Control charts (Shewhart / CUSUM / Holt-Winters) | - | ✓ |
| Batch process monitoring (MBPCA / MBPLS) | - | ✓ |
| Plotly diagnostics built in | - | ✓ |
Labeled DataFrame outputs |
partial | ✓ |
Installation
pip install process-improve # core (numpy, pandas, sklearn, statsmodels, patsy, pydantic, pyyaml, tqdm)
pip install 'process-improve[plotting]' # adds matplotlib, plotly, seaborn, ridgeplot
pip install 'process-improve[expt]' # adds pyDOE3 (designed experiments / DOE)
pip install 'process-improve[batch]' # adds openpyxl, scikit-image (batch process data IO)
pip install 'process-improve[mcp]' # adds the MCP server runtime
pip install 'process-improve[fast]' # adds numba (JIT speedups for batch alignment)
pip install 'process-improve[all]' # everything above (the pre-1.24.11 closure)
Requires Python 3.10 or newer. The core install pulls in numpy, pandas,
scipy, scikit-learn, statsmodels, patsy, pydantic, pyyaml, and
tqdm. Heavier optional surfaces (plotting, designed experiments, batch IO,
MCP server, numba JIT) live in extras so a caller who only needs, say,
detect_multivariate_outliers does not have to install Plotly or numba.
Quick start
PCA - Principal Component Analysis
import pandas as pd
from process_improve.multivariate.methods import PCA, MCUVScaler
X = pd.read_csv("your_data.csv", index_col=0)
X_scaled = MCUVScaler().fit_transform(X)
pca = PCA(n_components=3).fit(X_scaled)
print(pca.r2_cumulative_) # cumulative R² per component
pca.score_plot() # interactive Plotly figure
# Flag outliers using combined T² and SPE limits at 95% confidence
outliers = pca.detect_outliers(conf_level=0.95)
# Which variables drove the first observation off?
contrib = pca.score_contributions(pca.scores_.iloc[0].values)
PLS - Projection to Latent Structures
from process_improve.multivariate.methods import PLS, MCUVScaler
# Scale X and Y separately
scaler_x = MCUVScaler().fit(X)
scaler_y = MCUVScaler().fit(Y)
X_s, Y_s = scaler_x.transform(X), scaler_y.transform(Y)
pls = PLS(n_components=3).fit(X_s, Y_s)
print(pls.beta_coefficients_) # regression coefficients (K x M)
print(pls.r2_cumulative_) # cumulative R² for Y
print(pls.vip()) # VIP scores per X variable
# Predict new observations (sklearn-compatible: returns just y_hat)
y_pred = pls.predict(scaler_x.transform(X_new))
# Predict with full per-row diagnostics (scores, T², SPE, plus y_hat)
result = pls.diagnose(scaler_x.transform(X_new))
result.y_hat # point predictions
result.spe # squared prediction error
result.hotellings_t2 # Hotelling's T² for new observations
# Cross-validated component selection
cv_select = PLS.select_n_components(X_s, Y_s, max_components=6)
print(cv_select.n_components) # recommended number of components
print(cv_select.rmsecv) # RMSECV per component count
# Cross-validation with beta-coefficient confidence intervals
cv = pls.cross_validate(X_s, Y_s, cv="loo")
print(cv.beta_ci_lower, cv.beta_ci_upper) # 95% CI for each beta
print(cv.significant) # betas significantly != 0
print(cv.q_squared) # cross-validated R² (Q²)
DOE - multi-stage experimental strategy
from process_improve.experiments.factor import Factor, Response
from process_improve.experiments.strategy import recommend_strategy
factors = [
Factor(name="Temperature", low=25, high=40, units="degC"),
Factor(name="pH", low=5.0, high=7.5),
Factor(name="Glucose", low=10, high=50, units="g/L"),
]
strategy = recommend_strategy(
factors=factors,
responses=[Response(name="Yield", goal="maximize", units="g/L")],
budget=40,
domain="fermentation",
)
for s in strategy["stages"]:
print(s["stage_number"], s["design_type"], s["estimated_runs"])
Longer, fully-worked versions of each example live in the
Quickstart guide
and the process_improve/notebooks_examples/ folder.
New to designed experiments? The Applied DoE tutorial is an eight-module worked-solution series.
API design
PCA and PLS follow scikit-learn conventions: fit() returns self, fitted
attributes end with a trailing underscore (scores_, loadings_, spe_,
hotellings_t2_, r2_cumulative_, ...), and predict() returns an
sklearn.utils.Bunch with named fields (y_hat, spe, hotellings_t2, ...).
Inputs are accepted as pandas.DataFrame, and index/column labels are
preserved through fit and transform.
Documentation & learning resources
- API reference & user guide: https://kgdunn.github.io/process-improve/
- Applied DoE tutorial (8 modules): https://kgdunn.github.io/process-improve/applied_doe/index.html
- Companion textbook: Process Improvement using Data
- Hosted experiment-design tool: factori.al
- Local docs build:
cd docs && make html
Citing process-improve
If you use this package in academic work, please cite it:
@software{dunn_process_improve,
author = {Dunn, Kevin G.},
title = {{process-improve: Multivariate Analysis for Process Improvement}},
year = {2026},
version = {v1.21.4},
url = {https://github.com/kgdunn/process-improve}
}
A CITATION.cff file is included, so GitHub renders a "Cite this
repository" button in the sidebar.
Contributing
Bug reports, feature requests, and pull requests are welcome. See CONTRIBUTING.md for development setup, testing, and code style. Bugs and feature requests can be filed on the issue tracker.
License
MIT - see LICENSE for details.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file process_improve-1.51.0.tar.gz.
File metadata
- Download URL: process_improve-1.51.0.tar.gz
- Upload date:
- Size: 1.9 MB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
0fa8ebc5b4b92e76c15dcdc78318e31c09166e37996fbb646306437f35d6e6af
|
|
| MD5 |
e554bad3b2824ff1a8982fad316459f2
|
|
| BLAKE2b-256 |
822d5a20dd9987ce2025506476acc66c81bf17bf829b7c0bab9e5caf8400a15c
|
Provenance
The following attestation bundles were made for process_improve-1.51.0.tar.gz:
Publisher:
publish.yml on kgdunn/process-improve
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
process_improve-1.51.0.tar.gz -
Subject digest:
0fa8ebc5b4b92e76c15dcdc78318e31c09166e37996fbb646306437f35d6e6af - Sigstore transparency entry: 2019479485
- Sigstore integration time:
-
Permalink:
kgdunn/process-improve@751c8d779d35fe4b5aa3ac003900065c006f601c -
Branch / Tag:
refs/heads/main - Owner: https://github.com/kgdunn
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish.yml@751c8d779d35fe4b5aa3ac003900065c006f601c -
Trigger Event:
workflow_dispatch
-
Statement type:
File details
Details for the file process_improve-1.51.0-py3-none-any.whl.
File metadata
- Download URL: process_improve-1.51.0-py3-none-any.whl
- Upload date:
- Size: 1.9 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
8403a0a7f3aa009040696882ff8a8f8075382defeaeaa6ee01f7fce230ba98cd
|
|
| MD5 |
f9d7895179b3c1c96e280b00b418441f
|
|
| BLAKE2b-256 |
28d741c0e19ff12ddb183a358e5d208a1df16c692e53c542f68f5e76f09f7627
|
Provenance
The following attestation bundles were made for process_improve-1.51.0-py3-none-any.whl:
Publisher:
publish.yml on kgdunn/process-improve
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
process_improve-1.51.0-py3-none-any.whl -
Subject digest:
8403a0a7f3aa009040696882ff8a8f8075382defeaeaa6ee01f7fce230ba98cd - Sigstore transparency entry: 2019479554
- Sigstore integration time:
-
Permalink:
kgdunn/process-improve@751c8d779d35fe4b5aa3ac003900065c006f601c -
Branch / Tag:
refs/heads/main - Owner: https://github.com/kgdunn
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish.yml@751c8d779d35fe4b5aa3ac003900065c006f601c -
Trigger Event:
workflow_dispatch
-
Statement type:
