process-improve

Multivariate analysis, designed experiments, and process monitoring for Python. Built for the chemometrics, manufacturing, and pharma workflows where you need to know not just what fits, but is this observation normal, which variable moved, and how sure am I?

New here? The architecture overview (source) is the map of the codebase - package layout, the estimator stack, and the MCP tool layer.

What's new

The last few releases extend process-improve from offline model-building into end-to-end, on-line workflows. Highlights (full history in CHANGELOG.md):

Design, rather than predict. PLS.invert() (v1.61) solves for the inputs that reach a target quality, and returns the null space of equally valid recipes. OPLS reaches the same designs by separating that freedom while fitting. See the user guide or the longer book chapter.
Models that keep up with a drifting process. AdaptivePCA and AdaptivePLS (v1.55) fit once, then stream one observation at a time, re-learning the correlation structure and reporting how far the process has drifted, in units of components.
A DOE engine that goes past textbook designs. OMARS designs, D-/I-/A-optimal designs, fractional-cube CCDs, design augmentation (fixed_runs=), and an evaluate_design suite scoring any design on efficiency, aliasing, and prediction variance.
Sensory & descriptive panel analysis (process_improve.sensory): validate a panel, flag inconsistent assessors with the Mixed Assessor Model, and relate attributes to product covariates.
Robust regression (process_improve.regression): repeated-median and Theil-Sen estimators for data with outliers, plus OLS and fit_robust_lm.

What it does

process-improve provides production-grade implementations of the methods practitioners actually use on real plant and lab data:

PCA with SVD and NIPALS, plus native missing-value handling via Trimmed Score Regression
PLS regression with a fully sklearn-compatible API, VIP scores, and cross-validated diagnostics
TPLS - PLS for T-shaped (multi-block) data structures
Model inversion - PLS.invert() and OPLS solve for the inputs that reach a target quality, return the null space of equally valid designs, and report how far each design sits from the data that support it
Adaptive PCA / PLS - recursive, self-updating models for on-line process monitoring and soft sensing; they follow a drifting process one observation at a time and report how far it has moved
Outlier detection combining Hotelling's T² and SPE with an ESD-based test
Designed experiments - full-factorial, fractional-factorial, and response-surface designs; OMARS and D-/I-/A-optimal designs; a design-quality scorer (evaluate_design); and a multi-stage DOE strategy recommender
Process monitoring - Shewhart, CUSUM, and Holt-Winters control charts
Batch data analysis - alignment, feature extraction, and multivariate batch monitoring (MBPCA / MBPLS)
Sensory & descriptive panel analysis - panel validation, the Mixed Assessor Model, and attribute-to-product relations with a genuine-vs-proxy separation
Robust regression - repeated-median and Theil-Sen estimators for data with outliers
Interactive Plotly diagnostics bound directly to every fitted model

Outputs are pandas-native: scores, loadings, and predictions keep your row and column labels.

It is the companion package to the online textbook Process Improvement using Data.

Works alongside scikit-learn

process-improve is designed to sit next to scikit-learn, not replace it. It follows the same conventions (fit, predict, score, the _ suffix on fitted attributes), so its estimators drop straight into Pipeline, GridSearchCV, and cross_val_score. What it adds is the process-analytics layer on top: the diagnostics that tell you whether a new observation is normal, which variable moved, and how confident the prediction is.

Capability	scikit-learn	process-improve
PCA, PLS with sklearn-style API	✓	✓
Missing-data fitting (NIPALS / TSR)	-	✓
Hotelling's T² + SPE outlier limits	-	✓
Variable-level score contributions	-	✓
Cross-validated coefficient confidence intervals	-	✓
Multi-block models (TPLS)	-	✓
Model inversion: design inputs for a target	-	✓
On-line / adaptive monitoring (recursive PCA/PLS)	-	✓
Designed experiments, incl. OMARS & optimal	-	✓
Control charts (Shewhart / CUSUM / Holt-Winters)	-	✓
Batch process monitoring (MBPCA / MBPLS)	-	✓
Plotly diagnostics built in	-	✓
Labeled `DataFrame` outputs	partial	✓

Installation

pip install process-improve                    # core (numpy, pandas, sklearn, statsmodels, patsy, pydantic, pyyaml, tqdm)
pip install 'process-improve[plotting]'        # adds matplotlib, plotly, seaborn, ridgeplot
pip install 'process-improve[expt]'            # adds pyDOE3 (designed experiments / DOE)
pip install 'process-improve[batch]'           # adds openpyxl, scikit-image (batch process data IO)
pip install 'process-improve[mcp]'             # adds the MCP server runtime
pip install 'process-improve[fast]'            # adds numba (JIT speedups for batch alignment)
pip install 'process-improve[all]'             # everything above (the pre-1.24.11 closure)

Requires Python 3.10 or newer. The core install pulls in numpy, pandas, scikit-learn, statsmodels, patsy, pydantic, pyyaml, and tqdm (scipy arrives transitively via scikit-learn and statsmodels). Heavier optional surfaces (plotting, designed experiments, batch IO, MCP server, numba JIT) live in extras so a caller who only needs, say, detect_multivariate_outliers does not have to install Plotly or numba.

Use it from Claude

The designed-experiments tooling ships as a Claude Skill, so you can plan, generate, verify and analyse experiments in your own Claude account with no server involved:

/plugin marketplace add kgdunn/process-improve
/plugin install doe-designer@process-improve

The skill's first rule is that a design matrix is never written out by the model: it is generated from a catalogue and then verified, because a language model asked to produce a fractional factorial will often return one that looks right and is a lower resolution than it claims. See skills/README.md for the other install routes (local folder, claude.ai upload) and for the MCP server, which exposes the same tool registry without the workflow guidance.

Quick start

PCA - Principal Component Analysis

import pandas as pd
from process_improve.multivariate.methods import PCA, MCUVScaler

X = pd.read_csv("your_data.csv", index_col=0)
X_scaled = MCUVScaler().fit_transform(X)

pca = PCA(n_components=3).fit(X_scaled)
print(pca.r2_cumulative_)         # cumulative R² per component
pca.score_plot()                  # interactive Plotly figure

# Flag outliers using combined T² and SPE limits at 95% confidence
outliers = pca.detect_outliers(conf_level=0.95)

# Which variables drove the first observation off?
contrib = pca.score_contributions(pca.scores_.iloc[0].values)

PLS - Projection to Latent Structures

from process_improve.multivariate.methods import PLS, MCUVScaler

# Scale X and Y separately
scaler_x = MCUVScaler().fit(X)
scaler_y = MCUVScaler().fit(Y)
X_s, Y_s = scaler_x.transform(X), scaler_y.transform(Y)

pls = PLS(n_components=3).fit(X_s, Y_s)
print(pls.beta_coefficients_)     # regression coefficients (K x M)
print(pls.r2_cumulative_)         # cumulative R² for Y
print(pls.vip())                  # VIP scores per X variable

# Predict new observations (sklearn-compatible: returns just y_hat)
y_pred = pls.predict(scaler_x.transform(X_new))

# Predict with full per-row diagnostics (scores, T², SPE, plus y_hat)
result = pls.diagnose(scaler_x.transform(X_new))
result.y_hat                      # point predictions
result.spe                        # squared prediction error
result.hotellings_t2              # Hotelling's T² for new observations

# Cross-validated component selection: raw blocks in, each fold scales itself
cv_select = PLS.select_n_components(X, Y, max_components=6)
print(cv_select.n_components)     # recommended number of components
print(cv_select.rmsecv)           # RMSECV per component count

# Cross-validation with beta-coefficient confidence intervals
cv = pls.cross_validate(X_s, Y_s, cv="loo")
print(cv.beta_ci_lower, cv.beta_ci_upper)   # 95% CI for each beta
print(cv.significant)                       # betas significantly != 0
print(cv.q_squared)                         # cross-validated R² (Q²)

Model inversion - design the inputs for a quality you choose

from process_improve.multivariate.methods import PLS, OPLS

# X: acetic acid, H2S and lactic acid in 26 cheddar cheeses; y: their taste score
pls = PLS(n_components=2).fit(X, y)

design = pls.invert(y_desired=20.9)
print(design.x_new.round(2).to_list())   # [5.52, 5.56, 1.4], in the original units
print(design.hotellings_t2.round(2))     # 0.06: well inside the calibration data
print(design.null_space_dimension)       # 1: a line of designs, not a single recipe

# Walk that line. The recipe changes; the predicted taste does not.
for step in (-1.0, 0.0, 1.0):
    print(pls.invert(20.9, null_space_coordinates=[step]).x_new.round(2).to_list())
# [4.95, 6.1, 1.33]
# [5.52, 5.56, 1.4]
# [6.09, 5.02, 1.46]

# O-PLS separates that freedom while fitting, so inversion becomes one division.
opls = OPLS(n_orthogonal_components=1).fit(X, y)
print(opls.invert(y_desired=20.9).x_new.round(2).to_list())   # [5.46, 5.62, 1.39]

Both routes describe the same set of designs, and differ only in which point on it they report. The freedom is what you spend on cost, supply, or a regulatory window; the hotellings_t2 is what tells you when a design has walked past the evidence.

DOE - multi-stage experimental strategy

from process_improve.experiments.factor import Factor, Response
from process_improve.experiments.strategy import recommend_strategy

factors = [
    Factor(name="Temperature", low=25, high=40, units="degC"),
    Factor(name="pH", low=5.0, high=7.5),
    Factor(name="Glucose", low=10, high=50, units="g/L"),
]
strategy = recommend_strategy(
    factors=factors,
    responses=[Response(name="Yield", goal="maximize", units="g/L")],
    budget=40,
    domain="fermentation",
)
for s in strategy["stages"]:
    print(s["stage_number"], s["design_type"], s["estimated_runs"])

One-shot optimal & OMARS designs

Ask for a ready-to-run design table and score it, in two lines:

from process_improve.experiments import Factor, generate_design, evaluate_design

factors = [
    Factor(name="A", low=-1, high=1),
    Factor(name="B", low=-1, high=1),
    Factor(name="C", low=-1, high=1),
]

# An OMARS design: main effects clear of every second-order term
design = generate_design(factors, design_type="omars")

# Or a run-budgeted D-optimal design, then grade its quality
d_opt = generate_design(factors, design_type="d_optimal", budget=14)
print(evaluate_design(d_opt, metric="all"))   # D/I/G-efficiency, aliasing, prediction variance

On-line monitoring with Adaptive PCA

A static model goes stale the moment the process drifts. AdaptivePCA starts from an initial fit, then keeps learning as data streams in - flagging faults and reporting exactly how far the process has moved from where it was trained:

from process_improve.multivariate import AdaptivePCA

# Seed on a block of known-good ("common cause") data
monitor = AdaptivePCA(n_components=3).fit(X_reference)

# Feed live observations one row at a time
for _, row in X_stream.iterrows():
    result = monitor.update(row.to_numpy())
    if not result.in_control:
        print(f"Out-of-control point: SPE={result.spe:.2f}, T²={result.hotellings_t2:.2f}")

# How far has the model drifted from its training subspace? (in units of components)
print(monitor.distance_.tail())
print(monitor.center_shift_.tail())   # operating-point migration, in training-SD units

AdaptivePLS does the same for regression and soft sensing, and handles infrequently-sampled responses: the X-space model adapts every step while the regression part waits for the next lab result.

Longer, fully-worked versions of each example live in the Quickstart guide and the examples/ folder.

New to designed experiments? The Applied DoE tutorial is an eight-module worked-solution series.

API design

PCA and PLS follow scikit-learn conventions: fit() returns self, fitted attributes end with a trailing underscore (scores_, loadings_, spe_, hotellings_t2_, r2_cumulative_, ...), and predict() returns an sklearn.utils.Bunch with named fields (y_hat, spe, hotellings_t2, ...). Inputs are accepted as pandas.DataFrame, and index/column labels are preserved through fit and transform.

Documentation & learning resources

API reference & user guide: https://kgdunn.github.io/process-improve/
Applied DoE tutorial (8 modules): https://kgdunn.github.io/process-improve/applied_doe/index.html
Model inversion, end to end: the user-guide page, and the book chapter it is drawn from, Using a PLS model backwards
Companion textbook: Process Improvement using Data
Local docs build: cd docs && make html

Citing process-improve

If you use this package in academic work, please cite it. The CITATION.cff file carries the current version and release date, and GitHub renders a "Cite this repository" button in the sidebar with ready-made BibTeX and APA entries:

@software{dunn_process_improve,
  author  = {Dunn, Kevin G.},
  title   = {{process-improve: Multivariate Analysis for Process Improvement}},
  year    = {2026},
  url     = {https://github.com/kgdunn/process-improve}
}

Add the version field from CITATION.cff (or the release tag you installed) when citing a specific version.

Contributing

Bug reports, feature requests, and pull requests are welcome. See CONTRIBUTING.md for development setup, testing, and code style. Bugs and feature requests can be filed on the issue tracker.

License

MIT - see LICENSE for details.

Metadata

Release files for process-improve 1.94.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for process-improve 1.94.0
File	Size	Uploaded
process_improve-1.94.0.tar.gz	2.1 MB	Sep 13, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for process-improve 1.94.0
File	Interpreter	ABI	Platform	Reset
process_improve-1.94.0-py3-none-any.whl 2.2 MB Sep 13, 2026	Python 3	none	any	Details

Total release size: 4.4 MB

Release files / process_improve-1.94.0.tar.gz

Download URL	process_improve-1.94.0.tar.gz
Size	2.1 MB
Tags	Source
SHA-256 checksum How to use checksums	`34186a705d505afc8711f666e4ca32bd40927cd1ddf9fba9ba8158b316fe769a`
BLAKE2b-256 checksum How to use checksums	`210ffe5bc72795afc56f81f1911557839fe8ad7eedde1e5d4c3751a60d0d5424`
Upload date	Sep 13, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/kgdunn/process-improve
Publishing commit: github.com/kgdunn/process-improve/tree/47685e12d915cde22f439680fdaa288a7f0f3d41

Workflow

Publishing configuration: github.com/kgdunn/process-improve/blob/47685e12d915cde22f439680fdaa288a7f0f3d41/.github/workflows/publish.yml
Publishing logs: github.com/kgdunn/process-improve/actions/runs/34784803089/attempts/1

Artifact

Subject: process_improve-1.94.0.tar.gz
SHA-256 checksum: 34186a705d505afc8711f666e4ca32bd40927cd1ddf9fba9ba8158b316fe769a Verifying attestations

Release files / process_improve-1.94.0-py3-none-any.whl

Download URL	process_improve-1.94.0-py3-none-any.whl
Size	2.2 MB
Tags	Python 3
SHA-256 checksum How to use checksums	`567ac4fbfdcdb416bc82e3fee142286594a9717c0ec34684d1187bd779816555`
BLAKE2b-256 checksum How to use checksums	`be7353f9e348efd06baa83407a0c54590febf19762e41b7b34cf516d67ee8c0c`
Upload date	Sep 13, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/kgdunn/process-improve
Publishing commit: github.com/kgdunn/process-improve/tree/47685e12d915cde22f439680fdaa288a7f0f3d41

Workflow

Publishing configuration: github.com/kgdunn/process-improve/blob/47685e12d915cde22f439680fdaa288a7f0f3d41/.github/workflows/publish.yml
Publishing logs: github.com/kgdunn/process-improve/actions/runs/34784803089/attempts/1

Artifact

Subject: process_improve-1.94.0-py3-none-any.whl
SHA-256 checksum: 567ac4fbfdcdb416bc82e3fee142286594a9717c0ec34684d1187bd779816555 Verifying attestations