process-improve 1.17.0

Designed Experiments; Latent Variables (PCA, PLS, multivariate methods with missing data); Process Monitoring; Batch data analysis.

Process Improvement using Data

A pragmatic Python toolkit for industrial process data - multivariate analysis, designed experiments, and process monitoring, in one place.

What is this?

process-improve is the companion package to the online textbook Process Improvement using Data, and powers the statistical engine behind factori.al. It bundles the methods practitioners actually reach for on real plant and lab data - PCA / PLS with proper missing-data handling, designed experiments with a multi-stage strategy recommender, control charts, and batch-data tooling - behind an API that is sklearn-compatible where it makes sense and pandas-native throughout.

Highlights

🧪 Designed Experiments

• Full-factorial, fractional-factorial, and response-surface designs (built on pyDOE3)
• A DOE strategy recommender that plans a complete multi-stage program - screening → optimization → confirmation - from ~50 deterministic rules, with budget-aware allocation and domain-specific advice for fermentation, cell culture, pharma, and 5 other domains
• ANOVA, main-effects plots, linear-model fitting, and response optimization

📊 Latent Variable Methods

• PCA with SVD and NIPALS algorithms, plus missing-data via Trimmed Score Regression
• PLS regression with a fully sklearn-compatible API
• TPLS - PLS for T-shaped data structures
• Diagnostics: Hotelling's T², SPE, score contributions, VIP, and ESD-based outlier detection
• Cross-validation: PRESS / Wold's component selection, PLS RMSECV with validated explained variance, and PLS beta-coefficient error bars
• Interactive Plotly score, loading, SPE, and T² plots, bound directly to fitted models

📈 Process Monitoring

• Shewhart, CUSUM, and Holt-Winters control charts (regular and robust variants)
• Process-capability index Cpk

🔄 Batch Data Analysis

• DTW-based batch alignment, reference-batch selection, resampling
• 15+ batch feature extractors (mean, slope, area, elbow, rupture, crossings, robust variants, …)
• Format conversions between wide, melted, and dict-of-frames batch layouts

📐 Univariate & Robust Regression

• t-tests (paired, independent, plus DataFrame-aware helpers)
• ESD outliers, Sn estimator, MAD, normality tests, variance decomposition
• Robust regression: repeated-median slope and friends, for outlier-resistant fits

🎨 Visualization

• Plotly-backed plots that attach to fitted PCA / PLS models
• A backend-agnostic ChartSpec layer with Plotly and ECharts adapters
• DOE-specific plots: main-effects, design visualization

Installation

pip install process-improve

Requires Python 3.10 or newer.

Quick start

PCA - Principal Component Analysis

import pandas as pd
from process_improve.multivariate.methods import PCA, MCUVScaler

X = pd.read_csv("your_data.csv", index_col=0)
X_scaled = MCUVScaler().fit_transform(X)

pca = PCA(n_components=3).fit(X_scaled)
print(pca.r2_cumulative_)        # cumulative R² per component
pca.score_plot()                  # interactive Plotly plot

PLS - Projection to Latent Structures

from process_improve.multivariate.methods import PLS, MCUVScaler

# Scale X and Y separately
scaler_x = MCUVScaler().fit(X)
scaler_y = MCUVScaler().fit(Y)
X_s, Y_s = scaler_x.transform(X), scaler_y.transform(Y)

# Fit a PLS model
pls = PLS(n_components=3).fit(X_s, Y_s)

# Inspect results
print(pls.scores_)  # X scores (N x A)
print(pls.beta_coefficients_)  # Regression coefficients (K x M)
print(pls.r2_cumulative_)  # Cumulative R² for Y

# Predict new observations
result = pls.predict(scaler_x.transform(X_new))
print(result.y_hat)  # Predicted Y values
print(result.spe)  # SPE for new data
print(result.hotellings_t2)  # Hotelling's T² for new data

# Detect outliers and analyze contributions
outliers = pls.detect_outliers(conf_level=0.95)
contrib = pls.score_contributions(pls.scores_.iloc[0].values)

# Variable importance
print(pls.vip())  # VIP scores per X variable

# Cross-validated component selection
cv_select = PLS.select_n_components(X_s, Y_s, max_components=6)
print(cv_select.n_components)  # Recommended number of components
print(cv_select.rmsecv)        # RMSECV per component count

# Cross-validation with beta coefficient error bars
cv = pls.cross_validate(X_s, Y_s, cv="loo")
print(cv.beta_mean)       # Mean beta across LOO resamples
print(cv.beta_ci_lower)   # Lower 95% CI for each beta
print(cv.beta_ci_upper)   # Upper 95% CI for each beta
print(cv.significant)     # Which betas are significantly != 0
print(cv.q_squared)       # Cross-validated R² (Q²)
print(cv.rmse_cv)         # Cross-validated RMSE per Y variable

DOE - multi-stage experimental strategy

from process_improve.experiments.factor import Factor, Response
from process_improve.experiments.strategy import recommend_strategy

factors = [
    Factor(name="Temperature", low=25, high=40, units="degC"),
    Factor(name="pH", low=5.0, high=7.5),
    Factor(name="Glucose", low=10, high=50, units="g/L"),
]
strategy = recommend_strategy(
    factors=factors,
    responses=[Response(name="Yield", goal="maximize", units="g/L")],
    budget=40,
    domain="fermentation",
)
for s in strategy["stages"]:
    print(s["stage_number"], s["design_type"], s["estimated_runs"])

Longer, fully-worked versions of each example live in the Quickstart guide and the process_improve/notebooks_examples/ folder.

New to designed experiments? The Applied DoE tutorial is an eight-module worked-solution series that mirrors the 12-week DoE short course and shows the same workflow in Python with process-improve end to end - from a 2x2 first design through factorial scaling, fractional factorials, blocking, and 1-D / 2-D response-surface optimization.

API design

PCA and PLS follow scikit-learn conventions: fit() returns self, fitted attributes end with a trailing underscore (scores_, loadings_, spe_, hotellings_t2_, r2_cumulative_, …), and predict() returns an sklearn.utils.Bunch with named fields (y_hat, spe, hotellings_t2, …). Inputs are accepted as pandas.DataFrame, and index/column labels are preserved through fit and transform.

Documentation & learning resources

• API reference & user guide: https://kgdunn.github.io/process-improve/
• Applied DoE tutorial (8 modules): https://kgdunn.github.io/process-improve/applied_doe/index.html
• Companion textbook: Process Improvement using Data
• Hosted experiment-design tool: factori.al
• Local docs build: cd docs && make html

Contributing

Bug reports and feature requests are welcome on the issue tracker.

License

MIT - see LICENSE for details.