FTIR Model Training And Prediction Pipeline

xpectra is a reusable pipeline for microplastic FTIR classification, built around three preprocessing routes:

• norm: combined_norm_data.csv.xz
• deriv1: combined_norm_deriv1_data.csv.xz
• deriv2: combined_norm_deriv2_data.csv.xz

The preprocessing defaults, reproduced by xpectra-process-raw, are: wavelet denoising, aspls baseline correction, atmospheric interpolation with zero, spectral_moments normalization, and Savitzky-Golay derivatives with window_length=15, polyorder=3.

When installed from PyPI, the reusable workflows are available as command-line tools:

xpectra-process-raw --out-dir processed_data
xpectra-train --routes all
xpectra-predict path/to/raw_spectra.csv --output predictions.csv
xpectra-study --list

The notebook reproduction drivers are grouped under xpectra-study. Install the optional study dependencies for the heavier workflows:

python -m pip install "xpectra[study]"

Process Raw Data

Regenerate the three route files (.csv.xz) consumed by the study workflows. --out-dir defaults to processed_data/; pass --dry-run to build and verify without writing:

xpectra-process-raw               # writes to processed_data/
xpectra-process-raw --out-dir some/other/dir
xpectra-process-raw --dry-run     # build + verify only, no files written

Reproduce Notebook Results

List the available study workflows:

xpectra-study --list

Run one workflow:

xpectra-study loso
xpectra-study nb2-corrected
xpectra-study nb2-figures
xpectra-study contrastive
xpectra-study nb3-analysis
xpectra-study nb3-figures
xpectra-study nb4-analysis
xpectra-study nb4-figures
xpectra-study report

Preprocessing route

The study workflows and the nb2/nb3/nb4 analysis and figure commands accept --route {norm,deriv1,deriv2} (default norm). Every result table, figure, and cached embedding is written with the route as a filename prefix (for example norm_deep_ssl_loso.csv, deriv1_figure3_main_contrastive_transfer.png), so runs on different routes never overwrite one another:

xpectra-study deep-ssl --route norm
xpectra-study deep-ssl --route deriv1

The classical sweeps (calibrate, loso, ood, ssl) instead carry a route column and cover all three routes in a single unprefixed file.

The corrected Notebook 2 workflow reuses current nb2-corrected-v3 result tables by default. To deliberately rerun the complete nested analysis, use xpectra-nb2-corrected --force. Publication figures can be regenerated with xpectra-nb2-figures (also --route-aware).

The Notebook 3 publication workflow uses the cached contrastive encoder and embeddings from xpectra-study contrastive. Generate its canonical diagnostic tables with xpectra-nb3-analysis, then build the main and supplementary figure sets with xpectra-nb3-figures.

Notebook 4 validates robust field-spectrum clusters against polymer identity, carbonyl index, environmental status, and source-study provenance. Generate its canonical tables with xpectra-nb4-analysis and the publication figure sets with xpectra-nb4-figures.

Train And Save Models

Train all 41 Xpectrass models for all three routes:

xpectra-train --routes all

Train selected models only:

xpectra-train \
  --routes norm deriv1 deriv2 \
  --model "XGBoost (100)" \
  --model "Random Forest (100)"

Saved artifacts are written to:

models/
  norm/<model>.joblib
  deriv1/<model>.joblib
  deriv2/<model>.joblib
  training_summary.csv

Each artifact contains the fitted estimator, StandardScaler, LabelEncoder, class names, preprocessing settings, and exact training wavenumber order.

Predict From Raw CSV

Predict through all saved route artifacts:

xpectra-predict path/to/raw_spectra.csv \
  --routes all \
  --models-dir models \
  --output predictions.csv

Use only selected models:

xpectra-predict path/to/raw_spectra.csv \
  --routes norm deriv1 \
  --model "XGBoost (100)" \
  --output predictions.csv

If your CSV is already normalized, skip raw denoise/baseline/normalization and only interpolate/derive:

xpectra-predict path/to/normalized_spectra.csv \
  --input-stage normalized \
  --routes all \
  --output predictions.csv

If your CSV is already route-ready on the expected wavenumber grid:

xpectra-predict path/to/route_ready.csv \
  --input-stage route \
  --routes norm \
  --output predictions.csv

Override Preprocessing

The prediction CLI lets you override the preprocessing defaults:

xpectra-predict path/to/raw_spectra.csv \
  --denoising-method wavelet \
  --baseline-method aspls \
  --normalization-method spectral_moments \
  --interpolate-method zero

For absorbance-like inputs that should skip Xpectrass auto-conversion:

xpectra-predict path/to/raw_spectra.csv \
  --force-absorbance

For spectra stored on a 0-100 absorbance-like scale:

xpectra-predict path/to/raw_spectra.csv \
  --force-absorbance \
  --absorbance-scale-factor 100