FTIR-Prep: FTIR Preprocessing Framework

A modular and extensible framework for optimizing FTIR preprocessing pipelines for disease diagnosis.

🚀 Features

• Modular: Component-based reusable architecture
• Extensible: Easy addition of new preprocessing techniques
• Robust Pipeline Search: RobustPipelineSearch finds a preprocessing pipeline that performs consistently across a pool of classifier families (not just tuned to one), with time/trial and memory budgets, parallelization, and checkpointing to disk
• Robust Validation: Support for group-based cross-validation
• Configurable: Flexible pipeline configuration system
• Documented: Complete documentation with practical examples

📋 Supported Preprocessing Techniques

🔧 Baseline Correction

• Rubberband: Automatic correction using rubberband algorithm
• Polynomial: Correction using configurable order polynomials (1-6)
• Whittaker: Penalized least squares smoothing with lambda parameter
• ALS: Asymmetric Least Squares with lambda and p parameters
• ArPLS: Adaptive reweighted penalized least squares
• DrPLS: Doubly reweighted penalized least squares
• GCV Spline: Generalized cross-validation spline smoothing
• Gaussian Process: Baseline correction using Gaussian processes

📊 Normalization

• Min-Max: Individual Min-Max spectrum normalization
• Vector: L1, L2, or maximum normalization
• Amida I: Normalization based on amide I band peak (1600-1700 cm⁻¹)
• Area: Area under curve normalization

🎯 Smoothing

• Savitzky-Golay: Polynomial filter with configurable parameters
• Wavelets: Denoising using Daubechies wavelets (db2, db3, db4)
• Local Polynomial: LOWESS smoothing with configurable bandwidth
• Whittaker: Penalized least squares smoothing
• GCV Spline: Generalized cross-validation spline smoothing
• Flat: Flat window convolution smoothing
• Hanning: Hanning window convolution smoothing

📈 Derivatives

• First Derivative: First derivative calculation via Savitzky-Golay (order 1)
• Second Derivative: Second derivative calculation via Savitzky-Golay (order 2)

✂️ Wavelength Truncation

• Fingerprint Region: Keep only fingerprint region (900-1800 cm⁻¹)
• Fingerprint + Lipids: Keep fingerprint and lipids regions (900-1800, 2800-3050 cm⁻¹)

🔍 Model Explainability

• SHAP Analysis: Feature importance analysis using SHAP values

🏗️ Architecture

ftir_prep/
├── core/                    # Core functionalities
│   ├── pipeline.py         # Preprocessing pipeline
│   ├── evaluator.py        # Pipeline evaluation
│   └── explainer.py        # SHAP explainability analysis
├── preprocessing/           # Preprocessing techniques
│   ├── baseline.py         # Baseline correction
│   ├── normalization.py    # Normalization
│   ├── smoothing.py        # Smoothing
│   ├── derivatives.py      # Derivative calculation
│   └── truncation.py      # Wavelength truncation
├── optimization/            # Automatic optimization
│   ├── optuna_optimizer.py # Optuna integration (single-classifier search engine)
│   └── robust_search.py    # RobustPipelineSearch (recommended entry point)
├── utils/                   # Utilities
│   └── data_loader.py      # Data loading
└── config/                  # Configurations
    └── settings.py         # Default parameters

🚀 Installation

Requirements

• Python 3.8+
• pip (usually included with Python)

Installation via PyPI (Recommended - Simplest)

pip install ftir-prep

📖 Basic Usage

1. Data Loading

1.1 Separates into groups to guarantee that data from the same patient will be in the same fold in a future classification task

from ftir_prep import FTIRDataLoader

# Load FTIR data
data_loader = FTIRDataLoader(
    data_path="ftir_data.dat",
    wavenumbers_path="wavenumbers.dat"
)

X, y, wavenumbers = data_loader.load_data()

# Create groups var that will be used in classification task to indicate that patient's data must be in the same fold
groups = data_loader.create_groups(instances_per_group=3)

1.2 Slices the data to use only one spectra per patient. Data must be ordered by patient

from ftir_prep import FTIRDataLoader

# Load FTIR data
data_loader = FTIRDataLoader(
    data_path="ftir_data.dat",
    wavenumbers_path="wavenumbers.dat"
)

X, y, wavenumbers = data_loader.load_data(slice_size = 3) #use one of the triplicated spectra per patient

2. Pipeline Creation

from ftir_prep import FTIRPipeline, PipelineBuilder

# Using direct configuration
pipeline = FTIRPipeline()
pipeline.add_step('truncation', 'fingerprint_lipids')
pipeline.add_step('baseline', 'polynomial', polynomial_order=2)
pipeline.add_step('normalization', 'vector')


# Using PipelineBuilder (Fluent API)
pipeline = (PipelineBuilder()
            .add_truncation('fingerprint_lipids')
            .add_baseline('rubberband')
            .add_normalization('minmax')
            .add_smoothing('savgol', polyorder=2)
            .add_derivative('savgol',order=1)
            .build())

3. Execution and Evaluation

from ftir_prep import PipelineEvaluator

# Process data
X_processed, wavenumbers_processed = pipeline.process(X, wavenumbers)

# Evaluate pipeline
evaluator = PipelineEvaluator(classifier=None, # use default Random Forest
                              cv_method='StratifiedGroupKFold', # cross-validation strategy
                              cv_params={'n_splits': 3, #folds
                                        'shuffler': False,
                                        'random_state': 42})
results = evaluator.evaluate_pipeline(pipeline,
                                      X, y,
                                      groups, # groups var created previously
                                      wavenumbers=wavenumbers
)

print(f"Accuracy: {results['mean_accuracy']:.4f} ± {results['std_accuracy']:.4f}")

4. Automatic Optimization

RobustPipelineSearch is the recommended way to find an optimal preprocessing pipeline: it runs an Optuna search per classifier in a pool, cross-validates the resulting candidates on the other classifiers in that pool, and picks the pipeline that is most consistently good across all of them — more likely to generalize than one tuned for a single model.

Only X, y and wavenumbers are required. Everything else — including the classifier pool — falls back to sensible defaults:

from ftir_prep import RobustPipelineSearch, load_pipeline

selector = RobustPipelineSearch(X, y, wavenumbers)

best_pipeline, final_report = selector.run(n_trials=1000)  # or timeout=<seconds>

print(best_pipeline.steps)
print(final_report)  # validation score per classifier

Pass groups=groups with cv_method='StratifiedGroupKFold' (see Data Loading above) for group-based cross-validation instead of the StratifiedKFold default.

By default RobustPipelineSearch searches against a predefined pool of ~6-7 classifiers (LogisticRegression, SVC, ExtraTreesClassifier, KNeighborsClassifier, MLPClassifier, GaussianNB, and XGBClassifier if xgboost is installed) and picks the pipeline that generalizes best across all of them. Pass classifiers to use a single model instead (no cross-classifier robustness step then — just the best pipeline found for that model) or your own custom pool:

from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC

# Single model
selector = RobustPipelineSearch(
    X, y, wavenumbers,
    classifiers={"RandomForest": RandomForestClassifier(random_state=42)},
)

# Custom pool (with exactly 2 classifiers, candidates are ranked by the
# plain average of both scores instead of the usual robustness formula)
selector = RobustPipelineSearch(
    X, y, wavenumbers,
    classifiers={
        "RandomForest": RandomForestClassifier(random_state=42),
        "LogisticRegression": LogisticRegression(random_state=42, max_iter=1000),
        "SVC": SVC(),
    },
)

It also exposes time, trial and memory budgets, parallelization, and checkpointing to disk — useful for long, unattended runs:

selector = RobustPipelineSearch(
    X, y, wavenumbers,
    core_budget=4,       # cores for parallel search (default: sequential)
)

best_pipeline, final_report = selector.run(
    timeout=1800,             # wall-clock seconds, instead of n_trials
    memory_budget_mb=8192,    # optional RAM ceiling
    output_dir="search_results",  # checkpoints every stage to disk as it runs
)

# Reload the winning pipeline later, in another session
best_pipeline = load_pipeline("search_results/best_pipeline/best_pipeline.json")

n_trials and timeout are mutually exclusive. For finer control, each stage (search(), select_candidates(), cross_validate_candidates(), rank_candidates(), select_best()) can also be called individually instead of run() — see examples/pipeline_search/pipeline_search.py.

5. Model Explainability

from ftir_prep import FTIRExplainer

# Create explainer
explainer = FTIRExplainer(classifier=your_classifier)

# Analyze feature importance with SHAP
# It will save in output_dir a csv and a png with feature importance data
results = explainer.explain_model(
    X_processed, y, groups,
    split_method='stratified_group',
    feature_names=wavenumbers_processed,
    output_dir="shap_analysis"
)

🔬 Practical Examples

Pipeline Creation Examples

# Direct configuration example
python3 examples/create_pipeline/direct_configuration.py

# PipelineBuilder (Fluent API) example
python3 examples/create_pipeline/pipeline_builder.py

Pipeline Comparison Example

# Compare different preprocessing strategies
python3 examples/compare_pipelines/compare_pipelines.py

Pipeline Optimization Example

# Automatic pipeline optimization
python3 examples/pipeline_search/pipeline_search.py

Pipeline Loading Example

# Load and use saved pipelines
python3 examples/read_pipeline_from_file/read_pipeline_file.py

SHAP Explainability Example

# Feature importance analysis with SHAP
python3 examples/shap_analysis/explainer_example.py

🎯 Use Cases

Disease Diagnosis

• Analysis of FTIR spectra from biological samples
• Biomarker identification
• Automatic sample classification

Scientific Research

• Methodology comparison
• Protocol optimization
• Result validation

📚 Documentation

• Docstrings: Complete inline documentation
• Examples: Functional example code

👥 Authors

• Lucas Mendonça - Initial development - GitHub

⭐ If this project was useful to you, consider giving it a star on GitHub!