chemotools 0.0.11

Package to integrate chemometrics in scikit-learn pipelines

Welcome to chemotools! 🖖

This project is an implementation of spectral preprocessing techniques integrated with the widely popular sklearn API, providing you with an easy-to-use toolkit for analyzing and manipulating your data. With this integration, you can easily apply techniques such as spectral derivative, scatter correction or baseline removal to your datasets.

The goal of this project is to provide a comprehensive and user-friendly package for spectral preprocessing, with a focus on making it accessible to a wide range of users, from data scientists to researchers to curious learners. By integrating these techniques with the powerful sklearn API, I enable users to easily build machine learning and chemometric models on top of preprocessed data, making it possible to identify patterns and make predictions with greater accuracy.

In this repository, you will find a range of tools and resources for using spectral preprocessing techniques with the sklearn API, including code samples, documentation, and examples. I encourage contributions from the community, whether through bug reports, feature requests, or pull requests, to help us make this project even better.

In addition, I take great care to ensure that all functions are thoroughly unit tested for both API compatibility and functionality. I believe that unit testing is an essential part of any software development process, and it is particularly important for a project like this, which aims to provide reliable data analysis tools. I hope that this project will serve as a useful resource for anyone interested in learning more about spectral preprocessing and machine learning.

Thank you for your interest in this project, and I hope you find it useful for your data analysis and machine learning needs.

Table of contents

• Installation
• Integration with scikit-learn
• Scatter
  • Multiplicative scatter correction
  • Standard normal variate
  • Extended multiplicative scatter correction (COMING SOON)
• Derivative
  • Savitzky-Golay derivative
  • William Norris derivative
• Baseline
  • Linear baseline correction
  • Polynomial baseline correction
  • Cubic spline baseline correction
  • AirPls
  • Non-negative
• Scale
  • Min-max scaler
  • L-Norm scaler
• Smooth
  • Savitzky-Golay smoothing
  • Whittaker smoother
  • Mean filter
  • Median filter

Installation 🚀

This package is available on PyPI and can be installed using pip:

pip install chemotools

Integration with scikit-learn ✨

All preprocessing techniques in this package are compatible with scikit-learn and can be used in pipelines. For example, the following code creates a pipeline that performs multiplicative scatter correction, followed by a min-max scaling and a Savitzky-Golay smoothing:

from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

pipeline = make_pipeline(AirPls(), MultiplicativeScatterCorrection(), StandardScaler(with_std=False)) 
spectra_transformed = pipeline.fit_transform(spectra)

Scatter

This package contains three common algorithms for scatter correction in spectroscopy:

• Multiplicative scatter correction (MSC)
• Standard normal variate (SNV)
• Extended multiplicative scatter correction (EMSC)

Multiplicative scatter correction

Multiplicative scatter correction (MSC) is a preprocessing technique in spectroscopy that corrects for the influence of light scattering on spectral measurements by dividing each spectrum by a scatter reference spectrum. The current implementation, accepts three types of reference spectra:

• The mean spectrum of the dataset (default).
• The median spectrum of the dataset.
• A single spectrum that is used to correct all spectra in the dataset.

Usage example for a single reference spectrum:

Usage example for the mean spectrum:

from chemotools.scatter import MultiplicativeScatterCorrection

msc = MultiplicativeScatterCorrection()
spectra_msc = msc.fit_transform(spectra)

Usage example for the median spectrum:

from chemotools.scatter import MultiplicativeScatterCorrection

msc = MultiplicativeScatterCorrection(use_median=True)
spectra_msc = msc.fit_transform(spectra)

Usage example for a single reference spectrum:

from chemotools.scatter import MultiplicativeScatterCorrection

msc = MultiplicativeScatterCorrection(reference=reference_spectrum)
spectra_msc = msc.fit_transform(spectra)

Standard normal variate

Standard normal variate (SNV) is a preprocessing technique in spectroscopy that adjusts for baseline shifts and variations in signal intensity by subtracting the mean and dividing by the standard deviation of each spectrum.

Usage example for a single reference spectrum:

from chemotools.scatter import StandardNormalVariate

snv = StandardNormalVariate()
spectra_snv = snv.fit_transform(spectra)

Extended multiplicative scatter correction

Extended multiplicative scatter correction (EMSC) is a preprocessing technique in spectroscopy that corrects for the influence of light scattering and instrumental drift by fitting a mathematical model to a reference spectrum and using it to normalize all spectra in the dataset.

An implementation of the EMSC will be available soon 🤓.

Derivatives

This package contains two common algorithms for calculating derivatives in spectroscopy:

• Savitzky-Golay derivative
• William Norris derivative

Savitzky-Golay derivative

Savitzky-Golay derivative is a preprocessing technique in spectroscopy that calculates the derivative of a spectrum by fitting a polynomial to a window of adjacent points and calculating the derivative of the polynomial.

The following arguments can be set:

• window_size: int: The length of the window. Must be an odd integer number. Default: 5.
• polynomial_order: int: The order of the polynomial used to fit the samples. Must be less than window_size. Default: 2.
• derivative_order: int: The order of the derivative to compute. Default: 1.
• mode: str: The mode of the boundary. Default: 'nearest', available options: 'nearest', 'constant', 'reflect', 'wrap', 'mirror', 'interp'. See the official documentation for more information.

Usage example:

from chemotools.derivative import SavitzkyGolay

sg = SavitzkyGolay(window_size=15, polynomial_order=2, derivate_order=1)
spectra_derivative = sg.fit_transform(spectra)

William Norris derivative

William Norris derivative is a preprocessing technique in spectroscopy that calculates the derivative of a spectrum using finite differences.

The following arguments can be set:

• window_size: int: The length of the window. Must be an odd integer number. Default: 5.
• gap_size: int: The number of points between the first and second points of the window. Default: 3.
• derivative_order: int: The order of the derivative to compute. Default: 1.
• mode: str: The mode of the boundary. Default: 'nearest', available options: ‘reflect’, ‘constant’, ‘nearest’, ‘mirror’, ‘wrap’. See the official documentation for more information.

Usage example:

from chemotools.derivative import NorrisWilliams

nw = NorrisWilliams(window_size=15, gap_size=3, derivative_order=1)
spectra_derivative =   nw.fit_transform(spectra)

Baseline

Baseline correction is a preprocessing technique in spectroscopy that corrects for baseline shifts and variations in signal intensity by subtracting a baseline from a spectrum. The following algorithms are available:

• Linear baseline correction
• Polynomial baseline correction
• Cubic spline baseline correction
• Alternate iterative reweighed penalized least squares (AIRPLS) baseline correction
• Non-negative

Linear baseline correction

Linear baseline correction is a preprocessing technique in spectroscopy that corrects for baseline shifts and variations in signal intensity by subtracting a linear baseline from a spectrum. The current implementation subtracts a linear baseline between the first and last point of the spectrum.

Usage example:

from chemotools.baseline import LinearCorrection

lc = LinearCorrection()
spectra_baseline = lc.fit_transform(spectra)

Polynomial baseline correction

Polynomial baseline correction is a preprocessing technique in spectroscopy that approximates a baseline by fitting a polynomial to selected points of the spectrum. The selected points often correspond to minima in the spectra, and are selected by their index (not by the wavenumber). If no points are selected, the algorithm will select the first and last point of the spectrum.

The following arguments can be set:

• order: int The order of the polynomial used to fit the samples. Default: 1.
• indices: tuple The indices of the points to use for fitting the polynomial. Default: (0, -1). At the moment the indices need to be specified manually as a tuple because scikit-learn does not support mutable attributes in BaseEstimator. This tuple is transformed to a list when the transform method is called.

Usage example:

from chemotools.baseline import PolynomialCorrection

pc = PolynomialCorrection(order=2, indices=(0, 75, 150, 200, 337))
spectra_baseline = pc.fit_transform(spectra)

Cubic spline baseline correction

Cubic spline baseline correction is a preprocessing technique in spectroscopy that approximates a baseline by fitting a cubic spline to selected points of the spectrum. Similar to the PolynomialCorrection, the selected points often correspond to minima in the spectra, and are selected by their index (not by the wavenumber). If no points are selected, the algorithm will select the first and last point of the spectrum.

The following arguments can be set:

• indices: tuple The indices of the points to use for fitting the polynomial. Default: None. At the moment the indices need to be specified manually as a tuple because scikit-learn does not support mutable attributes in BaseEstimator. This tuple is transformed to a list when the transform method is called.

Usage example:

from chemotools.baseline import CubicSplineCorrection

cspl = CubicSplineCorrection(indices=(0, 75, 150, 200, 337))
spectra_baseline = cspl.fit_transform(spectra)

Alternate iterative reweighed penalized least squares (AIRPLS) baseline correction

It is an automated baseline correction algorithm that uses a penalized least squares approach to fit a baseline to a spectrum. The original algorithm is based on the paper by Zhang et al.. The current implementation is based on the Python implementation by zmzhang.

The following arguments can be set:

• nr_iterations: int The number of iterations before exiting the algorithm. Default: 15.
• lam: float smoothing factor. Default: 1e2.
• polynomial_order: int The order of the polynomial used to fit the samples. Default: 1.

Usage example:

from chemotools.baseline import AirPls

airpls = AirPls()
spectra_baseline = airpls.fit_transform(spectra)

Non-negative

Non-negative baseline correction is a preprocessing technique in spectroscopy that corrects for baseline by removing negative values from a spectrum. Negative values are either replaced by 0, or set to their absolute value.

The following arguments can be set:

• mode: str If 'zero', negative values are replaced by 0. If 'abs', negative values are set to their absolute value. _Default: 'zero'.

Usage example:

from chemotools.baseline import NonNegative

nnz = NonNegative(mode='zero')
nna = NonNegative(mode='abs')
spectra_nnz = nnz.fit_transform(spectra_baseline)
spectra_nna = nna.fit_transform(spectra_baseline)

Scale

Scale is a preprocessing technique in spectroscopy that scales the spectra. The following algorithms are available:

• MinMaxScaler: scales each spectrum by its minimum or maximum value.
• L-normalization: scales each spectrum by its L-norm.

MinMax scaler

MinMaxScaler is a preprocessing technique in spectroscopy that scales each spectrum by its minimum or maximum value.

The following arguments can be set:

• norm: str If 'min', the spectrum is scaled by its minimum value. If 'max', the spectrum is scaled by its maximum value. Default: 'max'.

Usage example:

from chemotools.scale import MinMaxScaler

minmax = MinMaxScaler(norm='max')
spectra_norm = minmax.fit_transform(spectra)

L-normalization

L-normalization is a preprocessing technique in spectroscopy that scales each spectrum by its L-norm.

The following arguments can be set:

• l-norm: int The L-norm to use. Default: 2.

Usage example:

from chemotools.scale import LNormalize

lnorm = LNormalize(l_norm=2)
spectra_norm = lnorm.fit_transform(spectra)

Smooth

Smooth is a preprocessing technique in spectroscopy that smooths the spectra. The following algorithms are available:

• Savitzky-Golay filter
• Whittaker smoother
• Mean filter
• Median filter

Savitzky-Golay filter

Savitzky-Golay filter is a preprocessing technique in spectroscopy that smooths the spectra by fitting a polynomial to the data. The current implementation is based on the scipy.signal.savgol_filter function.

The following arguments can be set:

• window_size: int: The length of the window. Must be an odd integer number. Default: 3.
• polynomial_order: int: The order of the polynomial used to fit the samples. Must be less than window_size. Default: 1.
• derivative_order: int: The order of the derivative to compute. Default: 1. 'constant', 'reflect', 'wrap', 'mirror', 'interp'. See the official documentation for more information.

Usage example:

from chemotools.smooth import SavitzkyGolayFilter

sgf = SavitzkyGolayFilter(window_size=15, polynomial_order=2)
spectra_norm = sgf.fit_transform(spectra)

Whittaker smoother

It is an automated smoothing algorithm that uses a penalized least squares approach to iteratively apply a smoothing operation to the data by minimizing a penalty function that balances the degree of smoothness and the fidelity to the original data.

The following arguments can be set:

• lam: float smoothing factor. Default: 1e2.
• differences: int The number of differences to use. Default: 1.

Usage example:

from chemotools.smooth import WhittakerSmooth

wtk = WhittakerSmooth(lam=10)
spectra_norm = wtk.fit_transform(spectra)

Mean filter

Mean filter is a preprocessing technique in spectroscopy that smooths the spectra by applying a mean filter. The current implementation is based on the scipy.ndimage.uniform_filter function.

The following arguments can be set:

• window_size: int: The length of the window. Must be an odd integer number. Default: 3.
• mode: str: The mode parameter determines how the array borders are handled, where 'constant', 'reflect', 'wrap', 'mirror', 'interp'. See the official documentation for more information. Default: 'nearest'.

Usage example:

from chemotools.smooth import MeanFilter

mean_filter = MeanFilter()
spectra_norm = mean_filter.fit_transform(spectra)

Median filter

Median filter is a preprocessing technique in spectroscopy that smooths the spectra by applying a median filter. The current implementation is based on the scipy.ndimage.median_filter function.

The following arguments can be set:

• window_size: int: The length of the window. Must be an odd integer number. Default: 3.
• mode: str: The mode parameter determines how the array borders are handled, where 'constant', 'reflect', 'wrap', 'mirror', 'interp'. See the official documentation for more information. Default: 'nearest'.

Usage example:

from chemotools.smooth import MedianFilter

median_filter = MedianFilter()
spectra_norm = median_filter.fit_transform(spectra)