Package to integrate chemometrics in scikit-learn pipelines
Project description
Welcome to chemotools! 🖖
This project is an implementation of spectral preprocessing techniques integrated with the widely popular sklearn API, providing you with an easy-to-use toolkit for analyzing and manipulating your data. With this integration, you can easily apply techniques such as spectral derivative, scatter correction or baseline removal to your datasets.
The goal of this project is to provide a comprehensive and user-friendly package for spectral preprocessing, with a focus on making it accessible to a wide range of users, from data scientists to researchers to curious learners. By integrating these techniques with the powerful sklearn API, I enable users to easily build machine learning and chemometric models on top of preprocessed data, making it possible to identify patterns and make predictions with greater accuracy.
In this repository, you will find a range of tools and resources for using spectral preprocessing techniques with the sklearn API, including code samples, documentation, and examples. I encourage contributions from the community, whether through bug reports, feature requests, or pull requests, to help us make this project even better.
In addition, I take great care to ensure that all functions are thoroughly unit tested for both API compatibility and functionality. I believe that unit testing is an essential part of any software development process, and it is particularly important for a project like this, which aims to provide reliable data analysis tools. I hope that this project will serve as a useful resource for anyone interested in learning more about spectral preprocessing and machine learning.
Thank you for your interest in this project, and I hope you find it useful for your data analysis and machine learning needs.
Table of contents
Installation 🚀
This package is available on PyPI and can be installed using pip:
pip install chemotools
Integration with scikit-learn ✨
All preprocessing techniques in this package are compatible with scikit-learn
and can be used in pipelines. For example, the following code creates a pipeline that performs multiplicative scatter correction, followed by a min-max scaling and a Savitzky-Golay smoothing:
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
pipeline = make_pipeline(AirPls(), MultiplicativeScatterCorrection(), StandardScaler(with_std=False))
spectra_transformed = pipeline.fit_transform(spectra)
Scatter
This package contains three common algorithms for scatter correction in spectroscopy:
- Multiplicative scatter correction (MSC)
- Standard normal variate (SNV)
- Extended multiplicative scatter correction (EMSC)
Multiplicative scatter correction
Multiplicative scatter correction (MSC) is a preprocessing technique in spectroscopy that corrects for the influence of light scattering on spectral measurements by dividing each spectrum by a scatter reference spectrum. The current implementation, accepts three types of reference spectra:
- The mean spectrum of the dataset (default).
- The median spectrum of the dataset.
- A single spectrum that is used to correct all spectra in the dataset.
Usage example for a single reference spectrum:
Usage example for the mean spectrum:
from chemotools.scatter import MultiplicativeScatterCorrection
msc = MultiplicativeScatterCorrection()
spectra_msc = msc.fit_transform(spectra)
Usage example for the median spectrum:
from chemotools.scatter import MultiplicativeScatterCorrection
msc = MultiplicativeScatterCorrection(use_median=True)
spectra_msc = msc.fit_transform(spectra)
Usage example for a single reference spectrum:
from chemotools.scatter import MultiplicativeScatterCorrection
msc = MultiplicativeScatterCorrection(reference=reference_spectrum)
spectra_msc = msc.fit_transform(spectra)
Standard normal variate
Standard normal variate (SNV) is a preprocessing technique in spectroscopy that adjusts for baseline shifts and variations in signal intensity by subtracting the mean and dividing by the standard deviation of each spectrum.
Usage example for a single reference spectrum:
from chemotools.scatter import StandardNormalVariate
snv = StandardNormalVariate()
spectra_snv = snv.fit_transform(spectra)
Extended multiplicative scatter correction
Extended multiplicative scatter correction (EMSC) is a preprocessing technique in spectroscopy that corrects for the influence of light scattering and instrumental drift by fitting a mathematical model to a reference spectrum and using it to normalize all spectra in the dataset.
An implementation of the EMSC will be available soon 🤓.
Derivatives
This package contains two common algorithms for calculating derivatives in spectroscopy:
- Savitzky-Golay derivative
- William Norris derivative
Savitzky-Golay derivative
Savitzky-Golay derivative is a preprocessing technique in spectroscopy that calculates the derivative of a spectrum by fitting a polynomial to a window of adjacent points and calculating the derivative of the polynomial.
The following arguments can be set:
window_size: int
: The length of the window. Must be an odd integer number. Default: 5.polynomial_order: int
: The order of the polynomial used to fit the samples. Must be less thanwindow_size
. Default: 2.derivative_order: int
: The order of the derivative to compute. Default: 1.mode: str
: The mode of the boundary. Default: 'nearest', available options:'nearest'
,'constant'
,'reflect'
,'wrap'
,'mirror'
,'interp'
. See the official documentation for more information.
Usage example:
from chemotools.derivative import SavitzkyGolay
sg = SavitzkyGolay(window_size=15, polynomial_order=2, derivate_order=1)
spectra_derivative = sg.fit_transform(spectra)
William Norris derivative
William Norris derivative is a preprocessing technique in spectroscopy that calculates the derivative of a spectrum using finite differences.
The following arguments can be set:
window_size: int
: The length of the window. Must be an odd integer number. Default: 5.gap_size: int
: The number of points between the first and second points of the window. Default: 3.derivative_order: int
: The order of the derivative to compute. Default: 1.mode: str
: The mode of the boundary. Default: 'nearest', available options:‘reflect’
,‘constant’
,‘nearest’
,‘mirror’
,‘wrap’
. See the official documentation for more information.
Usage example:
from chemotools.derivative import NorrisWilliams
nw = NorrisWilliams(window_size=15, gap_size=3, derivative_order=1)
spectra_derivative = nw.fit_transform(spectra)
Baseline
Baseline correction is a preprocessing technique in spectroscopy that corrects for baseline shifts and variations in signal intensity by subtracting a baseline from a spectrum. The following algorithms are available:
- Linear baseline correction
- Polynomial baseline correction
- Cubic spline baseline correction
- Alternate iterative reweighed penalized least squares (AIRPLS) baseline correction
- Non-negative
Linear baseline correction
Linear baseline correction is a preprocessing technique in spectroscopy that corrects for baseline shifts and variations in signal intensity by subtracting a linear baseline from a spectrum. The current implementation subtracts a linear baseline between the first and last point of the spectrum.
Usage example:
from chemotools.baseline import LinearCorrection
lc = LinearCorrection()
spectra_baseline = lc.fit_transform(spectra)
Polynomial baseline correction
Polynomial baseline correction is a preprocessing technique in spectroscopy that approximates a baseline by fitting a polynomial to selected points of the spectrum. The selected points often correspond to minima in the spectra, and are selected by their index (not by the wavenumber). If no points are selected, the algorithm will select the first and last point of the spectrum.
The following arguments can be set:
order: int
The order of the polynomial used to fit the samples. Default: 1.indices: tuple
The indices of the points to use for fitting the polynomial. Default: (0, -1). At the moment the indices need to be specified manually as a tuple becausescikit-learn
does not support mutable attributes inBaseEstimator
. This tuple is transformed to a list when thetransform
method is called.
Usage example:
from chemotools.baseline import PolynomialCorrection
pc = PolynomialCorrection(order=2, indices=(0, 75, 150, 200, 337))
spectra_baseline = pc.fit_transform(spectra)
Cubic spline baseline correction
Cubic spline baseline correction is a preprocessing technique in spectroscopy that approximates a baseline by fitting a cubic spline to selected points of the spectrum. Similar to the PolynomialCorrection
, the selected points often correspond to minima in the spectra, and are selected by their index (not by the wavenumber). If no points are selected, the algorithm will select the first and last point of the spectrum.
The following arguments can be set:
indices: tuple
The indices of the points to use for fitting the polynomial. Default: None. At the moment the indices need to be specified manually as a tuple becausescikit-learn
does not support mutable attributes inBaseEstimator
. This tuple is transformed to a list when thetransform
method is called.
Usage example:
from chemotools.baseline import CubicSplineCorrection
cspl = CubicSplineCorrection(indices=(0, 75, 150, 200, 337))
spectra_baseline = cspl.fit_transform(spectra)
Alternate iterative reweighed penalized least squares (AIRPLS) baseline correction
It is an automated baseline correction algorithm that uses a penalized least squares approach to fit a baseline to a spectrum. The original algorithm is based on the paper by Zhang et al.. The current implementation is based on the Python implementation by zmzhang.
The following arguments can be set:
nr_iterations: int
The number of iterations before exiting the algorithm. Default: 15.lam: float
smoothing factor. Default: 1e2.polynomial_order: int
The order of the polynomial used to fit the samples. Default: 1.
Usage example:
from chemotools.baseline import AirPls
airpls = AirPls()
spectra_baseline = airpls.fit_transform(spectra)
Non-negative
Non-negative baseline correction is a preprocessing technique in spectroscopy that corrects for baseline by removing negative values from a spectrum. Negative values are either replaced by 0, or set to their absolute value.
The following arguments can be set:
mode: str
If'zero'
, negative values are replaced by 0. If'abs'
, negative values are set to their absolute value. _Default:'zero'
.
Usage example:
from chemotools.baseline import NonNegative
nnz = NonNegative(mode='zero')
nna = NonNegative(mode='abs')
spectra_nnz = nnz.fit_transform(spectra_baseline)
spectra_nna = nna.fit_transform(spectra_baseline)
Scale
Scale is a preprocessing technique in spectroscopy that scales the spectra. The following algorithms are available:
- MinMaxScaler: scales each spectrum by its minimum or maximum value.
- L-normalization: scales each spectrum by its L-norm.
MinMax scaler
MinMaxScaler is a preprocessing technique in spectroscopy that scales each spectrum by its minimum or maximum value.
The following arguments can be set:
norm: str
If'min'
, the spectrum is scaled by its minimum value. If'max'
, the spectrum is scaled by its maximum value. Default:'max'
.
Usage example:
from chemotools.scale import MinMaxScaler
minmax = MinMaxScaler(norm='max')
spectra_norm = minmax.fit_transform(spectra)
L-normalization
L-normalization is a preprocessing technique in spectroscopy that scales each spectrum by its L-norm.
The following arguments can be set:
l-norm: int
The L-norm to use. Default:2
.
Usage example:
from chemotools.scale import LNormalize
lnorm = LNormalize(l_norm=2)
spectra_norm = lnorm.fit_transform(spectra)
Smooth
Smooth is a preprocessing technique in spectroscopy that smooths the spectra. The following algorithms are available:
- Savitzky-Golay filter
- Whittaker smoother
- Mean filter
- Median filter
Savitzky-Golay filter
Savitzky-Golay filter is a preprocessing technique in spectroscopy that smooths the spectra by fitting a polynomial to the data. The current implementation is based on the scipy.signal.savgol_filter
function.
The following arguments can be set:
window_size: int
: The length of the window. Must be an odd integer number. Default: 3.polynomial_order: int
: The order of the polynomial used to fit the samples. Must be less thanwindow_size
. Default: 1.derivative_order: int
: The order of the derivative to compute. Default: 1.'constant'
,'reflect'
,'wrap'
,'mirror'
,'interp'
. See the official documentation for more information.
Usage example:
from chemotools.smooth import SavitzkyGolayFilter
sgf = SavitzkyGolayFilter(window_size=15, polynomial_order=2)
spectra_norm = sgf.fit_transform(spectra)
Whittaker smoother
It is an automated smoothing algorithm that uses a penalized least squares approach to iteratively apply a smoothing operation to the data by minimizing a penalty function that balances the degree of smoothness and the fidelity to the original data.
The following arguments can be set:
lam: float
smoothing factor. Default: 1e2.differences: int
The number of differences to use. Default: 1.
Usage example:
from chemotools.smooth import WhittakerSmooth
wtk = WhittakerSmooth(lam=10)
spectra_norm = wtk.fit_transform(spectra)
Mean filter
Mean filter is a preprocessing technique in spectroscopy that smooths the spectra by applying a mean filter. The current implementation is based on the scipy.ndimage.uniform_filter
function.
The following arguments can be set:
window_size: int
: The length of the window. Must be an odd integer number. Default: 3.mode: str
: The mode parameter determines how the array borders are handled, where'constant'
,'reflect'
,'wrap'
,'mirror'
,'interp'
. See the official documentation for more information. Default:'nearest'
.
Usage example:
from chemotools.smooth import MeanFilter
mean_filter = MeanFilter()
spectra_norm = mean_filter.fit_transform(spectra)
Median filter
Median filter is a preprocessing technique in spectroscopy that smooths the spectra by applying a median filter. The current implementation is based on the scipy.ndimage.median_filter
function.
The following arguments can be set:
window_size: int
: The length of the window. Must be an odd integer number. Default: 3.mode: str
: The mode parameter determines how the array borders are handled, where'constant'
,'reflect'
,'wrap'
,'mirror'
,'interp'
. See the official documentation for more information. Default:'nearest'
.
Usage example:
from chemotools.smooth import MedianFilter
median_filter = MedianFilter()
spectra_norm = median_filter.fit_transform(spectra)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for chemotools-0.0.10-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 63dd82c7652752cf991a63e1a6c24a6949143ad9db439e2455c63e6fb190775b |
|
MD5 | 4fdd3b7a493bd1711acd6ae9aa52dabb |
|
BLAKE2b-256 | 411c33b6187868fcef4bb5033605e707c9a8c1d09c03e63d3b612462bd9b52b2 |