NIR-preprocess 0.0.4

Function to evaluate impact of NIR pre-processing techniques on spectral data

Function compare_preprocessing can be used on any NIR spectral data if Y values are available.
Y can include one or several variables.

Function evaluates impact of different pre-processing techniques and combinations using multiblock partial least squares (MBPLS). Each block in MBPLS is a pre-processed spectral data.

Different pre-processing techniques evaluated:

• baseline
• de-trend
• EMSC
• MSC
• SNV
• Savitzky Golay derivatives (different polynomial and derivatives orders can be tested as well as the size of the moving window)

Blocks in MBPLS include:

• pre-processing techniques and combinations (several techniques applied to same data)
• original spectral data (starting point)
• 20 blocks of random noise called false signals (reference for destroyed information)

Analyst can choose to only compare scatter corrections techniques or only derivatives or both. It is also possible to set the number of pre-processing techniques which can be applied to same data. By default, only a single pre-processing technique and a combination of 2 are tested. NB: EMSC and MSC can not be applied together

For MBPLS, analyst can choose:

• number of principal components
• to autoscale or center each block
• to autoscale or center Y

Blocks are represented in superloading plots.
Model performances (adjusted R2, RMSECV) and variable importance on projection (VIP) are calculated for each block by cross validation. Number of random picks for cross validation and number of lines predicted in each cross validation can be set by the analyst. Effective rank for each block is calculated as well.

Call function

combination, datasets, datasets0, R2_all, R2adj_all, RMSECV_all, VIP_all, Ef_all, Wt = compare_preprocessing(X0, y)

Input arguments

1. X0 (n x k) data to test pre-processing techniques on
2. y (n x m) property under study

Optional input arguments

3. nbPC: number of principal components for PLS in the MBPLS decomposition (default value=2)
4. nb_comb: maximum number of pre-processing techniques applied on same data (default value=2)
5. auto_x: autoscale data after applying pre-processing techniques if auto_x=1, if not data centered (default value=1)
6. auto_y: autoscale variables to predict if auto_y=1, if not centered (default value=1)
7. nb: number of random picks for cross validation
8. CVnb: number of samples predicted in each cross validation
9. only_sg: only test Savitzky-Golay if only_sg=1 (default value=0)
10. svg_order: Savitzky-Golay polynomials orders to test
11. svg_deriv: Savitzky-Golay derivatives orders to test
12. svg_window: Savitzky-Golay window sizes to test
13. sg_op: test only Savitzky-Golay pretreatments with the same order of polynomial and derivative if sg_op=0 (default value=0)

Outputs

1. combination: pre-processing options tested
2. datasets: data X0 after each pre-processing option and autoscaled or centered
3. datasets0: data X0 after each pre-processing option
4. R2_all: R2 values for each y variable predicted for each pre-processing option tested (block)
5. R2adj_all: Adjusted for each y variable predicted for each pre-processing option tested (block)
6. RMSECV_all: Root mean square error by cross validation for each y variable predicted for each pre-processing option tested (block)
7. VIP_all: PLS variable importance in projection for each y variable predicted for each pre-processing option tested (block)
8. W: Superloadings from MBPLS

Examples

Two full examples, along with datasets are provided in folder 'tests' of 'Download Files'. Please refer to 'NIR_preprocess_example.pdf' for full details

• Example 1: Artificial dataset
• Example 2: Corn dataset

Compatibility

compare_preprocessing tested on Python 3.8 using the following modules:

• numpy 1.19.2
• matplotlib 3.3.2
• copy
• itertools
• RG 0.0.66 (available at: https://pypi.org/project/RG/)