soilspecdl 0.0.1

Practical deep learning for soil infrared spectroscopy, built on the fastai framework.

SoilSpecDL

Practical deep learning for soil infrared spectroscopy, built on the fastai framework.

soilspecdl brings the fastai philosophy to soil spectroscopy: state-of-the-art deep learning models should be accessible to domain scientists, not just ML practitioners. If you are a soil scientist comfortable with R packages like resemble or prospectr, this package is designed with you in mind.

Built on top of fastai — a library renowned for making deep learning approachable without sacrificing performance — soilspecdl provides sensible defaults, minimal boilerplate, and a layered API that lets you go deeper when you need to. Training a CNN on a large MIR spectral library should take a few lines of code, not a PhD in machine learning.

soilspecdl is part of a broader ecosystem for soil spectroscopy in Python:

• soilspecdata — fetch and access large spectral libraries (KSSL, OSSL)
• soilspectfm — preprocessing and spectral transforms
• SoilSpecDL — deep learning modeling

Get Started

from soilspecdata.datasets.ossl import get_ossl
from soilspectfm.core import SNV, TakeDerivative
from sklearn.pipeline import Pipeline
import numpy as np

Load OSSL/USDA KSSL + ICRAF + … dataset large soil spectroscopy library using the SoilSpecData python package:

ossl = get_ossl()
mir_data = ossl.get_mir()
wns = mir_data.wavenumbers

X, y, ids = ossl.get_aligned_data(
    spectra_data=mir_data,
    target_cols='k.ext_usda.a725_cmolc.kg')

X.shape, y.shape, ids.shape, wns[:5], wns.shape

((57676, 1701),
 (57676, 1),
 (57676,),
 array([600, 602, 604, 606, 608]),
 (1701,))

A bit of pre-processing using SoilSpecTfm python package:

pipe = Pipeline([
    ('snv', SNV()),
    ('derivative', TakeDerivative()),
])
X_tfm = pipe.fit_transform(X)

# Log-transforming the target (Exchangeable Potassium)
y_tfm = np.log1p(y.flatten())

from soilspecdl.dataloader import get_spectral_dls

Create a Fastai-like data loader with nice batch preview:

dls, test_idx = get_spectral_dls(X_tfm, y_tfm, wns)
dls.show_batch(max_n=12, ncols=4)

Importing albinet et al. paper CNN architecture, fastai Learner, …

from soilspecdl.models import MirzaiCNN
from fastai.learner import Learner
from fastai.callback.schedule import lr_find
from fastai.losses import MSELossFlat
from fastai.metrics import R2Score

learn = Learner(dls, MirzaiCNN(), loss_func=MSELossFlat(), metrics=R2Score())

Finding “best” learning rate:

learn.lr_find()

Epoch 1/1 : |█-------------------| 5.07% [73/1441 00:00<00:12... 0.2062]

SuggestedLRs(valley=0.0020892962347716093)

learn.fit_one_cycle(40, lr_max=1e-3)

epoch     train_loss  valid_loss  r2_score  time    
0         0.110809    0.084269    0.394419  00:07                            
1         0.079789    0.069087    0.503516  00:15                            
2         0.074917    0.060016    0.568708  00:07                            
3         0.064447    0.064101    0.539350  00:07                            
4         0.054324    0.056243    0.595819  00:07                            
5         0.061540    0.059759    0.570552  00:06                            
6         0.057230    0.138288    0.006216  00:07                            
7         0.057897    0.121253    0.128639  00:08                            
8         0.052471    0.092067    0.338378  00:07                            
9         0.045639    0.044204    0.682335  00:15                             
10        0.048287    0.184855    -0.328426 00:06                             
11        0.043516    0.108474    0.220473  00:06                             
12        0.040384    0.038102    0.726186  00:07                             
13        0.040395    0.063995    0.540110  00:06                             
14        0.034513    0.057830    0.584413  00:06                             
15        0.037132    0.044052    0.683426  00:06                             
16        0.036089    0.051145    0.632458  00:07                             
17        0.032155    0.074059    0.467791  00:15                             
18        0.034771    0.040014    0.712450  00:08                             
19        0.031389    0.068343    0.508866  00:06                             
20        0.033233    0.066360    0.523116  00:06                             
21        0.032475    0.035962    0.741569  00:07                             
22        0.028055    0.037719    0.728938  00:15                             
23        0.029821    0.068363    0.508722  00:07                             
24        0.027841    0.030044    0.784094  00:07                             
25        0.025044    0.029944    0.784813  00:06                             
26        0.026742    0.033762    0.757377  00:06                             
27        0.023458    0.043821    0.685087  00:06                             
28        0.022406    0.031492    0.773691  00:14                             
29        0.024227    0.027881    0.799640  00:15                             
30        0.022663    0.028077    0.798226  00:06                             
31        0.020502    0.028358    0.796213  00:06                             
32        0.021859    0.026163    0.811988  00:06                             
33        0.019432    0.027430    0.802882  00:07                             
34        0.019836    0.025392    0.817526  00:07                             
35        0.018150    0.025433    0.817233  00:07                             
36        0.020574    0.025172    0.819103  00:07                             
37        0.019895    0.025199    0.818912  00:14                             
38        0.019346    0.025025    0.820163  00:15                             
39        0.019031    0.025060    0.819907  00:06                             

Developer Guide

If you are new to using nbdev here are some useful pointers to get you started.

Install soilspecai in Development mode

# make sure soilspecai package is installed in development mode
$ pip install -e .

# make changes under nbs/ directory
# ...

# compile to have changes apply to soilspecai
$ nbdev_prepare

Usage

Installation

Install latest from the GitHub repository:

$ pip install git+https://github.com/franckalbinet/soilspecdl.git

or from conda

$ conda install -c franckalbinet soilspecdl

or from pypi

$ pip install soilspecdl