raman-hypmap 0.0.6

Raman hyperspectral processing for Horiba instrument

raman_hypmap

Raman hyperspectral processing via python scripts tailoured for .txt output of Horiba instruments, e.g., the LabSpec 6 Spectroscopy Suite Software product. Theoretically this code will work on any hyperspectral Raman data if the .txt input is formatted correctly.

Three python modules are available:

• bgr_process — automated background reduction of entire map
• filter_window — using defined peaks to create spectral intensity maps
• hypmap_plot — interactive plot of intensity maps to create and save spectral masks

Referencing

If using this code, please reference the archive at zenodo.org

Please reference Wang and Dai (2016) for the iterative polynomial smoothing method used for background reduction of spectra bgr_process. Reference: Wang, T. and Dai, L., 2017. Background subtraction of Raman spectra based on iterative polynomial smoothing. Applied spectroscopy, 71(6), pp.1169-1179. doi: https://doi.org/10.1177/0003702816670915.

The progress bar used in this code is from user and full details are on stackoverflow

The pixel selector used for the hyperspectral map hypmap_plot is adapted from a matplotlib tutorial for creating a polygon selector.

License

The project is licensed under the GNU-v3 license.

Installation

To install the package, a .toml and .whl are provided. Download, navigate to the folder where the . .toml or .whl file is located and run pip install. Alternatively, the files can be imported as modules locally. Module are defined within hyprocess.py.

This package is also available at https://pypi.org/project/raman-hypmap/

Usage

Import a .txt hyperspectral map file from the horiba software (e.g., LabSpec 6). The file is structured as a tab deliminated file with the first row denoting the Raman Shift (cm^-1) values. Note this row is offset two spaces, given that each hyperspectral pixel is then stored as x-value, y-value, and spectral intensity-values where the latter has the same length as the list of Raman Shift values, created pairs. An exampe .txt fie is provided in example/example_data.txt to illustrate this.

The .txt file should be imported into a pandas dataframe.

Although optional, setting the coordinates of the map to start at (0,0) will avoid complications.

import pandas as pd

file = 'example/example_data.txt'

file = file.rsplit('.',1)[0]
_df = pd.read_csv(file+'.txt', sep='\t', index_col=False, decimal=',',header=None)
_df = _df.replace(',','', regex=True)
# set origin to zero
_df[0] = _df[0].sub(_df[0].min())
_df[1] = _df[1].sub(_df[1].min())

The python modules can then be run in order, for example:

from raman_hypmap import hyprocess

''' --- background reduction --- '''
_df_bgr = hyprocess.bgr_process(_df,file)
_df_bgr = pd.read_csv(file+'_bgr.csv', sep=',', index_col=False,header=None)
_df_raw = pd.read_csv(file+'_raw.csv', sep=',', index_col=False,header=None)

''' --- mineral filter --- '''
window_location = 'example/_window_ranges.csv'
hyprocess.filter_window(_df_bgr,file+'_bgr',window_location)

''' --- interactive hyperspectral map --- '''
_df_filtered = pd.read_csv(file+'_bgr_w_filter.csv', sep=',', index_col=False)
hyprocess.hypmap_plot(_df_filtered,_df_raw,file)

Note that each module imports the generated .csv file from the previous module as an input. These files will be created automatically when running the module to the local folder.

An example script showing this code is also available in: example/example.py

bgr_process

The input format is the standard .txt file generated by the Horiba LabSpec 6 software. However, this format can be recreated via a tab-delimited .txt file as shown below:

blank	blank	Raman-shift values (tab-delimited list)
x-coorindate	y-coordinate	Raman-intensity values (tab-delimited list)
...	...	...

Each spectra is background reduced via an iterative polynomial smoothing method Wang and Dai (2016) and written into a pandas dataframe. Three files are output and automatically saved as .csv files:

• example_data_raw.csv — reformatted data input saved as a .csv file
• example_data_bg.csv — the background subtracted from each spectra saved as a spectral file
• example_data_bgr.csv — the resulting background reduced spectra saved as a spectral file

Each file is saved in a identical structure to the table shown above, but comma-delimited in a format that can be opened and viewed in Microsoft Excel.

filter_window

This module needs a .csv file formatted as explained in the section above to operate correctly. The code will itterate over every spectra provided and take the maximum intensity value from this window. It will then output a hyperspectral grid of values using this window. That hyperspecral grid will be saved as an image .png file with greyscale values stretched to the maximum and minimum intensity values recorded.

The filter windows are imported from a standalone .csv file called _window_ranges.csv that can be open and edited in Microsoft Excel. The headings must not be changed. The format of the spredsheet is shown below:

phase	char_min	char_max	comp_min	comp_max	i_min	i_max
graphite	1510	1620	0	0	0	2000
apatite	950	975	0	0	0	2000
...	...	...	...	...	...	...

These numbers are important as they define how the python code operates:

• 'Phase' defines the name of the species
• 'char_min' and 'char_max' define the window that the filter will operate over. A new hyperspectral map will be created for every row of values.
• Sometimes spectra overlap and so 'comp_min' and 'comp_max' define a comparison window. the maximum intensity values defined in the 'char_' window must be larger than the maximum intensity value in the 'comp_' window otherwise a value of 0 will be given for that spectra.
• 'i_min' and 'i_max' work in a similar way to the above. The maximum intensity of the spectra in the window defined by 'char_' must be within the intensity thresholds defined by the 'i_min' and 'i_max' variables.

Each row of values will create and save a new hyperspectral map using that window. Examples derived from example/example.py using the current configuration of _window_ranges.csv are shown below:

In addition, this module will create a new _w_filter.csv spreadsheet that stores these values. This is formatted so that the first two columns store the x and y coordinates and each subsequent column is added for every species defined in _window_ranges.csv. An example of the output is provided example/example_output/example_data_bgr_w_filter.csv

hypmap_plot

This module needs two .csv files, a _w_filter.csv spreadsheet output generated by the filter_window module and a spectrum data file formatted using the bgr_process module. The module will then create a hyperspectral image with a superposed grid of points aligned with each pixel in the hyperspectral image. These points can be selected by drawing circles using the mouse. When highlighted, they will be black.

When pixels are selected in this way, pressing ENTER will generate a simple plot generated from every single pixel. Three lines are shown on the plot:

• The mean with ± 2 standard deviations with grey bars
• The mean smoothed using a Savitzky–Golay filter (2nd order polynomial and 15-wide window).
• The median of all selected pixels (Orange)

The location of the highlighted pixels will be saved as a .png file and the spectral mask will be saved as a tab-delimited .txt file. In addition, the .txt file saves the sum of all spectra. A second text file consisiting of two columns: (1) Raman shift values, vs (2) mean intensity will also be generated. This is a suitable input for additional automatic fitting software, such as IFORS. An example of these .txt files are provided example/example_output/example_data_spectra.txt.

This process can be repeated as long as a new selection is made. Every time you hit ENTER and plot the result of a new mask, the files automatically saving the spectra and location will be overwritten.