FAIRaman

FAIRaman is an open-source Python tool for converting biomedical Raman spectroscopy measurements into self-describing HDF5/NeXus files enriched with clinical, biobanking, provenance, and acquisition metadata.

The aim is simple: keep the spectrum, the sample context, the donor/sample annotation, and the acquisition parameters together in one structured file, so that Raman datasets can be reused, compared, integrated, and analysed beyond the laboratory where they were produced.

FAIRaman is designed for clinical and translational Raman spectroscopy, where the biological meaning of the sample is as important as the spectral matrix itself. In other words: no more lonely CSV files wandering through folders like tiny undocumented ghosts.

---

What FAIRaman does

FAIRaman converts raw Raman spectral data into FAIR-oriented output files by combining:

• PROJECT-level metadata
  study identity, funding, governance, authorship, licensing, accessibility, and keywords

• SAMPLE-level metadata
  sample identity, provenance, donor information, diagnosis, anatomical site, processing, and biobanking descriptors inspired by MIABIS v3

• ENTRY-level metadata
  Raman acquisition parameters, instrument information, laser settings, optical system, spectral data, coordinate provenance, and auxiliary images inspired by NeXus/NXraman

The output is an HDF5 file that contains both data and metadata in a predictable structure.

---

Why FAIRaman

Biomedical Raman spectroscopy is increasingly used in clinical research, but datasets are often difficult to reuse because spectra are stored separately from essential information such as:

• what biological material was measured
• where the sample came from
• how it was processed
• which donor or diagnosis it refers to
• how the spectrum was acquired
• under which governance and reuse conditions the data can be shared

FAIRaman addresses this problem by making Raman files self-describing and machine-readable from the beginning.

It is not proposed as a final universal standard. It is a pragmatic operational layer that adapts existing metadata ideas from ISA, MIABIS, and NeXus/NXraman to the specific needs of clinical Raman spectroscopy.

---

Key features

• Graphical user interface for batch conversion
• Support for Renishaw WDF files through renishawWiRE
• Support for RiverICon_SRM1 files through RiverD
• Support for vendor-neutral ASCII/CSV spectral files
• Export to HDF5/NeXus, JSON, and CSV
• Complete metadata schema written in every HDF5 file, even when fields are empty
• Automatic validation of converted data against the original source data
• Automatic first-guess mapping of TXT/Excel metadata fields to HDF5 paths
• Support for Italian and English metadata aliases
• Geometry-aware handling of Raman maps and point spectra
• Explicit coordinate provenance
• Storage of white-light images and acquisition-map overlays when available from WDF files
• Root-level provenance attributes including source_format and fairaman_version and hash
• Machine-readable data governance fields such as data_license, accessibility, and governance_reference

---

Installation

FAIRaman can be installed directly as a Python package or run from its source code. Both methods provide access to the graphical user interface (GUI) for converting Raman spectroscopy data into structured output formats.

Python 3.9 or later is recommended.

Option 1: Install FAIRaman as a Python package (Recommended)

This is the simplest method for users who want to use FAIRaman without downloading or modifying the source code.

Step 1. Install FAIRaman

Open a terminal (Command Prompt, PowerShell, or a Python environment terminal) and run:

pip install fairaman

This command installs the FAIRaman package and its declared dependencies from PyPI.

Step 2. Launch FAIRaman

After installation, start the application by running:

fairaman

The FAIRaman graphical user interface will open, allowing users to import Raman spectroscopy files, configure metadata, and export the converted data.

Note: Python and pip must already be installed and accessible from the terminal. The fairaman command must be executed in the Python environment where the package was installed.

---

Option 2: Run FAIRaman from source code (main.py)

This method is intended for developers and users who want to access, inspect, modify, or contribute to the FAIRaman source code.

Step 1. Clone the GitHub repository

Open a terminal and run:

git clone https://github.com/Maugeri-Nanomed-Lab/FAIRaman.git
cd FAIRaman

Step 2. Install the required dependencies

Install the main Python dependencies:

pip install numpy pandas h5py matplotlib pillow openpyxl

To enable Renishaw WDF support, install:

pip install renishawWiRE

If renishawWiRE is not installed, FAIRaman can still process supported ASCII/CSV files, but WDF conversion will be unavailable.

Step 3. Install FAIRaman in editable mode

From the repository's root directory, run:

pip install -e .

This installs FAIRaman in editable mode, allowing changes to the source code to be reflected without reinstalling the package.

Step 4. Launch FAIRaman

Run the main script:

python main.py

The FAIRaman graphical user interface will open and guide users through the conversion workflow.

---

Additional note for Linux users

On some Linux distributions, Tkinter, which is required for the graphical user interface, may need to be installed separately.

For Debian- or Ubuntu-based systems:

sudo apt-get install python3-tk

---

Quick Start

After installing FAIRaman using either method, launch the application.

For package installation:

fairaman

For source-code installation:

python main.py

Once the GUI opens, users can select their Raman input files, configure the available metadata, choose the desired output format, and start the conversion process.

FAIRaman supports the conversion of compatible Raman spectroscopy data into structured HDF5 files, with optional JSON and CSV outputs for metadata and spectral data, respectively.

Typical workflow

1. Select the input mode:

   • WDF mode for Renishaw .wdf files
   • ASCII mode for .txt, .csv, or .dat spectral files
   • RiverD mode for .txt spectral files

2. Select the input folder containing spectral files.

3. Select a shared metadata .txt file containing project- and assay-level metadata.

4. Optionally select an Excel metadata file containing sample-specific rows filled out manually by the users.

5. Review and adjust the automatic metadata mapping proposed by the GUI.

6. Select output formats (multiple formats can be selected simultaneously):

   • HDF5 / NeXus
   • JSON metadata sidecar
   • CSV spectral matrix

7. Run the conversion.

8. Check the tick mark (✓) to confirm successful conversion and validation.

---

Metadata input files

FAIRaman uses two complementary metadata sources.

1. Shared metadata TXT

The .txt file contains metadata common to the whole experiment or batch.

Expected format:

project_name: SpectraBREAST
project_id: GA101187508
funding: European Innovation Council
governance_reference: Ethics approval XYZ
author: Carlo Morasso
author_id: ORCID:0000-0001-9185-0198
data_license: CC BY-NC 4.0
accessibility: restricted
keywords: Raman spectroscopy; breast cancer; FAIR data
instrument_name: Renishaw inVia
laser_wavelength: 532
wavelength_units: nm
power: 10
power_units: mW
exposure_time: 1.2
exposure_time_units: s
accumulation_count: 32
substrate: CaF2

Lines starting with # are ignored.

2. Sample metadata Excel

The Excel file is used for sample-specific metadata. The first column is treated as the filename key and is matched against the spectral filename stem.

Example:

filename	sample_id	donor_id	diagnosis_code	diagnosis_ontology	anatomical_site	anatomical_site_code	anatomical_ontology	sample_type
sample_001.wdf	S001	D001	C50.9	ICD-10	breast	UBERON:0000310	UBERON	tissue section

The filename matching is case-insensitive and ignores the spectral file extension.

---

Supported input formats

Renishaw WDF

FAIRaman reads .wdf files through renishawWiRE.

It attempts to recover:

• spectral data
• Raman shift axis
• map geometry
• calibrated spatial coordinates when available
• laser wavelength when available
• white-light image
• acquisition-map overlay

WDF geometries are handled as:

Case	FAIRaman output
Regular rectangular map	regular_grid cube (ny, nx, n_wavenumbers)
Irregular or point-based acquisition	point_coordinates matrix (n_points, n_wavenumbers)
No physical coordinates available	synthetic index coordinates with explicit provenance

ASCII / CSV / DAT

FAIRaman automatically recognises four common tabular layouts:

Layout	Expected structure
Single spectrum	`raman_shift
Multi-spectrum wide matrix	`raman_shift
Spatial long format	`x
Spatial wide format	`x

Supported delimiters:

• whitespace
• comma
• semicolon
• tab

Raman shift axes are sorted in ascending order before export.

---

RiverD

FAIRaman reads .txt files generated by RiverD, supporting single-spectrum and multi-spectrum datasets.

Layout	Expected structure
Single spectrum	cm-1 | frame1
Multi-spectrum wide matrix	cm-1 | frame1 | frame2 | ... | frameN

In the multi-spectrum wide-matrix format:

• First column (cm-1): Raman shift values expressed in cm⁻¹.
• Subsequent columns (frame1 to frameN): Spectral intensity values corresponding to individual frames.
• Each row: Intensity measurements from all frames at a specific Raman shift.

The number of frames (N) is variable and depends entirely on the number of frames selected by the user in RiverD. FAIRaman accommodates files containing different numbers of frames.

Supported delimiters:

• Whitespace
• Comma
• Semicolon
• Tab

Raman shift values are sorted in ascending order before export.

Canonical data model

FAIRaman v1.5 introduces a canonical internal data model: a single contract between importers and the HDF5/NeXus writer.

Two coordinate modes are supported:

Mode	Spectral array	Coordinate representation
regular_grid	cube (ny, nx, nw)	independent x_axis and y_axis arrays
point_coordinates	spectra (n_points, nw)	one x and y coordinate per spectrum

Every dataset carries coordinate provenance:

Field	Meaning
coordinate_mode	regular_grid or point_coordinates
coordinate_source	source of the spatial information, e.g. WDF ORGN, ASCII x/y columns, or synthetic
coordinate_units	usually micrometers or index
coordinate_validated	whether coordinates were verified against spectral dimensions
reshape_applied	whether the spectral array was reshaped
reshape_source	source of the reshape operation
geometry_warning	optional warning about geometry handling

Before writing, validate_canonical() checks that spectra and coordinates describe the same geometry.

---

Output files

FAIRaman can generate three output types.

HDF5 / NeXus

The .h5 file is the primary FAIRaman output. It contains:

• the complete metadata hierarchy
• the Raman spectral data
• the Raman shift axis
• spatial coordinates
• coordinate provenance
• auxiliary images when available
• root-level provenance attributes

JSON sidecar

The .json file contains a human-readable copy of the mapped metadata.

CSV spectral matrix

The .csv file contains a geometry-aware flat export.

For regular_grid data:

x_um | y_um | wn_1 | wn_2 | ... | wn_n

For point_coordinates data:

point_id | x | y | wn_1 | wn_2 | ... | wn_n

---

HDF5 structure

FAIRaman organises the HDF5 file into three main groups.

FAIRaman.h5
│
├── PROJECT  (NXcollection)
│   ├── project_name
│   ├── project_id
│   ├── funding
│   ├── governance_reference
│   ├── author
│   ├── author_id
│   ├── data_license
│   ├── accessibility
│   └── keywords
│
├── SAMPLE  (NXsample)
│   ├── sample_id
│   ├── SAMPLE_INFO
│   │   ├── sample_provenance
│   │   ├── sample_type
│   │   ├── detailed_sample_type
│   │   ├── sample_source
│   │   ├── anatomical_site
│   │   ├── anatomical_site_code
│   │   ├── anatomical_ontology
│   │   ├── storage_temperature
│   │   ├── processing_method
│   │   ├── sample_creation_date
│   │   └── sample_notes
│   ├── SAMPLE_DONOR
│   │   ├── donor_id
│   │   ├── donor_sex
│   │   ├── donor_age
│   │   ├── diagnosis_code
│   │   ├── diagnosis_ontology
│   │   └── diagnosis_notes
│   └── SAMPLE_EVENT
│       ├── event_date
│       └── event_description
│
└── ENTRY  (NXentry; definition = NXraman)
    ├── title
    ├── experiment_type
    ├── run_type
    ├── start_time
    ├── data_type
    ├── measurement  (NXcollection)
    │   ├── exposure_time
    │   ├── exposure_time_units
    │   ├── substrate
    │   └── accumulation_count
    ├── instrument  (NXinstrument)
    │   ├── name
    │   ├── laser  (NXsource)
    │   │   ├── wavelength
    │   │   ├── wavelength_units
    │   │   ├── power
    │   │   ├── power_units
    │   │   └── filter
    │   └── optical_system  (NXoptics)
    │       └── lens
    ├── data  (NXdata)
    │   ├── intensity
    │   ├── raman_shift
    │   ├── x
    │   ├── y
    │   ├── point_id                 # point_coordinates mode only
    │   └── spectral_count
    └── auxiliary  (NXcollection)     # WDF mode only, when available
        ├── white_light  (NXdata)
        └── acquisition_map  (NXdata)

Root-level attributes include:

source_format
fairaman_version (FAIRaman software version)
HASH (content fingerprint used to verify file integrity)

---

Metadata mapping

The GUI includes an automatic first-guess mapper that suggests HDF5 targets for the fields found in TXT and Excel metadata files.

Examples:

Source field	Suggested HDF5 path
project_name, study_name	PROJECT.project_name
orcid, author_id	PROJECT.author_id
license, data_license	PROJECT.data_license
patient_id, paziente	SAMPLE.SAMPLE_DONOR.donor_id
age, eta, donor_age	SAMPLE.SAMPLE_DONOR.donor_age
icd10, diagnosis_code	SAMPLE.SAMPLE_DONOR.diagnosis_code
tissue, tessuto, anatomical_site	SAMPLE.SAMPLE_INFO.anatomical_site
uberon, anatomical_site_code	SAMPLE.SAMPLE_INFO.anatomical_site_code
protocol, processing_method	SAMPLE.SAMPLE_INFO.processing_method
laser, laser_nm, wavelength	ENTRY.instrument.laser.wavelength
power, laser_power	ENTRY.instrument.laser.power
exposure, integration_time	ENTRY.measurement.exposure_time
objective, obiettivo, lens	ENTRY.instrument.optical_system.lens

All suggestions can be manually reviewed and changed before conversion.

---

FAIR alignment

Findable

FAIRaman writes project identifiers, author identifiers, keywords, governance references, and accessibility information inside the file.

Accessible

The main output is HDF5, an open and widely supported format readable with tools such as h5py, HDFView, MATLAB, R, Origin, and other scientific software.

Interoperable

The metadata model combines:

Standard / resource	FAIRaman use
ISA logic	project / sample / assay organisation
MIABIS v3	clinical sample, donor, and biobanking descriptors
NeXus / NXraman	spectral data and Raman acquisition structure
UBERON	anatomical site coding
ICD-10	diagnosis coding

Reusable

The HDF5 file keeps spectral data, sample context, acquisition parameters, governance metadata, and provenance in one structured container. Missing fields are written explicitly as empty datasets, making absence visible instead of silent.

---

Data governance note

FAIRaman is intended for clinical and translational research workflows, but it does not anonymise data automatically.

Before sharing FAIRaman files:

• use pseudonymised donor and sample identifiers
• do not include direct personal identifiers
• check local ethics approval, data-use agreements, GDPR requirements, and institutional rules
• choose appropriate values for data_license, accessibility, and governance_reference

The software helps structure metadata. It does not magically make sensitive data shareable. Sadly, even HDF5 cannot repeal GDPR.

---

Notes and limitations

• The WDF acquisition-map overlay is intended for qualitative spatial context and may not represent exact spatial alignment, because some WDF image transformations are undocumented.
• ASCII files without physical coordinates are exported with synthetic coordinates and explicit coordinate provenance.
• FAIRaman currently focuses on Raman spectroscopy. Extension to related optical datasets, such as infrared or hyperspectral imaging, is conceptually possible but not yet implemented as a stable feature.
• A future RamanSPy adapter is planned for additional vendor formats such as Horiba, WiTec, and MATLAB files. For Renishaw WDF files, FAIRaman uses renishawWiRE directly because generic loaders may discard spatial coordinates, white-light images, or laser metadata.

---

Roadmap

Planned or desirable future developments include:

• packaging as an installable Python package
• command-line interface for headless batch conversion
• richer validation reports
• example notebooks for reading FAIRaman HDF5 files
• RamanSPy-based adapters for additional vendor formats
• community discussion on minimal recommended metadata fields
• extension of the schema to related optical modalities such as IR and hyperspectral imaging

---

Contributing

Contributions are welcome.

For major changes, please open an issue first to discuss the proposed modification.

When adding new importers or changing the data model, please keep the canonical data model consistent and ensure that validate_canonical() passes before writing HDF5 files.

---

Citation

If you use FAIRaman, please cite the associated manuscript when available:

Piccapietra D., Bonizzi A., Vazifehdan M., Madarshahi S.N.S., Piccotti F., Truffi M., Albasini S., Corsi F., Morasso C.
FAIRaman: making clinical Raman spectroscopy data FAIR through biobanking and NeXus standards.

Citation details will be updated after publication.

---

License

FAIRaman is released under the MIT License. See LICENSE.md for details.