A deep learning framework for metabolite identification using enhanced MS/MS data and multidimensional molecular embeddings

A deep learning framework for metabolite identification using enhanced MS/MS data and multidimensional molecular embeddings is designed to process mass spectrometry data, perform predictions using a trained Convolutional Neural Network (CNN) model, and match the predictions against a reference database to identify potential candidate molecules.

Features

• Data Preprocessing: Converts MSP files to structured DataFrames and preprocesses spectra data for model input.
• Model Prediction: Utilizes a pre-trained CNN model to predict molecular embeddings from spectra data.
• Candidate Matching: Matches predicted embeddings with a reference database to find top candidate molecules based on cosine similarity.
• Support for Multiple Input Types: Handles MSP files both with and without SMILES annotations, controlled via a configuration parameter.
• Configurable Parameters: Allows users to adjust parameters like intensity thresholds, resolution, and the number of top candidates via a YAML configuration file.
• Modular Codebase: Organized into separate modules for easy maintenance and scalability.

🧪 Sample Testing Guide

To test ChemEmbed, use the provided Jupyter notebook:

• Open ChemEmbed_Sample_Test.ipynb.
• Follow the step-by-step instructions included within the notebook.

This notebook demonstrates how to run ChemEmbed on sample data and verify the setup.

Setup and Configuration

Before running the pipeline, ensure that all dependencies are installed and properly configured. The application is controlled through a config.yaml file, which specifies all input/output paths and parameters.

Installation

pip install chemembed

That installs the code and its Python dependencies. The trained models and the reference database are distributed separately because of their size — see Models and reference database below.

To install from source instead:

git clone https://github.com/faizanurv/ChemEmbed.git
cd ChemEmbed
pip install .

Note that pip install chemembed pulls in PyTorch, which together with its CUDA libraries occupies roughly 5 GB. On a machine with a small /tmp you may need to redirect pip's scratch space, e.g. export TMPDIR=/path/with/space.

Models and reference database

ChemEmbed needs two model files and one reference database, none of which are part of the Python package. Download them from the v1.0.1 release:

File	Size	Purpose
model_positive.bin	1129.6 MB	CNN weights, positive mode ([M+H]+)
model_negative.bin	1129.6 MB	CNN weights, negative mode ([M-H]-)
chemembed_reference_mol2vec_300d_520k.pkl	705.6 MB	520,083 compounds, 300-dim mol2vec embeddings
chemembed_reference_mol2vec_300d_520k.parquet	542.5 MB	the same database as parquet (optional, needs pyarrow)

You need one reference database, not both. The parquet is smaller and safer to load — pandas.read_pickle executes code while unpickling, so a parquet file is preferable for anything downloaded over the network. It requires pyarrow (pip install pyarrow), which the package does not install by default. If you would rather not add that dependency, use the .pkl; both contain identical data.

mkdir -p data_model_files && cd data_model_files
BASE=https://github.com/faizanurv/ChemEmbed/releases/download/v1.0.1
curl -LO $BASE/model_positive.bin
curl -LO $BASE/model_negative.bin
curl -LO $BASE/chemembed_reference_mol2vec_300d_520k.parquet   # or the .pkl

These files are versioned independently of the code. They are published once, on the v1.0.1 release, and later code releases do not re-attach them — so the URLs above stay valid across releases and are only changed when the data itself changes.

Verify the downloads before use:

b376f329f1b3759e598c9e1fd2e6b252d9f63ed2316baaff3b57be387b0b8c73  model_positive.bin
8165286689322aef9c0486fe470bdafd68162adb9a644d8477810ce8f9fb51ad  model_negative.bin
6ddf348aa6be45db7f5122468b8814c410cbfe87d5b8befc22bc990bc8c561f0  chemembed_reference_mol2vec_300d_520k.pkl
41a88ef1c84e3f0bec2babfb6cac8941ae4fc37168622a94676a3e9c6c753df0  chemembed_reference_mol2vec_300d_520k.parquet

sha256sum -c <<< "b376f329f1b3759e598c9e1fd2e6b252d9f63ed2316baaff3b57be387b0b8c73  model_positive.bin"

The reference database is a parquet conversion of Supplementary File 2.pkl from the Zenodo deposit, DOI 10.5281/zenodo.14778518 (CC-BY-4.0). Embeddings, SMILES, InChIKeys and precursor masses are unchanged.

Scope. The reference database covers 520,083 compounds. Candidate matching can only return molecules that are in the database, so a compound outside it cannot be identified regardless of spectral quality, and results obtained with a differently sized reference are not directly comparable.

Quick check

chemembed --input_file_type without_smiles \
          --adduct "+" \
          --msp_file_positive your_spectra.msp \
          --model_path_positive data_model_files/model_positive.bin \
          --reference_database data_model_files/chemembed_reference_mol2vec_300d_520k.parquet \
          --top_n_candidates 5 \
          --prediction_results results.csv

Only the flags for the polarity you select are required: with --adduct "+" you need --msp_file_positive and --model_path_positive, and the negative-mode equivalents may be omitted.

How to Run

Execute the main script using the following command:

python main.py --config config.yaml

Note: If you do not specify the --config argument, the script will default to using config.yaml.

Batch annotation — chemembed_by_file.py (FAISS GPU edition)

main.py annotates a single MSP file. chemembed_by_file.py is the batch entry point: it walks a directory of samples, reads each sample's GNPS/SIRIUS MGF, and writes chemembed_annotation/{converted_spectra.msp, preprocessed_data.pkl, chemembed_results.csv} per sample.

cp chemembed_config.example.yml chemembed_config.yml   # then edit the paths
python -m chemembed.chemembed_by_file --config chemembed_config.yml

chemembed_config.yml is gitignored so machine-specific absolute paths are never committed; chemembed_config.example.yml documents every key.

Where the speed comes from

The batch path produces the same candidates as the original per-sample loop, but restructures the work:

1. Reference DB loaded once, before the sample loop, instead of once per sample — and the RDKit Precursormz computation over every reference row runs once rather than N_samples times.
2. One FAISS index, one search per sample. All of a sample's query embeddings are searched in a single GPU call (~1 ms) instead of a per-query scan of the full database.
3. Precursor m/z post-filter applied to the FAISS results, preserving the original ChemEmbed matching semantics (exact 3-decimal match). A dict-based fallback index handles sparse masses whose correct candidates fall outside the top faiss_k hits, so recall is not traded for speed.
4. Batched CNN inference on GPU (batch_size: 32) in place of batch_size=1 on CPU — typically 20–50× faster.
5. predict_without_smiles rebuilt to support batch_size > 1, which is what makes (4) possible.

reference_utils.py also gained a parquet-or-pickle reader and a self-validating Precursormz fast path: if the reference already carries a Precursormz column, a random sample of rows is re-derived from SMILES and checked against it for the requested adduct. Only if the check passes is the stored column trusted; a negative-mode request, a null-bearing column, or a differently-built database all fail the check and fall through to the original full recompute. Correctness is preserved in every case — only the common positive-mode path gets faster.

Performance keys

Key	Default	Notes
batch_size	32	CNN inference batch size. Raise to 64/128 if the GPU has memory.
num_workers	0	DataLoader worker processes. 4 is reasonable on a multi-core node.
faiss_k	200	FAISS candidates retrieved before the precursor post-filter. Raise if the run reports a high "Fallback used" count.
recompute	false	true re-runs samples that already have chemembed_results.csv.

Requirement

FAISS is required by this entry point and is not in requirements.txt, since the correct build depends on your hardware:

conda install -c pytorch faiss-gpu   # GPU (recommended)
conda install -c pytorch faiss-cpu   # CPU fallback

Without a GPU the code still runs — it builds a CPU index and reports CPU (no GPU detected) — but points 2 and 4 lose most of their benefit.

Per-scan variant — chemembed_per_spectrum.py

By default merge_spectra_by_feature collapses every MS2 scan of a feature into a single spectrum. chemembed_per_spectrum.py annotates each scan separately, which is useful when scans differ in collision energy:

python chemembed_per_spectrum.py \
    --mgf      <sample>_per_spectrum_combined.mgf \
    --manifest <sample>_spectrum_manifest.tsv \
    --config   chemembed_config.yml \
    --out      <sample>_chemembed_per_spectrum_results.tsv

It imports the model, reference cache, matching and post-processing from chemembed_by_file.py, so behaviour stays identical, and joins results back to the manifest to recover feature_id / scan_number / collision_energy / dissociation_method.

Run as a command line interface

If you use

pip install .

pip can install the prerequisites and create an executable script which can be run where all the settings of the config file need to be passes as arguments to the executable.

chemembed --input_file_type with_smiles \
            --adduct "+" \
            --msp_file_positive MSP_FILE_POSITIVE.msp \
            --msp_file_negative MSP_FILE_NEGATIVE.msp \
            --model_path_positive MODEL_PATH_POSITIVE.bin \
            --model_path_negative MODEL_PATH_NEGATIVE.bin \
            --reference_database REFERENCE_DATABASE.pkl \
            --intensity_threshold 1 \
            --tolerance 0.01 \
            --resolution  0.01 \
            --max_mz  700 \
            --top_n_candidates 7 \
            --preprocessed_data PREPROCESSED_DATA.pkl \
            --prediction_results PREDICTION_RESULTS.csv

Configuration of config.yaml File

The application is configured through a config.yaml file, which contains several sections:

Input Files

• msp_file: Path to the input MSP file containing spectra data.
• reference_database: Path to the reference database pickle file.
• model_path: Path to the pre-trained CNN model file.

Output Files

• preprocessed_data: Path where the preprocessed data will be saved (pickle format).
• prediction_results: Path for the final prediction results CSV file. Supports variable substitution for top_n_candidates.

Parameters

• top_n_candidates: Number of top candidate molecules to retrieve from the reference database. (default: 5)
• input_file_type: Type of the input MSP file. Options are 'with_smiles' and 'without_smiles'. (default: 'with_smiles')

Example config.yaml File

Edit the config.yaml file to set up paths and parameters based on your requirements. Key options include:

Input Files:

msp_file_positive: Path to the positive mode spectra file (input_spectra_with_smile.msp if containing SMILES, or input_spectra.msp otherwise).
msp_file_negative: Path to the negative mode spectra file (sample_negative_file_without_smile.msp if not containing SMILES, or sample_negative_file.msp otherwise).
reference_database: Path to the reference database file (e.g., sample_reference_database.pkl).
Model Files:
model_path_positive: Path to the trained positive mode model.
model_path_negative: Path to the trained negative mode model.
Output Files:

preprocessed_data: Path to save preprocessed data.
prediction_results: Path to save prediction results.
Parameters:

top_n_candidates – adjust as needed.
Input Type and Adduct:
input_file_type: Set as with_smiles or without_smiles.
adduct: Choose '-' for M-H or '+' for M+H based on the adduct type.

Using Different Input File Types

• with_smiles: Use this option if your MSP file includes SMILES strings for each spectrum. The pipeline will utilize SMILES information during data processing and candidate matching, including Tanimoto similarity calculations.

• without_smiles: Use this option if your MSP file does not include SMILES strings. The pipeline will process the data accordingly and perform candidate matching without relying on SMILES information.

MSP File Format Guide

1. MSP File with SMILES

This format includes SMILES notation, which provides the molecule's structure, followed by metadata and peak data.

Format:

smile: <smile >
Precursor: <precursor m/z value>
Adduct: <adduct type>
Num Peaks: <number of peaks>
<m/z> <intensity>
<m/z> <intensity> 
... 
##### Example:
smile: Clc1ccc(cc1)S(=O)(=O)NC2CC2
Precursor: 230.0048
Adduct: [M+H]+
Num Peaks: 16
63.9623 0.15257457101400854
64.9701 0.32720519597920816
...

2. MSP File without SMILES

This format does not include SMILES notation and instead begins with a unique identifier followed by metadata and peak data.

Format:

Name: <unique identifier>
Precursor: <precursor m/z value>
Adduct: <adduct type>
Num Peaks: <number of peaks>
<m/z> <intensity>
<m/z> <intensity>
...

##### Example:
Name: ID1
Precursor: 478.1471
Adduct: [M-H]-
Num Peaks: 101
130.9882 11.57
143.6251 2.41
...

Key Fields in MSP Files

• SMILES (if present): A line representing the molecular structure.
• Precursor: The m/z value of the precursor ion.
• Adduct: Specifies the adduct type (e.g., [M-H]-).
• Num Peaks: Total number of peaks in the entry.
• Peak Data: m/z and intensity values for each peak, with one peak per line.

Dependencies

Ensure you have the following libraries installed:

• Python 3.6+
• pandas >= 1.0.0
• numpy >= 1.18.0
• rdkit >= 2020.09.1
• torch >= 1.5.0
• scipy >= 1.4.0
• pyyaml >= 5.3.0
• argparse (Built-in)

Note: rdkit is best installed via conda due to its dependencies.

Custom Modules

Ensure the cnn_train module is present in your project and contains up_cnn_model.py and spectra_inference_dataset_loader.py.

Project Structure

ChemEmbed/
├── pyproject.toml                      # Package metadata and dependencies
├── LICENSE
├── README.md
├── config.yaml                         # Configuration for main.py
├── chemembed_config.example.yml        # Template for the batch entry point
├── main.py                             # Single-file run, reads config.yaml
├── chemembed_per_spectrum.py           # Per-scan variant (not packaged)
├── tests/                              # Regression tests
└── src/
    └── chemembed/                      # The installed package
        ├── __init__.py
        ├── cli.py                      # Command-line interface
        ├── chemembed_single_file.py    # Shared run() used by main.py and cli.py
        ├── chemembed_by_file.py        # Batch annotation, FAISS candidate search
        ├── data_processing.py          # Spectra preprocessing
        ├── model_utils.py              # Model loading and prediction
        ├── reference_utils.py          # Reference database loading and matching
        ├── cnn_train/
        │   └── up_cnn_model.py
        ├── data_loaders/
        │   ├── inference_dataset_loader.py          # 'with_smiles'
        │   └── spectra_inference_dataset_loader.py  # 'without_smiles'
        └── chembert_eval/              # ChemBERTa comparison used in the paper

Additional Information

Adjusting the Python Path

If your project relies on locally stored libraries or specific directories that are not installed in standard Python paths, you may need to adjust the Python path. Here's how you can do it within your script:

import sys

# Adjust the path to include the directory where your local libraries are stored
sys.path.append('path/to/your/library')

Variable Substitution in config.yaml

The prediction_results filename in config.yaml can include ${top_n_candidates} which will be replaced with the actual number specified in top_n_candidates.

Contributing

Contributions are welcome! Please open an issue or submit a pull request for any improvements or bug fixes.

License

This project is licensed under the MIT License.

Acknowledgements

• RDKit: Open-source cheminformatics software.
• PyTorch: Deep learning framework used for model implementation.

Enjoy using the Mass Spectrometry Data Processing and Prediction Pipeline!

If you have any questions or need assistance, feel free to reach out.