xllim 2.0.11

xLLiM is a C++ with Python bindings implementation of Gaussian Locally-Linear Mapping.

xLLiM

xLLiM is a C++ with Python bindings implementation of Gaussian Locally-Linear Mapping. It can be used to perform efficient inversion of high-dimensional models. xLLiM integrates features such as:

• Forward model functionals. These can be implemented as C++ or as pure Python functions. The forward model can be used to generate data following a distribution and refine GLLiM results by sampling the PDF using various strategies.
• Multi-initialization options
• Post-GLLiM refinement methods such as Importance Sampling (IS) and Iterative Mixture Importance Sampling (IMIS)
• Post-processing analysis, including confidence quantification on predictions, detection of multiple solutions, and permutation of predictions in case of signal regularity. Currently, xLLiM supports only Gaussian probability distributions, but other distributions may be added to it in the future.

xLLiM is distributed as a compiled shared library and can be installed via conda, pip, or as a Docker container. It is integrated in the Planet-GLLiM astrophysics application, that is distributed as a Docker image for local use and as a data processing service on Allgo-18.

Repository migration: xLLiM has moved to GitHub. The current repository is https://github.com/xllim-tools/xllim. The former GitLab repository (https://gitlab.inria.fr/xllim/xllim) is no longer maintained.

The xLLiM method was previously implemented in R and is available on CRAN. Other implementations of GLLiM also exist:

• A pure Python implementation: pyGLLiM
• A Julia implementation with IS and IMIS: Fast Bayesian Inversion

This xLLiM implementation is derived from Kernelo. Kernelo is obsolete and is no longer maintained.

Table of Contents

• Documentation and API reference
• Runtime Dependencies
  • Mandatory
  • Optional
• Platform support
• xLLiM installation options
  • 1. Conda-forge (recommended)
    • Installing a conda distribution
    • Installing xLLiM
  • 2. PyPI
  • 3. Docker / Apptainer (Singularity)
    • Docker
    • Apptainer / Singularity (HPC)
  • 4. Manual installation (optional)
• Using xLLiM
  • Python API vs Command-Line Interface
  • Quick start with the Python API
  • Command-Line Interface (xllim_cli)
• Further documentation
• Licence

Documentation and API reference

The API reference of the xLLiM module is available at xLLiM API reference. For more information you can find a complete scientific documentation in the Planet-GLLiM documentation.

Runtime Dependencies

Mandatory:

These packages are required for the core functionality of the library.

Name	version
Python	>=3.11, < 3.13
Numpy	>=2.0, < 3
h5py	>=3.8, < 4

For other python versions, see Other python versions.

Optional:

ENVI / GDAL Support

xllim requires GDAL specifically for ENVI hyperspectral file support. Because GDAL depends on native system libraries, installation can be complex.

Recommended (Most reliable)

Using Conda is the simplest way to handle linked C libraries:

conda install gdal

Alternative: pip + system package

If installing via apt or brew, the Python GDAL version must match the system GDAL version. If the versions do not match, the installation will fail.

Example on Ubuntu:

1. Install system libraries

sudo apt install libgdal-dev gdal-bin

2. Check the system version

gdal-config --version

3. Install the matching python wrapper (e.g., if version is 3.8.4)

pip install "gdal==3.8.4"

Platform support

The table below summarises which platforms and architectures have pre-built packages available for each distribution method.

Platform	pip	conda
Linux x86_64	✅	✅
Linux aarch64 (ARM)	—	✅
macOS x86_64	—	✅
macOS arm64 (Apple Silicon)	—	✅
Windows x86_64	—	✅

— = no pre-built package; source build may be possible (see Section 4).

Docker and Apptainer: the container image is built for Linux x86_64, but containers are designed to run on any host OS (Linux, macOS, or Windows) without requiring any native installation. If you are on macOS or Windows and do not want to manage a conda environment, this is the simplest path. See Section 3 for details.

xLLiM installation options

xLLiM can be installed and used in several ways, depending on your environment and preference.

1. Conda-forge (recommended)

The easiest way to install xLLiM is via a conda-compatible package manager. xLLiM is published on conda-forge, a community-maintained channel that provides up-to-date, cross-platform packages. The corresponding xllim-feedstock contains the conda packaging recipe and tracks each release. That is where the conda recipe, managing xllim conda builds, should be updated if needed (refer to xllim-feedstock README for instructions).

Installing a conda distribution

If you don't already have conda installed, several distributions are available. They all provide the conda command (or a drop-in equivalent) and can install packages from conda-forge:

Distribution	Description	Docs
Micromamba	Standalone, fast C++ reimplementation of conda. No base environment, no Python required. Defaults to conda-forge.	mamba.readthedocs.io
Miniforge	Community-maintained minimal installer. Defaults to conda-forge.	github.com/conda-forge/miniforge
Miniconda	Minimal installer from Anaconda. Defaults to the defaults channel (see warning below).	docs.conda.io
Anaconda	Full-featured distribution with many pre-installed packages and a GUI. Defaults to defaults channel (see warning below). Heavier download.	docs.anaconda.com

Recommended: Micromamba or Miniforge. Both default to conda-forge exclusively, so environments are clean and consistent. They are also significantly faster than the classic conda solver.

Example: installing Micromamba on Linux

"${SHELL}" <(curl -L micro.mamba.pm/install.sh)

This installs the micromamba binary and adds a shell initialisation block to your shell config. Restart your shell (or source ~/.bashrc), then verify:

micromamba --version

On macOS the same command works. On Windows, see the Micromamba installation docs.

Installing xLLiM

Once you have a conda distribution available, create and activate an environment (<my-env> is a placeholder):

# With conda or mamba
conda create -n <my-env>
conda activate <my-env>

# With micromamba
micromamba create -n <my-env>
micromamba activate <my-env>

Then install xLLiM (replace conda with micromamba if applicable). This will also install the required dependencies.

conda install -c conda-forge xllim

⚠️ Anaconda/Miniconda users: channel conflict warning. Anaconda and Miniconda default to Anaconda's proprietary defaults channel, which is not the same as conda-forge. Mixing channels (e.g., adding -c conda-forge to a defaults-based environment) can lead to subtle dependency conflicts and broken environments, because the two channels build packages independently and their ABI guarantees are not always compatible. If you use Anaconda or Miniconda, we strongly recommend creating a conda-forge-only environment before installing xLLiM:

conda create -n <my-env> -c conda-forge --override-channels
conda activate <my-env>
conda config --env --add channels conda-forge
conda config --env --set channel_priority strict

See the conda-forge documentation on channel conflicts for more details.

2. PyPI

Available wheels

We've built and tested wheels for Linux x86_64 (manylinux_2_28) - Python 3.11 and 3.12. If you're in one of these configurations, you can install xllim easily with:

pip install xllim

Mac/Windows

If you're on Mac or Windows, and are fine with Python 3.11/12, we recommend you either:

• install with conda (Section 1.)
• use our pre-built docker image (Section 3.)

Other Python versions

If you want to use a different Python version (3.10, 3.13, 3.14...): They've not been tested as some of xLLiM dependencies exclude these versions. It might be possible though. Then you may try:

• Building from the sdist (pip install xllim): when no pre-built wheel matches your Python version or platform, pip automatically falls back to downloading the source distribution and compiling the C++ extension on your machine. The Python-side build tools (scikit-build-core, cmake, ninja, numpy, pybind11) are installed automatically. However, you must first install the native system libraries manually (compiler, Armadillo, a BLAS/LAPACK implementation, Boost, carma - see Section 4. for details). .
• Compiling, building and installing from scratch (advanced users) (Section 4.)

3. Docker / Apptainer (Singularity)

Pre-built container images are published for every release. Two formats are available: a standard Docker image for workstations and cloud environments, and an Apptainer / Singularity SIF image for HPC clusters where Docker is unavailable.

Docker

A minimal Docker image based on python:3.11-slim is available, with xLLiM pre-installed.

Prerequisites: install Docker Engine (Linux/macOS) or Docker Desktop (Windows).

Pull the image:

docker pull ghcr.io/xllim-tools/xllim/xllim:latest

Note: specific versions are also available. See xLLiM GHCR.

Run your project by mounting your source directory:

docker run -it \
  -v $(pwd):/workspace \
  -w /workspace \
  ghcr.io/xllim-tools/xllim/xllim:latest \
  python main.py

You can also open an interactive shell:

docker run -it -v $(pwd):/workspace -w /workspace ghcr.io/xllim-tools/xllim/xllim:latest bash

Apptainer / Singularity (HPC)

Apptainer (formerly Singularity) is a container runtime designed for HPC environments where Docker is typically not available or not allowed. It runs rootless and is natively supported on most HPC schedulers (SLURM, PBS, etc.).

For every release, a SIF (Singularity Image Format) image is automatically built from the Docker image and published to the same GHCR registry with a -sif tag suffix.

Prerequisite: Apptainer must be installed on your system (typically already available on HPC clusters - check with your sysadmin).

Pull the latest SIF image:

apptainer pull xllim.sif oras://ghcr.io/xllim-tools/xllim/xllim:latest-sif

Or pull a specific version (replace <version> with the desired release tag):

apptainer pull xllim.sif oras://ghcr.io/xllim-tools/xllim/xllim:<version>-sif

Available versions: xLLiM GHCR.

Run your project by binding your working directory:

apptainer run -B $(pwd):/workspace --pwd /workspace xllim.sif python main.py

Or open an interactive shell:

apptainer shell -B $(pwd):/workspace xllim.sif

4. Manual installation (optional)

If you need to compile xLLiM from source (e.g., for an unsupported platform or Python version, or for development), please refer to the requirements below.

Build Dependencies

These are required only if you are compiling the library from source.

Name	Version	Notes
C++ compiler	C++17 support	System-installed (e.g., g++ >= 9, clang++ >= 5)
CMake	>= 3.21	Auto-installed by pip when using pip install .
Ninja	any	Auto-installed by pip when using pip install .
scikit-build-core	>= 0.7.0, < 1	Auto-installed by pip when using pip install .
Python	>= 3.11, < 3.13	System-installed; see Other Python versions for other versions
Numpy	>= 2	Auto-installed by pip when using pip install .
Pybind11	>= 2.12	Auto-installed by pip when using pip install .
Armadillo	>= 12.6, < 13	System-installed
Boost	>= 1.78, < 2	System-installed; components: system, thread, random
BLAS / LAPACK	any	System-installed; any conforming implementation is accepted (OpenBLAS, MKL, Apple Accelerate…)
Carma	>= 0.8.0	System-installed (see installation notes below)

Build from source

Since xLLiM contains C++ extensions with Python bindings, building from source requires native system libraries that pip cannot install automatically. The instructions below use a Debian/Ubuntu-based system as example - adapt package names for your distribution.

1. Clone the project

git clone https://github.com/xllim-tools/xllim.git
cd xllim

2. Install system dependencies

sudo apt update

# Compilation tools
sudo apt-get install -y --no-install-recommends g++ cmake ninja-build

# BLAS/LAPACK - OpenBLAS is used here as an example; any conforming implementation works (MKL, etc.)
sudo apt-get install -y --no-install-recommends libopenblas-dev liblapack-dev

# Armadillo
sudo apt-get install -y --no-install-recommends libarmadillo-dev

# Python development headers
sudo apt-get install -y --no-install-recommends python3-dev

# Boost (components: system, thread, random)
sudo apt-get install -y --no-install-recommends libboost-dev libboost-system-dev libboost-thread-dev libboost-random-dev

⚠️ Boost version: xLLiM requires Boost >= 1.78.0 (enforced by CMake). Ubuntu 24.04+ ships a compatible version out of the box. On older distributions (e.g., Ubuntu 22.04 ships 1.74), you will need to build Boost >= 1.78.0 from source:

curl -L https://archives.boost.io/release/1.78.0/source/boost_1_78_0.tar.gz -o boost.tar.gz
tar -xzf boost.tar.gz
cd boost_1_78_0
./bootstrap.sh --prefix=/usr/local
./b2 --with-system --with-thread --with-random link=static variant=release -j$(nproc) install
cd .. && rm -rf boost_1_78_0 boost.tar.gz

After building Boost to /usr/local, you must tell CMake where to find it by setting the CMAKE_ARGS environment variable before calling pip install .:

export CMAKE_ARGS="-DBoost_ROOT=/usr/local -DBoost_NO_SYSTEM_PATHS=ON -DBoost_USE_STATIC_LIBS=ON"

carma - not available in standard apt repositories; build from source:

curl -L https://github.com/RUrlus/carma/archive/refs/tags/v0.8.0.tar.gz -o carma.tar.gz
tar -xzf carma.tar.gz
cd carma-0.8.0 && mkdir build && cd build
cmake -DCARMA_INSTALL_LIB=ON ..
cmake --build . --config Release --target install
cd ../..
rm -rf carma-0.8.0 carma.tar.gz

Tip: if you are using conda, all of the above can be installed in one step:

conda install armadillo boost libblas liblapack carma

This pulls OpenBLAS by default; substitute a different blas build string (e.g., libblas * *mkl) to use an alternative BLAS implementation.

3. Build and install

pip install .

This uses scikit-build-core to drive the CMake build automatically. The Python-side build tools (scikit-build-core, cmake, ninja, numpy, pybind11) are fetched by pip as needed.

4. Verify the installation

python3
>>> import xllim

Manual CMake build (for development)

For C++ development or debugging, you can drive CMake directly:

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Debug -DXLLIM_BUILD_TESTS=ON
cmake --build build
ctest --test-dir build

The build type can be either Debug (-O0 -g) or Release (-O2 -DNDEBUG). To install the compiled module into your Python environment, use pip install . as described above.

Using xLLiM

xLLiM can be used in two ways: as a Python library (import xllim) for full programmatic control, or via the command-line interface (xllim_cli) for a guided, file-based workflow. Both interfaces use the same underlying C++ engine.

Python API vs Command-Line Interface

xLLiM exposes two interfaces that address different needs, and they can be used together.

The Python API (import xllim) gives you direct access to all objects and intermediate results. You pass parameters as function arguments, work with NumPy arrays in memory, and have full control over the workflow. It is the natural choice for Jupyter notebooks, custom scripts, or integration into a larger pipeline.

The CLI (xllim_cli) provides a guided, file-based workflow: all configuration, training data, and trained model parameters are stored in a single HDF5 file. You configure it once via interactive prompts, then run generate, train, and predict as separate commands. The resulting .h5 file is a self-contained, shareable experiment record, useful for reproducibility or when you want to run the full pipeline without writing code.

Quick start with the Python API

The examples/ directory contains two Jupyter notebooks that walk you through the full workflow:

• example.ipynb: Complete walkthrough using the basic TestModel: data generation, GLLiM training, prediction, importance sampling, and post-processing analysis with plots.
• example_hapke.ipynb: Same workflow applied to the HapkeModel photometric model using real BRDF geometry data.

For details on array shapes, dimension conventions (K, D, L, N, L_t, L_w), and parameter matrices (Pi, A, B, C, Gamma, Sigma), see the Useful information page in the documentation.

Command-Line Interface (xllim_cli)

The CLI provides a guided, no-code workflow where all configuration, training data, and trained model parameters are stored in a single HDF5 file.

After installing xllim, the xllim_cli command is available directly in your terminal:

xllim_cli --help

Windows with conda: on Windows, the xllim_cli console script may not work when xLLiM was installed via conda (the launcher embeds a build-time Python path that doesn't exist at runtime). Use python -m xllim.cli instead:

python -m xllim.cli --help

Available commands

Command	Description
edit <model.h5>	Interactively configure all sections (functional model, generator, GLLiM, prediction, importance sampling, output)
print <model.h5>	Print a summary of the HDF5 file contents (-v for detailed view)
generate <model.h5>	Generate synthetic training data using the configured functional model and generator
train <model.h5>	Train the GLLiM model using training data and configuration from the file
predict <model.h5> <observations> <output>	Run the full prediction workflow (load model, predict, optional IS, write results)
import <type> <source> <model.h5>	Import geometries (JSON/NPZ) or train_data (NPZ) into the HDF5 file
export <type> <model.h5> <output>	Export geometries, train_data, or trained_gllim to NPZ
copy <source.h5> <dest.h5>	Interactively copy sections between two HDF5 files

Typical CLI workflow

# 1. Create and configure a new model file (interactive prompts)
xllim_cli edit my_model.h5

# 2. (Optional) Import geometries from an existing JSON file
xllim_cli import geometries geometries.json my_model.h5

# 3. Generate training data
xllim_cli generate my_model.h5

# 4. Train the GLLiM model
xllim_cli train my_model.h5

# 5. Inspect the file
xllim_cli print my_model.h5 -v

# 6. Run predictions on observations
xllim_cli predict my_model.h5 observations.json results/ -f npz

# 7. Export trained parameters for use elsewhere
xllim_cli export trained_gllim my_model.h5 trained_params.npz

The edit command walks through six configuration sections in order:

1. Functional model: choose from Hapke, Shkuratov, External (custom Python class), or Test model, each with its own sub-options.
2. Data generator: dataset size N, generator type (sobol, random, latin hypercube), noise covariance, seed.
3. GLLiM model: number of components K, covariance types (full/diag/iso), initialisation parameters, and training variant (GLLiM or JGMM).
4. Importance sampling: IMIS parameters (N_zero, B, J, covariance, seed).
5. Prediction module: merging options (K_merged, threshold), IS flags.
6. Output: which results to write (fullGMM, mergedGMM, IS variants).

The predict command accepts observations in JSON, NPZ, or ENVI format (ENVI requires GDAL, see Optional dependencies). Results are written as NPZ files by default.

Further documentation

• API reference: Full module documentation (classes, methods, parameters).
• Useful information: Dimension conventions, array shapes, and workflow summaries.
• Planet-GLLiM documentation: Scientific background and application-level documentation.

Licence

This software is licensed under the GNU GPL-compatible CeCILL-C License. While the software is free, we would appreciate it if you send us an email at [e-mail] to let us know how you use it. Also, please contact us if the licence does not meet your needs.