zcmir

Multimodal image registration on the GPU, in the browser. zcmir aligns two images of the same region taken with different instruments or contrasts, such as EBSD maps to polarized light micrographs, near-infrared to RGB aerial photos, or fluorescence to bright field.

Run the app

Open it at https://zacharyvarley.github.io/zcmir/, or run it locally:

pip install zcmir
python -m zcmir

This serves the app on http://127.0.0.1:4180/ and opens it in your browser. Everything runs on your GPU inside the page, so your images never leave your machine. You need Chrome or Edge 137+ (WebGPU and JavaScript Promise Integration). --port picks another port (a free one is used if 4180 is taken), and --no-browser only prints the URL. pip also installs a zcmir command that does the same; like any package's command, it exists only inside the environment you installed into.

To install from a clone instead, run pip install . in it. The first build downloads Zig and wgpu-native into .deps/ and compiles for your machine, which takes a minute or two.

Open a moving and a fixed image (or drop both on the window), or pick one of the Examples, then press Auto. Each step also has its own card, settings and play button:

The Maps tab shows the similarity at every translation, and at every rotation × scale, around the current pose. Stack mode registers serial sections or a time series slice by slice. It carries each pose to the next slice, retries keypoints when the score drops, and flags slices that break from their neighbours' trend. A registered pair exports its transform and a displacement field (NRRD plus an ITK .tfm); a stack exports as a TIFF stack or PNG files.

How the similarity is computed

Mutual information (MI) is the standard similarity for images of different modalities. Heldmann et al. [1] and August and Kanade [2] originated estimating it densely with FFTs and the convolution theorem. Öfverstedt, Lindblad and Sladoje [3] made it fast for global alignment: binned MI at every translation, from FFT cross-correlations of the intensity-bin indicator images. Its cost grows with the number of bin pairs.

I proposed replacing binned MI with cheaper estimates built from a few moments, so the whole map takes a small, fixed number of FFTs. I first tried the Edgeworth approximation of Rubeaux et al. [4], which is very inexpensive in moments and FFTs. In my thesis [5] I extended it to fourth order from non-central moments (still available as the E4 score), but I found these approximations numerically unstable.

After exploring the literature I settled on square-loss mutual information (SMI) [6]. Each image is described by four channels: its rank r (histogram-equalized intensity), r², r³, and the Sobel gradient magnitude of r. With this linear basis, SMI is the squared norm of the whitened cross-covariance of the two images' channels over their overlap. Every term is a sum over the overlap, so the score at every translation takes 45 FFT cross-correlations, however many intensity levels the images have. The same moments give Refine its score, gradient and Gauss–Newton curvature.

The λmax score (in the Refine card's Score menu) also works well, especially for images without sharp boundaries. It is the Gaussian-copula MI of the top canonical correlation between [z, z², z³] of each image's normal scores.

POS-GIFT in zcmir

POS-GIFT [7] gives good matches across a huge variety of multimodal registration tasks, but its authors released it only as closed MATLAB code. I reverse engineered it, reimplemented it on the GPU, and modified it. zcmir's defaults differ from the authors' code as follows:

I chose these changes on EBSD ↔ BSE sections and rotated cross-modal pairs, where they reduced gross failures. Per-keypoint orientation is reliable within one modality but often wrong across modalities, hence the global rotation search. zcmir.register(moving, fixed, pg_released=True) reproduces the authors' behaviour, and the app's Detect card has switches for the rotation search and the keypoint map.

Python

The same engine is available as plain Python functions:

import zcmir

r = zcmir.register(moving, fixed)                 # keypoints → robust fit → refine (homography)
r.H, r.score                                      # 3×3 moving → fixed (1-based pixels)
warped = r.warp(moving)
r = zcmir.register(moving, fixed, spline=True)    # + a cubic B-spline
s = zcmir.register_stack(moving_slices, fixed_slices)
s.save_tiff("registered.tif", moving_slices)

Images are numpy arrays, or file paths if Pillow is installed (pip install zcmir[images]). zcmir.settings() lists every option, and every function accepts them as keywords.

Tutorials

Three notebooks in examples/ work through one example pair, two Landsat 8 views of a river in flood and dry seasons, with a known pose:

CI runs them on a GitHub M1 runner against the freshly built wheel. The executed notebooks, with every figure, are at https://zacharyvarley.github.io/zcmir/tutorials/ and attached to each release. To run them yourself, pip install zcmir matplotlib scipy jupytext, then open the .py files as notebooks or run python scripts/run_tutorials.py out/.

Building and contributing

The registration code is Zig and WGSL. It compiles to a native library (Vulkan, Metal or Direct3D 12 through wgpu-native) and to WebAssembly for the browser, so the app and the Python package run the same code. See CONTRIBUTING.md.

License

MIT (LICENSE). The wheels bundle wgpu-native (MIT / Apache-2.0). The example images keep their own terms, public domain (NIST, NASA / USGS) or CC BY 4.0 (sources).

References

1. S. Heldmann, O. Mahnke, D. Potts, J. Modersitzki, B. Fischer. Fast computation of mutual information in a variational image registration approach. Bildverarbeitung für die Medizin (BVM) 2004, pp. 448–452. doi:10.1007/978-3-642-18536-6_92
2. J. August, T. Kanade. The role of non-overlap in image registration. Information Processing in Medical Imaging (IPMI) 2005, LNCS 3565, pp. 713–724. doi:10.1007/11505730_59
3. J. Öfverstedt, J. Lindblad, N. Sladoje. Fast computation of mutual information in the frequency domain with applications to global multimodal image alignment. Pattern Recognition Letters 159 (2022) 196–203. doi:10.1016/j.patrec.2022.05.022
4. M. Rubeaux, J.-C. Nunes, L. Albera, M. Garreau. Medical image registration using Edgeworth-based approximation of mutual information. IRBM 35(3) (2014) 139–148. doi:10.1016/j.irbm.2013.12.004
5. Z. T. Varley. Algorithms for Crystallography in the Scanning Electron Microscope. PhD thesis, Carnegie Mellon University, 2024.
6. M. Sugiyama. Machine learning with squared-loss mutual information. Entropy 15(1) (2013) 80–112. doi:10.3390/e15010080
7. Z. Hou, Y. Liu, L. Zhang. POS-GIFT: A geometric and intensity-invariant feature transformation for multimodal images. Information Fusion 102 (2024) 102027. doi:10.1016/j.inffus.2023.102027