liblumif

Reference C99 implementation of LUMIF-1 (.lum), the lossless Lumen image format for astronomical FITS data. It reads and writes the format, so Siril, PixInsight, Python (ctypes / cffi) or any other program can use it. It has no dependency (revision 2 of the format dropped zstd).

• Format specification: docs/LUMIF-1-spec.md
• Rust reference: crates/lumen-core/src/io/lum/; golden files: crates/lumen-core/tests/golden/

What it does

decode	whole image, or any band of rows (only the touched tiles are read and decoded), from memory or streaming from a file; all payload modes, all predictors, range coder, palette, bit shift, CRC-32 checks
encode	planar float32 (mono / RGB / CFA, NaN and every bit pattern preserved), or raw u8/i16/u16/i32/f32/f64 buffers; u16-affine / palette / f32 payload picked automatically and verified sample by sample
FITS	lumif_encode_fits() (keeps the original header) and lumif_to_fits() rebuild the source FITS byte for byte (BITPIX 8/16/32/−32/−64, 2-D or 3 planes)
errors	every function returns a lumif_status; nothing aborts, prints or leaves partial output

Build

CMake ≥ 3.16, any C99 compiler. Tested with MSVC 2022 (generator "Visual Studio 17 2022").

cmake -S libs/liblumif -B build -G "Visual Studio 17 2022" -A x64    # Windows
cmake --build build --config Release
# Linux / macOS (gcc or clang):
cmake -S libs/liblumif -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build

Options: LUMIF_BUILD_TOOLS (ON, builds lumif-cat), LUMIF_SHARED (OFF), LUMIF_BUILD_FUZZ (OFF).

FMA warning for embedders. The affine u16 maps and the FITS scaling must be evaluated as separate IEEE double operations. The code stores intermediates in volatile doubles, and the CMake file adds -ffp-contract=off; if you compile the sources yourself, keep contraction off.

To use it from your own project: add include/ to the include path and compile src/*.c, or link the lumif CMake target.

Example

#include "lumif.h"

/* ---- read: whole image, then only rows 1000..1064 of plane 0 ---- */
lumif_reader *r;
lumif_status st = lumif_open_file("m33.lum", NULL, &r);          /* NULL = default limits */
if (st) { fprintf(stderr, "%s\n", lumif_strerror(st)); return 1; }
const lumif_info *in = lumif_get_info(r);
float *px = malloc((size_t)in->width * in->height * in->planes * sizeof *px);
st = lumif_read_all(r, px);                                       /* planar float32 */
st = lumif_read_rows(r, 0, 1000, 1064, px);                       /* partial: (64 * width) floats */
for (size_t i = 0; i < in->n_cards; i++)
    printf("%s = %s\n", in->cards[i].key, in->cards[i].value);
lumif_close(r);

/* ---- write: 16-bit sensor frame (raw u16, 6248 x 4176) ---- */
lumif_write_params p;
lumif_write_params_init(&p);
p.width = 6248; p.height = 4176; p.channels = LUMIF_CFA_RGGB;
uint8_t *out; size_t n;
st = lumif_encode_raw(&p, LUMIF_RAW_U16, raw_u16, 1.0, 0.0, &out, &n);
if (!st) { st = lumif_write_file("light.lum", out, n); lumif_free(out); }

/* ---- FITS in, FITS out (byte exact) ---- */
st = lumif_encode_fits(fits_bytes, fits_len, /*keep_origin*/ 1, /*effort*/ 1, &out, &n);
/* ... later ... */
lumif_open_memory(out, n, NULL, &r);
st = lumif_to_fits(r, &fits_again, &fits_again_len);              /* == fits_bytes */

Pixels are float32 planar. Integer FITS data map to float exactly as the format defines (((raw*bscale)+bzero)/range, see lumif_raw_to_float); LUMIF_RAW_U16 is v/65535. Note that lumif_encode_raw(LUMIF_RAW_F32 …) applies that FITS formula (-0.0 + 0.0 = +0.0); pass float pixels to lumif_encode() directly when you need -0.0 kept.

On Windows lumif_open_file / lumif_write_file take paths in the ANSI code page (fopen); for Unicode paths read the file yourself (e.g. _wfopen) and use lumif_open_memory. lumif_set_threads(r, n) (0 = one per CPU) decodes the tiles of a band on n threads (pthread / Win32; compile with -DLUMIF_NO_THREADS to remove): identical output, 16.8 MB M33 light 0.52 s -> 0.28 / 0.16 / 0.10 s with 2 / 4 / 8 threads. A reader is otherwise not thread-safe; use one reader per calling thread. All input is untrusted: sizes are overflow-checked and bounded by the file size and by lumif_limits (default 2³⁴ samples, 2²⁴ samples per tile, 64 MiB of header). Hostile files can still make the decoder spend time on large declared images; lower max_samples if that matters.

lumif-cat

lumif-cat info    f.lum                   geometry, payload, header cards
lumif-cat check   f.lum                   decode whole + by row bands, print pixel CRC-32
lumif-cat dump    f.lum out.f32           planar little-endian float32
lumif-cat to-fits f.lum out.fits          byte-exact FITS rebuild
lumif-cat encode  in.fits out.lum [--no-origin] [--fast]

Verification (no unit tests: real data)

python libs/liblumif/tools/make_synthetic.py D:/lumif_synth         # 15 small FITS covering every code path
python libs/liblumif/tools/verify.py --cat build/Release/lumif-cat.exe --lumen target/release/lumen.exe \
       --golden crates/lumen-core/tests/golden  D:/lumif_synth  D:/BenchData/fits --limit 8

For each FITS it checks that (1) the Rust encoder's .lum decodes in C to the identical FITS, (2) the C encoder's .lum is decoded by the Rust CLI to the identical FITS, (3) C → C, (4) row-band reads equal the whole-image read. Results of the last run: see "Results" below.

Fuzzing (-DLUMIF_BUILD_FUZZ=ON): lumif-fuzz mutates real .lum files (bit flips, interesting 32-bit values, truncation, chunk moves) and decodes them with small limits, the input sitting in front of a guard page so any over-read faults; lumif-fuzz-nocrc is the same with the checksums disabled so mutated payloads reach the range-coder / predictor paths. Run it with the Debug build (MSVC debug CRT checks heap overruns) or with ASan/UBSan where available.

Compression benchmark (public data)

47 public FITS images (744 MB; ESO 19, NOIRLab 16, MAST 12), one machine, external codecs on one thread. Every codec was decompressed and checked bit for bit; LUMIF-1 round-tripped 47/47 files. Ratio = FITS size / compressed size. Size = compressed size relative to LUMIF-1 on the same files (1.00 = equal, higher = larger). Speeds are MB/s of FITS data. Reproduce with tools/bench_lossless/.

16-bit integer frames (22 files, 12.7 MB average)

Codec	Files	Ratio	Size	Compress MB/s	Decompress MB/s
LUMIF-1	22	2.95	1.000	23.0	19.8
JPEG XL (lossless, effort 7)	22	2.85	1.037	2.5	26.7
JPEG XL (CFA planes split)	22	2.82	1.047	1.8	26.4
JPEG-LS (CharLS)	22	2.80	1.056	93.7	116.3
JPEG 2000 (lossless 5/3)	22	2.79	1.059	19.9	22.8
JPEG-LS (CFA planes split)	22	2.76	1.069	91.8	117.7
bzip2 -9	22	2.76	1.071	17.3	23.3
xz -9e (LZMA)	22	2.66	1.109	2.9	58.4
XISF zstd-19 + shuffle	22	2.63	1.123	7.4	871.3
fpack HCOMPRESS lossless	22	2.63	1.123	66.3	52.7
brotli -q 11	22	2.54	1.162	0.7	87.9
fpack Rice	22	2.53	1.166	167.4	142.5
XISF zlib-9 + shuffle	22	2.50	1.181	48.5	556.5
zstd -19	22	2.43	1.213	3.3	170.1
fpack GZIP-2 (shuffle)	22	2.24	1.319	47.1	116.3
gzip -9	22	2.12	1.392	5.6	113.4
XISF LZ4-HC + shuffle	22	2.08	1.417	64.8	3628.1
fpack GZIP-1	22	1.87	1.580	37.3	109.2

Float (f32) images (21 files, 18.7 MB average)

Codec	Files	Ratio	Size	Compress MB/s	Decompress MB/s
LUMIF-1	21	1.78	1.000	30.1	70.1
JPEG XL (lossless, effort 7)	17	1.80	1.011	18.0	77.4
XISF zstd-19 + shuffle	21	1.74	1.020	7.8	1939.7
XISF zlib-9 + shuffle	21	1.72	1.032	13.8	951.7
xz -9e (LZMA)	21	1.70	1.047	3.1	46.3
bzip2 -9	21	1.66	1.074	14.2	18.2
XISF LZ4-HC + shuffle	21	1.62	1.099	18.8	4832.8
brotli -q 11	20	1.51	1.114	0.6	90.9
fpack GZIP-2 (shuffle)	20	1.57	1.138	45.4	126.4
zstd -19	21	1.51	1.174	4.9	244.7
gzip -9	21	1.40	1.270	11.4	112.8
fpack GZIP-1	20	1.36	1.311	32.3	105.4

Codecs that could not round-trip some files: fpack gzip-1 (1), fpack gzip-2 (shuffle) (1); lumif-1 is the smallest on 8 of 21 files; rows with fewer files are compared on their own subset.

32-bit integer frames (4 files, 17.9 MB average)

Codec	Files	Ratio	Size	Compress MB/s	Decompress MB/s
LUMIF-1	4	4.35	1.000	42.9	40.5
JPEG-LS (CharLS)	4	4.24	1.028	159.3	184.7
JPEG-LS (CFA planes split)	4	4.22	1.031	159.6	188.2
JPEG 2000 (lossless 5/3)	4	4.22	1.032	34.4	38.2
JPEG XL (lossless, effort 7)	4	4.16	1.048	1.6	48.5
JPEG XL (CFA planes split)	4	4.14	1.052	1.5	47.8
fpack HCOMPRESS lossless	4	4.11	1.060	119.2	86.4
bzip2 -9	4	4.11	1.061	27.8	30.6
fpack Rice	4	3.98	1.093	300.7	194.2
XISF zstd-19 + shuffle	4	3.91	1.115	11.9	2901.9
XISF zlib-9 + shuffle	4	3.89	1.119	18.7	1116.5
xz -9e (LZMA)	4	3.89	1.120	2.7	82.1
brotli -q 11	4	3.70	1.177	0.8	168.6
XISF LZ4-HC + shuffle	4	3.65	1.191	33.3	10618.8
zstd -19	4	3.52	1.238	3.6	296.9
fpack GZIP-2 (shuffle)	4	3.40	1.281	67.3	141.6
gzip -9	4	2.87	1.520	1.3	152.1
fpack GZIP-1	4	2.38	1.828	41.6	123.0

By archive

Archive	Files	Avg FITS MB	LUMIF-1 ratio	fpack size / LUMIF-1	xz size / LUMIF-1
ESO	19	19	2.02	1.15	1.07
NOIRLab	16	16	3.63	1.09	1.10
MAST	12	11	1.56	1.08	1.04

What the encoder simplifies

The C encoder produces valid files that the Rust reader decodes identically, but is simpler:

simplification	cost
predictor chosen by residual bit-length only (Rust also kept the runner-up in the past, +0.01 %)	≈ 0.01 %
palette limited to 65536 distinct values (Rust 2²⁴)	rare
encoder single threaded	slower than the multi-threaded Rust encoder

Results

Measured before revision 2 (zstd removed); after it C and Rust files have identical sizes on the verification corpus (ratio 1.000) and the encoders gained 7-40 % speed.

Last run (MSVC 2022 Release build vs Rust lumen.exe of the same checkout):

• golden: 5/5 .lum decode in C to the committed .fits, byte for byte.
• FITS corpus: 5 golden + 15 synthetic (make_synthetic.py) + 61 public files of tools/bench_lossless (ESO, NOIRLab, HST, JWST; u16 / i16 / f32) + 5 real M33 files (16 MB light, master dark, 97 MB f32 stack, 65 MB RGB result): in all four directions (Rust enc → C dec, C enc → Rust dec, C → C, row bands = whole image) the FITS is byte-identical. 86/86 OK.
• Size, C encoder / Rust encoder (zstd level 3): median 1.002; camera frames 1.000–1.015; M33 light 6.33 vs 6.34 MB; 97 MB f32 stack 70.91 vs 70.90 MB; four VIMOS files are 0.3 % smaller. Worst cases: the 65 MB RGB result 1.055 (Rust picks a 32-bit palette / zstd where C codes f32), a JWST f32 rate frame 1.12, files below 0.1 MB (header-dominated, up to 1.9×; f32_const 1.57×), and f64 / scaled-i32 FITS whose origin residual is noisy (1.10×) or very regular (the synthetic i32_ramp: Rust's zstd makes the residual vanish, C stores it raw, 0.07 MB vs ~0 MB).
• Speed (MSVC /O2, M33 light 16.8 MB FITS): decode 0.5 s on 1 thread (26–38 MB/s of FITS-equivalent data), 0.08 s with lumif_set_threads(r, 0) on this machine, encode 1.5 s (5.6 s for 82 MB, 8.7 s for the 97 MB f32 stack). The Rust CLI is multi-threaded.
• Robustness: lumif-fuzz / lumif-fuzz-nocrc (50 000 mutated files of the golden + 30 real C/Rust-written .lum, clang 23 AddressSanitizer + UBSan, input before a guard page): no crash, no hang, no sanitizer report (the checksum-less variant decodes ~12 % of the mutants all the way through the decoders). 4 400 random-image encode → decode round trips (any float bit pattern, NaN, ±0, inf, palettes, shifts, CFA/RGB, odd sizes, random row bands) under the same sanitizers: 0 failures.
• Compilers: MSVC 2022 /W4 and clang 23 -Wall -Wextra: no warnings in the library. gcc was not available on this machine (the code is plain C99, no extensions).

Not done / known limits: encoder is single threaded; no FITS regeneration for a .lum without stored origin (lumif_to_fits returns LUMIF_ERR_NO_ORIGIN); FITS NaN payload round-trip assumes IEEE-754 NaN propagation of the host (x86-64 SSE verified); palette > 65 536 values and zstd tile codec not produced by the encoder (but both are decoded).

Python

python/lumif.py is a ctypes binding (stdlib + numpy, no compiler needed at install time).

cmake -S libs/liblumif -B build-dll -DLUMIF_SHARED=ON -DLUMIF_BUILD_TOOLS=OFF && cmake --build build-dll --config Release
export LUMIF_LIB=build-dll/Release/lumif.dll      # or copy the library next to lumif.py

import lumif
img = lumif.read("m33.lum", threads=0)            # img.pixels float32 (planes, h, w); img.info: cards, acq, ...
band = lumif.read("m33.lum", rows=(1000, 1064), plane=0)
lumif.write("copy.lum", img.pixels, channels=img.info["channels"], cards=img.info["cards"])
lumif.fits_to_lum("in.fits", "out.lum"); lumif.lum_to_fits("out.lum", "back.fits")   # byte-exact

python/check_binding.py LUMEN_EXE files... cross-checks the binding with the Rust CLI (pixels, byte-exact FITS both ways, row bands, numpy write path).