libxtm

libxtm is a C++ library and command-line utility for compression, encoding, decoding, and analysis of terrain/elevation data. It provides a specialized .xtm format designed for efficient storage and random-access retrieval of geospatial terrain data.

Features

• Specialized Terrain Compression: predictive coding tailored for terrain data, with optional wavelet coding.
• Predictors: six deterministic models — Left, Gradient, JPEG-LS, GAP, Polynomial, and Least Squares — chosen per block by a cost estimator.
• Quadtree Partitioning: 512×512 → 64×64 hierarchical partitioning driven by a rate-cost split rule.
• Split-Precision Coding: sub-meter scales encode meter and precision planes independently.
• Geospatial Support: GDAL-based I/O with full geotransform and WKT projection preservation.
• Region of Interest (ROI) Decoding: decode only a bounding region from the .xtm file without uncompressing the dataset.
• Comprehensive Analyzer: terrain statistics, predictor performance, and compression diagnostics via xtm analyze.

Architecture

• apps/xtm/ CLI — encode, decode, analyze, info, verify
• include/ public API
• src/
  • terrain quantization, NoData inpainting
  • partition quadtree 512→64 partitioning, block views
  • predictor predictor models (Left, Gradient, JpegLs, Gap, Polynomial, LeastSquares)
  • analyzer per-block predictor selection + statistics
  • transform CDF 5/3 integer wavelet
  • coding arithmetic coder, context modeling, pipeline
  • container .xtm format (header, block index, CRC32)
  • io GDAL reader/writer
• tests/ CTest suite

The encoder and the analyzer share one pipeline implementation — for_each_superblock (src/coding/Pipeline.cpp) owns superblock slicing, quadtree partitioning, and predictor selection for both, on top of the generic parallel_for_superblocks worker pool. See docs/pipeline.md for the full technical reference.

Requirements

To build and use libxtm, you will need:

• C++20 compiler (GCC or MSVC)
• CMake >= 3.21 and Ninja (when using the presets)
• GDAL library
• (Optional) GoogleTest (fetched automatically via CMake if tests are enabled and GTest is not found system-wide)
• (Optional) Python >= 3.10 dev headers — only when building the Python bindings (-DXTM_BUILD_PYTHON=ON); nanobind is fetched automatically via CMake

Building the Project

libxtm uses CMake. Two presets are provided in CMakePresets.json:

# Configure, build, and test with a preset (Ninja required)
cmake --preset release
cmake --build --preset release
ctest --preset release

Without presets, a plain configure works too (defaults to Release if no build type is given):

mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
cmake --build . -j

Build behavior worth knowing:

• Default build type: Release if none is specified.
• Warnings as errors: -Wall -Wextra -Wpedantic + -Werror (GCC, via XTM_WERROR) or /W4 /WX (MSVC).
• Release optimizations: -O3 -march=native -mtune=native -ftree-vectorize plus LTO when supported (GCC).
• ccache is used automatically when detected (zero-config; skipped silently otherwise).
• Output layout: executables in build/bin (xtm), libraries in build/lib (libxtm_core).
• Compile database: with -DCMAKE_EXPORT_COMPILE_COMMANDS=ON (on in both presets), compile_commands.json is symlinked to the project root for IDE/clangd.
• Install: cmake --install build (GNUInstallDirs; installs xtm_core, the xtm binary, and headers).

Build Options

Run the unit tests with ctest (via ctest --preset dev|release, or plain ctest inside the build directory); the suite covers lossless round-trips, ROI-vs-full-decode equality, thread determinism, and container/header corruption handling.

Usage

The project builds an executable named xtm:

Usage: xtm <command> [options]

Commands:
  analyze   Evaluate terrain statistics and compression diagnostics
  encode    Compress a raster into the .xtm format
  decode    Decompress an .xtm file (optionally a bounding region) to a raster
  info      Show .xtm file metadata
  verify    Verify a decode against source data

Analyze

xtm analyze <input.tif> [--scale <value>] [--wavelet]

Encode

Compress an input raster (e.g., GeoTIFF) into the xtm format:

xtm encode <input.tif> -o <output.xtm> [--scale <value>] [--pipeline predictor|wavelet] [--context simple|extended] [--disable-quadtree]

Decode

Decompress an xtm file back into a raster. Optionally decode a bounding region:

xtm decode <input.xtm> -o <output.tif> [--region x y w h]

Precision model

XTM is an error-bounded codec, not a bit-exact Float32 codec. Elevations are quantized to fixed-point integers before coding, so every sample round-trips within |z - z_hat| <= scale / 2 units (the CLI default scale is 1.0, i.e. meter precision; pass --scale 0.01 for centimeter precision at the cost of larger files). The quantization is lossless with respect to the quantized integer grid: decode reproduces the exact grid the encoder produced.

Note: the experimental --pipeline wavelet mode is only valid with --scale >= 1.0.

C++ API

Alongside the CLI, libxtm exposes a high-level file-oriented API in the xtm::api namespace (include/xtm/Api.hpp). The CLI commands are thin wrappers around these functions — the API is the single source of truth for the encode/decode/analyze/info/verify flows.

#include <xtm/Api.hpp>

// Encode with centimeter precision.
xtm::api::EncodeResult result = xtm::api::encode_file(
    "input.tif", "output.xtm",
    {0.01, xtm::coding::ContextModel::Simple,
     xtm::analyzer::PipelineType::Predictor, /*disable_quadtree=*/false,
     /*num_threads=*/0}); // 0 = hardware concurrency

// Decompress only a bounding region (x, y, width, height).
xtm::api::decode_file("output.xtm", "region.tif",
                      {128, 128, 512, 512, 0});

Option structs mirror the CLI flags:

• EncodeOptions —
  • precision,
  • context_model (Simple/Extended),
  • pipeline_type (Predictor/Wavelet),
  • disable_quadtree,
  • num_threads.
• DecodeOptions —
  • region_x/region_y/region_width/region_height (width or height 0 = full grid),
  • num_threads.

Results:

• EncodeResult (dimensions, total_blocks, predictor_counts mapping frozen PredictorId → block count, output_bytes, per-stage ms timings),
• DecodeResult (blocks_decoded, dimensions), FileInfo (header + block index, block_count, total_payload_bytes), VerifyResult (passed, counts of checked/mismatched pixels).

Error contract: invalid options throw std::invalid_argument; I/O, codec, and corrupt-file failures throw std::runtime_error.

Python Bindings

The same file-level surface is available from Python via nanobind — no numpy, no runtime pip dependencies. The module can be built in-tree and used through PYTHONPATH (as wired into CTest):

cmake -S . -B build/release -DXTM_BUILD_PYTHON=ON
cmake --build build/release --parallel
PYTHONPATH=build/release/lib python3 -c "import xtm; print(xtm.version())"

or installed as a wheel (build machine needs cmake, ninja, and GDAL dev headers, matching the C++ requirements):

pip install .

The artifact lands at <build>/lib/xtm.abi3.so (STABLE_ABI, portable across CPython ≥ 3.12). A plain-assert test script (tests/python/test_xtm_bindings.py) is wired into CTest as python_bindings and runs automatically with ctest when the option is on.

import xtm

# Encode a raster (centimeter precision) and inspect the result.
res = xtm.encode("grand_canyon.tif", "gc.xtm", precision=0.01)
print(res.total_blocks, res.output_bytes, res.predictor_counts)

# Metadata without decoding payloads.
info = xtm.info("gc.xtm")
print(info.width, info.height, info.precision, info.transform)

# Decode a bounding region (x, y, width, height); region=None = full grid.
xtm.decode("gc.xtm", "gc_roi.tif", region=(0, 0, 512, 512))

# Checksum-only verify, or decode-vs-source comparison when tif is given.
v = xtm.verify("gc.xtm", "grand_canyon.tif")
assert v.passed

Functions (all run with the GIL released):

Notes:

• Predictor IDs are frozen: 0=Gradient, 1=Left, 2=JpegLs, 3=Polynomial, 4=Gap, 5=LeastSquares (EncodeResult.predictor_counts keys).
• Invalid options raise ValueError; I/O, codec, and corrupt-file failures raise RuntimeError (C++ exceptions mapped in bindings/xtm.cpp).
• Do not build the module with the dev preset (TSan+UBSan) — Python is not TSan-instrumented; use the release preset.