XCalib

Camera-LiDAR cross-modal matching for edge devices, behind one small API:

• Match every camera box against every LiDAR box with a pretrained matcher, then pick the pairs: optional one-to-one assignment, gated by projection geometry once you pass a calibration (pair selection).
• Calibrate without targets (PnP/RANSAC), or nudge a site's existing matrix and keep the change only if the reprojection error improves (deployed site).
• Adapt on site without labels (one-shot), or keep learning from human-verified pairs with a gated continual refine (continual refine).
• Capture and review the frames the matcher is lost on, and fix the wrong pairs in a Qt window (review GUI).
• Deploy with ONNX/TensorRT export and train on HDF5 caches.

xcalib is the deployment package for the matching front-end studied in Position Encoding in Detection-Based LiDAR–Camera Matching: A Diagnostic Study at Infrastructure Sites (accepted at IEEE Sensors Letters). The same weights run on a lab workstation and an NVIDIA Jetson AGX Thor. It builds on the lab's prior targetless calibration framework CalibRefine (Cheng et al., IEEE TIM 2026, arXiv:2502.17648).

Status — publicly released and actively developed. The public API (xcalib.__all__) is stable across minor versions; internals may change without notice. Broader visibility is expected once the accompanying IEEE Sensors Letters paper is published. New versions are cut from main as tagged releases.

Docs: radar-lab.github.io/XCalib

Install

pip install XCalib            # inference
pip install "XCalib[onnx]"    # + ONNX export / onnxruntime parity checks
pip install "XCalib[train]"   # + h5py (HDF5 training / fine-tuning)
pip install "XCalib[gui]"     # + PySide6 (the xcalib-review window)

For normal use, install the wheel from PyPI once it is published; downloading or building from source is not required. Source checkouts are for development, paper validation, or local experimentation.

Requires Python 3.10+ (CI tests 3.10 through 3.14). The wheel pins torch>=2.2 only. On Jetson AGX Thor the default PyPI torch (>= 2.12, aarch64 + bundled CUDA 13) works out of the box; on x86 install the CUDA build you need first and pip will leave it alone. After install the xcalib CLI is on $PATH and configs resolve from package data — no checkout required.

For development, pixi drives everything (the manifest lives in pyproject.toml under [tool.pixi.*]):

pixi install            # per-platform torch (CUDA 12.8 on x86, PyPI CUDA-13 on Thor)
pixi run test           # full suite
pixi run smoke          # quick synthetic-frame smoke tests

Quickstart

Prefer a runnable, end-to-end walkthrough (install → Hub weights/data → match → calibrate, with inline overlays)? Open demo/xcalib_quickstart.ipynb locally or in Colab. The continual-refine flow has its own notebook, demo/refine_unification.ipynb.

import cv2, numpy as np
from xcalib import Matcher

matcher = Matcher.from_pretrained("crlite", site="a9_dataset_r02_s01")
# ... or local: Matcher.from_pretrained("crlite", weights="checkpoints/crlite_custom_best.pth")

image       = cv2.cvtColor(cv2.imread("frame.jpg"), cv2.COLOR_BGR2RGB)  # H,W,3 uint8
point_cloud = np.load("frame_lidar.npy").astype(np.float32)             # N,3 (X,Y,Z)
bboxes_2d   = np.array([[100, 200, 180, 320]], dtype=np.float32)        # K,4 (x1,y1,x2,y2)
bboxes_3d   = np.array([[5.0, -2.0, -1.5, 8.0, 0.5, 0.5]], dtype=np.float32)  # M,6

result = matcher.match(image, point_cloud, bboxes_2d, bboxes_3d,
                       top_k=5, validate="warn")
print(result.similarity)    # K x M float32 in [0, 1]
print(result.matches)       # [(img_idx, lid_idx, score), ...]

Everything a deployment needs hangs off the Matcher:

matcher.pair(...)                                  # alias of match()
matcher.build("onnx", output_dir="onnx/crlite")    # export + torch parity check
matcher.build("trt", precision="fp16")             # Jetson: trtexec + shape profiles

K = xcalib.CameraIntrinsics(fx=1418., fy=1422., cx=976., cy=606.)
calib = matcher.calibrate(image, pc, b2, b3, intrinsics=K)   # targetless PnP/RANSAC extrinsics

result = matcher.match(image, pc, b2, b3,          # result.assignment: optimal one-to-one pairs,
                       one_to_one=True,            #   gated by the calibration's projection
                       calibration=calib)

session = matcher.oneshot(K)                       # label-free adaptation:
session.observe(image, pc, b2, b3)                 #   confident matches -> projection
session.adapt(steps=50)                            #   -> pseudo-labels -> adapter updates
session.save("runs/site_adapted")

matcher.train("train.h5", "val.h5", epochs=10)     # fine-tune loaded weights
matcher.save_pretrained("runs/my_model")           # weights + config

A deployed site can nudge a matrix it already trusts, and save the frames the matcher gets lost on so a person can fix the pairs in xcalib-review:

rec = xcalib.SceneRecorder("capture/site1", camera="cam_front")
session = matcher.oneshot(K, recorder=rec)   # saves frames with detections but no confident pair
session.set_calibration(calib)               # start from the matrix the site already trusts
session.observe(image, pc, b2, b3)           # ... once per frame
session.calibrate(refine=True)               # nudge it; kept only if the median error drops

The whole flow is in deployed site.

Training and dataset helpers:

from xcalib import train, load_dataset

best = train("crlite", "a9_dataset_r02_s01", "a9_dataset_r02_s01",
             site="a9_dataset_r02_s01", epochs=100, warmup_epochs=5,
             optimizer="adam", scheduler="cosine")          # paper recipe
loader = load_dataset("a9_dataset_r02_s01", split="test")   # Hub or local cache

To keep improving a site's matcher from human-verified pairs instead of HDF5 caches, see Continual refine.

External configs work the same way as packaged configs, so users can keep a custom YAML next to custom weights:

matcher = Matcher.from_pretrained(
    "crlite",
    weights="runs/site42/best.pth",
    config="runs/site42/crlite_site42.yaml",   # outside the wheel/package
)

Architecture keys are read from that YAML before the model is built. Common knobs include embed_dim, token_len, top_k, crop_size, point_cloud_size, vit.depth, vit.num_heads, gnn.num_layers, gnn.hidden_dim, similarity_head.hidden_dims, and the CalibRefine dense.* widths. Changing shapes requires weights trained with the same cfg; released checkpoints should stay with their packaged configs.

The exact input contract (RGB uint8, finite float32 meters, shape and sync requirements) is specified in docs/protocol.md and enforced by validate="strict" | "warn" | "off".

See it in action

Rendered from the public A9 (TUMTraf intersection) test cache with real checkpoints (scripts/make_demo_media.py):

matcher.match() scores every camera x LiDAR detection pair and locks the confident argmax per camera box (green links span the image and the bird's-eye LiDAR view); below-threshold rows stay unmatched on purpose.

Those matches are what matcher.calibrate() feeds to PnP/RANSAC to recover the camera-LiDAR extrinsics without targets — once calibrated, the cloud and 3D detections land on the scene. One-shot adaptation (matcher.oneshot()) then uses that projection as a label-free supervisor to keep adapting the matcher on-site. Details and caveats: docs.

Model zoo

calibrefine is our standalone port of the Common Feature Discriminator from CalibRefine (Cheng et al., IEEE TIM 2026, arXiv:2502.17648), kept as the pairwise baseline the paper compares against.

Each released model ships with a matching checkpoint and config. See docs/hub.md for public loading and download conventions; docs/adding-models.md for how new models land here.

Datasets

HDF5 matching caches (schema: docs/hdf5-format.md):

• a9_dataset_r02_s01: public Hub artifact for the accepted-paper release.

xcalib pull-dataset --site a9_dataset_r02_s01 --split test

load_dataset(site, split) prefers a checkout-local cache and falls back to released Hub artifacts when available. The A9 caches derive from the TUM Traffic (A9) dataset (CC BY-NC-ND 4.0); users remain responsible for following upstream dataset terms.

The A9 caches also carry each LiDAR box as an oriented cuboid. By default the loader hands out the boxes exactly as the cache stores them, which is what every released weight and reported number uses. box_source="envelope" (accepted by load_dataset, train and Matcher.train) hands out the extents that enclose each cuboid instead, so the axis-aligned point-cloud crop keeps vehicles that are turned against the axes whole. Weights are tied to the source they were trained with; see 3D boxes: stored and envelope.

CLI

xcalib demo         --model crlite_vit_exp1 --weights ... --device cpu
xcalib export-onnx  --model crlite --site a9_dataset_r02_s01
xcalib build-trt    --model crlite --site a9_dataset_r02_s01 --precision fp16
xcalib pull-weights --model crlite --site a9_dataset_r02_s01
xcalib pull-dataset --site a9_dataset_r02_s01 --split test
xcalib version
xcalib-review --scene capture/site1 --camera cam_front   # review window; needs XCalib[gui]

Reproducing the paper

Validation commands, public A9 Table I numbers, frozen UTC reference artefacts, and latency benchmarks moved to docs/paper.md.

Public reproducibility is anchored on A9. UTC3/UTC4 entries are frozen reference artefacts for manuscript reconciliation and can only be rerun by partners who hold the institutional UTC HDF5 caches and weights; those data are not shipped in this public repository.

pixi run validate-a9          # public A9 Table I column
pixi run benchmark            # synthetic latency figure, no UTC data required

Frozen evaluation summaries cited by the supplemental: docs/evidence/index.md.

Repository layout

./
├── pyproject.toml            packaging + pixi env + ruff (single config file)
├── mkdocs.yml                docs site (GitHub Pages)
├── AGENTS.md                 entry point for contributors and coding agents
├── src/xcalib/               the package (ships in the wheel)
│   ├── engine/               matcher, wrappers, exporter, trt, trainer
│   ├── models/               architectures + registry
│   ├── oneshot/              PnP/RANSAC calibration + refinement, pseudo-labels, adapters
│   ├── continual/            corrections corpus, gated continual refine, revisions
│   ├── gui/                  `xcalib-review` Qt window + scene IO
│   ├── hub/                  Hub download helpers for weights + datasets
│   ├── data/                 cropping helpers + HDF5 loader (lazy h5py)
│   ├── utils/                config, io, metrics, torch_check
│   ├── cfg/                  packaged per-(model, site) YAML configs
│   ├── assign.py             one-to-one assignment + geometric gate
│   ├── capture.py            SceneRecorder: save frames for review
│   ├── calibration_refine.py physical-pose refinement candidate + validation gate
│   ├── visualization.py      matching + calibration overlays
│   ├── protocol.py           input contract (validate_frame_inputs, intrinsics)
│   └── cli.py                `xcalib` console entry point
├── scripts/                  public demos, smoke helpers, and docs guards
│   └── paper/                paper validation, export, TRT, and benchmark scripts
├── demo/                     quickstart + continual-refine notebooks, numbered scripts
├── tests/                    grouped pytest suite (smoke/unit/integration)
├── docs/                     docs site sources (see mkdocs.yml nav)
│   ├── agents/               contributor workflow, code index, conventions
│   └── evidence/             frozen JSON evidence linked from the paper docs
├── checkpoints/              optional local .pth files (gitignored)
└── datasets/                 optional local HDF5 caches (gitignored)

API stability

Only the symbols exported by xcalib.__all__ (Matcher, train, load_dataset, result dataclasses, pair-selection and deployed-site helpers, protocol types) are stable across minor versions. Anything imported from submodules is internal and may change without notice.

License and citation

Code is licensed under Apache-2.0. The A9/TUM Traffic dataset has its own CC BY-NC-ND 4.0 terms.

If you use this package in academic work, please cite the xcalib paper. The entry below is accepted-paper metadata; replace the paper URL / DOI once the publisher page is live:

@article{guo2026xcalib,
  author  = {Guo, Lihao and Tang, Jiahao and Bang, Tam and Zhang, Tianya and
             Harris, Austin and Sartipi, Mina and Cao, Siyang},
  title   = {Position Encoding in Detection-Based LiDAR--Camera Matching:
             A Diagnostic Study at Infrastructure Sites},
  journal = {IEEE Sensors Letters},
  year    = {2026},
  note    = {Accepted. Paper URL pending. Code:
             https://github.com/radar-lab/XCalib},
}

For the calibrefine baseline and the auto-calibration framework this work builds on, cite:

@article{cheng2026calibrefine,
  author  = {Cheng, Lei and Guo, Lihao and Zhang, Tianya and Bang, Tam and
             Harris, Austin and Hajij, Mustafa and Sartipi, Mina and
             Cao, Siyang},
  title   = {CalibRefine: Deep Learning-Based Online Automatic Targetless
             LiDAR-Camera Calibration With Iterative and Attention-Driven
             Post-Refinement},
  journal = {IEEE Transactions on Instrumentation and Measurement},
  volume  = {75},
  year    = {2026},
  note    = {arXiv:2502.17648. Code:
             https://github.com/radar-lab/Lidar_Camera_Automatic_Calibration},
}

Contact: Lihao Guo — leolihao@arizona.edu.