DS-MSP

Double Sphere & Multi-model Spherical-camera Platform — one interface to calibrate, convert, and reason about every wide-FOV camera model, from a single lens to a full rig.

A clean, tested, OpenCV-compatible Python platform for wide-FOV (fisheye / omnidirectional) cameras.

One interface covers calibration, model conversion, 3D geometry, multi-camera rigs, and hardware export — so you can swap camera models in a single line and convert between them without re-shooting images.

The namesake Double Sphere model in motion: each 3D point is traced through two spheres and projected to the fisheye image. Cross-checked against the library itself (std = 2e-16). Drive it live →

Install

pip install ds-msp                 # core library
pip install "ds-msp[calib]"        # + AprilGrid detector

What do you want to do?

Recipes

Every recipe below is a copy-paste starting point. Each links to a fuller guide; the code uses the real API.

Project & unproject

Build any model from its intrinsics, then project 3D points to pixels and unproject pixels back to unit bearing rays — exact inverses.

Every model also exposes analytic point and parameter Jacobians (no autodiff, no finite differences), so these drop straight into bundle adjustment.

import numpy as np
from ds_msp import DoubleSphereCamera

cam = DoubleSphereCamera(fx=711.57, fy=711.24, cx=949.18, cy=518.81,
                         xi=0.183, alpha=0.809, width=1920, height=1080)

pts_3d = np.array([[0., 0., 1.], [1., 1., 2.]])
px,   ok = cam.project(pts_3d)     # (N,2) pixels + validity mask
rays, ok = cam.unproject(px)       # (N,3) unit rays (exact inverse of project)

Round-trips to ~1e-13 px. The recipes below reuse this cam. → Multi-model guide

Pinhole image & new K

Rectify a raw fisheye frame to an undistorted pinhole image and get the new pinhole intrinsics K_new for it.

The balance knob trades field of view against crop:

• 0.0 keeps the widest FOV (more black border).
• 1.0 is the tightest crop (no border).

import cv2, ds_msp.cv as ds_cv

img = cv2.imread("frame.png")      # your raw fisheye frame

# object API — rectified pinhole image AND its new pinhole intrinsics
img_undist, K_new = cam.undistort_image(img)

# OpenCV-style equivalent, with explicit size + balance (0 = widest FOV … 1 = tightest crop)
K_new      = ds_cv.estimateNewCameraMatrixForUndistortRectify(cam.K, cam.D, (1920, 1080), balance=0.0)
img_undist = ds_cv.undistortImage(img, cam.K, cam.D, Knew=K_new)

Inverse-projection error across all undistort modes is < 0.00003 px. → Multi-model guide

2D/3D pose

cam.solve_pnp unprojects pixels through whichever of the 7 models you use, then selects a solver that matches the target geometry and visible sphere. Forward-only data uses the standard normalized-plane cv2.solvePnP(K=I, dist=None) path. If valid rays extend past 90°, non-coplanar targets use a bearing-vector DLT and coplanar boards use a bearing homography, so DS / UCM / EUCM / KB / OCam / DS+ share one full-sphere API.

Warning cv2.solvePnP only goes wrong if you hand it raw fisheye pixels with a pinhole K — the wrong distortion model for a wide-FOV lens. cam.solve_pnp avoids that by unprojecting first.

For noisy or mismatched correspondences, solve_pnp_robust(cam, ...) is the recommended multi-model estimator. It runs deterministic GNC-TLS directly on unit bearings, uses each model's analytic projection Jacobian to keep the noise bound locally calibrated in pixels at wide FOV, and returns a hard inlier mask. The legacy DoubleSphereCamera also exposes the same operation as a convenience method.

solve_pnp_ransac(cam, ...) remains available for callers that explicitly need classic random minimal-set consensus. On its bearing-capable path, the optional polish is full-sphere: a candidate is kept only when it preserves support and does not worsen the truncated all-data local-pixel score.

Note With the geometry-specific bearing support available, solve_pnp, solve_pnp_robust, and solve_pnp_ransac keep every model-valid bearing, including peripheral correspondences past 90°. A coplanar target needs at least 4 valid points; a non-coplanar full-sphere solve needs at least 6. The clean and compatibility APIs retain a normalized-plane forward-only fallback for an undersized non-coplanar bearing set; the recommended robust API reports failure instead.

from ds_msp import relative_pose, solve_pnp_ransac, solve_pnp_robust, triangulate_rays

# 3D pose from 2D–3D correspondences (PnP), straight on raw fisheye
points_3d = ...   # (N,3) world points (from your board / detector)
points_2d = ...   # (N,2) matching fisheye pixels
ok, rvec, tvec = cam.solve_pnp(points_3d, points_2d)

# noisy matches? deterministic GNC-TLS rejects gross outliers + returns an inlier mask
ok, rvec, tvec, inliers = solve_pnp_robust(cam, points_3d, points_2d)

# compatibility path when classic RANSAC behavior is explicitly required
ok, rvec, tvec, inliers = solve_pnp_ransac(cam, points_3d, points_2d, seed=0)

# 2D–2D → relative pose + triangulated 3D points, on bearing rays
px1 = ...         # (N,2) pixels in image 1
px2 = ...         # (N,2) matching pixels in image 2
f1, _ = cam.unproject(px1)
f2, _ = cam.unproject(px2)
R, t         = relative_pose(f1, f2)           # camera-2 pose relative to camera-1
pts_3d, _, _ = triangulate_rays(f1, f2, R, t)  # (N,3) triangulated points

Measured PnP + reprojection RMS on bundled real images: 0.43 px / 0.85 px. → Two-view geometry

Calibrate from scratch

calibrate() is model-agnostic — the same call fits any of the 7 models from a generic seed; only the seed class changes.

It runs manifold Levenberg–Marquardt with analytic Jacobians and an optional robust loss (huber / cauchy); per-view poses are re-seeded internally, so a rough focal is fine.

from ds_msp.calib import calibrate
from ds_msp.models import KannalaBrandtModel        # or DoubleSphereModel, DSPlusModel, EUCMModel, …

# your detected correspondences, per frame (e.g. from cv2 / AprilGrid):
X_world    = ...   # list of (M,3) board points
keypoints  = ...   # list of (M,2) detected pixels
visibility = ...   # list of (M,) bool masks
w, h = 1920, 1080  # image size

seed = KannalaBrandtModel(fx=0.5 * w, fy=0.5 * w, cx=0.5 * w, cy=0.5 * h)   # generic seed
result = calibrate(seed, X_world, keypoints, visibility, loss="huber", f_scale=1.0)
print(result["model"], result["rms_px"])           # calibrated model + reprojection RMS

On the capstone (real TUM-VI fisheye + AprilGrid) this matches the published intrinsics to ~0.02% focal at 0.08 px median. → Calibration capstone

Just have a folder of images, no correspondences yet? ds-msp-calibrate is a real console command from pip install ds-msp alone — detection (checkerboard / ChArUco / AprilGrid) + calibrate() above, config-driven, same pattern as the rig CLI below:

ds-msp-calibrate --init-config calib_config.yml   # write a starter config
ds-msp-calibrate --config calib_config.yml        # detect -> bundle-adjust -> report -> save

Prints the full reprojection-error distribution (not just one RMS number) and a PASS/WARN/FAIL verdict; ds_msp.calib.load_camera("camchain.yaml") loads the result straight back into a ready CameraModel instance. → Single-camera calibration guide

Convert between models (no images, no data)

Translate a calibration between any two models with no images and no recalibration: DS-MSP samples the image grid, unprojects through the source model, builds a linear seed for the target, then refines on pixel reprojection.

The result agrees to sub-pixel accuracy; the design follows Fisheye-Calib-Adapter (see Credits).

from ds_msp import convert
from ds_msp.models import DoubleSphereModel, KannalaBrandtModel

# convert() takes a model (not a Camera); same DS intrinsics as `cam` above
ds = DoubleSphereModel(fx=711.57, fy=711.24, cx=949.18, cy=518.81, xi=0.183, alpha=0.809)
kb, report = convert(ds, KannalaBrandtModel, width=1920, height=1080)   # DS → OpenCV fisheye
print(report["rms_px"])            # sub-pixel agreement; no images, no recalibration

→ Multi-model guide

Multi-camera rig extrinsics (any FOV)

Note Shipped in pip install ds-msp (ds_msp.rig) — validated on a real 8-camera rig and the MC-Calib Blender datasets (extrinsics within ~0.16% of ground truth).

pip install ds-msp alone gives you the ds-msp-calibrate-rig command; no repo clone needed. A git clone can use python scripts/calibrate_rig.py instead — same CLI either way.

A drop-in for MC-Calib: same calib_param.yml config schema, same output files (calibrated_cameras_data.yml, calibrated_objects_data.yml, …), interoperable both directions.

It calibrates the extrinsics (where each camera sits relative to the others) plus per-camera intrinsics from a folder of ChArUco images.

It adds one extension over MC-Calib: a different camera model per camera, so one bench can mix kb fisheye, radtan pinhole, and dsplus at any FOV.

pip install ds-msp
ds-msp-calibrate-rig --init-config calib_param.yml   # write a starter config
ds-msp-calibrate-rig --config calib_param.yml        # detect → reconstruct → bundle-adjust → MC-Calib output

Config options

• Point it at a camera_parameters YAML with fix_intrinsic=true to hold intrinsics fixed and solve extrinsics only.
• Point it at the same YAML without that flag to refine intrinsics too.
• Omit the YAML to calibrate everything from scratch.
• If the chosen model differs from the one in the YAML, the same lens is carried into it via convert() plus a warning.

Detected corners are saved to detected_keypoints_data.yml and reused via keypoints_path, so re-calibrating with different models takes seconds.

Measured accuracy

On the MC-Calib Blender datasets, extrinsics land within 0.16% of ground truth at sub-pixel reprojection, matching MC-Calib's own published per-camera RMS (0.92–1.07×). → Rig guide · Blender evaluation

Stereo depth

Estimate metric depth directly on raw fisheye with a sphere sweep — no rectification to pinhole, so the full wide FOV is kept.

Give the reference camera + image and the source views (each as (camera, image, R, t)), sweep a set of inverse-depth planes, and back-project to a point cloud.

from ds_msp.stereo import sphere_sweep, sweep_to_points, inverse_depth_samples

ref_cam, ref_img = cam, ...        # reference camera + its image  (... = your image)
sources = ...                      # list of (camera, image, R, t) for the other posed views
depths  = inverse_depth_samples(near=0.5, far=20.0, n=64)

depth, cost, valid = sphere_sweep(ref_cam, ref_img, sources, depths)
points = sweep_to_points(ref_cam, depth, valid)        # (N,3) metric point cloud

→ Stereo & extrinsics

Hardware LDC export

Generate a TI Jacinto LDC displacement-mesh lookup table (J7 / TDA4) directly from any calibrated model — the generator uses only the CameraModel contract, so every model in the library (and any you add) exports the same way:

from ds_msp.ldc import TI_LDC_MeshGenerator

res = TI_LDC_MeshGenerator(cam).generate_mesh_and_intrinsics(1920, 1080, downsample_factor=4, balance=0.5)
mesh_lut, K_new = res["mesh_lut"], res["K_new"]     # cam: DS, UCM, EUCM, KB, RadTan, OCam, DS⁺, ...

Isaac Sim camera LUT

Bake any calibrated model into NVIDIA RTX's generalized camera format: two float32 EXRs, an assignment manifest, and a ready Camera USDA.

# Existing DS-MSP/Kalibr calibration artifact
ds-msp lut --path ./results/camchain.yaml --camera cam0

# Or generate automatically at the end of calibration
ds-msp-calibrate --config calib_config.yml --isaac-lut

Direct named parameters are supported too; run ds-msp lut --list-models for every model's parameter names. → Isaac Sim LUT guide

Choosing a model

All 7 models live behind one CameraModel contract with analytic Jacobians, convert, and Kalibr + MC-Calib I/O.

Rule of thumb (directional, not a guarantee)

• ≥170° FOV → start with Double Sphere (the cheapest closed form that fits).
• ~120–150° → the spherical family (UCM/EUCM/DS) can collapse at these angles; if it does, use KB, or DS+ when you need a closed-form inverse.
• Always confirm with the median reprojection error, not the model class.

About DS+

• DS+ is the best-accuracy model tested and the only closed-form-invertible model that matches KB on hard lenses (~21% win on a demanding 158° full-FOV lens: 0.224 vs KB 0.285 px). The trade-off: its unproject is ~2× slower than KB (Ferrari quartic). Pick it when you need a differentiable / closed-form unproject with deterministic latency — not for raw throughput.
• KB stays the right default when unproject throughput rules and an iterative inverse is acceptable.

→ Choosing a model · DS+/EUCM+/KB case study (historical)

Verify it

Every claim is backed by a reproducible measurement; the live CI badge above is the authoritative suite status.

• Inverse projection (all undistort modes): mean error < 0.00003 px.
• 3D reconstruction of checkerboard corners: 0.835 mm mean; recovered 20.00 cm square (target 20.00).
• PnP + reprojection RMS on real test images: 0.43 px / 0.85 px.
• KB / RadTan vs OpenCV: agree to ~1e-13.
• Bundled DS calibration (both test views) converges to fx≈733.9, fy≈733.0, cx≈951.7, cy≈517.9, xi≈0.230, alpha≈0.817 at 0.57 px RMS. The reference intrinsics for this bundled data are fx≈711.6, xi≈0.183.

Note A single-view or two-view planar fit doesn't fully pin the DS focal/xi/alpha gauge (see are two models the same camera?). Judge the fit above by reprojection RMS, not by how closely the raw parameters match the reference.

pytest
bash verify_all.sh
python benchmarks/benchmark.py

Learn

DS-MSP doubles as a runnable curriculum in wide-FOV geometry. Every chapter in docs/learn/ prints a verifiable number — from the projection models through real calibration, model conversion, and the DS+/KB case study.

That case study openly documents a hypothesis the authors had to retract. Start at the Learn index.

Credits

DS-MSP stands on prior work, and credit stays prominent:

• Fisheye-Calib-Adapter — Sangjun Lee, arXiv:2407.12405. The conversion design and model set follow it.
• MC-Calib — Rameau, Park, Bailo, Kweon, CVIU 2022. The rig pipeline follows MC-Calib's design (config schema, result format, workflow); files interoperate both ways. Full credit to the MC-Calib authors.
• Double Sphere — Usenko, Demmel, Cremers, 3DV 2018 (arXiv:1807.08957; basalt-headers).
• Kannala-Brandt (2006), RadTan / Brown-Conrady (1966), OCamCalib (Scaramuzza), EUCM (Khomutenko et al. 2016), UCM (Geyer & Daniilidis / Mei & Rives).
• Kalibr (Furgale et al.), dscamera, and Muhammadjon Boboev (the original Python DS calibration this project grew from).
• Termynal — Ines Montani, MIT. Powers the animated terminal blocks in these docs; vendored from Typer, whose docs also inspired this site's style.

License

MIT-licensed (see LICENSING.md):

• MIT for the whole project — the generic library, the DS+ model, and the robust calibration/conversion engine.

SPDX: MIT. Please cite via CITATION.cff.