PanorAi

Convention-safe projection and computer vision for central spherical images.

PanorAi connects equirectangular panoramas (ERPs), local perspective views, unit rays, features, camera poses, and 3D structure without hiding coordinate, sampling, validity, or scale conventions. Use it as a geometry layer around existing planar models, or use its sphere-native image-processing and geometry algorithms when projection would be the wrong abstraction.

The 3.7 product is organized as six user-facing capability families. The families explain how capabilities compose; the granular contracts remain the unit of compatibility, testing, and future promotion.

panorai.graph is the new Experimental endpoint for materializing prepared pose, depth, semantic, association, and localization evidence as a replayable spatial-semantic graph. It does not run the perception chain or expose a GraphBuilder.from_images() shortcut. See the graph guide and API stability matrix.

Install

pip install panorai

PanorAi supports Python 3.11–3.14 and OpenCV >=4.9,<5. Install only the integrations you need:

pip install "panorai[torch]"      # Torch sampling and tensor workflows
pip install "panorai[deep-learning]"  # Experimental pretrained spherical FCN/CAM
pip install "panorai[deep-learning-depth]"  # Experimental spherical Metric3D CNN
pip install "panorai[pycolmap]"  # COLMAP database/reconstruction bridge
pip install "panorai[slam]"      # SLAM dependencies
pip install "panorai[pcd]"       # point-cloud I/O

Stable 3.x contracts center on panorai.geometry, modality-aware objects, sampling, blending, and the face-based feature core. Sphere-native image processing, spherical DoG, relative pose, dense stereo, native multiview, and SLAM are public but Experimental; their result models expose diagnostics so callers can reject unreliable outputs.

Choose a route

1. Projection

An ERP pixel is a sample of a ray, not a point on a flat image plane. PanorAi uses one explicit right-handed frame (+X right, +Y up, +Z front) and pixel-center convention across:

• ERP pixel ↔ longitude/latitude ↔ unit ray;
• sphere ↔ one gnomonic view with explicit center, FOV, roll, and resolution;
• ERP ↔ six-face cubemap or arbitrary-N overlapping face sets;
• nearest and bilinear sampling with seam wrapping and declared pole rules;
• overwrite, average, feather, Gaussian, and multiband blending;
• separate geometric support, source-data validity, and interpolation weight.

import numpy as np
from panorai.geometry import GnomonicProjector, GnomonicSpec

erp = np.zeros((512, 1024, 3), dtype=np.float32)
spec = GnomonicSpec(
    center_lat_deg=10,
    center_lon_deg=30,
    hfov_deg=90,
    vfov_deg=70,
    output_shape_hw=(256, 320),
)
projector = GnomonicProjector(spec, interpolation="bilinear")
view = projector.project(erp)
restored = projector.back_project(view, output_shape_hw=erp.shape[:2])

assert view.data.shape == (256, 320, 3)
assert restored.support_mask.shape == erp.shape[:2]

RGB, range, labels, and masks can share the same geometric grid while choosing the interpolation and invalid-data rule appropriate to each modality. Zero is data; it is never an implicit invalid marker.

Read: projection tutorial, custom projection pipeline, and the exact geometry v1 contract.

2. Image Processing

PanorAi exposes two intentionally different routes:

sphere-native: ERP ── spherical neighbourhood/filter ──► ERP

multiface:     ERP ── project ──► tangent faces ── planar operation ──►
               faces ── back-project + blend ──► ERP

The public APIs make that choice visible rather than implicit:

direct = spherical_gaussian_blur(erp, ksize=5, sigma=1.2, backend="auto")

multiface = EquirectangularImage(erp).process_views(
    lambda view: cv2.GaussianBlur(view, (5, 5), 1.2),
    layout="cube", size=256, fov=95, blend="gaussian",
)

On the bundled CC0 panorama resized to 512×1024, a 5×5 Gaussian (sigma=1.2) took 7.99 ms after warm-up through the direct native route and 104.25 ms through six 256² cube faces, OpenCV filtering, back-projection, and Gaussian blending. In a fixed 37° yaw-equivariance probe, MAE was 0.00113 and 0.00328, respectively. These routes are not bit-identical operators; see the full protocol and limitations and reproduce it with scripts/benchmark_image_processing_routes.py.

Histogram equalization and CLAHE

spherical_equalize_histogram is a first-party global equalizer. Its optional solid-angle weighting avoids over-counting the densely sampled polar rows of an ERP. PanorAi does not currently claim a sphere-native CLAHE implementation. You can apply OpenCV CLAHE to projected views through process_views, but the result depends on the face layout, FOV, overlap, and blender and is not equivalent to geodesic contextual equalization. A future spherical CLAHE needs seam- and pole-safe contextual regions, reproducible interpolation between regions, and evaluation on downstream feature/pose quality—not only visual contrast.

Read: image-processing tutorial and projection workflow how-to.

Pretrained spherical FCN/CAM and proxy-guided segmentation

The opt-in panorai.experimental.deep_learning namespace converts AlexNet, VGG16, and ResNet18 classification heads to fully convolutional form, replaces their spatial convolutions and pools with differentiable spherical operators, and reuses the exact Torchvision parameters without fine-tuning. First use automatically downloads the official DEFAULT checkpoints to the external Torch cache and verifies their URL hash prefix; PanorAi never packages those weights. An explicit prefetch command is also available:

python benchmarks/spherical_fcn_cam/download_models.py

Fully convolutional does not mean one output per input pixel: the original network stride and classifier kernel still determine the dense lattice. Read the tutorial for native-resolution inference, map interpretation, automatic model acquisition, and the planned output-stride/tangent-oracle comparison.

For the high-capacity ImageNet control, the same Experimental namespace loads the external, checksum-verified InternImage-G checkpoint and converts its DCNv3 samplers to differentiable local-tangent spherical samplers. The SphericalProxySegmenter.from_pretrained(...) facade combines that direct-ERP evidence with one transient SAM 2.1 chart/embedding at a time and returns all three mask alternatives, consensus/envelope masks, spherical instance fusion, unknown regions, and a deterministic panoptic map. Install .[deep-learning-internimage,deep-learning-sam]; no model bytes are bundled. This is automatic proxy-guided segmentation, not semantic ground truth; see the segmentation tutorial.

The same Experimental namespace now provides a checksum-pinned, adapter-only Metric3D-v1 ConvNeXt-Tiny/Hourglass loader. It requires explicit acceptance of upstream terms before downloading external source and weights, ports all learned spatial layers with exact parameter identity, and performs resize-free 2:1 ERP inference as radial range. Because the checkpoint has no separate published model-card license, PanorAi never redistributes it. See the spherical monocular-depth tutorial.

3. Detection and Feature Extraction

PanorAi separates where a keypoint is detected from how its local appearance is described.

For relative pose, SphericalFeaturePipeline.for_relative_pose() remains the canonical validated default. Direct spherical DoG is an explicit opt-in while its public-dataset detection, matching, and pose evaluation is completed.

The spherical detector avoids duplicate detections created by overlapping full faces and carries position, unit bearing, angular scale, response, octave, level, and localization validity. It refines discrete DoG extrema in position and scale using the second-order Taylor/Hessian step, rejects low-contrast and edge-like responses, assigns one or more tangent orientations, and only then materializes descriptor patches.

The contracts are deliberately descriptor-neutral:

• panorai-tangent-patches/v1 declares bearing/scale, angular context, projection, resolution, mask, and orientation;
• panorai-tangent-opencv-descriptor/v2 adapts the same geometric patch contract to SIFT, ORB, or AKAZE without making the detector “a SIFT detector.”

First-party C++ kernels accelerate compatible spherical convolutions and fuse the 3D tangent-neighbour extrema search without materializing a large neighbour tensor. OpenCV's own C++ implementations still own descriptor semantics and BF/FLANN distance search. The NumPy paths remain references for parity tests; native acceleration does not change the spherical geometry.

Read: features and matching, spherical feature how-to, and feature API.

4. Two-View Geometry

PanorAi can estimate a relative pose directly from matched unit bearings—no COLMAP process is involved:

ERP pair → features → descriptor matches → bearing pairs
         → 5-point Essential hypotheses → spherical MSAC/refit
         → cheirality + degeneracy/quality gates → R₂₁, direction(t₂₁)

The estimator scores tangent-Sampson residuals on the sphere, evaluates the four Essential decompositions, selects translation orientation using cheirality/parallax evidence, optionally refines rotation and translation direction, and reports rejection reasons. A returned hypothesis is not automatically a trustworthy pose: consume quality_report.accepted.

Monocular two-view geometry recovers the direction of translation, not its metric magnitude. Metric scale needs an external premise such as a known baseline, calibrated rig, depth, or a justified scene constraint.

Read: two-view geometry and the public-dataset benchmark.

5. Stereo Dense Reconstruction

Given two aligned central ERPs and a known metric relative pose, the Experimental stereo route searches inverse radial range directly along spherical epipolar geometry. It uses angular appearance support, optional coarse-to-fine search, cost aggregation, sub-hypothesis refinement, and bidirectional consistency.

The result keeps radial range, confidence, validity, selected cost, and consistency diagnostics separate. The supplied R,t is never silently re-estimated. If t has arbitrary unit length, the reconstructed range has the same arbitrary scale.

Read: dense-stereo tutorial, algorithm derivation, and current evidence.

6. Multi-View Geometry

For three or more panoramas, PanorAi offers two routes:

The native mapper optimizes angular tangent residuals without reconstructing ERP pixels during bundle adjustment. Its map is arbitrary-scale unless a metric prior is supplied. The PyCOLMAP route is useful when COLMAP interoperability is more important than keeping spherical equations first-party; it is an alternative pipeline, not a hidden dependency of the native mapper.

For registered metric cameras, Experimental metric landmark BA refines supplied spherical landmarks with a baseline gauge and soft radial-range priors. It reports sparse landmark support explicitly and does not produce a dense depth map. Read: multiview reconstruction and public-dataset benchmark.

Other essential capabilities

The six sections above cover the offline core. Two additional areas should not be forgotten:

• Visual SLAM: incremental ERP or calibrated fisheye tracking, keyframes, local mapping, relocalization, and loop correction. Start with the SLAM tutorial.
• Contracts and reproducibility: coordinate frames, units, pixel centers, interpolation, invalid-data rules, resolved configuration, provenance, and quality gates are part of the result—not incidental metadata. Read the geometry contract, stability tiers, and benchmark policy.

Evidence and project boundaries

The benchmarks and performance page separates microbenchmarks from end-to-end accuracy, reports Matterport3D and Stanford2D3D independently, and distinguishes accepted-result precision from coverage. Dataset licenses apply; PanorAi does not redistribute those datasets.

PanorAi owns spherical coordinates, projection, masks, provenance, spherical image-processing kernels, bearing geometry, and its native estimators. OpenCV owns SIFT/ORB/AKAZE descriptors and standard matchers; optional PyCOLMAP/COLMAP owns its downstream reconstruction route. See the capability map for the complete sphere-native, projection-domain, and hybrid boundary.

Reference

• Documentation home
• Tutorials in workflow order
• API reference
• Changelog
• License

PanorAi's distributed source is MIT licensed. Optional upstream projects, models, and datasets retain their own terms.