aruco-estimator 1.1.4

Aruco Scale Factor Estimation

Aruco Scale Factor Estimation for COLMAP (Work in Progress!)

About

This project aims to automatically compute the correct scale of a point cloud generated with COLMAP. By adding an aruco marker with into the scene it is possible to overcome the scale ambiguity from monocular SfM and SLAM. In the literature it is commonly suggested to manually measure a known distance of a calibration object in the 3D reconstruction and then scale the reconstruction accordingly or make photos from two known positions[1]. Since there is unfortunately no automated approach to calculate an accurate scaling factor, this project aims to be attached as a post processing step to the reconstruction process.

Setup

Before taking the images for the reconstruction, an aruco marker must be printed out. Care should be taken to ensure that the size of the marker fits the scene to be recognizable in the images. Additionally the marker is placed on a planar surface and does not have curvature (otherwise, the marker will be recognized incorrectly or not at all). The aruco marker can either be generated with ArUco-markers-with-OpenCV-and-Python or generated easily on this website. When taking the images, make sure that the aruco marker is visible in at least two images; otherwise, no solution exists. It is advisable to have at least five images with the aruco marker for an accurate solution (two pictures should already be enough for a sufficient approximation. However, blurry images or aruco markers that are too small could falsify the result). After the images have been acquired, the reconstructions process can be carried out using COLMAP. For the algorithm to work it only needs the result of the Structure from Motion part (extrinsic parameters). The MVS part for the dense reconstruction is used to visually verify the result.

Installation

PyPi

This repository is tested on Python 3.6+ and can be installed from PyPi

pip install aruco-estimator

Usage

Dataset

A simple dataset of an aruco marker on a door is provided. It is automatically downloaded by using

dataset = download.Dataset()
dataset.download_door_dataset()

API

from aruco_estimator.aruco_scale_factor import ArucoScaleFactor
from aruco_estimator import download
import os
import open3d as o3d

# Download example dataset. Door dataset is roughly 200 MB
dataset = download.Dataset()
dataset.download_door_dataset()

# Init & run pose estimation of corners in 3D & estimate mean L2 distance between the four aruco corners
aruco_scale_factor = ArucoScaleFactor(project_path=dataset.dataset_path)
aruco_distance = aruco_scale_factor.run()
print('Size of the unscaled aruco markers: ', aruco_distance)

# Calculate scaling factor, apply it to the scene and save scaled point cloud
dense, scale_factor = aruco_scale_factor.apply(true_scale=dataset.scale)  # scale in cm
print('Point cloud and poses are scaled by: ', scale_factor)
print('Size of the scaled (true to scale) aruco markers in meters: ', aruco_distance * scale_factor)

# Visualization of the scene and rays BEFORE scaling. This might be necessary for debugging
aruco_scale_factor.visualize_estimation(frustum_scale=0.2)
o3d.io.write_point_cloud(os.path.join(dataset.colmap_project, 'scaled.ply'), dense)
aruco_scale_factor.write_data()

Source

Get Repo

git clone https://github.com/meyerls/aruco-estimator.git
cd aruco-estimator
conda env create -f environment.yml
conda activate aruco_estimator
pip install -r requirements.txt

Usage of Command Line

usage: scale_estimator.py [-h] [--colmap_project COLMAP_PROJECT] [--dense_model DENSE_MODEL] [--aruco_size ARUCO_SIZE] [--visualize] [--point_size POINT_SIZE] [--frustum_size FRUSTUM_SIZE] [--test_data]

Estimate scale factor for COLMAP projects with aruco markers.

optional arguments:
  -h, --help                             show this help message and exit
  --colmap_project COLMAP_PROJECT        Path to COLMAP project
  --dense_model DENSE_MODEL              name to the dense model
  --aruco_size ARUCO_SIZE                Size of the aruco marker in cm.
  --visualize                            Flag to enable visualization
  --point_size POINT_SIZE                Point size of the visualized dense point cloud. Depending on the number of points in the model. Between 0.001 and 2
  --frustum_size FRUSTUM_SIZE            Size of the visualized camera frustums. Between 0 (small) and 1 (large)
  --test_data                            Download and try out test data

To test the code on your local machine try the example project by using:

python scale_estimator.py --test_data

Visualization

The visualization is suitable for displaying the scene, the camera frustums, and the casted rays. It is usefull to check whether the corners of the aruco marker are detected and positioned correctly. If the aruco markers are not detected correctly the aruco settings must be changed according to the scene to be more robust.

Limitation/Improvements

• Make package for PyPi
• Upload to PyPi during CI
• Make dataset available for download
• Put aruco marker detection in threads
• Scale poses of camera/extrinsics.
• Up to know only SIMPLE_RADIAL and PINHOLE camera models are supported. Extend all models
• Install CLI Tool vi PyPi
• Up to now only one aruco marker per scene can be detected. Multiple aruco marker could improve the scale estimation
• Different aruco marker settings should be investigated for different scenarios to make it either more robust to false detections
• Geo referencing of aruco markers with earth coordinate system using GPS or RTK
• Alternatives to aruco marker should be investigated.
• Are the corners from the aruco marker returned identical regarding the orientation in the image?

Acknowledgement

The Code to read out the binary COLMAP data is partly borrowed from the repo COLMAP Utility Scripts by uzh-rpg.

Trouble Shooting

• In some cases cv2 does not detect the aruco marker module. Reinstalling opencv-python and opencv-python-python might help Source

Resources

[1] Lourakis, M., & Zabulis, X. (n.d.). LNCS 8047 - Accurate Scale Factor Estimation in 3D Reconstruction. 2013

[2] Traa, J., Least-Squares Intersection of Lines. 2013