eval3r 0.2.1

Handy toolkit for saving, evaluating, and visualizing 3D reconstruction predictions.

eval3r

eval3r is a Python toolkit for saving, benchmarking, and visualizing 3D reconstruction results with explicit assumptions and reproducible workflows.

Motivation

Many reconstruction pipelines fail at the "last mile": prediction artifacts are hard to compare, evaluation settings are underspecified, and reproducing numbers from papers can be difficult.

eval3r addresses this with:

• a practical CLI (e3r) for evaluation tasks,
• strongly typed I/O for predictions and metrics,
• optional rendering and debug visualization,
• and end-to-end benchmark utilities.

Installation

pip install eval3r

Command-line overview

After installation, eval3r exposes the e3r command.

e3r --help
e3r -v        # print version

Primary commands:

• e3r metric ... — compute 2d/3d metrics for a prediction/GT pair
• e3r benchmark ... — evaluate many scenes and aggregate split-level metrics
• e3r render ... — render geometry and comparison outputs
• e3r mask ... — generate/inspect occlusion masks
• e3r validate <prediction_dir> — validate a prediction directory against manifest rules
• e3r inspect <prediction_dir> — inspect a prediction directory summary

Quickstart: single-scene geometry evaluation

Assume:

pred.ply   # prediction
gt.ply     # ground truth

1) Compute metrics

e3r metric geometry pred.ply --gt gt.ply

Typical outputs include Chamfer distance, accuracy/completeness, and thresholded precision/recall/F-score.

2) Always inspect overlap

e3r metric geometry pred.ply --gt gt.ply --debug-plot

If overlap is poor due to coordinate mismatch, metrics are not meaningful yet.

3) Select alignment intentionally

Use rigid alignment when scale is trustworthy:

e3r metric geometry pred.ply --gt gt.ply --align icp_se3 --debug-plot

Use similarity alignment when scale may drift (common in monocular systems):

e3r metric geometry pred.ply --gt gt.ply --align icp_sim3 --debug-plot

Use trajectory-driven alignment when poses are available:

e3r metric geometry pred.ply --gt gt.ply \
  --align traj_sim3 \
  --pred-poses pred_trajectory.txt \
  --gt-poses gt_trajectory.txt \
  --debug-plot

Quickstart: benchmark a dataset split

Expected prediction layout example:

preds_root/
  scene0000_00/
    mesh.ply
  scene0001_00/
    mesh.ply

Run benchmark:

e3r benchmark scannet \
  --pred-root preds_root \
  --gt-root /path/to/scannet \
  --split split.txt \
  --align sim3 \
  --workers 8 \
  --out results.json \
  --csv results.csv

Useful options:

• --samples 200000 points sampled per scene
• --thresholds 0.05 threshold(s) for F-score-like metrics
• --debug-plot to save visual diagnostics
• --pred-pattern to map custom prediction filenames

Trajectory-based benchmark alignment example:

e3r benchmark scannet \
  --pred-root preds_root \
  --gt-root /path/to/scannet \
  --split split.txt \
  --aligner traj_sim3 \
  --pred-pose-dir pred_poses \
  --pred-pose-file "{scene_id}.txt" \
  --pred-pose-convention T_wc

Prediction format and Python API

eval3r includes a manifest-driven prediction format so files are portable and self-described.

Minimal writer example:

import eval3r as e3r

with e3r.PredictionWriter(
    "outputs/scannet/scene0799_00",
    scene_id="scene0799_00",
    dataset="scannet",
    method="my_method",
    unit="m",
    coordinate_system="opengl",
    pose_convention="T_wc",
) as pred:
    pred.save_point_cloud(points, colors=colors)
    pred.save_mesh(vertices, faces)
    pred.save_poses(poses, timestamps=timestamps)
    pred.save_metadata({"checkpoint": "ckpt.pth"})

When key metadata is missing, the writer records explicit placeholders instead of guessing, enabling downstream validation and auditability.

Supported workflows

• Scene-level metric inspection during model development.
• Dataset-level benchmark reporting for ablations and papers.
• Reproducible artifact exchange between teams using manifest-backed prediction directories.
• Alignment experiments (icp_se3, icp_sim3, trajectory-driven variants) with debug plots.

Documentation

• Docs site: https://eval3r.readthedocs.io/
• Repository: https://github.com/xingruiy/eval3r

Development

pip install -e .[dev]
pytest

License and dataset disclaimer

eval3r is released under the MIT License.

This project does not redistribute third-party datasets such as ScanNet, Replica, DTU, ETH3D, Tanks & Temples, or TUM RGB-D. You must obtain and use those datasets under their original licenses.