euler

euler-eval

Score depth, RGB, ray and 3D point-map predictions against the same ground truth — from one JSON config.

Comparing predictions to ground truth is rarely one line of code. The files have to be paired, the ground truth may be radial where the prediction is planar, the model may only be correct up to scale, and every modality wants a different metric set.

euler-eval takes a JSON config naming the ground truth and one or more prediction datasets, and writes an eval.json per modality: files paired by ID, decoded from dataset metadata rather than convention, aligned when the prediction is not metric, and scored with the metric set that modality deserves.

flowchart LR
    C["ds-crawler<br/><i>indexes files</i>"] --> H["dataset-head.json<br/><i>euler-dataset-contract</i>"]
    H --> L["euler-loading<br/><i>pairs GT + prediction by ID</i>"]
    L --> E["euler-eval<br/><b>scores the pair</b>"]
    E --> J["eval.json<br/><i>per modality, per dataset</i>"]
    E -.->|optional| T["euler-train<br/><i>experiment logging</i>"]

It is the consuming end of that pipeline and does none of the earlier steps itself: a path in the config is a euler-loading path, and how to decode it — loader, units, radial or planar depth, point-cloud column layout — is read from the dataset's own index. Metric keys are built with euler-metric-naming, so every result is self-describing to downstream tools.

Install

pip install euler-eval                  # core
pip install "euler-eval[fid]"           # + clean-fid RGB FID backend
pip install "euler-eval[logging]"       # + euler-train experiment logging

Python 3.9 or newer. CUDA is used automatically when available, and every metric also runs on CPU.

Quick start

Describe the ground truth and what to score against it:

{
  "gt": {
    "rgb":   { "path": "/data/gt/rgb" },
    "depth": { "path": "/data/gt/depth" }
  },
  "datasets": [
    {
      "name": "model_a",
      "rgb":   { "path": "/data/model_a/rgb" },
      "depth": { "path": "/data/model_a/depth" }
    }
  ]
}

euler-eval config.json --batch-size 32

Ground truth and prediction files are matched by ID rather than by directory order, so an incomplete prediction set scores the frames it does have instead of silently comparing the wrong pairs. Results are written next to each prediction modality:

{
  "depth": {
    "eval": {
      "metric": {
        "standard": {
          "image_mean": { "absrel": 0.081, "rmse": 3.42, "delta1": 0.93 }
        }
      }
    }
  }
}

Every path accepts euler-loading's inline selectors, so splits and archives need no extra plumbing — "/data/muses.zip:test#scope=rgb" reads the test split straight out of the archive. Full schema: Configuration.

What it evaluates

Modality	Config key	Scored against	Headline metrics
Depth	`depth`, `relative_depth`, `affine_depth`	dense GT depth	`absrel`…`delta3`, PSNR/SSIM/LPIPS/FID/KID, surface-normal consistency, depth-edge F1
Sparse depth	`depth` + `gt.sparse_depth`	a LiDAR-style point cloud, projected into the prediction plane	pointwise depth metrics at projected points
RGB	`rgb`	GT RGB	PSNR, SSIM, LPIPS, FID, SCE, edge F1, tail errors, HF energy ratio, depth-binned photometric error; optional NIQE/FADE dehazing metrics
Rays	`rays`	GT ray direction map	ρ_A (angular-accuracy AUC), angular error, threshold percentages
Points-3D	`points_3d`	GT point map, GT depth unprojected on the fly, or a sparse point cloud	3D EPE/RMSE/δ, radial-vs-lateral decomposition, true-3D normals and edge F1, Chamfer / F-score

A run evaluates whichever modalities are configured on both sides; a depth-only prediction is a perfectly ordinary run. The full inventory, with the key each metric is written under, is in Metrics.

What you get


ID-based pairing	GT and prediction frames are matched by file ID and hierarchy, not by filename luck. Calibration and pose files are matched by tree position and shared.
Metadata-driven decoding	Radial vs planar depth, value ranges, point-cloud column layout and loader choice all come from the dataset index — no per-dataset flags.
Honest spaces	Relative models are scored `native` and `metric` (after scale/shift or a similarity gauge), so a scale failure never reads as a geometry failure.
Benchmark depth bins	`--benchmark-depth-range MIN MAX` adds near/mid/far bins in square-root depth space, additive to the regular metrics.
Sky masking	`--mask-sky` drops sky pixels using GT segmentation — from the metrics and from the alignment fit.
Per-file + aggregate	Dataset-level numbers plus per-image metrics in dataset-hierarchy order, in the same file.
Sanity checks	Results are validated against configurable thresholds; implausible ranges, degenerate inputs and scale mismatches are reported instead of quietly scored.
Structured metric names	Each result carries a `metricSet` envelope declaring its namespace, axes, units and metric directions.
Composable domains	`--domain dehazing` supplements the stable core set with prediction-only NIQE and FADE; the registry is ready for additional domains.
euler-train logging	Optional: register each evaluation in an experiment run with full package provenance.

In-training validation

The same depth-metric semantics are available for in-memory predictions, so a training loop's validation numbers match what the CLI reports later — against dense or sparse ground truth, without writing predictions to disk:

from euler_eval import DepthValidationAggregator, evaluate_sparse_depth_sample

aggregator = DepthValidationAggregator()
for sample in val_dataset:
    result = evaluate_sparse_depth_sample(
        model(sample["rgb"]),        # (H, W) metres
        sample["sparse_depth"],      # (N, C>=3) lidar points
        intrinsics,
        lidar_to_camera,
        alignment="none",            # "affine" for relative depth
    )
    aggregator.update(result)

aggregator.summary()["standard"]["image_mean"]["absrel"]

Multi-process runs reduce sufficient statistics rather than averages — see In-training validation.

Documentation

Guide	Covers
Configuration	The config file, `gt` and `datasets`, paths and selectors, sparse-depth and points-3d ground truth, euler-train logging
CLI reference	Every flag, worked examples, device selection, offline cache warmup
Spaces & alignment	`native` vs `metric`, depth affine fitting, points-3d gauge alignment, benchmark depth bins
Metrics	The full metric inventory per modality
Results & output	Anatomy of `eval.json`, per-file metrics, the sanity-check report
In-training validation	The programmatic API for scoring in-memory predictions

Start with the concepts page for how the pieces fit together.

Development

git clone https://github.com/d-rothen/euler-eval.git
cd euler-eval
uv sync --extra dev     # or: pip install -e ".[dev]"
uv run pytest

See CONTRIBUTING.md for the test layout and release process.

License

MIT © Daniel Rothenpieler

Metadata

Release files for euler-eval 2.28.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for euler-eval 2.28.0
File	Size	Uploaded
euler_eval-2.28.0.tar.gz	199.6 kB	Sep 13, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for euler-eval 2.28.0
File	Interpreter	ABI	Platform	Reset
euler_eval-2.28.0-py3-none-any.whl 164.1 kB Sep 13, 2026	Python 3	none	any	Details

Total release size: 363.7 kB

Release files / euler_eval-2.28.0.tar.gz

Download URL	euler_eval-2.28.0.tar.gz
Size	199.6 kB
Tags	Source
SHA-256 checksum How to use checksums	`e28440c9d6d2396c654939e38c3086e98c68fcf0b0a5b3eba71d2e0d9c3fe51f`
BLAKE2b-256 checksum How to use checksums	`b518ad37beba8f9e55c13c4207ad98b02b0a0a6e98dd27999deb3d70e9712c85`
Upload date	Sep 13, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`uv/0.12.13 {"installer":{"name":"uv","version":"0.12.13","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}`

Release files / euler_eval-2.28.0-py3-none-any.whl

Download URL	euler_eval-2.28.0-py3-none-any.whl
Size	164.1 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`a3d2a7ab0ddbc6080b6013b67fd33f072a6e7d0936472f8b62cc334101cae5e6`
BLAKE2b-256 checksum How to use checksums	`e995b54b496ded09d8f8be8a530782959b9d51abc131e25008861c70075124f5`
Upload date	Sep 13, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`uv/0.12.13 {"installer":{"name":"uv","version":"0.12.13","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}`