mote_error 0.1.0

Decomposable evaluation metrics for document understanding pipelines — the MOTe error for layout parsing, and the Character Error Vector for page-level OCR

mote_error is the successor to cotescore. The layout metric has been reframed from the COTe score into the MOTe error; see Coming from the COTe score? below.

Overview

Document Layout Analysis (DLA) is the process of parsing a page into meaningful elements, typically using machine learning models. Traditional evaluation metrics such as IoU, F1, and mAP were designed for 3D-to-2D image projections (e.g. photographs) and can give misleading results for natively 2D printed documents.

Document understanding pipelines generally have two stages. The first stage parses the page into individual visual elements or regions, effectively focusing on answering where is the text?. The second stage is the Optical Character Recognition, where the text is transcribed in to machine readable format, answering what does the text say?. When the output text quality is poor conventional metrics can struggle to identify the root cause.

The mote_error library provides two decomposable metrics one for each stage of the pipeline.

For a conceptual introduction to these metrics there is a blog post on the COTe score (the original framing that MOTe reframes as an error), a blog post on the CEV, and two tutorials available to run for free on a lightning ai studio (written for cotescore).

Layout parsing — the MOTe error

• Structural Semantic Units (SSUs) — a relational labelling approach that shifts focus from the physical bounding boxes of regions to the semantic structure of the content.
• MOTe error — a decomposable error that breaks page-parsing quality into interpretable components:
  • Missing — ground-truth area that no prediction covers (the complement of Coverage)
  • Overlap — redundant predictions covering the same ground-truth area
  • Trespass — predictions that cross semantic boundaries into a different SSU
  • excess — a support metric for predictions that fall on background or white-space

$$\mathrm{MOTe} = \mathcal{M} + \mathcal{O} + \mathcal{T}$$

0 is a perfectly parsed page and 1 is a page with no predictions. Values above 1 are possible, and mean that Overlap plus Trespass outweigh the area the predictions cover. Excess is reported alongside but is not part of the sum.

MOTe is more informative than traditional metrics, reveals distinct model failure modes, and remains useful even when explicit SSU labels are unavailable. Because it works on per-instance masks it also applies to instance and panoptic segmentation (where Overlap is zero by construction).

Coming from the COTe score? MOTe is the same decomposition reframed as an error. With default weights MOTe = 1 − COTe and Missing = 1 − Coverage, so previously computed results convert directly.

Page-level OCR — the Character Error Vector

Character Error Rate assumes the text was parsed perfectly. When it was not, the reading order is wrong, the alignment CER depends on becomes meaningless, and the metric breaks down precisely when a diagnosis is most needed.

• Character Error Vector (CEV) — a bag-of-characters evaluator that needs no alignment, and decomposes page-level OCR error into parsing, OCR and interaction components.
• Two instantiations ship with the library: SpACER (Spatially Aware Character Error Rate), a count-based metric analogous to CER, and CDD, a character-distribution distance built on the Jensen–Shannon distance.

Because the components separate the stages, the CEV can say whether the transcription was let down by the parser or by the OCR engine. This library is the reference implementation for the CEV paper.

Installation

pip install mote_error

That is all you need to compute MOTe and CEV scores on predictions you already have. Running the layout models themselves requires extra dependencies — see Optional extras at the end.

Quick Start

The library ships with a bundled limerick case study that you can use to try it out immediately.

from mote_error import mote, load_limerick_example, extract_ssu_boxes
from mote_error.adapters import boxes_to_gt_ssu_map, boxes_to_pred_masks, compute_canvas

# Load the bundled example: ground-truth dict, document image, and example predictions
ground_truth, image, pred_boxes = load_limerick_example()

h, w = image.shape[:2]
canvas_w, canvas_h = compute_canvas(w, h)

# Build tagged SSU-level GT boxes and rasterize to a 2-D SSU id map
gt_boxes = extract_ssu_boxes(ground_truth)
gt_ssu_map = boxes_to_gt_ssu_map(gt_boxes, w, h, canvas_w, canvas_h)
preds = boxes_to_pred_masks(pred_boxes, w, h, canvas_w, canvas_h)

# Compute the MOTe error; the result is a named tuple
result = mote(gt_ssu_map, preds)
print(f"MOTe={result.mote:.3f}  M={result.missing:.3f}  O={result.overlap:.3f}  "
      f"T={result.trespass:.3f}  e={result.excess:.3f}")

---

MOTe pixel-state visualisation: each pixel in the document is classified as Covered (green), Overlap (yellow), Trespass (red), Trespass AND Overlap (purple), Missing (grey), or Excess (blue). Produced by notebooks/limerick_analysis.py.

---

See notebooks/limerick_analysis.py for a full worked example comparing MOTe against F1 and mean IoU at different granularity levels.

Core Metrics — MOTe

Function	Description
mote(gt_ssu_map, preds)	Returns a MOTeResult named tuple (mote, missing, overlap, trespass, excess) [0, ∞)
missing(gt_ssu_map, preds)	Fraction of GT area no prediction covers [0, 1]
coverage(gt_ssu_map, preds)	Fraction of GT area covered, 1 − missing [0, 1]
overlap(gt_ssu_map, preds)	Redundant prediction area within GT [0, ∞)
trespass(gt_ssu_map, preds)	GT area covered by predictions assigned to a different SSU [0, ∞)
excess(gt_ssu_map, preds)	Predicted area falling on background, over total background [0, 1]
mote_class(gt_ssu_map, ssu_to_class, preds)	Per-class interaction matrices and per-GT-class Missing (ClassMOTeResult)

All functions are importable directly from mote_error:

from mote_error import mote, missing, coverage, overlap, trespass, excess, mote_class

mote accepts weight_missing, weight_overlap and weight_trespass to reweight the sum; equal weights are the default because each component measures the same spatial area. When ground truth is available as boxes, pass a GTBoxes and an (M, 4) array of predicted boxes to mote to use the analytic fast path, which never rasterises. A page with no SSUs has Missing, Overlap and Trespass of 0.

The conventional instance-matching baselines MOTe is compared against in the paper — iou, mean_iou and f1 — remain available from mote_error.layout if you want to reproduce those comparisons.

For an alternative visual overview of what the different elements of the MOTe error mean please see notebooks/metrics_exploration.py

Core Metrics — the CEV

The CEV compares four bags of characters. Which ones you can build determines which components you get:

Symbol	Key	What it is
$Q$	gt	all ground-truth characters on the page
$R$	parsing	ground-truth characters falling inside predicted regions
$S^*$	ocr	OCR run on ground-truth regions
$S$	total	OCR run on predicted regions

From those come the four components — d_pars (parsing alone), d_ocr (OCR given perfect regions), d_int (their interaction), and d_total (the whole pipeline).

Function	Description
spacer(reference, prediction)	Macro SpACER between two token Counters [0, ∞): $(\hat{E} + D + I) / 2C$ with net deletions $D$ and insertions $I$ at page level
spacer_micro(ref_boxes, pred_boxes)	Per-box SpACER: $D$ and $I$ accumulated per box, so a deletion in one box is not cancelled by an insertion in another
spacer_decomp(named_dict)	Four-way SpACER decomposition (macro and micro) from page or per-box text
cdd_decomp(named_dict)	Four-way CDD decomposition; any (Counter, Counter) -> float metric can be substituted
spacer_decomp_spatial(gt_chars, pred_regions, ...)	As above, building $R$ by joining character positions to predicted regions
cdd_decomp_spatial(gt_chars, pred_regions, ...)	Spatial CDD equivalent
jsd_distance(p, q)	Square root of the Jensen–Shannon divergence [0, 1] — the default CDD metric
text_to_counter(text, mode)	Build a character (or word, for SpAWER) frequency Counter

from mote_error import spacer_decomp, cdd_decomp

bags = {"gt": gt_text, "ocr": ocr_of_gt_regions, "total": ocr_of_predicted_regions}
sp = spacer_decomp(bags)
cd = cdd_decomp(bags)
print(sp.d_ocr_macro, sp.d_total_macro)

Components whose inputs are absent come back as None rather than failing. Building $R$ needs character-level positions, so a dataset with only region-level text yields d_ocr and d_total while d_pars and d_int are None. That pair is still enough to triage: combining MOTe with the ratio d_ocr / d_total predicts the dominant error source with an F1 of 0.91, using only values that are cheap to obtain. The *_spatial functions recover the full vector when character boxes are available.

Visualisation

from mote_error import compute_mote_masks, visualize_mote_states
import matplotlib.pyplot as plt

masks = compute_mote_masks(gt_ssu_map, preds)

fig, ax = plt.subplots()
visualize_mote_states(image, masks, ax=ax)
plt.show()

Datasets

The library includes loaders for the datasets used in the two papers, all sharing one annotation format:

from mote_error.dataset import (
    NCSEDataset, DocLayNetDataset, HNLA2013Dataset,
    SpiritualistDataset, HierTextDataset, DocBankDataset,
)

# Bundled toy example (no download required)
from mote_error import load_limerick_example
chars_df, image, pred_boxes = load_limerick_example()

Examples

The primary interactive example is the Marimo notebook at notebooks/limerick_analysis.py. It demonstrates:

• Visualising SSU, line, and character-level bounding boxes
• The granularity-mismatch problem and how MOTe handles it
• Side-by-side comparison of MOTe, F1, and mean IoU
• MOTe pixel-state visualisation

To run the notebook (requires Marimo):

pip install marimo
marimo edit notebooks/limerick_analysis.py

End-to-end walkthrough on a real dataset

There are two tutorial notebooks, covering MOTe and CEV that provide hands on experience of the metrics. The tutorials use the marimo library, the NCSEv2 test set and require a GPU. A 1-click reproducible lightning.ai studio is available to run for free here

1. notebooks/Tutorial_1_the_MOTe.py — scores two layout models (DocLayout-YOLO and Docling Heron) with MOTe, compares it against F1 and mean IoU, and writes a folder of MOTe pixel-state diagnostics per model.
2. notebooks/Tutorial_2_the_CEV.py — runs EasyOCR and Tesseract over both models' regions and scores them with the CEV (SpACER and CDD), then uses MOTe together with d_ocr / d_total to attribute each page's error to parsing or to OCR.

Both notebooks take the images folder and the ground-truth CSV as plain string paths at the top, so they can be pointed at your own dataset. The layout models need a GPU and the doclayout-yolo, easyocr and pytesseract packages (plus the tesseract binary).

marimo edit notebooks/Tutorial_1_the_MOTe.py
marimo edit notebooks/Tutorial_2_the_CEV.py

Optional extras — running the benchmark models

Only needed if you want to run the document layout models yourself, for example to reproduce the papers' benchmarks. Scoring existing predictions needs none of this.

The benchmarking extras include either PyTorch or PaddlePaddle — do not install both in the same environment. These frameworks require different CUDA versions and will conflict:

Extra	Use case	GPU framework
mote_error[benchmarks]	Torch-based DLA models (DocLayout-YOLO, Heron)	PyTorch
mote_error[paddle-benchmark]	PaddleOCR / PP-DocLayout	PaddlePaddle

# Torch-based benchmarks
pip install "mote_error[benchmarks]"

# PaddlePaddle benchmarks (install PaddlePaddle first, separately)
# See: https://www.paddlepaddle.org.cn/install/quick
pip install "mote_error[paddle-benchmark]"

Note: PaddlePaddle must be installed before mote_error[paddle-benchmark] — it is not listed as a pip dependency because the correct wheel depends on your CUDA version. Follow the PaddlePaddle installation guide to get the right version.

Questions and Bug Reports

If you have questions, find a bug, or want to request a feature, please open an issue on GitHub.

Citation

If you use mote_error in your research, please cite the paper for the metric you used.

MOTe — layout parsing. MOTe reframes the COTe score from this paper as an error (MOTe = 1 − COTe); until the MOTe paper is out, please cite:

Bourne, Jonathan, Mwiza Simbeye, and Ishtar Govia. “The COTe Score: A Decomposable Framework for Evaluating Document Layout Analysis Models.” arXiv:2603.12718. Preprint, arXiv, March 13, 2026. https://doi.org/10.48550/arXiv.2603.12718.

@misc{bourne2026cote,
  title         = {The {COTe} Score: A Decomposable Framework for Evaluating {Document Layout Analysis} Models},
  author        = {Bourne, Jonathan and Simbeye, Mwiza and Govia, Ishtar},
  year          = {2026},
  month         = mar,
  publisher     = {arXiv},
  doi           = {10.48550/arXiv.2603.12718},
}

CEV / SpACER — page-level OCR:

Bourne, Jonathan, Mwiza Simbeye, and Joseph Nockels. "The Character Error Vector: Decomposable Errors for Page-Level OCR Evaluation." arXiv:2604.06160. Preprint, arXiv, April 2026. https://doi.org/10.48550/arXiv.2604.06160.

@misc{bourne2026charactererrorvector,
  title         = {The {Character Error Vector}: Decomposable errors for page-level {OCR} evaluation},
  author        = {Bourne, Jonathan and Simbeye, Mwiza and Nockels, Joseph},
  year          = {2026},
  month         = apr,
  eprint        = {2604.06160},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  doi           = {10.48550/arXiv.2604.06160},
}