ritn3d-floorplan-eval 0.1.0

Evaluation toolkit for floor plan detection models — IoU on walls, F1 on doors/windows, room matching, standard report formats. By the Ritn3D team.

floorplan-eval

A standardized evaluation toolkit for floor plan detection models — IoU on walls, precision / recall / F1 on doors and windows, polygon overlap on rooms. Schema-agnostic: works on any model's predictions as long as they're expressed in the simple JSON schema documented below.

If you're working on floor plan recognition — whether that's a classical CV pipeline, a U-Net wall-segmentation model, a transformer- based room-detection model, or anything else — you've probably noticed there's no widely-agreed-upon evaluation harness in the wild. Most research papers ship one-off scripts. Most production teams reinvent the same metrics. This library is the same five-file harness, written once, that we hope becomes the shared baseline.

Install

pip install floorplan-eval

CLI

floorplan-eval evaluate \
    --pred  my_model_prediction.json \
    --truth ground_truth.json

Output (markdown):

# Floor plan detection — evaluation report

## Walls
- Mean IoU: 0.847
- Matched: 12
- Unmatched: 1

## Doors and windows
| Type | Precision | Recall | F1 | TP | FP | FN |
|---|---|---|---|---|---|---|
| door | 0.800 | 1.000 | 0.889 | 4 | 1 | 0 |
| window | 1.000 | 0.857 | 0.923 | 6 | 0 | 1 |

## Rooms
- Mean IoU: 0.788
- Matched: 4
- Unmatched: 0

For machine-readable output, add --format json.

JSON schema

A floor plan is a small JSON document with three lists. Every model can adopt this with a thin adapter rather than the library trying to be clever about model output formats.

{
  "image_size": [800, 600],            // optional, [width, height] in px
  "walls": [
    {"start": [50, 50], "end": [750, 50], "thickness": 4}
    // thickness defaults to 1 if omitted; used as the buffer width for IoU
  ],
  "doors_windows": [
    {"type": "door",   "position": [400, 300], "width": 30},
    {"type": "window", "position": [200, 50],  "width": 50}
    // width defaults to 0 (point-only landmark)
  ],
  "rooms": [
    {"label": "living_room",
     "polygon": [[50, 50], [400, 50], [400, 550], [50, 550]]}
    // label is optional
  ]
}

A complete sample lives in examples/sample_floorplan.json.

Library

from floorplan_eval import (
    load_floorplan,
    walls_iou,
    doors_windows_f1,
    rooms_match,
    format_report,
)

pred = load_floorplan("pred.json")
truth = load_floorplan("truth.json")

results = {
    "walls": walls_iou(pred, truth),
    "doors_windows": doors_windows_f1(pred, truth, match_radius=20.0),
    "rooms": rooms_match(pred, truth),
}

print(format_report(results, output_format="markdown"))

Metrics — what's measured and how

Walls (walls_iou)

Each wall is a line segment plus a thickness. The library buffers each segment to a rectangular polygon at half-thickness on each side, then computes IoU between matched predicted and ground-truth polygons. Matching is greedy by centroid distance: each predicted wall picks its nearest unused ground-truth wall as a candidate, IoU is computed, and the highest-IoU pairing wins. Unmatched ground-truth walls contribute 0 to the average — so a model that predicts only half the walls is penalized.

Doors and windows (doors_windows_f1)

Reported as precision / recall / F1 separately for doors and windows. A predicted opening counts as a true positive if a ground-truth opening of the same type sits within match_radius pixels (default 20). Doors and windows are evaluated separately so the model gets credit for type discrimination, not just position.

Rooms (rooms_match)

Rooms are closed polygons. The library computes IoU between every predicted-vs-ground-truth pair, then greedily 1:1 matches the highest IoUs. Unmatched ground-truth rooms contribute 0. Room labels (kitchen, bedroom, etc.) are stored on the schema but not currently used for matching — purely geometric. Label-aware matching is on the roadmap.

What's in scope

Feature	Status
Wall IoU metric	✓
Door / window F1	✓
Room IoU + greedy matching	✓
JSON Floorplan schema	✓
Markdown / JSON report formats	✓
CLI	✓

What's out of scope (for now)

• Per-class room labeling metric (e.g. confusion matrix kitchen vs bedroom). Roadmap.
• Multi-floor evaluation. Treat each floor as a separate floor plan.
• Free-form annotation formats (COCO, LabelMe). Convert to this schema first.
• Model inference. This is a metrics library only — bring your own predictions.

Compatibility

• Python 3.9+
• Tested on Linux, macOS, Windows
• Pure Python, depends on shapely and numpy only

Why this exists

Floor plan recognition has been an active research area for over a decade (RPLAN, CubiCasa5K, R2V, DeepFloorplan, the parametric design papers from late 2010s, more recent transformer-based work). Yet every paper ships its own evaluation script, and every production team writes a slightly different one. The community has no shared baseline, which makes comparing two approaches across two papers genuinely difficult.

This library is a first cut at filling that gap. The schema is intentionally minimal — five concepts, four fields — so that adopting it requires only a thin adapter on whatever output format your model produces. We hope it gets contributed to and improved.

License

MIT — see LICENSE.

Maintained by

Ritn3D — a floor-plan-to-3D-model tool. We needed a way to measure detection accuracy across model iterations and noticed nobody else had one we could pip-install. So we wrote this. Contributions, schema improvements, and citations welcome.