Belegant

BelegBench

🇨🇭 CH · 🇩🇪 DE · 🇦🇹 AT — Belege in DE / FR / IT

BelegBench ist der Benchmark hinter Belegant — der On-Prem Beleg-AI für den DACH-Raum. Von den Machern von German-OCR.

Benchmark für die Extraktion von Rechnungen, Gutschriften und Kleinbetragsbelegen aus der Schweiz, Deutschland und Österreich — 100 % synthetisch, mit vollständiger Ground Truth, prüfsummen-korrekten Kennungen (IBAN, QR-Referenz, RF, CHE-UID, USt-IdNr., ATU) und echten, scannbaren QR-Codes (Swiss QR bzw. EPC-QR/GiroCode).

Datensatz: huggingface.co/datasets/keyvan-ai/belegbench — Configs ch, de, at (v2, 3 × 3 334 = 10 002 Belege) und ch-v1 (300 Belege, v1-Referenz).

Warum

Bisher gab es kein öffentliches Benchmark für Belege mit den Spezifika aller drei Länder — QR-IBAN und QR-Referenz (CH), § 14 UStG-Pflichtangaben, Steuernummer-Landesformate und § 13b (DE), § 11 UStG, UID-Pflicht über 10 000 € und 13 %-Satz (AT) — und vor allem keines, das misst, was für Treuhänder und Steuerberater zählt: die Halluzinationsrate. Die Ground Truth enthält nur, was auf dem Beleg gedruckt ist; jedes „ergänzte" Feld ist eine Halluzination.

Benchmark laufen lassen

pip install belegbench
# HF-Datensatz, Config de, 300 Belege, Belegant-Pipeline (Schema + QR-Oracle + Verifier)
python -m belegbench.run_eval --model belegant-4b --base-url http://localhost:11434/v1 --config de --n 300 --mode verified
# rohes Basismodell, Config ch
python -m belegbench.run_eval --model qwen3.5:4b --config ch --n 300 --mode raw

Modi: raw (Prompt + freies JSON) · schema (Constrained Decoding mit striktem JSON-Schema) · verified (Schema + QR-Oracle-Merge + deterministischer Verifier = Belegant-Pipeline).

Metriken (belegbench/metrics.py): Field-Level Exact Match, Halluzinationsrate, Verify-Catch-Rate, Parse-Fehler, Latenz — gesamt und je Feld. Ergebnisse landen als JSON in belegbench/results/.

Ergebnisse

v2 (CH / DE / AT, je n = 100, Modus verified)

Modell Modus Config n Field-EM Halluzinationsrate Parse-Fehler
German-OCR-3-8 verified ch 97 95.8 % 6.7 % 0/97
German-OCR-3-8 verified de 100 95.5 % 4.7 % 0/100
German-OCR-3-8 verified at 100 95.3 % 5.0 % 0/100

Je Feld (Field-EM, Auszug):

Feld AT CH DE
invoice_number 99 % 99 % 99 %
invoice_date 99 % 99 % 99 %
total 98 % 98 % 100 %
creditor_iban 98 % 98 % 97 %
supplier_vat_uid 93 % 93 % 92 %
due_date 81 % 77 % 81 %
line_item_count 100 % 99 % 100 %
vat_rates 98 % 96 % 99 %
payment_reference 96 % – 98 %
customer_number 96 % 91 % 94 %

v1 (CH, n = 300, fairer 4B-Vergleich — Config ch-v1)

Modell Field-EM Halluzinationsrate Parse-Fehler
Belegant-4B 84.7 % 0.0 % 3/300
Qwen3.5-4B (roh) 75.8 % 1.2 % 15/100
gemma3:4b (roh) 16.5 % 15.9 % 76/100

Trilingual: FR 81.4 %, IT 83.0 %. Viral-Grafik: belegbench/viral_table.html.

Paket

  • belegbench/schema.py — Extraktionsschema (35 Felder, pydantic) + striktes JSON-Schema für Constrained Decoding
  • belegbench/checks.py — deterministische Prüfungen (IBAN mod-97, QRR, RF/ISO 11649, CHE-UID, DE USt-IdNr., ATU), QR-Oracle-Parser (SPC/EPC), Verifier
  • belegbench/metrics.py — Field-EM / Halluzinationsrate / Verify-Catch-Rate
  • belegbench/run_eval.py — Eval-Harness (OpenAI-kompatible Endpoints, Ollama, vLLM, llama.cpp)

Lizenz

Apache-2.0 · von den Machern von German-OCR

Project links

Data verified by PyPI on Aug 27, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for Keyvan from gravatar.com Keyvan

Credits

Author: Keyvan Hardani

GitHub Statistics

Data verified by PyPI on Aug 27, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

Apache-2.0
View SPDX License List

Requires

Python >=3.11

Provides Extra

belegant

Tags

benchmark invoice qr-bill swiss germany austria girocode ocr document-ai synthetic-data

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

belegbench-0.2.0.tar.gz (18.3 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

belegbench-0.2.0-py3-none-any.whl (18.5 kB view details)

Uploaded Python 3

File details

Details for the file belegbench-0.2.0.tar.gz.

File metadata

  • Download URL: belegbench-0.2.0.tar.gz
  • Upload date:
  • Size: 18.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for belegbench-0.2.0.tar.gz
Algorithm Hash digest
SHA256 9f8d907c8cc6146e5cb046b074a15c7725b7d1465a7c468777f3475ef7c75ce3
MD5 9257f16b67fd1afd46c52080ae8ca3a6
BLAKE2b-256 47be6faa7c2c51143b0cbdf282ea8af415fc7a1ea077997955a735a92dc078c2

See more details on using hashes here.

Provenance

The following attestation bundles were made for belegbench-0.2.0.tar.gz:

Publisher: publish.yml on Keyvanhardani/belegbench

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file belegbench-0.2.0-py3-none-any.whl.

File metadata

  • Download URL: belegbench-0.2.0-py3-none-any.whl
  • Upload date:
  • Size: 18.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for belegbench-0.2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 09a2582d6dd089d9da2fe5282d317fdceb099080df09427cfc03068c039e0dfc
MD5 76732d25bd154420b43366d6312e5a61
BLAKE2b-256 4d2750e1041a33cc93c2333fdf9faa763f74cd3df8367b6a0d8565d42752faf8

See more details on using hashes here.

Provenance

The following attestation bundles were made for belegbench-0.2.0-py3-none-any.whl:

Publisher: publish.yml on Keyvanhardani/belegbench

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.2.0 This release

2 files

0.1.1

2 files

0.1.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page