Docling-eval

Evaluate Docling on variety of datasets.

Features

Evaluate docling on various datasets. You can use the cli

terminal %> docling-eval --help
                                                                                                                                                                                                                                                
 Usage: docling_eval [OPTIONS] COMMAND [ARGS]...                                                                                                                                                                                                
                                                                                                                                                                                                                                                
 Docling Evaluation CLI for benchmarking document processing tasks.                                                                                                                                                                             
                                                                                                                                                                                                                                                
╭─ Options ────────────────────────────────────────────────────────────────────────────╮
│ --help          Show this message and exit.                                          │
╰──────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────────────╮
│ create        Create both ground truth and evaluation datasets in one step.          │
│ create-eval   Create evaluation dataset from existing ground truth.                  │
│ create-gt     Create ground truth dataset only.                                      │
│ evaluate      Evaluate predictions against ground truth.                             │
│ visualize     Visualize evaluation results.                                          │
╰──────────────────────────────────────────────────────────────────────────────────────╯

Benchmarks

• General
  • DP-Bench benchmarks: Text, layout, reading order and table structure evaluation on the DP-Bench dataset.
  • OmniDocBench benchmarks: Text, layout, reading order and table structure evaluation on the OmniDocBench dataset.
• Layout
  • DocLayNetV1 Benchmarks: Text and layout evaluation on the DocLayNet v1.2 dataset.
• Table-Structure
  • FinTabnet Benchmarks: Table structure evaluation on the FinTabNet dataset.
  • PubTabNet benchmarks: Table structure evaluation on the PubTabNet dataset.
  • Pub1M benchmarks: Table structure evaluation on the Pub1M dataset.

On our list for next benchmarks:

• OmniOCR
• Hyperscalers
• CoMix
• DocVQA
• rd-tablebench
• BigDocs-Bench

Contributing

Please read Contributing to Docling for details.

License

The Docling codebase is under MIT license. For individual model usage, please refer to the model licenses found in the original packages.

IBM ❤️ Open Source AI

Docling-eval has been brought to you by IBM.