🔬 SciFigBench

Evaluating VLMs' Understanding of Scientific Method Figures.

News · Overview · Quick Start · License & Citation

🔥 News

• [Oct. 20, 2026] 🎉 SciFigBench is officially released! Following a second round of human verification, we further strengthened the benchmark's reliability and revised approximately 5.8% of its reference answers.
• [Aug. 21, 2026] 🎉 Our paper, SciFigBench: A Benchmark for Scientific Method Figure Understanding, was accepted to EMNLP 2026 Main Conference!
• [May 26, 2026] 📝 We submitted SciFigBench to the May 2026 ARR cycle, targeting EMNLP 2026.

👋 Overview

SciFigBench is a diagnostic benchmark for evaluating Vision-Language Models (VLMs) on scientific method figure understanding. It consists of 4,352 real-world scientific method figures and 40,670 QA pairs, organized into five fine-grained tasks across three core capabilities: Dense Content Perception, Visual Relation Comprehension, and Information-Flow Tracing.

By separately evaluating text recognition, arrow-induced structural relations, and information-flow reasoning, SciFigBench helps identify where VLMs struggle to understand scientific method figures.

Five diagnostic tasks across three complementary capabilities.

SciFigBench also provides human-verified content and structural annotations for further research. The full dataset is available on Hugging Face.

Note on reported results. The paper reports results on an earlier QA version. Retrospective evaluations show modest score changes, with relative model performance trends and main conclusions remaining largely consistent.

🚀 Quick Start

🤖 Using a Coding Agent? Follow AGENTS.md to quickly set up SciFigBench and evaluate your VLM with tools like Codex, Claude Code, or Cursor.

1. Installation

Install SciFigBench from source:

git clone https://github.com/LeNcyer/SciFigBench.git
cd SciFigBench
pip install .

Install via PyPI:

pip install scifigbench

2. One-call Evaluation

Provide your own callable VLM (model(prompt, image) -> str). SciFigBench automatically handles data downloading, input rendering, model inference, and evaluation.

from scifigbench import SciFigBench

model = MyVLM(...)  # Your callable VLM

# Quick Test (default): 5 QA per task, up to 25 calls.
# Partial scores only, NOT official benchmark results.
result = SciFigBench(model=model)
print(result.scores)

# Full Evaluation: all 40,670 QA pairs.
# WARNING: May incur substantial API costs.
result = SciFigBench(
    model=model,
    max_samples_per_task=None,
    confirm_full_run=True,
)
print(result.scores)

For advanced configuration and resuming evaluations, see AGENTS.md.

3. Download the Dataset

The benchmark dataset is hosted on Hugging Face. Download the QA pairs and images from the latest public version:

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="LeNcyer/SciFigBench",
    repo_type="dataset",
    revision="main",  # Latest public version; this branch can change.
    local_dir="./data",
    allow_patterns=["QA/*", "images/*.jpg", "checksums.sha256"],
)

The toolkit also supports dataset downloads:

scifigbench download --repo-id LeNcyer/SciFigBench --revision main --data-root ./data

4. Run Your VLM

Use your preferred VLM to answer the benchmark questions. SciFigBench provides the official prompts and a rendering utility for generating query-specific visual inputs.

import json
from scifigbench import prepare_model_payload

with open("data/QA/qa_edge_level_verification.jsonl", encoding="utf-8") as f:
    qa = json.loads(next(f))

payload = prepare_model_payload(qa, data_root="./data")

# Pass payload["prompt"] and payload["image"] to your VLM.

Save your model's raw responses as a JSONL file, with one prediction per line:

{"qid": "cf32f75bc3d9", "response": "true"}
{"qid": "0aaf24e73a36", "response": "1"}

Expected answer text by task:

5. Evaluate Predictions

Evaluate all five tasks using the official SciFigBench metrics:

scifigbench evaluate \
    --predictions predictions.jsonl \
    --task all \
    --data-root ./data

For additional commands and Python API details, refer to AGENTS.md or run scifigbench --help.

✍️ License & Citation

The SciFigBench Evaluation Toolkit is released under the MIT License. See LICENSE.

If you find our work helpful, please use the following citation.

@inproceedings{wu2026scifigbench,
  title     = {SciFigBench: A Benchmark for Scientific Method Figure Understanding},
  author    = {Wu, Lijia and Li, Zheng and Liu, Yuxuan and Zhang, Jingwen and Zhao, Penghai and Li, Xiang},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing},
  year      = {2026},
  eprint    = {2610.27087},
  archivePrefix = {arXiv},
  url       = {https://arxiv.org/abs/2610.27087}
}