drvd-bench 0.2.0

Benchmark toolkit for DrVD

DrVD-Bench: Do Vision-Language Models Reason Like Human Doctors in Medical Image Diagnosis?

paper ｜ kaggle ｜ huggingface ｜ github

This repository is the official implementation of the paper: DrVD-Bench: Do Vision-Language Models Reason Like Human Doctors in Medical Image Diagnosis?

Introduction

Vision–language models (VLMs) exhibit strong zero-shot generalization on natural images and show early promise in interpretable medical image analysis. However, existing benchmarks do not systematically evaluate whether these models truly reason like human clinicians or merely imitate superficial patterns.
To address this gap, we propose DrVD-Bench, the first multimodal benchmark for clinical visual reasoning. DrVD-Bench consists of three modules: Visual Evidence Comprehension, Reasoning Trajectory Assessment, and Report Generation Evaluation, comprising 7 789 image–question pairs.
Our benchmark covers 20 task types, 17 diagnostic categories, and five imaging modalities—CT, MRI, ultrasound, X-ray, and pathology. DrVD-Bench mirrors the clinical workflow from modality recognition to lesion identification and diagnosis.
We benchmark 19 VLMs (general-purpose & medical-specific, open-source & proprietary) and observe that performance drops sharply as reasoning complexity increases. While some models begin to exhibit traces of human-like reasoning, they often rely on shortcut correlations rather than grounded visual understanding. DrVD-Bench therefore provides a rigorous framework for developing clinically trustworthy VLMs.

DrVD-Bench Usage Instructions

0. Installation

pip install drvd-bench

---

1. Function get_drvd_data

Functionality

Reads JSONL, generates the final prompt, and returns an iterator of (img_path, prompt, origin_data).

Note

This code only returns an iterator for retrieving data and does not include model inference.

Parameters

• jsonl_path (str | Path): Path to the dataset JSONL file
• image_root (str | Path): Root directory of the dataset
• data_type ({“single”, “joint”}): Question type, default “single”. Use “joint” only for joint_qa.jsonl; all other JSONL files should use “single”
• verbose (bool): Whether to show a progress bar; default is True

Returns

Iterator[(img_path:str, prompt:str, record:dict)]

Example

from drvd_bench import get_drvd_data
for img, prompt, original_data in get_drvd_data(
    "test_drvd_pypi/data/visual_evidence_qa.jsonl",
    "test_drvd_pypi/data",
    data_type="single"
):
    print(img, prompt[:80] + "…")

---

2. Function map_result

Functionality

Calls the DeepSeek Chat API to map open-ended responses to options and writes back to JSONL.

Note

Use only when the model instruction-following capability is insufficient and for non-report generation tasks. Not needed if the model instruction-following is strong.

Parameters

• api_key (str): DeepSeek API Key; refer to the DeepSeek official website for details
• input_path (str | Path): Path to the original prediction JSONL file
• output_path (str | Path): Path to save the mapped JSONL file
• base_url (str): Private deployment URL; typically does not need modification
• show_preview (int): Preview the first N results; default is 5

Example

from drvd_bench import map_result
map_result(
    api_key="sk-…",
    input_path="pred.jsonl",
    output_path="mapped.jsonl"
)

---

3. Function compute_choice_metric

Functionality

Calculates accuracy for multiple-choice questions.

Note

Your JSONL file must contain an answer key for the original data’s answer. The model’s result must be stored in the key model_response and should be a single option, such as A, B, C, D, etc.

Parameters

• jsonl_path (str | Path): Path to the prediction result JSONL file
• mode ({“single”, “joint”}): Question type, default “single”. Use “joint” only for joint_qa.jsonl; all other multiple-choice JSONL result files should use “single”

Example

from drvd_bench import compute_choice_metric
compute_choice_metric(
    "visual_evidence_qa_pred.jsonl",
    mode="single"
)

---

4. Function compute_report_generation_metric

Functionality

Computes BLEU and BERTScore F1 for report generation tasks.

Note

Your JSONL file must contain an answer key for the original data’s answer. The model’s result must be stored in the key model_response.

Parameters

• api_key (str): DeepSeek API Key
• json_path (str | Path): Path to the JSONL result file containing answer / model_response fields for report generation
• base_url (str): Private deployment URL; typically does not need modification

Example

from drvd_bench import compute_report_generation_metric
compute_report_generation_metric(
    api_key="sk-…",
    json_path="report_generation_preds.jsonl"
)

Contact

• Tianhong Zhou · [e-mail]
• Yin Xu · [e-mail]
• Yingtao Zhu · [e-mail]