crimson-score 0.2.0

CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation

CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation

Paper | Model (HuggingFace)

CRIMSON is a clinically grounded evaluation framework for chest X-ray report generation that assesses reports based on diagnostic correctness, contextual relevance, and patient safety. Unlike prior metrics, CRIMSON incorporates full clinical context, including patient age, indication, and guideline-based decision rules. CRIMSON evaluates only abnormal findings, excluding normal findings from scoring. The framework categorizes errors into a comprehensive taxonomy covering false findings, missing findings, and eight attribute-level errors (e.g., location, severity, measurement, and diagnostic overinterpretation). Each finding is assigned a clinical significance level (urgent, actionable non-urgent, non-actionable, or expected/benign), based on a guideline developed in collaboration with attending cardiothoracic radiologists, enabling severity-aware weighting that prioritizes clinically consequential mistakes over benign discrepancies. Findings are weighted as 1.0 (urgent), 0.5 (actionable non-urgent), 0.25 (non-actionable), or 0.0 (expected/benign), and attribute errors are weighted as 0.5 (significant) or 0.0 (negligible). After weighting, the framework produces a score in the range of (-1, 1], where 1 represents a perfect report, 0 indicates the report is no more informative than a normal template, and negative scores indicate more weighted errors than correct findings.

Installation

Step 1: Create and activate a dedicated environment (Conda or Python venv):

# Option 1: Conda
conda create -n crimson python=3.12 -y
conda activate crimson

# Option 2: Python venv
python -m venv crimson
source crimson/bin/activate

python -m pip install --upgrade pip

Step 2: Install CRIMSON:

pip install crimson-score

Or install from source:

git clone https://github.com/rajpurkarlab/CRIMSON.git
cd CRIMSON
pip install -e .

Usage

By default, CRIMSON uses the fine-tuned MedGemmaCRIMSON model via HuggingFace.

To use OpenAI GPT models instead, set your API key:

export OPENAI_API_KEY="your-openai-api-key"

Scoring a report pair

from CRIMSON import CRIMSONScore

# Default: uses the HuggingFace MedGemmaCRIMSON model
scorer = CRIMSONScore()
result = scorer.evaluate(
    reference_findings="Cardiomegaly. Small bilateral pleural effusions.",
    predicted_findings="Normal heart size. Small left pleural effusion.",
)

print(f"CRIMSON Score: {result['crimson_score']:.2f}")
print(f"False findings: {result['error_counts']['false_findings']}")
print(f"Missing findings: {result['error_counts']['missing_findings']}")
print(f"Attribute errors: {result['error_counts']['attribute_errors']}")

Scoring with patient context

Providing patient context (age, indication) enables context-dependent clinical significance assignment — for example, missing aortic atherosclerosis in an 82-year-old with routine preoperative evaluation is expected/benign, but in a 25-year-old with chest pain it may be actionable.

result = scorer.evaluate(
    reference_findings="Bibasilar atelectasis. Mild cardiomegaly. Aortic atherosclerosis with vascular calcification.",
    predicted_findings="Bibasilar atelectasis. Mild cardiomegaly.",
    patient_context={
        "Age": "82",
        "Indication": "Routine preoperative evaluation",
    },
)

Batch evaluation

Score multiple report pairs in a single call. For HuggingFace and vLLM backends this runs batched GPU inference, which is significantly faster than calling evaluate() in a loop.

results = scorer.evaluate_batch(
    reference_findings_list=[
        "Cardiomegaly. Small bilateral pleural effusions.",
        "No acute cardiopulmonary abnormality.",
        "Right lower lobe pneumonia.",
    ],
    predicted_findings_list=[
        "Normal heart size. Small left pleural effusion.",
        "No acute cardiopulmonary abnormality.",
        "Right lower lobe consolidation concerning for pneumonia.",
    ],
    patient_contexts=[
        {"Age": "72", "Indication": "Shortness of breath"},
        None,
        {"Age": "45", "Indication": "Cough and fever"},
    ],
)

for i, result in enumerate(results):
    print(f"Sample {i}: {result['crimson_score']:.2f}")

Using vLLM (faster inference)

For large-scale evaluation, vLLM provides significantly faster inference via continuous batching:

pip install crimson-score[vllm]

scorer = CRIMSONScore(api="vllm")
result = scorer.evaluate(reference_findings="...", predicted_findings="...")

Using an OpenAI model (requires API key)

scorer = CRIMSONScore(api="openai", model_name="gpt-5.2")
result = scorer.evaluate(reference_findings="...", predicted_findings="...")

Reference

@article{baharoon2026crimson,
  title={CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation},
  author={Baharoon, Mohammed and Heintz, Thibault and Raissi, Siavash and Alabbad, Mahmoud and Alhammad, Mona and AlOmaish, Hassan and Kim, Sung Eun and Banerjee, Oishi and Rajpurkar, Pranav},
  journal={arXiv preprint arXiv:2603.06183},
  year={2026}
}