COVALE: Compositional Open-Vocabulary Anatomical Localization Evaluation

Text-based metrics can tell whether two anatomical descriptions use similar words, but they often miss whether the descriptions point to the same place. COVALE turns each description into a 3D region in an anatomical atlas, then measures how much the regions overlap. This makes it easy to compare generated descriptions, evaluate systems, and provide spatial feedback during model training.

Quick start

pip install covale

import os

from covale import COVALE

os.environ["OPENAI_API_KEY"] = "your-api-key"

score = COVALE().score(
    reference="left frontal lobe",
    candidate="left frontal region",
)

Table of contents

• Quick start
• Calculate COVALE
• Additional resources
• License

Calculate COVALE

Create an evaluator and pass equally sized lists of reference and candidate descriptions:

from covale import COVALE

covale_evaluator = COVALE(
    method="llm",
    model="gpt-6-astra",
    concurrency=4,
)
results = covale_evaluator(
    refs=[
        "left frontal lobe",
        "right temporal lobe",
    ],
    hyps=[
        "left frontal region",
        "right temporal region",
    ],
)
print(results["dice"])

Change model to select an OpenAI model. Increase concurrency to evaluate independent pairs in parallel.

Choose an evaluation method

Set method based on how descriptions should be compared:

Evaluate reports

Add extract_findings=True when the inputs are sentences or reports rather than isolated anatomical locations:

report_evaluator = COVALE(
    extract_findings=True,
)
results = report_evaluator(
    refs=reference_reports,
    hyps=generated_reports,
)

COVALE extracts anatomically localized findings, aligns compatible findings one to one, evaluates each matched location, and gives no credit for missing or extra findings. Location strings remain the default and skip finding extraction.

Inspect detailed results

Add output_mode="detailed" to include per-sample scores, localization expressions, timings, and categorized failures:

detailed_evaluator = COVALE(
    output_mode="detailed",
)
results = detailed_evaluator(refs=refs, hyps=hyps)

Additional resources

License

COVALE is available under the MIT License.