crystal-metrics 0.1.0

Transparent multimodal reasoning metrics from the CRYSTAL benchmark (Match F1, Ordered Match F1, accuracy).

crystal-metrics

Transparent multimodal reasoning metrics from the CRYSTAL benchmark — Match F1, Ordered Match F1, Precision, Recall, and multi-format Accuracy.

pip install crystal-metrics          # core metrics
pip install crystal-metrics[judge]   # + optional LLM judge

from crystal_metrics import MLLMReasoningEvaluator

evaluator = MLLMReasoningEvaluator()  # all-distilroberta-v1, tau=0.35 (paper defaults)
m = evaluator.evaluate_single(
    predicted_steps=["Three objects on a table", "The middle one is smallest", "Answer C"],
    reference_steps=["There are three objects", "Compare their sizes", "Middle is smallest", "Select C"],
    alpha=0.3,  # enable Ordered Match F1
)
print(m.match_f1, m.precision, m.recall, m.ordered_match_f1)

See the docs for installation, quickstart, metric definitions, and the CLI.