crystal-metrics
Transparent multimodal reasoning metrics from the CRYSTAL benchmark.
Your model gets the right answer. But does it actually reason? Standard
benchmarks only check the final answer, so a lucky guess scores the same as
sound reasoning. crystal-metrics scores the reasoning chain itself —
step-level precision/recall, ordering, and answer accuracy.
What it measures
| Metric | Measures |
|---|---|
| Match F1 | Step-level F1 of predicted vs. reference reasoning steps via semantic-similarity matching |
| Precision | Fraction of predicted steps that match a reference step (few wrong things) |
| Recall | Fraction of reference steps that were covered (completeness — the hard part) |
| Ordered Match F1 | Match F1 penalized for out-of-order reasoning (Kendall's τ or LIS ratio) |
| Accuracy | Multi-format final-answer correctness (yes/no, numeric, multiple choice, free text) |
It also ships the RL reward functions used to train models on CRYSTAL —
Causal Process Reward (CPR) and Semantic Process Reward (SPR) — in
crystal_metrics.rewards (pure Python, model-agnostic). See the
rewards docs.
Install
pip install crystal-metrics # core metrics (no LLM required)
pip install crystal-metrics[judge] # + optional LLM judge for free-form answers
Requires Python 3.8+. The default embedding model all-distilroberta-v1 is
downloaded and cached on first use.
Quickstart
from crystal_metrics import MLLMReasoningEvaluator
evaluator = MLLMReasoningEvaluator() # all-distilroberta-v1, threshold τ=0.35 (paper defaults)
m = evaluator.evaluate_single(
predicted_steps=[
"Three objects sit on the table",
"The middle console is the smallest",
"Therefore the answer is C",
],
reference_steps=[
"There are three objects in the image",
"Compare the sizes of the three objects",
"The middle object is smallest",
"Select option C",
],
alpha=0.3, # enable Ordered Match F1 (0 = order-insensitive)
)
print(f"Match F1: {m.match_f1:.3f}")
print(f"Precision: {m.precision:.3f}")
print(f"Recall: {m.recall:.3f}")
print(f"Ordered Match F1: {m.ordered_match_f1:.3f}")
Answer accuracy
from crystal_metrics import AccuracyCalculator
calc = AccuracyCalculator(use_llm_grader=False) # rule-based, no LLM
acc = calc.evaluate_dataset(predictions, references)
print(acc["overall_accuracy"], acc["type_statistics"])
The optional LLM judge (for free-form text) needs the [judge] extra and any
OpenAI-compatible endpoint (e.g. a local Ollama server):
calc = AccuracyCalculator(use_llm_grader=True, llm_model="gpt-oss:120b",
base_url="http://localhost:11434/v1")
Command line
crystal-metrics evaluate predictions.json references.json --alpha 0.3
=== CRYSTAL metrics ===
samples : 3
match_f1 : 0.5524
precision : 0.6667
recall : 0.4722
ordered_match_f1 : 0.4952
accuracy : 0.6667
Data format
// predictions
{"<id>": {"question": "...", "reasoning_steps": ["..."], "answer": "..."}}
// references
{"<id>": {"reference_steps": ["..."], "answer": "..."}}
Paper defaults
| Setting | Value | Source |
|---|---|---|
| Embedding model | all-distilroberta-v1 |
Paper §4.3 |
| Similarity threshold τ | 0.35 | Ablation-validated |
Recommended alpha |
0.3 | Paper |
| Numeric tolerance | ε_abs = 0.05, ε_rel = 0.10 | Paper Eq. (2) |
Documentation
Benchmark: 🤗 waybarrios/CRYSTAL · Project: github.com/waybarrios/crystal
Citation
@misc{barrios2026crystal,
title = {Beyond Final Answers: CRYSTAL Benchmark for Transparent
Multimodal Reasoning Evaluation},
author = {Wayner Barrios and SouYoung Jin},
year = {2026},
eprint = {2603.13099},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2603.13099}
}
License
MIT — see the project repository.
Metadata
Release files for crystal-metrics 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| crystal_metrics-0.2.0.tar.gz | 33.7 kB | Without provenance | Details for crystal_metrics-0.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| crystal_metrics-0.2.0-py3-none-any.whl | Python 3 | none | any | Details for crystal_metrics-0.2.0-py3-none-any.whl |
Total release size: 60.0 kB
Release files / crystal_metrics-0.2.0.tar.gz
| Download URL | crystal_metrics-0.2.0.tar.gz |
|---|---|
| Size | 33.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
72801b1bfc5ac3abd8cb65608a5ca8c2e002d46c0c1ad71cb8cf817112020229
|
|
BLAKE2b-256 checksum How to use checksums |
237458212368d54cc364a183bb6caa01c3a291f4cfa8268a782a5576d98494ba
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.2
|
Release files / crystal_metrics-0.2.0-py3-none-any.whl
| Download URL | crystal_metrics-0.2.0-py3-none-any.whl |
|---|---|
| Size | 26.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
3281619e5b1791030426813c296dea9dfbe08e2391c39e6213dfca17b6101972
|
|
BLAKE2b-256 checksum How to use checksums |
725ab688371aebc5e6bae29a4d4af498b5922bd14aa7c763a52244092baeb15e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.2
|
Security for crystal-metrics 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.