CaptionEvalKit-for-VLMs

Reproducible, all-in-one image captioning evaluation for VLMs.

• For metric developers:
  • 🤔 Need a reliable way to reproduce Kendall's tau for captioning metrics?
  • 😄 Evaluate metrics and reproduce reported results with a single command!
• For VLM developers:
  • 🤔 Tired of preparing separate dependency environments for each metric?
  • 😄 Score VLM-generated captions using a comprehensive set of established captioning metrics.

CaptionEvalKit currently supports:

• LLM-free metrics: Polos, CLIPScore, PAC-S, RefCLIPScore, RefPAC-S, and more
• LLM-as-a-Judge metrics: FLEUR, RefFLEUR, VELA, and EXPERT
• Classic captioning metrics: BLEU, ROUGE-L, METEOR, CIDEr, SPICE, and JaSPICE
• Benchmarks: Composite, Flickr8k-Ex, Flickr8k-CF, Polaris, Nebula, and LongCap-Arena

Table of Contents

• Install
• For VLM Developers
• For Metric Developers
• Reproduce Reported Results
• Reproduction Status
• Supported Metrics
• Supported Benchmarks
• Data and Assets
• Development
• Citation

Install

Requirements: Python 3.10+, git, and uv. Java is also required for METEOR/SPICE through pycocoevalcap. JaSPICE requires Docker; CaptionEvalKit builds and starts the local JaSPICE server automatically when needed.

From PyPI or a built wheel:

pip install capevalkit
capevalkit doctor
capevalkit list-metrics

From a source checkout:

git clone --recursive https://github.com/YuigaWada/CaptionEvalKit-for-VLMs.git
cd CaptionEvalKit-for-VLMs
uv tool install --editable "$PWD" --force
capevalkit list-metrics

~/.cache/capevalkit/
  runtime/<lock-digest>/
    metrics/
    metrics/upstreams/
    benchmarks/expected/
    overlays/
  uv/
  huggingface/

CAPEVALKIT_HOME=/scratch/capevalkit capevalkit doctor

For Metric Developers

Benchmark existing metrics, or evaluate your own metric without adopting a fixed metric signature.

When changing upstream submodule revisions for a release, regenerate the runtime lock:

python scripts/generate_upstream_lock.py

capevalkit benchmark \
  --metric clipscore \
  --benchmark composite \
  --output outputs/clipscore/composite.json

capevalkit suite \
  --metrics clipscore \
  --benchmarks composite,flickr8k-ex,flickr8k-cf,nebula,polaris \
  --output-dir outputs/clipscore

[metric]
name = "mymetric"
python = ">=3.10,<3.12"
module = "capevalkit.metrics.mymetric"

[repository]
dir = "metrics/upstreams/mymetric"
uv_project = "metrics/upstreams/mymetric"

[runner]
command = ["python", "score.py"]

[project]
name = "mymetric"
version = "0.1.0"
requires-python = ">=3.10,<3.12"
dependencies = []

--predictions PREDICTIONS.jsonl
--references REFERENCES.jsonl
--output OUTPUT.json

capevalkit benchmark \
  --metric mymetric \
  --benchmark composite \
  --output outputs/mymetric/composite.json

import capevalkit.api as capeval

class MyMetric:
    def __call__(self, samples):
        return {
            sample.id: float(bool(sample.prediction and sample.references))
            for sample in samples
        }

result = capeval.evaluate_metric(
    benchmark="flickr8k-cf",
    metric=MyMetric(),
    metric_name="MyMetric",
    output="outputs/mymetric/flickr8k-cf.json",
)

The callable receives CaptionSample objects and returns {sample_id: score}. Your metric can keep any internal signature.

For VLM Developers

Evaluate saved captions from files, or run your caption model on your own images.

{"id": "0001", "caption": "A dog runs through grass.", "image": "0001.jpg"}
{"id": "0002", "caption": "A person rides a bicycle.", "image": "0002.jpg"}

{"id": "0001", "references": ["A dog runs outside.", "A dog is in a grassy field."]}
{"id": "0002", "references": ["A cyclist rides on a road.", "A person rides a bike."]}

capevalkit score \
  --metric clipscore \
  --predictions predictions.jsonl \
  --references references.jsonl \
  --image-dir images \
  --output outputs/clipscore.json

{
  "CLIPScore": 0.73,
  "RefCLIPScore": 0.81,
  "per_item": {
    "0001": {"CLIPScore": 0.70, "RefCLIPScore": 0.78}
  }
}

Run these examples with uv run python from the repository, or install capevalkit into your own Python environment.

import capevalkit.api as capeval

def predict(batch):
    return ["A dog runs through grass." for _ in batch.images]

results = capeval.evaluate_caption_model(
    images=["images/0001.jpg", "images/0002.jpg"],
    metrics=["cider", "clipscore"],
    predict=predict,
    references=[
        ["A dog runs outside.", "A dog is in a grassy field."],
        ["A cyclist rides on a road.", "A person rides a bike."],
    ],
    batch_size=8,
    output_dir="outputs/my-model",
)

If captions are already generated, pass image-caption pairs directly:

import capevalkit.api as capeval

results = capeval.evaluate_captions(
    pairs=[
        {
            "id": "0001",
            "image": "images/0001.jpg",
            "caption": "A dog runs through grass.",
            "references": ["A dog runs outside.", "A dog is in a grassy field."],
        },
        {
            "id": "0002",
            "image": "images/0002.jpg",
            "caption": "A person rides a bicycle.",
            "references": ["A cyclist rides on a road.", "A person rides a bike."],
        },
    ],
    metrics=["cider", "clipscore"],
    output_dir="outputs/my-captions",
)

For manual caption-model control:

import capevalkit.api as capeval

def predict(batch):
    return ["A dog runs through grass." for _ in batch.images]

with capeval.CaptionEvalRun(
    images=["images/0001.jpg", "images/0002.jpg"],
    metrics=["cider", "clipscore"],
    references=[
        ["A dog runs outside.", "A dog is in a grassy field."],
        ["A cyclist rides on a road.", "A person rides a bike."],
    ],
    output_dir="outputs/my-model",
) as run:
    for batch in run.iter_batches(batch_size=8):
        run.record(batch.ids, predict(batch))

    results = run.evaluate()

Reproduce Reported Results

Preview the default reproducibility suite:

capevalkit all_reproduce --dry-run

Run one verified pair:

capevalkit all_reproduce \
  --metrics clipscore \
  --benchmarks composite

Run a launch smoke test for every default pair:

capevalkit all_reproduce --smoke --jobs 4 --gpu-jobs 1

--smoke runs one sample per pair and checks launch/output writing only. Omit it for full correlations.

Reproduction Status

Legend: ✅ reproduced, ⚠️ not reproduced, - no default target. For LongCap-Arena, unreproduced targets are also shown as -.

Supported Metrics

Supported Benchmarks

Data and Assets

Benchmark datasets are cached on first use under <runtime-root>/.hf-cache/benchmarks/. In a source checkout, <runtime-root> is the repository root; in a wheel install, it is $CAPEVALKIT_HOME/runtime/<lock-digest>.

Model files and checkpoints are downloaded on first use by the corresponding metric runner or upstream library. First-use downloads and runtime setup print Preparing..., Downloading..., and cache-completion status lines. Direct HTTP assets and checkpoints also report byte progress.

Set IC_EVAL_REFRESH_HF_CACHE=1 to refresh cached benchmark rows and extracted images.

data/
  composite/
    en_test_composite_da2.csv
    images/
  flickr8k/
    flickr8k.json
    crowdflower_flickr8k.json
    images/
  nebula/
    images/
  polaris/
    images/

Development

uv run python -m unittest discover -s tests

Repository map:

.
├── capevalkit/
│   ├── api.py                 public Python API endpoint
│   ├── interfaces/            CLI entrypoint, Python API, presenters
│   ├── application/           use cases and verification services
│   ├── domain/                evaluation/reproduction policies and value objects
│   └── infrastructure/        runtime, metric execution, assets, manifests, benchmark loaders
├── metrics/
│   ├── */metric.toml          metric manifests
│   └── upstreams/*            upstream metric repositories
├── overlays/
│   └── metrics/upstreams/*    uv overlays for upstream repositories
└── benchmarks/
    └── expected/              default all_reproduce expected values

Citation

If you use this toolkit, cite the original metric and benchmark papers for the implementations and reported values you rely on.