SommBench: Assessing Sommelier Expertise of Language Models

SommBench is a multilingual benchmark for evaluating sommelier-level wine knowledge in large language models. It spans three complementary tasks — Wine Theory Q&A, Food & Wine Pairing, and Wine Feature Completion.

Quick Start

uv add sommbench

Create a .env file with your API credentials:

API_KEY="your-api-key"
API_BASE="https://api.openai.com/v1"

Run the full benchmark suite:

sommbench run "lm_studio/qwen3.5-0.8b-mlx"

Usage

Command Line

sommbench <COMMAND> <MODEL> [OPTIONS]

Commands

Examples

Run all benchmarks:

sommbench run "gpt-4o" --output ./benchmark_results/

Single task — WTQA on all languages:

sommbench wtqa "gpt-4o"

Specify languages for WFC (Italian and German):

sommbench wfc "gemini-1.5-pro" -l it -l de --batch-size 64 --output wfc_results.json

Multiple passes for FWP:

sommbench fwp "claude-sonnet-4-6" --num-passes 3 --output fwp_results.json

Extra model parameters:

sommbench run "lm_studio/my-model" --model-params '{"stop": ["<|im_end|>"], "max_tokens": 512}'

Inline credentials:

sommbench wtqa "gpt-4-turbo" --api-key "sk-..." --api-base "https://api.openai.com/v1"

Options

sommmbech --help

Python API

Full suite — run all benchmarks in one call:

from pathlib import Path
from sommbench import run

run(
    model="gpt-4o",
    output=Path("results/"),
    api_key="sk-...",
    api_base="https://api.openai.com/v1",
)

Single task — run one benchmark with custom options:

import json
from sommbench import run_wtqa_benchmark

results = run_wtqa_benchmark(
    model="gpt-4o",
    language=["en", "de"],
    num_passes=2,
    api_key="sk-...",
    api_base="https://api.openai.com/v1",
    batch_size=16,
)

with open("wtqa_results.json", "w", encoding="utf-8") as f:
    json.dump(results, f, indent=4, ensure_ascii=False)

Granular workflow — run each task individually and compute the composite score:

from sommbench import (
    compute_sommbench_score,
    run_fwp_benchmark,
    run_wfc_benchmark,
    run_wtqa_benchmark,
)

api_key = "sk-..."
api_base = "https://api.openai.com/v1"

wtqa = run_wtqa_benchmark(model="gpt-4o", language=None, num_passes=1, api_key=api_key, api_base=api_base)
fwp  = run_fwp_benchmark(model="gpt-4o", num_passes=1, api_key=api_key, api_base=api_base)
wfc  = run_wfc_benchmark(model="gpt-4o", language=["en", "de", "it"], num_passes=1, api_key=api_key, api_base=api_base)

score = compute_sommbench_score(wtqa, fwp, wfc)
print(score)
# {'sommbench_score': 0.6821, 's_wtqa': 0.8012, 's_fwp': 0.5432, 's_wfc': 0.7019}

License

This project is licensed under the MIT License. See LICENSE for details.

Citation

If you use SommBench in your research, please cite:

@misc{brach2025sommbench,
    title={SommBench: Assessing Sommelier Expertise of Language Models},
    author={William Brach and Peter Hartman and Marek Šuppa},
    year={2025},
    eprint={2603.12117},
    archivePrefix={arXiv},
    primaryClass={cs.CL},
    url={https://arxiv.org/abs/2603.12117},
}