mosaic-bench

Run the MOSAIC benchmark — nine validated psychometric questionnaires and four ethical dilemmas — against any LLM. Test data is pulled directly from the MOSAIC repository at a pinned commit and cached locally, so there is nothing to download by hand.

Install

pip install "mosaic-bench[openai]"     # OpenAI, Groq, DeepSeek, vLLM, Ollama, ...
pip install "mosaic-bench[hf]"         # HF Inference Providers
pip install "mosaic-bench[local]"      # offline, via transformers

Run

export OPENAI_API_KEY=sk-...
mosaic --model gpt-4o --test mfq2 sdo --runs 10

Or pass the key inline:

mosaic --model gpt-4o --api-key sk-... --test mfq2

Any OpenAI-compatible endpoint works without code changes:

mosaic --backend openai --model qwen/qwen3-32b \
       --base-url https://api.groq.com/openai/v1 --api-key gsk_...

mosaic --backend openai --model my-finetune \
       --base-url http://localhost:8000/v1 --api-key dummy   # local vLLM

Anything on the Hub:

export HF_TOKEN=hf_...
mosaic --backend hf --model meta-llama/Llama-3.3-70B-Instruct --test mfq2

Fully offline, no keys:

mosaic --backend local --model Qwen/Qwen3-8B --test mfq2 --runs 3

Output

mfq2 — gpt-4o
  authority                      21.00 ± 1.00
  care                           24.33 ± 2.08
  equality                       19.67 ± 5.13
  ! 2.1% of answers imputed — {'ok': 340, 'unparsable': 8}

Two files per test under results/<model>/:

• <test>_<variant>.jsonl — one record per invocation: raw response, run index, temperature, timestamp, provider status.
• <test>_<variant>_scored.jsonl — the same records plus parsed_score, parse_status and imputed.

Runs are resumable: re-running the same command skips (item, run) pairs already on disk, and raising --runs only fills in the missing repetitions.

Python API

from mosaic_bench import build_backend, run_test

backend = build_backend("gpt-4o", api_key="sk-...")
summary = run_test(backend, "mfq2", runs=10)
print(summary["scores"], summary["quality"])

Notes on scoring

• Answers are parsed leniently but validated against each test's declared range; out-of-range values are rejected rather than accepted.
• Unparsable answers are imputed with the test's neutral score, and the imputed fraction is reported alongside every result.
• Provider content filters are recorded as content_filtered and excluded from aggregation — never converted into a score.
• Reverse-keyed items are flipped using scale_min + scale_max.

Publishing

Until this is on PyPI, it installs straight from the repository:

pip install "git+https://github.com/EricaCoppolillo/MOSAIC#egg=mosaic-bench[openai]"

To publish to PyPI, from the package root:

python -m build
python -m twine upload dist/*

Prefer PyPI Trusted Publishing over an API token: it authenticates the GitHub Actions workflow via OIDC, so no long-lived secret lives in the repo. Configure the publisher once at pypi.org/manage/account/publishing/, then tag a release.