Skip to main content

cupel cupel

separates precious LLMs from base LLMs

score local and cloud LLMs with custom prompts and a configurable judge

PyPI - Version Python - Version LICENSE

cupel leaderboard — models ranked by score
a cupel is the small dish used in a fire assay to separate precious metal from base metal

install

curl -fsSL https://cupel.run/install | bash

or

pip install cupel

the UI is bundled in the package

quick start

cupel

opens a browser at localhost:8042

first time cupel is started

ships with example data (8 models scored by Claude Opus 4.6 on 8 prompts) — the dashboard is populated on first launch

  • LLM-assisted authoring — describe what you want to test, an LLM drafts the prompt and 0–3 rubric
  • local + cloud — oMLX, Ollama, LM Studio, SGLang, OpenRouter, Anthropic, OpenAI
  • configurable judge — any model can score responses on a 0–3 rubric with reasoning
  • thinking model support — separates <think> blocks from answers, only judges the response
  • multi-turn + tool calling — multi-step conversations with injected tool results
  • speed tracking — tok/s and response times per model
  • auto-discovery — probes known ports for local inference servers

leaderboard

leaderboard with score vs. speed, overall accuracy, and per-category breakdowns

cupel dashboard — score vs speed scatter plot with leaderboard overall accuracy — horizontal bar chart of all models

category fingerprint — radar chart comparing models across categories

score models

select models from discovered providers, filter by prompt category, choose a judge model, and start the run

bench it — run evals from the browser

progress updates via SSE as each prompt completes

running multiple models

author prompts

describe what to test, select a category and difficulty — an LLM generates the title, prompt text, and 0–3 rubric. edit before saving

authoring a prompt authored prompt with rubric

results

each run is saved as JSON with the model, judge, timestamp, and per-prompt scores with judge reasoning
results can be sorted, tagged, muted and expanded to inspect individual evaluations

results — browse, tag, and manage all scored runs

judge

set a default judge in UI settings or in config.yml:

judge:
  model: claude-opus-4-6

scores are 0–3:

score meaning
3 correct and insightful
2 correct but shallow
1 partially correct
0 wrong or hallucinated

prompt format

{
  "id": 14,
  "category": "math_estimation",
  "title": "Model Memory from Quantization",
  "prompt": "A model has 70B parameters. Estimate memory for FP16, 8-bit, and 4-bit.",
  "rubric": {
    "3": "FP16: ~140GB, 8-bit: ~70GB, 4-bit: ~35GB. Shows the math.",
    "2": "Correct for 2 of 3, or all correct but no explanation.",
    "1": "Gets the direction right but wrong numbers.",
    "0": "Wrong math or doesn't understand quantization."
  }
}

multi-turn prompts

for tool calling and conversations, use turns instead of prompt:

{
  "id": 21,
  "title": "Tool Calling — School Status Check",
  "turns": [
    {
      "messages": [
        {"role": "system", "content": "You have tools: get_grades(name), ..."},
        {"role": "user", "content": "How are both kids doing?"}
      ]
    },
    {
      "inject_after": [
        {"role": "user", "content": "Tool results: get_grades(\"phoebe\") => ..."}
      ]
    }
  ],
  "rubric": { "3": "Emits correct tool calls, synthesizes results...", "..." : "..." }
}

thinking models

cupel handles <think> blocks automatically — separates thinking from the answer, only judges the response:

thinking: null   # model default (recommended)
thinking: 0      # disable
thinking: 4096   # explicit budget

providers

cloud providers can be added from presets (Anthropic, OpenRouter, OpenAI) or as custom endpoints. the settings page fetches model lists from a provider's API (includes per-token pricing for OpenRouter), validates API keys, and tests connections

settings — add cloud providers, fetch models with pricing

cupel auto-discovers local servers on known ports:

port server
8000 oMLX / vLLM
11434 Ollama
1234 LM Studio
30000 SGLang
8080 llama.cpp

API keys

each provider gets its own env var. put them in .env or ~/.cupel/.env:

OMLX_API_KEY=4242
ANTHROPIC_API_KEY=sk-ant-...
OPENROUTER_API_KEY=sk-or-...
OPENAI_API_KEY=sk-proj-...

or configure in config.yml:

providers:
  - name: openrouter
    api_url: https://openrouter.ai/api/v1/chat/completions
    api_key_env: OPENROUTER_API_KEY
    models: [google/gemini-2.5-pro, deepseek/deepseek-r1]

  - name: anthropic
    api_url: https://api.anthropic.com/v1/messages
    api_key_env: ANTHROPIC_API_KEY
    models: [claude-opus-4-6, claude-sonnet-4-6]

CLI

cupel                                  # open dashboard
cupel run                              # collect responses
cupel run --models "Qwen3.5-27B-8bit"  # specific model
cupel run --prompts 18-22              # specific prompts
cupel judge eval-results/*.json        # score with judge
cupel judge eval-results/*.json --judge-model gemma-4-26b-a4b-it-4bit
cupel init                             # create config.yml + eval-set

development

git clone https://github.com/tolitius/cupel.git && cd cupel
pip install -e .
uvicorn cupel.server:app --reload --port 8042

vanilla JS frontend (Preact + HTM from CDN). no build step.

license

Copyright © 2026 tolitius

Distributed under the Apache 2.0 License.

Metadata

Release files for cupel 0.1.90

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for cupel 0.1.90
File Size Uploaded Provenance Actions
cupel-0.1.90.tar.gz 5.3 MB Without provenance Details for cupel-0.1.90.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for cupel 0.1.90
File Interpreter ABI Platform Actions
cupel-0.1.90-py3-none-any.whl Python 3 none any Details for cupel-0.1.90-py3-none-any.whl

Total release size: 5.6 MB

Release files / cupel-0.1.90.tar.gz

Download URL cupel-0.1.90.tar.gz
Size 5.3 MB
Tags Source
SHA-256 checksum
How to use checksums
80576035ca1783054bd43d57087fa36b8e89a2e59275003f595d9975984b990c
BLAKE2b-256 checksum
How to use checksums
3e3f9771d28728c88cf22092d1dde07abbc0fb22aba42da32b3e1d6ad564b291
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.6

Release files / cupel-0.1.90-py3-none-any.whl

Download URL cupel-0.1.90-py3-none-any.whl
Size 252.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
3f15774f70f0d9de88dd9c2ece34bc52e63f1664c26bdee8dc2a2f2258803552
BLAKE2b-256 checksum
How to use checksums
1b997d48ba981a0f29de2b7479525ea73dfb5ba493e3e40d1269dbdbebc6e8e0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.6

Security for cupel 0.1.90

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.90 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page