Skip to main content

LemonMatrix

A benchmarking control plane for Lemonade instances. One profile = one Lemonade instance, auto-discovered and reused as the environment fingerprint for every run on that machine. See schema/result.schema.json for the canonical result schema.

Live public leaderboard: lemonmatrix.github.io -- results and the site's own generator live in the sibling LemonMatrix/lemonmatrix.github.io repo; this repo is code (CLI, library, tests, schema) only. Community submissions go through a PR adding a result JSON under results/<profile>/ there, gated by CI running lemonmatrix validate-submission against the schemas in this repo.

Install

pip install lemonmatrix

Python 3.10 or newer. The step-by-step guide walks from a running Lemonade to a result on the leaderboard.

To work on LemonMatrix itself:

git clone https://github.com/Bekhouche/LemonMatrix.git
cd LemonMatrix
pip install -e ".[dev]"

Quickstart

# Find a Lemonade instance running on this machine and save it as a profile.
lemonmatrix profile detect --save my-machine

# Or add one you already know the address of.
lemonmatrix profile add my-machine --host 192.168.1.42 --port 13305 --token $LEMONADE_API_KEY

lemonmatrix profile list
lemonmatrix profile show my-machine

# Run a sweep and write a schema-conformant result JSON under results/.
# OS isn't a flag -- it's derived from the profile's own discovered
# environment, since a different OS is a different profile, not a per-run
# setting.
lemonmatrix run --profile my-machine \
  --model Llama-3.1-8B-Instruct-GGUF --model-class dense --quant Q4_K_M \
  --context-length 4096 --engine igpu --backend llama.cpp-vulkan \
  --power-state plugged

# Browse profiles and results, or launch new sweeps, from a local dashboard.
lemonmatrix dashboard

lemonmatrix profile detect --subnet 192.168.1.0/24 extends the scan to a LAN range for the multi-machine fleet case. Lemonade has no announce/broadcast mechanism, so this is a bounded port scan, not magic discovery -- capped at 1024 hosts per scan.

Router cost runs

A collection.router that mixes local and cloud candidates answers one question: how much of the cloud bill does local hardware take over? Give the router a mixed prompt set and LemonMatrix records where each request went and prices it:

lemonmatrix run --profile my-machine --run-type router \
  --model user.My-Router --context-length 4096 --power-state plugged \
  --prompt-set router-mix --trials 10

The result's metrics.routing block holds the share of requests per candidate, the share that stayed local, the cost of the run, and the cost of sending every request to the most expensive candidate instead. Prices are USD per million tokens. They come from Lemonade's own estimated_cost when the candidate carries cost metadata (cloud models get it from their provider), and non-cloud candidates count as free. --cost MODEL=IN,OUT sets or overrides a price, --cost-baseline MODEL picks the comparison model, and --prompts-file takes your own prompts (one per line, or JSONL).

After the trials, each prompt the router sent elsewhere goes once to the baseline model, and the baseline is priced on the tokens that model actually wrote. This matters for reasoning models, which can write ten times the tokens of a small local model for the same prompt. The extra requests run outside the measured window; --estimate-baseline skips them and reuses the routed model's token counts, and the result's baseline_method says which was used.

Submitting results

lemonmatrix submit results/my-machine/<run_id>.json opens a pull request that adds the file to the public leaderboard (LemonMatrix/lemonmatrix.github.io). It validates the file against the same schemas the leaderboard's CI uses, forks the repo (or reuses your fork), keeps the file's results/<profile>/... path, pushes and opens the PR with the GitHub CLI. --dry-run shows the PR without touching GitHub.

Importing other tools' results

lemonmatrix import FILE... reads results from lemonade_lab (as published in amdaibenchmarks) and LemonMetrics report.json files, which carry Ryzen AI NPU/hybrid rows and energy estimates. Each file is normalized into results/imported/<tool>/ and listed with lemonmatrix imported-results. These numbers did not go through LemonMatrix's validity checks, so they are marked validated: false and never appear on the leaderboard. Host names, commands, URLs and raw power samples are not copied. When several backends report the same number for one model, the import warns that the backend switch may not have taken effect.

Dashboard

lemonmatrix dashboard starts a local Flask app (default http://127.0.0.1:5050) with:

  • A sortable, filterable leaderboard over every result under results/.
  • Profile management: detect, add, and inspect environment fingerprints.
  • A form to launch a new sweep against a saved profile (runs synchronously, same as the CLI's run command).

It's a single-user local tool with no job queue -- submitting a run blocks the page until the sweep finishes.

Development

pytest tests/

Tests run against a fake Lemonade server (tests/conftest.py) modeled on the documented /api/v1/* response shapes, so they don't require real AMD hardware.

Metadata

Release files for lemonmatrix 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for lemonmatrix 0.1.0
File Size Uploaded Provenance Actions
lemonmatrix-0.1.0.tar.gz 179.0 kB Without provenance Details for lemonmatrix-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for lemonmatrix 0.1.0
File Interpreter ABI Platform Actions
lemonmatrix-0.1.0-py3-none-any.whl Python 3 none any Details for lemonmatrix-0.1.0-py3-none-any.whl

Total release size: 346.5 kB

Release files / lemonmatrix-0.1.0.tar.gz

Download URL lemonmatrix-0.1.0.tar.gz
Size 179.0 kB
Tags Source
SHA-256 checksum
How to use checksums
e41d2fa12de80c4280ef513a9df7a43b40cc92f2e325d7fabbda50772b70a894
BLAKE2b-256 checksum
How to use checksums
63802d5cc2401c956ce983fdebdd85943c0dedde316ec8ddc8f88572b8a94dc4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.7

Release files / lemonmatrix-0.1.0-py3-none-any.whl

Download URL lemonmatrix-0.1.0-py3-none-any.whl
Size 167.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
c199eea5bb9015072c2c7da6b37bb547c54f78eb4b48740add2a30890a48b4bc
BLAKE2b-256 checksum
How to use checksums
87dc62d7a3954d5143955d323e162483598b76587a2137cc33ccc0b40c5ab8a6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.7

Security for lemonmatrix 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page