LLMRig

Know what your rig can run.

The open compatibility and performance intelligence layer for local AI.

LLMRig determines viable ways to run AI models on real hardware. It keeps model, artifact, quantization, runtime, context, and measured performance separate so missing evidence does not become a confident recommendation.

Install the isolated CLI and run the safest useful first command:

pipx install llmrig
llmrig solve qwen3:0.6b

Default solve is read-only: it does not download a model, install or start a runtime, execute inference, or write benchmark output. Measurement is an explicit step with --verify, and only already-local candidates are eligible.

hardware
  → model and artifact resolution
  → runtime compatibility and local availability
  → read-only solve
  → optional measured verification
  → reproducible evidence

LLMRig is Qwen-first today. Curated setup uses Ollama. Local race measurement also supports llama.cpp and MLX-LM when compatible artifacts and runtimes are already present; LLMRig does not install those native runtimes or download their artifacts.

Install

pipx is recommended for the CLI:

pipx install llmrig
pipx upgrade llmrig
llmrig --version

If pipx is unavailable, use a virtual environment. This also avoids PEP 668 errors from system-managed Python installations.

python3 -m venv .venv
source .venv/bin/activate  # macOS/Linux
python -m pip install llmrig

On Windows PowerShell:

python -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install llmrig

LLMRig supports Python 3.9+ on macOS, Linux, and Windows and has no third-party Python runtime dependencies. It can also run directly from a source checkout with python3 llmrig.py.

Solve first

Analyze a curated model or exact owner/repository Hugging Face identifier:

llmrig solve MODEL
llmrig solve MODEL --json
llmrig solve MODEL --context 32768
llmrig solve MODEL --local-artifact llama.cpp=/path/to/model.gguf
llmrig solve MODEL --local-artifact mlx-lm=/path/to/model-directory

Hugging Face resolution reads metadata and file listings but does not download weights. Native artifacts are inspected only at paths explicitly supplied with --local-artifact; LLMRig does not scan arbitrary filesystem locations.

Solve reports these evidence dimensions independently:

compatible != local
local != executable
executable != measurable
measurable != measured
measured != recommended

Unknown is not false. An observed Ollama name is local evidence, not independent content attestation. A native model association is supplied by the user: LLMRig checks its local structure but does not attest its identity, weights, quality, or quantization. Private native paths stay out of public solve results.

Solve does not calculate a universal score, infer quality from throughput, or promise a winner. It recommends a planning candidate only when the available evidence supports exactly one runnable configuration with no comparable unresolved alternative. An inconclusive result is valid.

Verify explicitly

llmrig solve MODEL \
  --local-artifact llama.cpp=/path/to/model.gguf \
  --verify

solve --verify executes only compatible, already-local, executable, measurable candidates. It requires at least two comparable configurations and reuses the unchanged race-v2 workload, two-sample rule, 5% comparison threshold, and balanced Pareto decision. It does not pull models, install or start runtimes, scan for artifacts, or write passports. Fewer than two candidates makes verification unavailable without executing inference; a competitor failure invalidates the comparison; multiple Pareto members remain inconclusive.

Solve exit codes are:

• 0: analysis completed, including a completed but inconclusive verification
• 1: an operational solve or attempted verification failure
• 2: invalid/unresolvable input or unavailable requested verification

Minimal Python SDK

import llmrig

result = llmrig.solve("qwen3:0.6b")
print(result.plan.recommendation_status)

The stable entry point is:

llmrig.solve(
    model,
    *,
    context=None,
    local_artifacts=(),
    verify=False,
) -> llmrig.SolveResult

It prints nothing and never exits the process. local_artifacts accepts RUNTIME=PATH strings. Invalid or unresolvable input raises SolveInputError; operational failures raise SolveEngineError. SolveResult and SolveCandidate are the deliberate public result contracts. _llmrig is private implementation.

Measurement commands

The measurement commands share local execution machinery but answer different questions:

Race two already-local native configurations, or combine one with an installed equivalent Ollama build:

llmrig race MODEL \
  --local-artifact llama.cpp=/path/to/model.gguf \
  --local-artifact mlx-lm=/path/to/model-directory

race never installs or downloads. At most one explicit artifact per native runtime is accepted. A GGUF target must be a non-empty .gguf file. An MLX-LM target must have an immediate non-empty config.json and at least one immediate non-empty model*.safetensors file. These structural checks do not prove the runtime can load the artifact or that differently packaged artifacts have identical weights or quality.

Race exit 0 means at least two competitors were measured successfully, 1 means execution failed and invalidated the comparison, and 2 means the race is unavailable or unresolved. Results within 5% are inconclusive. There is no composite result or model-quality claim.

Choose an explicit objective from the same measured race path:

llmrig choose MODEL --objective generation
llmrig choose MODEL --objective prompt --json
llmrig choose MODEL --objective latency
llmrig choose MODEL --objective balanced

The default balanced objective recommends only a unique measured-performance Pareto leader. choose exit 0 means a recommendation exists, 1 means race execution failed, and 2 means the decision is unavailable or inconclusive.

Inspect the unranked frontier directly:

llmrig optimize MODEL
llmrig optimize MODEL --json

The active dimensions are generation throughput, prompt-evaluation throughput, and normalized inference latency. A dimension is omitted globally if any successful competitor lacks two positive finite samples. Missing data is never converted to zero. The frontier measures performance only; it does not infer quality, accuracy, or reasoning.

Run the Ollama-specific full benchmark:

llmrig bench --model qwen3.8:27b-mlx --context 32768 --runs 2
llmrig bench --all-installed --context 32768 --runs 2
llmrig bench --model qwen3.8:27b-mlx --passport benchmark.passport.json

bench unloads resident Ollama models before measurement and unloads its target afterward. It records throughput, Ollama-reported context, accelerator residency when available, memory snapshots, and three lightweight correctness smoke tests. Reports default to the ignored benchmarks/ directory. They are local configuration checks, not academic quality benchmarks, and memory readings are snapshots rather than peak measurements.

Benchmark passports

A passport is a versioned, privacy-filtered JSON record of one measured execution configuration. Export one with bench --passport FILE or one per successfully measured competitor with race --passport-dir DIR.

llmrig passport verify benchmark.passport.json

Verification is offline and read-only. It checks schema, SHA-256 identifiers and configuration fingerprints, aggregates, impossible states, and known privacy constraints. These hashes are deterministic identity and integrity checks—not signatures, independent attestations, benchmark certification, or proof that a claimed measurement is true. User-supplied native artifact IDs are path-independent and do not attest file contents.

Passports are exact when configuration fingerprints match. comparable_with_warnings means the logical model and workload match but artifact, format, quantization, runtime, runtime version, or hardware differs. not_comparable means the logical model or workload differs. None of these labels ranks results. A failed race exports no competitor passports.

Other commands

Inspect hardware and Ollama readiness:

llmrig doctor
llmrig doctor --json

Check three-state compatibility for a curated ID or exact Hugging Face repository:

llmrig can qwen3.8:27b-mlx
llmrig can owner/repository --json

can exits 0 for compatible, 1 for incompatible, and 2 for unknown or unresolvable. Memory fit is a conservative planning estimate, not a prediction of performance.

Inspect the Qwen-first catalog:

llmrig models --offline --fit
llmrig models --fit
llmrig models --refresh --fit
llmrig models --all --fit

The curated snapshot is the only layer eligible for automatic setup. Live discovery is informational and cannot establish package size, fit, runtime compatibility, or installation trust from a repository name.

Use the curated recommendation/setup workflow when desired:

llmrig recommend --category official --preference balanced
llmrig setup --model qwen3.8:27b-mlx --context 32768

setup is mutating: after confirmation it may pull a curated Ollama artifact and run the full benchmark. Running bare llmrig starts the interactive version of that workflow. LLMRig does not install Ollama itself.

Run offline project checks, or explicitly include live discovery:

llmrig check
llmrig check --online

Evidence and scope

LLMRig models compatibility as:

hardware × model × artifact × quantization × runtime × context × measured performance

Important boundaries:

• verified, inferred, measured, estimated, and unknown are not interchangeable
• discovery does not establish installation trust
• local does not imply executable, measurable, measured, or recommended
• benchmark evidence outranks planning heuristics
• throughput does not establish model quality
• there is no universal model/runtime score
• prediction and prediction-versus-measurement calibration are not implemented yet

Official curated entries are upstream Qwen models distributed through the selected local backend. Community reduced-refusal entries are third-party derivatives. Their behavior does not imply better reasoning, accuracy, safety, or reliability; review their provenance and licensing before use.

Development

python3 -m compileall -q llmrig.py _llmrig
python3 -m unittest discover -s tests -v
python3 llmrig.py --version
python3 llmrig.py solve --help
python3 llmrig.py check

See CONTRIBUTING.md for architecture and evidence invariants and RELEASING.md for the release sequence.

Repository layout

llmrig/
├── .github/
│   ├── ISSUE_TEMPLATE/
│   ├── dependabot.yml
│   ├── PULL_REQUEST_TEMPLATE.md
│   └── workflows/
│       ├── ci.yml
│       ├── codeql.yml
│       └── publish-to-pypi.yml
├── _llmrig/                 # private implementation package
├── assets/llmrig-terminal.svg
├── tests/
├── CHANGELOG.md
├── CITATION.cff
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── LICENSE
├── README.md
├── RELEASING.md
├── SECURITY.md
├── llmrig.py                # CLI and small public SDK facade
└── pyproject.toml

Roadmap

Future work may include broader model-family coverage, a RigGraph representation, privacy-preserving community benchmark evidence, prediction and calibration, and stable runtime adapter/plugin interfaces. These are directions, not current product claims. New functionality must preserve provenance and must not turn discovery metadata into automatic installation trust.

License and references

LLMRig is released under the MIT License.

• Qwen on Hugging Face
• Qwen3.8 repository
• Hugging Face Hub API
• Ollama documentation
• Ollama generate API
• Ollama running-model API