CRE-Router
Implementation of the paper Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving.
Production LLM serving trades accuracy against cost. CRE-Router routes each query to the most cost-effective model in a pool, then escalates low-quality outputs to a stronger model:
- Stage 1 (clustering-based routing). Queries are embedded and clustered offline; each cluster is routed to the model that minimizes a cost-adjusted error score (Error + lambda * Cost), with lambda tuned once to satisfy a latency (TPOT) budget.
- Stage 2 (quality-estimation cascade). A lightweight ModernBERT classifier inspects each efficient-model output and escalates low-quality answers up an ordered ladder of stronger models.
Both stages train only on task-correctness labels obtainable from standard benchmark evaluation; no extra annotation is required.
Install
pip install "cre-router[full]"
full installs the whole pipeline on one machine. Narrower installs are
available through the serve, qe, and eval extras; see the
installation guide.
Usage
The workflow is driven by the cre CLI, one stage per step:
cre cluster → cre evaluate → cre fit → cre qe-train → cre serve
Run cre <stage> --help for options. Runnable end-to-end quickstarts (a
no-GPU routing-table demo and a full serving walkthrough) are in the
README.
Links
- Source and documentation: https://github.com/ymoslem/CRE-Router
- Paper: https://arxiv.org/abs/2606.27457
- Reproducing the paper: https://github.com/ymoslem/CRE-Router/blob/main/REPRODUCE.md
Apache-2.0 licensed.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file cre_router-0.2.1.tar.gz.
File metadata
- Download URL: cre_router-0.2.1.tar.gz
- Upload date:
- Size: 83.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
2bd88733e1b2ff642fd3794c1168fc1c2b76824c644f7eb2af9b41b3fdef0237
|
|
| MD5 |
217c3fcf45a4e7994bb8d9dec66d64ef
|
|
| BLAKE2b-256 |
7e5135e1e35c4568ed1d674fd0952d3fa8c9696ea5c26d7f016ef239d7642533
|
Provenance
The following attestation bundles were made for cre_router-0.2.1.tar.gz:
Publisher:
release.yml on ymoslem/CRE-Router
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
cre_router-0.2.1.tar.gz -
Subject digest:
2bd88733e1b2ff642fd3794c1168fc1c2b76824c644f7eb2af9b41b3fdef0237 - Sigstore transparency entry: 2467271655
- Sigstore integration time:
-
Permalink:
ymoslem/CRE-Router@7e6277bc7d1f64e5af562db1b9fa77886ca1aca9 -
Branch / Tag:
refs/tags/v0.2.1 - Owner: https://github.com/ymoslem
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@7e6277bc7d1f64e5af562db1b9fa77886ca1aca9 -
Trigger Event:
push
-
Statement type:
File details
Details for the file cre_router-0.2.1-py3-none-any.whl.
File metadata
- Download URL: cre_router-0.2.1-py3-none-any.whl
- Upload date:
- Size: 53.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
9f98c0a092818dfe3a713e5ef8a6a081951d2772592ac636bf58018ed04c6f11
|
|
| MD5 |
67a8f2e87359693d2cabfa79e0ea8c15
|
|
| BLAKE2b-256 |
015bbdd82da6946f5200b04c88205660246fce4afaf9cd1fbc6ef8fb111fe9a4
|
Provenance
The following attestation bundles were made for cre_router-0.2.1-py3-none-any.whl:
Publisher:
release.yml on ymoslem/CRE-Router
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
cre_router-0.2.1-py3-none-any.whl -
Subject digest:
9f98c0a092818dfe3a713e5ef8a6a081951d2772592ac636bf58018ed04c6f11 - Sigstore transparency entry: 2467271659
- Sigstore integration time:
-
Permalink:
ymoslem/CRE-Router@7e6277bc7d1f64e5af562db1b9fa77886ca1aca9 -
Branch / Tag:
refs/tags/v0.2.1 - Owner: https://github.com/ymoslem
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@7e6277bc7d1f64e5af562db1b9fa77886ca1aca9 -
Trigger Event:
push
-
Statement type:
