clef-router 0.2.0

Route prompts with Cloudflare's Clef decision model.
Send each prompt to the affordable model when it is enough, and to the frontier model when it matters.

Get started · How it works · Benchmarks · clef vs laya · Limitations

Quickstart

pip install clef-router
export CLEF_ACCOUNT_ID=your_cloudflare_account_id
export CLEF_API_TOKEN=your_cloudflare_api_token

Run it as a proxy and point any OpenAI client at it:

clef-router --port 8000

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed")

completion = client.chat.completions.create(
    model="auto",  # the router chooses; your value is ignored
    messages=[{"role": "user", "content": "Say hi in three words"}],
)
print(completion.choices[0].message.content)  # the routing decision as JSON
print(completion.extensions["clef"]["tier"])  # "cheap" or "frontier"

The proxy is a decision service. It returns the routing decision (tier, confidence, reason, token usage) in an OpenAI-shaped response, and your client calls the chosen model, exactly like laya-router's decision headers.

Prefer in-process? The library and the CLI do the same thing without HTTP:

from clef_router import ClefRouter

with ClefRouter() as router:
    routing = router.route("Design a rate limiter")
print(routing.tier)            # "frontier"
print(routing.reason)          # "team selected frontier"
print(routing.decision.usage.input_tokens)

clef-route "Design a rate limiter"
# tier:       frontier
# reason:     team selected frontier
# model:      clef-flash

Async clients, custom question sets, and raw decisions are one import away:

from clef_router import AsyncClefRouter

async with AsyncClefRouter() as router:
    decision = await router.decide({"invoice_total": 1250, "due_days": 45})

How it works

   your app (OpenAI SDK)          your app (library / CLI)
          |                               |
          v                               v
   +---------------------+      +----------------+
   | clef-router proxy   |      | clef_router    |
   | POST /v1/chat/...   |      | ClefRouter     |
   | POST /v1/decide     |      |                |
   +----------+----------+      +-------+--------+
              |                          |
              +------------+-------------+
                           v
        +--------------------------------------+
        |  Clef decision model  (one call)     |
        |  questions: team, urgency            |
        |  answers:   probabilities per option |
        +------------------+-------------------+
                           |
        team=cheap & conf>=0.45 --> CHEAP tier
        otherwise (low confidence, urgent,
        unknown answer)----------> FRONTIER tier

One forward pass scores every option of every question, so a decision costs a single small request. The policy is fail-safe on purpose:

1. When the team answer names a tier, that tier wins, unless its confidence is below min_confidence (default 0.45), which escalates to frontier.
2. With no usable team answer, the urgency probability routes: 0.5 or higher goes frontier.
3. Anything the router cannot parse escalates. It never silently picks the cheap tier without evidence.

Benchmarks

Two different things get measured, and mixing them up is the oldest trick in routing marketing. We keep them separate.

What we measured (this repo, evals/run_eval.py, committed results):

What Cloudflare measured (Decision Index 0.2.1, from the model card; that suite scores the decision model itself, not this router):

Reproduce our numbers offline (no credentials, seconds to run):

python evals/run_eval.py                          # writes evals/results/routing-eval-mock.json
python evals/run_eval.py --mode api               # real end-to-end numbers; needs credentials

The repository ships a regression gate so a routing change cannot silently drop accuracy:

- uses: ./.github/actions/regression-gate
  with:
    current: evals/results/current.json
    baseline: evals/results/baselines/routing-eval-mock.json
    metrics: |
      metrics.accuracy:max
      metrics.escalations.escalation_precision:max:0.05

clef vs laya

Honest reading: laya wins on latency because it runs on your machine. clef-router wins on decision quality, which shows up as fewer wrong routes on genuinely hard prompts. If your workload is latency-sensitive at the routing step and mostly easy prompts, laya is a great fit. If misroutes cost more than 30 ms, use clef-router.

Context compaction

clef_router.compactor uses the same one-pass trick for RAG: one Clef request scores the relevance of up to 64 retrieved documents, then a token budget decides what survives. Documents are kept verbatim or cut with a reason; nothing is rewritten.

from clef_router.compactor import compact

result = compact(
    "invoice processing rules",
    documents=retrieved_chunks,
    budget=2048,
)
print(result.stats.savings_pct)   # e.g. 61.3
print(result.kept[0].score)       # 0..3 relevance
print(result.cut[2].reason)       # "score 0 below min_score 1"

Framework adapters ship as extras (pip install "clef-router[compactor]"):

from clef_router.compactor.langchain import ClefDocumentCompressor   # LangChain
from clef_router.compactor.llamaindex import ClefNodePostprocessor   # LlamaIndex

Configuration

Environment variables, with CLEF_* winning over CLOUDFLARE_* fallbacks:

Retries are structured: timeouts and connection errors retry, HTTP 401/403 raise ClefAuthError immediately, 429 raises ClefRateLimitError with the server's Retry-After once the budget is spent. Every HTTP-derived error carries the status code, the Cloudflare error code, and the cf-ray request id when present.

Limitations

Stated plainly, because routing libraries that hide these waste your time.

• The proxy decides; it does not complete. It never forwards your prompt to a chat model. Clients that want the answer call the chosen model themselves.
• No streaming. POST /v1/chat/completions is request/response. SSE pass-through is on the roadmap.
• The committed accuracy number is policy accuracy on 42 labeled fixtures, not a model benchmark. Use --mode api with real credentials for end-to-end numbers on your own workload, or run the Decision Index suite for model quality.
• The tier mapping reads the team and urgency question ids. Custom question sets work with decide(); route() still expects those ids.
• Latency depends on your network to Cloudflare. The 38.8 ms median is Cloudflare's measurement; add your round trip.
• Images are accepted by the API (decide(), max 4) but the routing question set is text-only today.
• Running Cloudflare/clef weights locally is possible (Apache-2.0) but the model card lists a single H200 as the tested environment; the Kaggle kernel in evals/kaggle-kernel/ measures policy overhead on CPU instead.

Reproduce on Kaggle

evals/kaggle-kernel/ holds a kernel that clones this repository and reruns the offline evaluation on clean Kaggle hardware, writing /kaggle/working/routing-eval.json:

KAGGLE_API_TOKEN=... python -m kaggle kernels push -p evals/kaggle-kernel

Development

git clone https://github.com/Gjusev/clef-router.git && cd clef-router
pip install -e ".[dev]"
make test          # offline suite, integration deselected
make lint          # ruff
make eval          # regenerate evals/results/routing-eval-mock.json
make assets        # regenerate logo, social card, diagrams, demo video
pytest -m integration   # live API tests; needs credentials

src/clef_router/     client (sync+async), config, errors, models, transport
src/clef_router/     server (FastAPI proxy), compat (in-process OpenAI)
src/clef_router/     compactor (core + LangChain/LlamaIndex adapters)
evals/               dataset, runner, committed results, Kaggle kernel
scripts/             asset generation, regression check CLI
tests/               offline suite; @pytest.mark.integration opts into the live API

License

Apache-2.0. See LICENSE. Clef is a Cloudflare model; its weights carry the same license on the model page.