clef-router 0.3.0

Route prompts with Cloudflare's Clef decision model.
Send each prompt to the affordable model when it is enough, and to the frontier model when it matters.

Get started · How it works · Benchmarks · clef vs laya · Limitations

Quickstart

pip install clef-router
export CLEF_ACCOUNT_ID=your_cloudflare_account_id
export CLEF_API_TOKEN=your_cloudflare_api_token

Run it as a proxy and point any OpenAI client at it:

clef-router --port 8000

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed")

completion = client.chat.completions.create(
    model="auto",  # the router chooses; your value is ignored
    messages=[{"role": "user", "content": "Say hi in three words"}],
)
print(completion.choices[0].message.content)  # the routing decision as JSON
print(completion.extensions["clef"]["tier"])  # "cheap" or "frontier"

The proxy is a decision service. It returns the routing decision (tier, confidence, reason, token usage) in an OpenAI-shaped response, and your client calls the chosen model, exactly like laya-router's decision headers.

Prefer in-process? The library and the CLI do the same thing without HTTP:

from clef_router import ClefRouter

with ClefRouter() as router:
    routing = router.route("Design a rate limiter")
print(routing.tier)            # "frontier"
print(routing.reason)          # "team selected frontier"
print(routing.decision.usage.input_tokens)

clef-route "Design a rate limiter"
# tier:       frontier
# reason:     team selected frontier
# model:      clef-flash

Async clients, custom question sets, and raw decisions are one import away:

from clef_router import AsyncClefRouter

async with AsyncClefRouter() as router:
    decision = await router.decide({"invoice_total": 1250, "due_days": 45})

How it works

   your app (OpenAI SDK)          your app (library / CLI)
          |                               |
          v                               v
   +---------------------+      +----------------+
   | clef-router proxy   |      | clef_router    |
   | POST /v1/chat/...   |      | ClefRouter     |
   | POST /v1/decide     |      |                |
   +----------+----------+      +-------+--------+
              |                          |
              +------------+-------------+
                           v
        +--------------------------------------+
        |  Clef decision model  (one call)     |
        |  questions: team, urgency            |
        |  answers:   probabilities per option |
        +------------------+-------------------+
                           |
        team=cheap & conf>=0.45 --> CHEAP tier
        otherwise (low confidence, urgent,
        unknown answer)----------> FRONTIER tier

One forward pass scores every option of every question, so a decision costs a single small request. The policy is fail-safe on purpose:

1. When the team answer names a tier, that tier wins, unless its confidence is below min_confidence (default 0.45), which escalates to frontier.
2. With no usable team answer, the urgency probability routes: 0.5 or higher goes frontier.
3. Anything the router cannot parse escalates. It never silently picks the cheap tier without evidence.

Benchmarks

Two different things get measured, and mixing them up is the oldest trick in routing marketing. We keep them separate.

What we measured (this repo, evals/run_eval.py, committed results):

What Cloudflare measured (Decision Index 0.2.1, from the model card; that suite scores the decision model itself, not this router):

Reproduce our numbers offline (no credentials, seconds to run):

python evals/run_eval.py                          # writes evals/results/routing-eval-mock.json
python evals/run_eval.py --mode api               # real end-to-end numbers; needs credentials

The repository ships a regression gate so a routing change cannot silently drop accuracy:

- uses: ./.github/actions/regression-gate
  with:
    current: evals/results/current.json
    baseline: evals/results/baselines/routing-eval-mock.json
    metrics: |
      metrics.accuracy:max
      metrics.escalations.escalation_precision:max:0.05

clef vs laya

Honest reading: laya wins on latency because it runs on your machine. clef-router wins on decision quality, which shows up as fewer wrong routes on genuinely hard prompts. If your workload is latency-sensitive at the routing step and mostly easy prompts, laya is a great fit. If misroutes cost more than 30 ms, use clef-router.

Context compaction

clef_router.compactor uses the same one-pass trick for RAG: one Clef request scores the relevance of up to 64 retrieved documents, then a token budget decides what survives. Documents are kept verbatim or cut with a reason; nothing is rewritten.

from clef_router.compactor import compact

result = compact(
    "invoice processing rules",
    documents=retrieved_chunks,
    budget=2048,
)
print(result.stats.savings_pct)   # e.g. 61.3
print(result.kept[0].score)       # 0..3 relevance
print(result.cut[2].reason)       # "score 0 below min_score 1"

Framework adapters ship as extras (pip install "clef-router[compactor]"):

from clef_router.compactor.langchain import ClefDocumentCompressor   # LangChain
from clef_router.compactor.llamaindex import ClefNodePostprocessor   # LlamaIndex

Configuration

Environment variables, with CLEF_* winning over CLOUDFLARE_* fallbacks:

Retries are structured: timeouts and connection errors retry, HTTP 401/403 raise ClefAuthError immediately, 429 raises ClefRateLimitError with the server's Retry-After once the budget is spent. Every HTTP-derived error carries the status code, the Cloudflare error code, and the cf-ray request id when present.

Operate the proxy

docker build -t clef-router .
docker run -p 8000:8000 -e CLEF_ACCOUNT_ID=... -e CLEF_API_TOKEN=... clef-router

• "stream": true works: the decision arrives as OpenAI-shaped SSE chunks (role, decision JSON, stop, [DONE]), and the clef extension rides in the first chunk.
• GET /metrics serves Prometheus text: clef_router_requests_total by tier and status, plus a clef_router_routing_seconds histogram.
• CLEF_DECISION_LOG=/path/decisions.jsonl appends one JSON line per decision (tier, confidence, reason, latency, token usage, prompt preview) for post-hoc calibration of the confidence gate.

Limitations

Stated plainly, because routing libraries that hide these waste your time.

• The proxy decides; it does not complete. It never forwards your prompt to a chat model. Clients that want the answer call the chosen model themselves.
• The committed policy number is accuracy on 42 labeled fixtures. The GPU rerun measures the real model on the same 42 prompts (92.9%), which is still a small set: neither number is a Decision-Index-grade benchmark. Use --mode api with real credentials to measure your own workload.
• The tier mapping reads the team and urgency question ids. Custom question sets work with decide(); route() still expects those ids.
• Latency depends on your network to Cloudflare. The 38.8 ms median is Cloudflare's measurement; add your round trip.
• Images are accepted by the API (decide(), max 4) but the routing question set is text-only today.
• Running Cloudflare/clef-flash weights locally is possible (Apache-2.0, 9.4B) and is exactly what evals/kaggle-kernel-gpu/ does on one T4 in 4-bit; expect ~500 ms per decision there versus 38.8 ms median on Cloudflare's unquantized datacenter GPUs.

Reproduce on Kaggle

Two public kernels rerun the evaluation on Kaggle hardware:

• evals/kaggle-kernel-gpu/ — GPU (T4): downloads the real clef-flash weights, routes every labeled prompt through the model, and writes routing-gpu-eval.json. Needs a GPU-verified account; about 15 minutes and a slice of your 30 h weekly quota.
• evals/kaggle-kernel/ — CPU: replays the committed fixtures through the policy pipeline in about two minutes, no GPU or credentials.

KAGGLE_API_TOKEN=... python -m kaggle kernels push -p evals/kaggle-kernel-gpu

Development

git clone https://github.com/Gjusev/clef-router.git && cd clef-router
pip install -e ".[dev]"
make test          # offline suite, integration deselected
make lint          # ruff
make eval          # regenerate evals/results/routing-eval-mock.json
make assets        # regenerate logo, social card, diagrams, demo video
pytest -m integration   # live API tests; needs credentials

src/clef_router/     client (sync+async), config, errors, models, transport
src/clef_router/     server (FastAPI proxy), compat (in-process OpenAI)
src/clef_router/     compactor (core + LangChain/LlamaIndex adapters)
evals/               dataset, runner, committed results, Kaggle kernel
scripts/             asset generation, regression check CLI
tests/               offline suite; @pytest.mark.integration opts into the live API

License

Apache-2.0. See LICENSE. Clef is a Cloudflare model; its weights carry the same license on the model page.