pydecide

One Python client for typed decisions, with a confidence gate between the cheap model and the good one.

Routing, triage and guardrail calls do not need a chat model. A decision model takes a state and typed questions (pick one of these, score this, yes or no) and answers with probabilities, in milliseconds. pydecide speaks to all of them through one API: TypeSafe's hosted Jev, OpenRouter, OpenAI's Decisions endpoint, the open-weight laya family on PyTorch and MLX, any sentence-transformers CrossEncoder, and a JSON-prompted LLM as a last resort.

The reason to use it over any single backend's SDK is the chain. A Client takes an ordered list of backends and a confidence policy. A backend that errors, or that answers below its threshold, hands the question to the next one, so a local model carries the traffic and a hosted model sees only what the local one could not answer. Every response records the route it took.

Install

uv tool install pydecide   # the decide CLI on your PATH, from any directory
uv add pydecide            # inside a project

With pip: pip install pydecide / pipx install pydecide. That is the CLI, the hosted backends and one dependency, httpx. decide serve additionally needs the server extra.

Local model backends need an extra:

uv tool install "pydecide[all]"

With pip: pip install "pydecide[all]". Quote the brackets in zsh, since it otherwise tries to glob them. This pulls in PyTorch and MLX, about 2 GB.

The mlx extra depends on laya-mlx, which requires Python 3.11 or newer; on 3.10 the extra installs nothing and the laya_mlx backend is unavailable.

Quickstart: zero configuration

Install a local backend and decide ask works right away, with no environment variables at all:

uv tool install "pydecide[mlx]"    # Apple Silicon
# or
uv tool install "pydecide[laya]"   # any platform (PyTorch)

decide ask "I had a rough day, everything broke" --noul "Is the writer doing well?" --choice good,bad

NAME    TYPE    ANSWER  PROBABILITIES
choice  choice  bad     good=0.01 bad=0.99
noul    noul    false   0.01
backend=laya_mlx route=laya_mlx:ok latency=56.5ms

This is the real output of that command, run with laya_mlx installed and none of DECIDE_LOCAL_MODEL, TYPESAFE_API_KEY, OPENROUTER_API_KEY, OPENAI_API_KEY, DECIDE_LLM_BASE_URL or DECIDE_BACKENDS set.

Run decide init to interactively save an explicit backend chain and confidence threshold in decide.json. Both the CLI and Python client read that file from the current directory. Setup never saves credentials or loads models.

The Python client works the same way. Client.from_env() reads the saved configuration, or discovers an installed local backend when there is no file. API keys alone never select a hosted backend. Pass an explicit env= mapping instead (e.g. in tests) to configure from something other than os.environ:

from decide import Client, Choice, Score, Noul

client = Client.from_env()

r = client.decide(
    state={"ticket": "I was charged twice, please refund."},
    questions={
        "team": Choice(
            "Which team handles this?",
            {
                "billing": "charges, refunds",
                "engineering": "bugs and outages",
                "sales": "pricing and upgrades",
            },
        ),
        "severity": Score("How severe is this?", ["minor", "degraded", "blocked"]),
        "refund": Noul("Does the customer ask for money back?"),
    },
)

r.choices["team"].choice  # "billing"
r.choices["team"].probabilities  # {"billing": 0.9763, "engineering": 0.0107, "sales": 0.013}
r.scores["severity"].score  # 1.6284  (expected level index, 0..len(levels)-1)
r.scores["severity"].probabilities  # [0.0802, 0.2112, 0.7086]
r.nouls["refund"].noul  # 0.8919
r.meta.backend  # "laya_mlx"
r.meta.latency_ms  # 50.0
r.meta.route  # ["laya_mlx:ok"]

This was run against the laya_mlx backend with no environment variables set at all, using its own default checkpoint; every value above is the real output of that run, not illustrative.

Python accepts the same configuration directly or by file:

from decide import Client, Config

client = Client(config="decide.json")
# Or configure in code:
client = Client(
    config=Config(
        backends=["laya_mlx", "typesafe"],
        min_confidence=0.85,
    )
)

That chain explicitly allows hosted fallback to TypeSafe; provide its credential through TYPESAFE_API_KEY. Without a config, only laya_mlx (preferred) or laya is discovered. DECIDE_BACKENDS remains an explicit override for from_env and the CLI. DECIDE_LOCAL_ONLY=1 refuses server backends even when explicitly selected. With no configured backend or local installation, the client raises ConfigError with setup instructions.

See Configuration for the file format and precedence rules.

AsyncClient has the same surface, awaited: await client.decide(...), await client.decide_batch(...), AsyncClient.from_env(...).

Documentation

Latency

Measured on an Apple M4 (32 GB) with the laya_mlx backend (laya-mlx 0.4.0) on its default checkpoint, aac6fef/laya-mlx, in process, after three warm-up requests. One Client.decide call, timed by meta.latency_ms, 20 samples:

Loading the model and answering the first request took 1,304 ms, once. A decide CLI invocation on the same machine takes 0.7 to 1.3 s of wall clock, most of it interpreter startup and loading the model again, while the decision inside it measures 56 to 144 ms. Measure in process if you are comparing against a hosted API.

Your numbers will differ with the machine, the checkpoint and the length of the state, so measure before you rely on any of this. --verbose prints the backend, the route and the latency of the call:

decide ask "I was charged twice" --choice billing,engineering --verbose

NAME    TYPE    ANSWER   PROBABILITIES
choice  choice  billing  billing=0.99 engineering=0.01
backend=laya_mlx route=laya_mlx:ok latency=45.4ms

That one is a cold call with no warm-up, which is why it is slower than the table above.

What the probabilities mean

Every probability in a ChoiceAnswer, ScoreAnswer or NoulAnswer is whatever the backend reported; decide does not calibrate, smooth or verify it. What that means differs by backend: typesafe, openrouter, openai_decisions and laya/laya_mlx are purpose-built decision models, but their outputs are still self-reported by the model and not audited by this library. The crossencoder backend turns a relevance reranker's raw logits into a softmax or sigmoid, so the result is a normalized score forced to distribute mass over the supplied candidates, not a calibrated probability; a Choice will still pick a winner even when every candidate is a bad fit, and a Noul of 0.9 does not mean the condition holds 90% of the time. The llm backend is the least trustworthy of all: it prompts a general chat model to estimate its own confidence in JSON, with no guarantee the model attends to every candidate or keeps its numbers well calibrated. Treat all of this accordingly, as a signal to gate and fall back on, not as ground truth.

Status

The public API may still change before a 1.0 release. The current version is the one in the PyPI badge above, and the changelog says what changed.

License

MIT, see LICENSE.