pydecide

decide is one Python client for typed decisions (Choice, Score and Noul) over any "System One" decision model: TypeSafe's hosted Jev, OpenRouter's Decisions endpoint, the open-weight laya family (PyTorch and MLX), any sentence-transformers CrossEncoder, and a JSON-prompted LLM fallback. The core of the library is the fallback chain: a Client takes an ordered list of these backends and a confidence policy, and if a backend errors or answers with low confidence, the next backend in the chain is tried, so a cheap local model can back off to a hosted one only when it's unsure.

The library also ships a small HTTP server (see Server below) that speaks TypeSafe's wire protocol. laya now ships its own laya-serve as of 0.3.7, so this server isn't the reason to reach for decide if you only run laya. Its purpose is chaining several backends (local and hosted) behind one TypeSafe-compatible endpoint.

Already using typesafe-sdk?

Swap the import. Your questions, your system_one calls and the response object stay the same, and with no other change the hosted API is the only backend, exactly as before.

from decide import TypeSafeClient  # was: from typesafe_sdk import TypeSafeClient

Add fallback= and the call keeps answering when the hosted API does not. Here the hosted tier is a transport that always returns 503, so the local model takes over:

import httpx
from decide import TypeSafeClient
from typesafe_sdk import Choice

hosted_down = httpx.MockTransport(lambda request: httpx.Response(503, text="down"))
client = TypeSafeClient(api_key="sk-demo", transport=hosted_down, fallback=["laya_mlx"])
result = client.system_one(
    "I was charged twice for the same order, please refund the duplicate.",
    {
        "team": Choice(
            instructions="Which team owns this?", criteria={"billing": None, "engineering": None}
        )
    },
)
print(result.choices["team"].choice, round(result.choices["team"].confidence, 3))
print(result.decide.backend, result.decide.model)
print(result.decide.route)

billing 0.99
laya_mlx aac6fef/laya-multilingual-mlx
['typesafe:error', 'laya_mlx:ok']

The result is a subclass of the SDK's own SystemOneResponse, so existing code that reads answers, choices or usage keeps working; result.decide adds the backend, model, latency and route. Questions can be the SDK's Choice, Score and Noul, decide's own, or a mix. typesafe-sdk is not a dependency of pydecide; install it yourself to use this layer. AsyncTypeSafeClient works the same way.

Errors keep the vendor types: a failed hosted call raises the matching TypeSafeAPIError subclass (TypeSafeInternalServerError for a 503, TypeSafeAPIConnectionError for a transport failure, and so on), so existing except clauses still catch it. With a fallback and every tier failing, the error is both a TypeSafeError and a decide.AllBackendsFailed.

retry= takes a typesafe_sdk.RetryPolicy and retries the hosted tier before the chain moves on. It honours max_retries, backoff_initial, backoff_max, backoff_jitter, http_statuses, api_connection_error and api_timeout_error; it ignores respect_retry_after, exceptions, predicate and timeout. Not supported yet: response_model=, http_client=, and per-call retry=, extra_headers=, extra_body= and timeout= (each raises NotImplementedError).

Install

uv tool install pydecide   # the decide CLI on your PATH, from any directory
uv add pydecide            # inside a project

With pip: pip install pydecide / pipx install pydecide. Either way, this includes the CLI, the hosted backends and the server.

Local model backends need an extra:

uv tool install "pydecide[all]"

With pip: pip install "pydecide[all]". Quote the brackets in zsh, since it otherwise tries to glob them. This pulls in PyTorch and MLX, about 2 GB.

The mlx extra depends on laya-mlx, which requires Python 3.11 or newer; on 3.10 the extra installs nothing and the laya_mlx backend is unavailable.

Quickstart: zero configuration

Install a local backend and decide ask works right away, with no environment variables at all:

uv tool install "pydecide[mlx]"    # Apple Silicon
# or
uv tool install "pydecide[laya]"   # any platform (PyTorch)

decide ask "I had a rough day, everything broke" --noul "Is the writer doing well?" --choice good,bad

NAME    TYPE    ANSWER  PROBABILITIES
choice  choice  bad     good=0.01 bad=0.99
noul    noul    false   0.01
backend=laya_mlx route=laya_mlx:ok latency=33.1ms

This is the real output of that command, run with laya_mlx installed and none of DECIDE_LOCAL_MODEL, TYPESAFE_API_KEY, OPENROUTER_API_KEY, OPENAI_API_KEY, DECIDE_LLM_BASE_URL or DECIDE_BACKENDS set.

The Python client works the same way. Client.from_env() builds a backend chain from whatever is installed and configured in the process environment; with a local backend installed, it needs no variables either. Pass an explicit env= mapping instead (e.g. in tests) to configure from something other than os.environ:

from decide import Client, Choice, Score, Noul

client = Client.from_env()

r = client.decide(
    state={"ticket": "I was charged twice, please refund."},
    questions={
        "team": Choice(
            "Which team handles this?",
            {
                "billing": "charges, refunds",
                "engineering": "bugs and outages",
                "sales": "pricing and upgrades",
            },
        ),
        "severity": Score("How severe is this?", ["minor", "degraded", "blocked"]),
        "refund": Noul("Does the customer ask for money back?"),
    },
)

r.choices["team"].choice  # "billing"
r.choices["team"].probabilities  # {"billing": 1.0, "engineering": 0.0, "sales": 0.0}
r.scores["severity"].score  # 1.5522  (expected level index, 0..len(levels)-1)
r.scores["severity"].probabilities  # [0.0197, 0.4083, 0.5719]
r.nouls["refund"].noul  # 0.9461
r.meta.backend  # "laya_mlx"
r.meta.latency_ms  # 37.1
r.meta.route  # ["laya_mlx:ok"]

This was run against the laya_mlx backend with no environment variables set at all, using its own default checkpoint; every value above is the real output of that run, not illustrative.

from_env picks the chain in this order, using whatever is installed and/or configured: laya_mlx or laya, whichever is importable, with its own default model unless DECIDE_LOCAL_MODEL overrides it; typesafe (if TYPESAFE_API_KEY is set); openrouter (if OPENROUTER_API_KEY is set); openai_decisions (if OPENAI_API_KEY is set and DECIDE_OPENAI_DECISIONS is not 0); llm (if DECIDE_LLM_BASE_URL or OPENAI_API_KEY is set). A local backend, when installed, is tried first, with the hosted backends as fallback. Set DECIDE_BACKENDS="laya,typesafe" to override the order explicitly. If nothing is importable or configured, from_env raises ConfigError telling you to install a local backend or set a hosted one's API key.

AsyncClient has the same surface, awaited: await client.decide(...), await client.decide_batch(...), AsyncClient.from_env(...).

Backends

crossencoder is built by hand, for example:

from decide import Client, Choice
from decide.backends.crossencoder import CrossEncoderBackend

backend = CrossEncoderBackend("cross-encoder/ms-marco-MiniLM-L6-v2")
client = Client([backend])

r = client.decide(
    "I was charged twice for the same order last week, please refund the duplicate charge.",
    {
        "team": Choice(
            "Which team should handle this?",
            {
                "billing": "Charges, invoices, payment problems, refunds",
                "eng": "Bugs, crashes, broken features",
                "shipping": "Delivery status, delays, lost packages",
            },
        )
    },
)
r.choices["team"].choice  # "billing"
r.choices["team"].probabilities  # {"billing": 0.947, "eng": 0.019, "shipping": 0.034}

Environment variables: overrides and hosted-backend keys

None of these are required to get started. See Quickstart: zero configuration above. They either override a local backend that from_env already auto-selects once it's installed, or supply the API key a hosted backend needs to be auto-selected at all.

Gating and fallback

from decide import Gate

Gate(
    min_confidence=0.0,  # top probability of a Choice, max(noul, 1-noul) for a Noul
    per_question=None,  # optional {"question_name": threshold} overrides
    per_type=None,  # optional {"choice"/"score"/"noul": threshold} overrides
    on_error="next",  # or "raise" to stop the chain on the first BackendError
)

For a question with a threshold set at more than one level, the most specific one wins: per_question[name], else per_type[<the question's type>], else min_confidence. Score answers are never gated regardless of which of these sets a threshold for "score".

For each backend in order: call it. On BackendError with on_error="next", append "<name>:error" to meta.route and try the next backend (with on_error="raise", the error propagates immediately instead). If the response fails the gate, append "<name>:low_confidence:<question>=<value><threshold>" and try the next backend. If it passes, append "<name>:ok" and return. Score answers are never gated (there is no single confidence number for an expected value over levels).

If every backend is exhausted without a passing response, the best response seen so far (highest minimum confidence across its gated answers) is returned, with meta.route[-1] == "<name>:accepted_low_confidence". If no backend produced any response at all, Client.decide raises AllBackendsFailed(route, errors).

A chain of the hosted Jev model, then OpenAI Decisions, then a local model is three backends in order. The chain below is illustrative, and was not run against the live APIs:

from decide import Client
from decide.backends import load

client = Client(
    [
        load("typesafe", api_key="..."),
        load("openai_decisions", api_key="..."),
        load("laya_mlx"),
    ]
)

Route strings you will see in meta.route:

• "<name>:ok": the backend answered and passed the gate.
• "<name>:error": the backend raised a BackendError.
• "<name>:low_confidence:<question>=<value><threshold>": the backend answered but at least one gated question fell below its threshold.
• "<name>:accepted_low_confidence": appended once, at the end of the route, when no backend passed the gate and the best low-confidence response was returned instead.

Client.decide_batch/AsyncClient.decide_batch run the same chain per input state, preserving input order; a state that clears the gate on an earlier backend is not sent to later ones. Client.decide_batch uses a backend's real batch path when capabilities().batch is true, looping decide per request otherwise. AsyncClient.decide_batch always loops adecide per state, concurrently via asyncio.gather; it does not use a backend's batch path in v1. If any state in the batch is left unrouted, AllBackendsFailed carries partial (every Response that did resolve, keyed by input index) and failed (the route so far for every state that did not), so the resolved siblings are not silently lost.

Choosing a threshold

Jevals, an independent Jev benchmark, measured Jev's own accuracy against the confidence threshold it would take to reach it, over 1,500 decisions per task:

Banking77, a Choice task (overall accuracy 79.7%):

PubMedQA, a yes/no Noul task (overall accuracy 91.3%):

Two things follow from this. First, thresholds below about 0.85 filter almost nothing, because most of the accuracy gain from gating shows up only once the threshold climbs well past the model's overall accuracy. Second, Jev clips its probabilities to the 0.01-0.99 range, so a yes/no answer rarely exceeds 0.98; a single min_confidence applied to both Choice and Noul answers therefore behaves very differently across the two. A 0.96 threshold still keeps 59% of choice answers but only about 11% of yes/no answers. Use per_type to set separate thresholds:

from decide import Gate

Gate(per_type={"choice": 0.85, "noul": 0.95})

Server: point TypeSafe's SDK at a local model

laya ships its own laya-serve command as of 0.3.7, so if you only run laya behind a single endpoint, that's the more direct option. Reach for decide serve instead when you want to chain several backends, local or hosted, behind one TypeSafe-compatible endpoint, with the same fallback and gating behavior as the Python client.

DECIDE_LOCAL_MODEL=aac6fef/laya-multilingual-mlx decide serve --backends laya_mlx --port 8811

GET /health reports liveness and the configured backend names; GET /v1/models lists them in an OpenAI-style shape; POST /v1/systemone takes a TypeSafe SystemOneRequest body and returns a SystemOneResponse-shaped body plus a decide extension carrying our own backend/route metadata. Errors come back as {"error": {"message": ..., "type": ...}} with a matching HTTP status.

If --api-key/DECIDE_API_KEY is set, requests must send a matching Authorization: Bearer <token> header; the key must be ASCII, since HTTP header bytes are latin-1 decoded by the server before comparison.

A raw request against the running server above:

curl -s -X POST http://127.0.0.1:8811/v1/systemone \
  -H "Content-Type: application/json" \
  -d '{
    "state": {"ticket": "I was charged twice, please refund."},
    "questions": {
      "team": {"type": "choice", "instructions": "Which team handles this?",
               "criteria": {"billing": "charges, refunds", "engineering": "bugs and outages", "sales": "pricing and upgrades"}},
      "severity": {"type": "score", "instructions": "How severe is this?", "criteria": ["minor", "degraded", "blocked"]},
      "refund": {"type": "noul", "instructions": "Does the customer ask for money back?"}
    }
  }'

{
  "model": "aac6fef/laya-multilingual-mlx",
  "answers": {
    "team": {"type": "choice", "choice": "billing", "confidence": 1.0,
              "probabilities": {"billing": 1.0, "engineering": 0.0, "sales": 0.0}},
    "severity": {"type": "score", "score": 1.5522, "confidence": 0.5719,
                 "legend": {"0": "minor", "1": "degraded", "2": "blocked"},
                 "probabilities": {"0": 0.0197, "1": 0.4083, "2": 0.5719}},
    "refund": {"type": "noul", "noul": 0.9461}
  },
  "usage": {"input_tokens": 0, "output_tokens": 0},
  "decide": {"backend": "laya_mlx", "latency_ms": 50.5, "route": ["laya_mlx:ok"]}
}

And the same request through TypeSafe's own SDK, pointed at the local server (no API key is needed since this server has none configured, but the SDK requires a non-empty string):

from typesafe_sdk import TypeSafeClient, Choice, Score, Noul

client = TypeSafeClient(api_key="anything", base_url="http://127.0.0.1:8811")

resp = client.system_one(
    state={"ticket": "I was charged twice, please refund."},
    questions={
        "team": Choice(
            instructions="Which team handles this?",
            criteria={
                "billing": "charges, refunds",
                "engineering": "bugs and outages",
                "sales": "pricing and upgrades",
            },
        ),
        "severity": Score(
            instructions="How severe is this?", criteria=["minor", "degraded", "blocked"]
        ),
        "refund": Noul(instructions="Does the customer ask for money back?"),
    },
)

model='aac6fef/laya-multilingual-mlx' usage=Usage(input_tokens=0, output_tokens=0)
answers={'team': ChoiceAnswer(type='choice', choice='billing', confidence=1.0,
  probabilities={'billing': 1.0, 'engineering': 0.0, 'sales': 0.0}),
 'severity': ScoreAnswer(type='score', score=1.5522, confidence=0.5719,
  legend={0: 'minor', 1: 'degraded', 2: 'blocked'},
  probabilities={0: 0.0197, 1: 0.4083, 2: 0.5719}),
 'refund': NoulAnswer(type='noul', noul=0.9461)}

TypeSafe's SDK parsed our server's response without modification: it is a genuine SystemOneResponse, not a hand-shaped dict.

CLI

--choice, --score and --noul question names are optional. An unnamed flag is auto-named after the flag itself (choice, score, noul); a second unnamed flag of the same type becomes choice2, score2, noul2, and so on:

decide ask "I had a rough day" --noul "Is the writer doing well?" --choice good,bad

NAME    TYPE    ANSWER  PROBABILITIES
choice  choice  bad     good=0.07 bad=0.93
noul    noul    false   0.02
backend=laya_mlx route=laya_mlx:ok latency=65.3ms

Name a question explicitly with NAME=... to control what shows up in the table and in --json; named and unnamed questions can be mixed on the same command line:

decide ask "I was charged twice, please refund." \
  --choice "team=billing,engineering,sales" \
  --score "severity=minor,degraded,blocked" \
  --noul "refund=Does the customer ask for money back?"

NAME      TYPE    ANSWER    PROBABILITIES
team      choice  billing   billing=0.93 engineering=0.02 sales=0.05
severity  score   degraded  1.09 (degraded)
refund    noul    true      0.87
backend=laya_mlx route=laya_mlx:ok latency=34.8ms

--json prints a machine-readable payload instead of the table, and writes only that JSON to stdout, so decide ask ... --json | jq . is safe to use in scripts. --min-confidence and --model are also available on ask. By default, ask and serve set HF_HUB_DISABLE_PROGRESS_BARS=1 (unless it's already set) so a local backend's model-loading progress bars don't mix into that output; pass --verbose to see them. When shown, that progress goes to stderr, never stdout.

decide backends

NAME              INSTALLED  CONFIGURED  INSTALL
typesafe          yes        no
openrouter        yes        no
openai_decisions  yes        no
laya              yes        default
laya_mlx          yes        default
crossencoder      yes        n/a
llm               yes        no

Every backend with INSTALLED no gets an INSTALL column naming the exact command to add it, e.g. pip install "pydecide[laya]". For laya and laya_mlx, CONFIGURED reads default when installed with no DECIDE_LOCAL_MODEL override (it will be auto-selected with its own default model), yes when DECIDE_LOCAL_MODEL is set, and no only when the backend isn't installed at all.

decide serve --backends a,b --host 127.0.0.1 --port 8811 [--api-key TOKEN] [--min-confidence FLOAT] runs the HTTP server described above.

What the probabilities mean

Every probability in a ChoiceAnswer, ScoreAnswer or NoulAnswer is whatever the backend reported; decide does not calibrate, smooth or verify it. What that means differs by backend: typesafe, openrouter, openai_decisions and laya/laya_mlx are purpose-built decision models, but their outputs are still self-reported by the model and not audited by this library. The crossencoder backend turns a relevance reranker's raw logits into a softmax or sigmoid, so the result is a normalized score forced to distribute mass over the supplied candidates, not a calibrated probability; a Choice will still pick a winner even when every candidate is a bad fit, and a Noul of 0.9 does not mean the condition holds 90% of the time. The llm backend is the least trustworthy of all: it prompts a general chat model to estimate its own confidence in JSON, with no guarantee the model attends to every candidate or keeps its numbers well calibrated. Treat all of this accordingly, as a signal to gate and fall back on, not as ground truth.

Status

pydecide is at 0.2.0. The public API may still change before a 1.0 release.

License

MIT, see LICENSE.