A decision model takes a state (text or JSON) and one or more named questions, and answers each with a probability for every option in a single forward pass. No text is generated, so there is no sampling and no temperature. Three question types cover most decisions:

Endor serves these models and lets you fine-tune them on your own labeled data, with a training loop you drive from Python while Endor runs the GPUs.

Install

pip install endor
export ENDOR_API_KEY=edk_...

Python 3.10 and up. Dependencies: httpx and pydantic.

The SDK calls production (https://api.endor.thetokencompany.com). For another environment, such as staging, set ENDOR_BASE_URL=https://staging.api.endor.thetokencompany.com or pass EndorClient(base_url=...).

API keys belong to an organization (edk_..., created in the dashboard); everything you create belongs to that org.

Decisions

Every decision goes through a project, which records it. Create one with a base model and it answers at once:

import endor
from endor import Choice, Noul, Score

client = endor.EndorClient()
client.projects.create("tickets", base_models=["decider-2b"])   # once

res = client.system_one(
    {"subject": "Charged twice", "body": "I was billed two times for my March invoice."},
    {
        "department": Choice(instructions="Which team should handle `body`?",
                             criteria={"billing": "Payments, refunds", "technical": "Bugs, outages", "sales": None}),
        "urgent": Noul(instructions="Does the customer need an answer today?"),
        "frustration": Score(instructions="How frustrated is the customer?", criteria=["Calm", "Annoyed", "Angry"]),
    },
    model="tickets/v1",          # a saved model; "tickets/decider-2b" for the base model
)

res.choices["department"].choice          # "billing"
res.choices["department"].probabilities   # {"billing": 0.91, "technical": 0.07, "sales": 0.02}
res.nouls["urgent"].noul                  # 0.83
res.scores["frustration"].score           # 1.4 (expected level), plus .probabilities and .legend
res.model, res.usage.input_tokens         # the model that answered, e.g. "tickets/v1"

A bare base model id such as "decider-2b" raises ModelRequiresProjectError (422 model_requires_project). There is no default model: pass model= or set one with EndorClient(model="tickets") or ENDOR_DEFAULT_MODEL, else system_one raises EndorError before sending anything.

Questions can also be plain dicts in the same shape. Every question is answered independently against the same state. Refer to parts of the state with backticked paths, as in "Is `body` angry?".

Decisions are TypeSafe's API (POST /v1/systemone). The SDK has its own Noul, Choice, Score, answer and SystemOneResponse types, which send and read exactly TypeSafe's wire format, so it doesn't depend on typesafe-sdk; an unmodified TypeSafe client also works against Endor by changing base_url and model.

Typed responses. Subclass SystemOneResponse to get one attribute per question:

from endor import ChoiceAnswer, NoulAnswer, SystemOneResponse

class Triage(SystemOneResponse):
    department: ChoiceAnswer
    urgent: NoulAnswer

triage = client.system_one(state, questions, model="tickets", response_model=Triage)
triage.department.choice, triage.urgent.noul

Benchmarks. Pass exclude_from_training=True to system_one (or EndorClient(exclude_from_training=True) for every call) to keep decisions out of a managed project's continuous learning, for example a benchmark or an evaluation. They are still answered, billed and logged.

Images. On a base model that reads images (client.base_models(): modalities includes "image"), put endor.Image objects anywhere in the state. PNG, JPEG or WebP, at most 8 per decision and 5 MB each; their tokens are billed like text. Other base models answer 422 unsupported_modality.

state = {"screenshot": endor.Image.from_path("checkout.png"), "goal": "Pay for the order"}
client.system_one(state, {"pay": Noul(instructions="Is the Pay button visible?")}, model="shop")

Image.from_bytes(data, media_type=None) and Image.from_pil(image, format="PNG") (needs Pillow) build one too.

The catalog. client.models.list() returns every name you can pass as model (GET /v1/models): for each project with a base model, "<project>" and "<project>/<base id>" for each of its base models, then "<project>/<name>" for each saved model (.kind is project, base or model). client.base_models() returns the base models (GET /v1/base_models) with their option limits, modalities, hf_repo, contract and prices (price_per_mtok_decide, price_per_mtok_decide_continuous_learning for managed projects, price_per_gpu_hour). Call a base model through a project that has it: "<project>/<base id>".

Async. await client.system_one_async(...) and await client.models.list_async(); async with EndorClient() closes the connections.

Projects

Everything you create lives in a project, keyed by its name. Each project has a kind, set at creation and never changed, and any number of base models, each called "<project>/<base id>":

A custom project holds datasets, runs, the models those runs save (named <project>/<name>), and evaluations:

project = client.projects.get_or_create("tickets", base_models=["decider-2b"])    # kind="custom"
project.datasets.upload("train", endor.data.load_rows("train.jsonl"))   # rows: see "Data format" below
project.datasets.upload("heldout", endor.data.load_rows("heldout.jsonl"))
baseline = project.evaluate("decider-2b", "heldout").result()   # names resolve in the project: "tickets/decider-2b"

project.add_base_model("gev-26b")         # "tickets/gev-26b" answers at once
project.info_.base_models                 # ["decider-2b", "gev-26b"]
project.remove_base_model("gev-26b")      # "tickets/gev-26b" stops answering; models trained on it keep working

project.info_ has kind, base_models (in the order they were added) and paused (managed projects only). project.update(description=...) changes the description. project.models.list() starts with the base models (kind == "base", name the base id), then the saved models (kind == "saved"), each with loss, its training loss when it was saved. runs.create() trains on the project's first base model, or on base_model=; a base the project doesn't have yet is added to it. get_or_create adds the base_models an existing project doesn't have (and removes none); it raises ValueError when the existing project has another kind. base_model= on create or get_or_create raises TypeError: pass base_models=[...].

Managed projects. Endor trains new versions from the project's decisions, named YYYY-MM-DD-N, and "tickets" serves the newest one. The training pipeline is coming soon: until then a managed project serves its first base model. Its decisions are billed at the managed price (50% more) from the start.

project = client.projects.create("triage", kind="managed", base_models=["decider-2b"])
project.update(paused=True)    # stops learning; keeps serving its newest version, at the managed price

Model names you save are lowercase letters, digits, ., _ or -, starting with a letter or digit, up to 63 characters, and can't be base or a base model id. save_checkpoint raises ValueError for a bad name (or base) before sending it; the API refuses a base model id.

A dataset row is a decision request with labels:

{"state": {"body": "I was charged twice"},
 "questions": {"department": {"type": "choice", "criteria": {"billing": null, "technical": null}},
               "urgent": {"type": "noul", "instructions": "Does this need an answer today?"}},
 "labels": {"department": "billing", "urgent": true}}

Labels can be hard ("billing", true, a level index) or soft (a distribution over the options).

Data format

A row has state (text or JSON), questions (named Noul, Choice or Score objects, as system_one takes), and optionally labels (question name → label), id and weight (default 1). Up to 50,000 rows per dataset. Each labeled question becomes one training example; unlabeled questions are skipped.

Every option has an id, used in labels, training targets and answer probabilities: "false"/"true" for a noul, the criteria keys for a choice, "0"…"n-1" (the level index) for a score.

endor.data.load_rows(path) reads .jsonl or a .json array and also accepts single-question rows, {"state", "question", "expected"} or {"state", "question", "label"?, "target"?}, each becoming one question named decision. endor.data.save_rows(path, rows) writes .jsonl.

Fine-tune in a few lines

from endor.recipes import SupervisedConfig, supervised

cfg = SupervisedConfig(project="tickets", model_name="v1")   # base_model="decider-2b" by default
result = supervised.train(cfg, endor.data.load_rows("train.jsonl"))

result.base_metrics["accuracy"], result.final_metrics["accuracy"]   # held-out: "tickets/decider-2b", then yours
client.system_one(state, questions, model=result.model)             # "tickets/v1"

The recipe holds out 10% of the rows, scores the untuned base model ("<project>/<base_model>", added to the project if it doesn't have it) on them, trains one epoch (learning rate 1e-4 with warmup then linear decay, batches of 16), scores the held-out rows during and after training, and saves the final model. Every number appears on the run's dashboard page. SupervisedConfig has the knobs: base model, rank, learning rate and schedule, batch size, epochs, loss, how often to evaluate. The recipe sets the run's total_steps from the data size and epochs, so the dashboard shows a progress bar with an ETA, and it prints a progress line (step 37/120 (31%) · loss 0.4120 · ~3 min left; progress=False turns it off).

Training costs and limits

• Billing. Training is billed by the second for the GPU time your training calls use, one price per GPU-hour for every base model; start-up, model loading and the time between calls are free, and each run session costs at least 60 GPU-seconds. Decisions are billed per 1M input tokens, per base model. client.base_models() shows the prices; client.usage(starting_on, ending_before) returns hourly decide rows (input tokens) and train rows (GPU-seconds) with their cost.
• Idle timeout. After 2 minutes without calls, a run saves its state and frees its GPU (status idle). The next call resumes it in a few seconds. After 1 hour without calls, the run is closed; saved models are kept.
• 5 open runs per org. Every run that isn't closed counts, idle ones included. A sixth runs.create raises LimitReachedError (409, code limit_reached), whose message lists the open runs; close one (run.close() or endor runs close RUN_ID). It is not retried.
• Other limits. 7 projects per org, 20 API keys and 50 models per project also raise LimitReachedError once reached. client.whoami().limits lists every limit with its value.
• Close your runs. Use with project.runs.create(...) as run: (or async with): the run is closed when the block ends, also on errors. runs.create(wait=True) closes the run if you interrupt it while it provisions, and the recipes close their run whatever happens. run.close() refuses later calls, lets calls already accepted finish (status closing), then releases the GPU (status closed); it returns at once, or waits with close(wait=True).
• Balance. When your org's balance is used up (a new org starts with none), decisions, new runs, training calls and server-side evaluations raise InsufficientBalanceError (402, code insufficient_balance) until you add credit in the dashboard under Billing. A call the server already accepted still finishes.

Or drive the loop yourself

rows = endor.data.load_rows("train.jsonl")
train, heldout = endor.data.split(rows, holdout=0.1)
train, heldout = endor.data.rows_to_datums(train), endor.data.rows_to_datums(heldout)

with project.runs.create(rank=16) as run:          # on the project's first base model, or base_model="..."
    for batch in endor.data.batches(train, 16):
        fb = run.forward_backward(batch)            # gradients accumulate on the trainer
        opt = run.optim_step(learning_rate=1e-4)    # AdamW step, then zero gradients
        out, step = endor.gather(fb, opt)           # submit both, then wait once
    scored = run.forward(heldout).result()          # option probabilities with the current adapter
    metrics = endor.metrics.decision_metrics(scored.probabilities, [d.target for d in heldout])
    run.log({"heldout/accuracy": metrics["accuracy"]})   # at the run's current step
    model = run.save_checkpoint("v2", include_optimizer=True).result()   # "tickets/v2", usable at once

resumed = project.runs.create(from_model="v2", include_optimizer=True)   # continue exactly where v2 stopped

A run created with from_model inherits that model's base model and LoRA settings (rank, alpha, which layers are trained); the SDK sends only the settings you pass explicitly, and one that conflicts with the model is a 422. A fresh run defaults to rank 16, alpha 32, attention and MLP layers, no readout.

• A Datum is one state, one question, a Target (a hard label or soft probs) and a weight. Each labeled question of a row becomes one datum.
• forward, forward_backward, optim_step and save_checkpoint return futures. Call .result(), await them, or wait for several with endor.gather(...). Each has an _async variant that submits without blocking. A future can stay pending while a GPU starts (usually under a minute); .result() keeps waiting.
• Gradients are the weighted mean over every datum since the last optim_step, so splitting a batch across calls changes nothing. optim_step with nothing accumulated fails with no_gradients. Batches over 1,024 datums are split for you; every datum is checked locally before any chunk is sent, so a bad datum never leaves half a batch applied.
• Losses: cross_entropy (hard or soft targets) and brier.
• optim_step takes AdamParams (betas, epsilon, weight decay, gradient clipping) and the learning rate, which you pass every step: the schedule is yours.
• run.forward scores data with the current adapter without saving a model. project.evaluate(model, dataset) scores a saved model on a stored dataset server-side.
• run.close() releases the trainer (a with block does it for you). Unsaved progress is lost; the SDK warns when you close with optimizer steps newer than your last save.
• Creating a run provisions a trainer, usually in under a minute. runs.create(wait=True) (the default) blocks and logs progress on the endor logger; with wait=False the run returns at once and run.ready is the future.
• If the trainer's GPU dies, pending calls fail with trainer_lost and the run becomes failed (run.info().failure says why). Start a new run with from_model= your last saved model.
• Progress: pass runs.create(total_steps=N) (or later run.set_total_steps(N)) and the dashboard shows a progress bar with an ETA; without it, the step count and steps per minute. run.info() has step, total_steps, seconds_per_step (median over the last 20 steps), progress (0 to 1) and eta_seconds.

Weights & Biases

Endor logs runs to Weights & Biases from its servers; your training code doesn't change. Connect W&B once in the dashboard (Settings > Integrations, with a W&B API key that Endor keeps in AWS Secrets Manager), then turn logging on per project (its Settings tab, or project.update(wandb={"enabled": True})), to the W&B project endor-<project name>. Override it per run with project.runs.create(wandb=True) or wandb=False. Each run becomes a W&B run with the same id, with the base model, LoRA settings and your config, and every run metric (train/loss, train/accuracy, train/grad_norm, train/lr, run.log) against the Endor step; it is finished when the run closes. run.info().wandb_url links to it.

Teacher distillation

Label unlabeled rows with a teacher of your choice, then fine-tune on its answer distributions:

from endor.recipes import DistillConfig, SupervisedConfig, distill

class MyTeacher:
    def label(self, rows):                        # one {question name: label} per row
        return [{"department": {"billing": 0.7, "technical": 0.3, "sales": 0.0}} for _ in rows]

cfg = DistillConfig(SupervisedConfig(project="tickets", model_name="distilled"), teacher=MyTeacher())
result = distill.distill(cfg, unlabeled_rows, eval_rows=gold_rows)

Download a trained model

A saved model is a LoRA adapter (PEFT) for its base model. Download its files to run it yourself:

files = project.models.download("v1", "./tickets-v1")   # include_optimizer=True adds optimizer.pt

This writes adapter_model.safetensors, adapter_config.json, endor_manifest.json and, if the model has one, readout.safetensors into the folder. optimizer.pt (Adam state) is only needed to resume training. Each file comes straight from storage over a link that expires after 10 minutes, and is checked against its size and SHA-256 (when the API gives one); a file that fails raises DownloadError and is not left behind. The links are never returned or logged, and your API key is never sent to storage. From the shell: endor models download tickets/v1 -o ./tickets-v1.

Errors and retries

Every API error is an endor.APIError subclass named after its status (NotFoundError, ConflictError, UnprocessableEntityError, RateLimitError, ...) with the server's stable code (unknown_model, invalid_datum, seq_conflict, ...), the offending param when there is one, and the request_id. InsufficientBalanceError (402, insufficient_balance) means your org's balance is used up: add credit in the dashboard. A few codes have their own subclass: LimitReachedError (409 limit_reached) and NoBaseModelError (409 no_base_model), both ConflictErrors, and ModelRequiresProjectError (422 model_requires_project), an UnprocessableEntityError. A training operation that fails on the trainer raises OperationFailedError from .result(), with codes such as oom, no_gradients or trainer_lost. Connection problems raise APIConnectionError and APITimeoutError.

Requests that fail with a connection error, a timeout, a 408, a 429 or a 5xx are retried with exponential backoff and Retry-After, except limit_reached and insufficient_balance, which waiting doesn't fix. Configure it with RetryPolicy:

client = endor.EndorClient(retry=endor.RetryPolicy(max_retries=5, timeout=120))
client = endor.EndorClient(retry=endor.RetryPolicy(max_retries=0))      # no retries

A base model that has been idle starts cold: its first decision waits a few minutes while the model starts. The SDK waits up to 330 s for a decision (30 s for other calls). If the model is still not ready after 5 minutes, the API answers 503 warming_up with Retry-After, and the SDK retries after that wait.

Retries are safe: creates carry an Idempotency-Key, and a run's compute calls carry a sequence number the server checks, so a retried request never runs twice.

Configuration

EndorClient also takes headers for every request, retry, gzip=False to stop compressing large bodies, capture=False to stop sending each run's settings and your git commit hash to the dashboard, and your own httpx client or transport.

What the SDK sends

Besides your API key and the request itself, every request carries:

• User-Agent: endor-python/<version> (python <version>; <os>; <cpu>), with ; cli added for the endor command;
• X-Endor-SDK: endor-python, X-Endor-SDK-Version: <version>, X-Endor-Runtime: python/<version> (<os>; <cpu>) and X-Endor-SDK-Interface: python or cli;
• X-Endor-SDK-Method (the SDK call, e.g. run.forward_backward), X-Endor-SDK-Recipe (inside a recipe), X-Endor-Client-Request-Id (the same on every retry of one call) and X-Endor-Retry-Count (on retries).

runs.create also sends the run's settings and, unless capture=False, your git commit hash and a dirty flag (never file contents).

For every request, the API stores the request id, your org, the calling key, the route, the status and the latency, plus a client record with one field per header: user_agent, sdk, sdk_version, runtime, interface, method, recipe, client_request_id and retry_count (each cut to 200 characters). It also keeps the request body, and the response body when it is JSON and at most 256 KB, in object storage (S3). The secret API key is never stored. You can add your own headers with EndorClient(headers=...); the ones above can't be overridden.

CLI

endor whoami
endor base-models
endor projects list | create NAME [--base-model BASE ...] [--kind custom|managed] | delete NAME
endor projects add-base-model NAME BASE | remove-base-model NAME BASE
endor datasets list PROJECT | upload PROJECT NAME FILE | delete PROJECT NAME
endor runs list PROJECT | show RUN_ID | close RUN_ID
endor models list PROJECT | info MODEL | download MODEL [-o DIR] [--include-optimizer] | set-ttl MODEL SECONDS|none | delete MODEL
endor eval PROJECT MODEL DATASET
endor usage --start 2026-10-01 --end 2026-10-06 [--project P] [--csv]

MODEL is <project>/<name>, or <project>/<base id> for a base model of the project. Every command takes -f json. Training itself happens in Python. endor runs close frees a slot when a script left a run open. usage --csv columns are hour, kind, project, base_model, model, training_run_id, input_tokens, gpu_seconds, continuous_learning, price_per_mtok, base_cost_usd, continuous_learning_cost_usd, cost_usd; continuous_learning is true for a managed project's decisions.

Reference

Every public member has a docstring with the details.

License

MIT