Oragon

Run AI models inside your own software — on a laptop, your own server or your own datacenter, CPU or GPU. Your data never leaves the machine.

pip install oragon

That is the whole install. The engine is inside the package: no compiler, no separate download, no signup, no API key.

from oragon import Oragon, Engine, LocalModel, Settings, user

client = Oragon(engine=Engine.EMBEDDED, model=LocalModel.QWEN2_5_0_5B)

reply = client.chat.generate(
    model=LocalModel.QWEN2_5_0_5B,
    messages=[user("Name one primary color.")],
)
print(reply.choices[0].message.content)

The model loads once and stays ready, so the second request does not pay for the first one's startup. Requests run side by side: up to four at once for free, and more than four on a paid plan. When more arrive than can run, they wait in a queue instead of failing. How that works.

Why Oragon

• It tells you when it doesn't know. You get valid data or a clear error, the exact label you asked for or "no match", a cited answer or an honest refusal. Never a guess dressed up as an answer.
• It fails like production software. Every error is a typed exception that says whether it is safe to retry. Memory is checked before a model loads, and context length before a word is generated.
• It handles many requests at once. Requests are batched together, shared prompts are cached, extra requests wait in a bounded queue, and a request turned away is retried for you.
• It runs anywhere. The same code works on a laptop, a server or a GPU. Moving between them is a setting, not a rewrite.
• Your data stays yours. No telemetry, ever. The full list of what touches the network is below.

Quick start

oragon try                # download a small model and chat with it
oragon try --tunnel       # the same chat at a public HTTPS link — open it on your phone
oragon try extract        # turn text into validated JSON
oragon try rag            # ask questions about your documents
oragon serve              # run an OpenAI-compatible API, with a browser playground
oragon doctor             # check what this machine can run

The chat runs against a live OpenAI-compatible API the whole time, so any OpenAI client in any language can join in while it is up.

The first run downloads the model. Disk space is checked first, progress is shown, and the file is verified. Every later run finds it on disk. To fetch ahead of time (CI images, air-gapped machines), run oragon download qwen2.5-0.5b. Set ORAGON_AUTO_DOWNLOAD=0 to stop the SDK from ever going online; a missing model then becomes a clear error that names the download command.

If anything goes wrong, oragon doctor reports the platform, memory, engine, models and settings in one paste. ORAGON_DEBUG=1 turns on detailed logging.

The browser playground

oragon serve also serves a playground page from the engine itself. Open http://127.0.0.1:8080/ in a browser and you get:

• a chat pane to talk to the model,
• a structured-output demo that turns text into validated JSON,
• a live console showing what the engine is doing right now: tokens per second, lanes in use, requests waiting, requests answered, time to first word, how much of each prompt came from cache, and how full the cache and GPU memory are.

The page uses nothing from the internet, so it works the same on an air-gapped machine. When the server needs a token, the link oragon serve prints already carries it. With --tunnel, it also prints a public playground link you can send to someone.

Many requests at once

Oragon is built to serve traffic, not one chat window. This is what happens when requests pile up, in order:

1. Several run at the same time. The engine has lanes, and each lane generates one request at a time. Up to 4 lanes are free, in every install. More than 4 needs a paid plan. Requests in different lanes are batched into the same passes over the model, so four requests cost far less than four times one. A long prompt is read in chunks, so it cannot freeze the others.
2. Shared prompts are computed once. Requests that start with the same text, such as a shared system prompt, share its cache across lanes. In one test, eight agents sharing an instruction went from 11 s to 2.2 s.
3. The rest wait in a queue. When every lane is busy, a new request waits and starts as soon as a lane frees up. Nothing is dropped, and nothing is cut off partway through an answer.
4. The queue has limits, so it cannot sink the machine. By default it holds 256 requests per lane, capped by a memory budget sized from free RAM. It also has a time limit: a request that would wait more than 60 seconds is turned away when it arrives, instead of being answered a minute late.
5. A request that is turned away is never half-done. It gets an immediate overloaded answer and nothing was generated, so sending it again is safe.
6. The SDK retries it for you. A turned-away request is retried automatically, up to three times, with short randomized pauses (about two seconds in total) so a thousand callers do not all retry at once. Streams are retried too: a refusal always comes before the first word, so a retry never repeats output. You only see EngineBusyError when the overload lasts. Before that, an EngineBusyWarning on requests that succeeded tells you pressure is building while you can still act on it.
7. Over HTTP, it is a 429. The API returns the status OpenAI clients already understand, so the official OpenAI libraries retry it on their own. A server that is shutting down returns 503, finishes what is running, and then exits.

On a GPU, the same rules cover memory. New work is turned away before the card fills, while running requests finish, so there is no out-of-memory crash to recover from. A request whose prompt and answer would not fit the cache waits for room instead of being started and pushed out halfway.

You can change all of it. On the free tier, slots goes up to 4: asking for more runs with 4 and says so. It is not an error, and nothing else changes. A paid plan raises the limit.

client = Oragon(
    engine=Engine.EMBEDDED,
    model=LocalModel.QWEN2_5_1_5B,
    slots=4,             # lanes: 1-4 free; more than 4 needs a paid plan
    max_queue=512,       # how many may wait for a lane
    max_queue_wait=10,   # seconds a caller will wait; 0 = always wait, never turn away
)

client.stats()   # lanes busy, requests waiting, queue limits, cache reuse
client.ping()    # health check; answers between words, even when busy

AsyncOragon has the same behaviour: asyncio.gather over many calls really does overlap, and the engine batches them. Over HTTP, the same figures are at /metrics (Prometheus) and /v1/status (JSON).

Measured on a 16 GB laptop with no GPU: 16 requests at once, each streaming at 12+ tokens/s with its first word in about a second. In a flood of 6,000 requests arriving at the same instant, every one was either answered or told overloaded right away. None were left hanging.

What you can do

Everything in this section is free.

Three engines, one API

Oragon(engine=Engine.EMBEDDED, model=...)   # the model runs in a process this one owns
Oragon(engine=Engine.SHARED,   model=...)   # one copy of the model per machine (paid)
Oragon(engine=Engine.HOSTED,   model=...)   # the Oragon API (not available yet)

All three have the same generate(), the same response type and the same errors, so where the model runs is a deployment choice, not a rewrite.

Get structured data

Pass a schema. You get data that matches it, or an ExtractionError — nothing in between.

rows = client.chat.generate(
    model=LocalModel.QWEN2_5_1_5B,
    messages=[user(invoice_text)],
    response_schema={"type": "array", "items": {
        "type": "object",
        "required": ["date", "amount"],
        "properties": {"date": {"type": "string"}, "amount": {"type": "number"}},
    }},
)

The schema is enforced while the model writes, so output that would break it cannot be produced.

Make a decision

You get back one of the labels you offered, or matched=False. It is never stretched to fit the closest-looking label.

decision = client.chat.generate(
    model=LocalModel.QWEN2_5_1_5B,
    messages=[user("Is 'J. Smith' the same person as 'John Smith'?")],
    response_format={"type": "choice", "choices": ["SAME", "DIFFERENT"]},
)
print(decision.choice, decision.matched)

For harder calls, Settings(thinking=True) shows the model its own answer and asks it to check. Its first answer is the floor, so this cannot do worse than leaving it off:

decision = client.chat.generate(
    model=LocalModel.QWEN2_5_1_5B,
    messages=[user("Meeting started 9:45, ran 100 min — over before noon?")],
    response_format={"type": "choice", "choices": ["YES", "NO"]},
    settings=Settings(temperature=0.0, thinking=True),
)
print(decision.reason)   # the reasoning, for your audit log
print(decision.choice)   # the verdict, for your code

Use tools

Give it plain Python functions and act() runs the whole loop for you. The tool's description is read from its signature, type hints and docstring.

def get_weather(city: str) -> dict:
    """Current weather for a city."""
    return {"city": city, "temp_c": 22}

reply = client.chat.act(
    model=LocalModel.QWEN2_5_1_5B,
    messages="What's the weather in Lagos?",
    tools=[get_weather],
)
print(reply.choices[0].message.content)

• If a tool fails, the model sees the error and tries again.
• Pass messages as a list and the conversation is kept, so you can answer a follow-up question and carry on.
• Several tool calls in one round run in parallel.
• on_tool and on_round let you watch what it does.

Prefer to run the loop yourself? generate(tools=...) uses the same format as the OpenAI client.

Ask questions about your documents

from oragon.rag import DocumentQA, EngineRetriever, ingest

retriever = EngineRetriever()
ingest(["handbook.md", "policy.pdf"], retriever)

qa = DocumentQA(client, model=LocalModel.QWEN2_5_1_5B, retriever=retriever)
result = qa.ask("How many vacation days do new hires get?")

if result.refused:
    print("The documents don't say.")
else:
    print(result.text)               # with citations like [1]
    for hit in result.sources:       # the exact place in each file
        print(hit.chunk.source, hit.chunk.start, hit.chunk.end)

It searches by words and by meaning, entirely on your machine. A small embedding model runs inside the same engine, so there is no second service to run. If nothing relevant is found, it refuses instead of guessing, and a citation the model made up is flagged rather than quietly fixed.

Stream and run async

for chunk in client.chat.generate(model=..., messages=[...], stream=True):
    if isinstance(chunk, str):
        print(chunk, end="", flush=True)

from oragon import AsyncOragon

client = AsyncOragon(engine=Engine.EMBEDDED, model=LocalModel.QWEN2_5_1_5B)
reply = await client.chat.generate(model=..., messages=[...])

Long conversations

A conversation too long for the model raises ContextOverflowError before anything is generated. It is never silently cut. Pass overflow="slide" to drop the oldest turns instead (the system prompt is kept):

client.chat.generate(model=..., messages=history, overflow="slide")

Fast without tuning

• Tuned to your machine. On first run the engine measures this machine and sizes its own work to match. The measurement is saved, so it happens once.
• Prompt reuse. A shared system prompt or a growing conversation is read from cache rather than recomputed. Watch reply.usage.cached_tokens climb.
• Warm start. Oragon(warm_prompt=SYSTEM_PROMPT) reads a long standing instruction at startup, so even the first request is fast: 10 s down to 0.5 s in our test.
• Conversations that survive a restart. Pass session="..." and a conversation's computed state is kept on disk. The next request only processes what is new. The answer is the same with or without it.
• Repeated questions answered instantly. oragon serve --cache auto (or Oragon(answer_cache="auto")) answers a request it has already answered, word for word, without generating it again: under a millisecond instead of seconds. Only exact matches are served. --cache measure reports how much it would save on your traffic before you turn it on. Off by default.
• Speculative decoding. draft_model= lets a small model speed up a large one, up to 1.56× on a 3B model. It checks whether this helps on your machine and switches itself off when it would not.

oragon doctor tells you what this machine will do before you choose a model: its measured memory speed, the generation speed to expect from each model, and how much context each one has room for.

Use your own model

oragon import ./my-finetune/       # a training checkpoint folder
oragon import my-model.gguf        # or a model file

It is converted and optimized for you, and your original file is never changed. If it cannot be converted correctly, it is refused with the reason.

Serve it over HTTP

oragon serve                               # this machine only: 127.0.0.1:8080
oragon serve --expose                      # other machines on your network
oragon serve --tunnel                      # anyone: a public HTTPS link

--expose and --tunnel create an access token for the run unless you pass --auth-token. Any OpenAI client, in any language, can now use your model:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key=TOKEN)
client.chat.completions.create(model="qwen2.5-1.5b", messages=[...])

Route	What it is
/	the browser playground
/v1/chat/completions	chat, with streaming, tools and response_format
/v1/embeddings	vectors, for a vector database you already run
/v1/models	what is loaded
/health	liveness
/v1/status, /metrics	load, queue and cache figures (behind the token, when one is set)

When the engine cannot do something, it says so instead of improvising. Asking for a model it has not loaded is a 404, not a quiet swap. Binding to a network address without a token is refused at startup.

Handle errors

All errors share a base class, ApiError, and say whether a retry is safe.

from oragon import (
    ApiError, ContextOverflowError, EngineBusyError, EngineDiedError,
    EngineTimeoutError, ExtractionError, InsufficientMemoryError,
)

Error	What it means
EngineBusyError	Still overloaded after the automatic retries. Nothing was started; retry later
InsufficientMemoryError	The model will not fit. Caught before loading, not a crash
ContextOverflowError	The conversation is too long; see overflow="slide"
ExtractionError	The output still did not match your schema after one corrected retry
EngineDiedError	The engine stopped mid-request; the next call starts a new one
EngineTimeoutError	The request took longer than your timeout

See what your app did — locally

Every request is summarized into a store on this machine: timings, token counts, models, error codes, tool names. Prompts and answers are never stored, and nothing is ever uploaded.

oragon stats             # last 7 days: latency, error rates, tool calls
oragon stats --json
oragon obs clear         # delete the store; always safe

ORAGON_OBSERVABILITY=0 turns recording off entirely.

More

• Reproducible answers — Oragon(reproducible=True): an answer never depends on what was asked before it.
• Audit export — oragon audit export --format jsonl (or cef) for your audit system.
• See a model's chat template — oragon template qwen2.5-1.5b.
• LangChain — pip install 'oragon[langchain]', then from oragon.llm import OragonChat.

Servers and GPUs

The same engine runs on a server or in a datacenter.

pip install "oragon[gpu]"          # Linux, Python 3.10+, an NVIDIA driver
oragon serve Qwen/Qwen2.5-1.5B-Instruct --gpu auto --expose

The engine starts a GPU model server, waits for it to load, and serves the API in front of it. Typed errors, the queue, licensing and document search all stay the same.

• Tuned to the card. --gpu-profile auto (the default) reads the card's memory and sets itself up. small, balanced and throughput choose explicitly.
• It does not thrash. If the GPU starts running out of room, the engine admits fewer requests until it recovers, instead of slowing everyone down.
• It reserves before it starts. A request is only started when its prompt and answer fit. Otherwise it waits for a running one to finish.
• It restarts itself. If the model server crashes, it is replaced, and queued requests wait for it.

On CPU servers, --kv-pool auto sizes one shared cache from the free memory. Requests that share a prompt store it once, so more lanes fit in the same memory, and requests that do not fit wait instead of failing. None of this changes an answer.

Running several machines? Each response carries an X-Oragon-Affinity header. Route on it, and requests with the same system prompt land on the machine that already has it cached. Several machines then behave like one deployment instead of several cold starts.

No GPU is required. CPU is a first-class path, not a fallback. If a GPU is there it is used automatically; pass cpu=True to force CPU and compare.

Benchmarks

Measured, not estimated. CPU figures come from a 16 GB MacBook (Apple M4) running qwen2.5-1.5b — the smallest machine Oragon is built for. Servers do better.

Oragon on its own

Test	Result
16 requests at once, CPU only	each gets 12+ tokens/s, first word in 1 s
16 requests at once, one Tesla T4 GPU	224 tokens/s total, slowest request still 59.9 tokens/s
6,000 requests at the same instant	every one answered or told "busy" — none left hanging
600,000 requests in three floods, never restarted	0 lost, 0 unexplained errors, memory flat, health check in 0.02 ms
First request with a long system prompt	10 s → 0.5 s when warmed at startup
8 agents sharing one prompt	11 s → 2.2 s with caching
The same question asked again, answer cache on (0.5B model)	0.5 ms instead of 1.3 s
Document search over 10,000 passages	7 ms per search; 150 of 150 citations pointed at the exact text
Same model, with Oragon's checks around it	7/18 → 11/18 correct on graded tasks
Invoice extraction on documents that aren't invoices	5/7 handled correctly, against 2/7 for the model alone
Startup, small model	first answer 0.8 s after launch

Oragon and Ollama, side by side

Same model file, same machine, one running at a time (Ollama 0.33.3).

Test	Oragon	Ollama
1,000 different requests, CPU, both at 16 parallel	done in 810 s	1,244 s — Oragon 1.54× faster
6,000 requests at the same instant, one CPU	none left hanging, 17 answers/s	4,757 got no answer, 8 answers/s
16 requests at once, one Tesla T4 GPU	224 tokens/s, slowest 60 tokens/s	140 tokens/s, slowest 25 tokens/s
First read of a long instruction, CPU	10.9 s	23.8 s — 2.2× faster
16 requests at once, CPU, default settings	9.3 s	20.1 s
16 requests at once, Apple M4, default settings	10.9 s	15.7 s
Startup to first word	0.96 s	1.61 s
Invoice extraction, correct outcome	5/7	2/7
One request, generation speed	92 tokens/s	94 tokens/s — a tie
Repeated long prompt (caching)	0.17 s	0.18 s — a tie
Accuracy on the same model	14/36	14/36 — a tie
One request on the T4, time to first word	228 ms	24 ms — Ollama ahead

Ollama answers one request at a time by default; with its parallel setting raised it matches us on the Apple M4. The 1,000-request, 6,000-request, T4 and long-instruction results hold with Ollama tuned. Ollama spent about 14% more work per request on the 1,000-request run, so not all of that gap is engine.

Speed per request is set by your hardware, and answer quality by the model — what Oragon adds is everything around it.

How it compares

	Oragon	Ollama
Install	pip install oragon	a separate installer, then a client library
Runs on	laptop or server, CPU or GPU	laptop or server, CPU or GPU
Many requests at once	up to 4 lanes free (more on a paid plan), a bounded queue, automatic retry	one at a time by default
When it is full	answers 429 at once; the SDK retries for you	requests can wait with no answer
Asking for JSON	valid data, or a clear error	text you check yourself
Asking for a category	the exact label, or an honest "no match"	text you have to interpret
Questions over your documents	included, with citations, and it says when it doesn't know	build it yourself
When something breaks	typed errors that say whether to retry	a status code and a message
Model too big for the machine	you are told before it loads	it fails partway through loading
Your data	stays on the machine	stays on the machine

Ollama is a good way to try models. Oragon is built for the next step: software you hand to someone else, running on a machine you do not control, where an odd answer from the model has to be something your code can catch.

Platforms

Python 3.9 to 3.13.

Platform	Supported	Speeds up with
macOS 11+, Apple silicon	✅	Apple GPU
Linux x86-64 (Ubuntu 24.04+, Debian 13+, Fedora 40+)	✅	NVIDIA, AMD or Intel GPU (via Vulkan), or CPU
Linux arm64 (same versions)	✅	CPU
RHEL 9, Ubuntu 22.04, Debian 12	coming soon
Windows, Intel Macs	coming soon

Every feature works the same on every platform; only the speed-up differs. No GPU needed. If one is there, it is used automatically. One install covers every machine: there is no separate GPU package to choose. On an unsupported platform, pip installs the older 0.0.1 placeholder instead — if you see version 0.0.1, that is why.

Pricing

Everything above is free — no signup, no limits on tokens, and commercial use is allowed within the free tier.

	Free	Paid
Up to 4 lanes (requests generating at once), plus the queue	✅	✅
Every feature on this page, including the HTTP API and GPU serving	✅	✅
Many processes sharing one model (Engine.SHARED)	—	✅
More than 4 lanes	—	✅
Oragon's own licensed models, encrypted on disk	—	✅

If four at a time in one process is enough, you never need to pay. That is intended, not a loophole: you can ship a desktop app to someone who never signs up, never connects, and never tells you their name.

Your data

The complete list of what crosses the network, so you can check it rather than trust it:

• Your prompts and answers never leave the machine. There is no code path that could send them, and they are never used to train anyone's model.
• No telemetry, no crash reports, no update checks.
• Model downloads — only the model you asked for, only incoming. ORAGON_AUTO_DOWNLOAD=0 turns even that off.
• Licence check-in — only on a machine with an activated licence. It sends the licence key, usage counts (token totals, model names, timings — never content) and a device ID. The device ID is a short hash, not the machine's own identifier, and ORAGON_DEVICE_ID lets you set it yourself. No licence, no check-in.

Licences work offline for their whole validity period, so air-gapped deployments are fully supported.

Not yet supported

• Images, audio and video — text only for now.
• The hosted engine. Asking for it raises an error rather than pretending. Code written today will move to it by changing one word.
• Some platforms — see Platforms.

Who builds this

Oragon Labs — an AI research lab building AI that runs where people and their data already are: on their own machines, in their own country, at a price local economies can sustain. No GPU required, no data leaving your control, no single company or server as a point of failure.

Learn more

• Documentation: https://www.oragonlabs.com
• Contact: hello@oragonlabs.com

Already have an Oragon account? The same package is on your private index, so you can install it from your dashboard.