Oragon

Run AI models inside your own software — on a laptop, your own server or your own datacenter, CPU or GPU. Your data never leaves the machine.

pip install oragon

That is the whole install. The engine is inside the package: no compiler, no separate download, no signup, no API key.

from oragon import Oragon, Engine, LocalModel, user

client = Oragon(engine=Engine.EMBEDDED, model=LocalModel.QWEN2_5_0_5B)

reply = client.chat.generate(
    model=LocalModel.QWEN2_5_0_5B,
    messages=[user("Name one primary color.")],
)
print(reply.choices[0].message.content)

The model loads once and stays ready. Requests run side by side — four at once on the free tier, and as many as your machine holds on a paid plan.

Why Oragon

• It tells you when it doesn't know. You get valid data or a clear error, the exact label you asked for or "no match", a cited answer or an honest refusal. Never a guess dressed up as an answer.
• It fails like production software. Every error is a typed exception that says whether it is safe to retry. Memory is checked before a model loads.
• It behaves like a server. Requests are batched, shared prompts are cached, and when it is full it says "busy, try again" instead of hanging.
• It runs anywhere. The same code works on a laptop or a server.
• Your data stays yours. No telemetry, ever.

Quick start

oragon try            # download a small model and chat with it
oragon try extract    # turn text into validated JSON
oragon try rag        # ask questions about your documents
oragon serve          # run an OpenAI-compatible API on this machine
oragon doctor         # check what this machine can run

The first run downloads the model and checks it. Set ORAGON_AUTO_DOWNLOAD=0 to stop the SDK from ever going online.

What you can do

Everything in this section is free.

Get structured data

Pass a schema. You get data that matches it, or an ExtractionError — nothing in between.

rows = client.chat.generate(
    model=LocalModel.QWEN2_5_1_5B,
    messages=[user(invoice_text)],
    response_schema={"type": "array", "items": {
        "type": "object",
        "required": ["date", "amount"],
        "properties": {"date": {"type": "string"}, "amount": {"type": "number"}},
    }},
)

Make a decision

You get back one of the labels you offered, or matched=False.

decision = client.chat.generate(
    model=LocalModel.QWEN2_5_1_5B,
    messages=[user("Is 'J. Smith' the same person as 'John Smith'?")],
    response_format={"type": "choice", "choices": ["SAME", "DIFFERENT"]},
)
print(decision.choice, decision.matched)

For harder calls, Settings(thinking=True) has the model check its own answer first.

Use tools

Give it plain Python functions and act() runs the whole loop for you.

def get_weather(city: str) -> dict:
    """Current weather for a city."""
    return {"city": city, "temp_c": 22}

reply = client.chat.act(
    model=LocalModel.QWEN2_5_1_5B,
    messages="What's the weather in Lagos?",
    tools=[get_weather],
)
print(reply.choices[0].message.content)

• If a tool fails, the model sees the error and tries again.
• Pass messages as a list and the conversation is kept, so you can answer a follow-up question and carry on.
• Several tool calls in one round run in parallel.
• on_tool and on_round let you watch what it does.

Prefer to run the loop yourself? generate(tools=...) uses the same format as the OpenAI client.

Ask questions about your documents

from oragon.rag import DocumentQA, EngineRetriever, ingest

retriever = EngineRetriever()
ingest(["handbook.md", "policy.pdf"], retriever)

qa = DocumentQA(client, model=LocalModel.QWEN2_5_1_5B, retriever=retriever)
result = qa.ask("How many vacation days do new hires get?")

if result.refused:
    print("The documents don't say.")
else:
    print(result.text)               # with citations like [1]
    for hit in result.sources:       # the exact place in each file
        print(hit.chunk.source, hit.chunk.start, hit.chunk.end)

It searches by words and by meaning, entirely on your machine. If nothing relevant is found, it refuses instead of guessing.

Stream and run async

for chunk in client.chat.generate(model=..., messages=[...], stream=True):
    if isinstance(chunk, str):
        print(chunk, end="", flush=True)

AsyncOragon has the same API with await, and parallel requests are batched together.

Use your own model

oragon import ./my-finetune/       # a training checkpoint folder
oragon import my-model.gguf        # or a model file

It is converted and optimized for you. If it cannot be converted correctly, it is refused with the reason.

Serve it over HTTP

oragon serve --auth-token "$TOKEN"

Any OpenAI client, in any language, can now use your model:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key=TOKEN)

Handle errors

All errors share a base class, ApiError, and say whether a retry is safe.

from oragon import (
    ApiError, ContextOverflowError, EngineBusyError, EngineDiedError,
    EngineTimeoutError, ExtractionError, InsufficientMemoryError,
)

• EngineBusyError — nothing was started; retry.
• InsufficientMemoryError — the model will not fit; caught before loading.
• ContextOverflowError — the conversation is too long. Pass overflow="slide" to drop the oldest turns instead.

More

• Speculative decoding — draft_model= lets a small model speed up a large one, up to 1.56× on a 3B model. It switches itself off when it would not help.
• Conversations that survive restarts — pass session="...".
• Reproducible answers — Oragon(reproducible=True).
• Audit export — oragon audit export --format jsonl.
• Local usage stats — oragon stats. Never uploaded.
• LangChain — pip install 'oragon[langchain]'.

Servers and GPUs

The same engine runs on a server or in a datacenter.

pip install "oragon[gpu]"          # Linux, Python 3.10+, an NVIDIA driver
oragon serve Qwen/Qwen2.5-1.5B-Instruct --gpu auto --expose

• It tunes itself to the GPU it finds.
• If the GPU fills up, it admits fewer requests instead of slowing down.
• If the model server crashes, it restarts it.
• Running several machines? Each response carries an X-Oragon-Affinity header — route on it, and similar requests land on the machine that has them cached.

Benchmarks

Measured, not estimated. CPU figures come from a 16 GB MacBook (Apple M4) running qwen2.5-1.5b — the smallest machine Oragon is built for. Servers do better.

Oragon on its own

Test	Result
16 requests at once, CPU only	each gets 12+ tokens/s, first word in 1 s
16 requests at once, one Tesla T4 GPU	224 tokens/s total, slowest request still 59.9 tokens/s
6,000 requests at the same instant	every one answered or told "busy" — none left hanging
600,000 requests in three floods, never restarted	0 lost, 0 unexplained errors, memory flat, health check in 0.02 ms
First request with a long system prompt	10 s → 0.5 s when warmed at startup
8 agents sharing one prompt	11 s → 2.2 s with caching
Document search over 10,000 passages	7 ms per search; 150 of 150 citations pointed at the exact text
Same model, with Oragon's checks around it	7/18 → 11/18 correct on graded tasks
Invoice extraction on documents that aren't invoices	5/7 handled correctly, against 2/7 for the model alone
Startup, small model	first answer 0.8 s after launch

Oragon and Ollama, side by side

Same model file, same machine, one running at a time (Ollama 0.33.3).

Test	Oragon	Ollama
1,000 different requests, CPU, both at 16 parallel	done in 810 s	1,244 s — Oragon 1.54× faster
6,000 requests at the same instant, one CPU	none left hanging, 17 answers/s	4,757 got no answer, 8 answers/s
16 requests at once, one Tesla T4 GPU	224 tokens/s, slowest 60 tokens/s	140 tokens/s, slowest 25 tokens/s
First read of a long instruction, CPU	10.9 s	23.8 s — 2.2× faster
16 requests at once, CPU, default settings	9.3 s	20.1 s
16 requests at once, Apple M4, default settings	10.9 s	15.7 s
Startup to first word	0.96 s	1.61 s
Invoice extraction, correct outcome	5/7	2/7
One request, generation speed	92 tokens/s	94 tokens/s — a tie
Repeated long prompt (caching)	0.17 s	0.18 s — a tie
Accuracy on the same model	14/36	14/36 — a tie
One request on the T4, time to first word	228 ms	24 ms — Ollama ahead

Ollama answers one request at a time by default; with its parallel setting raised it matches us on the Apple M4. The 1,000-request, 6,000-request, T4 and long-instruction results hold with Ollama tuned. Ollama spent about 14% more work per request on the 1,000-request run, so not all of that gap is engine.

Speed per request is set by your hardware, and answer quality by the model — what Oragon adds is everything around it.

Platforms

Python 3.9 to 3.13.

Platform	Supported	Speeds up with
macOS 11+, Apple silicon	✅	Apple GPU
Linux x86-64 (Ubuntu 24.04+, Debian 13+, Fedora 40+)	✅	NVIDIA, AMD/Intel (Vulkan), or CPU
Linux arm64 (same versions)	✅	CPU
RHEL 9, Ubuntu 22.04, Debian 12	coming soon
Windows, Intel Macs	coming soon

No GPU needed. If one is there, it is used automatically. On an unsupported platform, pip installs the older 0.0.1 placeholder instead.

Pricing

Everything above is free — no signup, no limits on tokens.

	Free	Paid
One process, 4 requests at once	✅	✅
Every feature on this page	✅	✅
Many processes sharing one model (Engine.SHARED)	—	✅
More than 4 requests at once	—	✅
Oragon's own licensed models	—	✅

If four at a time in one process is enough, you never need to pay.

Your data

• Your prompts and answers never leave the machine.
• No telemetry, no crash reports, no update checks.
• The only network calls: downloading a model you asked for, and — only on a machine with an activated licence — a check-in that sends usage counts, never content. No licence, no check-in.

Not yet supported

• Images, audio and video — text only for now.
• The hosted engine.

Learn more

• Documentation: https://www.oragonlabs.com
• Contact: hello@oragonlabs.com

Already have an Oragon account? The same package is on your private index, so you can install it from your dashboard.