jinfer

Local AI inference for Python. Chat, vision, audio, embeddings, reranking, speech and transcription in one wheel: no server, no ML stack, no GPU needed.

jinfer is jinfer, the JVM inference engine, for Python: libjinfer, the engine, every ported model architecture and its native kernels compiled into one shared library with a C API, which the wheel carries. A GGUF model runs in your process, on the CPU, from one load call, and comes back as a typed object with only the methods the file supports. toknroll is the only dependency, for the tokenizer.

pip install jinfer

import jinfer

model = jinfer.load("LiquidAI/LFM2.5-350M-GGUF:Q8_0")       # a LanguageModel; the file decides
reply = model.chat("Why is the sky blue?", max_tokens=200)   # already generating
for text in reply:                                           # stream it ...
    print(text, end="", flush=True)
reply.text, reply.usage.prompt_ms                            # ... or just read it

@dataclass
class Person:
    name: str
    born: int
person = model.chat("Ada Lovelace, born 1815.", schema=Person).parsed   # a Person, guaranteed

def weather(city: str) -> dict:
    """Current weather for a city."""
    return {"celsius": 18}
conversation = model.conversation(system="Be brief.", tools=[weather])
print(conversation.chat("Weather in Zurich?").text)          # runs weather, then answers
branch = conversation.fork(); conversation.rewind()          # free: nothing is recomputed

Why jinfer

• Three lines to a reply, and the reply is an object. load, chat, .text; no messages list to assemble for a first question, no ["choices"][0]["message"]["content"] to dig through, no server to start first.
• Everything local in one package. Chat, vision, audio in, embeddings, reranking, speech synthesis and transcription, each a method on the class the model file calls for, from one 19 MB wheel; no torch, no transformers, no daemon.
• Structured output that cannot come back malformed. schema=Person takes a dataclass, a TypedDict, a Pydantic class, list[T], Literal or Enum, the engine enforces it while decoding, and reply.parsed is the Python value; there is nothing to validate and retry.
• Tools are functions. The signature is the schema and the docstring the description; a conversation runs the calls itself, between the words of the streamed answer.
• The engine keeps the conversation. A turn costs only its new tokens, fork and rewind recompute nothing, prepare warms a prompt before the user types, and a prompt cache file keeps it across processes.
• Stopping is not losing. Close a reply early and its partial text, usage and finish reason stay readable, in a conversation too.
• Typed errors, keyword options, model defaults. InvalidArgument is a ValueError, NotFound a FileNotFoundError; a typo in an option is a TypeError at the call; an option left out takes the model file's value, never the library's.
• One engine, three faces. The same code serves Java, this package, and jinfer.h, twenty-nine C functions any language calls.

Where it trails, so you can decide: the CPU only, the architectures listed below rather than every GGUF, WAV or PCM into transcribe, and eight options on purpose, the rest left to the model file. The comparison, use case by use case, has every snippet.

Requirements

• Python 3.10 or later, and the toknroll wheel it pins, installed with it.
• The wheel carries the library: Linux x86-64 and aarch64 with glibc 2.28 or later, macOS 14 or later on Apple silicon, Windows 10 or later on x86-64. On x86-64 the library is built for x86-64-v3, a CPU with AVX2.
• curl on the PATH, to download a model named by a hub reference; a local file needs nothing.
• ffmpeg on the PATH, to decode an image or an audio file given to a chat; a WAV file for transcription needs nothing.
• espeak-ng (or espeak) on the PATH, for speech synthesis with Kokoro voices.
• RAM for the model file, which is read whole, plus its context.

Models

Any GGUF of a ported architecture, in Q4_0, Q4_1, Q5_0, Q5_1, Q4_K, Q5_K, Q6_K, Q8_0, MXFP4, NVFP4, Q1_0, TQ1_0, TQ2_0, F32, F16 or BF16; Q8_0 is the recommended balance of quality and speed.

The jinfer README keeps the authoritative table, with links to each family.

Models are hub references (owner/repo:QUANT, downloaded once and cached) or local paths. load returns the class the file calls for, LanguageModel, Embedder, Reranker, Speaker or Transcriber, each with only the methods that apply. Every call returns a plain typed object; nothing is a dict you have to know the shape of, and nothing in the API mentions tokens, JSON or OpenAI field names.

Errors are jinfer.JinferError subclasses: InvalidArgument (also a ValueError), NotFound (also a FileNotFoundError), Cancelled; each has .code. jinfer.OpenAI(ref) is a migration aid: code written for the OpenAI SDK's chat.completions.create and embeddings.create runs in-process, unchanged.

Behaviour worth knowing

• A reply is already generating when chat returns. Iterating and the properties are two views of the same reply: read .text after a partial iteration and you get the whole reply.
• Closing a reply early cancels it, and the reply keeps what was produced: .text is the partial text, .finish_reason is "cancelled", .usage counts what ran. In a conversation the partial text joins the history; a turn cancelled before its first word leaves no trace.
• A conversation turn with functions among the tools streams too: the calls run as you read the reply, between the words, and the text you iterate is the model's answer after them. reply.usage is the whole turn's.
• Temperature 0 gives the same tokens every run. Two different kernels can still land on different sides of a near-tie between two tokens, so a test pins behaviour, not prose.
• A conversation's turns are served from the engine's kept state, so a turn costs only its new tokens; reply.usage.cached_tokens shows it. A fork shares the prefix; a rewind recomputes nothing and keeps the system prompt; prepare() computes the prefix before the user types. With prompt_cache=path, what a model prepared is written when it closes and read by the next one.
• Structured output is enforced while decoding, so malformed JSON cannot come back. Validate meaning if you need to; there is no format to retry.
• Quiet by default: only warnings and errors reach logging. A download shows its progress on stderr, jinfer's live board on a terminal and a line at each tenth elsewhere; JINFER_PROGRESS=off, or -Djinfer.progress=off through configure, turns it off entirely. With progress off, JINFER_LOG=info logs each download instead, a start and a done record, shown once the application logs at INFO (logging.basicConfig(level=logging.INFO)); JINFER_LOG=off silences the library. Ctrl-C during a download ends the process; the download resumes next time. repr(model) names the kernels and threads that actually run.
• Any thread may call. A model may be shared between threads; a reply, speech or conversation belongs to one thread at a time. async for on a reply or speech steps in a worker thread; wrap one-shot calls in asyncio.to_thread.
• A model's weights stay alive while any reply or speech made from it lives; close() drops your reference. Context managers close; so does garbage collection.
• Embeddings and audio are float32 array.array buffers, scores are floats: the standard library, whoever installs the package. np.asarray(vector) and torch.frombuffer(speech.audio, dtype=torch.float32) are zero-copy.
• jinfer.configure("-Djinfer.threads=8", "-Djinfer.cache.sessions=16", "-Djinfer.cache.budget=8g", "-Djinfer.offline=true", "-XX:MaxHeapSize=4g") passes the runtime the options a java command line would, once and before the first load; the engine's jinfer.* and jam.* properties are all reachable this way. A later call is refused, and a rejected option fails this one. JINFER_OPTIONS="-Djinfer.threads=4" in the environment holds the same options and overrides them, so the box a program runs on can retune it without a code change.
• A scalar schema (int, str, bool, Literal, Enum) travels as {"value": ...} on the wire, so the model's first token is anchored; reply.parsed is the value, reply.text the JSON. Objects and lists go as they are.

From C

The wheel is the C library too: jinfer.h, toknroll.h, libjinfer and a pkg-config file live in the installed package, and the package says where.

On Linux and macOS, compile with Python's standard library, preserving each output line as one compiler argument:

import subprocess

flags = subprocess.check_output(
    ["jinfer-config", "--cflags", "--libs"], text=True,
).splitlines()
subprocess.run(["cc", "chat.c", *flags], check=True)

jinfer-config is the wheel's console script, in the venv's bin, with --cflags, --libs, --pkgconfigdir and --version, as numpy-config and python3-config have them. Compiler arguments are unquoted, one per line; keep spaces within each line rather than splitting on whitespace.

For pkg-config, point it at the installed package:

PKG_CONFIG_PATH="$(jinfer-config --pkgconfigdir)" pkg-config --cflags --libs jinfer

pkg-config emits shell-escaped flags; in a Python build script, parse that output with shlex.split() rather than splitlines().

On Windows the flags are cl's, /I and /LIBPATH with jinfer.lib, one per line, and jinfer.dll sits beside the import library. The C API is jinfer.h; nothing else is a tool here, jbang jinfer@qxoticai is.

With toknroll

libjinfer carries its own toknroll: a model's tokenizer lives in jinfer's runtime, and model.tokenizer is a toknroll.Tokenizer served by libjinfer. The toknroll wheel is a dependency for that class alone, pinned to the exact toknroll inside libjinfer; the class checks the library's version when it binds, so a mismatched environment fails at import jinfer naming both versions rather than later. jinfer.configure and JINFER_OPTIONS configure the model's tokenizers too, -Dtoknroll.* options included, as they configure everything in libjinfer. toknroll.configure and TOKNROLL_OPTIONS configure libtoknroll only, which jinfer never loads. Using both wheels in one process is fine: toknroll.load(...) starts libtoknroll, a second runtime with its own heap and threads, the two never interact, the order they start in does not matter, and every tokenizer remembers the library serving it. A knob both need, such as a GGUF pre-tokenizer registration, is given to both.

Examples

One script a use case, in examples/, each runnable as it is with the model it names, downloaded on the first run:

A companion is a second file a model needs, named at load: jinfer.load(ref, companions={"media": "<ref>"}) for a vision model's projector, {"voice": "<ref>"} for a speaker's voice. The architectures the library carries are jinfer's ports, listed in the jinfer documentation.

From a checkout

make python in jinfer/jinfer-native stages the freshly built library into the package, then pip install python/. JINFER_LIBRARY=/path/to/libjinfer.so points the package at any build. pydantic is optional. The C API the package binds is jinfer.h, with toknroll.h for the tokenizer, usable from any language with a foreign function interface.

Links

• jinfer: the engine, the ports, the Java API, the server and the framework providers.
• libjinfer: the shared library, its C API jinfer.h, this package and its examples.
• Documentation, the Python comparison, and issues.

License

Apache 2.0, by Quixotic AI, for the package and the library's own code. The library in the wheel is built with Oracle GraalVM and carries its Native Image runtime under the GraalVM Free Terms and Conditions, spelled out in the wheel's NOTICE: use it as you like, commercially too; a product sold for a fee that bundles a copy of it is not covered by those terms, and takes a build with GraalVM CE, see the libjinfer README.