oxillama-py

Python bindings for OxiLLaMa — high-performance LLM inference from Python.

Part of the OxiLLaMa workspace — a Pure Rust LLM inference engine.

What It Provides

EngineConfig — configuration dataclass for thread count, context size, tokenizer path, and sampler defaults
Engine — load a GGUF model and generate text; releases the GIL during inference
AsyncEngine — async/await interface; streams tokens to Python coroutines without blocking the event loop
SamplerConfig — all ten sampler knobs with greedy() and mirostat_v2() static constructors
SpeculativeConfig / SpeculativeEngine — draft + target model pair for faster generation
Lora — load a LoRA adapter and hot-swap it onto an Engine
Tokenizer — first-class tokenizer object with encode, decode, encode_batch, apply_chat_template
CancellationToken — cooperative cancellation handle accepted by generate() and generate_streaming()
Structured exception hierarchy: OxiLlamaError → LoadError, GenerateError, TokenizerError, GrammarError, QuantError, KvCacheFullError, GpuUnavailableError (raised if GPU offload is ever requested and unavailable — there is currently no constructor kwarg to request it from Python, see below)
Full Python type annotations (.pyi stubs) and docstrings
Wheels built with maturin (ABI3, Python 3.8+)
Optional numpy interop (embed_numpy(), embed_batch_numpy(), forward_logits_numpy()) via numpy feature

Status

Version: 0.1.4 — Tests: 131 Rust unit tests passing (cargo nextest -p oxillama-py --all-features); see TODO.md for the separate Python pytest suite count

Installation

pip install maturin
maturin develop --release          # in-place development install
# or
maturin build --release            # build a wheel
pip install target/wheels/oxillama_py-*.whl

Usage

import oxillama_py as ox

# Load model
engine = ox.Engine("llama-3.2-3b.Q4_K_M.gguf")

# Basic generation (GIL is released during the Rust inference call)
output = engine.generate(
    prompt="Tell me about the Rust programming language.",
    max_new_tokens=256,
    temperature=0.8,
    top_p=0.95,
)
print(output)

# Streaming generation with a callback
engine.generate_streaming(
    "Explain quantum computing.",
    max_tokens=256,
    callback=lambda tok: print(tok, end="", flush=True),
)

# Async engine (non-blocking, event-loop friendly)
import asyncio

async def run():
    aengine = ox.AsyncEngine("llama-3.2-3b.Q4_K_M.gguf")
    result = await aengine.generate("Hello async world", max_new_tokens=64)
    print(result)

asyncio.run(run())

# Cooperative cancellation
token = ox.CancellationToken()
engine.generate_streaming("Tell me a story", max_tokens=1024,
                          callback=print, cancel_token=token)
token.cancel()  # stop from another thread

# Speculative decoding: 3-8x faster on large models
draft  = ox.Engine("llama-3.2-1b.Q4_K_M.gguf")
target = ox.Engine("llama-3.2-8b.Q4_K_M.gguf")
spec   = ox.SpeculativeEngine(draft=draft, target=target, gamma=4)
output = spec.generate("Once upon a time", max_new_tokens=512)
print(output)

# LoRA adapter
lora   = ox.Lora.load("my-adapter.gguf")
engine.apply_lora(lora)
output = engine.generate("Write a haiku.", max_new_tokens=64)
engine.remove_lora()

# Tokenizer
tokenizer = ox.Tokenizer.from_file("tokenizer.json")
ids = tokenizer.encode("Hello, world!")
text = tokenizer.decode(ids)

# HuggingFace Hub loader
engine = ox.Engine.from_hub("meta-llama/Llama-3.2-3B-GGUF")

Feature Flags

Feature	Default	Description
`numpy`	no	numpy interop for `embed_numpy()`, `embed_batch_numpy()`, `forward_logits_numpy()`
`hub`	no	HuggingFace Hub loader (`Engine.from_hub()`) — required for the Hub example above

License

Apache-2.0 — COOLJAPAN OU (Team Kitasan)

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

oxillama-0.1.4.tar.gz (1.9 MB view details)

Uploaded Aug 17, 2026 Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

oxillama-0.1.4-cp38-abi3-win_amd64.whl (3.0 MB view details)

Uploaded Aug 17, 2026 CPython 3.8+Windows x86-64

oxillama-0.1.4-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB view details)

Uploaded Aug 17, 2026 CPython 3.8+manylinux: glibc 2.17+ x86-64

oxillama-0.1.4-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.9 MB view details)

Uploaded Aug 17, 2026 CPython 3.8+manylinux: glibc 2.17+ ARM64

oxillama-0.1.4-cp38-abi3-macosx_11_0_arm64.whl (2.8 MB view details)

Uploaded Aug 17, 2026 CPython 3.8+macOS 11.0+ ARM64

oxillama-0.1.4-cp38-abi3-macosx_10_12_x86_64.whl (2.9 MB view details)

Uploaded Aug 17, 2026 CPython 3.8+macOS 10.12+ x86-64

File details

Details for the file oxillama-0.1.4.tar.gz.

File metadata

Download URL: oxillama-0.1.4.tar.gz
Upload date: Aug 17, 2026
Size: 1.9 MB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for oxillama-0.1.4.tar.gz
Algorithm	Hash digest
SHA256	`2419017802b1604e0452e1a9e5b3a83e4acb2e0c779ee489d1149d3402408700`
MD5	`33895a8a69bd628d773f1258f5fe3023`
BLAKE2b-256	`1743f3895cc712260d0e534ec194f508f0eb02fe844fe3b363cf226331398d3f`

See more details on using hashes here.

Provenance

The following attestation bundles were made for oxillama-0.1.4.tar.gz:

Publisher: pypi-publish.yml on cool-japan/oxillama

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: oxillama-0.1.4.tar.gz
- Subject digest: 2419017802b1604e0452e1a9e5b3a83e4acb2e0c779ee489d1149d3402408700
- Sigstore transparency entry: 2498501943
- Sigstore integration time: Aug 17, 2026
Source repository:
- Permalink: cool-japan/oxillama@98f17b848daefc0dcbda4d3c64681df30a4a991b
- Branch / Tag: refs/tags/v0.1.4
- Owner: https://github.com/cool-japan
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: pypi-publish.yml@98f17b848daefc0dcbda4d3c64681df30a4a991b
- Trigger Event: push

File details

Details for the file oxillama-0.1.4-cp38-abi3-win_amd64.whl.

File metadata

Download URL: oxillama-0.1.4-cp38-abi3-win_amd64.whl
Upload date: Aug 17, 2026
Size: 3.0 MB
Tags: CPython 3.8+, Windows x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for oxillama-0.1.4-cp38-abi3-win_amd64.whl
Algorithm	Hash digest
SHA256	`9ae0fd067a068f7eecac542e878400df7e735426104a54d3dec1b4de98c54f6b`
MD5	`a4b98d7f3c1e4ff795f16e4104e05f52`
BLAKE2b-256	`b2e708585e3c77f3e353a1d8a5b289908d8dc56fefc2a286a30a90bae9a1a0ad`

See more details on using hashes here.

Provenance

The following attestation bundles were made for oxillama-0.1.4-cp38-abi3-win_amd64.whl:

Publisher: pypi-publish.yml on cool-japan/oxillama

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: oxillama-0.1.4-cp38-abi3-win_amd64.whl
- Subject digest: 9ae0fd067a068f7eecac542e878400df7e735426104a54d3dec1b4de98c54f6b
- Sigstore transparency entry: 2498501965
- Sigstore integration time: Aug 17, 2026
Source repository:
- Permalink: cool-japan/oxillama@98f17b848daefc0dcbda4d3c64681df30a4a991b
- Branch / Tag: refs/tags/v0.1.4
- Owner: https://github.com/cool-japan
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: pypi-publish.yml@98f17b848daefc0dcbda4d3c64681df30a4a991b
- Trigger Event: push

File details

Details for the file oxillama-0.1.4-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

Download URL: oxillama-0.1.4-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Upload date: Aug 17, 2026
Size: 3.1 MB
Tags: CPython 3.8+, manylinux: glibc 2.17+ x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for oxillama-0.1.4-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm	Hash digest
SHA256	`60be8a0b457943e399371fe957b560edeb09c233f2eeea6aa879043060260792`
MD5	`d7c9eaaed7c3543ed2139a4b790ae0bf`
BLAKE2b-256	`5483a9b34378409ba2f24d2193506632663f5733a311be62a9259e5b9448c8f4`

See more details on using hashes here.

Provenance

The following attestation bundles were made for oxillama-0.1.4-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: pypi-publish.yml on cool-japan/oxillama

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: oxillama-0.1.4-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Subject digest: 60be8a0b457943e399371fe957b560edeb09c233f2eeea6aa879043060260792
- Sigstore transparency entry: 2498501951
- Sigstore integration time: Aug 17, 2026
Source repository:
- Permalink: cool-japan/oxillama@98f17b848daefc0dcbda4d3c64681df30a4a991b
- Branch / Tag: refs/tags/v0.1.4
- Owner: https://github.com/cool-japan
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: pypi-publish.yml@98f17b848daefc0dcbda4d3c64681df30a4a991b
- Trigger Event: push

File details

Details for the file oxillama-0.1.4-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

Download URL: oxillama-0.1.4-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Upload date: Aug 17, 2026
Size: 2.9 MB
Tags: CPython 3.8+, manylinux: glibc 2.17+ ARM64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for oxillama-0.1.4-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm	Hash digest
SHA256	`96cab8e48f57117e16fd5ea7a5f59a0e064728a98145bdaaaf3bb98d1cb14fa0`
MD5	`5ec40261e7f48d460a2e6a7ba21bb7c9`
BLAKE2b-256	`ead7a4b896a724015332be3f898d93c3e752a62ee350a0aa77c686cd06584d22`

See more details on using hashes here.

Provenance

The following attestation bundles were made for oxillama-0.1.4-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: pypi-publish.yml on cool-japan/oxillama

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: oxillama-0.1.4-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
- Subject digest: 96cab8e48f57117e16fd5ea7a5f59a0e064728a98145bdaaaf3bb98d1cb14fa0
- Sigstore transparency entry: 2498501971
- Sigstore integration time: Aug 17, 2026
Source repository:
- Permalink: cool-japan/oxillama@98f17b848daefc0dcbda4d3c64681df30a4a991b
- Branch / Tag: refs/tags/v0.1.4
- Owner: https://github.com/cool-japan
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: pypi-publish.yml@98f17b848daefc0dcbda4d3c64681df30a4a991b
- Trigger Event: push

File details

Details for the file oxillama-0.1.4-cp38-abi3-macosx_11_0_arm64.whl.

File metadata

Download URL: oxillama-0.1.4-cp38-abi3-macosx_11_0_arm64.whl
Upload date: Aug 17, 2026
Size: 2.8 MB
Tags: CPython 3.8+, macOS 11.0+ ARM64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for oxillama-0.1.4-cp38-abi3-macosx_11_0_arm64.whl
Algorithm	Hash digest
SHA256	`80c8bfc5d0ce6281665c8fbfc246ac5723d9d069c3eb0f1cc4afaae2cddb7564`
MD5	`7812f7de77cf2c7c4848820595e36f2a`
BLAKE2b-256	`cf70d6add9d0fe527b6cd8a6fca91efb497d0197a06ab0ff2e033b0249d5563c`

See more details on using hashes here.

Provenance

The following attestation bundles were made for oxillama-0.1.4-cp38-abi3-macosx_11_0_arm64.whl:

Publisher: pypi-publish.yml on cool-japan/oxillama

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: oxillama-0.1.4-cp38-abi3-macosx_11_0_arm64.whl
- Subject digest: 80c8bfc5d0ce6281665c8fbfc246ac5723d9d069c3eb0f1cc4afaae2cddb7564
- Sigstore transparency entry: 2498501957
- Sigstore integration time: Aug 17, 2026
Source repository:
- Permalink: cool-japan/oxillama@98f17b848daefc0dcbda4d3c64681df30a4a991b
- Branch / Tag: refs/tags/v0.1.4
- Owner: https://github.com/cool-japan
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: pypi-publish.yml@98f17b848daefc0dcbda4d3c64681df30a4a991b
- Trigger Event: push

File details

Details for the file oxillama-0.1.4-cp38-abi3-macosx_10_12_x86_64.whl.

File metadata

Download URL: oxillama-0.1.4-cp38-abi3-macosx_10_12_x86_64.whl
Upload date: Aug 17, 2026
Size: 2.9 MB
Tags: CPython 3.8+, macOS 10.12+ x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for oxillama-0.1.4-cp38-abi3-macosx_10_12_x86_64.whl
Algorithm	Hash digest
SHA256	`de66cd2609f624b40fc82c6d2df79f9bcc206c4f1c76a91805ccbfece5711177`
MD5	`3a000258a7f808e710443fb4211755d8`
BLAKE2b-256	`fb1eaf4e593c9c76468d4ceed937a46b8ff25c6b26b8ce5ecddde4370bb6a479`

See more details on using hashes here.

Provenance

The following attestation bundles were made for oxillama-0.1.4-cp38-abi3-macosx_10_12_x86_64.whl:

Publisher: pypi-publish.yml on cool-japan/oxillama

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: oxillama-0.1.4-cp38-abi3-macosx_10_12_x86_64.whl
- Subject digest: de66cd2609f624b40fc82c6d2df79f9bcc206c4f1c76a91805ccbfece5711177
- Sigstore transparency entry: 2498501960
- Sigstore integration time: Aug 17, 2026
Source repository:
- Permalink: cool-japan/oxillama@98f17b848daefc0dcbda4d3c64681df30a4a991b
- Branch / Tag: refs/tags/v0.1.4
- Owner: https://github.com/cool-japan
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: pypi-publish.yml@98f17b848daefc0dcbda4d3c64681df30a4a991b
- Trigger Event: push

Release history Release notifications | RSS feed

This release

0.1.4 This release

Aug 17, 2026 6 files

May 5, 2026 6 files

Apr 25, 2026 6 files

Apr 24, 2026 6 files

Apr 15, 2026 6 files

oxillama 0.1.4

oxillama-py

What It Provides

Status

Installation

Usage

Feature Flags

License

Key dates

1 maintainer

License

Requires

Classifiers

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

Release history Release notifications | RSS feed