OpenInfer

A developer-friendly, extensible open-source runtime for local LLM inference.

OpenInfer sits between low-level libraries like Hugging Face Transformers (where you configure everything by hand) and heavyweight production inference engines like vLLM or TensorRT-LLM (powerful but complex). It gives you a simple API for local inference while handling model loading, device detection, and configuration internally.

Status: early / alpha (v0.1). Package name: open-infer (PyPI) / open_infer (import).

Installation

pip install open-infer                 # library only
pip install open-infer[api]            # + REST API server
pip install open-infer[ui]             # + browser playground UI
pip install open-infer[api,ui]         # everything

Usage

As a library

from open_infer import InferenceEngine

engine = InferenceEngine("Qwen/Qwen2.5-1.5B-Instruct")
response = engine.generate("Explain KV Cache.")

Device (cuda / mps / cpu) is auto-detected. Models load in float16 on GPU/MPS, float32 on CPU.

As a REST API

open-infer-api

Starts a server at http://127.0.0.1:8000 (OPEN_INFER_API_HOST / OPEN_INFER_API_PORT to override). The model loaded at startup comes from OPEN_INFER_MODEL (default: Qwen/Qwen2.5-1.5B-Instruct).

Interactive docs at http://127.0.0.1:8000/docs.

As the Playground UI

open-infer-api    # in one terminal — the UI talks to the API, it doesn't replace it
open-infer-ui      # in another terminal

A browser-based chat UI (OPEN_INFER_API_URL to point it at a different API). Pick a model from the sidebar's curated presets (all ≤4GB in float16) or paste a custom Hugging Face model ID.

Features

• Automatic hardware detection (CUDA / Apple Silicon MPS / CPU)
• One-line model loading via InferenceEngine
• Repetition-safe generation (repetition_penalty, no_repeat_ngram_size)
• REST API with live model switching, no restart needed
• Browser playground UI with curated, memory-safe model presets

Roadmap

OpenInfer is early — current focus is a stable core (InferenceEngine, REST API, UI). Planned next: streaming responses, KV cache, continuous batching, quantization, and additional backends (GGUF, ONNX, MLX) beyond the current Transformers backend.

License

Apache 2.0 — see LICENSE.