llm-inference-engine

A unified Python interface for LLM inference across multiple backends. Separates model configuration (how a model behaves) from inference engines (where it runs), so you can swap backends without changing application code.

Table of Contents

• Installation
• Quick Start
• LLM Configs
  • BasicLLMConfig
  • ReasoningLLMConfig
  • Qwen3LLMConfig
  • OpenAIReasoningLLMConfig
  • Passing extra_body for Hybrid Thinking Models
• Inference Engines
  • OpenAIInferenceEngine
  • AzureOpenAIInferenceEngine
  • VLLMInferenceEngine
  • SGLangInferenceEngine
  • OpenRouterInferenceEngine
  • OllamaInferenceEngine
  • HuggingFaceHubInferenceEngine
  • LiteLLMInferenceEngine
• Chat Methods
• End-to-End Examples
• Message Logging
• Config Cheat Sheet

Installation

pip install llm-inference-engine

Then install the client library for your backend:

Backend	Install
OpenAI / Azure OpenAI	pip install openai
vLLM (OpenAI-compatible)	pip install openai
SGLang (OpenAI-compatible)	pip install openai
OpenRouter	pip install openai
Ollama	pip install ollama
Hugging Face Hub	pip install huggingface-hub
LiteLLM	pip install litellm

Quick Start

from llm_inference_engine import BasicLLMConfig, OpenAIInferenceEngine

config = BasicLLMConfig(max_new_tokens=1024, temperature=0.5)
engine = OpenAIInferenceEngine(model="gpt-4.1-mini", config=config)

response = engine.chat([{"role": "user", "content": "Hello!"}])
print(response["response"])

LLM Configs

Configs control model behavior: token limits, temperature, message preprocessing, and response postprocessing. Pick the config that matches your model type.

BasicLLMConfig

For standard non-reasoning models (GPT-4.1, Llama, Qwen3-instruct, etc.).

from llm_inference_engine import BasicLLMConfig

config = BasicLLMConfig(max_new_tokens=2048, temperature=0.7)

Response format: {"response": "...", "tool_calls": [...]}

ReasoningLLMConfig

For always-on reasoning models that wrap thinking in <think>...</think> tags (DeepSeek-R1, Qwen3.5-thinking, GPT-OSS, Gemma 4, etc.). Automatically parses thinking and response content.

from llm_inference_engine import ReasoningLLMConfig

config = ReasoningLLMConfig(max_new_tokens=4096, temperature=0.6)

Response format: {"reasoning": "...", "response": "...", "tool_calls": [...]}

Custom thinking tokens (if the model uses different delimiters):

config = ReasoningLLMConfig(thinking_token_start="<reasoning>", thinking_token_end="</reasoning>")

Qwen3LLMConfig

For Qwen3 hybrid thinking models. Appends /think or /no_think tokens to system and user messages to toggle thinking mode. This is the token-based approach that works across all serving engines.

from llm_inference_engine import Qwen3LLMConfig

# Enable thinking
config = Qwen3LLMConfig(thinking_mode=True, max_new_tokens=4096)

# Disable thinking for faster responses
config = Qwen3LLMConfig(thinking_mode=False, max_new_tokens=2048)

Response format: {"reasoning": "...", "response": "...", "tool_calls": [...]} (reasoning is empty when thinking is disabled)

OpenAIReasoningLLMConfig

For OpenAI o-series models (o4-mini etc.). Handles reasoning effort, removes unsupported temperature parameter, and concatenates system prompts into user prompts (required by these models).

from llm_inference_engine import OpenAIReasoningLLMConfig

config = OpenAIReasoningLLMConfig(reasoning_effort="low")   # "low", "medium", or "high"

Response format: {"reasoning": "...", "response": "...", "tool_calls": [...]}

Passing extra_body for Hybrid Thinking Models

Some models (Gemma 4, Qwen3.5, etc.) use server-side thinking control via the extra_body parameter in the OpenAI chat completion API. Since extra_body format depends on the serving engine (vLLM, SGLang, etc.) rather than the model itself, this is handled through kwargs on any config class rather than a dedicated config.

All config classes accept arbitrary kwargs that get passed through to the API call:

Gemma 4 with thinking enabled (vLLM / SGLang):

from llm_inference_engine import ReasoningLLMConfig

config = ReasoningLLMConfig(
    max_new_tokens=4096,
    temperature=1.0,
    extra_body={
        "chat_template_kwargs": {"enable_thinking": True}
    }
)

Gemma 4 with thinking disabled:

from llm_inference_engine import BasicLLMConfig

config = BasicLLMConfig(
    max_new_tokens=2048,
    extra_body={
        "chat_template_kwargs": {"enable_thinking": False}
    }
)

Qwen3.5 with thinking enabled (vLLM):

from llm_inference_engine import ReasoningLLMConfig

config = ReasoningLLMConfig(
    max_new_tokens=4096,
    extra_body={
        "chat_template_kwargs": {"enable_thinking": True}
    }
)

Qwen3.5 with thinking disabled (vLLM):

from llm_inference_engine import BasicLLMConfig

config = BasicLLMConfig(
    max_new_tokens=2048,
    extra_body={
        "chat_template_kwargs": {"enable_thinking": False}
    }
)

Tip: Use ReasoningLLMConfig when thinking is enabled (it parses <think> tags in the response). Use BasicLLMConfig when thinking is disabled.

Inference Engines

Engines define where the model runs. Each engine maps config parameters to the backend's expected format (e.g., max_new_tokens becomes max_completion_tokens for OpenAI, num_predict for Ollama).

OpenAIInferenceEngine

from llm_inference_engine import OpenAIInferenceEngine

engine = OpenAIInferenceEngine(model="gpt-4.1-mini", config=config)

Set OPENAI_API_KEY environment variable, or pass api_key= directly.

AzureOpenAIInferenceEngine

from llm_inference_engine import AzureOpenAIInferenceEngine

engine = AzureOpenAIInferenceEngine(
    model="gpt-4.1-mini",
    api_version="2025-03-01-preview",
    azure_endpoint="https://your-resource.openai.azure.com/",
    api_key="your-api-key"
)

Set AZURE_OPENAI_API_KEY environment variable, or pass api_key= directly.

Set AZURE_OPENAI_ENDPOINT environment variable, or pass azure_endpoint= directly.

VLLMInferenceEngine

For models served via vLLM.

from llm_inference_engine import VLLMInferenceEngine

engine = VLLMInferenceEngine(
    model="Qwen/Qwen3.5-35B-A3B",
    api_key="",                                  # optional for local servers
    base_url="http://localhost:8000/v1"           # default
)

SGLangInferenceEngine

For models served via SGLang.

from llm_inference_engine import SGLangInferenceEngine

engine = SGLangInferenceEngine(
    model="google/gemma-4-26B-A4B-it",
    api_key="",                                   # optional for local servers
    base_url="http://localhost:30000/v1"           # default
)

OpenRouterInferenceEngine

from llm_inference_engine import OpenRouterInferenceEngine

engine = OpenRouterInferenceEngine(model="openai/gpt-oss-120b")

Set OPENROUTER_API_KEY environment variable, or pass api_key= directly.

Reasoning field key compatibility

OpenAI-compatible servers have exposed the model's reasoning/thinking text under different JSON keys across versions (e.g. vLLM/SGLang historically used reasoning_content, while OpenRouter and newer builds use reasoning). By default each engine probes the known keys in order (reasoning_content, reasoning, reason) and uses the first one present, so reasoning is picked up regardless of server version. If your server exposes it under a non-standard key, override the probe order per instance:

engine = VLLMInferenceEngine(
    model="Qwen/Qwen3.5-35B-A3B",
    reasoning_keys=["my_custom_reasoning_field", "reasoning_content"],
)

This applies to OpenAIInferenceEngine, AzureOpenAIInferenceEngine, VLLMInferenceEngine, SGLangInferenceEngine, and OpenRouterInferenceEngine.

OllamaInferenceEngine

from llm_inference_engine import OllamaInferenceEngine

engine = OllamaInferenceEngine(
    model_name="qwen3:27b",
    num_ctx=4096,                                 # context length
    keep_alive=300                                 # seconds to hold model in memory
)

HuggingFaceHubInferenceEngine

from llm_inference_engine import HuggingFaceHubInferenceEngine

engine = HuggingFaceHubInferenceEngine(model="meta-llama/Llama-4-Scout-17B-16E-Instruct")

LiteLLMInferenceEngine

from llm_inference_engine import LiteLLMInferenceEngine

engine = LiteLLMInferenceEngine(
    model="openai/gpt-4.1-mini",
    api_key="your-api-key"
)

Chat Methods

All engines support four chat methods:

Method	Sync/Async	Returns
chat()	Sync	dict
chat_stream()	Sync	Generator[dict]
chat_async()	Async	dict
chat_async_stream()	Async	AsyncGenerator[dict]

Sync

# Non-streaming
response = engine.chat(messages)
print(response["response"])

# With verbose output (prints to terminal in real-time)
response = engine.chat(messages, verbose=True)

# Streaming
for chunk in engine.chat_stream(messages):
    print(chunk["data"], end="", flush=True)  # chunk["type"] is "reasoning" or "response"

Async

import asyncio

async def main():
    # Non-streaming
    response = await engine.chat_async(messages)
    print(response["response"])

    # Streaming
    stream = await engine.chat_async_stream(messages)
    async for chunk in stream:
        print(chunk["data"], end="", flush=True)

asyncio.run(main())

Streaming Chunk Format

Streaming methods yield dicts with type and data keys:

{"type": "reasoning", "data": "Let me think..."}    # thinking content
{"type": "response", "data": "The answer is 42."}    # response content
{"type": "tool_calls", "data": [{"name": "...", "arguments": "..."}]}  # tool calls

End-to-End Examples

GPT-4.1 via Azure OpenAI

from llm_inference_engine import BasicLLMConfig, AzureOpenAIInferenceEngine

config = BasicLLMConfig(max_new_tokens=1024, temperature=0.5)
engine = AzureOpenAIInferenceEngine(
    model="gpt-4.1-mini",
    api_version="2025-03-01-preview",
    azure_endpoint="https://your-resource.openai.azure.com/"
)

response = engine.chat([{"role": "user", "content": "Summarize this document."}])
print(response["response"])

GPT-OSS Reasoning via OpenRouter

from llm_inference_engine import OpenAIReasoningLLMConfig, OpenRouterInferenceEngine

config = ReasoningLLMConfig(reasoning_effort="medium")
engine = OpenRouterInferenceEngine(model="openai/gpt-oss-120b", config=config)

response = engine.chat([
    {"role": "system", "content": "You are a math tutor."},
    {"role": "user", "content": "Prove that sqrt(2) is irrational."}
])
print(response["reasoning"])  # chain-of-thought
print(response["response"])   # final answer

GPT-5.4 via OpenAI

from llm_inference_engine import ReasoningLLMConfig, OpenAIInferenceEngine

config = ReasoningLLMConfig(max_new_tokens=4096, temperature=0.7)
engine = OpenAIInferenceEngine(model="gpt-5.4-mini", config=config)

response = engine.chat([{"role": "user", "content": "Write a Python function to merge two sorted lists."}])
print(response["response"])

Qwen3.5 with Thinking via vLLM

from llm_inference_engine import ReasoningLLMConfig, VLLMInferenceEngine

config = ReasoningLLMConfig(
    max_new_tokens=4096,
    temperature=0.6,
    extra_body={"chat_template_kwargs": {"enable_thinking": True}}
)
engine = VLLMInferenceEngine(model="Qwen/Qwen3.5-35B-A3B", base_url="http://localhost:8000/v1")

response = engine.chat([{"role": "user", "content": "What causes tides?"}])
print(response["reasoning"])
print(response["response"])

Gemma 4 VLM with Thinking + High Resolution for OCR (vLLM)

For vision tasks with Gemma 4, use mm_processor_kwargs in extra_body to control the vision token budget per image. Supported values: 70, 140, 280 (default), 560, or 1120 tokens. Higher values preserve more detail for OCR (https://huggingface.co/google/gemma-4-E4B).

Note: max_soft_tokens can also be set at server launch via --mm-processor-kwargs '{"max_soft_tokens": 1120}'. The extra_body approach below overrides per request.

import base64
from llm_inference_engine import ReasoningLLMConfig, VLLMInferenceEngine

config = ReasoningLLMConfig(
    max_new_tokens=4096,
    temperature=1.0,
    top_p=0.95,
    extra_body={
        "chat_template_kwargs": {"enable_thinking": False}, # OCR doesn't require thinking
        "mm_processor_kwargs": {"max_soft_tokens": 1120} # max detail for OCR 
        "top_k": 64
    }
)
engine = VLLMInferenceEngine(model="google/gemma-4-27b-it", base_url="http://localhost:8000/v1")

with open("document.png", "rb") as f:
    b64_image = base64.b64encode(f.read()).decode()

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64_image}"}},
            {"type": "text", "text": "Extract all text from this document."}
        ]
    }
]

response = engine.chat(messages)
print(response["reasoning"])  # Gemma's thinking process
print(response["response"])   # extracted text

Gemma 4 VLM via SGLang

from llm_inference_engine import ReasoningLLMConfig, SGLangInferenceEngine

config = ReasoningLLMConfig(
    max_new_tokens=4096,
    temperature=1.0,
    extra_body={
        "chat_template_kwargs": {"enable_thinking": True},
        "mm_processor_kwargs": {"max_soft_tokens": 1120}
    }
)
engine = SGLangInferenceEngine(model="google/gemma-4-27b-it", base_url="http://localhost:30000/v1")

Async Batch Processing with Rate Limiting

import asyncio
from llm_inference_engine import BasicLLMConfig, VLLMInferenceEngine

config = BasicLLMConfig(max_new_tokens=512)
engine = VLLMInferenceEngine(
    model="Qwen/Qwen3.5-32B",
    max_concurrent_requests=10,
    max_requests_per_minute=60,
    config=config
)

questions = ["What is Python?", "What is Rust?", "What is Go?"]

async def main():
    tasks = [
        engine.chat_async([{"role": "user", "content": q}])
        for q in questions
    ]
    responses = await asyncio.gather(*tasks)
    for q, r in zip(questions, responses):
        print(f"Q: {q}\nA: {r['response']}\n")

asyncio.run(main())

Message Logging

Use MessagesLogger to capture full conversation history (prompts + responses).

from llm_inference_engine import MessagesLogger

logger = MessagesLogger(store_images=False)  # replaces image URLs with "[image]"

response = engine.chat(messages, messages_logger=logger)

# Retrieve logged conversations
for conversation in logger.get_messages_log():
    for msg in conversation:
        print(f"{msg['role']}: {msg['content'][:100]}")

Token counts and call metadata

The last message of each logged call is the assistant response. Besides role, content, and reasoning, it carries:

{"role": "assistant", "content": "...", "reasoning": "...",
 "token_count": {"input": 18, "output": 28, "total": 46, "cached": 0},
 "finish_reason": "stop",                 # e.g. "stop", "length", "tool_calls"
 "timestamp": "2026-09-21T14:03:12.482Z", # UTC, when the request was sent
 "latency": 1.70}                         # seconds to the complete response (streaming: to the final chunk)

Sum the counts across all logged calls:

logger.get_token_usage()
# {"input": 105, "output": 95, "total": 200, "cached": None}

• Counts come from the backend. A count the backend does not report is None; nothing is estimated.
• output includes reasoning tokens. Backends do not split them consistently, so they are not logged separately.
• cached is reported by OpenAI and Azure. vLLM reports it only when served with --enable-prompt-tokens-details, and SGLang only with --enable-cache-report. Ollama and HuggingFace Hub do not report it.

Streaming changes the request. Servers report token counts for a streamed call only when asked. When a messages_logger is passed to a streaming call (chat_stream, chat_async_stream, or chat(verbose=True)), OpenAI-schema engines add stream_options={"include_usage": True} to the request. The server then sends one extra final chunk that carries the counts. Requests made without a logger are unchanged. Ollama always reports counts on its final chunk, so its requests are never changed.

Config Cheat Sheet

Model	Config	Key Settings
GPT-4.1, GPT-5.4	BasicLLMConfig	max_new_tokens, temperature
o4-mini, GPT-OSS	OpenAIReasoningLLMConfig	reasoning_effort
Qwen3 (hybrid)	Qwen3LLMConfig	thinking_mode
Qwen3.5 (hybrid, vLLM)	ReasoningLLMConfig	extra_body={"chat_template_kwargs": {"enable_thinking": True}}
Gemma 4 (hybrid)	ReasoningLLMConfig	extra_body={"chat_template_kwargs": {"enable_thinking": True}}
DeepSeek-R1	ReasoningLLMConfig	defaults
Llama, Mistral	BasicLLMConfig	max_new_tokens, temperature