tinfer-ai 0.1.0

Tinfer — Tiny Inference Engine. Run LLMs locally with GPU acceleration.

Tinfer — Tiny Inference Engine

Run LLMs locally with GPU acceleration. No dependencies, no cloud, no API keys.

Install

pip install tinfer

Quick Start

CLI Chat

tinfer -m model.gguf -p "Hello, what is AI?"

Server + WebUI

tinfer-server -m model.gguf --port 8080
# Open http://localhost:8080 for the chat interface

Python API

from tinfer import Server, chat

# Start server programmatically
with Server("model.gguf", port=8080, n_gpu_layers=-1) as s:
    response = chat("What is artificial intelligence?")
    print(response)

HTTP API (OpenAI-compatible)

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Hello"}], "max_tokens": 100}'

Features

• CLI — Interactive chat and text completion
• Server — HTTP server with built-in WebUI
• API — OpenAI-compatible /v1/chat/completions endpoint
• GPU — CUDA acceleration out of the box
• Quantized — Run GGUF models (Q4, Q5, Q8) efficiently
• MoE — Mixture of Experts model support
• Vision — Multimodal model support

Python Client

from tinfer import chat, complete, models, set_server

# Point to a running server
set_server("http://localhost:8080")

# Simple chat
print(chat("Explain quantum computing in 3 sentences"))

# Chat with system prompt
print(chat([
    {"role": "system", "content": "You are a pirate."},
    {"role": "user", "content": "What's the weather?"}
]))

# List models
for m in models():
    print(m["id"])