offline-intelligence 0.1.7

Private On-Device Inference Engine — run LLMs locally with zero configuration

Offline Intelligence

Private On-Device Inference Engine — run LLMs locally with zero configuration.

Installation

pip install offline-intelligence

Quick Start

from offline_intelligence import OfflineIntelligence

# Create SDK — auto-detects your hardware (NVIDIA, AMD, Intel, Apple Silicon)
sdk = OfflineIntelligence()

# Download inference engine (first run only, ~200MB, stored in AppData)
sdk.ensure_ready()

# Load a GGUF model and start inference
sdk.load_model("path/to/model.gguf")

# Chat
response = sdk.chat("What is the capital of France?")
print(response["content"])

# Cleanup
sdk.stop()
sdk.close()

Context Manager

from offline_intelligence import OfflineIntelligence

with OfflineIntelligence() as sdk:
    sdk.ensure_ready()
    sdk.load_model("model.gguf")
    
    # Simple string input
    response = sdk.chat("Hello!")
    print(response["content"])
    
    # Full message format
    response = sdk.chat([
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain quantum computing in one sentence."}
    ])
    print(response["content"])

Configuration

sdk = OfflineIntelligence({
    "ctx_size": 4096,       # Context window size
    "gpu_layers": 28,       # GPU layers to offload (0 = CPU only)
    "threads": 8,           # CPU threads
    "batch_size": 512,      # Batch size for prompt processing
    "env_file": ".env",     # Optional .env file (does NOT pollute your environment)
})

How It Works

1. OfflineIntelligence() — Detects your GPU hardware, creates data directories. Instant.
2. ensure_ready() — Downloads the best inference engine for your hardware. First run only.
3. load_model(path) — Spawns a local inference server on a free port. Manages all DLLs automatically.
4. chat(messages) — Sends requests to the local server. All inference stays on your machine.

No data leaves your device. No API keys. No internet required after initial setup.

Status

from offline_intelligence import SdkStatus

sdk = OfflineIntelligence()
print(sdk.status)  # SdkStatus.NOT_STARTED

sdk.ensure_ready()
print(sdk.status)  # SdkStatus.DEGRADED (engine ready, no model)

sdk.load_model("model.gguf")
print(sdk.status)  # SdkStatus.READY

Platform Support

Platform	GPU Support
Windows x64	NVIDIA (CUDA), AMD (HIP), Intel (SYCL), Vulkan, CPU
macOS ARM64	Apple Metal
macOS x64	CPU
Linux x64	NVIDIA (CUDA), AMD (ROCm), Vulkan, CPU

License

Apache 2.0