Boole

Local-first LLM inference for Python. Run GGUF models on your own hardware via llama.cpp — get cloud-SDK ergonomics without the cloud bill.

pip install boole

Why Boole

Most inference SDKs assume every call leaves your machine. You pay per token, per second of GPU time, per cold start — even for workloads your own laptop or workstation could handle in milliseconds. Boole flips the default: inference runs locally unless you tell it not to.

• ~10x cheaper by default — no metered API calls for work your hardware can already do.
• No cold starts — models load once into a long-lived local process, not a fresh container on every request.
• No data leaves your machine — prompts, context, and outputs stay local unless you explicitly opt into remote burst.
• Familiar shape — App, Function, and Sandbox primitives will feel immediately natural if you've used a serverless inference SDK before.
• Burst when you need to — for models too large for local hardware, or workloads that need to scale past one machine, the same function can transparently hand off to remote compute (opt-in, planned).

Quickstart

from boole import App

app = App(name="my-app")

generate = app.function(
    {"model": "TheBloke/Mistral-7B-Instruct-v0.2-GGUF", "quant": "Q4_K_M"},
    lambda ctx, prompt: ctx.llm.generate(prompt),
)

result = generate.call("Write a haiku about GPUs")
print(result)

The first call downloads and caches the GGUF weights to ~/.boole/models; every call after that loads from disk and runs entirely on your machine.

You can also use Function as a decorator:

@app.function({"model": "TheBloke/Mistral-7B-Instruct-v0.2-GGUF", "quant": "Q4_K_M"})
def generate(ctx, prompt: str) -> str:
    return ctx.llm.generate(prompt)

Core concepts

Streaming generation

def handler(ctx, prompt: str):
    return ctx.llm.stream(prompt)


stream_generate = app.function({"model": "...", "quant": "Q4_K_M"}, handler)

for token in stream_generate.stream("Write a haiku about GPUs"):
    print(token, end="", flush=True)

Running untrusted code in a Sandbox

sandbox = app.sandbox(timeout_ms=5000, memory_limit_mb=512)
result = sandbox.exec("python3", ["-c", "print(1 + 1)"])
print(result.stdout)

Async usage

Function.call() is synchronous by default. If you're already inside an event loop (e.g. a FastAPI handler), use acall() instead:

result = await generate.acall("Write a haiku about GPUs")

Platform support

Boole uses llama-cpp-python bindings to talk to llama.cpp directly, with GPU offload where available.

llama-cpp-python publishes prebuilt wheels for common platform/CUDA combinations; unsupported combinations fall back to compiling llama.cpp from source on install (a C++ compiler is required in that case).

Configuration

from boole import Client

client = Client(
    model_cache_dir="~/.boole/models",  # where GGUF files are stored
    default_backend="llama-cpp",  # inference backend
)

app = App(name="my-app", client=client)

Roadmap

• Local inference via llama.cpp / GGUF (llama-cpp-python)
• App / Function / Sandbox primitives
• RemoteBurst — opt-in remote fallback for oversized models / scaled workloads
• Structured output / grammar-constrained generation helpers
• Async-native backend (non-blocking generation without a thread pool)

Contributing

Issues and PRs welcome. See CONTRIBUTING.md for local dev setup.

pip install -e ".[dev]"
pytest
ruff check .