pylolm (LoLm)

Local LLMs on Android, from Python. An Android-friendly take on llama-cpp-python, built on llama.cpp and designed for Kivy, Flet and Termux.

• Zero Python dependencies: a thin ctypes layer over a small C shim.
• llama-cpp-python-style API: Llama(...), create_chat_completion(...), create_completion(...).
• Streaming generators, stop strings, a stop() button for your UI.
• Multi-turn chat that reuses the KV cache, so long conversations stay fast.
• Aimed at small GGUF models (0.5B-2B) on phones.

Status: alpha (0.1.0). Text generation only. TTS / STT / OCR are on the roadmap.

Install

pip install pylolm
pylolm get-libs      # downloads the prebuilt native libraries (seconds)
pylolm doctor        # checks that everything loads

get-libs fetches the Android arm64 libraries attached to the matching GitHub Release, checks their SHA-256 checksum, and installs them to ~/.pylolm/lib. These are built by CI to run on a wide range of phones.

Prefer to compile for your exact device, or get-libs is not available for your setup?

pkg install cmake clang git
pylolm build-native  # clones llama.cpp and builds on the device (slow: ~30 min)

Libraries are searched in $PYLOLM_LIB_DIR, then pylolm/lib/, then ~/.pylolm/lib/.

Quick start

from pylolm import Llama

llm = Llama("/sdcard/models/Qwen2.5-0.5B-Instruct-Q4_K_M.gguf", n_ctx=2048)

messages = [{"role": "user", "content": "Explain GGUF in two sentences."}]

# whole answer
out = llm.create_chat_completion(messages, max_tokens=200)
print(out["choices"][0]["message"]["content"])

# streaming
for chunk in llm.create_chat_completion(messages, max_tokens=200, stream=True):
    piece = chunk["choices"][0]["delta"].get("content")
    if piece:
        print(piece, end="", flush=True)

Try it from the terminal: pylolm chat /path/to/model.gguf. Use pylolm info /path/to/model.gguf to see a model's details and chat template, and --chat-template NAME if its template is not recognised.

API

Errors: LibraryNotFoundError, ModelLoadError, ContextOverflowError, GenerationError (all subclass PyLoLmError).

usage in a non-streaming result includes cached_prompt_tokens: how many prompt tokens were reused from the previous turn.

Chat templates. Every chat model expects its own prompt format. pylolm handles this for you: it first tries llama.cpp's built-in templates (ChatML/Qwen, Llama 3, Gemma 2/3, ...), and if the model's template is not one of those (typical for brand-new models) it renders the template stored inside the GGUF file itself, in Python. That second path needs the small optional jinja2 package: pip install "pylolm[jinja]".

Use the chat_template= argument to override: "jinja" (always use the model's own template), a built-in name such as "chatml" or "gemma", or your own Jinja string. llm.format_chat(messages) shows the exact prompt text that will be sent.

Debugging a model: pylolm info model.gguf (no native libraries needed) prints its metadata and chat template, and pylolm template model.gguf loads it and shows how prompts get formatted.

Using it in apps

UI threads. Generation blocks, so run it off the UI thread. A streaming generator is easiest:

import threading

def worker():
    for chunk in llm.create_chat_completion(messages, stream=True):
        text = chunk["choices"][0]["delta"].get("content")
        if text:
            Clock.schedule_once(lambda dt, t=text: append_to_label(t))  # Kivy
            # page.run_thread / page.update(...) for Flet

threading.Thread(target=worker, daemon=True).start()

Call llm.stop() from your Stop button; breaking out of the loop also stops generation.

Flet. flet build apk installs packages from PyPI using the Android platform tag, so the android_24_arm64_v8a wheel is what ships the native libraries. This path needs on-device verification for your Flet/Python version.

Kivy / python-for-android. p4a does not consume Android wheels yet. For now, bundle the .so files with your app and point PYLOLM_LIB_DIR at them before importing pylolm:

import os
os.environ["PYLOLM_LIB_DIR"] = os.path.join(os.path.dirname(__file__), "pylolm_libs")
import pylolm

A proper p4a recipe is on the roadmap.

Building from source

git clone https://github.com/aura-way/pylolm && cd pylolm
pip install -e ".[dev]"
python -m pytest                                   # pure-Python tests, no model needed
PYLOLM_MODEL=/path/model.gguf python -m pytest     # + integration tests

Native libraries: src/pylolm/native/build_termux.sh (on a phone) or build_android_ndk.sh (cross-compile). The llama.cpp version is pinned in src/pylolm/native/LLAMA_CPP_REF.

Releasing to PyPI

1. Set LLAMA_CPP_REF to a llama.cpp tag/commit you have tested.
2. Bump version in pyproject.toml and add a CHANGELOG.md entry.
3. On PyPI, add a trusted publisher for this repo (workflow release.yml, environment pypi).
4. git tag v0.1.0 && git push --tags. CI builds the sdist, a pure wheel, and the Android arm64 wheel, publishes them to PyPI, and attaches pylolm-libs-android-arm64-v8a.tar.gz (+ .sha256) to the GitHub Release for pylolm get-libs. The tag must match the version in pyproject.toml.

Roadmap

• p4a recipe for Kivy; verified Flet build
• whisper.cpp (speech-to-text) module
• TTS and OCR modules
• GPU backends (Vulkan/OpenCL) as an opt-in

License

MIT. llama.cpp is MIT-licensed too; model licenses are your responsibility.