pylolm (LoLm)

Local LLMs on Android, from Python. An Android-friendly take on llama-cpp-python, built on llama.cpp and designed for Kivy, Flet and Termux.

• Tiny footprint: a thin ctypes layer over a small C shim; the only Python dependency is jinja2.
• llama-cpp-python-style API: Llama(...), create_chat_completion(...), create_completion(...).
• Streaming generators, stop strings, a stop() button for your UI.
• Multi-turn chat that reuses the KV cache, so long conversations stay fast.
• Aimed at small GGUF models (0.5B-2B) on phones.

Status: alpha (0.1.0). Text generation only. TTS / STT / OCR are on the roadmap.

Install

pip install pylolm

That is all. It brings everything with it:

• Chat-template support (jinja2) is installed as a normal dependency.
• The native libraries come inside the Android arm64 wheel, which pip selects automatically on Android (including Termux with Python 3.13+).
• If your Python ends up with the library-free wheel instead (older Termux Pythons, building from source), pylolm downloads the prebuilt libraries once, on first use (checksum-verified, from the matching GitHub Release) and tells you it is doing so. This never happens inside an app (APK), and you can switch it off with PYLOLM_AUTO_DOWNLOAD=0.

Check the result any time with pylolm doctor.

Other ways to get the libraries: pylolm get-libs (download now), or compile for your exact device (slow, ~30 min):

pkg install cmake clang git
pylolm build-native

Libraries are searched in $PYLOLM_LIB_DIR, pylolm/lib/, site-packages/opt/lib/, the app's native-library folder, then ~/.pylolm/lib/.

Quick start

from pylolm import Llama

llm = Llama("/sdcard/models/Qwen2.5-0.5B-Instruct-Q4_K_M.gguf", n_ctx=2048)

messages = [{"role": "user", "content": "Explain GGUF in two sentences."}]

# whole answer
out = llm.create_chat_completion(messages, max_tokens=200)
print(out["choices"][0]["message"]["content"])

# streaming
for chunk in llm.create_chat_completion(messages, max_tokens=200, stream=True):
    piece = chunk["choices"][0]["delta"].get("content")
    if piece:
        print(piece, end="", flush=True)

Try it from the terminal: pylolm chat /path/to/model.gguf. Use pylolm info /path/to/model.gguf to see a model's details and chat template, and --chat-template NAME if its template is not recognised.

API

Errors: LibraryNotFoundError, ModelLoadError, ContextOverflowError, GenerationError (all subclass PyLoLmError).

usage in a non-streaming result includes cached_prompt_tokens: how many prompt tokens were reused from the previous turn.

Chat templates. Every chat model expects its own prompt format. pylolm handles this for you: it first tries llama.cpp's built-in templates (ChatML/Qwen, Llama 3, Gemma 2/3, ...), and if the model's template is not one of those (typical for brand-new models) it renders the template stored inside the GGUF file itself, in Python (using jinja2, which is installed with pylolm).

Use the chat_template= argument to override: "jinja" (always use the model's own template), a built-in name such as "chatml" or "gemma", or your own Jinja string. llm.format_chat(messages) shows the exact prompt text that will be sent.

Debugging a model: pylolm info model.gguf (no native libraries needed) prints its metadata and chat template, and pylolm template model.gguf loads it and shows how prompts get formatted.

Using it in apps

UI threads. Generation blocks, so run it off the UI thread. A streaming generator is easiest:

import threading

def worker():
    for chunk in llm.create_chat_completion(messages, stream=True):
        text = chunk["choices"][0]["delta"].get("content")
        if text:
            Clock.schedule_once(lambda dt, t=text: append_to_label(t))  # Kivy
            # page.run_thread / page.update(...) for Flet

threading.Thread(target=worker, daemon=True).start()

Call llm.stop() from your Stop button; breaking out of the loop also stops generation.

Flet. flet build apk installs your dependencies for Android from PyPI, so the android_24_arm64_v8a wheel is picked automatically. Its libraries sit in opt/lib/, which is the folder Flet's packager copies into the app's native-library directory. Add this to your app's pyproject.toml:

[project]
dependencies = ["flet", "pylolm"]

[tool.flet.android]
legacy_packaging = true      # extract the .so files so llama.cpp can find its CPU backends
target_arch = ["arm64-v8a"]  # pylolm ships arm64 libraries only

A complete streaming chat app is in examples/flet_chat. Check with the Diagnostics button (or pylolm.doctor) that the native libraries loaded.

Kivy / python-for-android. p4a does not consume Android wheels yet. For now, bundle the .so files with your app and point PYLOLM_LIB_DIR at them before importing pylolm:

import os
os.environ["PYLOLM_LIB_DIR"] = os.path.join(os.path.dirname(__file__), "pylolm_libs")
import pylolm

A proper p4a recipe is on the roadmap.

Building from source

git clone https://github.com/aura-way/pylolm && cd pylolm
pip install -e ".[dev]"
python -m pytest                                   # pure-Python tests, no model needed
PYLOLM_MODEL=/path/model.gguf python -m pytest     # + integration tests

Native libraries: src/pylolm/native/build_termux.sh (on a phone) or build_android_ndk.sh (cross-compile). The llama.cpp version is pinned in src/pylolm/native/LLAMA_CPP_REF.

Releasing to PyPI

1. Set LLAMA_CPP_REF to a llama.cpp tag/commit you have tested.
2. Bump version in pyproject.toml and add a CHANGELOG.md entry.
3. On PyPI, add a trusted publisher for this repo (workflow release.yml, environment pypi).
4. git tag v0.1.0 && git push --tags. CI builds the sdist, a pure wheel, and the Android arm64 wheel, publishes them to PyPI, and attaches pylolm-libs-android-arm64-v8a.tar.gz (+ .sha256) to the GitHub Release for pylolm get-libs. The tag must match the version in pyproject.toml.

Roadmap

• p4a recipe for Kivy; verified Flet build
• whisper.cpp (speech-to-text) module
• TTS and OCR modules
• GPU backends (Vulkan/OpenCL) as an opt-in

License

MIT. llama.cpp is MIT-licensed too; model licenses are your responsibility.