geistr

Run language models on your own computer: download, chat, serve. One small binary, no Python, no Docker, no account. The model runs locally, on the CPU or the GPU.

Full manual: docs/GUIDE.md: every command, chat command, setting and file, the HTTP API, and what to do when something fails.

Install

curl -fsSL https://www.geisten.net/download/geistr/install.sh | sh

Or with Homebrew:

brew install geisten/tap/geistr

Both work on Linux (x86_64, arm64) and macOS (Apple Silicon). The script downloads the latest release, checks it against SHA256SUMS and puts geistr in /usr/local/bin.

Without sudo, or a specific release (the server keeps the last 5, from 0.3.2 on; another one is reported as "no release …"):

curl -fsSL https://www.geisten.net/download/geistr/install.sh | PREFIX=~/.local sh
curl -fsSL https://www.geisten.net/download/geistr/install.sh | GEISTR_VERSION=v0.4.0 sh

Requirements

The Linux binaries are fully static and run on any distribution. The CPU needs x86-64-v3 (AVX2: Intel since 2013, AMD since 2015) or ARMv8.2 with dotprod (Raspberry Pi 5, AWS Graviton 2 and newer); on an older CPU geistr says so and exits. To build from source instead, see docs/DEVELOPMENT.md.

Quick start

geistr chat                    # the first time: the models that run here, the best
                               # first; choose with ↑↓ and Enter, it downloads, verifies
                               # and chats

Or choose yourself:

geistr catalog                 # which models fit this computer
geistr chat qwen3-0.6b         # talk to it; not installed yet: Enter downloads it (640 MB)

geistr pull qwen3-0.6b downloads without starting anything: for scripts and servers (without a terminal geistr never downloads unasked), the embedding model of --files, and with no id, the update of all installed models after a new geistr.

Or a single answer, for scripts:

geistr run qwen3-0.6b "Name three prime numbers."
echo "Summarize: …" | geistr run qwen3-0.6b

Models

geistr catalog lists the models geistr knows, the best suited to this computer first (installed or not does not move a row), marks what is installed (✓), what is available (↓) and whether each fits your RAM (⚠/✗), with the speed measured here on ⚙ CPU and ⚡ GPU.

model	download	RAM
smollm2-360m	0.4 GB	2 GB
qwen3-0.6b	0.6 GB	4 GB
qwen35-0.8b	0.8 GB	4 GB
bitnet-2b	1.2 GB	4 GB
gemma4-e2b	3.1 GB	8 GB
gemma4-e4b	5.0 GB	16 GB
bonsai2-27b-pq2	7.2 GB	24 GB
qwen38-27b-q4	16 GB	32 GB
qwen38-27b-q8	29 GB	48 GB

Every download is checked against its SHA-256 and resumes where it stopped. Any other .gguf file works too: geistr chat path/to/model.gguf.

Chat

In a terminal, geistr chat renders Markdown, tables, math and clickable links, shows the speed after each answer and continues your last conversation (see below). Once the context is half full, the prompt says how full (⚙ 62% >, yellow from 80 %, red from 95 %); /info always shows it. Type / for the commands:

/model gemma4-e2b    switch the model, keep the conversation (best fit first)
/gpu /cpu /auto      switch the processor (Metal on macOS, Vulkan on Linux)
/temp 0.7            sampling temperature
/system Be brief.    system prompt (/system off removes it)
/info                what runs now, and how full the context is
/files ~/Notes       answers from your files (/files off: none)
/retry               the last answer again (at temperature 0: once at 0.7)
/copy  /copy code    the last answer, or its last code block, to the clipboard
/save                keep model, processor, temperature and system prompt
/clear  /exit

key	does
Ctrl-J, Alt-Enter, \ + Enter	a new line in the message (Enter sends)
Esc	stop the answer
Ctrl-C	clear the line; twice on an empty line: exit
↑ ↓	earlier lines, also from earlier chats; → takes the hint
Ctrl-R	search earlier lines (type to narrow, Ctrl-R for older, Enter takes it)
?	the shortcuts

Your conversation stays

Close the terminal, reboot, come back tomorrow: geistr chat picks up where you left off, and the model still knows what you talked about.

↻ 6 · „Compare Mars and Venus in a small table…“ · /clear new
⚡ >

That is what makes a local model useful for real work: you can think a problem through over several sessions, refer back to an earlier answer, or switch to a bigger model mid-conversation without explaining everything again. It is also private by design: each conversation is a plain file on your computer (readable only by you, one JSON message per line), never sent anywhere.

A long conversation stays fast to resume: the model re-reads only the newest part (about 2000 tokens, geistr config resume_tokens), and the ↻ line says how much. The whole conversation stays in the file. When a conversation outgrows the model's context, the oldest messages are left out, and the chat says so (↥ 12 oldest messages left out …).

• /clear starts a new conversation and keeps the old file.
• geistr chat --new starts fresh once; geistr config resume off always.
• Piped chats (echo … | geistr chat) keep nothing, so scripts stay reproducible.

Ask your own files

geistr pull bitnet-embed-0.6b      # the embedding model, 428 MB, once
geistr index ~/Notes               # text, Markdown, PDF (with pdftotext)
geistr chat --files ~/Notes        # or /files ~/Notes in a chat

Each question gets the most relevant passages of your files put in front of it, and the answer shows where they came from (sources: garden/shed.md). With the reranker installed (geistr pull qwen3-reranker-0.6b, 639 MB) only the passages that answer the question go in front of it and only their files are named; for a question your files do not answer it says nothing relevant in your files. Without it, the four nearest passages are used and named. geistr run --files ~/Notes "…" does the same once. The index is updated by file hash whenever you use the folder, so only new and changed files are read again. It stays on your computer, readable only by you, in the data folder (index/); hidden files and folders are left out.

Decisions

Some questions have a fixed set of answers: which team gets a ticket, is a text urgent. geistr decide reads the model's choice among your options from one pass over the prompt, without writing an answer:

$ geistr decide gemma4-e2b "I was charged twice. Which team handles this?" billing technical sales
billing
  billing    1.00
  technical  0.00
  sales      0.00

Piped, it prints only the chosen option, for scripts. The probabilities are the model's own, read from the option keys; nothing else can come out.

With --threshold 0.9, a decision below that probability is not taken: geistr decide exits with 3, or with --escalate chat the same model writes the answer (thinking, if it can), or with --escalate bonsai2-27b-pq2 a larger model decides. --max-tokens N caps the chat answer and --temperature T applies your calibration. Where to set the threshold and the cap is your application's trade between speed and quality, and it depends on the task and the model. docs/DECISIONS.md shows how to measure it, with one measured curve.

It works with every chat model: the prompt follows the model's own chat template. For gemma4-e2b and bonsai2-27b-pq2 the decisions are also checked against an independent engine (llama.cpp); for other models geistr says "unverified". EXPERIMENTAL: details, limits and evidence in docs/DECISIONS.md.

Use it from other tools

Keep a model loaded and talk to it over the OpenAI or Ollama API, e.g. from Open WebUI, editor plugins or the OpenAI SDK:

geistr serve gemma4-e2b --http          # http://127.0.0.1:11434
geistr serve --http                     # every installed model, one loaded at a time

curl http://127.0.0.1:11434/v1/chat/completions \
  -d '{"messages":[{"role":"user","content":"Hi"}]}'

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:11434/v1", api_key="unused")
reply = client.chat.completions.create(model="gemma4-e2b",
                                       messages=[{"role": "user", "content": "Hi"}])
print(reply.choices[0].message.content)

In Open WebUI, add http://127.0.0.1:11434 as an Ollama connection (or …/v1 as OpenAI). Clients that resend the whole conversation pay only for what is new.

Without a model, geistr serve --http offers every installed model (Open WebUI lists them) and the request's model chooses. It is built for small computers: only one model is in memory. A request for another model waits while the loaded one is unloaded completely and the requested one is loaded (qwen3-0.6b ⇄ gemma4-e2b: 0.5–0.7 s on an M1 Max's GPU; not yet measured on a Raspberry Pi 5); its conversations are then processed anew, which takes longer, but the answers are the same.

Endpoints, the socket, and security

endpoint
POST /v1/chat/completions	OpenAI: messages, temperature, max_tokens / max_completion_tokens, stop, stream (SSE, stream_options.include_usage), tools / tool_choice
POST /v1/embeddings	OpenAI: input (a text or a list), encoding_format (float, base64); with an embedding model
GET /v1/models	the model, or without one every installed model
POST /api/chat	Ollama: messages, stream (NDJSON, default), options.temperature, num_predict, stop, tools
POST /api/embed	Ollama: input (a text or a list); with an embedding model
GET /api/tags, /api/version, /	the model(s), the version, a health check

There is no authentication. By default geistr listens on loopback only and answers only requests addressed to this computer (Host check, against DNS rebinding). A web page in a browser may use it only if it comes from this computer too (Origin check: another site's page cannot start answers); GEISTR_ORIGINS=https://ui.example,… (or *) allows more, like Ollama's OLLAMA_ORIGINS. --http=0.0.0.0:11434 makes it reachable for everyone who can reach the computer; geistr warns about that. One request is answered at a time; a client that disconnects stops its answer.

Tool calling works with Qwen3 models (e.g. qwen3-0.6b): send tools, get tool_calls back (finish_reason: tool_calls; streamed too), send the results as role: tool messages, and the model answers with them, as in the OpenAI SDK's function-calling example. tool_choice: "none" turns tools off for a request. A model without a tool format refuses tools (400).

Images work with Gemma 4 E2B (gemma4-e2b): geistr pull gemma4-e2b also fetches its vision tower (the vision tensors of the original model, a 337 MB vision_tower.safetensors next to it, SHA-256 verified). Send an image_url part with a data: URL (OpenAI) or images (Ollama), PNG, JPEG or BMP, one per message; the image goes with the last message, and the conversation keeps it for the follow-up questions. Other models refuse images (400).

Embeddings: geistr serve bitnet-embed-0.6b --http answers /v1/embeddings (the OpenAI SDK's embeddings.create works) and Ollama's /api/embed: unit-length vectors of 1024 dimensions, multilingual. For retrieval, embed the question and the documents as they are, with nothing before them, as --files does: on scripts/retrieval-bench.py (#157) that put the right file among the four nearest passages most often (97 %), ahead of an Instruct: …\nQuery: task before the question (92 %) or query: / passage: (89–94 %). An embedding model does not chat, and a chat model does not embed (400). geistr serve <model> runs that one model, the one in /v1/models; a request that names another gets 404 (model_not_found) instead of this model's answer. geistr serve --http without a model loads the one a request names; one that is not installed gets 404 (a service downloads nothing: geistr pull it).

Without --http, geistr serve listens on a Unix socket only (owner only, 0600), and geistr chat --socket chats through it. --chats N sets how many conversations the service keeps (default 2), --max-request=MiB the size of one request (default 16). The options and the socket protocol: docs/GUIDE.md.

Settings

Settings are remembered between runs; geistr config shows them and the file they live in.

geistr config processor gpu                # auto (default), cpu, gpu
geistr config temperature 0.7              # 0 to 2
geistr config system "Answer in German."   # system prompt for every new chat
geistr config markdown off                 # plain text
geistr config stats off                    # no speed line
geistr config resume off                   # always start new, keep nothing
geistr config history off                  # don't keep typed lines between chats
geistr config threads 8                    # CPU threads (default: one per physical core)

--cpu / --gpu choose the processor for one run, --threads N the CPU threads, --models DIR another model folder.

All commands

command	does
geistr catalog [--installed | --available] [--json]	the models, their state, fit and speed
geistr pull <id>	download, resume, verify
geistr pull	update the installed models after a geistr update
geistr run <model> [prompt…]	one answer to stdout; the prompt from stdin if none
geistr chat [<model>] [--new]	interactive; without a model the last one
geistr serve <model> [--http[=ADDR:PORT]] [--socket=PATH] [--chats N] [--max-request=MiB]	the model as a service
geistr serve --http[=ADDR:PORT] […]	every installed model as a service, one loaded at a time
geistr chat --socket[=PATH]	chat with that service
geistr config [key [value]]	settings
geistr bench [model…]	tokens/s on CPU and GPU (default: every installed model)
geistr bench --compare [A [B]]	two engine versions' speeds side by side
geistr --version	geistr, catalog revision, engine commit

<model> is a catalog id or a path to a .gguf file. Options: --cpu / --gpu, --models DIR, --catalog FILE. Exit codes: 0 ok, 1 error, 2 usage, 130 cancelled.

Every complete answer records its speed in speed.tsv in the data folder; geistr catalog shows the median of the last ten geistr bench runs (of the last ten answers before the first).

Update

brew upgrade geistr, or run the install command again. The model list ships inside geistr, so new models come with a new geistr. Afterwards geistr catalog marks installed models whose file changed with ⟳ and a file of the right size with another checksum (damaged?) with ✗, and geistr pull fetches them (the old file stays until the new one is verified).

Python and C

geistr is built on geist-runtime, an embeddable model runner on geistlib with one C API (include/geistr.h) and a Python package (pip install geistr; every release also has its wheels at https://www.geisten.net/download/geistr/vX.Y.Z/):

import geistr

with geistr.chat("gemma4-e2b", system="Answer briefly.") as chat:
    for piece in chat.send("What is the capital of France?"):   # streamed
        print(piece, end="", flush=True)
    print(chat.ask("And of Italy?"))           # only the new message is processed

Using geistr: docs/GUIDE.md. Building, the library, tests and CI: docs/DEVELOPMENT.md. API reference and design: docs/API.md.