koda

Run any LLM locally. One command.

Koda downloads and runs quantized LLMs on your machine. No cloud, no API keys, no Docker. It speaks the Ollama and OpenAI protocols, so any compatible client works out of the box.

koda pull llama3.2
koda run llama3.2

Inspired by Ollama. Built with llama.cpp + FastAPI.

Requirements

• Python 3.12+
• RAM: 4 GB minimum (8 GB+ recommended for 7B models)
• Disk: varies by model — 2–5 GB per model
• GPU: optional but recommended — CUDA (Linux/Windows) or Metal (Apple Silicon)

Install

macOS / Linux

curl -fsSL https://raw.githubusercontent.com/rjcuff/koda/main/install.sh | bash

The installer detects your platform and automatically builds llama-cpp-python with GPU support (CUDA or Metal) if available. After install, run:

source ~/.bashrc   # or ~/.zshrc on zsh
koda version

Windows

irm https://raw.githubusercontent.com/rjcuff/koda/main/install.ps1 | iex

Manual

git clone https://github.com/rjcuff/koda
cd koda
python3 -m venv .venv && source .venv/bin/activate
pip install -e .

GPU support (optional but faster)

CUDA (Linux / Windows):

CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --no-cache-dir
pip install -e .

Apple Silicon (Metal):

CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python --no-cache-dir
pip install -e .

CPU-only works fine if you skip the above — inference is just slower.

Quick Start

# 1. Download a model
koda pull llama3.2

# 2. Chat interactively
koda run llama3.2

# 3. Or start the API server
koda serve

Type /bye to exit an interactive session.

Commands

Available Models

All models use Q4_K_M quantization — a good balance of quality and size. Run koda list --available to see the current list.

API Server

koda serve
# Listening on http://127.0.0.1:11434

The server implements both the Ollama and OpenAI API protocols, so you can point any compatible client at it without any code changes.

Ollama-compatible endpoints

# List downloaded models
curl http://localhost:11434/api/tags

# Text generation
curl http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3.2", "prompt": "Why is the sky blue?"}'

# Chat completion
curl http://localhost:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3.2", "messages": [{"role": "user", "content": "Hello"}]}'

# List running models
curl http://localhost:11434/api/ps

# Pull a model
curl http://localhost:11434/api/pull \
  -H "Content-Type: application/json" \
  -d '{"name": "llama3.2"}'

# Delete a model
curl -X DELETE http://localhost:11434/api/delete \
  -H "Content-Type: application/json" \
  -d '{"name": "llama3.2"}'

OpenAI-compatible endpoints

Drop-in replacement for any client that supports a custom base URL:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="koda")

response = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Hello"}],
)
print(response.choices[0].message.content)

# List models
curl http://localhost:11434/v1/models

# Chat completion
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3.2", "messages": [{"role": "user", "content": "Hello"}]}'

Streaming works on both protocols — set "stream": true in the request body.

Python Library

Use Koda directly in your Python code. No server, no daemon, no subprocess.

from koda import Koda

k = Koda()

# Download a model (no-op if already present)
k.pull("llama3.2")

# Chat — returns a string
reply = k.chat("llama3.2", [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is 2 + 2?"},
])
print(reply)

# Streaming chat — returns a token iterator
for token in k.chat("llama3.2", [{"role": "user", "content": "Tell me a story"}], stream=True):
    print(token, end="", flush=True)

# Raw text completion (no chat template applied)
text = k.generate("llama3.2", "The capital of France is")
print(text)

# Streaming completion
for token in k.generate("llama3.2", "Once upon a time", stream=True):
    print(token, end="", flush=True)

# Manage loaded models
print(k.models())    # all models in the registry
print(k.loaded())    # models currently in memory
k.unload("llama3.2") # free memory

Custom context window:

k = Koda(n_ctx=8192)

Kodafile

A Kodafile is a YAML config file that defines a model's behavior — useful for project-specific assistants or repeatable setups.

koda create            # writes a Kodafile template in the current directory
koda run --kodafile Kodafile

Kodafile format:

base: llama3.2
system: You are a concise coding assistant. Respond in plain text, no markdown.
parameters:
  n_ctx: 8192
  temperature: 0.7
  top_p: 0.9
  repeat_penalty: 1.1

Project Structure

koda/
├── koda/
│   ├── api.py          # Python library API (Koda class)
│   ├── cli.py          # CLI commands: pull, list, run, serve, create
│   ├── config.py       # Paths and defaults (~/.koda/)
│   ├── inference.py    # Model loading + thread-safe in-memory cache
│   ├── kodafile.py     # Kodafile YAML config format
│   ├── pull.py         # HuggingFace model downloads
│   ├── registry.py     # Model name → HuggingFace repo mapping
│   └── server.py       # FastAPI server — Ollama + OpenAI APIs
├── install.sh          # macOS / Linux one-line installer
├── install.ps1         # Windows one-line installer
└── pyproject.toml

Models are stored in ~/.koda/models/.

Stack

Contributing

See CONTRIBUTING.md for how to add models, endpoints, and commands.

License

MIT