kiapi

English | 日本語

Summary

kiapi is a local API server that uses Apple Silicon and MLX to provide generative AI capabilities for LLM agents.

API

See: kiapi API Docs

Requirements

• macOS / Apple Silicon
• Python >=3.12,<3.13
• uv (optional, recommended for isolated CLI installs and faster venv/package setup in kiapi activate)
• mise (used for development)
• Docker (when using the Web capability)
• Enough disk capacity for model weights and Docker images

kiapi is developed mainly for personal use on a Mac Studio M4 Max 128GB. Some or all features may work on other Apple Silicon environments, but they are not the primary verification target.

The memory budget can be specified with KIAPI_MEMORY_LIMIT_GB. If omitted, kiapi automatically uses 80% of installed memory as the effective budget on startup. If a model's required memory does not fit in that budget, requests return 503 as an insufficient memory budget error.

kiapi activate --all uses a little under 600GB of disk capacity, including model weights and Docker images. At first, it is recommended to use kiapi activate to set up only the capabilities you need.

Remote Job Relay

The optional GCP relay lets a kiapi node inside a closed network receive API work without exposing an inbound socket. Firebase Realtime Database carries small notifications, while Cloud Storage carries request and response bodies. Install kiapi with the relay-gcp extra to enable it:

python3.12 -m pip install --upgrade "kiapi[relay-gcp]"
uv tool install --python 3.12 "kiapi[relay-gcp]"

export KIAPI_RELAY_GCP_DATABASE_URL="https://PROJECT.firebaseio.com"
export KIAPI_RELAY_GCP_BUCKET="PRIVATE_RELAY_BUCKET"

# Uses Application Default Credentials by default.
kiapi run --relay gcp

Each node generates its own node_id on first start, persists it in the user data directory, and publishes a liveness heartbeat under liveness. A requester picks the most recently seen node and writes sessions/{session_id}/request.json in GCS, then writes a notification to nodes/{node_id}/requests/{session_id} in RTDB. The relay reports queued, running, and the terminal result below the requester node's responses path.

• Requests are dispatched directly to the in-process FastAPI app and handled one at a time by the relay.
• JSON responses use response.json. Binary responses write response.body before response.json.
• response.json uses a GCS create-only generation precondition. A completed response found after restart is reported without executing the API again.
• The terminal RTDB response and request deletion use one atomic multi-location update.
• Startup installs a session-scoped lifecycle rule that deletes session objects after one day. Set KIAPI_RELAY_GCP_MANAGE_BUCKET_LIFECYCLE=false when infrastructure manages this rule.

Use a dedicated bucket and narrowly scoped RTDB/GCS permissions. Google credentials are configured through kiarina-lib-google.

See GCP Relay setup for resource creation, IAM, authentication, configuration, and verification steps.

For local relay verification without GCP, use local. It uses the same in-process dispatch path but stores notifications and payloads under a local directory:

export KIAPI_RELAY_LOCAL_ROOT="/tmp/kiapi/relay"
export KIAPI_RELAY_LOCAL_PREFIX="private/kiapi"

kiapi run --relay local

The requester writes {root}/{prefix}/sessions/{session_id}/request.json, then writes {root}/{prefix}/nodes/{node_id}/requests/{session_id}.json with {"session_id":"...","source_node_id":"..."}. The relay writes bridge status to {root}/{prefix}/nodes/{source_node_id}/responses/{session_id}.json and stores the committed response in the session directory.

Local Storage

kiapi mainly writes to these local paths at runtime.

Other model weights and library caches are managed by Hugging Face, mflux, Docker, or each library/tool. kiapi generally does not move them into its own storage location.

Security

By default, kiapi run starts on 127.0.0.1:8000. When --host 0.0.0.0 is specified, the server may be reachable from other machines, so use it only on trusted networks.