ailia LLM Python API

!! CAUTION !! “ailia” IS NOT OPEN SOURCE SOFTWARE (OSS). As long as user complies with the conditions stated in License Document, user may use the Software for free of charge, but the Software is basically paid software.

About ailia LLM

ailia LLM is a library for running LLMs on edge devices. It provides bindings for C++ and Unity.

Install from pip

You can install the ailia LLM free evaluation package with the following command.

pip3 install ailia_llm

Install from package

You can install the ailia LLM from Package with the following command.

python3 bootstrap.py
pip3 install ./

Usage

import ailia_llm

import os
import urllib.request

model_file_path = "gemma-2-2b-it-Q4_K_M.gguf"
if not os.path.exists(model_file_path):
	urllib.request.urlretrieve(
		"https://storage.googleapis.com/ailia-models/gemma/gemma-2-2b-it-Q4_K_M.gguf",
		model_file_path
	)

model = ailia_llm.AiliaLLM()
cpu_backend = next(backend for backend in model.get_backend_list()
                   if backend["backend_name"] == "CPU")
model.set_backend(cpu_backend["idx"])  # Optional; call before open().
model.open(model_file_path, n_ctx=8192)

messages = []
messages.append({"role": "system", "content": "語尾に「わん」をつけてください。"})
messages.append({"role": "user", "content": "あなたの名前は何ですか？"})

stream = model.generate(messages)

text = ""
for delta_text in stream:
	text = text + delta_text
print(text)

if model.context_full():
	raise Exception("Context full")

messages.append({"role": "assistant", "content": text})

Tool Use (JSON API)

Use ailiaLLMSetPromptJson from the first user message while tools are configured. The legacy ailiaLLMSetPrompt and ailiaLLMSetMultimodalPrompt return INVALID_STATE until tools are cleared. Define tools with ailiaLLMSetTools, set the JSON messages array, and call ailiaLLMGenerate until done. Deltas remain available through ailiaLLMGetDeltaText for previews; retrieving or concatenating them is optional. Retrieve assistant JSON with ailiaLLMGetResponseJsonSize / ailiaLLMGetResponseJson, append the object to history, execute complete calls, and append tool results with matching tool_call_id. Set the next JSON prompt to continue.

The SDK accumulates generated output. Reads do not consume it; a new prompt resets it.

Structured history

[
 {"role":"user","content":"Weather in Tokyo?"},
 {"role":"assistant","content":"","tool_calls":[
  {"id":"call_0","type":"function","function":{"name":"get_weather","arguments":"{\"city\":\"Tokyo\"}"}}
 ]},
 {"role":"tool","tool_call_id":"call_0","content":"Snow, -3 C"}
]

Content is not reparsed as tool syntax. Supply assistant thinking in reasoning_content. Arguments must be a string containing a JSON object. Call IDs/names must be nonempty; IDs are unique per assistant, but may be reused in later turns. Consecutive tool results match the preceding assistant by ID, not order. Unknown IDs, duplicate results and conflicting names are rejected. Tool content must be a string; serialize objects first.

Images and audio

Load a compatible projector with ailiaLLMOpenMultimodalProjectorFileA and use ordered user content parts:

[{"role":"user","content":[
 {"type":"text","text":"Describe this image and audio."},
 {"type":"image","file_path":"/path/to/image.jpg"},
 {"type":"audio","file_path":"/path/to/audio.wav"}
]}]

Each image/audio part specifies exactly one of file_path or data. Data is standard padded Base64 containing encoded file bytes (JPEG/PNG, WAV/MP3/FLAC, etc.). No URL fetching, video or raw RGB/PCM is supported. Media works with or without tools and is re-evaluated each prompt. An unloaded/incompatible projector returns INVALID_STATE. Query image/audio support with ailiaLLMGetMultimodalCapabilities.

Incomplete output and errors

GetResponseJson rejects unfinished calls/thinking with PARSE_ERROR. It never repairs arguments or returns executable partial results. SetPromptJson rejects incomplete arguments, invalid JSON/types/ID associations and empty arrays with INVALID_ARGUMENT. If GetResponseJson returns PARSE_ERROR, do not append the failed assistant. Append a retry user message to the existing history; no partial content/reasoning retrieval is needed. Do not invent tool results.

After clearing tools with SetTools(NULL), SetPromptJson still accepts historical tool calls and results for summaries or final answers. Retrieve response JSON before clearing tools, which invalidates the output parser. Use finally to clear tools even when an exception occurs. An unset prompt or changed tools/thinking returns INVALID_STATE; before generation an initialized prompt returns an empty assistant. File failures return ERROR_FILE_API, invalid Base64 returns INVALID_ARGUMENT, and undecodable media buffers return ERROR_BUFFER_API.

# Tool Use with SDK-buffered responses. Deltas are optional previews.
import json
import ailia_llm

model = ailia_llm.AiliaLLM()
model.open("gemma-4-E2B-it-Q4_K_M.gguf", n_ctx=8192)
model.set_tools([{"type":"function","function":{
    "name":"get_weather", "description":"Get the current weather.",
    "parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}
}}])
messages = [{"role":"user","content":"What is the weather in Tokyo?"}]
for turn in range(8):
    for delta in model.generate_json(messages):
        print(delta, end="", flush=True)  # optional preview; no accumulation
    response = model.get_response_json()  # raises on incomplete output
    messages.append(response)
    if not response.get("tool_calls"):
        print(response["content"])
        break
    for call in response["tool_calls"]:
        args = json.loads(call["function"]["arguments"])
        # Validate and execute the tool in the application. Mock result here.
        result = {"city": args["city"], "weather": "sunny"}
        messages.append({"role":"tool", "tool_call_id":call["id"],
                         "content":json.dumps(result)})
model.set_tools(None)

API specification

https://github.com/ailia-ai/ailia-sdk