rishi

rishi lets you use a language model from Python with the same calls, whether the model runs on your machine or on a remote server. Each way of running a model is a backend. Four backends run models locally: LiteRT-LM, llama.cpp, MLX and Ollama. Three backends send the conversation to another program or server: a hosted API such as OpenAI or Anthropic, your Claude Code installation, and GitHub Copilot. Each of these seven has its own message format and its own way to pass images and tool calls.

rishi uses one conversation format for all of them. The conversation is called the history. The history is a Python list of message dicts in the OpenAI format, such as {'role': 'user', 'content': 'hi'}. It is stored in chat.hist. Every backend reads and writes that list. You can start a conversation on a small local model, give the list to a hosted model, and the hosted model continues the conversation.

The parts that all backends share come from Urai, which is published on PyPI as uraiyadal. These are the Chat class, tool calling, streaming, options and recorded replies. rishi adds the seven backends and registers them with Urai. from rishi import * gives you the names from Urai and from rishi.

Install

The base package includes LiteRT-LM and the three backends that send the conversation to another program or server. The other local backends need optional extras. Install only the extras you use:

pip install rishi                # LiteRT-LM, hosted APIs, Claude Code and GitHub Copilot
pip install 'rishi[llama]'       # llama.cpp, for GGUF model files
pip install 'rishi[mlx]'         # MLX, on Apple Silicon Macs
pip install 'rishi[mlx-vlm]'     # MLX vision and audio models
pip install 'rishi[ollama]'      # Ollama; on Linux before Python 3.14 this also installs zstandard
pip install 'rishi[all]'         # every extra above

You can combine extras, as in rishi[llama,mlx]. rishi imports a backend module only when a chat first needs it. A missing extra causes no error until you ask for that backend. The hosted backends use their usual credentials. Hosted APIs need an API key in the environment. Claude Code needs a signed-in claude command. Copilot needs a GitHub sign-in.

Chat

Chat(model) opens one conversation. The model name selects the backend. rishi recognizes a .litertlm build, a -GGUF repo, an mlx-community repo, an Ollama hf.co/ id, and a hosted name such as gpt-5.1. A prefix such as claude/ or copilot/ selects the backend directly. With no model, Chat uses LiteRT-LM and its default model, Gemma 4 E2B. The core page lists the exact rules.

Call the chat with a prompt to send one turn. The call returns the reply, a dict that Urai calls a Resp. Each call appends the prompt and the reply to chat.hist. The second question below is therefore asked with the first exchange in the history. resp_text returns the answer text from a reply. close() frees the model memory.

chat = Chat(gemma4_e2b)
print(resp_text(chat('Give one fact about lobsters.')))
print(resp_text(chat('Give one more.')))
chat.close()

Lobsters are crustaceans, which means they have a hard exoskeleton and eight legs.
Lobsters are known for their ability to change their color and texture to blend in with their surroundings, a behavior called camouflage.

Streaming

Pass stream=True to receive a reply as it is generated. The call then returns an iterator of markdown strings instead of one reply. Urai renders the events of every backend into the same markdown. Thinking and tool calls look the same on every backend. Pass stream='raw' to receive the chunk dicts that the markdown is built from.

chat = Chat(gemma4_e2b)
for markdown in chat('Write a two-line poem about rain.', stream=True):
    print(markdown, end='', flush=True)
chat.close()

Soft drops tap on the pane,
Washing the world clean again.

Images

To ask about a picture, pass a list instead of a string. The list holds the prompt text and the image. The image can be a Path, raw bytes or a PIL.Image. Every backend that accepts images accepts the same list. The model must accept image input. Gemma 4 on LiteRT-LM does. The page of each backend names other models.

chat = Chat(gemma4_e2b)
image = repo_root()/'nbs/images.jpeg'
display(Image.open(image))
print(resp_text(chat(['Describe this image in one sentence.', image])))
chat.close()

A medium-sized, reddish-brown German Shepherd is looking attentively off to the side with its mouth slightly open.

Audio

Pass audio in the same list as the prompt, as a file path or as bytes. The model needs an audio encoder. Gemma 4 on LiteRT-LM has one. Some llama.cpp and MLX models have one. Claude Code, Ollama and text-only models raise an error. They do not drop the clip.

chat = Chat(gemma4_e2b)
audio = repo_root()/'nbs/speech.wav'
print(resp_text(chat(['Transcribe this clip.', audio])))
chat.close()

Dancing in the masquerade, idol truth in plain sight jaded, pop, roll, click, dot, who will I be today or not? But such a tide as moving seems asleep, too full for sound and foam, when that drew from out the boundless deep turns again home, twilight and evening bell and after that.

One conversation across seven backends

Every backend uses the same history format. A conversation can move between backends. The example below has seven legs. Each leg opens a new Chat with messages=history, sends one prompt, and passes the longer history to the next leg. The first leg uses Gemma on LiteRT-LM. Its prompt gives the code word “amber”. The next six legs use Qwen3 on Ollama, Qwen3 on MLX, Qwen3 on llama.cpp, GPT-4.1 through the hosted API, Claude Haiku through Claude Code, and GPT-4.1 through Copilot. Each of them is asked for the code word and answers “amber”. It can know this only from the history.

The example downloads four local models. The last three legs need their credentials.

legs = [
    ('LiteRT', gemma4_e2b, {}),
    ('Ollama', f'ollama/{ollama_qwen}', {'think': True}),
    ('MLX', mlx_qwen, {}),
    ('llama.cpp', llama_qwen, {'n_ctx': 4096}),
    ('cloud', 'gpt-4.1', {}),
    ('Claude', 'claude/haiku', {}),
    ('Copilot', 'copilot/gpt-4.1', {}),
]

history = []
for i, (name, model, options) in enumerate(legs):
    chat = Chat(model, messages=history, **options)
    prompt = (
        'The code word is amber. Start a travel log with one short sentence.'
        if i == 0 else
        f'What is the code word? Add one short sentence saying {name} received this conversation.'
    )
    reply = chat(prompt)
    print(f'{name}: {resp_text(reply)}')
    history = list(chat.hist)
    chat.close()

print([m['role'] for m in history])

LiteRT: Amber awaits.
Ollama: The code word is **amber**. Ollama received this conversation.

Fetching 9 files:   0%|                                                                                                                 | 0/9 [00:00<?, ?it/s]Fetching 9 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:00<00:00, 1912.59it/s]

MLX: The code word is **amber**. MLX received this conversation.

llama_context: n_ctx_seq (4096) < n_ctx_train (40960) -- the full capacity of the model will not be utilized

llama.cpp: The code word is **amber**. llama.cpp received this conversation.
cloud: The code word is amber. Cloud received this conversation.
Claude: The code word is amber. Claude received this conversation.
Copilot: The code word is amber. Copilot received this conversation.
['user', 'assistant', 'user', 'assistant', 'user', 'assistant', 'user', 'assistant', 'user', 'assistant', 'user', 'assistant', 'user', 'assistant']

Backend guides

Each backend has a page for the details that apply only to it.