rishi

rishi lets you talk to a language model from Python with the same few calls, whether the model runs on your own machine or on someone else’s server. Each way of running a model is a backend. Four run models locally: LiteRT-LM, llama.cpp, MLX and Ollama. Three send the conversation elsewhere: to a hosted API such as OpenAI’s or Anthropic’s, to your Claude Code installation, or to GitHub Copilot. Each of these has its own message format, its own way of passing images and tool calls, and its own idea of what a conversation is.

rishi keeps one conversation format for all of them. The conversation, called the history, is a plain Python list of message dicts in the OpenAI shape ({'role': 'user', 'content': 'hi'}), stored on chat.hist. Every backend reads and writes that list, so you can start a conversation on a small local model and hand the list to a hosted one, which carries on as though it had been there from the start.

The parts every backend shares come from Urai, published on PyPI as uraiyadal: the Chat class, tool calling, streaming, options and recorded replies. rishi adds the seven backends and registers them with Urai. from rishi import * gives you Urai’s names as well as rishi’s.

Install

The base package includes LiteRT-LM and the three backends that send the conversation elsewhere. The other local runtimes are extras, so you install only the ones you use:

pip install rishi                # LiteRT-LM, hosted APIs, Claude Code and GitHub Copilot
pip install 'rishi[llama]'       # llama.cpp, for GGUF model files
pip install 'rishi[mlx]'         # MLX, on Apple Silicon Macs
pip install 'rishi[mlx-vlm]'     # MLX vision and audio models
pip install 'rishi[ollama]'      # Ollama; on Linux before Python 3.14 this adds zstandard to unpack Ollama
pip install 'rishi[all]'         # every extra above

Extras combine, as in rishi[llama,mlx]. A backend module is imported only when a chat first needs it, so an extra you never installed costs nothing until you ask for that backend. The hosted backends use their usual credentials: an API key in the environment for hosted APIs, a signed-in claude command for Claude Code, and a GitHub sign-in for Copilot.

Chat

Chat(model) opens one conversation. The model name decides the backend. rishi recognises a .litertlm build, a -GGUF repo, an mlx-community repo, an Ollama hf.co/ id, or a hosted name such as gpt-5.1. A prefix such as claude/ or copilot/ names the backend outright. With no model at all you get LiteRT-LM and its default model, Gemma 4 E2B. The core page lists the exact rules.

Calling the chat with a prompt sends one turn and returns the reply, a dict Urai calls a Resp. Each call appends the prompt and the reply to chat.hist, so the second question below is asked with the first exchange in view. resp_text pulls the answer text out of a reply, and close() frees the model’s memory when you are done.

chat = Chat(gemma4_e2b)
print(resp_text(chat('Give one fact about lobsters.')))
print(resp_text(chat('Give one more.')))
chat.close()

Lobsters are crustaceans, which means they have a hard exoskeleton and eight legs.
Lobsters are known for their ability to change their color and texture to blend in with their surroundings, a behavior called camouflage.

Streaming

A long reply is easier to follow as it arrives. Pass stream=True and the call returns an iterator of markdown strings instead of one reply. Urai renders every backend’s events into the same markdown, so thinking and tool calls look alike whichever backend produced them. Pass stream='raw' if you want the chunk dicts that markdown is built from.

chat = Chat(gemma4_e2b)
for markdown in chat('Write a two-line poem about rain.', stream=True):
    print(markdown, end='', flush=True)
chat.close()

Soft drops tap on the pane,
Washing the world clean again.

Images

To ask about a picture, pass a list instead of a string, with the prompt text and the image side by side. The image can be a Path, raw bytes or a PIL.Image, and every backend that takes images accepts the same list. The model has to be one that can see. Gemma 4 on LiteRT-LM can, and each backend’s page names others.

chat = Chat(gemma4_e2b)
image = repo_root()/'nbs/images.jpeg'
display(Image.open(image))
print(resp_text(chat(['Describe this image in one sentence.', image])))
chat.close()

A medium-sized, reddish-brown German Shepherd is looking attentively off to the side with its mouth slightly open.

Audio

Audio goes in the same way, as a file path or bytes next to the prompt. The model needs an audio encoder. Gemma 4 on LiteRT-LM has one, and so do some llama.cpp and MLX models. Claude Code, Ollama and text-only models raise an error instead of dropping the clip.

chat = Chat(gemma4_e2b)
audio = repo_root()/'nbs/speech.wav'
print(resp_text(chat(['Transcribe this clip.', audio])))
chat.close()

Dancing in the masquerade, idol truth in plain sight jaded, pop, roll, click, dot, who will I be today or not? But such a tide as moving seems asleep, too full for sound and foam, when that drew from out the boundless deep turns again home, twilight and evening bell and after that.

One conversation across seven backends

Because every backend keeps the same history format, a conversation can move between them. The example below gives a code word to Gemma on LiteRT-LM, then passes the history along a chain: Qwen3 on Ollama, on MLX and on llama.cpp, GPT-4.1 through the hosted API, Claude Haiku through Claude Code, and GPT-4.1 through Copilot. Each leg opens a new Chat with messages=history, asks for the code word, and hands the longer history to the next leg. Every leg answers “amber”, which it could only know from the history it was given.

Running it downloads four local models, and the last three legs need their credentials.

legs = [
    ('LiteRT', gemma4_e2b, {}),
    ('Ollama', f'ollama/{ollama_qwen}', {'think': True}),
    ('MLX', mlx_qwen, {}),
    ('llama.cpp', llama_qwen, {'n_ctx': 4096}),
    ('cloud', 'gpt-4.1', {}),
    ('Claude', 'claude/haiku', {}),
    ('Copilot', 'copilot/gpt-4.1', {}),
]

history = []
for i, (name, model, options) in enumerate(legs):
    chat = Chat(model, messages=history, **options)
    prompt = (
        'The code word is amber. Start a travel log with one short sentence.'
        if i == 0 else
        f'What is the code word? Add one short sentence saying {name} received this conversation.'
    )
    reply = chat(prompt)
    print(f'{name}: {resp_text(reply)}')
    history = list(chat.hist)
    chat.close()

print([m['role'] for m in history])

LiteRT: Amber awaits.
Ollama: The code word is **amber**. Ollama received this conversation.

Fetching 9 files:   0%|                                                                                                                 | 0/9 [00:00<?, ?it/s]Fetching 9 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:00<00:00, 1912.59it/s]

MLX: The code word is **amber**. MLX received this conversation.

llama_context: n_ctx_seq (4096) < n_ctx_train (40960) -- the full capacity of the model will not be utilized

llama.cpp: The code word is **amber**. llama.cpp received this conversation.
cloud: The code word is amber. Cloud received this conversation.
Claude: The code word is amber. Claude received this conversation.
Copilot: The code word is amber. Copilot received this conversation.
['user', 'assistant', 'user', 'assistant', 'user', 'assistant', 'user', 'assistant', 'user', 'assistant', 'user', 'assistant', 'user', 'assistant']

Backend guides

Each backend has a page for what is particular to it.