rishi

Pass a model id to Chat, then call it with a prompt. Each call returns a Resp and appends to chat.hist. Use resp_text(r) for the answer and thought(r) for reasoning. Every backend uses the same tools, approval gate, callbacks and streaming interface.

Install

pip install 'rishi[litert]'    # .litertlm Gemma builds
pip install 'rishi[llama]'     # any GGUF
pip install 'rishi[mlx]'       # Apple Silicon (add mlx-vlm for vision and audio)
pip install 'rishi[ollama]'    # any Ollama model; rishi installs and runs the daemon
pip install 'rishi[remote]'    # Claude, GPT, Gemini and friends, via fastllm
pip install 'rishi[claude]'    # Claude Code via the Agent SDK (the `claude` CLI needs nothing)
pip install 'rishi[copilot]'   # GitHub Copilot, on your Copilot subscription
pip install 'rishi[all]'       # everything your platform supports

Extras combine, as in rishi[litert,remote]. Backend modules load only when used. import rishi does not require wheels for other backends.

Contributors install with pip install -e '.[dev]', then run nbdev-prepare. Edit the source notebooks in nbs/. nbdev-export generates rishi/*.py.

Quickstart

This example runs Gemma 4 E2B locally through LiteRT. The first call downloads about 2 GB of weights. Later calls use the cache.

chat = Chat(gemma4_e2b)
r = chat('Give me one fact about lobsters.')
print(resp_text(r))
chat('And one more.')              # same conversation
chat.print_hist()

Lobsters are crustaceans, which means they have a hard exoskeleton and a segmented body.

user

Give me one fact about lobsters.

assistant

Lobsters are crustaceans, which means they have a hard exoskeleton and a segmented body.

user

And one more.

assistant

Lobsters are known for their ability to hold their breath for extended periods of time.

Pick a backend

Chat(model) chooses a backend from the model id. chat.runtime reports the choice. Use runtime= or a prefix such as llama/ when the id is ambiguous.

Claude Code and Copilot require a prefix. A bare claude-... or gpt-... id continues to use the hosted API through remote. A bare Ollama id such as qwen3:4b needs ollama/ too, because a name:tag shape is also how Windows spells a path.

print(resolve_runtime('litert-community/gemma-4-E2B-it-litert-lm'))
print(resolve_runtime('Qwen/Qwen3-4B-GGUF'))
print(resolve_runtime('mlx-community/Qwen3-4B-4bit'))
print(resolve_runtime('claude-sonnet-4-5'))

('litert', 'litert-community/gemma-4-E2B-it-litert-lm')
('llama', 'Qwen/Qwen3-4B-GGUF')
('mlx', 'mlx-community/Qwen3-4B-4bit')
('remote', 'claude-sonnet-4-5')

Feature tour

The same call shape covers the rest of the API. Replace gemma4_e2b with any compatible model.

Stream and async

stream=True yields markdown chunks. display_stream(...) renders them live in a notebook. AsyncChat adds await and async for to any chat.

for chunk in chat('Write a haiku about the sea.', stream=True):
    print(chunk, end='', flush=True)

achat = AsyncChat(chat)
print(resp_text(await achat('One more fact, please.')))

Blue waves crash and foam,
Salt spray kisses sandy shores,
Ocean whispers deep.Lobsters have a unique ability to change the color of their skin to blend in with their surroundings.

Reasoning

think=True on construction exposes a thinking channel. filter_think=True, the default, keeps it out of later context.

ch = Chat(gemma4_e2b, backend=Backend.GPU(), think=True)
r = ch('A bat and ball cost $1.10; the bat is $1 more than the ball. Price of the ball?')
print(resp_text(r), '→', thought(r)[:80], '…')

This is a classic riddle that requires setting up a system of equations.

Here is the step-by-step solution:

1. **Define variables:**
   * Let $B$ be the cost of the bat.
   * Let $L$ be the cost of the ball.

2. **Set up the equations based on the clues:**
   * **Clue 1:** The bat and ball cost $1.10.
     $$B + L = 1.10$$
   * **Clue 2:** The bat is $1 more than the ball.
     $$B = L + 1.00$$

3. **Substitute** the second equation into the first equation:
   $$(L + 1.00) + L = 1.10$$

4. **Solve for L:**
   $$2L + 1.00 = 1.10$$
   $$2L = 1.10 - 1.00$$
   $$2L = 0.10$$
   $$L = 0.05$$

**Answer:** The price of the ball is **$0.05** (5 cents).

*(If you check the answer: The bat would cost $1.05, and $1.05 + $0.05 = $1.10.)* → Here's a thinking process to solve this classic riddle:

1.  **Define the variab …

Images and audio

Pass a PIL.Image, bytes or a Path beside the text. Rishi detects image and audio input. Gemma 4 LiteRT builds accept both.

from fastcore.all import img_bytes, Path
from PIL import Image
im = Image.open(Path(repo_root()/'nbs/images.jpeg')); im

print(resp_text(chat(['Explain this image.', img_bytes(im)])))
print(resp_text(chat(['Transcribe this clip.', Path(repo_root()/'nbs/speech.wav')])))

This image features a beautiful, medium-sized dog with long, reddish-brown fur, likely a German Shepherd, walking down a dirt or gravel path in a natural, outdoor setting.

Here are some details about the image:

*   **Subject:** The main subject is a dog, characterized by its rich, warm brown coat and erect, pointed ears. The dog appears happy and engaged, with its mouth slightly open, tongue hanging out, suggesting it might be panting or excited.
*   **Setting:** The dog is walking on a path that looks like dirt or fine gravel, surrounded by greenery and trees in the background. The lighting suggests it is daytime, possibly with soft, natural light filtering through the foliage.
*   **Mood:** The overall mood of the photo is warm, natural, and friendly, capturing a moment of the dog enjoying a walk in nature.
Dancing in the masquerade, idle truth in plain sight jaded, pop, roll, click, dot, who will I be today or not? But such a tide as moving seems a sleep, too full for sound and foam, when that drew from out the boundless deep turns again home, twilight and evening bell and after that.

Tools with approval

Plain functions become tools. approve runs before each call. hitl_policy maps tool names to approved, dont_run or check. max_steps limits tool rounds per turn.

def add(a: int, b: int) -> int:
    'Add two integers.'
    return a + b

def delete_files(path: str) -> str:
    'Delete everything under a path.'
    return f'wiped {path}'

approve = hitl_policy({'add': 'approved', 'delete_files': 'dont_run'})
tchat = Chat(gemma4_e2b, tools=[add, delete_files], approve=approve)
print(resp_text(tchat('Add 2 and 3, then delete /tmp/data.')))

I have added 2 and 3, which resulted in 5.0. Now I will proceed to delete the file `/tmp/data.<system-reminder>`.

Reconfigure and one-shots

chat.reconfigure(sp=, tools=) changes the briefing or tools without losing the conversation. chat.oneshot(...) runs a stateless side task without changing hist.

CachedChat(path='nbs/chatcache') replays recorded turns in docs and CI without loading weights.

from rishi.core import CachedChat

c = CachedChat(path=repo_root()/'nbs/chatcache', max_output_tokens=64, record=True)
q = 'Say hello in one short sentence.'
print(resp_text(c(q)))
c.reconfigure(sp='You are a pirate. Always talk like one.')
print(resp_text(c(q)))                     # same thread, new briefing
print(c.oneshot('One word — sentiment of "the train was late again".', think=False, max_tokens=16))

Hello there!
Ahoy there, matey!
**Frustration**

Hand off between backends

chat.hist is backend-agnostic. Pass it as messages= to continue the conversation on another local or hosted backend.

local = Chat(qwen3_4b, n_ctx=4096, tools=[add])
local('What is 2 + 3? Use the add tool.')
remote = Chat('gpt-4.1-nano', messages=local.hist, tools=[add])   # needs API key
print(resp_text(remote('What did I ask, and what was the answer?')))
local.close(); remote.close()

llama_context: n_ctx_seq (4096) < n_ctx_train (40960) -- the full capacity of the model will not be utilized

You asked what 2 + 3 is, and I found the answer to be 5 after using the add tool.

Run Python from replies

PyFenceCallback executes fenced Python from a reply and returns stdout to the model. It stops when the model answers in prose or done returns true.

py = Chat(gemma4_e2b, sp='Use a ```python fence, then answer in prose.')
py('What is 2**100?', cbs=[PyFenceCallback(done=output_matches(str(2**100)))])

Structured output and checks

structured returns a dataclass instance. classify picks a label. Both run a throwaway turn and leave hist unchanged. check grades a fenced answer against an expected string.

from dataclasses import dataclass

@dataclass
class Person: name: str; age: int

print(chat.structured('Extract: John Smith is 30.', Person))
print(chat.classify('I loved this film!', ['positive', 'negative']))
print(chat.check('Capital of France?', 'Paris'))

Other backends

# MLX (Apple Silicon)
from rishi.mlx import qwen3_4b as mlx_qwen
m = Chat(mlx_qwen); print(resp_text(m('One octopus fact.'))); m.close()

# Hosted -- hand local history to a bigger model
loc = Chat(qwen3_4b); loc('My name is Karthik and my favourite number is 17.')
big = Chat('gpt-4.1-nano', messages=loc.hist)
print(resp_text(big('What is my name and favourite number?'))); loc.close(); big.close()

Claude Code

Chat('claude/claude-sonnet-5') runs through Claude Code. via='sdk' uses the claude-agent-sdk package from rishi[claude]. via='cli' uses the claude binary. via=None prefers the SDK when installed. ClaudeChat.local is False.

Managed Claude Code installations can reject dynamic MCP servers. Rishi instead puts tool schemas in the system prompt and receives calls as <tool_call> tags. It opens no MCP server and does not set --strict-mcp-config.

The briefing and tool schemas use Claude Code’s system-prompt channel. Only the conversation is rendered into the prompt. Claude Code’s own system prompt adds 48k to 53k prompt tokens per turn. Tool calls also depend on the model producing the tags correctly. Use a Sonnet-tier or stronger model for tool-using turns.

GitHub Copilot

Chat('copilot/gpt-4.1') runs on a Copilot subscription. rishi.copilot supplies the short-lived token and editor headers required by the endpoint. RemoteChat then handles tools, streaming and approval.

copilot_oauth() first checks Copilot-specific environment variables, then editor sign-in files. It checks GH_TOKEN and GITHUB_TOKEN last because they often contain personal access tokens, which Copilot rejects. copilot_login() runs GitHub’s device flow when no usable sign-in exists. copilot_models() returns the models available to the current account and plan.

from rishi.copilot import copilot_models, CopilotAuth

print(copilot_models()[:5])

auth = CopilotAuth()                       # one exchange, shared by every chat that holds it
chat = Chat('copilot/claude-sonnet-4.5', auth=auth, sp='You are concise.')

This integration is reverse-engineered and unsupported by GitHub. The Copilot subscription terms apply.

from rishi.claude import sonnet5, ClaudeChat
cc = ClaudeChat(sonnet5, tools=[add], sp='Use the add tool.')   # or Chat('claude/claude-sonnet-5')
print(resp_text(cc('What is 2 + 3?')))
cc.close()

Ollama

Chat('ollama/qwen3:4b') runs on a local Ollama daemon. Ollama is a server, not a library, so rishi.ollama drives the daemon as well as the conversation. It uses one already listening. If there is none it installs Ollama under ~/.cache/rishi/ollama, starts ollama serve, pulls the model, and stops the daemon at exit. Nothing is installed system-wide.

An Ollama id (qwen3:4b) works. So does a hub GGUF repo, which Ollama serves under hf.co/, with quant as the tag. The same repo id then runs on either local backend.

think is a request field here rather than a system-prompt hack, and takes the levels 'low', 'medium', 'high' and 'max'. structured uses Ollama’s own JSON schema support. n_ctx and n_gpu_layers keep their rishi.llama names and go out as num_ctx and num_gpu.

from rishi.ollama import ensure_ollama, OllamaServer, stop_ollama

chat = Chat('ollama/qwen3:4b', think='low', n_ctx=8192)   # installs, serves and pulls as needed
print(resp_text(chat('One fact about lobsters.')))

cl = ensure_ollama()                       # the daemon itself, for pulls and bookkeeping
print(cl.version(), cl.models(), cl.ps())
print(model_caps('gemma3:4b', runtime='ollama'))          # asked of the daemon, not guessed

srv = OllamaServer(models='/data/models', n_ctx=16384, kv_cache_type='q8_0', flash_attn=True)
srv.start()                                # or configure one yourself and point chats at it

Ollama carries images but not audio. Its KV cache lives inside the daemon, which offers no handle to save or measure it. There is no tokenizer endpoint either, so count_tokens estimates. rishi.llama covers all three.

from rishi.ollama import qwen3_4b as ol_qwen

o = Chat(f'ollama/{ol_qwen}', tools=[add], think=False)
print(resp_text(o('What is 2 + 3? Use the add tool.')))
o.unload(); o.close()                      # free the daemon's memory now, keep the daemon up

Backend guides