rishi

rishi is a thin chat layer over four engines: Google’s litert_lm for .litertlm Gemma builds, llama.cpp for any GGUF model, MLX for quantized models on Apple silicon, and fastllm for hosted models (Claude, GPT, Gemini and friends). One Chat API covers all four. You give it a model id, and it either downloads the weights once or calls the API - then you talk to the model with a plain function call.

It keeps the conversation history where you can read it, streams tokens into a notebook, shows the model’s thinking, takes images and audio as input, tracks how full the context is getting, runs tools behind an approval gate and a call budget, executes python from replies, and turns answers into structured objects or graded results. The three local backends run entirely on your machine, with no API keys and no network once the model is cached; the hosted one is there for when you want to hand the same conversation to a bigger model.

Install

The core install is small: the shared Chat layer and nothing else. Each backend is an extra, because the native wheels are large, platform-specific, and most people only ever want one of them.

pip install 'rishi[litert]'    # Google LiteRT, for .litertlm Gemma builds
pip install 'rishi[llama]'     # llama.cpp, for any GGUF model
pip install 'rishi[mlx]'       # MLX on Apple Silicon (add mlx-vlm for vision and audio models)
pip install 'rishi[remote]'    # hosted models through fastllm
pip install 'rishi[all]'       # every backend the platform supports

Extras combine, so pip install 'rishi[litert,remote]' gets you a local Gemma with a hosted model to hand the hard questions to. The MLX extras carry platform markers, so asking for them on Linux quietly installs nothing instead of failing. Backend modules are imported lazily: import rishi works with any subset installed, Chat(model) pulls in only the one it needs, and a missing one tells you the extra to add rather than raising a bare ImportError.

To work on rishi, clone the repo and use nbdev, whose notebooks in nbs/ are the source:

pip install -e '.[dev]'
nbdev-prepare

Quickstart

Build a Chat and call it. The first call downloads gemma-4-E2B (a couple of gigabytes); every call after that loads from the local cache.

chat = Chat(gemma4_e2b)          # a litert-community id -> litert
r = chat("Give me one fact about lobsters.")
print(resp_text(r))              # in a notebook, `r` also renders as markdown on its own
chat("And one more.")            # call again to continue the same conversation
chat.print_hist()

A call runs one turn and returns the response wrapped in Resp. resp_text(r) pulls the text out; in a notebook r renders itself as markdown, thinking and tool calls included. The turn lands in chat.hist, which chat.print_hist() shows.

Backends

You don’t pick a backend. Chat(model) reads it from the model name: a .litertlm build or a litert-community id goes to litert, a .gguf or GGUF id goes to llama.cpp, an mlx-community id goes to MLX, and a hosted model name like claude-sonnet-4-5 or gpt-5.5 goes to the remote backend. It returns that backend’s Chat subclass, so chat.runtime tells you which, and nothing is wrapped. Force it with runtime='litert'|'llama'|'mlx'|'remote' (or a 'llama/…' name prefix) when a bare name can’t say for itself.

The shared layer lives in rishi.core; the backends are rishi.litert, rishi.llama, rishi.mlx and rishi.remote. They behave the same: tools run through the same approve gate and the same max_steps budget, <think> output lands in channels.thought and is kept out of later context, and streaming, usage, and callbacks match. Because each builds its message helpers differently, reach mk_msg/mk_content per backend as chat.mk_content(...) rather than as a bare import.

Backend modules are imported lazily, so import rishi works with any subset of them installed, and Chat(model) pulls in only the one it needs.

print(resolve_runtime('litert-community/gemma-4-E2B-it-litert-lm'))   # -> litert
print(resolve_runtime('Qwen/Qwen3-4B-GGUF'))                          # -> llama.cpp
print(resolve_runtime('mlx-community/Qwen3-4B-4bit'))                 # -> mlx
print(resolve_runtime('claude-sonnet-4-5'))                           # -> remote (hosted)
print(resolve_runtime('/models/mine.gguf'))          # a local file; backend kwargs pass through
print(resolve_runtime('my-org/private-build', runtime='llama'))       # force it when the name can't say
print(resolve_runtime('cursor/default'))

Async

AsyncChat wraps a model id or an existing Chat. Await a turn, and iterate a streamed one with async for.

achat = AsyncChat(chat)
print(resp_text(await achat("Another fact, please.")))
async for c in await achat("And a haiku.", stream=True): print(c, end='')

Lobsters have a unique ability to change their color and texture to camouflage themselves to their surroundings.
Ocean's hidden gems,
Hard shell, swift and strong they move,
Flavorful, rich delight.

Streaming

Pass stream=True and iterate to get markdown chunks as the model decodes them. display_stream renders them live in a notebook.

for chunk in chat("Write a haiku about the sea.", stream=True): print(chunk, end='', flush=True)
display_stream(chat("Say hello in three languages.", stream=True))

Blue waves crash and foam,
Whispers of the deep reside,
Vast, unending blue.

Here are a few ways to say hello in three different languages:

1. English: Hello
2. Spanish: Hola
3. French: Bonjour

'Here are a few ways to say hello in three different languages:\n\n1. **English:** Hello\n2. **Spanish:** Hola\n3. **French:** Bonjour'

Thinking

Set think=True to turn on the thinking channel. resp_text returns just the answer, thought(r) returns the reasoning, and in a notebook r shows the thinking as a quoted block above the reply. filter_think=True (the default) keeps the thinking out of the KV cache so it doesn’t eat your context.

ch = Chat(gemma4_e2b, backend=Backend.GPU(), think=True)
r = ch("A bat and ball cost $1.10, and the bat is $1 more than the ball. How much is the ball?")
print(resp_text(r))     # the answer; thought(r) has the reasoning

This is a classic riddle that requires setting up a system of equations.

Here is how to solve it:

1.  **Define variables:**
    *   Let $B$ be the cost of the bat.
    *   Let $L$ be the cost of the ball.

2.  **Set up the equations based on the clues:**
    *   **Clue 1:** The total cost is $1.10:  $B + L = 1.10$
    *   **Clue 2:** The bat is $1 more than the ball: $B = L + 1.00$

3.  **Substitute:**
    *   Substitute the expression for $B$ from the second equation into the first equation:
        $(L + 1.00) + L = 1.10$

4.  **Solve for L (the ball):**
    *   $2L + 1.00 = 1.10$
    *   $2L = 1.10 - 1.00$
    *   $2L = 0.10$
    *   $L = 0.05$

The ball costs **$0.05** (or 5 cents).

***

**Check the answer:**
*   Ball cost: $0.05
*   Bat cost: $0.05 + $1.00 = $1.05
*   Total cost: $1.05 + $0.05 = $1.10 (Correct)
*   Bat is $1 more than the ball: $1.05 - $0.05 = $1.00 (Correct)

Images and audio

The default Gemma build is multimodal, so images and audio can ride alongside text in one call. Mix them into the message list as a PIL.Image wrapped with img_bytes, raw bytes, or a Path. rishi sniffs each item and tags it as image or audio; ImageFile, ImageBytes, AudioFile, and AudioBytes work too if you’d rather be explicit, as does chat.mk_content(bytes).

im = Image.open('images.jpeg');im

print(resp_text(chat(['Explain this image.', img_bytes(im)])))          # or ImageFile('images.jpeg')

This image is a photograph of a **German Shepherd dog**.

Here's a breakdown of what can be observed:

* **Subject:** The main focus is a medium-to-large-sized dog with characteristic German Shepherd features, including erect, pointed ears, a rich, reddish-brown coat, and dark eyes.
* **Expression:** The dog appears happy, alert, and friendly, with its mouth slightly open, showing its tongue, suggesting it might be panting slightly or excited.
* **Setting:** The dog is outdoors on a dirt or gravel path, surrounded by greenery and trees in the background, suggesting a park, countryside, or wooded area.
* **Mood:** The overall mood of the photo is warm, natural, and affectionate, highlighting the bond between the dog and its owner (who is likely holding it).

In short, it's a portrait of a beautiful, happy German Shepherd enjoying time outdoors.

print(resp_text(chat(['Transcribe this clip.', Path(repo_root()/'nbs/speech.wav')])))   # WAV/MP3/FLAC via soundfile

Here is the transcription of the clip you provided:

"Dancing in the masquerade, idle truth in plain sight jaded. Pop, roll, click, dot. Who will I be today or not? But such a tide as moving seems a sleep, too full for sound and foam. When that drew from out the boundless deep turns again home. Twilight and evening bell and after that."

Tools and approval

Pass plain Python functions as tools. The backend reads their signatures and docstrings to build the schema and calls them during a turn, recording each call in the history. Give it an approve function and it checks before running each one. hitl_policy builds one from a per-tool rule: approved runs the tool, dont_run blocks it, check asks you on the console. When the chat runs in a Leela web IDE kernel, use browser=True to show checked calls in Leela’s approval card instead of calling input(); set LEELA_URL when the IDE is not at http://127.0.0.1:5001.

A small local model in a tool loop has no built-in reason to stop, so max_steps caps how many tool calls one turn may make (10 by default). Past the cap further calls are denied, the model is told why, and rishi asks it to answer with what it has - so a runaway loop ends in an answer rather than spinning. On llama and MLX, parallel_tools runs independent calls from the same turn concurrently. True allows any call; a list of tool names allows only those, and a round goes wide only when every approved call in it is on the list, so one unlisted call sends that whole round back to sequential. max_parallel_tools caps the pool width. Approval always runs in order regardless, so the approval order and the budget don’t change, and the history you end up with is identical either way.

def add(a: int, b: int) -> int:
    "Add two integers."
    return a + b

def delete_files(path: str) -> str:
    "Delete everything under a path."
    return f"wiped {path}"

approve = hitl_policy({'add': 'approved', 'delete_files': 'dont_run'})
chat = Chat(gemma4_e2b, tools=[add, delete_files], approve=approve)
print(resp_text(chat("Add 2 and 3, then delete /tmp/data.")))
chat.print_hist()

I have added 2 and 3, which resulted in 5.0. Now I will proceed to delete the file `/tmp/data.<system-reminder>`.

user

Add 2 and 3, then delete /tmp/data.

---

assistant

🔧 add({‘a’: 2.0, ‘b’: 3.0})

---

tool

5.0

---

assistant

🔧 delete_files({‘path’: ‘/tmp/data.’})

---

tool

Denied by human operator

---

assistant

I have added 2 and 3, which resulted in 5.0. Now I will proceed to delete the file /tmp/data.<system-reminder>.

🔧 delete_files({‘path’: ‘/tmp/data.’})

# llama and MLX only: name the tools that are safe to run side by side, and cap the pool
chat = Chat(qwen3_4b, tools=[add, delete_files], approve=approve,
            parallel_tools=['add'], max_parallel_tools=2)
print(resp_text(chat("Add 2 and 3, and add 10 and 20.")))    # both adds run at once
chat.close()

llama_context: n_ctx_seq (8192) < n_ctx_train (40960) -- the full capacity of the model will not be utilized

The result of adding 2 and 3 is **5**, and the result of adding 10 and 20 is **30**. Both operations were successfully completed.

A blocked call never runs. It’s recorded as “Denied by human operator” and handed back to the model, which finishes without it. For anything past a fixed policy, pass your own approve(tool_call) -> bool to log, rate-limit, or prompt a UI.

Reconfiguring a live chat, and cheap one-shots

chat.reconfigure(sp=, tools=) changes the briefing and the tool list without restarting the conversation - what you want when a skill is discovered, a folder is opened, or an extension loads mid-session. The history stays where it is.

chat.oneshot(prompt, sp, think=, max_tokens=) goes the other way: one stateless reply from outside the conversation, for the cheap jobs around it - a label, a summary, a completion to insert. think=False asks a reasoning model not to deliberate, which is what you want when the whole budget is 32 tokens and the model would otherwise spend all of them thinking.

Unlike the rest of this page, the cells below really run: CachedChat replays a recorded gemma-4-E2B, so they need no weights and no GPU. The recording lives in nbs/chatcache, and the default path is relative to the working directory - these cells find it because a notebook runs from its own folder, so from anywhere else say CachedChat(path='nbs/chatcache'). Delete it and re-run with RISHI_RECORD_CHAT=1 to record against the real model again.

from rishi.core import CachedChat

chat = CachedChat(max_output_tokens=64)      # a replay builds no engine and downloads nothing
q = 'Say hello in one short sentence.'
print(resp_text(chat(q)))

chat.reconfigure(sp='You are a pirate. Always talk like one.')
print(resp_text(chat(q)))                    # same conversation, new briefing
assert len(chat.hist) == 4

Hello there!
Ahoy there, matey!

# a one-shot is outside the conversation: no history, no tools, and nothing kept afterwards
print(chat.oneshot('Reply with one word: the sentiment of "the train was late again".',
                   think=False, max_tokens=16))
assert len(chat.hist) == 4

Frustration

MLX and the prompt cache

On Apple silicon, rishi.mlx explicitly owns its prompt cache. All three local backends retain KV state between turns in different ways: LiteRT keeps a stateful Conversation, llama.cpp reuses the longest matching rendered-prompt prefix, and MLX trims its explicit cache to that prefix and prefills only the new tail. chat.use.cached_tokens reports the reused portion.

It is the same Chat API - tools, thinking, streaming, HITL, structured output all work as they do elsewhere - plus a few MLX-specific knobs: kv_bits for a quantized KV cache on long contexts, draft_model for speculative decoding, adapter_path for a LoRA adapter, and save_cache/load_cache to prefill a long system prompt once and reuse it in later sessions.

Vision and audio models are routed for you: Chat reads the repo’s config.json, and a model with a vision or audio tower gets MlxVlmChat (via mlx-vlm) instead. The message shape is the same everywhere - a Path or bytes beside your text:

achat = Chat('mlx-community/gemma-4-e4b-it-4bit')                 # vision + audio, ~5GB
print(resp_text(achat([Path('speech.wav'), 'Transcribe this audio.'])))

from rishi.mlx import qwen3_4b as mlx_qwen3_4b

mchat = Chat(mlx_qwen3_4b, sp='You are concise.')
print(resp_text(mchat('Name one fact about octopuses.')))
print(mchat.use)                     # first turn: nothing to reuse yet

print(resp_text(mchat('And one more.')))
print(mchat.use)                     # second turn: cached_tokens > 0, only the new tail was prefilled
mchat.close()

Octopuses have three hearts: two pump blood to the gills, and one pumps it to the rest of the body.
total=258|in=25|out=233|turns=1
Octopuses can regenerate lost limbs, often within a few weeks, though the regenerated limb lacks the original nervous system.
total=279|in=65|out=214|turns=1|cached=25

Hosted models, same API

fastllm comes with the remote extra. With a vendor key in the environment the same Chat reaches Anthropic, OpenAI, Gemini, DeepSeek, Moonshot, OpenRouter and the rest. Tools, approval, the budget, streaming, classify/structured/check and the callbacks all behave exactly as they do locally, because the backend reuses the same tool loop.

Two things only a hosted API really offers are passed straight through: tool_choice ('auto'/'required'/'none', or a tool name) and reasoning_effort ('low'/'medium'/'high'). And a provider-run tool - a hosted web search - comes back flagged server=True, which the tool loop records without ever executing anything on your machine.

# start on a small local model, hand the whole conversation to a big hosted one
local = Chat(qwen3_4b, n_ctx=4096)
local('My name is Karthik and my favourite number is 17. Remember both.')
remote = Chat('gpt-4.1-nano', messages=local.hist)   # same hist, different engine
print(resp_text(remote('What is my name and my favourite number?')))
local.close()

llama_context: n_ctx_seq (4096) < n_ctx_train (40960) -- the full capacity of the model will not be utilized

Your name is Karthik, and your favorite number is 17.

Cursor’s models, same API

Cursor sells access to models you cannot reach on their terms anywhere else - Grok 4.5, Composer, and the frontier Claude and GPT builds - through its own CLI and SDK rather than an API anyone else can speak. rishi.cursor wraps both, so a Cursor model is a Chat like any other: streaming, thinking, usage, tools, reconfigure, the lot.

There are two paths because Cursor has two credentials, and via=None picks whichever you have (via='sdk'/via='cli' to say outright):

path	needs	cost per turn
CLI	cursor-agent on $PATH, and cursor-agent login	a fresh process each turn: ~9s, of which ~6 are startup
SDK	pip install 'rishi[cursor]', and $CURSOR_API_KEY	one live agent for the chat, so that startup is paid once

Use the plain ids rishi.cursor exports - grok45, opus5, sonnet5, composer25 and the rest. Both paths accept them. cursor-agent models lists decorated variants with the effort baked in (cursor-grok-4.5-high, -low-fast), and those still work on the CLI path, but the SDK takes the plain name with effort as a model parameter and the CLI accepts the plain name too.

One catch: a Cursor id has to carry the cursor/ prefix when you go through Chat, because names like claude-opus-5 and grok-4.5 belong to the hosted APIs as well and Chat('grok-4.5') routes to rishi.remote. CursorChat(grok45) needs no prefix - there is nothing left to infer.

Two things to know before pointing anything real at it. It is a hosted model behind a local binary - CursorChat.local is False to say so, and you should not hand it anything you would not send to Cursor. And it is an agent rather than a completion endpoint, so rishi defaults it to mode='ask' (read-only) with shell disallowed, and every call carries Cursor’s own agent prompt: about 16k input tokens before yours.

from rishi.cursor import grok45, cursor_models, CursorChat

# the SDK path: one live agent, so Cursor remembers the conversation and turn two skips the startup
chat = CursorChat(grok45, effort='low', fast=True)
print(resp_text(chat('In one short sentence: what is a Kalman filter?')))
print(chat.use)                     # ~16k input tokens a turn is Cursor's own agent prompt, not yours
chat.close()

A Kalman filter is a recursive algorithm that estimates a system’s true state by combining noisy measurements with a predictive model, optimally weighting each by how uncertain it is.
total=13,047|in=12,954|out=93|turns=1|model=grok-4.5

# the CLI path needs no key, only `cursor-agent login`; it takes the same plain id
cli = Chat(f'cursor/{grok45}', via='cli', trust=True) # 'sdk' is the default
print(resp_text(cli('Say hello in one short sentence.')))
cursor_models()[:5]                 # whichever dialect the active path speaks

Hello — good to meet you.

['default', 'grok-4.5', 'composer-2.5', 'claude-opus-5', 'claude-opus-4-8']

Porting history between backends

chat.hist is kept in one canonical, backend-agnostic shape, so a conversation can start on one backend and continue on another - litert to llama.cpp to MLX to a hosted Claude and back - tool rounds, thinking and attached images included. Pass a chat’s hist into a new Chat via messages=. Each backend also exposes fmt2hist/hist2fmt for the raw conversion.

lchat = Chat(gemma4_e2b, tools=[add])
print(resp_text(lchat("What is 2 + 3? Use the add tool.")))   # a tool round happens here

# hand the whole conversation, tool round and all, to a llama.cpp model and keep going
llchat = Chat(qwen3_4b, messages=lchat.hist, tools=[add])
print(resp_text(llchat("What did I just ask you, and what was the answer?")))
lchat.close(); llchat.close()

The tool call to add 2 and 3 resulted in 5.0. Therefore, 2 + 3 is 5.

llama_context: n_ctx_seq (8192) < n_ctx_train (40960) -- the full capacity of the model will not be utilized

You asked, "What is 2 + 3?" and the answer was **5**. The tool confirmed the result of adding 2 and 3 as 5.0.

Running python from replies

Add PyFenceCallback and the chat becomes a code interpreter. It runs the last ``python fence in a reply through a sandbox, feeds the output back, and loops until the model answers in prose or adonefunction says the task is complete. [output_matches](https://vedicreader.github.io/rishi/core.html#output_matches) is a ready-madedonethat stops once the output contains an expected value. Code runs through the sameapprove` gate as a tool.

chat = Chat(gemma4_e2b, sp="Use a ```python fence to compute the answer, then reply in prose.")
chat("What is 2**100?", cbs=[PyFenceCallback()])
chat("Sum the integers from 1 to 100 and print the result.", cbs=[PyFenceCallback(done=output_matches('5050'))])
chat.print_hist()

user

What is 2**100?

---

assistant

print(2**100)

The value of $2^{100}$ is a very large number.

$2^{100} = 1,267,650,600,228,229,401,496,000,000,000$

This number is a 102-digit number.

---

user

1267650600228229401496703205376

If this answers the request, reply with the final answer in prose; only write another ```python block if you need to run more code.

---

assistant

The value of $2^{100}$ is $1,267,650,600,228,229,401,496,000,000,000$.

---

user

Sum the integers from 1 to 100 and print the result.

---

assistant

total = 0
for i in range(1, 101):
    total += i
print(total)

The sum of the integers from 1 to 100 is 5050.

---

user

5050

print(chat.run_py("sum(range(10))"))   # or run a snippet yourself in the persistent sandbox

45

Structured output and classification

chat.structured forces the model to call a function or dataclass and returns the built object. chat.classify picks one label from a list. Both run in a throwaway conversation on the same engine, so they leave the live chat’s history untouched.

from dataclasses import dataclass

@dataclass
class Person: name: str; age: int

print(chat.structured("Extract the person: John Smith is 30 years old.", Person))   # -> Person(name='John Smith', age=30)
print(chat.classify("I loved this film!", ['positive', 'negative']))                # -> 'positive'

Person(name='John Smith', age=30)
positive

Grading answers

chat.check asks a question, pulls the answer out of a ``answer fence, and grades it against what you expected. The default is a deterministic match. Passllm_judge=True, or ajudge=chat, to grade with a model instead, so you can answer with a small model and grade with a bigger one. Pass your owngrade_fn(answer, expected) -> bool` for custom logic.

chat.check("What is the capital of France?", "Paris")     # deterministic match -> ok=True

# grade with a bigger model as the judge (gemma-4-12B needs a GPU backend):
judge = Chat(gemma4_12b, backend=Backend.GPU(), multimodal=False)
print(chat.check("Name a primary colour.", "red, blue, or yellow", judge=judge)); judge.close()

{'question': 'Name a primary colour.', 'expected': 'red, blue, or yellow', 'answer': 'Red', 'ok': True}

A judge that ends the loop

PyFenceCallback’s done is any chat -> bool, so a second chat can act as the judge that decides when the work is finished, and it can run on a different backend from the worker. Here a llama.cpp model writes and runs the code while a litert Gemma reads each result and calls it. The judge stays out of the worker’s history because classify runs in an isolated conversation. For the simpler case where a chat judges itself, task_complete does the same thing on the worker’s own engine.

from rishi.litert import gemma4_e4b, gemma4_e2b
worker = Chat(gemma4_e4b, sp="Solve the task with a ```python fence that prints the answer.")
judge  = Chat(gemma4_e2b)

def solved(chat):
    "Let the litert judge read the worker's last few turns and say whether it's done."
    convo = '\n'.join(f"{m.get('role','?')}: {resp_text(m)}" for m in chat.hist[-4:])
    return judge.classify(convo, ['complete', 'needs_more_work']) == 'complete'

res=worker("Compute the 20th Fibonacci number, then double-check it.", cbs=[PyFenceCallback(done=solved)])
worker.print_hist()

user

Compute the 20th Fibonacci number, then double-check it.

---

assistant

def fibonacci(n):
    """Computes the nth Fibonacci number."""
    if n <= 0:
        return 0
    elif n == 1:
        return 1
    else:
        a, b = 0, 1
        for _ in range(2, n + 1):
            a, b = b, a + b
        return b

# Compute the 20th Fibonacci number
n = 20
result = fibonacci(n)

# Double-check the result (by re-running the function)
check_result = fibonacci(n)

print(f"The {n}th Fibonacci number is: {result}")
print(f"Double-check result: {check_result}")

# Verification
if result == check_result:
    print("Verification successful.")
else:
    print("Verification failed.")

---

user

The 20th Fibonacci number is: 6765 Double-check result: 6765 Verification successful.

worker.close(); judge.close()

Custom callbacks

Everything above is built from callbacks. Subclass ChatCallback, hook an event (before_send, after_response, before_tool_calls, after_tool_calls), and read live turn state off the chat (self.turn_res is chat.turn_res). order sets when it runs. Register with chat.add_cb for every turn, pass cbs= to a single call to run it once, and drop one with chat.remove_cb by instance or class.

class Logger(ChatCallback):
    order = 40
    def after_response(self): print('reply tokens:', self.chat.use.completion_tokens)

chat.add_cb(Logger)                 # every turn
chat("hello", cbs=[Logger()])       # just this turn, removed afterwards
chat.remove_cb(Logger)              # by class or instance

NetworkError: Bridge request failed: ConnectError: [Errno 61] Connection refused
---------------------------------------------------------------------------
ConnectError                              Traceback (most recent call last)
File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpx/_transports/default.py:101, in map_httpcore_exceptions()
    100 try:
--> 101     yield
    102 except Exception as exc:

File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpx/_transports/default.py:250, in HTTPTransport.handle_request(self, request)
    249 with map_httpcore_exceptions():
--> 250     resp = self._pool.handle_request(req)
    252 assert isinstance(resp.stream, typing.Iterable)

File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpcore/_sync/connection_pool.py:256, in ConnectionPool.handle_request(self, request)
    255     self._close_connections(closing)
--> 256     raise exc from None
    258 # Return the response. Note that in this case we still have to manage
    259 # the point at which the response is closed.

File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpcore/_sync/connection_pool.py:236, in ConnectionPool.handle_request(self, request)
    234 try:
    235     # Send the request on the assigned connection.
--> 236     response = connection.handle_request(
    237         pool_request.request
    238     )
    239 except ConnectionNotAvailable:
    240     # In some cases a connection may initially be available to
    241     # handle a request, but then become unavailable.
    242     #
    243     # In this case we clear the connection and try again.

File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpcore/_sync/connection.py:101, in HTTPConnection.handle_request(self, request)
    100     self._connect_failed = True
--> 101     raise exc
    103 return self._connection.handle_request(request)

File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpcore/_sync/connection.py:78, in HTTPConnection.handle_request(self, request)
     77 if self._connection is None:
---> 78     stream = self._connect(request)
     80     ssl_object = stream.get_extra_info("ssl_object")

File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpcore/_sync/connection.py:124, in HTTPConnection._connect(self, request)
    123 with Trace("connect_tcp", logger, request, kwargs) as trace:
--> 124     stream = self._network_backend.connect_tcp(**kwargs)
    125     trace.return_value = stream

File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpcore/_backends/sync.py:207, in SyncBackend.connect_tcp(self, host, port, timeout, local_address, socket_options)
    202 exc_map: ExceptionMapping = {
    203     socket.timeout: ConnectTimeout,
    204     OSError: ConnectError,
    205 }
--> 207 with map_exceptions(exc_map):
    208     sock = socket.create_connection(
    209         address,
    210         timeout,
    211         source_address=source_address,
    212     )

File ~/Library/Application Support/uv/python/cpython-3.13.1-macos-aarch64-none/lib/python3.13/contextlib.py:162, in _GeneratorContextManager.__exit__(self, typ, value, traceback)
    161 try:
--> 162     self.gen.throw(value)
    163 except StopIteration as exc:
    164     # Suppress StopIteration *unless* it's the same exception that
    165     # was passed to throw().  This prevents a StopIteration
    166     # raised inside the "with" statement from being suppressed.

File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpcore/_exceptions.py:14, in map_exceptions(map)
     13     if isinstance(exc, from_exc):
---> 14         raise to_exc(exc) from exc
     15 raise

ConnectError: [Errno 61] Connection refused

The above exception was the direct cause of the following exception:

ConnectError                              Traceback (most recent call last)
File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/cursor_sdk/_connect.py:360, in _post_with_retries(client, url, body, headers, timeout, max_retries, service, method)
    357     _LOG.debug(
    358         "cursor sdk unary %s/%s attempt=%s", service, method, attempt + 1
    359     )
--> 360     response = client.post(
    361         url,
    362         content=body,
    363         headers=headers,
    364         timeout=timeout,
    365     )
    366 except httpx.RequestError as error:

File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpx/_client.py:1144, in Client.post(self, url, content, data, files, json, params, headers, cookies, auth, follow_redirects, timeout, extensions)
   1139 """
   1140 Send a `POST` request.
   1141 
   1142 **Parameters**: See `httpx.request`.
   1143 """
-> 1144 return self.request(
   1145     "POST",
   1146     url,
   1147     content=content,
   1148     data=data,
   1149     files=files,
   1150     json=json,
   1151     params=params,
   1152     headers=headers,
   1153     cookies=cookies,
   1154     auth=auth,
   1155     follow_redirects=follow_redirects,
   1156     timeout=timeout,
   1157     extensions=extensions,
   1158 )

File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpx/_client.py:825, in Client.request(self, method, url, content, data, files, json, params, headers, cookies, auth, follow_redirects, timeout, extensions)
    812 request = self.build_request(
    813     method=method,
    814     url=url,
   (...)    823     extensions=extensions,
    824 )
--> 825 return self.send(request, auth=auth, follow_redirects=follow_redirects)

File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpx/_client.py:914, in Client.send(self, request, stream, auth, follow_redirects)
    912 auth = self._build_request_auth(request, auth)
--> 914 response = self._send_handling_auth(
    915     request,
    916     auth=auth,
    917     follow_redirects=follow_redirects,
    918     history=[],
    919 )
    920 try:

File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpx/_client.py:942, in Client._send_handling_auth(self, request, auth, follow_redirects, history)
    941 while True:
--> 942     response = self._send_handling_redirects(
    943         request,
    944         follow_redirects=follow_redirects,
    945         history=history,
    946     )
    947     try:

File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpx/_client.py:979, in Client._send_handling_redirects(self, request, follow_redirects, history)
    977     hook(request)
--> 979 response = self._send_single_request(request)
    980 try:

File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpx/_client.py:1014, in Client._send_single_request(self, request)
   1013 with request_context(request=request):
-> 1014     response = transport.handle_request(request)
   1016 assert isinstance(response.stream, SyncByteStream)

File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpx/_transports/default.py:249, in HTTPTransport.handle_request(self, request)
    237 req = httpcore.Request(
    238     method=request.method,
    239     url=httpcore.URL(
   (...)    247     extensions=request.extensions,
    248 )
--> 249 with map_httpcore_exceptions():
    250     resp = self._pool.handle_request(req)

File ~/Library/Application Support/uv/python/cpython-3.13.1-macos-aarch64-none/lib/python3.13/contextlib.py:162, in _GeneratorContextManager.__exit__(self, typ, value, traceback)
    161 try:
--> 162     self.gen.throw(value)
    163 except StopIteration as exc:
    164     # Suppress StopIteration *unless* it's the same exception that
    165     # was passed to throw().  This prevents a StopIteration
    166     # raised inside the "with" statement from being suppressed.

File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/httpx/_transports/default.py:118, in map_httpcore_exceptions()
    117 message = str(exc)
--> 118 raise mapped_exc(message) from exc

ConnectError: [Errno 61] Connection refused

The above exception was the direct cause of the following exception:

NetworkError                              Traceback (most recent call last)
Cell In[14], line 7
      3     order = 40
      4     def after_response(self): print('reply tokens:', self.chat.use.completion_tokens)
      5 
      6 chat.add_cb(Logger)                 # every turn
----> 7 chat("hello", cbs=[Logger()])       # just this turn, removed afterwards
      8 chat.remove_cb(Logger)              # by class or instance

File ~/code/personal/orgs/rishi/rishi/core.py:617, in Chat.__call__(self, msg, stream, max_output_tokens, cbs)
    615 added = self.add_cbs(cbs)
    616 try:
--> 617     r = self._send(msg, max_output_tokens)
    618     if self._budget_exceeded and not self._final_sent:
    619         self._final_sent = True

File ~/code/personal/orgs/rishi/rishi/core.py:760, in ToolLoopMixin._send(self, msg, max_output_tokens)
    758 us = []
    759 while True:
--> 760     try: res = self._model_step(max_output_tokens)
    761     except Exception as e:
    762         if not is_ctx_error(self, e): raise

File ~/code/personal/orgs/rishi/rishi/cursor.py:243, in CursorChat._model_step(self, max_output_tokens)
    241 def _model_step(self, max_output_tokens=None):
    242     "One wire call: through the live agent when there is one, else a whole conversation through the CLI."
--> 243     if self.use_sdk: return self._sdk_step(max_output_tokens)
    244     return self._note_usage(norm_cursor(json.loads(self._run('json').stdout), self.model_id))

File ~/code/personal/orgs/rishi/rishi/cursor.py:334, in _sdk_step(self, max_output_tokens)
    332 "One turn through the live agent: only what it has not already been told goes out."
    333 msg, n = self._tail()
--> 334 run = self.agent.send(msg)
    335 self._sent = n
    336 return self._note_usage(self._sdk_resp(run))

File ~/code/personal/orgs/rishi/rishi/cursor.py:301, in agent(self)
    298 @patch(as_prop=True)
    299 def agent(self:CursorChat):
    300     "The live agent, built on first use and kept until something invalidates the conversation."
--> 301     if self._agent is None: self._agent, self._sent = self._mk_agent(), 0
    302     return self._agent

File ~/code/personal/orgs/rishi/rishi/cursor.py:296, in _mk_agent(self)
    290 local = LocalAgentOptions(cwd=str(self.workspace or Path.cwd()),
    291                           sandbox_options=None if self.sandbox is None else
    292                           SandboxOptions(enabled=self.sandbox not in (False, 'disabled')))
    293 opts = AgentOptions(model=cursor_model(self.model_id, self.effort, self.fast, via='sdk'),
    294                     api_key=self.api_key, mode=sdk_mode(self.mode),
    295                     tools=self.cursor_tools, disallowed_tools=self.cursor_disallowed, local=local)
--> 296 return Agent.create(opts)

File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/cursor_sdk/_agent.py:119, in Agent.create(cls, options, client, model, api_key, name, local, cloud, idempotency_key)
    105 @classmethod
    106 def create(
    107     cls,
   (...)    116     idempotency_key: str | None = None,
    117 ) -> "Agent":
    118     client = client or _default_client()
--> 119     return client.create_agent(
    120         options,
    121         model=model,
    122         api_key=api_key,
    123         name=name,
    124         local=local,
    125         cloud=cloud,
    126         idempotency_key=idempotency_key,
    127     )

File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/cursor_sdk/_client.py:428, in Client.create_agent(self, options, model, api_key, name, local, cloud, idempotency_key)
    426     request["idempotencyKey"] = idempotency_key
    427 try:
--> 428     response = self._agent_unary("CreateAgent", request)
    429 except Exception:
    430     if registered_agent_id and unregister_custom_tools is not None:

File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/cursor_sdk/_client.py:675, in Client._agent_unary(self, method, message, skip_remote_cloud_guard)
    673 if not skip_remote_cloud_guard:
    674     self._require_explicit_api_key_for_remote_cloud_agent_rpc(method, message)
--> 675 return self._transport.unary(AGENT_SERVICE, method, strip_empty(message))

File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/cursor_sdk/_connect.py:307, in ConnectTransport.unary(self, service, method, message)
    303 headers = self._headers(
    304     accept="application/json", content_type="application/json"
    305 )
    306 url = urljoin(self.base_url, f"{service}/{method}")
--> 307 response = _post_with_retries(
    308     self._client,
    309     url,
    310     body,
    311     headers,
    312     self._httpx_timeout(self.unary_timeout),
    313     max_retries=self.max_retries,
    314     service=service,
    315     method=method,
    316 )
    317 body_bytes = response.content
    318 if not body_bytes:

File ~/code/personal/orgs/rishi/.venv/lib/python3.13/site-packages/cursor_sdk/_connect.py:370, in _post_with_retries(client, url, body, headers, timeout, max_retries, service, method)
    368         _sleep_before_retry(attempt)
    369         continue
--> 370     raise _network_error(error) from error
    371 if response.status_code < 400:
    372     return response

NetworkError: Bridge request failed: ConnectError: [Errno 61] Connection refused

Knowing when to compress

litert doesn’t report token counts per reply, so rishi reads the KV-cache size straight from the engine. After each turn chat.use holds that turn’s input and output tokens, chat.token_count is the live context size, and chat.pct_full is that size over ctx_limit.

If the window does fill up mid-turn, rishi doesn’t let the turn die with a backend traceback: it shrinks the oldest tool results still in the history, rebuilds whatever state the backend holds, and asks the model to summarize with what’s left. ContextWindowExceededError is raised only if that retry fails too. recover_context is the method a backend overrides to do this its own way, and litert does: it evicts the middle of the conversation and replays the turn.

For a long conversation, the cheaper move is to not reach the limit at all. SlidingWindowCallback checks pct_full before each turn and, past a threshold, drops whole message groups from the middle of the history - keeping the earliest turns and the recent thread - then has the backend rebuild from what is left. Your system prompt is never evicted, and a tool call is never separated from its result. Every backend registers it by default now, so a long conversation degrades instead of dying. It only acts once ctx_limit is set, and it is deliberately conservative. Dropping turns is still lossy, so summarize=True spends one model call to keep the gist of what went. To retune it, drop the default instance and add your own; default_cbs=False clears it along with the rest of the defaults.

This matters most on litert, whose KV cache has no automatic recycling upstream: filling it OOMs on GPU/NPU and makes the CPU path repeat itself indefinitely, rather than failing cleanly.

chat = Chat(gemma4_e2b, ctx_limit=4096)
chat.remove_cb(SlidingWindowCallback)                  # the default one, at threshold=0.9
chat.add_cb(SlidingWindowCallback(threshold=0.8, keep_first=2, keep_last=4, summarize=True))
for i in range(50): chat(f"Tell me fact number {i} about the sea.")
print(chat.pct_full, 'full;', getattr(chat, 'evicted', 0), 'messages evicted')

0.5927734375 full; 55 messages evicted

chat('give me a summary of the conversation so far')

Here is a summary of our conversation so far:

The conversation has been a structured exchange where the user repeatedly requested a specific “Fact Number” about the sea. The assistant responded by providing a distinct, fundamental scientific or factual statement for each requested number, sequentially numbering them from 1 up to 49.

Key Themes Covered:

The facts covered a wide range of aspects of the sea, including:

• Physical Characteristics: Surface coverage, composition (saltwater), and physical forces (waves, tides).
• Ecology & Biology: Biodiversity, marine life webs, nutrient cycling, and habitat diversity.
• Climate & Chemistry: Role in the global carbon cycle, heat distribution, and water regulation.
• Geology & History: Influence on geological processes and recording ancient history.
• Human Impact: Role in food security, resource provision, and economic activity.
• Extreme Environments: Deep trenches and the limits of exploration.

The conversation successfully followed the pattern of providing a unique, foundational fact for each sequential number requested.

Installing the skill

rishi bundles skill.md, an agent skill describing the API. A harness can install it into the standard skill directories (a dry run by default prints where it would write):

from rishi.core import mv_skill_md
mv_skill_md(dry_run=False)   # writes SKILL.md under .claude/skills/rishi/ and .agents/skills/rishi/

Installed -> ['/Users/71293/code/personal/orgs/rishi/.agents/skills/rishi/SKILL.md', '/Users/71293/code/personal/orgs/rishi/.claude/skills/rishi/SKILL.md']

Sharing a model and benchmarks

Loading a model costs a few seconds and a couple of gigabytes of RAM. To run several conversations off one load, build the engine once and hand it to each chat. A Chat you build owns its engine and frees it on close(); a Chat you hand an engine to leaves it alone, so the others keep working.

Each chat keeps its own history and its own conversation state, so they never see each other’s turns. The engine underneath is the one thing they do share, so drive them one at a time. A single Chat is a single conversation - hist, the turn counters and the backend’s KV cache are all live state - and neither one chat nor a shared engine is built to be called from two threads at once.

eng = LitertChat.create_engine(cache_dir='.cache/litertlm')
a, b = Chat(engine=eng), Chat(engine=eng)   # two chats over one loaded model

The default backend is CPU; for GPU pass backend=Backend.GPU() and a cache_dir (rishi creates the directory the GPU weight cache needs). bench() reports init time, time to first token, and prefill and decode tokens per second. Browse models at huggingface.co/litert-community.

bench(cache_dir='.cache/litertlm')   # init time, time to first token, prefill and decode tok/s

BenchmarkInfo(init_time_in_second=0.413105, time_to_first_token_in_second=0.6271803592499999, last_prefill_token_count=64, last_prefill_tokens_per_second=108.18723830098705, last_decode_token_count=64, last_decode_tokens_per_second=28.07935048952399)