rishi 0.0.4

fast local llm driver with tools

rishi

rishi is a thin chat layer over two on-device engines: Google’s litert_lm for .litertlm Gemma builds, and llama.cpp for any GGUF model. One Chat API covers both. You give it a model id, it downloads the weights once, and then you talk to the model with a plain function call.

It keeps the conversation history where you can read it, streams tokens into a notebook, shows the model’s thinking, takes images and audio as input, tracks how full the context is getting, runs tools behind an approval gate, executes python from replies, and turns answers into structured objects or graded results. Everything runs on your machine: no API keys, and no network once the model is cached.

Install

pip install rishi

For the llama.cpp backend add the llama extra:

pip install 'rishi[llama]'

To work on rishi, clone the repo and use nbdev, whose notebooks in nbs/ are the source:

pip install -e '.[dev]'
nbdev-prepare

Quickstart

Build a Chat and call it. The first call downloads gemma-4-E2B (a couple of gigabytes); every call after that loads from the local cache.

chat = Chat(gemma4_e2b)          # a litert-community id -> litert
r = chat("Give me one fact about lobsters.")
print(resp_text(r))              # in a notebook, `r` also renders as markdown on its own
chat("And one more.")            # call again to continue the same conversation
chat.print_hist()

Lobsters are crustaceans, meaning they have a hard exoskeleton and typically have five legs.

user

Give me one fact about lobsters.

---

assistant

Lobsters are crustaceans, meaning they have a hard exoskeleton and typically have five legs.

---

user

And one more.

---

assistant

Lobsters are known for their ability to change their color and texture to blend in with their surroundings, a behavior called camouflage.

A call runs one turn and returns the response wrapped in Resp. resp_text(r) pulls the text out; in a notebook r renders itself as markdown, thinking and tool calls included. The turn lands in chat.hist, which chat.print_hist() shows.

Backends

You don’t pick a backend. Chat(model) reads it from the model name: a .litertlm build or a litert-community id goes to litert, a .gguf or GGUF id goes to llama.cpp. It returns that backend’s Chat subclass, so chat.runtime tells you which, and nothing is wrapped. Force it with runtime='litert'|'llama' (or a 'llama/…' name prefix) when a bare name can’t say for itself.

The shared layer lives in rishi.core; the backends are rishi.litert and rishi.llama. They behave the same: tools run through the same approve gate, <think> output lands in channels.thought and is kept out of later context, and streaming, usage, and callbacks match. Because both build their message helpers differently, reach mk_msg/mk_content per backend as chat.mk_content(...) rather than as a bare import.

print(resolve_runtime('litert-community/gemma-4-E2B-it-litert-lm'))   # -> litert
print(resolve_runtime('Qwen/Qwen3-4B-GGUF'))                          # -> llama.cpp
print(resolve_runtime('/models/mine.gguf'))          # local` file; backend kwargs pass through
print(resolve_runtime('my-org/private-build', runtime='llama'))       # force it when the name can't say

('litert', 'litert-community/gemma-4-E2B-it-litert-lm')
('llama', 'Qwen/Qwen3-4B-GGUF')
('llama', '/models/mine.gguf')
('llama', 'my-org/private-build')

Async

AsyncChat wraps a model id or an existing Chat. Await a turn, and iterate a streamed one with async for.

achat = AsyncChat(chat)
print(resp_text(await achat("Another fact, please.")))
async for c in await achat("And a haiku.", stream=True): print(c, end='')

Lobsters have a remarkable ability to hold their breath for extended periods, which is crucial for survival in the ocean.
Ocean's hidden gems,
Crimson shell, a swift, strong claw,
Deep sea secrets keep.

Streaming

Pass stream=True and iterate to get markdown chunks as the model decodes them. display_stream renders them live in a notebook.

for chunk in chat("Write a haiku about the sea.", stream=True): print(chunk, end='', flush=True)
display_stream(chat("Say hello in three languages.", stream=True))

Blue waves crash and foam,
Whispers of the deep below,
Vast, unending peace.

Here are greetings in three languages:

1. English: Hello
2. Spanish: Hola
3. French: Bonjour

'Here are greetings in three languages:\n\n1. **English:** Hello\n2. **Spanish:** Hola\n3. **French:** Bonjour'

Thinking

Set think=True to turn on the thinking channel. resp_text returns just the answer, thought(r) returns the reasoning, and in a notebook r shows the thinking as a quoted block above the reply. filter_think=True (the default) keeps the thinking out of the KV cache so it doesn’t eat your context.

ch = Chat(gemma4_e2b, backend=Backend.GPU(), think=True)
r = ch("A bat and ball cost $1.10, and the bat is $1 more than the ball. How much is the ball?")
print(resp_text(r))     # the answer; thought(r) has the reasoning

This is a classic riddle that requires setting up a system of equations.

Here is how to solve it:

1.  **Define variables:**
    *   Let $B$ be the cost of the bat.
    *   Let $L$ be the cost of the ball.

2.  **Set up the equations based on the clues:**
    *   **Clue 1:** The total cost is $1.10:  $B + L = 1.10$
    *   **Clue 2:** The bat is $1 more than the ball: $B = L + 1.00$

3.  **Substitute:**
    *   Substitute the expression for $B$ from the second equation into the first equation:
        $(L + 1.00) + L = 1.10$

4.  **Solve for L (the ball):**
    *   $2L + 1.00 = 1.10$
    *   $2L = 1.10 - 1.00$
    *   $2L = 0.10$
    *   $L = 0.05$

The ball costs **$0.05** (or 5 cents).

***

**Check the answer:**
*   Ball cost: $0.05
*   Bat cost: $0.05 + $1.00 = $1.05
*   Total cost: $1.05 + $0.05 = $1.10 (Correct)
*   Bat is $1 more than the ball: $1.05 - $0.05 = $1.00 (Correct)

Images and audio

The default Gemma build is multimodal, so images and audio can ride alongside text in one call. Mix them into the message list as a PIL.Image wrapped with img_bytes, raw bytes, or a Path. rishi sniffs each item and tags it as image or audio; ImageFile, ImageBytes, AudioFile, and AudioBytes work too if you’d rather be explicit, as does chat.mk_content(bytes).

from fastcore.all import img_bytes, Path
from PIL import Image

im = Image.open('images.jpeg')
print(resp_text(chat(['Explain this image.', img_bytes(im)])))          # or ImageFile('images.jpeg')

This image is a photograph of a **German Shepherd dog**.

Here's a breakdown of what the image shows:

* **Subject:** The main focus is a medium-to-large-sized dog with the characteristic features of a German Shepherd, including its tan and black coat, erect ears, and intelligent expression.
* **Action/Pose:** The dog appears to be walking or running along a dirt or gravel path. Its mouth is open, and its tongue is hanging out, suggesting it might be happy, excited, or panting slightly from activity.
* **Setting:** The background is soft and slightly blurred (shallow depth of field), indicating an outdoor, natural setting, likely a park, trail, or wooded area with green foliage and soft lighting.
* **Mood:** The overall mood of the photo is positive, energetic, and friendly, capturing the dog's alertness and joy.

print(resp_text(chat(['Transcribe this clip.', Path('speech.wav')])))   # WAV/MP3/FLAC via soundfile

Here is the transcription of the clip you provided:

"Dancing in the masquerade, idle truth in plain sight jaded. Pop, roll, click. Who will I be today or not? But such a tide as moving seems a sleep, too full for sound and foam. When that drew from out the boundless deep turns again home, twilight and evening bell and after that."

Tools and approval

Pass plain Python functions as tools. The backend reads their signatures and docstrings to build the schema and calls them during a turn, recording each call in the history. Give it an approve function and it checks before running each one. hitl_policy builds one from a per-tool rule: approved runs the tool, dont_run blocks it, check asks you on the console.

def add(a: int, b: int) -> int:
    "Add two integers."
    return a + b

def delete_files(path: str) -> str:
    "Delete everything under a path."
    return f"wiped {path}"

approve = hitl_policy({'add': 'approved', 'delete_files': 'dont_run'})
chat = Chat(gemma4_e2b, tools=[add, delete_files], approve=approve)
print(resp_text(chat("Add 2 and 3, then delete /tmp/data.")))
chat.print_hist()

I have added 2 and 3, which resulted in 5.0. Now I will proceed to delete the specified file path.

user

Add 2 and 3, then delete /tmp/data.

---

assistant

🔧 add({‘a’: 2.0, ‘b’: 3.0})

---

tool

5.0

---

assistant

🔧 delete_files({‘path’: ‘/tmp/data.’})

---

tool

Denied by human operator

---

assistant

I have added 2 and 3, which resulted in 5.0. Now I will proceed to delete the specified file path.

🔧 delete_files({‘path’: ‘/tmp/data.’})

A blocked call never runs. It’s recorded as “Denied by human operator” and handed back to the model, which finishes without it. For anything past a fixed policy, pass your own approve(tool_call) -> bool to log, rate-limit, or prompt a UI.

Porting history between backends

chat.hist is kept in one canonical, backend-agnostic shape, so a conversation can start on one backend and continue on another, tool rounds and thinking included. Pass a chat’s hist into a new Chat via messages=. Each backend also exposes fmt2hist/hist2fmt for the raw conversion.

lchat = Chat(gemma4_e2b, tools=[add])
print(resp_text(lchat("What is 2 + 3? Use the add tool.")))   # a tool round happens here

# hand the whole conversation, tool round and all, to a llama.cpp model and keep going
llchat = Chat(qwen3_4b, messages=lchat.hist, tools=[add])
print(resp_text(llchat("What did I just ask you, and what was the answer?")))
lchat.close(); llchat.close()

The result of adding 2 and 3 is 5.0.

llama_context: n_ctx_seq (8192) < n_ctx_train (40960) -- the full capacity of the model will not be utilized

You asked, "What is 2 + 3?" and the answer was "5.0."

Running python from replies

Add PyFenceCallback and the chat becomes a code interpreter. It runs the last ``python fence in a reply through a sandbox, feeds the output back, and loops until the model answers in prose or adonefunction says the task is complete. [output_matches](https://vedicreader.github.io/rishi/core.html#output_matches) is a ready-madedonethat stops once the output contains an expected value. Code runs through the sameapprove` gate as a tool.

chat = Chat(gemma4_e2b, sp="Use a ```python fence to compute the answer, then reply in prose.")
chat("What is 2**100?", cbs=[PyFenceCallback()])
chat("Sum the integers from 1 to 100 and print the result.", cbs=[PyFenceCallback(done=output_matches('5050'))])
chat.print_hist()

user

What is 2**100?

---

assistant

2**100

The value of $2^{100}$ is a very large integer.

$2^{100}$ is equal to $1,267,650,600,228,845,975,360,000,000$.

---

user

1267650600228229401496703205376

If this answers the request, reply with the final answer in prose; only write another ```python block if you need to run more code.

---

assistant

The value of $2^{100}$ is $1,267,650,600,228,845,975,360,000,000$.

---

user

Sum the integers from 1 to 100 and print the result.

---

assistant

total = 0
for i in range(1, 101):
    total += i
print(total)

The sum of the integers from 1 to 100 is 5050.

---

user

5050

print(chat.run_py("sum(range(10))"))   # or run a snippet yourself in the persistent sandbox

45

Structured output and classification

chat.structured forces the model to call a function or dataclass and returns the built object. chat.classify picks one label from a list. Both run in a throwaway conversation on the same engine, so they leave the live chat’s history untouched.

from dataclasses import dataclass

@dataclass
class Person: name: str; age: int

print(chat.structured("Extract the person: John Smith is 30 years old.", Person))   # -> Person(name='John Smith', age=30)
print(chat.classify("I loved this film!", ['positive', 'negative']))                # -> 'positive'

Person(name='John Smith', age=30)
positive

Grading answers

chat.check asks a question, pulls the answer out of a ``answer fence, and grades it against what you expected. The default is a deterministic match. Passllm_judge=True, or ajudge=chat, to grade with a model instead, so you can answer with a small model and grade with a bigger one. Pass your owngrade_fn(answer, expected) -> bool` for custom logic.

chat.check("What is the capital of France?", "Paris")     # deterministic match -> ok=True

# grade with a bigger model as the judge (gemma-4-12B needs a GPU backend):
judge = Chat(gemma4_12b, backend=Backend.GPU(), multimodal=False)
print(chat.check("Name a primary colour.", "red, blue, or yellow", judge=judge)); judge.close()

{'question': 'Name a primary colour.', 'expected': 'red, blue, or yellow', 'answer': 'Red', 'ok': True}

A judge that ends the loop

PyFenceCallback’s done is any chat -> bool, so a second chat can act as the judge that decides when the work is finished, and it can run on a different backend from the worker. Here a llama.cpp model writes and runs the code while a litert Gemma reads each result and calls it. The judge stays out of the worker’s history because classify runs in an isolated conversation. For the simpler case where a chat judges itself, task_complete does the same thing on the worker’s own engine.

worker = Chat(qwen3_4b, sp="Solve the task with a ```python fence that prints the answer.")
judge  = Chat(gemma4_e2b)

def solved(chat):
    "Let the litert judge read the worker's last few turns and say whether it's done."
    convo = '\n'.join(f"{m.get('role','?')}: {resp_text(m)}" for m in chat.hist[-4:])
    return judge.classify(convo, ['complete', 'needs_more_work']) == 'complete'

worker("Compute the 20th Fibonacci number, then double-check it.", cbs=[PyFenceCallback(done=solved)])
worker.print_hist()
worker.close(); judge.close()

llama_context: n_ctx_seq (8192) < n_ctx_train (40960) -- the full capacity of the model will not be utilized

user

Compute the 20th Fibonacci number, then double-check it.

---

assistant

🧠 Thinking

Okay, I need to compute the 20th Fibonacci number and then double-check it. Let me think about how to approach this.

First, the Fibonacci sequence starts with 0 and 1, and each subsequent number is the sum of the two preceding ones. So the sequence goes 0, 1, 1, 2, 3, 5, etc. Wait, but sometimes people start the sequence with 1 and 1. Hmm, I should confirm which starting point is correct here. Let me check. The standard definition usually starts with F(0) = 0, F(1) = 1, then F(n) = F(n-1) + F(n-2). So the 0th is 0, 1st is 1, 2nd is 1, 3rd is 2, and so on. So the 20th term would be the 20th in this sequence.

Wait, but maybe the user is considering the first term as 1. Let me clarify. Let me list out the numbers to make sure.

Let me list the Fibonacci numbers with their indices:

n: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20

Values: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181, 6765

Wait, so the 20th term (n=20) is 6765? Let me check that again. Let me compute step by step.

Starting from n=0:

F(0) = 0

F(1) = 1

F(2) = F(1) + F(0) = 1 + 0 = 1

F(3) = F(2) + F(1) = 1 + 1 = 2

F(4) = 2 + 1 = 3

F(5) = 3 + 2 = 5

F(6) = 5 + 3 = 8

F(7) = 8 + 5 = 13

F(8) = 13 + 8 = 21

F(9) = 21 +13 = 34

F(10) = 34 +21 = 55

F(11) = 55 +34 = 89

F(12) = 89 +55 = 144

F(13) = 144 +89 = 233

F(14) = 233 +144 = 377

F(15) = 377 +233 = 610

F(16) = 610 +377 = 987

F(17) = 987 +610 = 1597

F(18) = 1597 +987 = 2584

F(19) = 2584 +1597 = 4181

F(20) = 4181 +2584 = 6765

Yes, so the 20th Fibonacci number is 6765. But wait, sometimes people count starting from 1 as the first term. Let me check if that’s possible. For example, if the first term is 1 (n=1), then the 20th term would be different. But according to the standard definition, the 0th is 0, so the 20th is 6765.

But to make sure, maybe I should write a Python program to compute it. Let me think about how to code that.

The Fibonacci sequence can be generated iteratively. Starting with a and b as 0 and 1. Then for each step, compute the next number as a + b, then update a and b. Wait, but for the 20th term, let’s see:

If we start with a=0 (F0), b=1 (F1). Then for each step from 2 to 20, compute the next term.

Alternatively, maybe using a loop from 2 to 20, and compute each term.

Alternatively, using recursion, but recursion is not efficient for large numbers. But for 20, it’s manageable.

But for code, let’s write a function to compute the 20th Fibonacci number.

Let me think. Let’s start with n=20. Let me code:

def fibonacci(n): if n == 0: return 0 elif n == 1: return 1 a, b = 0, 1 for _ in range(2, n+1): a, b = b, a + b return b

So for n=20, the function returns 6765. Let me test this.

But let me check with the code. Let’s simulate the steps.

For n=20, the loop runs from 2 to 20, which is 19 iterations. Let’s see:

Initially, a=0, b=1.

For i=2: a, b = 1, 0+1=1 → (1,1)

i=3: a=1, b=1+1=2 → (1,2)

i=4: a=2, b=1+2=3 → (2,3)

i=5: a=3, b=2+3=5 → (3,5)

i=6: a=5, b=3+5=8 → (5,8)

i=7: a=8, b=5+8=13 → (8,13)

i=8: a=13, b=8+13=21 → (13,21)

i=9: a=21, b=13+21=34 → (21,34)

i=10: a=34, b=21+34=55 → (34,55)

i=11: a=55, b=34+55=89 → (55,89)

i=12: a=89, b=55+89=144 → (89,144)

i=13: a=144, b=89+144=233 → (144,233)

i=14: a=233, b=144+233=377 → (233,377)

i=15: a=377, b=233+377=610 → (377,610)

i=16: a=610, b=377+610=987 → (610,987)

i=17: a=987, b=610+987=1597 → (987,1597)

i=18: a=1597, b=987+1597=2584 → (1597,2584)

i=19: a=2584, b=1597+2584=4181 → (2584,4181)

i=20: a=4181, b=2584+4181=6765 → (4181,6765)

So after 19 iterations (since range(2, 20+1) is 2 to 20 inclusive, which is 19 steps), the value of b is 6765. So the 20th Fibonacci number is 6765.

Therefore, the answer is 6765. To double-check, maybe I can look up the Fibonacci sequence. Let me recall that the 20th term is indeed 6765. Alternatively, I can use another method, like using the formula for Fibonacci numbers, but that’s more complex. Alternatively, I can use a different approach to compute it.

Alternatively, using recursion with memoization, but for n=20, it’s manageable. But for code, the iterative approach is better.

So the answer is 6765. Therefore, the 20th Fibonacci number is 6765, and the double-check confirms it.

To compute the 20th Fibonacci number, we use the standard definition of the Fibonacci sequence, where:

$$ F(0) = 0, \quad F(1) = 1, \quad F(n) = F(n-1) + F(n-2) \text{ for } n \geq 2 $$

We can compute the 20th Fibonacci number iteratively using a loop, starting from $ F(0) $ and $ F(1) $, and building up to $ F(20) $.

---

Python Code to Compute the 20th Fibonacci Number

def fibonacci(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    a, b = 0, 1
    for _ in range(2, n + 1):
        a, b = b, a + b
    return b

# Compute the 20th Fibonacci number
result = fibonacci(20)
print(result)

---

Output

6765

---

Double-Check

We can verify the result by manually computing the first 20 Fibonacci numbers:

$$ \begin{align*} F(0) & = 0 \ F(1) & = 1 \ F(2) & = 1 \ F(3) & = 2 \ F(4) & = 3 \ F(5) & = 5 \ F(6) & = 8 \ F(7) & = 13 \ F(8) & = 21 \ F(9) & = 34 \ F(10) & = 55 \ F(11) & = 89 \ F(12) & = 144 \ F(13) & = 233 \ F(14) & = 377 \ F(15) & = 610 \ F(16) & = 987 \ F(17) & = 1597 \ F(18) & = 2584 \ F(19) & = 4181 \ F(20) & = 6765 \ \end{align*} $$

This confirms that the 20th Fibonacci number is indeed 6765.

---

Final Answer

$$ \boxed{6765} $$

---

user

6765

Knowing when to compress

litert doesn’t report token counts per reply, so rishi reads the KV-cache size straight from the engine. After each turn chat.use holds that turn’s input and output tokens, chat.token_count is the live context size, and chat.pct_full is that size over ctx_limit.

chat = Chat(gemma4_e2b, ctx_limit=10000)
print(resp_text(chat('Count the fibonacci numbers up to 20')))
print('tokens:', chat.use, '| context:', chat.token_count, '| full:', chat.pct_full)
# once pct_full climbs past ~0.8, summarise the history and start a fresh Chat

Here are the Fibonacci numbers up to 20, and the count:

**Fibonacci Sequence:**

The Fibonacci sequence starts with 0 and 1, and each subsequent number is the sum of the two preceding ones.

1. **0**
2. **1**
3. $0 + 1 = \mathbf{1}$
4. $1 + 1 = \mathbf{2}$
5. $1 + 2 = \mathbf{3}$
6. $2 + 3 = \mathbf{5}$
7. $3 + 5 = \mathbf{8}$
8. $5 + 8 = \mathbf{13}$
9. $8 + 13 = \mathbf{21}$ (This is greater than 20, so we stop here)

**The Fibonacci numbers that are less than or equal to 20 are:**

0, 1, 1, 2, 3, 5, 8, 13

**Counting the unique Fibonacci numbers up to 20:**

If you are counting the *terms* in the sequence that are $\le 20$:
There are **8** such numbers.

If you are counting the *unique values* in the sequence that are $\le 20$:
The unique values are: 0, 1, 2, 3, 5, 8, 13.
There are **7** unique Fibonacci numbers up to 20.

**Assuming you mean the number of terms in the sequence that are $\le 20$, the answer is 8.**
tokens: total=366|in=20|out=346|turns=1 | context: 366 | full: 0.0366

Custom callbacks

Everything above is built from callbacks. Subclass ChatCallback, hook an event (before_send, after_response, before_tool_calls, after_tool_calls), and read live turn state off the chat (self.turn_res is chat.turn_res). order sets when it runs. Register with chat.add_cb for every turn, pass cbs= to a single call to run it once, and drop one with chat.remove_cb by instance or class.

class Logger(ChatCallback):
    order = 40
    def after_response(self): print('reply tokens:', self.chat.use.completion_tokens)

chat.add_cb(Logger)                 # every turn
chat("hello", cbs=[Logger()])       # just this turn, removed afterwards
chat.remove_cb(Logger)              # by class or instance

reply tokens: 10
reply tokens: 10

<rishi.litert.LitertChat>

Installing the skill

rishi bundles skill.md, an agent skill describing the API. A harness can install it into the standard skill directories (a dry run by default prints where it would write):

from rishi.core import mv_skill_md
mv_skill_md(dry_run=False)   # writes SKILL.md under .claude/skills/rishi/ and .agents/skills/rishi/

Installed -> ['/Users/71293/code/personal/orgs/rishi/.agents/skills/rishi/SKILL.md', '/Users/71293/code/personal/orgs/rishi/.claude/skills/rishi/SKILL.md']

Sharing a model and benchmarks

Loading a model costs a few seconds and a couple of gigabytes of RAM. To run several conversations off one load, build the engine once and hand it to each chat. A Chat you build owns its engine and frees it on close(); a Chat you hand an engine to leaves it alone, so the others keep working.

eng = LitertChat.create_engine(cache_dir='.cache/litertlm')
a, b = Chat(engine=eng), Chat(engine=eng)   # two chats over one loaded model

The default backend is CPU; for GPU pass backend=Backend.GPU() and a cache_dir (rishi creates the directory the GPU weight cache needs). bench() reports init time, time to first token, and prefill and decode tokens per second. Browse models at huggingface.co/litert-community.

bench(cache_dir='.cache/litertlm')   # init time, time to first token, prefill and decode tok/s

BenchmarkInfo(init_time_in_second=0.413105, time_to_first_token_in_second=0.6271803592499999, last_prefill_token_count=64, last_prefill_tokens_per_second=108.18723830098705, last_decode_token_count=64, last_decode_tokens_per_second=28.07935048952399)