dummybot 0.1.0

A scripted LLM test double that verifies what your pipeline feeds the model, not just what it does with the response.

dummybot

A scripted LLM test double that verifies what your pipeline feeds the model — not just what it does with the response.

Most LLM mocking looks like this:

def fake_model(prompt):
    return "42"

That proves your code can handle "42". It proves nothing about the prompt: did the retrieved context actually make it in? Did the system instructions survive your template refactor? Did an internal debug blob leak into the model's input? In LLM pipelines, most real bugs are feed bugs — the model was called with the wrong thing — and a return-only fake is blind to all of them.

Dummybot treats the model seam as a contract boundary:

• Trigger-keyed dispatch — each scripted line answers only prompts containing its trigger substring, in script order, with per-line use limits.
• Prompt input assertions — required substrings, forbidden substrings, and any-of groups checked against the actual prompt your pipeline sent.
• Typed faults, never silent drift — an unmatched call, a failed assertion, or a partially-consumed script raises a structured error naming exactly what went wrong (with a prompt digest and the available triggers).
• Completion contract — assert that every expected model call actually happened; the pytest fixture does this automatically at teardown.
• Scripted failures — lines can simulate provider errors (timeouts, refusals) so you can test your error handling deterministically.
• Zero dependencies — pure standard library; pytest is only needed for the optional fixture.

Install

pip install dummybot

Quickstart

from dummybot import Dummybot, PromptAssertion, Script, ScriptLine

script = Script(
    name="rag-happy-path",
    lines=(
        ScriptLine(
            line_id="answer",
            trigger="USER QUESTION:",           # matches as a substring of the prompt
            response="Paris",
            assertions=(
                PromptAssertion.requires("RETRIEVED CONTEXT:", "Eiffel"),
                PromptAssertion.forbids("INTERNAL_DEBUG"),
            ),
        ),
    ),
)

bot = Dummybot(script)

# Patch it in wherever your pipeline calls its model provider:
answer = my_rag_pipeline(question="Where is the Eiffel Tower?", model_call=bot)

assert answer == "Paris"
bot.assert_all_expected_calls_consumed()   # every expected model call happened

If the pipeline drops the retrieved context on the floor, you don't get a mysterious wrong answer — you get:

DummybotFault: [input_assertion_failed] input_assertion_failed
  assertion_label: 'required_substrings'
  line_id: 'answer'
  missing_required_substrings: ('RETRIEVED CONTEXT:', 'Eiffel')
  prompt_digest: 'sha256:…'

The pytest fixture

Installing the package registers a dummybot fixture (a factory). Bots built through it are automatically completion-checked at test teardown:

def test_pipeline_calls_planner_then_answerer(dummybot):
    bot = dummybot({
        "name": "two-step",
        "lines": [
            {"line_id": "plan", "trigger": "PLAN THE TASK", "response": '{"steps": ["look up"]}'},
            {"line_id": "answer", "trigger": "FINAL ANSWER", "response": "done",
             "assertions": [{"required_substrings": ["steps"]}]},
        ],
    })
    run_pipeline(model_call=bot)
    # teardown fails the test if either line was never used

Opt out per-bot with dummybot(script, autocheck=False).

Scripts as JSON

Scripts are plain data — keep them next to your tests and load them:

from dummybot import load_script
bot = Dummybot(load_script("tests/scripts/rag_happy_path.json"))

{
  "schema": "dummybot_script.v1",
  "name": "rag-happy-path",
  "lines": [
    {
      "line_id": "answer",
      "trigger": "USER QUESTION:",
      "response": "Paris",
      "max_uses": 1,
      "assertions": [
        {"required_substrings": ["RETRIEVED CONTEXT:"], "forbidden_substrings": ["INTERNAL_DEBUG"]}
      ]
    }
  ]
}

Simulating model failures

script = Script(name="outage", lines=(
    ScriptLine(line_id="down", trigger="USER QUESTION:", fault_type="model_unavailable"),
))

bot = Dummybot(script)                      # raises ScriptedModelError on match
bot = Dummybot(script,                      # or raise the exception type your seam catches
               fault_factory=lambda line, prompt: TimeoutError("scripted outage"))

Shaping the return value for your seam

By default a matched line returns plain text (raw_response if set, else response — use raw_response to include <think> blocks your seam is supposed to strip). If your model-call seam expects a richer object, pass a response_factory:

def ollama_shaped(line, prompt, kwargs):
    return {
        "response": line.response,
        "model": kwargs.get("model", "qwen3:4b"),
        "prompt_eval_count": 0,
        "eval_count": 0,
    }

bot = Dummybot(script, response_factory=ollama_shaped)

Helpers for common response shapes:

from dummybot import json_response, thinking_json_response

json_response({"answer": 42})                       # deterministic sorted-key JSON
thinking_json_response("let me think", {"answer": 42})  # <think>…</think> + JSON

Semantics worth knowing

• Matching order: first line in script order whose trigger is in the prompt and that has uses remaining. Exhausted lines fall through to later lines — so two lines with the same trigger model a two-step conversation.
• max_uses is both a cap and an expectation: max_uses=2 means "matches at most twice, and the completion contract expects exactly two uses." max_uses=0 means unlimited and optional.
• Failed assertions don't consume the line — fix the feed and the same line is still there.
• Scripted fault lines do consume the line — the call happened; it's recorded in bot.calls and satisfies the completion contract.
• bot.calls records every matched call: line id, full prompt, prompt digest, and the kwargs your pipeline passed (model name, temperature, …).

Origin

Dummybot was extracted from the test infrastructure of Tamago, a local-first LLM orchestration runtime, where it replaces the model at the Ollama caller seam so that ~2,800 deterministic tests can pin down not just what the pipeline does with model output, but what every stage feeds the model. The feed-assertion idea earned its keep there — most of the bugs it caught were prompts that silently lost or leaked context.

License

MIT