langgraph-scenario-lab

Behavioral scenario testing for production LangGraph applications: write a scenario as plain Python, run it against the real graph, and assert on the execution trace.

scenario → real graph execution → trace → assertions → report.

Why

Features

• Scenarios as plain Python — no YAML, no JSON, no custom DSL.
• Trace-based assertions — answer, tool calls, node path, state (incl. nested result.state["a"]["b"].equals(...)), status events, errors, sources.
• Fault injection — lab.fail / lab.mock to test retries, fallbacks, and network failures against the real graph.
• LLM mocking — lab.mock_llm("classifier", {...}) replaces the model call inside a graph node with a canned response for deterministic routing tests; the default always uses the real model.
• Budget assertions — result.llm.calls <= 3, result.llm.tokens <= 5000, result.latency < 10, result.retries, result.interrupts.
• Record / Replay — run --mode record once, then --mode replay for fast, deterministic regression checks without re-contacting tools.
• Multi-turn scenarios — lab.scenario().user(...).run(...).
• Custom domains — product-specific entities exposed as result.<name>, registered via ScenarioLab(domains=[...]).
• Rich terminal report — colourised live progress, exit code 0/1/2.
• pytest integration — an autouse lab fixture; same suite convention.

Install

uv pip install langgraph-scenario-lab
# or
pip install langgraph-scenario-lab

Requires Python ≥ 3.11.

Quickstart

A suite is a directory with lab.py (builds the lab) and scenarios/ (auto-collected test_*.py files):

scenarios_lab/
├── lab.py                  # def build_lab() -> ScenarioLab
└── scenarios/
    └── test_small_talk.py

lab.py:

from langgraph_scenario_lab import ScenarioLab

def build_lab() -> ScenarioLab:
    from examples.demo_graph import build_graph
    return ScenarioLab(build_graph())

scenarios/test_small_talk.py:

from langgraph_scenario_lab import ScenarioLab


def test_small_talk(lab: ScenarioLab) -> None:
    result = lab.run("Hi")

    result.answer.matches(r"hi|help")
    result.tools.never_called("search_knowledge")
    result.status.matches(r"^Thinking")
    result.path.contains("answer")

Run it:

scenario-lab run                 # live, against the real graph
scenario-lab run --mode record   # live + save recordings under the suite
scenario-lab run --mode replay   # deterministic: no graph calls, no tool calls

Scenario Lab
────────────────────────────────────────
✓ test_small_talk
8 passed, 0 failed

See docs/quickstart.md for the full walkthrough.

CLI

scenario-lab run [target] [--mode live|record|replay] [--record-dir DIR]
                 [--replay-dir DIR] [--jsonl FILE] [--quiet] [--log-file FILE]
                 [--log-level debug|info|warning|error] [--name-filter SUBSTR]
scenario-lab version

• target — suite directory (default scenarios_lab/), a scenarios/ subdirectory, a test_*.py file, or file::test_name.
• --mode record / --mode replay implement the record/replay cycle.
• --jsonl appends one JSON line (scenario, status, trace) per run.
• --name-filter re-runs only the scenarios whose name contains the substring.
• Exit codes: 0 all passed, 1 failures, 2 suite/usage error.

A failing assertion prints a red FAILURES section with the expected vs actual (reproduce with scenario-lab run examples/demo_suite/scenarios/test_failing_example.py):

Full reference in docs/cli.md.

Writing scenarios

def test_rag(lab: ScenarioLab) -> None:
    result = lab.run("What is Nimbus?")
    result.answer.contains("Nimbus")
    result.sources.nonempty()
    result.tools.called("search_knowledge")
    result.nodes.visited("query_rewrite")
    result.status.sequence(r"^Thinking", r"^Searching", r"^Composing")

Fault injection

def test_retry(lab: ScenarioLab) -> None:
    lab.fail("search_knowledge", TimeoutError("flaky"), times=1)
    result = lab.run("What is flaky search?")
    result.errors.expected("search_knowledge")
    assert result.invocations["search_knowledge"] == 2  # retried on the graph

def test_mock(lab: ScenarioLab) -> None:
    lab.mock("search_knowledge", return_value="Nimbus fake docs")
    result = lab.run("What are the mock docs?")
    result.answer.contains("Nimbus fake docs")

Multi-turn

def test_context(lab: ScenarioLab) -> None:
    scenario = lab.scenario()
    scenario.user("My favorite color is blue.")
    result = scenario.run("What is my favorite color?")
    result.state.has("messages")

Custom domains

Expose product entities as result.<key> without touching the core:

# scenarios_lab/domains.py
from langgraph_scenario_lab import DomainModel

class StateSourcesDomain(DomainModel):
    key = "my_sources"
    def bind(self, trace):
        return list(trace.state.get("sources") or [])

def my_sources() -> StateSourcesDomain:
    return StateSourcesDomain()

# scenarios_lab/lab.py
def build_lab() -> ScenarioLab:
    from domains import my_sources
    return ScenarioLab(build_graph(), domains=[my_sources()])

# scenarios/test_domain.py
def test_sources(lab: ScenarioLab) -> None:
    result = lab.run("What is Nimbus?")
    assert result.my_sources == ["https://docs.nimbus.local/nimbus"]

pytest integration

The package registers a pytest11 plugin. With the suite on disk:

pytest scenarios_lab/scenarios

The autouse lab fixture serves the suite's lab, clears faults/mocks between tests, and auto-discovers lab.py walking up from the test file. Pass an explicit suite with --scenario-suite path/to/suite.

See docs/pytest-plugin.md.

Record / replay

1. Record a golden baseline:

   scenario-lab run --mode record        # saves scenarios_lab/recordings/*.json
   

2. Change the prompt or graph, then replay the old trace against the new assertions (no tool calls, fully deterministic):

   scenario-lab run --mode replay
   

Recordings are JSON traces keyed by the last human/user message; per query the cleanest (error-free) recording wins.

Diagnostics

• result.diff(baseline) renders a BEFORE/AFTER execution diff when behavior changed (node path, statuses, tool calls, errors, answer).
• --jsonl gives machine-readable per-scenario results.
• --log-file + --log-level mirror the rich progress to a durable log.

About the fixtures

• examples/demo_graph.py — a compact Nimbus/Vega RAG graph used by the docs walkthrough and the dogfood suite.
• examples/support_graph.py — a non-trivial example: a support chat with intent routing, knowledge-base retrieval + ranking, subscription and escalation tools, a RetryPolicy, extra state keys (intent, queries, source_documents, sources), and status stream events. Read its docstring to see where each asserted-on piece of data comes from.
• examples/demo_suite/ — a richly commented scenario suite (15 cases) covering the assertion catalog — answer, tools, path, status, state, errors, sources, custom domains, fault injection, multi-turn — wired against support_graph. Run it with scenario-lab run examples/demo_suite.
• scenarios_lab/ is the dogfood suite used by the tests and the CLI.

Development

uv sync --extra dev
ruff check && ruff format --check
uv run python -m pytest tests/ scenarios_lab/scenarios/ -q

CI (.github/workflows/ci.yml) runs the same checks on Python 3.11–3.13 and smoke-installs the built wheel in a clean venv. Releasing is tag-driven: publish.yml builds and publishes to PyPI via trusted publishing whenever a v* tag is pushed.

uv build                      # local wheel + sdist sanity check
uvx --from build twine check dist/*  # verify long description metadata
git tag v0.2.0 && git push origin v0.2.0

Documentation

• Changelog
• Constitution — the design contract the code is built against
• Quickstart
• Example suite: support chat — 15 commented scenarios against a non-trivial graph
• CLI reference
• Execution modes: live / record / replay
• Assertion catalog
• Fault injection
• Custom domains
• pytest plugin
• Architecture
• API reference

License

MIT