langgraph-chat-agent-eval 0.1.1

Evaluation framework for LangGraph chatbots on multi-turn conversations

langgraph-chat-agent-eval

Test your LangGraph agents the way users actually use them — across multi-turn conversations, with deterministic tool mocks and an LLM judge that scores every step.

A pytest-based evaluation framework for testing LangGraph chat agents end-to-end.

Motivation

Unit tests and mocked LLM calls can verify that your agent's plumbing is wired correctly, but they don't tell you whether the agent actually behaves well. This library fills that gap: it runs real LLM calls against your compiled graph, mocks tool responses so tests are deterministic, and uses an LLM-as-judge to assess whether the agent completed its goal and how good the response was.

The result is a quantitative signal — tool recall, response quality, latency — that you can track over time in CI and use to catch regressions before they reach production.

Features

• Multi-turn conversations — define scenarios with multiple steps; the evaluator simulates a user when the agent needs clarification
• Deterministic tool mocks — replace any tool's coroutine with a fixed return value or callable, without touching the agent code
• Arg-level assertions — verify not just that a tool was called, but that it was called with the right arguments
• LLM-as-judge — works with any LangChain-compatible model (OpenAI, Anthropic, Google, etc.)
• JSON output — results saved per run, easy to diff in CI or feed into dashboards
• Terminal viewer — eval-view renders a color-coded report from any result file

Installation

pip install langgraph-chat-agent-eval

The package depends on langgraph and langchain-core. It does not depend on any specific LLM provider — bring your own.

Quick start

1. Define a scenario

# tests/evaluation/scenarios/my_scenario.py
from langgraph_chat_agent_eval import (
    EvaluationScenario, ConversationStep, ExpectedToolCall, ToolMock
)

def build_my_scenario() -> EvaluationScenario:
    return EvaluationScenario(
        name="search_and_create",
        description="User searches for items then creates a new one",
        shared_mocks=[
            ToolMock(
                tool_name="search_items",
                return_value=[{"id": "1", "name": "Widget"}],
            ),
        ],
        steps=[
            ConversationStep(
                name="search",
                user_message="Find all widgets",
                expected_tools=[
                    ExpectedToolCall("search_items", with_args={"query": "widgets"}),
                ],
                expected_answer="Agent lists the available widgets",
            ),
            ConversationStep(
                name="create",
                user_message="Create a new widget called Gadget",
                expected_tools=[
                    ExpectedToolCall("create_item", with_args={"name": "Gadget"}),
                    ExpectedToolCall("search_items", required=False),  # optional re-list
                ],
                expected_answer="Agent confirms the widget was created",
                tool_mocks=[
                    ToolMock("create_item", return_value={"id": "2", "name": "Gadget"}),
                ],
            ),
        ],
    )

2. Wire up fixtures

# tests/evaluation/conftest.py
import pytest
from langchain_openai import ChatOpenAI           # or any other provider
from langgraph.checkpoint.memory import MemorySaver
from langchain_core.tools import BaseTool

from langgraph_chat_agent_eval import Evaluator
from my_agent import build_graph, get_tools, AgentConfig

pytest_plugins = ["langgraph_chat_agent_eval.fixtures"]


@pytest.fixture(scope="session")
def eval_context() -> AgentConfig:
    return AgentConfig()

@pytest.fixture(scope="session")
def eval_graph(eval_context: AgentConfig):
    return build_graph(eval_context, checkpointer=MemorySaver())

@pytest.fixture(scope="session")
def all_tools(eval_context: AgentConfig) -> list[BaseTool]:
    return get_tools(eval_context)

@pytest.fixture(scope="session")
def evaluator() -> Evaluator:
    return Evaluator(llm=ChatOpenAI(model="gpt-4o", temperature=0))

3. Write the test

# tests/evaluation/test_my_scenario.py
import pytest
from langgraph_chat_agent_eval import ConversationRunner
from scenarios.my_scenario import build_my_scenario

@pytest.mark.asyncio
@pytest.mark.evaluation
async def test_my_scenario(eval_context, eval_graph, evaluator, all_tools):
    runner = ConversationRunner(eval_graph, evaluator, all_tools, eval_context)
    metrics = await runner.run_scenario(build_my_scenario())

    assert metrics.all_steps_completed
    assert metrics.avg_tool_recall >= 0.8
    assert metrics.avg_response_quality >= 0.6

4. Run

If using plain pip:

pytest tests/evaluation/ -m evaluation -v

If using uv:

uv run pytest tests/evaluation/ -m evaluation -v

Results are saved to tests/evaluation/output/YYYYMMDD_HHMMSS_<scenario>.json.

Viewing results

If using plain pip:

eval-view                           # shows latest run for each scenario
eval-view path/to/result.json       # shows a specific file

If using uv:

uv run eval-view
uv run eval-view path/to/result.json

The viewer prints a summary panel, a per-step table, and a full conversation replay with tool call details and color-coded scores.

Running the example

The examples/ directory contains a working end-to-end example with a simple notes agent (two tools: search_notes and create_note). It is a good starting point to verify your setup and understand how the framework fits together.

git clone https://github.com/your-org/langgraph-chat-agent-eval
cd langgraph-chat-agent-eval
uv sync

Set your OpenAI API key (the example uses ChatOpenAI — swap in any provider you prefer):

export OPENAI_API_KEY=sk-...

Run the evaluation:

uv run pytest examples/ -v

View the results:

uv run eval-view

See examples/README.md for a walkthrough of what each file does.

Core concepts

EvaluationScenario

A sequence of ConversationSteps representing a complete user workflow. Shared mocks apply to all steps and can be overridden per step.

ConversationStep

One logical goal within the conversation. The runner drives the agent through up to max_turns (default 5), with the evaluator acting as the user when follow-ups are needed.

ExpectedToolCall

Declares a tool the agent should call. Use required=False for optional tools (calling them earns recall credit; skipping them doesn't penalize). Use with_args to verify specific arguments were passed — matching is a shallow subset check.

ExpectedToolCall("search", required=True, with_args={"query": "widgets", "limit": 10})

ToolMock

Replaces a tool's async coroutine with a stub. Use return_value for a fixed response, or side_effect for a callable (useful for stateful sequences) or an exception (to test error handling).

ToolMock("search", return_value=[{"id": "1"}])
ToolMock("flaky_tool", side_effect=TimeoutError("upstream down"))
ToolMock("stateful", side_effect=iter([result_1, result_2]).__next__)

Evaluator

Wraps any LangChain BaseChatModel and exposes two methods used internally by the runner:

• check_step_completion — judges whether the agent's response satisfied the step goal
• evaluate_step — scores response quality 0.0–1.0 on correctness, tool usage, clarity, and goal achievement

from langchain_anthropic import ChatAnthropic
evaluator = Evaluator(llm=ChatAnthropic(model="claude-opus-4-6"))

ConversationRunner

Drives a scenario against a compiled graph. Each instance gets a unique thread_id for state isolation.

runner = ConversationRunner(
    graph=eval_graph,
    evaluator=evaluator,
    all_tools=all_tools,
    context=eval_context,
    scenario_timeout_seconds=120,   # optional wall-clock cap
)
metrics = await runner.run_scenario(scenario)

Metrics

run_scenario returns a ConversationMetrics object:

Field	Description
all_steps_completed	True only if every step passed
avg_tool_recall	Mean of per-step recall (matched required tools / total required)
avg_response_quality	Mean of LLM-judged quality scores (0.0–1.0)
total_turns	Total agent turns across all steps
steps	List of StepMetrics with per-step detail

Pytest fixtures reference

Import with pytest_plugins = ["langgraph_chat_agent_eval.fixtures"] and override in your conftest.py:

Fixture	Scope	Notes
eval_context	session	Your agent's config object — must override
eval_graph	session	Compiled StateGraph with MemorySaver — must override
all_tools	session	Full tool list for mock lookup — must override
evaluator	session	Evaluator instance — must override
_clear_tool_cache	function	autouse no-op; override to reset tool state between tests

LangGraph assumptions

The framework assumes your graph:

1. Has a messages key in its state holding BaseMessage objects
2. Uses interrupt() before tool confirmation steps (auto-approved with Command(resume="y"))
3. Uses async BaseTool instances (mocking replaces tool.coroutine)
4. Is compiled with a MemorySaver checkpointer (state persists across turns within a test)