Agent-Harness

Open-source test harness for AI agents that take real-world actions.

Problem

Teams can observe agents in traces and score outputs with existing tools, but they lack a shared, pytest-friendly way to assert the sequence of tool calls, arguments, and safety properties on a run. Agent-Harness provides trace-oriented assertions and a CLI so those checks can run in CI without calling real APIs by default. It is complementary to observability and LLM evaluation stacks.

Install

Core package (editable, from repo root):

pip install -e "."

LangGraph adapter, dev tooling, and optional resource/cost helpers (matches CI and typical agent projects):

pip install -e ".[langgraph,dev]"

[dev] includes pytest-asyncio, typing stubs, and pulls [resource] (tokencost) per pyproject.toml. Other extras: [openai], [anthropic], [crewai], [live], [compliance], [langfuse], [arize], or [all].

PyPI (alpha): The published package is pytest-agentharness. To match CI and the LangGraph example:

pip install "pytest-agentharness[langgraph,dev]==0.1.0a2"

The GitHub repository name is Agent-Harness; the Python package import remains agentharness.

Quickstart

Behavioral checks use @scenario, the run fixture, and assertions on run.trace. For a trace that includes tool arguments (not only tool names), run the bundled LangGraph example test from the repo root:

from agentharness import (
    assert_approval_gate,
    assert_arg_lte,
    assert_called_before,
    scenario,
)


@scenario("examples/01_customer_support_langgraph/scenarios/happy_path.yaml")
def test_happy_path(run):
    assert_called_before(run.trace, "lookup_order", "issue_refund")
    assert_arg_lte(run.trace, tool="issue_refund", arg="amount", value=100)
    assert_approval_gate(run.trace, tool="issue_refund")

pip install -e ".[langgraph,dev]"
python -m pytest examples/01_customer_support_langgraph/test_refund_agent.py::test_happy_path -q

The example package overrides the run fixture so YAML steps execute under LangGraph with recorded args. More detail: examples/01_customer_support_langgraph/README.md.

What Agent-Harness is not

• Not a monitoring or observability platform (use LangFuse or Arize Phoenix for that)
• Not a full LLMOps platform
• Not framework-specific (not a LangChain product)
• Not an LLM benchmark (not SWE-bench or WebArena)
• Not a replacement for LangSmith, DeepEval, or TruLens — complementary behavioral testing over traces

Available assertions

CLI

agentharness run <scenario.yaml>
python -m agentharness run <scenario.yaml>

Optional --mode mock|live (default mock).

Demo and screenshots

Agent-Harness is terminal-first (pytest + CLI)—there is no separate web UI to demo. Step-by-step commands and a short live script: docs/demo.md.

Pytest — example happy-path (test_happy_path):

CLI — agentharness run on the safety scenario (mock):

Replay + diff — cassette comparison:

Roadmap

Phase 0 (foundation, pytest plugin, assertions, LangGraph adapter, CLI run, example agent) is complete. 0.1.0a2 is on PyPI as an alpha (pytest-agentharness). Phase 1 continues with additional adapters (e.g. OpenAI, CrewAI), multi-run statistical mode, and follow-on releases. A fuller public roadmap is planned before the Phase 1 launch milestone.

License

Apache License 2.0 — see LICENSE and pyproject.toml.

Contributing

See CONTRIBUTING.md for workflow and review expectations.