Reenact

Deterministic tests for LLM agents, and an eval cache that only pays for what a change touched.

Reenact records a multi-step agent run once - every model call, tool call, and MCP call - and keeps it as a test fixture. On every pull request, reenact check re-runs the agent the pull request contains against those recordings, offline: the model's responses come from the recording, while the requests, tool calls, and control flow are the current code's own. When nothing the agent sends has changed, the test is deterministic, free, and needs no API key. When something has, Reenact shows exactly what changed - and can re-run just those scenarios against the real model.

Install

pip install reenact

reenact[langchain] adds LangChain and LangGraph support; reenact[anthropic] adds the client that model-judged criteria use.

Quickstart

Write one entrypoint that runs your agent with its model client and tools taken from rt; Reenact calls it both to record a scenario and to check it:

import reenact

def run(rt: reenact.Session, *, question: str) -> None:
    client = rt.anthropic()                       # records, or answers from the recording
    search = rt.tool(search_docs, side_effect="read_only")
    reply = rt.tool(post_reply, side_effect="mutating", stub=fake_post_reply)
    my_agent(client, tools=[search, reply], question=question)

Name it in a suite with each scenario's arguments, cassette, and checks, then:

reenact record evals/suite.toml    # run each scenario for real, write its cassette
reenact check evals/suite.toml     # re-run the current agent against them, offline

reenact init scaffolds the entrypoint, a suite skeleton, and a pull-request workflow; reenact suggest drafts the checks from a recording.

What you get

• Tests of the agent's plumbing with the model frozen - $0, no API key.
• Request snapshots - a change to what the agent sends is reported field by field.
• Side-effect safety for every tool - a mutating tool never runs again on replay.
• A live re-run for what changed - --on-divergence live re-runs only the scenarios whose requests changed against the real model, and gates on their checks.
• Baseline-relative gating - fail only on drift, including a check deleted to go green; a GitHub Action posts a sticky comment and a merge-gating check-run.

Works with the Anthropic SDK, OpenAI SDK, LiteLLM, LangChain, and LangGraph.

Documentation

Full docs, the demo, and the GitHub Action: https://github.com/Manas-33/Reenact

License

MIT