verbatim-relay

Test your chat agent through Claude Code or Codex. The harness model does not retype one message or one reply.

verbatim-relay is a test harness for chat agent development. You talk to your agent in the coding harness where you already work. The relay sends each message to your agent byte for byte, and shows each reply byte for byte. The model does not run while you talk. At the end, the model reads the exact record and evaluates your agent: business logic, tone, accuracy.

It adapts to your app through a thin entry in .verbatim-relay/, and your app's code does not change. The cost is plumbing: you write the entry once, and you change it when the start or the wiring of your app changes.

How it works

tester ──> harness ──> relay ──────> tap ──stdin/stdout──> entry ──> your app
                         │            │
                         ▼            ▼
                    relay.jsonl    tap.jsonl
            (what you typed      (what the agent
             and saw)             received and sent)
                         │            │
                         └─> audit <──┘   exit 0 clean, 1 break, 2 invalid record

Two processes write two records, and the audit compares them byte for byte. It reports 7 break classes (SPEC.md section 3.3). It fails closed: it never reports clean on a record that it cannot read. 53 audit cases, 20 contract cases, 10 trace cases, 11 seal cases and 7 receiver cases in conformance/ test the spec (cases, tests).

A test also records what your app did. It copies the session files of the app's own Agent SDK or Codex sessions. It records the backend calls, the direct model calls and the OpenTelemetry spans of the app. docs/architecture.md shows each part.

Install

uv tool install verbatim-relay

It needs Python 3.10 or later, and it has no runtime dependencies (pyproject.toml). Version 0.2.0 or later has tests with an entry. If verbatim-relay --version shows 0.1.0, install from the repo: uv tool install --force git+https://github.com/mohanraj00/verbatim-relay.

Quick start

In a clone of this repo, with the toy shop agent and the hook kit in Claude Code:

verbatim-relay init claude-code --entry "python3 examples/toy-shop/agent.py"
verbatim-relay check        # one message through the entry: PASS
verbatim-relay view         # in a second terminal: each reply shows here

In Claude Code, type the prompt verbatim-relay start, then your test messages, then verbatim-relay end. The model then evaluates the test and writes report.md. Audit the records:

verbatim-relay audit --tap .verbatim-relay/tests/<test-id>/tap.jsonl --relay .verbatim-relay/tests/<test-id>/relay.jsonl

docs/getting-started.md shows each step with its real output. For the plugin, which shows each reply in the chat, read docs/how-to/claude-code.md. For Codex, read docs/how-to/codex.md. For your own app, ask the harness model: "Run verbatim-relay setup and connect a test to this app" (docs/how-to/connect-your-agent.md).

Results

Data: plugin, hook kit in Claude Code, hook kit in Codex, evaluation. Method: docs/results.md.

Under pressure, the mechanism had 0 breaks in 1,000 turns. The test had 40 scripted conversations in each harness, 5 or 20 turns long. They had refusals, HTTP 500 errors, questions back to the tester and ambiguous messages. Each turn was a new harness call. I registered the design before the first run. Method, data and the one deviation: docs/results.md.

Docs

The full map is docs/index.md. Read docs/limits.md before you trust a result: the deny is best effort, and the entry is not audited.

Contribute

Read CONTRIBUTING.md and the code of conduct. Report a security problem as SECURITY.md says.

License

Apache-2.0. See LICENSE.