verbatim-relay

Test your chat agent through Claude Code or Codex. The harness model does not retype one message or one reply.

verbatim-relay is a test harness for chat agent development. You talk to your agent in the coding harness where you already work. The relay sends each message to your agent byte for byte, and shows each reply byte for byte. The model does not run while you talk. At the end, the model reads the exact record and evaluates your agent: business logic, tone, accuracy.

It adapts to your app through a thin entry in .verbatim-relay/, and your app's code does not change. The cost is plumbing: you write the entry once, and you change it when the start or the wiring of your app changes.

How it works

tester ──> harness ──> relay ──────> tap ──stdin/stdout──> entry ──> your app
                         │            │
                         ▼            ▼
                    relay.jsonl    tap.jsonl
            (what you typed      (what the agent
             and saw)             received and sent)
                         │            │
                         └─> audit <──┘   exit 0 clean, 1 break, 2 invalid record

• Harness: Claude Code or Codex, where you type your test messages.
• Relay: the plugin or the hook kit in the harness. It carries each message and each reply, and the model writes neither.
• Tap: a proxy in front of your agent. It forwards each byte with no change, and records what the agent received and sent.
• Entry: a thin wrapper that starts your app and speaks one JSON line in and one JSON line out.
• Audit: a program that compares the two records byte for byte, and names each break.

docs/architecture.md shows this flow with each part, and has diagrams of one turn, the test lifecycle and the trace.

Two processes write two records, and the audit compares them byte for byte. It reports 7 break classes (SPEC.md section 3.3). It fails closed: it never reports clean on a record that it cannot read. 53 audit cases, 20 contract cases, 10 trace cases, 11 seal cases and 7 receiver cases in conformance/ test the spec (cases, tests).

A test also records what your app did. It copies the session files of the app's own Agent SDK or Codex sessions. It records the backend calls, the direct model calls and the OpenTelemetry spans of the app. docs/architecture.md shows each part.

Install

uv tool install verbatim-relay

It needs Python 3.10 or later, and it has no runtime dependencies (pyproject.toml). The docs are for version 0.3.0. To upgrade from an earlier version, end each running test first. Then run uv tool upgrade verbatim-relay (CHANGELOG.md). docs/how-to/upgrade.md gives each step.

Quick start

Select one route: try the toy shop, connect your app, or choose a relay.

Try the toy shop

In a clone of this repo, with the toy shop agent and the hook kit in Claude Code:

verbatim-relay init claude-code --entry "python3 examples/toy-shop/agent.py"
verbatim-relay check        # one message through the entry: PASS proves the connection, not the reply
verbatim-relay view         # in a second terminal: each reply shows here

In Claude Code, type the prompt verbatim-relay start, then your test messages, then verbatim-relay end. The end of the test writes audit.json in the test folder .verbatim-relay/tests/<test-id>/. Then the model evaluates the test and writes report.md. docs/getting-started.md shows each step with its real output.

Optional: to see the audit again, run it by hand:

verbatim-relay audit --tap .verbatim-relay/tests/<test-id>/tap.jsonl --relay .verbatim-relay/tests/<test-id>/relay.jsonl

Connect your app

In the harness, in the project folder of your app, type !verbatim-relay setup. Then type the prompt "Follow the verbatim-relay setup guide, and connect a test to this app." docs/how-to/test-your-app.md gives each step from the install to the results, with the expected output. To write the entry yourself, read docs/how-to/connect-your-agent.md.

Choose a relay

The plugin shows each reply in the chat of Claude Code. The hook kit works in Claude Code and in Codex, and shows each reply in verbatim-relay view. docs/how-to/choose-a-relay.md compares the 2 relays. Then read the guide of your relay: Claude Code plugin, Claude Code hook kit or Codex.

Results

Data: plugin, hook kit in Claude Code, hook kit in Codex, evaluation. Method: docs/results.md.

Under pressure, the mechanism had 0 breaks in 1,000 turns. The test had 40 scripted conversations in each harness, 5 or 20 turns long. They had refusals, HTTP 500 errors, questions back to the tester and ambiguous messages. Each turn was a new harness call. I registered the design before the first run. Method, data and the one deviation: docs/results.md.

Docs

The how-to rows are in the order of the tasks of a test: choose, connect, run, read the results, fix. The row after them has the tasks that maintain verbatim-relay.

The full map is docs/index.md. Read docs/limits.md before you trust a result: the deny is best effort, and the entry is not audited.

Contribute

Read CONTRIBUTING.md and the code of conduct. Report a security problem as SECURITY.md says.

License

Apache-2.0. See LICENSE.