Status

convy is not released yet. This README describes the first release, which is being built now. Nothing below can be installed from PyPI until 0.1.0 is out.

Why convy

Before a team puts an agent in front of people, it wants to know how the agent behaves in a real conversation — not on a single prompt, but over several turns with someone who asks vague questions, changes their mind or never gives the details up front. And when there are several agents, or several builds of one, it wants to compare them on the same conversations.

convy does exactly that and nothing more:

• Any agent, connected in minutes. An agent with a JSON API needs a few lines of settings. Anything else — streaming, a two-step login, a custom protocol — is two small Python classes.
• Conversations, not prompts. A scenario tells a simulated user who to be and what to want; the judge checks plain-language claims about the dialogue.
• Numbers for a decision. Pass rate over repeated attempts, the agent's own tokens and its response time per turn, side by side for every agent and version.
• Small and readable. About a thousand lines, no evaluation framework and no containers underneath.

How it works

             scenario                                       claims
                │                                              │
                ▼                                              ▼
       ┌─────────────────┐  message  ┌─────────┐       ┌─────────────┐
       │ simulated user  │ ────────▶ │  agent  │       │    judge    │──▶ verdict
       │     (model)     │ ◀──────── │         │       │   (model)   │
       └─────────────────┘  answer   └─────────┘       └─────────────┘
                │     time and tokens of every turn           ▲
                └───────────────── transcript ────────────────┘

For every attempt convy opens a fresh conversation with the agent, lets the simulated user talk to it until the user is done or the turns run out, and asks the judge whether every claim holds. Each attempt is written to a journal as soon as it ends, and convy report turns the journals into one static HTML page.

Quick start

uvx convy init my-bench              # a project with examples
cd my-bench
uv run convy run echo --smoke        # check convy itself: free, no models called
cp .env.example .env                 # add the address and key of your model gateway
uv run convy run echo                # a real run with the simulated user and the judge
uv run convy report                  # results/index.html

convy init creates a project of your own:

my-bench/
├── gateway.py        # the model gateway, shared by models.py and the agents
├── models.py         # the models that play the user and the judge
├── agents/           # one file per agent
├── scenarios/        # one YAML file per scenario
└── results/          # journals and the report

Connect an agent

Add agents/support_bot.py. For an agent with a JSON API, settings are enough:

from pydantic import SecretStr
from convy import Env, JsonAgent, JsonEndpoint


class BotEnv(Env):
    bot_token: SecretStr  # BOT_TOKEN in .env


env = BotEnv()
agent = JsonAgent(
    JsonEndpoint(
        "https://bot.example.com/api/chat",
        headers={"Authorization": f"Bearer {env.bot_token.get_secret_value()}"},
    ),
    body={"session_id": "{session}", "message": "{text}"},
    reply="data.answer",
    tokens=("data.usage.input_tokens", "data.usage.output_tokens"),
)

convy sends the body on every turn with {text} replaced by the user's message and {session} by an id that stays the same for the whole conversation. For an agent that does not remember the dialogue, put "{history}" where it expects the list of messages. reply and tokens are paths in the JSON answer.

Files of the project import each other, as in a script run from its folder: settings that several agents share go into one module, such as gateway.py in a new project (from gateway import gateway).

Then check the connection — the agent answers for real, nothing else is called:

uv run convy run support_bot --smoke

Anything a JSON template cannot express is two classes in the same file: an agent that opens a conversation, and a conversation that answers a message.

from contextlib import asynccontextmanager

import httpx2

from convy import Answer, Message, Usage


class SupportBot:
    @asynccontextmanager
    async def conversation(self):
        async with httpx2.AsyncClient(base_url="https://bot.example.com") as http:
            login = await http.post("/login", json={"user": "convy"})
            yield SupportChat(http, login.raise_for_status().json()["token"])


class SupportChat:
    def __init__(self, http: httpx2.AsyncClient, token: str):
        self.http = http
        self.token = token

    async def answer(self, message: Message) -> Answer:
        response = await self.http.post(
            "/chat",
            json={"text": message.text},
            headers={"Authorization": f"Bearer {self.token}"},
        )
        data = response.raise_for_status().json()
        return Answer(data["text"], Usage(input=data["tokens_in"], output=data["tokens_out"]))


agent = SupportBot()

There is no need to catch errors in answer: any exception counts as a failed turn, and convy records it in the dialogue. The text of an exception from your agent goes to the journal and the report, so keep keys out of it; Env's own errors never show the values read.

A client without async

convy's interface is async. If your agent's client only has blocking calls, run them in a thread with asyncio.to_thread, so other conversations go on while one waits:

import asyncio
from contextlib import asynccontextmanager

from support_sdk import Client  # a client with blocking calls only

from convy import Answer, Message


class SupportBot:
    @asynccontextmanager
    async def conversation(self):
        client = Client("https://bot.example.com")
        session = await asyncio.to_thread(client.start_session)
        try:
            yield SupportChat(client, session)
        finally:
            await asyncio.to_thread(client.close)


class SupportChat:
    def __init__(self, client: Client, session: str):
        self.client = client
        self.session = session

    async def answer(self, message: Message) -> Answer:
        text = await asyncio.to_thread(self.client.send, self.session, message.text)
        return Answer(text)


agent = SupportBot()

When a turn runs out of time, convy stops waiting for it, but a thread cannot be stopped: the call goes on in the background until it returns. Give the client a timeout of its own if it has one.

Tokens counted from the start

Some agents report only the tokens spent so far, not those of one answer. Read the counter before and after the turn; the difference is the turn's tokens:

import httpx2

from convy import Answer, Message, Usage


class SupportChat:
    def __init__(self, http: httpx2.AsyncClient):
        self.http = http

    async def answer(self, message: Message) -> Answer:
        before = await self.spent()
        response = await self.http.post("/chat", json={"text": message.text})
        after = await self.spent()
        text = response.raise_for_status().json()["text"]
        return Answer(text, Usage(input=after[0] - before[0], output=after[1] - before[1]))

    async def spent(self) -> tuple[int, int]:
        """The tokens this conversation has spent so far: input and output."""
        usage = (await self.http.get("/usage")).raise_for_status().json()
        return usage["input_tokens"], usage["output_tokens"]

The counter must belong to the conversation. A counter shared by conversations that run at the same time mixes their tokens; run such an agent with --parallel 1.

Corporate certificates

An agent or a model gateway behind a corporate certificate authority takes Tls in its endpoint: ca is an extra root certificate, cert and key a client certificate. Declare the paths in your Env as pydantic's FilePath, so a wrong path stops convy before the run (error: …, exit code 2) instead of failing every attempt:

from pydantic import FilePath
from convy import Env, JsonAgent, JsonEndpoint, Tls


class CorpEnv(Env):
    corp_ca: FilePath  # CORP_CA in .env


env = CorpEnv()
agent = JsonAgent(
    JsonEndpoint("https://bot.corp.example/api/chat", tls=Tls(ca=str(env.corp_ca))),
    body={"message": "{text}"},
    reply="answer",
)

FilePath does not expand ~: write the full path. A file that exists but is damaged shows only on connection — check an agent with --smoke.

Scenarios

A scenario is a YAML file in scenarios/:

id: clarify-backup
max_turns: 6
user: |
  You need a backup script. Start short: "Make me a backup script".
  Give details only when the assistant asks for them:
  back up ~/photos to /mnt/backup, once a day.
judge:
  - Before giving a finished script, the agent asked what to back up and where to
  - The final script copies ~/photos to /mnt/backup and runs once a day

user tells the simulated user who to be and what to want; judge lists claims that must all hold for the scenario to pass. An attempt in which the agent fails a turn does not pass, whatever it said before. When one of convy's own models fails instead — it does not answer, or the judge answers without a verdict in JSON ("pass" of true or false) — the attempt gets no verdict and is left out of the pass rate. Write scenarios in any language — the simulated user speaks the language of its instructions.

uv run convy run support_bot -k 3                  # every scenario, three attempts each
uv run convy run support_bot --scenarios 'refund-*'
uv run convy run support_bot new_bot               # several agents, one after another
uv run convy run support_bot --turn-timeout 60

--turn-timeout (600 by default) is the seconds for each step of the agent: opening a conversation, a turn, closing. A step that takes longer fails the attempt as an agent error.

The report

results/index.html is a single static page that works offline:

• agents — for each agent and version: the share of attempts passed, every attempt as a mark (● passed, ✕ failed, ○ no verdict), the agent's tokens and time per scenario, its answer time, and problems such as agent errors; the best value in each column is in bold;
• scenarios — scenarios × agents, split into those where the agents' results differ and those where they are the same; a filter such as refund-* narrows the list;
• a result — select a cell to read each attempt: the claims, the judge's reasoning, and the dialogue with the time and tokens of each answer. The address keeps the selected result, so a link to the page opens it.

Tokens and time are the agent's own: the simulated user and the judge are not counted.

Run from Python

Everything the command does is in the library. A run is a Bench played against an agent into a journal; the report is built from the journals:

import asyncio
from datetime import datetime
from pathlib import Path

from convy import Bench, JsonlJournal, Models, Report, RunHeader, Runs, Scenario
from convy.fakes import Echo, FakeModel

scenario = Scenario(
    id="greet",
    max_turns=3,
    instructions="Say hello to the assistant, then thank it.",
    claims=("The agent answered the greeting",),
)
models = Models(  # fakes: nothing is called; use OpenAiModel for real ones
    user=FakeModel("Hello!", "Thank you!", "###STOP###"),
    judge=FakeModel('{"pass": true, "reason": "it answered"}'),
)
bench = Bench((scenario,), models)
header = RunHeader(
    agent="echo",
    version="",
    user=models.user.name,
    judge=models.judge.name,
    attempts=bench.attempts,
    planned=bench.planned(),
    started=datetime.now().astimezone(),
)
runs = Path("results/runs")
outcomes = asyncio.run(bench.run(Echo(), JsonlJournal(runs, header)))
Path("results/index.html").write_text(Report(Runs(runs)).html(), encoding="utf-8")

Put your own agent in place of Echo(). The fakes in convy.fakes are public too, for testing your agent classes without a model.

FAQ

Does convy need to know which model my agent uses? No. convy talks to the agent, not to its model. Only convy's own models — the simulated user and the judge — are set in models.py.

My agent does not report tokens. Then the report shows time and results without tokens. If tokens matter, ask for them in the agent's answer, or read them from the model gateway the agent uses.

What do the exit codes mean?

• 0 — every attempt finished;
• 1 — an attempt ended with an agent error or a failure of convy's models;
• 2 — the project could not be loaded, and nothing ran.

Is it safe to run convy on someone else's project? convy run executes models.py and agents/*.py. Treat a project like its tests: run only what you trust. See the security policy.

Documentation

Contributing

Issues and pull requests are welcome. Read CONTRIBUTING.md first, and follow the Code of Conduct. Report vulnerabilities privately, as the security policy describes.

License

MIT