generalcontext

Record your AI agent's model calls for General Context. We read your agent's production conversations and find where it goes wrong, and each failure you confirm becomes an RL and eval task built from the conversation. Your tasks together are your benchmark.

Recording never gets in your agent's way: calls are sent in the background, a failure on our side is logged and dropped, never raised, and there are no runtime dependencies.

Install

pip install generalcontext
export GENERALCONTEXT_API_KEY=gck_...   # from your workspace's Connect page

Record

import generalcontext

generalcontext.initialize(app_name="support-agent")   # once per process, at startup

# Group every call in one conversation under the id you already use for it
with generalcontext.session(conversation_id, user_id=customer_id, channel="chat"):
    reply = handle(message)

generalcontext.flush()   # before a serverless function, job or script exits

If your app serves several companies of its own, pass account= to session() (the id or name you already use for that company) so conversations are grouped by account.

initialize() records every call these clients make in the process, sync and async, streamed or not, from clients created before or after it. Call it in every process, including workers.

Voice (OpenAI Realtime). Connections from client.realtime.connect() (sync and async, needs openai[realtime]) are recorded too. Open the connection inside session(...) and the whole call lands in that conversation. Each response.done is recorded as one openai-realtime call shaped like a Responses call: the instructions and tools, the conversation so far as text (caller turns from the input transcription, agent turns from the output transcript, tool calls and their results), and a voice field with speech_stopped_at, first_audio_at, interrupted and truncated_at_ms. Turn on input audio transcription in the session to record what the caller said. Audio is never recorded. Pass a connection to wrap() to record only that one.

Python 3.9 to 3.13. Supported client versions, redaction, outcomes and events, and every option: SDK reference.

Run your benchmark

Each task runs anywhere, with no access to your systems: it carries what your agent's tools returned in production, a model plays the person, and a judge grades the rubric for a reward from 0 to 1.

pip install generalcontext openai anthropic
generalcontext benchmark pull --out benchmark.json   # needs a key that can run the benchmark
generalcontext evals run benchmark.json --agent my_agent:handle --trials 3 --upload

my_agent:handle is your agent, called as handle(messages, toolbox). See Tasks and environments and the command line.

License

Apache License 2.0. See LICENSE and NOTICE.