Skip to main content

modmex-ai

Lightweight, provider-neutral agents, flows, tools, and voice runtimes for Python.

CI Coverage PyPI Python Versions License

modmex-ai is a small runtime for applications that need an agent to reason, call tools, hand off work, and return a controlled result. It fits equally well in HTTP APIs, CLIs, background workers, WebSocket servers, notebooks, desktop applications, voice systems, and event-driven/serverless workloads.

Its core has no provider SDK dependency. OpenAI is the first complete adapter; the core contracts remain reusable for another text, speech, or realtime provider.

The basic execution model is intentionally simple:

input -> Flow -> one or more Agents -> output, tools, handoffs, and events

What it provides

  • Agent for instructions, structured output, tools, guardrails, and streaming.
  • Flow for multi-agent routing, handoffs, continuation, and sync/async execution.
  • Sessions, snapshots, optimistic concurrency contracts, and approval checkpoints.
  • Realtime and chained voice contracts that keep STT, reasoning, and TTS composable.
  • Provider-neutral tracing, usage, evals, and host-owned observability.

Install the core package in any Python application:

pip install modmex-ai

Add only the optional transport needed by a deployment:

pip install 'modmex-ai[async]'      # native async HTTP
pip install 'modmex-ai[realtime]'  # WebSocket-based Realtime adapters

Quick start

from modmex import BaseModel
from modmex_ai import Agent, FakeModel, Flow


class TriageOutput(BaseModel):
    intent: str
    confidence: float


triage = Agent(
    name="triage",
    instructions="Classify the incoming message.",
    output_type=TriageOutput,
    handoffs=["support"],
)

flow = Flow(
    name="analyze-message",
    entrypoint=triage,
    agents=[],
    model=FakeModel(['{"intent":"support","confidence":0.91}']),
)

result = flow.run("I need help")
result.output.model_dump()

The same Flow can be called by a REST handler, a queue consumer, a CLI command, or a long-lived WebSocket connection. Event-driven systems are a natural fit, but they are not required.

Multi-agent delegation uses handoff function tools, not a field in the model output. A handoff defaults to transfer_to_agent_name; its typed input is validated and delivered to an on_handoff callback before the next agent continues with the conversation.

Multimodal inputs

Messages can combine typed text, document, and image parts without exposing a provider payload in application code. FileInput and ImageInput accept a URL, a provider file id, or inline bytes/Base64 as appropriate. The provider adapter decides which source it supports.

The OpenAI Responses adapter maps these parts to the official input_text, input_file, and input_image request items. For documents stored privately, prefer a short-lived signed URL and create it only for the model request; do not persist it in conversation state or publish it in events.

from modmex_ai import FileInput, InputDetail, Message, TextInput
from modmex_ai.models import ModelRequest
from modmex_ai.providers.openai import OpenAIResponsesModel

request = ModelRequest(
    messages=[
        Message(
            role="user",
            content=[
                TextInput(text="Extract the organization name and expiration date."),
                FileInput(
                    url=presigned_file_url,
                    filename="agreement.pdf",
                    media_type="application/pdf",
                    detail=InputDetail.HIGH,
                ),
            ],
        )
    ]
)

response = OpenAIResponsesModel("gpt-5.6", api_key="...").complete(request)
print(response.output_text)

OpenAIChatModel deliberately rejects FileInput and ImageInput until that adapter implements their distinct wire format. Use OpenAIResponsesModel for multimodal input.

Durable sessions and approvals

The core provides portable SessionSnapshot and PersistedFlowState models, plus optimistic-concurrency store contracts. A host chooses the database and owns the lifecycle; this keeps the framework usable in a web application, a worker, or serverless infrastructure without importing a storage SDK into the core.

Tools can require an externally verified approval. When a protected tool is requested, a Flow raises FlowSuspended with the exact pending tool call and a serializable checkpoint. Persist it, collect the signed approval in the host, then resume the Flow without asking the model to decide again.

For Lambda-oriented hosts, install the optional connector integration and use the DynamoDB stores from modmex-ai:

pip install 'modmex-ai[lambda]'
from modmex_ai.persistence import DynamoDbDurableSessionStore
from modmex_lambda.connectors.dynamodb import Connector

store = DynamoDbDurableSessionStore(Connector("sessions-table"))

Other applications can implement the same DurableSessionStore and FlowStateStore contracts with Postgres, Redis, or their existing storage.

Realtime voice sessions

VoiceAgentSession is the common contract for voice sessions. There are two implementations with distinct responsibilities:

  • OpenAIRealtimeSession para speech-to-speech por WebSocket/SIP.
  • ChainedVoiceSession para conservar el control explícito de STT → Agent/Flow → TTS.

Ambas exponen eventos normalizados, uso acumulado, herramientas y handoffs. El adaptador de OpenAI conserva el payload específico del proveedor únicamente en su frontera; el core usa tool_call_id para no confundir llamadas de tools con identificadores de llamadas telefónicas.

Install the optional WebSocket transport only in workloads that need it:

pip install 'modmex-ai[realtime]'

The OpenAI adapter reuses Agent tools, handoffs, tool guardrails, context, usage, and traces. It does not depend on the OpenAI Agents SDK. For a SIP call that was already accepted by your server, connect the worker using its realtime_call_id:

import asyncio

from modmex_ai import Agent
from modmex_ai.providers.openai import (
    OpenAIRealtimeClient,
    OpenAIRealtimeSessionConfig,
)


async def run_voice_call(realtime_call_id: str) -> None:
    agent = Agent(
        name="support",
        instructions="Help the caller concisely and use tools when needed.",
    )
    client = OpenAIRealtimeClient()
    session = await client.connect(
        agent=agent,
        realtime_call_id=realtime_call_id,
        config=OpenAIRealtimeSessionConfig(voice="marin"),
    )
    async with session:
        await session.configure()
        await session.create_response(
            instructions="Greet the caller and ask how you can help."
        )
        async for event in session.events():
            if event.type == "response.done":
                print(session.usage.model_dump())


asyncio.run(run_voice_call("rtc_u1_example"))

When the model emits a function call, the session validates and executes the registered Modmex-AI tool, sends its output to the socket, and asks Realtime to continue. A transfer_to_* handoff updates the active agent's instructions and tool surface in the same Realtime session.

For a SIP accept endpoint, build the accept body from the exact same session configuration used after the WebSocket connects. That prevents model, voice, instructions and tool definitions from drifting between both phases:

accept_payload = OpenAIRealtimeSessionConfig(voice="marin").to_accept_payload(agent)

Chained voice pipeline

Use a chained session when the application must choose its transcription and speech providers, or needs text-agent behavior between audio turns. A Flow is preferred when the conversation can hand off to another agent.

session = ChainedVoiceSession(
    flow=conversation_flow,
    speech_to_text=transcriber,
    text_to_speech=synthesizer,
    context={"workspace_id": "workspace-1"},
)

audio_reply = await session.process_audio(caller_audio)
async for event in session.voice_events():
    # transcript_final, tool_finished, handoff_completed,
    # response_completed, session_ended, or error
    handle_voice_event(event)

The STT/TTS providers are small async protocols, so concrete provider SDKs stay outside the core package. Close either session with a VoiceTerminationReason to make lifecycle reporting explicit.

For bounded audio turns, the OpenAI adapters use the Audio API directly through the lightweight HTTP client—no provider SDK is installed:

from modmex_ai.providers.openai import (
    OpenAISpeechProvider,
    OpenAITranscriptionProvider,
)

session = ChainedVoiceSession(
    flow=conversation_flow,
    speech_to_text=OpenAITranscriptionProvider(model="gpt-4o-mini-transcribe"),
    text_to_speech=OpenAISpeechProvider(voice="marin", response_format="wav"),
)

To continue a chained conversation in another process, persist the host's Session implementation and session.continuation. The continuation tracks the active agent and provider state; the host owns durable history storage. For a small self-contained host, SessionSnapshot.from_session(session) can serialize an InMemorySession and later restore it with snapshot.to_memory_session().

ChainedVoiceSession.process_audio_stream() supports providers that implement streaming STT and TTS contracts. It emits transcript deltas, waits for a final transcript before invoking the text flow, then yields playable audio chunks. The callable adapters make it possible to plug in a provider-specific function without making that provider a dependency of modmex-ai.

Native async models

Models can optionally expose async def acomplete(request). Agent.run_async and Flow.run_async use it directly, including async tools; existing sync models retain their compatible worker-thread fallback. The built-in OpenAI clients expose acomplete() as well. Install the optional native transport to make them select httpx automatically:

pip install 'modmex-ai[async]'

Without that extra, the compatible stdlib transport runs in a worker and keeps the event loop responsive.

Realtime transcription

For live microphone or telephony media, use the transcription-only WebSocket adapter instead of a speech-to-speech agent session. It emits Transcription deltas as audio arrives and one final value after manual commit:

from modmex_ai.providers.openai import OpenAIRealtimeTranscriptionProvider

transcriber = OpenAIRealtimeTranscriptionProvider()
async for transcript in transcriber.transcribe_stream(pcm_chunks, context=context):
    render(transcript.text, final=transcript.is_final)

The adapter uses 24 kHz PCM by default and requires modmex-ai[realtime]. For completed audio turns, OpenAITranscriptionProvider remains the simpler request-based alternative.

For a long-lived call, open one reusable session and commit each user turn:

session = await transcriber.connect()
await session.append_audio(pcm_chunk)
await session.commit_turn()
async for transcript in session.events():
    correlate(transcript.item_id, transcript.text, transcript.is_final)

Turn cancellation and provider retries

ChainedVoiceSession accepts VoiceTurnOptions(max_provider_retries=...). Retries apply only to STT/TTS provider I/O, never to the text flow, tools, or handoffs. Call cancel_current_turn() to cooperatively stop a streaming turn before its next provider chunk; hosts can still cancel the owning task when an immediate interruption of a blocked transport is required.

After every completed, cancelled, timed-out, or failed bounded turn, ChainedVoiceSession.last_turn contains a VoiceTurnOutcome. Its metrics separate STT, agent, TTS, total latency, and provider retry count. Pass a VoiceTurnObserver to export those outcomes to the host's own logs or metrics backend without adding an observability dependency to this package.

Model Context Protocol (MCP)

MCPClient is the low-level protocol client. MCPClientConnection adds discovery, tool adaptation, caching and lifecycle management so an Agent can consume one or more remote MCP servers through mcp_clients:

from modmex_ai import Agent, MCPClientConnection, MCPClientManager

async with MCPClientManager([
    MCPClientConnection("https://example.com/mcp"),
]) as mcp_clients:
    agent = Agent(
        name="assistant",
        instructions="Use the available remote tools when appropriate.",
        mcp_clients=mcp_clients,
    )
    result = await agent.run_async("Help me with my request")

Use client.list_tools() directly only when the application needs to manage the remote tool catalog itself. Resources and prompts remain explicit:

connection = mcp_clients[0].client
menu = await connection.read_resource("pizza://menu")
prompt = await connection.get_prompt("assistant")

The client supports MCP 2026-07-28, pagination, structured tool results, modern metadata/headers and buffered SSE responses. It does not create transport sessions. The MCP server transport and Lambda/API Gateway integration are provided by modmex-lambda.

Concurrent MCP discovery and tool calls

Async agent runs and MCPClientManager connect to MCP servers concurrently. Each connection caches its tool catalog. Discovery and cursor pagination within one server stay ordered.

Set Agent(..., parallel_tool_calls=True) to execute the tool calls returned in one model response concurrently in run_async and arun_stream. The default is False. Enable this for independent async tools, including MCP calls; synchronous functions still run on the event loop. Tools share the run context, so avoid parallel calls that depend on each other's results or mutate the same state. Streaming progress and finish events arrive as tools produce them; results sent back to the model retain the original call order. A failed or cancelled batch cancels and awaits its remaining tasks. Synchronous runs remain sequential.

Execution resilience and latency (0.5)

Existing agents retain fail-fast tool handling unless explicitly configured:

from modmex_ai import Agent, ModelSettings, RunTimeoutError, execution_budget

agent = Agent(
    name="driver",
    instructions="Use tools to perform operations; never claim unconfirmed writes succeeded.",
    model=model,
    tools=tools,
    tool_error_policy="return_error",
    execution_timeout=25,
    settings=ModelSettings(reasoning_effort="low", max_tokens=500),
)
try:
    result = await agent.run_async("Report arrival")
except RunTimeoutError:
    # The application persists pending work and sends its own deterministic fallback.
    # A timed-out write may already have completed: verify before retrying.
    ...

return_error converts ToolExecutionError/ToolValidationError into sanitized function outputs with ok: false, an error code, and retryable: false. The model can then respond without exposing internal exception text. There are no automatic tool retries. Approvals, remote MCP requests for human input, guardrail blocks, cancellation, unknown tools and execution deadlines still propagate. With recovery enabled, remote MCP tools use their buffered call path even in agent streaming; model streaming remains available, but MCP progress is not emitted.

Execution deadlines are optional and shared with nested agents and HTTP retries. Wrap a whole Flow in with execution_budget(seconds): to share a single deadline across handoffs. Async execution cancels cooperative operations when the budget expires. Synchronous execution checks deadlines between operations and bounds HTTP timeouts; it cannot forcibly interrupt arbitrary blocking Python functions or undo writes. The same limitation applies to blocking synchronous functions called inside async code and synchronous model fallbacks running in worker threads. Closing a stream early is still the caller's responsibility. Do not hold a stream open while running unrelated work in the same task: its execution budget remains active until it is closed.

Trace events include model response duration, first streamed event latency, message count, UTF-8 context/schema sizes, and tool duration/outcome. Byte counts are not token estimates. New metrics do not include prompt text, arguments or exception messages; existing application tracing policies still apply. Non-streaming model failures produce model_error with exception type and elapsed time. Attach existing Trace observers to export these metrics; supply RetryConfig(on_retry=...) for sanitized retry events.

HTTP retries honor Retry-After (seconds or HTTP date), skip exhausted billing quota, respect the active execution budget and support optional jitter_seconds. An explicitly empty status_codes set disables retries. Streams are never replayed after output starts. Transport/network timeouts retain their existing fail-fast policy. No HTTP connection pooling was replaced: AsyncHttpClient already owns/reuses its transport explicitly.

reasoning_effort is omitted by default and maps to Responses reasoning.effort or Chat Completions reasoning_effort. Model support and accepted effort levels vary; choose compatible settings explicitly. The library validates known effort names but does not maintain a model capability catalog or silently remove sampling parameters. See https://developers.openai.com/api/docs/guides/reasoning.

These mechanisms do not send user messages, escalate business cases, change an agent's permissions, or compact business context automatically. Those remain application policies.

Incremental context compaction

Compaction is opt-in and provider-neutral. It is independent of Agent/Flow execution, so hosts can compact the exact business context they assemble, without changing agents.

from modmex_ai import ContextCompactor, CompactionSnapshot, execution_budget

compactor = ContextCompactor(
    model=summary_model,  # Host chooses the model and credentials.
    threshold=24_000,     # Serialized UTF-8 bytes by default, not tokens.
    keep_recent_turns=4,
    max_input_size=32_000,
    max_summary_bytes=6_000,
    preservation_instructions="Preserve load IDs, assignments and unresolved requests.",
)
with execution_budget(25):  # Shared budget for compaction AND the ensuing agent run.
    result = await compactor.compact_async(full_history, previous=saved_snapshot, trace=trace)
    # Persist result.snapshot using your repository and optimistic concurrency control.
    # Supply result.messages as agent input; do not add the previous summary again.
    # If still over your allowed budget, apply an explicit host policy before executing.
    response = await agent.run_async(result.messages)

Pass the full, append-only history every time, together with the last successful CompactionSnapshot (serializable with model_dump()). The snapshot stores summary, covered item count and a digest of the covered prefix. Changed/truncated history or a snapshot from another history is rejected; the host must reset/rebuild it explicitly. Already covered messages are replaced by the existing summary before checking the threshold, and only new complete turns enter the next incremental summary request. Persist the snapshot only for the matching history revision; concurrent turns require host-side synchronization. The original history is never mutated or deleted.

System/developer messages are preserved verbatim, including those inside the covered prefix. Summaries are injected as user-role historical data, never elevated to privileged instructions. Recent user turns are kept verbatim; pending calls and parallel tool-call blocks are never split. Supply tool history as SessionItem function_call/output (or handoff_call/output) pairs with unique call IDs. Orphan/duplicate outputs are rejected; legacy Message(role="tool") records require explicit conversion by the host.

A single bounded batch is compacted per call. Large turns are not truncated to fit the summarizer. Results report unchanged, compacted, blocked or failed, sizes, duration, usage and a safe reason. A compacted result can still exceed the threshold: the host must check after_size and choose whether to compact again, defer or reject execution. Supply a deterministic, monotonic measure=your_token_counter and thresholds in matching units for token budgets; max_summary_bytes remains a byte cap. Empty, oversized, or non-reducing summaries never replace the old snapshot. Provider failures retain the previous effective context and unsummarized messages; cancellation and RunTimeoutError propagate. Usage from completed summarizer responses is reported even if the summary is rejected. Failed provider calls may incur usage unavailable to the library. Summarization is lossy: critical authorization and transaction state must always be reloaded from authoritative application data.

The compactor does not change provider_state or server-managed conversations. Hosts must rebuild/clear incompatible provider state when replacing raw history with a summary; otherwise old history can be sent twice. Compaction does not enable automatic persistence, age filtering, permissions or TMS-specific retention rules. Existing SessionSnapshot.compact remains a separate legacy item-trimming utility and is not automatically replaced.

Metadata

Release files for modmex-ai 0.5.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for modmex-ai 0.5.0
File Size Uploaded Provenance Actions
modmex_ai-0.5.0.tar.gz 131.9 kB With provenance Details for modmex_ai-0.5.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for modmex-ai 0.5.0
File Interpreter ABI Platform Actions
modmex_ai-0.5.0-py3-none-any.whl Python 3 none any Details for modmex_ai-0.5.0-py3-none-any.whl

Total release size: 232.2 kB

Release files / modmex_ai-0.5.0.tar.gz

Download URL modmex_ai-0.5.0.tar.gz
Size 131.9 kB
Tags Source
SHA-256 checksum
How to use checksums
a7baec7ed60be51d9a840a3e7f3b39060b773a851bdfb9be0f9b89dacfe1a319
BLAKE2b-256 checksum
How to use checksums
cfb6f6c47e5f48c4caeaadd8b7d4ad3ac50aa5f23a2dfe181e4f0374213b4c89
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.

Transparency log

Release files / modmex_ai-0.5.0-py3-none-any.whl

Download URL modmex_ai-0.5.0-py3-none-any.whl
Size 100.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
7be72a349e8c759c6e77ea8d5e7ee37edf7998e1448c1c7abf014ea8bbf7898c
BLAKE2b-256 checksum
How to use checksums
63e9ad2ca353f4c9c6eaf4f71e2d7bd7b39445d05884a232dd71cf0f3c1513a7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.

Transparency log

Security for modmex-ai 0.5.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for modmex_ai-0.5.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page