polyrt

Unified Python interface for LLMs across local and cloud backends.

polyrt is a typed adapter library that exposes a single small surface — generate, stream, and their async variants — backed by Anthropic's Claude, OpenAI's GPT models, and local Apple-Silicon models via MLX. It does not route, it does not orchestrate, it does not run a server: it gives you one consistent function call shape and one consistent return type, regardless of which backend you point it at.

Install

Python 3.11+ required.

Quickstart

Sync

"""Synchronous one-shot generation."""

import polyrt

result = polyrt.generate(
    "claude",
    model="claude-haiku-4-5",
    messages=[{"role": "user", "content": "Say hello in one word."}],
    max_tokens=20,
)
print(result.text)
print(f"Tokens: {result.usage.input_tokens} in, {result.usage.output_tokens} out")

Async streaming

"""Async streaming generation."""

import asyncio

import polyrt


async def main() -> None:
    async for event in polyrt.astream(
        "openai",
        model="gpt-4.1-mini",
        messages=[{"role": "user", "content": "Count to five."}],
        max_tokens=40,
    ):
        if event.type == "token":
            print(event.delta, end="", flush=True)
        elif event.type == "done" and event.usage is not None:
            print(f"\n\n[{event.usage.total_tokens} tokens]")


asyncio.run(main())

Structured output

"""Schema-constrained structured output."""

from pydantic import BaseModel

import polyrt


class City(BaseModel):
    name: str
    country: str
    population: int


result = polyrt.generate(
    "claude",
    model="claude-haiku-4-5",
    messages=[{"role": "user", "content": "Pick a notable city and return its data."}],
    schema=City,
)
assert isinstance(result.parsed, City)
print(f"{result.parsed.name}, {result.parsed.country} — pop {result.parsed.population:,}")

Backends

polyrt.generate("anthropic", ...) and polyrt.generate("claude", ...) are equivalent — pick whichever name reads better in your call site.

Schema enforcement

When you pass schema=SomeModel, polyrt asks the backend to constrain its output to that Pydantic model and parses the result for you. The strategy differs by backend:

• Claude — passes the schema as a single tool with tool_choice forced to that tool, then parses the tool's input payload.
• OpenAI — uses Structured Outputs (response_format with type=json_schema, strict=true) so the model is constrained at decode time.
• MLX — not yet supported in v0.1; raises NotSupportedError. v0.2 will add support via outlines or lm-format-enforcer.

Response.parsed is your validated Pydantic instance when it works, None otherwise.

What polyrt is not

• Not a router that picks a backend for you
• Not an agent framework
• Not a model server or proxy
• Not a fine-tuning toolkit
• Not a prompt template engine

It is a typed adapter. If you want one of the things above, this is the layer you would build it on top of.

Adding a backend

1. Subclass Backend (or duck-type the protocol) and implement the seven methods.
2. Call polyrt.registry.register("<name>", YourFactory) at module import time.
3. Either ship inside polyrt/backends/<name>.py, or expose your factory as a polyrt.backends entry point so users get it via pip install.

See docs/backends.md for the full contract.

License

Apache 2.0. See LICENSE.

Acknowledgments

• anthropic-sdk-python
• openai-python
• mlx-lm
• litellm — prior art for the unified-interface idea.