aid-runtime-python

The Python runtime for AID Framework: the behavior layer that apps generated by the py-fastapi target import. Generated code is thin glue; this is where the behavior lives.

Distribution name aid-runtime-python, import package aid_runtime. The two differ deliberately: a distribution name is conventionally hyphenated, and a Python import package cannot be. That asymmetry is asserted by the test suite so it cannot drift.

Install

pip install aid-runtime-python

Requires Python 3.11 or newer. Dependencies are httpx, pydantic, pydantic-ai-slim and opentelemetry-api.

Two optional extras:

pip install "aid-runtime-python[pgvector]"   # the pgvector store adapter
pip install "aid-runtime-python[deepagents]" # the deepagents agent engine, pinned

Quick start

from aid_runtime import (
    Budget,
    CompletionRequest,
    CostProfile,
    ModelGateway,
    ModelSpec,
    OpenAICompatibleProvider,
    PinnedModel,
)

spec = ModelSpec(
    name="gpt-4o-2024-08-06",
    provider="openai-compatible",
    cost=CostProfile(input_per_1k_usd=0.0025, output_per_1k_usd=0.01),
    context_window=128_000,
    max_output_tokens=4_096,
)

gateway = ModelGateway(
    providers={"openai-compatible": OpenAICompatibleProvider(api_key=...)},
    aliases={"small": PinnedModel(alias="small", spec=spec)},
    budget=Budget(max_usd=5.0, scope="nightly"),
)

result = await gateway.generate("small", CompletionRequest.prompt("hello"))

result.text, result.usage, result.cost_usd, result.alias, result.attempts, result.fell_back

What it provides

249 names are exported. The layering is acyclic and one-directional: errors and types import nothing else in the package, gateway is imported only by the package __init__ and the eval runner, and no core module imports a higher layer back.

Model pinning

An alias resolves to an exact provider model id, including a pinned version. That is not cosmetic: a baseline recorded against gpt-4o means nothing if the alias floats to gpt-4o-2026-… on the next run, so an unknown alias is an error (UnknownModelAliasError) rather than a silent fall-through to a provider default.

• resolve(alias) returns the PinnedModel (alias + spec); provider_for(alias) its provider.
• require(alias, capabilities) asserts the capabilities and returns the pinned model, checking both sides — a model that supports tool calling behind an adapter that does not serialise tool definitions cannot tool-call — and raises CapabilityMismatchError naming what is missing.
• generate(alias, request, *, fallbacks=(), budget=None) tries the alias then each fallback in the order given, and reports the aliases actually attempted. Fallbacks are ordered, not conditional.
• Every charged call opens a span and updates the ledger. CostLedger tracks per-call LedgerEntry records and yields a SpendSummary; Budget (max_usd, max_total_tokens, scope) is enforced before the call, and estimate_usage / estimate_cost_usd price a request from a model's CostProfile when a caller wants to check before spending.

Decisions are not chat

DecisionClient calls TypeSafe AI's Jev (POST /v1/systemone), which takes a state plus typed questions and returns typed answers instead of prose. It is deliberately not in providers/: every provider there maps messages to text, and Jev has no text surface, so a pipeline cannot select it as the model that writes an answer.

from aid_runtime import DecisionClient, DecisionQuestion

async with DecisionClient() as client:
    response = await client.decide(
        model="jev-1.14.0",
        state={"message": message},
        questions={
            "triage": DecisionQuestion(
                type="choice",
                instructions="Which team should handle this",
                criteria=("billing", "technical"),
            )
        },
    )
    response.answers["triage"].choice

The key is TYPESAFE_API_KEY (Vercel's spelling, TYPESAFE_AI_API_KEY, is accepted too); TYPESAFE_BASE_URL overrides the host. Two rules matter more than they look: a noul answer carries noul and no confidence field at all, so a branch on confidence must refuse it rather than default to zero; and decide(..., reproducible=True) refuses the moving aliases jev-latest and jev-preview, which is what eval scoring needs. Only 429/529 are retried, honouring retry-after.

The eval gate measures statistics, not equality

A model is expected to vary between runs, so a metric that compares two runs for equality is either always red or measuring nothing. Non-determinism is not a bug to be fixed here; it is the property the gate is built to be robust against.

• Dataset — versioned cases with inputs, expectations, tags and metadata; from_records / from_json / from_path, to_json, and lookups by id (case) and by tag (by_tag).
• Metrics — ExactMatch, NormalizedContains, JsonSchemaValid, and LlmJudge. metric_names and default_metrics cover the common set.
• run_evaluation(dataset, runner, metrics, runs=5) samples the runner N times and produces an EvalRun of per-run metric means, so a single unlucky run does not decide the verdict. evaluate_once is the single-pass primitive; a DatasetRun gives per-case outcomes and a rendered summary().
• EvalBaseline.from_run(run, model=..., note=...) stores a baseline as JSON: per metric the sample count, mean and spread. A committed example ships at runtimes/python/tests/fixtures/capitals-baseline.json.
• BaselineGate(thresholds).evaluate(run, baseline) compares the current sample against the baseline using a one-sided normal test. Each Threshold carries a metric, an optional absolute min_mean, a max_regression allowance, and an alpha significance level. A metric passes by clearing min_mean, by being within max_regression of the baseline, or by regressing only within sampling noise.

Two deliberate limits:

• LlmJudge is implemented and never CI-required. A judge is itself a model call, and a gate that depends on one non-deterministic call inside the gate you are trying to trust is the problem this design avoids. gateway_judge(gateway, alias) builds the judge call when a caller wants it.
• GateConfigurationError refuses a threshold on a model-derived metric, because an N-run sample of a probabilistic judge cannot carry a fixed threshold honestly.

cost is the enforced budget. A latency p95 budget belongs with dashboards, not here.

Retrieval

retrieval/ is an IR-driven layer: binding_from_spec(retriever, embedding=..., store=...) folds the IR's retrievers, embeddings and stores entries into one RetrieverBinding, which a Retriever then ingests and searches through. The runtime consumes the names the IR emits, field for field.

• Three protocols, in the same style as providers.Provider: Embedder, Reranker, VectorStore.
• FakeDeterministicEmbedder maps text to a stable unit vector with a hash — keyless and offline, which is what lets a retrieval gate run in CI. OpenAICompatibleEmbedder is the one hosted adapter: a single POST to /embeddings over httpx, with the retry, timeout and error-chaining behavior of the HTTP provider.
• LexicalReranker is the only reranker. It is deterministic and lexical; there is no hosted rerank adapter.
• Chunk ids are content hashes, so re-ingesting the same input produces the same ids and a no-op diff. Ingest is held to idempotence rather than to the byte-identical codegen guarantee.
• Pinning. embed_model names an IR embeddings entry and resolves at codegen time to an exact model_id plus its recorded width; the resolved id, width and store kind travel into the generated code. An alias is never resolved at runtime, because a baseline recorded against a floating model means nothing.
• Cost. Embedding calls charge the existing CostLedger through the existing Budget, using the catalog's CostProfile for the pinned model. There is no retrieval-specific budgeting machinery.

RetrievalMetric (RecallAtK, MeanReciprocalRank, CitationGrounding) is deliberately a separate protocol from the shipped Metric, which scores a single string prediction. Retrieval scores a retrieved set: recall@k and MRR need the ranked ids, and citation grounding needs the passages behind the citations. Retrieval is therefore sampled through RetrievalGate.sample(...), producing a standard EvalRun, and then judged by the unchanged BaselineGate. Retrieval metrics are deterministic, so they pass the gate's own eligibility check.

pgvector

PgVectorStore is a thin adapter over psycopg and the vector extension.

• It is locally provisioned and not covered by CI. No CI job starts a database; the in-memory adapter carries the store tests, and both adapters share the same VectorStore protocol and similarity semantics. compose.pgvector.yml in the repository brings up a database for local use.
• The [pgvector] extra is import-safe when absent: importing aid_runtime or aid_runtime.retrieval never imports the driver, and loading the adapter raises a ConfigurationError naming the extra.

Control flow is plain async

Pipeline is a list of named async (or plain) steps over a PipelineContext that is frozen, so a step cannot mutate what an earlier step observed — it returns a new context, which keeps a failed run's state inspectable. A step may return a context, a mapping, or None. Each step runs in its own span, a step raising anything aborts the run with the original exception chained (PipelineError.__cause__), and StopPipeline is the supported early exit.

Pydantic AI is used for structured output and nothing else (structured.build_agent). The orchestration library stays out of the control-flow path deliberately, so it remains swappable, and no graph or orchestration dependency is taken at all: the agent loop in agents/ is this package's own.

Tracing follows the OTel GenAI semantic conventions: gen_ai.* attributes for the operation, model, provider, request parameters and usage, with AID-specific values namespaced aid.* so the two never collide. A failing call records its exception exactly once.

The agent runtime

AgentRuntime.from_ir(document, providers=..., ...) reads the emitted IR and runs an agent to a stop condition: model call, tool calls, observations, until a terminal answer, max_steps, or a budget breach. Model calls go through ModelGateway, so fallback, cost accounting and tracing apply without a second implementation.

Transfer and delegation are different, and the API says which. _transfer (handoffs) hands the conversation to the target and never resumes the caller, so RunResult.agent_id is the target and path grows. _delegate (delegates) runs the target in an isolated [system, user] context and returns its answer as a single tool observation, so RunResult.agent_id is the caller and path is unchanged. Both refuse to re-enter an agent already on the path.

A budget breach fails hard. BudgetExceededError stops the run; the gateway preflight and the ledger both enforce the ceiling. An agent declaring on_budget_exhausted: "degrade" (spec v0.2) instead steps exactly one rung down its declared alias ladder — no retry loop, and the build-time gate rejects the declaration when no strictly cheaper alias exists. Each degraded step is recorded in StepRecord.degraded_from, and a checkpoint carries its current alias so a resumed run re-routes from where it stopped.

Memory is in-process plus one store-backed adapter. thread scope is the checkpointer unit: trimmed to its token_budget, never written to a store. user and global scopes read and write the store and render as one extra system message, so injection is visible in the request rather than hidden.

Human-in-the-loop pauses by returning a checkpoint (completed steps, pending call, current alias), and resume(checkpoint, decision) continues it. The pause is recorded as an aid.hitl span event, not an attribute, because a run can pause more than once.

Limits, stated plainly

These are refused explicitly rather than approximated — each raises FeatureDeferredError or a ConfigurationError naming the reason:

• ModelGateway.stream — the transport is request/response only, and no provider advertises Capability.STREAMING.
• Retrieval: non-file sources, semantic chunking, incremental refresh, retriever filters, a hosted reranker, and the qdrant / pinecone / weaviate / custom store kinds.
• Escalation routing and third-party orchestration frameworks.

Development

From a checkout of the repository:

pip install -e ".[dev]"
pytest
ruff check .
ruff format --check .
mypy

mypy runs strict over src and tests, so the tests are type-checked too.

Non-responsibility: no app-specific prompts, tools, or rubrics. Those belong to the generated app's business/ tier — the seam this runtime is called from.

See also

• @aid-framework/runtime — the same runtime in TypeScript, module for module
• @aid-framework/generator-py-fastapi — the generator that emits apps against this runtime

License

Apache-2.0. Code generated by AID Framework is not subject to this license.