AID runtime — Python

The Python runtime library: the behavior layer imported by apps generated by the py-fastapi generator. Generated code is thin glue; this is where the behavior lives (docs/design.md §4, principle 1).

Distribution name aid-runtime-python, import package aid_runtime. The two differ deliberately: a distribution name is conventionally hyphenated, and a Python import package cannot be. That asymmetry is asserted in tests/test_package.py so it cannot drift.

Requires Python 3.11+.

pip install -e ".[dev]"        # the runtime plus its test/lint/type tooling
pytest -q
pip install -e ".[pgvector]"   # only if you want the pgvector store adapter

What it provides

194 names are exported from aid_runtime. The layering is acyclic and one-directional: errors and types import nothing else in the package, gateway is imported only by the package __init__ and the eval runner, and no core module imports a higher layer back.

Model pinning

An alias resolves to an exact provider model id, including a pinned version. That is not cosmetic: a baseline recorded against gpt-4o means nothing if the alias floats to gpt-4o-2026-… on the next run, so an unknown alias is an error (UnknownModelAliasError) rather than a silent fall-through to a provider default.

from aid_runtime import (
    Budget,
    CompletionRequest,
    CostProfile,
    ModelGateway,
    ModelSpec,
    OpenAICompatibleProvider,
    PinnedModel,
)

spec = ModelSpec(
    name="gpt-4o-2024-08-06",
    provider="openai-compatible",
    cost=CostProfile(input_per_1k_usd=0.0025, output_per_1k_usd=0.01),
    context_window=128_000,
    max_output_tokens=4096,
)

gateway = ModelGateway(
    providers={"openai-compatible": OpenAICompatibleProvider(api_key=...)},
    aliases={"small": PinnedModel(alias="small", spec=spec)},
    budget=Budget(max_usd=5.0, scope="nightly"),
)

result = await gateway.generate("small", CompletionRequest.prompt("hello"))
result.text, result.usage, result.cost_usd, result.alias, result.attempts, result.fell_back

• resolve(alias) returns the PinnedModel (alias + spec); provider_for(alias) its provider. require(alias, capabilities) asserts the capabilities and returns the pinned model, checking both sides — a model that supports tool calling behind an adapter that does not serialise tool definitions cannot tool-call — and raises CapabilityMismatchError naming what is missing.
• generate(alias, request, *, fallbacks=(), budget=None) tries the alias then each fallback in the order given, and reports the aliases actually attempted on the result. Fallbacks are ordered, not conditional.
• Every charged call opens a span and updates the ledger. CostLedger tracks per-call LedgerEntry records and yields a SpendSummary; Budget (max_usd, max_total_tokens, scope) is enforced before the call, and estimate_usage / estimate_cost_usd price a request from a model's CostProfile when a caller wants to check before spending.

The eval harness gates on statistics, not equality

Phase 0 ratifies sampling-based gating: a model is expected to vary between runs, so a metric that compares two runs for equality is either always red or measuring nothing. Non-determinism here is not a bug to be fixed; it is the property the gate has to be robust against.

• Dataset — versioned cases with inputs, expectations, tags and metadata; from_records / from_json / from_path, to_json, and lookups by id (case) and by tag (by_tag).
• Metrics — ExactMatch, NormalizedContains, JsonSchemaValid, and LlmJudge. metric_names and default_metrics cover the common set.
• run_evaluation(dataset, runner, metrics, runs=5) samples the runner N times and produces an EvalRun of per-run metric means, so a single unlucky run does not decide the verdict. evaluate_once is the single-pass primitive; a DatasetRun gives per-case outcomes and a rendered summary().
• EvalBaseline.from_run(run, model=..., note=...) stores a baseline as JSON: per metric the sample count, mean and spread. runtimes/python/tests/fixtures/capitals-baseline.json is a committed example.
• BaselineGate(thresholds).evaluate(run, baseline) compares the current sample against the baseline using a one-sided normal test, with each Threshold carrying metric, an optional absolute min_mean, a max_regression allowance, and an alpha significance level. A metric passes by clearing min_mean, by being within max_regression of the baseline, or by regressing only within sampling noise.

Two deliberate limits:

• LlmJudge is implemented and never CI-required. A judge is itself a model call, and a gate that depends on one non-deterministic call inside the gate you are trying to trust is the problem this design avoids. gateway_judge(gateway, alias) builds the judge call when a caller wants it.
• GateConfigurationError refuses a threshold on a model-derived metric, because an N-run sample of a probabilistic judge cannot carry a fixed threshold honestly.

cost is the enforced budget; a latency p95 budget belongs with Phase 2 dashboards, not here.

Retrieval

retrieval/ is an IR-driven layer: binding_from_spec(retriever, embedding=..., store=...) folds the IR's retrievers, embeddings and stores entries into one RetrieverBinding, which a Retriever then ingests and searches through. The runtime consumes the names the IR emits, field for field.

• Three protocols, in the same style as providers.Provider: Embedder, Reranker, VectorStore. Phase 1 ships the implementations listed below and no others.
• FakeDeterministicEmbedder maps text to a stable unit vector with a hash — keyless and offline, which is what lets a retrieval gate run in CI. OpenAICompatibleEmbedder is the one hosted adapter: a single POST to /embeddings over httpx, with the retry, timeout and error-chaining behaviour of the HTTP provider.
• LexicalReranker is the only reranker. It is deterministic and lexical; there is no hosted rerank adapter.
• Chunk ids are content hashes, so re-ingesting the same input produces the same ids and a no-op diff. Ingest is therefore held to idempotence rather than to the byte-identical codegen guarantee.
• Pinning. embedModel names an IR embeddings entry and resolves at codegen time to an exact modelId plus its recorded width; the resolved id, width and store kind travel into the generated code. An alias is never resolved at runtime, because a baseline recorded against a floating model means nothing.
• Cost. Embedding calls charge the existing CostLedger through the existing Budget, using the catalog's CostProfile for the pinned model. There is no retrieval-specific budgeting machinery.

RetrievalMetric (RecallAtK, MeanReciprocalRank, CitationGrounding) is deliberately a separate protocol from the shipped Metric, which scores a single string prediction. Retrieval scores a retrieved set: recall@k and MRR need the ranked ids, and citation-grounding needs the passages behind the citations. Widening Metric would have changed an interface the existing gate already threshold-checks, for every existing implementation — so retrieval is sampled through RetrievalGate.sample(...) instead, producing a standard EvalRun, and then judged by the unchanged BaselineGate. Retrieval metrics are deterministic, so they pass the gate's own eligibility check.

pgvector

PgVectorStore is a thin adapter over psycopg and the vector extension. Two consequences worth stating:

• It is locally provisioned and not covered by any gate. No CI job starts a database, and no required check touches one; the in-memory adapter carries the store gates, and the two adapters share the same VectorStore protocol and the same similarity semantics. compose.pgvector.yml beside this README brings up a database for local use.
• The [pgvector] extra is import-safe when absent: importing aid_runtime or aid_runtime.retrieval never imports the driver, and the adapter loading it raises a ConfigurationError naming the extra.

Control flow is plain async

Pipeline is a list of named async (or plain) steps over a PipelineContext that is frozen, so a step cannot mutate what an earlier step observed — it returns a new context, which keeps a failed run's state inspectable. A step may return a context, a mapping, or None. Each step runs in its own span, a step raising anything aborts the run with the original exception chained (PipelineError.__cause__), and StopPipeline is the supported early exit.

Pydantic AI is used for structured output and nothing else (structured.build_agent). The orchestration library stays out of the control-flow path deliberately, so it remains swappable, and no graph/orchestration dependency is taken at all: the agent loop in agents/ is this package's own, and escalation routing and third-party orchestration frameworks remain out of scope (docs/design.md §14).

Tracing follows the OTel GenAI semantic conventions: gen_ai.* attributes for the operation, model, provider, request parameters and usage, with AID-specific values namespaced aid.* so the two never collide. A failing call records its exception exactly once.

The agent runtime

AgentRuntime.from_ir(document, providers=..., ...) reads the emitted IR and runs an agent to a stop condition: model call, tool calls, observations, until a terminal answer, maxSteps, or a budget breach. Model calls go through ModelGateway, so fallback, cost accounting and tracing apply without a second implementation.

Transfer and delegation are different, and the API says which. _transfer (handoffs) hands the conversation to the target and never resumes the caller, so RunResult.agent_id is the target and path grows. _delegate (delegates) runs the target in an isolated [system, user] context and returns its answer as a single tool observation, so RunResult.agent_id is the caller and path is unchanged. Both refuse to re-enter an agent already on the path.

A budget breach fails hard. BudgetExceededError stops the run; the gateway preflight and the ledger both enforce the ceiling. An agent declaring onBudgetExhausted: 'degrade' (v0.2) instead steps exactly one rung down its declared alias ladder — no retry loop, and the build-time gate rejects the declaration when no strictly cheaper alias exists. Each degraded step is recorded in StepRecord.degraded_from, and a checkpoint carries its current alias so a resumed run re-routes from where it stopped.

Memory is in-process plus one store-backed adapter. thread scope is the checkpointer unit: trimmed to its tokenBudget, never written to a store. user and global scopes read and write the store and render as one extra system message, so injection is visible in the request rather than hidden.

Human-in-the-loop pauses by returning a checkpoint (completed steps, pending call, current alias), and resume(checkpoint, decision) continues it. The pause is recorded as an aid.hitl span event, not an attribute, because a run can pause more than once.

Development

Four gates, all run by the python job in .github/workflows/ci.yml over real content:

pytest
ruff check .
ruff format --check .
mypy

mypy runs strict over src and tests, so the tests are type-checked too. The python job is a real gate but is advisory: the main ruleset requires the node, workflows and licenses contexts only.

tests/fixtures/capitals-baseline.json is regenerated from FakeProvider(accuracy=0.6, seed=0) over 10 runs — the fake is seeded and deterministic per call, so the committed baseline reproduces exactly while a fresh sample of it still varies, which is exactly the property the gate has to tolerate.

Non-responsibility: no app-specific prompts, tools, or rubrics. Those belong to the generated app's business/ tier (docs/design.md §8.4, the seam).

Status: Phase 0 scope implemented, plus the Phase 1 retrieval runtime. One thing deferred and undeclared, on purpose: ModelGateway.stream raises FeatureDeferredError, and no provider advertises Capability.STREAMING or target declares the capability, because the transport is request/response-only. Retrieval likewise fails loudly rather than approximating: non-file sources, semantic chunking, incremental refresh, retriever filters, a hosted reranker, and the qdrant / pinecone / weaviate / custom store kinds all raise FeatureDeferredError. Also deferred (§14): escalation routing, memory, guardrails, prompt/semantic caching, rate limiting, token buckets, and adapters beyond the one HTTP provider and the fake.

Emitting retrieval into a generated app and indexing it from the CLI are later layers, so this package adds no entry point for either.

Reference: docs/design.md §9, §12, §14, §17 #2.