ByoAI Runtime (byoai-runtime)

Bring Your Own Infrastructure (BYOI). ByoAI Brings the Runtime.

ByoAI Runtime is an infrastructure-agnostic AI agent engine and workflow execution layer for Python. It connects directly to your existing Redis clusters, vector databases (pgvector, Pinecone, Qdrant), LLMs, and telemetry pipelines without requiring data migrations, vector re-indexing, database schema alterations, or vendor lock-in.

⚡ Why ByoAI Runtime?

Most AI frameworks force engineering teams to adapt their database schemas, re-embed millions of vectors, and rewrite state management logic. ByoAI Runtime adapts to your existing stack instead.

• 🔌 Zero Vector Re-indexing (Schema Mapping): Connect directly to existing vector tables using declarative column mapping.
• 🌲 Cross-Provider AST Filter Parser: Pass unified JSON filters; ByoAI translates them on the fly into native target dialects (pgvector JSONB, Pinecone $eq, Qdrant payload filters).
• 🔒 Non-Invasive Cache Isolation: Isolates internal runtime keys (byoai:*) while using pattern-mapped readers to read existing chat histories safely.
• 🛡️ Resilient Provider Routing & Fallbacks: Native rate-limit management, retries, and dynamic model failovers (e.g., OpenAI ➔ Azure OpenAI ➔ Ollama).
• 📊 Zero-SaaS Telemetry: Native OpenTelemetry (OTLP) trace emission directly to your existing Grafana, Datadog, or Honeycomb collectors.

🏗️ Architecture & Execution Loop

ByoAI Runtime executes as an unopinionated, process-level orchestrator sitting above your existing production data layers:

                          ┌──────────────────────────┐
                          │   runtime.execute()      │
                          └─────────────┬────────────┘
                                        │
           ┌────────────────────────────┼────────────────────────────┐
           ▼                            ▼                            ▼
┌──────────────────────┐    ┌──────────────────────┐    ┌──────────────────────┐
│  Redis Key Reader    │    │  AST Filter Parser   │    │ Dynamic Model Router │
│ (App Chat Ingestion) │    │ (Schema Mapping DB)  │    │ (Failover / Retry)   │
└──────────┬───────────┘    └───────────┬──────────┘    └───────────┬──────────┘
           │                            │                            │
           ▼                            ▼                            ▼
   Existing Redis DB            Existing Vector DB           LLM APIs / Inference
 (No keys overwritten)       (No vector re-indexing)      (Existing API keys)

🚀 Quickstart

1. Installation

pip install --pre byoai-runtime

--pre is required while the package is in pre-release (0.1.0a1).

2. Hello world

import asyncio
from byoai import Runtime

async def main():
    runtime = Runtime(llm={"provider": "openai", "model": "gpt-4o"})  # reads $OPENAI_API_KEY
    result = await runtime.execute("What are our enterprise SLA terms?")
    print(result.content, result.usage.total_tokens, result.cached)
    await runtime.close()

asyncio.run(main())

No cache, no vector store, no telemetry — just a provider. Everything else in this README is opt-in: add cache=/vector_store=/semantic_cache=/telemetry= only for what you need, when you need it. The next example shows all of them wired up against real infrastructure. See Getting Started for environment variables, system_prompt=, and async with Runtime(...).

3. Execution Example (Production Setup)

from byoai import Runtime

# Connect to existing infrastructure without altering schemas or keys
runtime = Runtime(
    cache={
        "provider": "redis",
        "url": "redis://redis.internal:6379",
        "namespace": "byoai:",  # Isolates ByoAI state
        "session_reader": {
            "pattern": "app:users:{user_id}:chat_history", # Ingests existing history
            "format": "json"
        }
    },
    vector_store={
        "provider": "pgvector",
        "dsn": "postgresql://user:pass@localhost:5432/production_db",
        "table": "document_embeddings",
        "schema_map": {
            "id": "doc_id",
            "embedding": "embedding_v2",
            "content": "raw_text",
            "metadata": "payload_json"
        }
    },
    llm={
        "provider": "openai",
        "model": "gpt-4o",
        "fallback": {
            "provider": "azure_openai",
            "endpoint": "https://prod.openai.azure.com",
            "deployment": "gpt-4-prod",
        }
    },
    telemetry={
        "provider": "opentelemetry",
        "endpoint": "http://otel-collector.internal:4317"  # your existing collector
    },
)

# Execute through the runtime (async, from any async framework)
result = await runtime.execute(
    "What are our enterprise SLA terms?",
    user_id="usr_9912",
    filters={"department": {"$eq": "legal"}}  # Translated automatically to JSONB SQL
)

print(result.content, result.usage.total_tokens, result.cached)

4. Drop into an existing FastAPI app

from fastapi import Depends, FastAPI
from byoai import Runtime
from byoai.integrations.fastapi import attach, get_runtime, stream_response

app = FastAPI()               # your existing app
attach(app, Runtime(llm={"provider": "openai", "model": "gpt-4o"}))

@app.post("/ask")
async def ask(body: dict, rt: Runtime = Depends(get_runtime)):
    result = await rt.execute(body["query"])
    return {"content": result.content, "usage": result.usage.__dict__}

@app.post("/ask/stream")      # Server-Sent Events token streaming
async def ask_stream(body: dict, rt: Runtime = Depends(get_runtime)):
    return stream_response(rt, body["query"])

See examples/fastapi_app/ for a runnable app with events, caching, and fallback.

5. Semantic (intent) caching

Serve similar questions from cache — not just identical ones. One embedding call (~15ms) replaces the whole LLM round-trip when intent matches:

runtime = Runtime(
    llm={"provider": "openai", "model": "gpt-4o"},
    cache={"provider": "redis", "url": "redis://redis.internal:6379"},  # exact match
    semantic_cache={"provider": "memory", "threshold": 0.92},           # intent match
    embedder={"provider": "openai", "model": "text-embedding-3-small"},
)

await runtime.execute("What are our enterprise SLA terms?")   # LLM call (~800ms)
await runtime.execute("Tell me about our enterprise SLAs")    # intent hit (~16ms)

Measured ~50× faster on intent hits; lookups stay sub-millisecond to ~30k cached answers (benchmarks/RESULTS.md).

For production, back the intent cache with your existing Redis so hits are shared across every worker/replica and survive restarts:

semantic_cache={"provider": "redis", "url": "redis://redis.internal:6379",
                "threshold": 0.92, "capacity": 10_000, "ttl": 3600}

Entries live in one byoai:-namespaced Redis stream; each worker keeps a local numpy mirror and syncs incrementally, so similarity math never leaves the process. Redis Cluster and Sentinel are supported everywhere Redis is ("mode": "cluster" or "mode": "sentinel" + sentinels/service_name).

🛠️ Core Capabilities

1. Zero-Migration Schema Mapping

No need to run migration scripts or duplicate tables. Define a schema_map during initialization to bridge ByoAI to your existing table structures:

vector_config = {
    "provider": "pgvector",
    "dsn": "...",
    "table": "enterprise_knowledge",
    "schema_map": {
        "id": "uuid",
        "embedding": "vector_768",
        "content": "body_text",
        "metadata": "attributes_json"
    }
}

2. AST Filter Translation

Avoid provider-specific query lock-in. Pass standard logical filter expressions and ByoAI compiles them into native query dialects:

                      [ AST Filter Parser ]
                                │
       ┌────────────────────────┼────────────────────────┐
       ▼                        ▼                        ▼
pgvector (SQL / JSONB)    Pinecone (JSON Dict)     Qdrant (Payload Filter)
`attributes_json->>'dept'  `{"dept": {"$eq":      `FieldCondition(key="dept",
 = 'legal'`                "legal"}}`              match=MatchValue("legal"))`

3. Non-Invasive State Management

ByoAI writes operational artifacts (semantic cache, execution traces, intent plans) under its isolated key namespace while reading existing user sessions read-only:

cache_config = {
    "provider": "redis",
    "url": "redis://localhost:6379",
    "namespace": "byoai:",  # All writes go to byoai:cache:*, byoai:planner:*
    "session_reader": {
        "pattern": "session:{user_id}:messages",
        "format": "json"
    }
}

📊 Framework Comparison

📦 Supported Adapter Ecosystem

Cache & Memory

• Redis (Standalone, Cluster, Sentinel)
• Valkey
• In-Memory (Dev/Testing)

Vector Databases

• PostgreSQL + pgvector
• Pinecone
• Qdrant
• Anything else via a byoai.vector_stores plugin — see Vector stores.

LLM Providers

• OpenAI
• Anthropic (direct API, AWS Bedrock, or Google Vertex AI)
• Azure OpenAI
• Google Gemini
• Ollama / vLLM / LiteLLM
• OpenRouter / Any OpenAI-compatible REST endpoint

Observability

• OpenTelemetry (Datadog, Grafana, Honeycomb, Jaeger, New Relic) — gRPC or HTTP OTLP.

Transports

One execution, five ways in — all share the same payload/result dialect:

• FastAPI — byoai.integrations.fastapi (HTTP, SSE, WebSocket)
• Robyn (Rust-powered) — byoai.integrations.robyn (HTTP, SSE, WebSocket)
• MCP — byoai.integrations.mcp: expose the runtime as a tool any MCP client (Claude Desktop, another agent) can call, over stdio or streamable HTTP — with a streaming tool variant (live token deltas as progress notifications)
• Queue workers — byoai.workers: RuntimeWorker + RedisStreamQueue/MemoryJobQueue
• Or embed Runtime directly in any async Python process

Configuration

Every adapter's every setting — timeouts, retry classification, connection pooling, TTLs, capacity bounds, batch sizes, and more — is documented in CONFIGURATION.md.

🧩 Agent Context Cache

A standalone proxy, separate from Runtime, that sits in front of the Anthropic API. Point Claude Code (or any Anthropic API client) at it and it injects prompt-cache breakpoints and dedupes repeated large text blocks within a session, cutting token spend without any client-side changes.

pip install --pre "byoai-runtime[agent-context-cache]"
byoai-cache                                    # runs in the foreground
export ANTHROPIC_BASE_URL=http://localhost:8787

Prefer to keep it running without holding a terminal open? Start it detached:

byoai-cache start      # background; survives closing the terminal
byoai-cache status     # running (pid …) → http://localhost:8787
byoai-cache stop

start writes its pid and logs under ~/.byoai/ (proxy.pid, proxy.log). Both byoai-cache and the longer byoai-agent-context-cache are the same command. Override the bind address/port with --host / --port (or the BYOAI_HOST / BYOAI_PORT env vars). It listens on :8787 by default and uses Redis for session/dedup state if REDIS_URL is set (falls back to an in-process store otherwise). Full env var reference in CONFIGURATION.md.

Reaching the proxy from a remote client (ngrok)

localhost only works for a client on the same machine. To route a remote client — Claude's web/mobile apps, a phone, a cloud agent — through the proxy, expose it with a tunnel such as ngrok:

BYOAI_PROXY_TOKEN=$(openssl rand -hex 16) byoai-cache start   # gate it first
ngrok http 8787                                               # public https URL

Set BYOAI_PROXY_TOKEN before exposing the proxy. Without it, a public URL is an open relay to api.anthropic.com (and, if you configured the OpenAI-compat backend, anyone could spend your BYOAI_OPENAI_COMPAT_API_KEY). With it set, every request must carry the token, supplied either way:

• Header — x-byoai-proxy-token: <token> (for clients that allow custom headers).
• URL path — put the token in the base URL, for clients that only let you set one: ANTHROPIC_BASE_URL=https://<id>.ngrok-free.app/<token>. The leading /<token> segment is stripped before routing.

/health stays reachable without the token so a tunnel can probe liveness. For a private, always-on setup between your own devices, Tailscale (point clients at the proxy host's tailnet IP) avoids a public endpoint entirely.

Inspecting token-savings data

The proxy keeps a durable SQLite log of per-request usage and tokenizer-verified benchmark samples at BYOAI_SQLITE_PATH (default ~/.byoai/byoai_runtime.db). The /v1/stats, /v1/stats/benchmark, /v1/stats/permanent, and /v1/stats/history endpoints expose these numbers as JSON. To browse the raw tables without writing SQL, open the file in sqlite-web, a small browser-based SQLite viewer:

pip install sqlite-web
sqlite-web ~/.byoai/byoai_runtime.db   # opens a UI at http://localhost:8080

sqlite-web is an optional dev convenience, not a dependency of byoai-runtime.

Keeping a long-running proxy small

Two things would otherwise grow with uptime, and both are now bounded.

The SQLite log is pruned to the last BYOAI_RETENTION_DAYS (default 90) once on every start. A day of heavy Claude Code use adds a few thousand rows and under a megabyte, so without a window the file reaches a few hundred MB a year and the unfiltered SUM() queries behind /v1/stats/permanent slow down as the table grows. To prune a proxy that has been up for months without restarting it:

byoai-cache prune              # delete old rows, then reclaim the freed space
byoai-cache prune --days 30
byoai-cache prune --no-vacuum  # delete only; safe against a running proxy

Reclaiming space is skipped when fewer than 1,000 rows were deleted — rewriting the whole file costs more than the space it returns. The command says so when it skips.

The delete itself is safe to run against a live proxy: WAL mode keeps in-flight readers unblocked. Reclaiming the freed space is not — VACUUM rewrites the whole file under an exclusive lock, and a concurrent write from the running proxy can fail with "database is locked". Either stop the proxy first, or pass --no-vacuum and let the space be reused by future rows. Startup pruning never vacuums for this reason.

Set BYOAI_RETENTION_DAYS=0 to keep every row.

In-process dedup state is capped on two axes: 500 concurrent sessions and 5,000 content hashes per session, oldest evicted first. Evicting a hash costs at most one re-send of a block nobody has referenced in a long time. Redis, when configured, holds the same state; the in-process copy is kept as a complete mirror so an outage degrades dedup quality instead of breaking requests. Those two caps are what bound it — before them, a single long conversation could accumulate hashes for its entire 8-hour lifetime.

Contributing

ByoAI welcomes AI-assisted development as well as human contributions. See CONTRIBUTING.md for dev setup, checks, the AI-assisted-development policy, and the PR process, and our Code of Conduct. To report a vulnerability, see SECURITY.md rather than opening a public issue.

📄 License & Enterprise Security

byoai-runtime is distributed under the Apache License 2.0.

• Enterprise Safe: Permissive license with explicit patent grants and trademark protections. Pre-approved for enterprise compliance scanners (Snyk, FOSSA, Mend).
• Data Privacy: Runs strictly in-process within your infrastructure. Zero data is transmitted to external servers beyond your configured model and database providers.

🌐 Community & Documentation

• Documentation: ravikings.github.io/byoai-runtime
• GitHub Repository: github.com/ravikings/byoai-runtime
• PyPI Package: pypi.org/project/byoai-runtime
• Changelog: CHANGELOG.md
• Contributing: CONTRIBUTING.md
• Security: SECURITY.md