agentflow-runtime 1.3.0

Real-time data platform serving context to AI agents

AgentFlow

Real-time data platform for AI agents. Live entity lookups, typed contracts, dual-language SDKs, and release-gated delivery.

Why this exists

Most agent demos work until they have to answer from live business state. Support, ops, and merch workflows need current orders, metrics, and health signals while the conversation is happening, not a stale warehouse snapshot and not a pile of one-off service adapters.

AgentFlow turns that problem into one serving boundary:

• streaming ingestion for operational events
• a semantic layer that exposes entities, metrics, and query endpoints
• typed contracts so SDKs and callers know what shape to expect
• Python and TypeScript clients that speak the same API surface

Highlights

• Release-line gate: 752 passed, 4 skipped on 2026-05-04; GitHub environments staging and production have required reviewers. The 2026-04-27 audit closure sprint (Codex p1–p9 + Opus) shipped six commits closing all P0/P1/P2 findings — see docs/audits/2026-04-27/README.md and Release Readiness for the live status
• Sub-second entity lookups in the checked-in baseline: entity p50 38-55 ms, entity p99 290-320 ms, aggregate p50 56 ms at 50 users for 60s
• Historical performance remediation is documented: the serving path moved from an original ~26,000 ms baseline to the current 43-55 ms release range
• Dual SDK parity for Python and TypeScript, including retry policies, circuit breakers, batching, pagination, and contract pinning
• Postgres/MySQL CDC path through Debezium and Kafka Connect, with local compose, Helm manifests, and canonical CDC normalization
• Security hardening in the hot path: parameterized queries, sqlglot AST validation for NL-to-SQL, and a Bandit baseline gate for new findings only
• Release workflow coverage: chaos smoke on PRs, performance regression gate, contract drift checks, and a Terraform apply workflow with OIDC-ready auth

Quick start

Upgrading from v1.0.x? See the v1.1 migration guide before installing.

Prerequisites:

• Python 3.11+
• make
• Docker Compose (make demo starts Redis)

PowerShell 7+:

git clone https://github.com/brownjuly2003-code/agentflow.git
cd agentflow
. .\scripts\setup.ps1
make demo

macOS / Linux:

git clone https://github.com/brownjuly2003-code/agentflow.git
cd agentflow
source ./scripts/setup.sh
make demo

make demo seeds local data, starts Redis, and serves the API on http://localhost:8000. Swagger UI is available at http://localhost:8000/docs.

Try it:

curl http://localhost:8000/v1/entity/order/ORD-20260404-1001

curl -X POST http://localhost:8000/v1/query \
  -H "Content-Type: application/json" \
  -d '{"question":"Show me top 3 products"}'

Local demo runs without API-key enforcement unless you explicitly configure AGENTFLOW_API_KEYS_FILE.

Architecture

Event sources -> Kafka -> Flink -> Iceberg ----\
                                                -> Semantic layer -> FastAPI -> Agent / SDK
Local demo   -> local_pipeline -> DuckDB ------/

Stack:

• Ingestion: Kafka producers, Debezium/Kafka Connect CDC, and a local synthetic pipeline
• Processing: Flink plus validation and enrichment stages
• Storage: Iceberg for production-shaped tables, DuckDB for the local serving path
• Serving: FastAPI, contract registry, lineage, search, and operational endpoints
• Orchestration: Dagster
• IaC: Terraform, Helm, Docker Compose, and a Fly.io demo config

See docs/architecture.md for the detailed design, trade-offs, and deployment topologies.

CDC source capture is standardized on Debezium/Kafka Connect; downstream consumers use the canonical AgentFlow CDC contract defined in ADR 0005.

What's inside

Area	Files
API core	src/serving/api/
Semantic layer	src/serving/semantic_layer/
Python SDK	sdk/agentflow/
TypeScript SDK	sdk-ts/src/
Agent integrations	integrations/agentflow_integrations/ (LangChain, LlamaIndex, CrewAI, MCP)
Flink jobs	src/processing/flink_jobs/
Test suites	tests/
Planning trail	docs/plans/
Public site	site/
IaC	infrastructure/terraform/, infrastructure/dv2/, helm/, k8s/
DV2.0 warehouse	warehouse/agentflow/dv2/ (hubs / links / satellites + X5 loader)

Documentation

• Interactive Technical Walkthrough - MkDocs Material guide with Mermaid architecture, API, SDK, deployment, observability, and troubleshooting pages
• Architecture - system context, data flow, failure modes
• Operational Runbook - local stack, CDC capture, incident response, and maintenance commands
• API Reference - endpoint-by-endpoint examples for curl, Python, and TypeScript
• Security Audit - threat model, controls, and evidence
• Competitive Analysis - positioning and trade-offs
• CDC Deployment Plan - Debezium/Kafka Connect rollout decisions and implementation trail
• Glossary - interview-ready explanations of the core technical terms
• Release Readiness - checked release evidence through v1.1.0 and current external-gate handoffs
• Customer Discovery Tracker - synthetic/modelled PMF planning surface and real-evidence gap
• Pricing Validation Plan - pricing/WTP questions, evidence gates, and current real-evidence gap
• AWS OIDC Setup - Terraform apply readiness handoff
• Immutable Retention Evidence Handoff - external audit-retention evidence checklist
• Production CDC Source Onboarding - production CDC decision record and no-go gates
• External Gate Evidence Intake Checklist - required owner evidence before blocked external gates can close
• Public Production-Hardware Benchmark Plan - benchmark evidence plan for c8g.4xlarge+
• External Pen-Test Attestation Handoff - third-party pen-test evidence checklist
• Audit History - baseline-to-release remediation trail
• Publication Checklist - final GitHub publishing checklist
• Fly.io Demo Deploy - minimal hosted demo instructions
• DV2.0 Multi-Branch Extension - Data Vault 2.0 model for mid-market e-com with 5 locations / 3 jurisdictions; schema, end-to-end flow, live demo evidence, k8s manifests
• Contributing - development and PR expectations
• Changelog - project release notes

Development

# verified release slice
python -m pytest tests/unit tests/integration tests/sdk -q

# benchmark and regression gate
python scripts/run_benchmark.py
python scripts/check_performance.py --baseline docs/benchmark-baseline.json --current .artifacts/load/results.json --max-regress 20

# benchmark trend: [.github/perf-history.json](.github/perf-history.json) is appended on every main push;
# render the history locally with `make perf-plot` (writes docs/perf/history.html).

# contracts and security
python scripts/generate_contracts.py --check
bandit -r src sdk --ini .bandit --severity-level medium -f json -o .tmp/bandit-current.json
python scripts/bandit_diff.py .bandit-baseline.json .tmp/bandit-current.json

Status

v1.1.0 is published to PyPI, npm, and GitHub. The 2026-04-27 audit closure sprint landed six commits on main that close all P0/P1/P2 findings from the Claude Opus + Codex p1–p9 audits: tenant isolation across the control plane, SQL guard centralization, entity allowlist enforcement on every read surface, secrets scrubbed and rotated, helm runAsNonRoot / NetworkPolicy / PodDisruptionBudget, npm lockfile + npm audit clean, vulnerable dep bumps (dagster>=1.13.1, langchain-core>=1.2.22), trivy pinned, OpenAPI drift gate, branch protection with 12 required status checks, GitHub Actions environment reviewers, and Python SDK alignment with the server v1 contract (F1–F10). Recent local full-suite verification: 752 passed, 4 skipped on 2026-05-04 after clarifying the external-gate handoff. The post-v1.1 CDC operationalization for Debezium / Kafka Connect is checked in, while production source onboarding remains pending; see docs/release-readiness.md. Remaining external gates are AWS OIDC role setup for real Terraform apply, external immutable audit retention if claimed beyond local hash-chain evidence, production CDC source onboarding, real PMF/pricing evidence, public benchmark publication on production hardware, external pen-test attestation, and legacy npm NPM_TOKEN revocation after a successful new-package trusted-publish run. npm Trusted Publishing readback for the new package is complete. A project-local Pi skill for evidence intake lives at .pi/skills/external-gate-evidence-intake.

Screenshots

Admin UI	API docs

Landing page	Benchmark run

Capture notes and publish-time checks are listed in docs/publication-checklist.md.

License

MIT. See LICENSE.

Credits

Built as a data-engineering reference project during the 2026-04-10 -> 2026-04-20 release cycle, with the full implementation trail preserved in docs/plans/.