Skip to main content

AgentFlow

Event-native metrics layer: business metrics that move when events happen — measured 1.1 s p50 event-to-metric freshness on production defaults. Live entity lookups, typed contracts, dual-language SDKs, and release-gated delivery for people, dashboards, services, and AI agents alike.

Release gate codecov Python License

Why this exists

BI on a replica answers yesterday's questions. Support, ops, and merch workflows need current orders, metrics, and health signals at the moment of decision — not a stale warehouse snapshot, not a pile of one-off service adapters, and not a cache that quietly serves 30-second-old numbers.

AgentFlow's axis is event → live metric: every metric declares which events move it (a contract-tested lineage graph), and the serving layer keeps reads fresh by invalidating its cache when events arrive — a measured behavior, not a slogan (docs/freshness-benchmark.md). One serving boundary on top of that axis:

  • streaming ingestion for operational events (validated, enriched, journaled)
  • a semantic layer that exposes entities, metrics, lineage, and query endpoints
  • typed, versioned contracts — each metric ships with its source events and a staleness budget
  • Python and TypeScript clients that speak the same API surface

Consumers are whoever needs the number now: humans, dashboards, downstream services, and AI agents — agents are one consumer, not the product.

Highlights

  • Measured event-to-metric freshness — an event entering the pipeline is reflected in GET /v1/metrics/* in 1.06 s p50 / 1.99 s p95 on production defaults (event-driven cache invalidation, no webhook registration), tunable to 238 ms p50; a plain TTL cache on the same pipeline sits at ~15 s. Reproducible via python scripts/benchmark_freshness.py → freshness benchmark
  • Lineage as a contract — all six metrics declare their source events, serving table, and a 2.5 s p95 staleness budget in versioned contracts, exposed through /v1/catalog and /v1/contracts and pinned by tests against the actual write path
  • Published release line through v1.6.0 on PyPI (agentflow-runtime, agentflow-client) and npm (@yuliaedomskikh/agentflow-client) via OIDC Trusted Publishers with SLSA provenance on every artifact
  • Tested and gated — 1,500+ unit tests plus a broad Windows no-Docker suite; CI enforces 13 required status checks (lint, schema, unit, integration, helm, perf, terraform, bandit, safety, npm-audit, trivy, contract, build-smoke) through branch protection
  • Dual SDK parity across Python and TypeScript — retries, circuit breakers, batching, pagination, contract pinning, idempotency keys, as_of historical reads — over sub-second entity lookups (p50 38–55 ms, p99 167 ms on local hardware)
  • Security in the hot path — tenant isolation on every read surface, parameterized queries, sqlglot AST validation for NL-to-SQL, fail-closed auth, secret scrubbing, and a Bandit gate for new findings
  • Production-shaped extras — two CDC paths (hardened Debezium/Kafka Connect + a ClickHouse per-branch fan-out), on-call runbooks, and a narrated demo of the DV2 multi-branch warehouse

Quick start

Upgrading from v1.0.x? See the v1.1 migration guide before installing.

Prerequisites:

  • Python 3.11+
  • make
  • Docker Compose (make demo starts Redis and the ClickHouse serving store)

PowerShell 7+:

git clone https://github.com/brownjuly2003-code/agentflow.git
cd agentflow
. .\scripts\setup.ps1
make demo

macOS / Linux:

git clone https://github.com/brownjuly2003-code/agentflow.git
cd agentflow
source ./scripts/setup.sh
make demo

make demo starts Redis and ClickHouse, seeds demo data through the full pipeline (validated events land in the ClickHouse serving store), and serves the API on http://localhost:8000. Swagger UI is available at http://localhost:8000/docs.

Try it:

curl http://localhost:8000/v1/entity/order/ORD-20260404-1001

curl -X POST http://localhost:8000/v1/query \
  -H "Content-Type: application/json" \
  -d '{"question":"Show me top 3 products"}'

Local demo runs without API-key enforcement unless you explicitly configure AGENTFLOW_API_KEYS_FILE.

Architecture

Event sources -> Kafka -> Flink -> Iceberg --------\
                                                    -> Semantic layer -> FastAPI -> Agent / SDK
Local demo   -> local_pipeline -> ClickHouse ------/
                       (DuckDB stays the local lake / test store)

Stack:

  • Ingestion: Kafka producers, Debezium/Kafka Connect CDC, and a local synthetic pipeline
  • Processing: Flink plus validation and enrichment stages
  • Storage: Iceberg for production-shaped tables; ClickHouse is the serving store (ADR 0006 — ReplacingMergeTree upserts, final=1 reads), DuckDB the local-dev / test store
  • Serving: FastAPI, contract registry, lineage, search, and operational endpoints
  • Orchestration: Dagster
  • IaC: Terraform, Helm, Docker Compose, and a Fly.io demo config

See docs/architecture.md for the detailed design, trade-offs, and deployment topologies.

CDC source capture is standardized on Debezium/Kafka Connect; downstream consumers use the canonical AgentFlow CDC contract defined in ADR 0005.

What's inside

Area Files
API core src/serving/api/
Semantic layer src/serving/semantic_layer/
Python SDK sdk/agentflow/
TypeScript SDK sdk-ts/src/
Agent integrations integrations/agentflow_integrations/ (LangChain, LlamaIndex, CrewAI, MCP)
Flink jobs src/processing/flink_jobs/
Test suites tests/
Design decisions docs/decisions/ (ADRs)
Public site site/
IaC infrastructure/terraform/, infrastructure/dv2/, helm/, k8s/
DV2.0 warehouse warehouse/agentflow/dv2/ (hubs / links / satellites + real-dataset loader)

Documentation

Core

Deep dives

Development

# verified release slice
python -m pytest tests/unit tests/integration tests/sdk -q

# benchmark and regression gate
python scripts/run_benchmark.py
python scripts/check_performance.py --baseline docs/benchmark-baseline.json --current .artifacts/load/results.json --max-regress 20

# benchmark trend: [.github/perf-history.json](.github/perf-history.json) is appended on every main push;
# render the history locally with `make perf-plot` (writes docs/perf/history.html).

# contracts and security
python scripts/generate_contracts.py --check
bandit -r src sdk --ini .bandit --severity-level medium -f json -o .tmp/bandit-current.json
python scripts/bandit_diff.py .bandit-baseline.json .tmp/bandit-current.json

Status

v1.6.0 is the current release line — PyPI agentflow-runtime / agentflow-client and npm @yuliaedomskikh/agentflow-client, all published via OIDC Trusted Publishers with SLSA provenance attestations. CI on main is green across all 13 required checks.

The v1.1.0 → v1.6.0 arc landed in six increments on top of a security audit-closure sprint:

  • v1.1.0 — audit closure: tenant isolation across every read surface, SQL guard centralized on sqlglot, entity allowlist enforcement, fail-closed auth, secret rotation, Helm hardening, OpenAPI drift gate, and the required status checks.
  • v1.2.0 — DV2 multi-branch warehouse: 38 Data Vault 2.0 tables (8 hubs / 8 links / 22+ satellites), an Argo Workflows dv2-refresh template, a dbt project (3 mart models + 12 tests), and per-branch CDC fan-out via ClickHouse MaterializedPostgreSQL.
  • v1.3.0 — helm/kafka-connect hardening matched to helm/agentflow (NetworkPolicy + PDB + securityContext), live Helm validation across both charts, and the narrated DV2 demo (terminal + web-UI + dbt docs).
  • v1.4.0 — maintenance: on-call runbooks, SECURITY.md, issue/PR templates, contract/DORA CI hardening, repo hygiene, and a dependency wave (mypy, Terraform AWS provider, TypeScript, GitHub Actions, Vitest). No runtime API changes from v1.3.0.
  • v1.5.0 — security & correctness hardening: argon2id key hashing with an O(1) peppered lookup index (M-C4), an NL→SQL guard bypass fix (typed read_csv / read_parquet scan functions now denied in projection position), sqlglot control-byte and mutation-target repairs, and a strict-mypy expansion across the orchestration and freshness slices. No public API changes.
  • v1.6.0 — the architecture-fixing release: ClickHouse becomes the shipped serving engine (pipeline sink, ReplacingMergeTree row versions, backend-routed event scan, a dedicated CI E2E lane against a real ClickHouse), PII protection moves from the removed app-level string-parse gate to engine-enforced vault governance (fail-closed column grants, per-jurisdiction officer roles, row policies, SQL SECURITY DEFINER views — every live adversarial probe green), plus the vendored NL→SQL generation engine (LangGraph, routed through GraceKelly), the DV2 raw vault on PostgreSQL with LISTEN/NOTIFY freshness, the MinIO-backed PyIceberg catalog, and the OpenSSF Scorecard channel (5.8 → 7.0).

The tagged line and main are in sync as of v1.6.0. See the changelog for full detail.

Scope

This is a reference data-engineering project. The streaming, warehouse, and deployment artifacts (Flink, Iceberg, Helm, Terraform, k8s) are exercised against a local pipeline and a kind cluster in CI rather than a managed cloud. Wiring it to a live production source needs inputs that live outside the repo — CDC source onboarding (runbook ready in docs/operations/cdc-production-onboarding.md), a public benchmark on production-grade hardware, and an external pen-test attestation.

Screenshots

Admin UI API docs
AgentFlow admin UI AgentFlow API docs
Landing page Benchmark run
AgentFlow landing page AgentFlow benchmark terminal

Capture notes and publish-time checks are listed in docs/publication-checklist.md.

License

MIT. See LICENSE.

Credits

Built as a data-engineering reference project. Initial release cycle 2026-04-10 → 2026-04-20, with post-audit hardening and the DV2 extension landing through v1.4.0. Architecture decisions are recorded as ADRs in docs/decisions/.

Metadata

Release files for agentflow-runtime 2.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for agentflow-runtime 2.0.0
File Size Uploaded Provenance Actions
agentflow_runtime-2.0.0.tar.gz 262.7 kB With provenance Details for agentflow_runtime-2.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for agentflow-runtime 2.0.0
File Interpreter ABI Platform Actions
agentflow_runtime-2.0.0-py3-none-any.whl Python 3 none any Details for agentflow_runtime-2.0.0-py3-none-any.whl

Total release size: 537.9 kB

Release files / agentflow_runtime-2.0.0.tar.gz

Download URL agentflow_runtime-2.0.0.tar.gz
Size 262.7 kB
Tags Source
SHA-256 checksum
How to use checksums
aa3edb519ac4a0f7a516f1717c7e43f315929711e2bb37abd6d854cae4c95c8e
BLAKE2b-256 checksum
How to use checksums
f27fed28149ea1fc08d6d740631d252b6c7a81b1cd9f40ddb046188c2614e7c9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.13

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jul 6, 2026.

Transparency log

Release files / agentflow_runtime-2.0.0-py3-none-any.whl

Download URL agentflow_runtime-2.0.0-py3-none-any.whl
Size 275.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
5033147788046ad2dcf829d96c7964f4d0914204273fd47666cfe3c5618b7d86
BLAKE2b-256 checksum
How to use checksums
45d2ec10b8a41b651e62b0e45d671e01d7122c66183b155e9667285ebc026b08
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.13

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jul 6, 2026.

Transparency log

Security for agentflow-runtime 2.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for agentflow_runtime-2.0.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jul 6, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page