zeptodb 0.1.2

Python client and DataFrame integration for ZeptoDB — ultra-low latency columnar time-series database

⚡ ZeptoDB

Agent Memory for Live Time-Series Data

Ingest millions of events per second. Retrieve the evidence, context, and cache entries AI agents need before they act.

Website · Agent Memory · Quick Start · Benchmarks · Docs · Community

---

What is ZeptoDB?

ZeptoDB is an in-memory columnar database purpose-built for live time-series analytics and AI agent memory.

It handles high-throughput ingestion and real-time analytical queries simultaneously — without trade-offs between the two.

The core engine is hardware-software co-optimized: Highway SIMD vectorization, LLVM JIT compilation, lock-free ring buffers, NUMA-aware allocation, and UCX/RDMA networking — all working together to eliminate unnecessary copies, allocations, and cache misses. The AI memory layer adds agent-scoped context for applications built on live time-series data: client-supplied embedding storage, parallel filtered context retrieval, optional experimental ANN candidate generation, token-budget assembly, and exact/semantic prompt cache lookup without calling LLM providers from the server. Agent examples live under examples/agent_memory and include provider cache, LangGraph-style memory, optional provider adapters, AgentOps telemetry, and agent-attached time-series demos for finance, IoT, observability, robotics, and game/live-ops.

New to the project? Start with the website:

Path	What it gives you
Agent Memory	How live time-series becomes agent context, cache, and replay
Python Agent Memory Quickstart	Copy-paste memory, context, cache, and write-back flow
Benchmarks	Ingestion, query latency, zero-copy Python, and Agent Memory numbers
Features	Time-series core, memory layer, APIs, storage, and security
Quick Start	First local query and client setup

┌─────────────────────────────────────────────────────────────┐
│  Clients: HTTP API · Python DSL · C++ API · Arrow Flight    │
├─────────────────────────────────────────────────────────────┤
│  AI Memory: vertical context · agent telemetry · cache      │
├─────────────────────────────────────────────────────────────┤
│  SQL Engine: Parser (1.5μs) · AST Optimizer · Executor      │
├─────────────────────────────────────────────────────────────┤
│  Execution: Highway SIMD · LLVM JIT · Partition-parallel    │
│  ASOF JOIN · Window JOIN · xbar · EMA · VWAP                │
├─────────────────────────────────────────────────────────────┤
│  Ingestion: Lock-free MPMC Ring Buffer · WAL · Feed Handlers│
├─────────────────────────────────────────────────────────────┤
│  Storage: Arena Allocator · Column Store · Tiered (→S3)     │
├─────────────────────────────────────────────────────────────┤
│  Cluster: Consistent Hashing · RF=2 · Auto Failover         │
├─────────────────────────────────────────────────────────────┤
│  Security: TLS · JWT/OIDC · RBAC · Audit (SOC2/MiFID II)   │
└─────────────────────────────────────────────────────────────┘

---

🔍 Why ZeptoDB?

	ZeptoDB	kdb+	ClickHouse	TimescaleDB	QuestDB
Ingestion	5.52M evt/s	~5M evt/s	~100K evt/s	~50K evt/s	~1M evt/s
Filter 1M rows	272μs	~300μs	~10ms	~50ms	~2ms
ASOF JOIN	✅ Native	✅ Native	❌ UDF	❌ Manual	✅ Native
SQL	✅ Standard	❌ q lang	✅ Dialect	✅ PostgreSQL	✅ Dialect
Python zero-copy	522ns	❌ IPC only	❌	❌	❌
License cost	Free (BUSL-1.1)	$100K–500K/yr	Free (Apache 2.0)	Free (Apache 2.0)	Free (Apache 2.0)

TL;DR: kdb+ class time-series performance, standard SQL, zero-copy Python, and a native Agent Memory layer for context retrieval, prompt cache, and AgentOps telemetry.

---

🚀 Quick Start

Method	Command
Binary	Download from GitHub Releases
Homebrew	brew install ZeptoDB/tap/zeptodb
Docker	docker run -p 8123:8123 zeptodb/zeptodb:0.0.1
PyPI	pip install zeptodb
Source	Build instructions below

Binary

# amd64
curl -LO https://github.com/ZeptoDB/ZeptoDB/releases/latest/download/zeptodb-linux-amd64-0.0.1.tar.gz
tar xzf zeptodb-linux-amd64-0.0.1.tar.gz
./zeptodb-linux-amd64-0.0.1/zepto_http_server --port 8123

# arm64 (AWS Graviton)
curl -LO https://github.com/ZeptoDB/ZeptoDB/releases/latest/download/zeptodb-linux-arm64-0.0.1.tar.gz

Note: Prebuilt binaries require runtime libraries (LLVM 19, Arrow, etc.). See the Binary Installation Guide for prerequisites and troubleshooting.

Docker

docker run -p 8123:8123 zeptodb/zeptodb:0.0.1

# Insert data
curl -X POST http://localhost:8123/ \
  -d "INSERT INTO trades VALUES (1, 1714000000000000000, 185.50, 100)"

# Query
curl -X POST http://localhost:8123/ \
  -d "SELECT vwap(price, volume), count(*) FROM trades WHERE symbol = 'AAPL'"

Python

import zeptodb

db = zeptodb.Pipeline()
db.start()
db.ingest(symbol=1, price=185.50, volume=100)
db.drain()

# Zero-copy numpy access (522ns)
prices = db.get_column(symbol=1, name="price")

Build from Source

# Dependencies (Amazon Linux 2023 / Fedora)
sudo dnf install -y clang19 clang19-devel llvm19-devel \
  highway-devel numactl-devel ucx-devel ninja-build lz4-devel

mkdir -p build && cd build
cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=clang-19 -DCMAKE_CXX_COMPILER=clang++-19
ninja -j$(nproc)

./zepto_http_server --port 8123

📖 Full guide: Quick Start · Agent Memory · Python Reference · SQL Reference

---

📊 Performance

Single node. End-to-end latencies including SQL parsing. No cherry-picking.

Operation	Latency	Notes
Ingestion throughput	5.52M events/sec	Lock-free MPMC ring buffer
Filter 1M rows	272μs	Highway SIMD vectorized scan
VWAP 1M rows	532μs	Fused price×volume aggregation
GROUP BY (8 threads)	248μs	Partition-parallel scatter/gather
EMA 1M rows	2.2ms	Streaming exponential moving average
Window SUM 1M rows	1.36ms	Prefix-sum O(n) algorithm
xbar (1M → 3,334 bars)	11ms	Time-bucketed OHLCV
SQL parse	1.5–4.5μs	Recursive descent, zero allocation
Python column access	522ns	Zero-copy shared memory
Indexed lookup (g#/p#)	3.3μs	274× faster than full scan
HDB flush to disk	4.8 GB/s	LZ4 compressed
Partition routing	2ns	Consistent hash ring

---

💡 SQL Examples

-- 5-minute OHLCV candlestick bars
SELECT xbar(timestamp, 300000000000) AS bar,
       first(price) AS open, max(price) AS high,
       min(price) AS low, last(price) AS close,
       sum(volume) AS volume
FROM trades WHERE symbol = 'AAPL'
GROUP BY xbar(timestamp, 300000000000)

-- ASOF JOIN (point-in-time lookup)
SELECT t.price, q.bid, q.ask
FROM trades t
ASOF JOIN quotes q
ON t.symbol = q.symbol AND t.timestamp >= q.timestamp

-- EMA with delta
SELECT symbol, price,
       EMA(price, 20) OVER (PARTITION BY symbol ORDER BY timestamp) AS ema20,
       DELTA(price) OVER (ORDER BY timestamp) AS price_change
FROM trades

-- Window JOIN (time-range aggregation)
SELECT t.price, wj_avg(q.bid) AS avg_bid
FROM trades t
WINDOW JOIN quotes q ON t.symbol = q.symbol
AND q.timestamp BETWEEN t.timestamp - 5000000000 AND t.timestamp + 5000000000

-- Materialized view (incremental, updated on ingest)
CREATE MATERIALIZED VIEW ohlcv_5min AS
  SELECT symbol, xbar(timestamp, 300000000000) AS bar,
         first(price) AS open, max(price) AS high,
         min(price) AS low, last(price) AS close,
         sum(volume) AS vol
  FROM trades
  GROUP BY symbol, xbar(timestamp, 300000000000)

-- Storage tiering
ALTER TABLE trades SET STORAGE POLICY
  HOT 1 HOURS WARM 24 HOURS COLD 30 DAYS DROP 365 DAYS

Full SQL reference: SQL_REFERENCE.md — INSERT, UPDATE, DELETE, CASE WHEN, LIKE, UNION, CTE, subqueries, and more.

---

🏗️ Use Cases

Domain	Why ZeptoDB	Key Features
Finance / HFT	Sub-ms tick processing, kdb+-class perf	ASOF JOIN, xbar, EMA, VWAP
Quant Research	Backtest in Python, execute in C++	Zero-copy numpy, Polars DSL
Agentic AI	Add agents to time-series workflows without losing operational context	Agent-scoped memory, context assembly, exact/semantic cache
Crypto / DeFi	24/7 multi-exchange streaming	Binance feed handler, real-time agg
IoT / Manufacturing	High-frequency sensor ingestion and OPC-UA integration	OPC-UA client/server, Telegraf output, Historical Access, DELTA/RATIO, time-bar agg, LZ4
Physical AI / Robotics	Typed ROS 2 telemetry, rosbag2 replay, feature store	ROS 2 roadmap, ROS 2 setup, edge guide, use cases, spatial SQL
Autonomous Vehicles	Sensor fusion, driving log replay	ASOF JOIN, Parquet HDB, parallel scan
Observability	High-cardinality metrics	SQL + Grafana, TTL + S3 tiering

---

⚙️ Optimization Stack

Hardware	Software
Highway SIMD — 256/512-bit vectorized scans NUMA-aware — memory pinned to local node UCX / RDMA — kernel-bypass networking Arena allocator — zero GC, zero fragmentation	LLVM JIT — runtime expression compilation Lock-free MPMC — zero-contention ingestion Columnar storage — cache-friendly sequential access Partition-parallel — 3.48× scaling at 8 threads

---

🔒 Enterprise Security

Feature	Details
TLS/HTTPS	OpenSSL 3.2, cert/key PEM
Authentication	API Key (SHA256) + JWT/OIDC (HS256/RS256, JWKS auto-fetch)
Authorization	RBAC: 5 roles + symbol-level ACL + multi-tenancy
Rate Limiting	Token bucket per-identity + per-IP
Secrets	Vault KV v2 → K8s secrets → env var (priority chain)
Audit Log	7-year retention, SOC2/EMIR/MiFID II compliant

---

🚢 Deployment

# Docker
docker run -p 8123:8123 zeptodb/zeptodb

# Helm
helm install zeptodb ./deploy/helm/zeptodb

# Bare-metal (systemd)
./deploy/scripts/install_service.sh

Guides: Production Deployment · Kubernetes · Bare-metal Tuning

---

🔄 Migration

Migrate from existing databases with built-in tooling:

./zepto-migrate --source kdb+ --hdb-path /data/hdb --target localhost:8123

Supported: kdb+ (HDB loader, q→SQL) · ClickHouse (DDL/query conversion) · DuckDB (Parquet) · TimescaleDB (hypertable conversion)

---

💬 Community

• Website — docs, benchmarks, comparisons, and Agent Memory guides
• Discord — questions, discussions, real-time help
• GitHub Discussions — design proposals, Q&A, ideas
• Twitter/X — release announcements, benchmarks

---

🛠️ Built with Kiro

ZeptoDB is developed using Kiro (AI coding assistant, IDE + CLI) on top of a large existing C++20 codebase. Three Kiro features do most of the heavy lifting:

1. Multi-agent orchestration — an orchestrator (zeptodb-dev) delegates to zepto-developer, zepto-reviewer, and zepto-qa subagents. Reviewer is tool-restricted to read-only; QA runs builds on x86_64 and aarch64 in parallel with review. See .kiro/agents/.
2. MCP + Skills — six keyword-activated SKILL.md knowledge packs (design-doc index, layer patterns, edge-case catalog, doc-update checklist, review checklist, cross-arch verification) plus a Playwright MCP server for Web UI smoke tests. See .kiro/skills/ and .kiro/settings/mcp.json.
3. Persistent context — .kiro/KIRO.md (project rulebook) + per-agent .kiro/context/*.md are auto-loaded on every session via .kiro/settings.json. The repo itself becomes the prompt — no re-pasting rules, no doc-code drift.

Full case study, workflow diagrams, and learnings: docs/KIRO_USAGE.md.

---

🤝 Contributing

We welcome contributions of all sizes — from typo fixes to new features.

• 🏷️ Check out issues labeled good-first-issue for easy starting points
• 📖 Read CONTRIBUTING.md for build instructions and guidelines
• 💬 Join Discord to discuss ideas before starting large changes

---

If ZeptoDB is useful to you, consider giving it a ⭐ — it helps others discover the project.

---

📄 License

Business Source License 1.1 (BUSL-1.1) — You can use, modify, and distribute ZeptoDB freely, including in production. The only restriction: you cannot offer it as a commercial database-as-a-service (DBaaS).

• Change Date: 2030-04-01 — on this date, the license automatically converts to Apache 2.0
• SPDX: BUSL-1.1

For commercial licensing inquiries: [e-mail]