zeptodb 0.0.3

Python client and DataFrame integration for ZeptoDB — ultra-low latency columnar time-series database

⚡ ZeptoDB

In-Memory Time-Series Database for High-Throughput Workloads

Ingest millions of events per second. Analyze them in microseconds.

Quick Start · Performance · SQL Examples · Docs · Contributing

---

What is ZeptoDB?

ZeptoDB is an in-memory columnar database purpose-built for time-series analytics at scale.

It handles high-throughput ingestion and real-time analytical queries simultaneously — without trade-offs between the two.

The engine is hardware-software co-optimized: Highway SIMD vectorization, LLVM JIT compilation, lock-free ring buffers, NUMA-aware allocation, and UCX/RDMA networking — all working together to eliminate unnecessary copies, allocations, and cache misses.

┌─────────────────────────────────────────────────────────────┐
│  Clients: HTTP API · Python DSL · C++ API · Arrow Flight    │
├─────────────────────────────────────────────────────────────┤
│  SQL Engine: Parser (1.5μs) · AST Optimizer · Executor      │
├─────────────────────────────────────────────────────────────┤
│  Execution: Highway SIMD · LLVM JIT · Partition-parallel    │
│  ASOF JOIN · Window JOIN · xbar · EMA · VWAP                │
├─────────────────────────────────────────────────────────────┤
│  Ingestion: Lock-free MPMC Ring Buffer · WAL · Feed Handlers│
├─────────────────────────────────────────────────────────────┤
│  Storage: Arena Allocator · Column Store · Tiered (→S3)     │
├─────────────────────────────────────────────────────────────┤
│  Cluster: Consistent Hashing · RF=2 · Auto Failover         │
├─────────────────────────────────────────────────────────────┤
│  Security: TLS · JWT/OIDC · RBAC · Audit (SOC2/MiFID II)   │
└─────────────────────────────────────────────────────────────┘

---

🚀 Quick Start

Docker (fastest)

docker run -p 8123:8123 zeptodb/zeptodb:0.0.1

# Insert data
curl -X POST http://localhost:8123/ \
  -d "INSERT INTO trades VALUES (1, 1714000000000000000, 185.50, 100)"

# Query
curl -X POST http://localhost:8123/ \
  -d "SELECT vwap(price, volume), count(*) FROM trades WHERE symbol = 'AAPL'"

Build from Source

# Dependencies (Amazon Linux 2023 / Fedora)
sudo dnf install -y clang19 clang19-devel llvm19-devel \
  highway-devel numactl-devel ucx-devel ninja-build lz4-devel

mkdir -p build && cd build
cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=clang-19 -DCMAKE_CXX_COMPILER=clang++-19
ninja -j$(nproc)

./zepto_http_server --port 8123

Python

import zeptodb

db = zeptodb.Pipeline()
db.start()
db.ingest(symbol=1, price=185.50, volume=100)
db.drain()

# Zero-copy numpy access (522ns)
prices = db.get_column(symbol=1, name="price")

📖 Full guide: Quick Start · Python Reference · SQL Reference

---

📊 Performance

Single node. End-to-end latencies including SQL parsing. No cherry-picking.

Operation	Latency	Notes
Ingestion throughput	5.52M events/sec	Lock-free MPMC ring buffer
Filter 1M rows	272μs	Highway SIMD vectorized scan
VWAP 1M rows	532μs	Fused price×volume aggregation
GROUP BY (8 threads)	248μs	Partition-parallel scatter/gather
EMA 1M rows	2.2ms	Streaming exponential moving average
Window SUM 1M rows	1.36ms	Prefix-sum O(n) algorithm
xbar (1M → 3,334 bars)	11ms	Time-bucketed OHLCV
SQL parse	1.5–4.5μs	Recursive descent, zero allocation
Python column access	522ns	Zero-copy shared memory
Indexed lookup (g#/p#)	3.3μs	274× faster than full scan
HDB flush to disk	4.8 GB/s	LZ4 compressed
Partition routing	2ns	Consistent hash ring

---

💡 SQL Examples

-- 5-minute OHLCV candlestick bars
SELECT xbar(timestamp, 300000000000) AS bar,
       first(price) AS open, max(price) AS high,
       min(price) AS low, last(price) AS close,
       sum(volume) AS volume
FROM trades WHERE symbol = 'AAPL'
GROUP BY xbar(timestamp, 300000000000)

-- ASOF JOIN (point-in-time lookup)
SELECT t.price, q.bid, q.ask
FROM trades t
ASOF JOIN quotes q
ON t.symbol = q.symbol AND t.timestamp >= q.timestamp

-- EMA with delta
SELECT symbol, price,
       EMA(price, 20) OVER (PARTITION BY symbol ORDER BY timestamp) AS ema20,
       DELTA(price) OVER (ORDER BY timestamp) AS price_change
FROM trades

-- Window JOIN (time-range aggregation)
SELECT t.price, wj_avg(q.bid) AS avg_bid
FROM trades t
WINDOW JOIN quotes q ON t.symbol = q.symbol
AND q.timestamp BETWEEN t.timestamp - 5000000000 AND t.timestamp + 5000000000

-- Materialized view (incremental, updated on ingest)
CREATE MATERIALIZED VIEW ohlcv_5min AS
  SELECT symbol, xbar(timestamp, 300000000000) AS bar,
         first(price) AS open, max(price) AS high,
         min(price) AS low, last(price) AS close,
         sum(volume) AS vol
  FROM trades
  GROUP BY symbol, xbar(timestamp, 300000000000)

-- Storage tiering
ALTER TABLE trades SET STORAGE POLICY
  HOT 1 HOURS WARM 24 HOURS COLD 30 DAYS DROP 365 DAYS

Full SQL reference: SQL_REFERENCE.md — INSERT, UPDATE, DELETE, CASE WHEN, LIKE, UNION, CTE, subqueries, and more.

---

🏗️ Use Cases

Domain	Why ZeptoDB	Key Features
Finance / HFT	Sub-ms tick processing, kdb+-class perf	ASOF JOIN, xbar, EMA, VWAP
Quant Research	Backtest in Python, execute in C++	Zero-copy numpy, Polars DSL
Crypto / DeFi	24/7 multi-exchange streaming	Binance feed handler, real-time agg
IoT / Manufacturing	High-frequency sensor ingestion	DELTA/RATIO, time-bar agg, LZ4
Autonomous Vehicles	Sensor fusion, driving log replay	ASOF JOIN, Parquet HDB, parallel scan
Observability	High-cardinality metrics	SQL + Grafana, TTL + S3 tiering

---

⚙️ Optimization Stack

Hardware	Software
Highway SIMD — 256/512-bit vectorized scans NUMA-aware — memory pinned to local node UCX / RDMA — kernel-bypass networking Arena allocator — zero GC, zero fragmentation	LLVM JIT — runtime expression compilation Lock-free MPMC — zero-contention ingestion Columnar storage — cache-friendly sequential access Partition-parallel — 3.48× scaling at 8 threads

---

🔒 Enterprise Security

Feature	Details
TLS/HTTPS	OpenSSL 3.2, cert/key PEM
Authentication	API Key (SHA256) + JWT/OIDC (HS256/RS256, JWKS auto-fetch)
Authorization	RBAC: 5 roles + symbol-level ACL + multi-tenancy
Rate Limiting	Token bucket per-identity + per-IP
Secrets	Vault KV v2 → K8s secrets → env var (priority chain)
Audit Log	7-year retention, SOC2/EMIR/MiFID II compliant

---

🚢 Deployment

# Docker
docker run -p 8123:8123 zeptodb/zeptodb

# Helm
helm install zeptodb ./deploy/helm/zeptodb

# Bare-metal (systemd)
./deploy/scripts/install_service.sh

Guides: Production Deployment · Kubernetes · Bare-metal Tuning

---

🔄 Migration

Migrate from existing databases with built-in tooling:

./zepto-migrate --source kdb+ --hdb-path /data/hdb --target localhost:8123

Supported: kdb+ (HDB loader, q→SQL) · ClickHouse (DDL/query conversion) · DuckDB (Parquet) · TimescaleDB (hypertable conversion)

---

🤝 Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

---

📄 License

Business Source License 1.1 — Production use permitted, except offering as a commercial DBaaS. Changes to Apache 2.0 on 2030-04-01.

For commercial licensing: [e-mail]