Wormhole

A semantic layer runtime written in Zig. Define dimensions, measures and joins once, then query them from Python on DuckDB, PySpark or pandas through one native library.

• SQL engines (DuckDB, Spark): Wormhole compiles the query to engine SQL.
• pandas: Wormhole executes the plan itself on Arrow buffers in Zig (Wormhole.query), checked against DuckDB with a differential test.

Models are boring-semantic-layer style YAML and queries are its declarative query dicts (dimensions, measures, filters, order_by, limit, time_grain, time_range).

Status

Try it

pip install ziglang==0.16.0 duckdb pyyaml pyarrow
python -m ziglang build -Doptimize=ReleaseSafe
python tests/conformance_duckdb.py

import sys; sys.path.insert(0, "python")
from wormhole import Wormhole

wh = Wormhole("spec/models/flights.yml")
print(wh.emit_sql({
    "model": "flights_joined",
    "dimensions": ["carriers.name"],
    "measures": ["flight_count", "delay_per_flight"],
    "filters": [{"field": "flights_joined.origin", "operator": "in", "values": ["JFK", "LGA"]}],
    "order_by": [["flight_count", "desc"]],
    "limit": 5,
}))  # SQL for DuckDB; pass dialect="spark" for Spark SQL

Layout

src/expr.zig      scalar expression parser and SQL writer
src/sql.zig       manifest + query -> SQL (DuckDB, Spark)
src/c_api.zig     C ABI (include/wormhole.h)
python/wormhole   ctypes wrapper and YAML -> manifest translation
spec/             shared conformance suite; expected results recorded from boring-semantic-layer on DuckDB
tests/            conformance runners
docs/design.md    design note

Known v1 limits

• Joins: type: one (many-to-one, LEFT) only. No join_many, self-joins, or nested joins.
• Measures: count, sum, mean, min, max, nunique, and arithmetic over other measures. Mixed forms such as _.x.sum() / _.count() are rejected when the YAML is loaded.
• A filter may not mix dimension and measure fields; having= is not supported yet.
• order_by always sorts NULLs last.
• A filter on a name that is neither a dimension nor a measure is treated as a raw column of the model's table, so a typo there surfaces as an engine error.

References: sm8 (Scala) and boring-semantic-layer (Python).

Packaging

python tools/build_wheels.py cross-compiles libwormhole with zig build and writes one py3-none-<platform> wheel per target into dist/ (Linux wheels target glibc 2.17). Wheels contain no compiled Python extension, so one wheel serves every Python 3 version. The distribution name is wormhole-semantic because wormhole on PyPI belongs to another project; you still import wormhole. Nothing is published to PyPI yet.

Benchmarks

python tests/bench.py (2,000,000 flight rows, median of 3 runs, ms, one machine, single-threaded executor):

Reading it: on pandas data the native executor is 3 to 8 times faster than handing the same DataFrames to DuckDB on each call, because DuckDB converts them every time. Once the data sits in DuckDB tables, DuckDB is 3 to 4 times faster than the native executor (which does not use multiple threads yet), and Wormhole's SQL runs as fast as boring-semantic-layer's.