semql-engine

In-process executor for semql FederatedPlan results. Runs each per-backend fragment via a caller-supplied Adapter, materialises the rows into in-memory DuckDB, then runs the plan's merge SQL against the assembled tables.

semql core stays sans-io. semql-engine is the opt-in package that turns a FederatedPlan into result rows when you want the cross-source execution done for you.

Install

pip install semql-engine

Quickstart

import duckdb
from semql import Catalog, Dialect, compile_federated_query
from semql_engine import DuckDBAdapter, Engine

catalog = Catalog([...])  # cubes spanning multiple backends
plan = compile_federated_query(query, catalog.as_dict())

engine = Engine()
engine.register(Dialect.POSTGRES, my_pg_adapter)
engine.register(Dialect.BIGQUERY, my_bq_adapter)
result = engine.run(plan)
result.columns  # ['region', 'revenue', ...]
result.rows     # [(...), ...]   — or stream with engine.iter_rows(plan)

What it does

For every fragment in the plan, the engine calls the adapter registered for that backend with (sql, params). It loads the resulting rows into a DuckDB table named frag_<i> (matching FederatedPlan.fragments indices) and finally applies plan.merge_spec — rendered to DuckDB SQL by semql_engine.merge — to produce the merged shape.

Single-fragment plans (single-backend queries that went through compile_federated_query anyway) work transparently — the merge is a pass-through.

Adapters

An Adapter is anything with execute(sql, params) -> AdapterResult where AdapterResult carries columns: list[str] and an iterable of row dicts. Built-ins:

• DuckDBAdapter(con) — runs the SQL inside an existing DuckDB connection. Useful for local CSV / Parquet enrichment cubes.
• DBAPIAdapter(con) — wraps any PEP-249 connection (psycopg, mysql, sqlite, etc).

Bring your own for warehouses that need a vendor SDK.

Semi-joins

A cross-backend semi-join (restrict an outer dimension to the value set of an inner query, shipped as a value list rather than a join) compiles to a SemiJoinPlan via semql.compile_semi_join_query. Run it with run_semi_join(plan, engine) — it executes each inner plan, projects the key column to a value list, and runs the outer query with that list bound as an IN / NOT IN filter:

from semql import compile_semi_join_query
from semql_engine import Engine, run_semi_join

plan = compile_semi_join_query(query, catalog.as_dict())
result = run_semi_join(plan, engine)  # same ExecutionResult shape as run()

Scope

v1 mirrors compile_federated_query v1:

• Sum / count / avg supported (avg is decomposed at compile and recomposed in the merge SQL); other aggregations are refused by the compiler before the engine ever sees them.
• Equality bridge joins only.
• No compare mode, no boolean where tree across backends.

The engine itself is small; most of the federation logic lives in semql.federate.

Status

Early development. The Adapter contract is stable; the federation shape mirrors compile_federated_query v1 (sum / count / avg and equality bridge joins only).