Opteryx Core

Opteryx Core is the SQL execution engine behind opteryx.app. It is a fork of Opteryx with a smaller, more opinionated API and configuration surface, shaped around the workloads used by the hosted service.

This library is designed for fast, read-heavy analytical queries over columnar data. It handles SQL parsing, planning, predicate pushdown, projection pruning, and execution so you can query datasets from Python without standing up a separate warehouse.

Query planning is Python; query execution is native. Once the planner has produced a physical plan, the engine runs it in compiled code end to end — scan, operators, scheduling, and dispatch — and neither PyArrow nor NumPy is present anywhere in the engine.

This project is opinionated toward the needs of opteryx.app. It is still useful as a standalone library if you want to query local Parquet, NDJSON, CSV, and .skene datasets, embed SQL into a Python service or notebook, or experiment with engine internals directly.

Requirements

• Python 3.11 or later
• A C/C++ toolchain for local source builds
• Rust/Cargo for the Rust extension in src/

Hardware Support

Wheels are built against a fixed instruction-set floor. A CPU below the floor will fail with an illegal-instruction error rather than run slowly.

• x86-64: Haswell (2013) or later — AVX2, BMI1/2, FMA, PCLMUL.
• ARM (AArch64): ARMv8-A with the AES/PMULL crypto extension. Plain ARMv8-A is not enough.

On Linux you can check a machine with grep -m1 -o -w aes /proc/cpuinfo — no output means the CPU is below the floor.

Building from source on a Raspberry Pi 5

Any Raspberry Pi 5 runs the published wheels. Building from source needs more:

• 8 GB RAM or more. The largest translation unit peaks at 5.3 GB on its own, so 2 GB and 4 GB boards cannot build. On 8 GB, build with make compile JOBS=2; the default (one job per core, 4) runs out of memory.
• GCC 14: Debian 13 (trixie) / Raspberry Pi OS trixie. GCC 12 (bookworm) rejects vmull_p64 under -march=armv8-a+crc+aes, because its arm_neon.h gates PMULL behind +crypto.
• NVMe storage recommended. The source tree, without the large benchmark datasets, is about 4.2 GB.
• Toolchain: Python 3.14, Rust (rustup), build-essential, pkg-config, libssl-dev, libnghttp2-dev, and clang (the native tests in make q call clang++ directly).

A full make compile JOBS=2 takes about an hour on a Pi 5 (8 GB, NVMe).

Install

pip install opteryx-core

Import it as:

import opteryx

Quick Start: Query Local Files

If your current working directory contains local Parquet data, the simplest way to use Opteryx Core is to register a local workspace and query it with dot-separated names.

import opteryx
from opteryx.connectors import DiskConnector

opteryx.register_workspace("data", DiskConnector)

session = opteryx.session()

for morsel in session.execute_to_morsels("SELECT id, name FROM data.planets WHERE id < 5"):
    print(morsel)

Results arrive as Draken morsels — batches of columns — streamed as the engine produces them, so a large result never has to fit in memory at once. A morsel prints as a table, and carries num_rows, column_names and column(name).to_pylist() for getting at the values. Once the stream has been read to the end, session.rowcount is the number of rows it delivered.

In this model, dataset names are resolved relative to the current working directory. For example, data.planets resolves to ./data/planets, and Opteryx Core reads the files it finds there, detecting the format from their extension. See File Formats for what it can read.

There is also a command line, for querying without writing Python:

python -m opteryx "SELECT id, name FROM data.planets WHERE id < 5"

What It Is For

• Powering the execution layer used by opteryx.app
• Running analytical SQL against local Parquet, CSV, JSONL, and .skene datasets
• Embedding a query engine inside Python applications, scripts, notebooks, and services
• Working on engine internals such as planning, native execution, and file-format performance
• Using the file engine on its own, via the rugo wheel

Local Development

The supported local build path is the repository Makefile:

make dev-install
make compile
make q

Useful targets:

Do not use pip install . as the primary development build path; make compile matches the layout expected by this repository.

Repository Layout

Distributions

This is a single repository that produces two wheels from one source tree. They are packagings of the same sources, not separate forks, so they cannot drift: the extension definitions are single-sourced in build_common.py.

draken is not published separately; it ships inside both. skene is not published separately either; it ships inside opteryx-core only, and the rugo wheel does not carry it. Opteryx never depends on the published rugo wheel; rugo is intrinsic to it, and the standalone wheel is a separate packaging of the same code.

Wheels are built in CI, never locally. For local development use make compile, as above.

File Formats

Datasets are read by extension, and a dataset is one format throughout — a directory mixing formats is an error rather than a best-effort read.

• Parquet — the default for stored data and for interchange, read through rugo
• CSV and JSONL/NDJSON — read through rugo
• Avro — object container files, read through rugo's native Avro reader with the read_avro() table function; it is not a stored-dataset format
• .skene — the draken-native format. It stores one or more row groups of draken vectors losslessly, including the things Parquet drops: an IPv4 column round-trips as a UINT32 refined by an IPV4 logical descriptor rather than losing the refinement, and dictionary encoding and layout hints are restored rather than re-derived. It is deliberately not portable and no foreign reader is promised, so Parquet remains the right choice for interchange; .skene is for cases where the draken-native round trip is what matters. See skene/FORMAT.md for the specification.

Parquet, CSV, JSONL, and Avro files can also be named directly with the read_parquet(), read_csv(), read_jsonl(), and read_avro() table functions. A read_avro() glob takes its schema from the first file, and every other file is read as that schema — a field a file lacks is NULL (or its default), an int widens to a long, and a file that cannot be read that way fails the query naming it. There is no read_skene() — skene datasets are read through a registered workspace like any other dataset.

Best With Opteryx Catalog

Opteryx Core works best when paired with the opteryx_catalog library. That is the intended model for named datasets, catalog-backed tables, and the general experience used in opteryx.app.

Typical setup:

import os

import opteryx

from opteryx import set_default_connector
from opteryx.connectors import OpteryxConnector
from opteryx_catalog import OpteryxCatalog

set_default_connector(
    OpteryxConnector,
    catalog=OpteryxCatalog,
    firestore_project=os.environ["GCP_PROJECT_ID"],
    firestore_database=os.environ["FIRESTORE_DATABASE"],
    gcs_bucket=os.environ["GCS_BUCKET"],
)

Once configured, you can query catalog-backed datasets using dot-separated names such as public.space.planets or opteryx.ops.billing.

For local data, Opteryx Core is typically used through registered workspaces such as testdata, scratch, or data. Queries refer to datasets by dot-separated names relative to the workspace root, for example testdata.planets, testdata.satellites, or scratch.signals.

Where It Fits

Opteryx Core is best thought of as an embedded analytical engine rather than a full end-user platform. If you want a hosted experience, multi-tenant service features, and the broader product workflow, use opteryx.app. If you want the core engine in your own environment, this package gives you that engine directly. If you want the intended table-resolution model, pair it with opteryx_catalog.

Contributing

Third-party benchmark material

The Apache-2.0 licence covers Opteryx-Core's own code. It does not cover the benchmark queries, data and results derived from ClickBench (CC BY-NC-SA 4.0), TPC-H, the Join Order Benchmark and the H2O db-benchmark (MPL-2.0), which keep the terms of their upstream sources. TPC-H results here are derived from TPC-H, are not audited, and are not comparable to published TPC-H results. See THIRD_PARTY_BENCHMARKS.md for attributions, licences and changes made.

If you use Opteryx-Core yourself, we want to hear from you.

• Use it on your own datasets
• Raise bugs when queries, schemas, or performance do not behave as expected
• Open pull requests for fixes, tests, docs, or performance improvements
• Share repro cases, failing queries, and edge-case Parquet files

This project is being actively built, and outside usage helps make it better.

Docs: https://docs.opteryx.app/ Source: https://github.com/mabel-dev/opteryx-core License: Apache-2.0