Opteryx Core
Opteryx Core is the SQL execution engine behind opteryx.app. It is a fork of Opteryx with a smaller, more opinionated API and configuration surface, shaped around the workloads used by the hosted service.
This library is designed for fast, read-heavy analytical queries over Parquet-backed data. It handles SQL parsing, planning, predicate pushdown, projection pruning, and execution so you can query datasets from Python without standing up a separate warehouse.
This project is opinionated toward the needs of opteryx.app. It is still useful as a standalone library if you want to query local Parquet, NDJSON, and CSV datasets, embed SQL into a Python service or notebook, or experiment with engine internals directly.
Requirements
- Python 3.13
- A C/C++ toolchain for local source builds
- Rust/Cargo for the Rust extension in
src/
Install
pip install opteryx-core
Import it as:
import opteryx
Quick Start: Query Local Files
If your current working directory contains local Parquet data, the simplest way to use Opteryx Core is to register a local workspace and query it with dot-separated names.
import opteryx
from opteryx.connectors import DiskConnector
opteryx.register_workspace("data", DiskConnector)
session = opteryx.session()
result = session.execute_to_arrow(
"SELECT id, name FROM data.planets WHERE id < 5"
)
print(result)
In this model, dataset names are resolved relative to the current working directory. For example, data.planets resolves to ./data/planets, and Opteryx Core reads the Parquet files it finds there.
What It Is For
- Powering the execution layer used by
opteryx.app - Running analytical SQL against local Parquet-backed datasets
- Embedding a query engine inside Python applications, scripts, notebooks, and services
- Working on engine internals such as planning, execution, and Parquet performance
Local Development
The supported local build path is the repository Makefile:
make dev-install
make compile
make q
Useful targets:
| Target | Purpose |
|---|---|
make compile |
Clean in-place build of Cython, C++, and Rust extensions |
make c |
Incremental extension build |
make q |
Fast SQL shape smoke test |
make test |
Full pytest suite after compiling |
make dt |
Draken native unit tests |
make check |
Ruff and import-order checks without modifying files |
Do not use pip install . as the primary development build path; make compile matches the layout expected by this repository.
Repository Layout
| Path | Purpose |
|---|---|
opteryx/ |
Python package, planner, operators, connectors, expression evaluation, and Cython modules |
draken/ |
Native columnar vector substrate used by the execution engine |
rugo/ |
Internal Parquet and JSONL reader used by scans and metadata paths |
src/ |
Rust extension code, currently including the SQL dialect integration |
tests/ |
Unit, integration, fuzzing, sqllogictest, and benchmark harnesses |
testdata/ |
Local datasets and benchmark fixtures |
dev/ |
Development, release, vendoring, and analysis scripts |
scratch/ |
Experimental prototypes and one-off investigations |
third_party/ |
Vendored native dependencies |
Best With Opteryx Catalog
Opteryx Core works best when paired with the opteryx_catalog library. That is the intended model for named datasets, catalog-backed tables, and the general experience used in opteryx.app.
Typical setup:
import os
import opteryx
from opteryx import set_default_connector
from opteryx.connectors import OpteryxConnector
from opteryx_catalog import OpteryxCatalog
set_default_connector(
OpteryxConnector,
catalog=OpteryxCatalog,
firestore_project=os.environ["GCP_PROJECT_ID"],
firestore_database=os.environ["FIRESTORE_DATABASE"],
gcs_bucket=os.environ["GCS_BUCKET"],
)
Once configured, you can query catalog-backed datasets using dot-separated names such as public.space.planets or opteryx.ops.billing.
For local data, Opteryx Core is typically used through registered workspaces such as testdata, scratch, or data. Queries refer to datasets by dot-separated names relative to the workspace root, for example testdata.planets, testdata.satellites, or scratch.signals.
Where It Fits
Opteryx Core is best thought of as an embedded analytical engine rather than a full end-user platform. If you want a hosted experience, multi-tenant service features, and the broader product workflow, use opteryx.app. If you want the core engine in your own environment, this package gives you that engine directly. If you want the intended table-resolution model, pair it with opteryx_catalog.
Contributing
If you use Opteryx-Core yourself, we want to hear from you.
- Use it on your own datasets
- Raise bugs when queries, schemas, or performance do not behave as expected
- Open pull requests for fixes, tests, docs, or performance improvements
- Share repro cases, failing queries, and edge-case Parquet files
This project is being actively built, and outside usage helps make it better.
Docs: https://docs.opteryx.app/ Source: https://github.com/mabel-dev/opteryx-core License: Apache-2.0
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file opteryx_core-0.9.49.tar.gz.
File metadata
- Download URL: opteryx_core-0.9.49.tar.gz
- Upload date:
- Size: 12.1 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
9e4d66bc5a9f2ef661b8fb818c1d5961ad86c8fb9dc3bdaa3eceda8e0189ff8c
|
|
| MD5 |
94113f590872e5ab336b06e0f3d132c6
|
|
| BLAKE2b-256 |
13653f9bbf39daf02ef15094c13def93a415ef672923782f179e9d4141420e5f
|
File details
Details for the file opteryx_core-0.9.49-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.
File metadata
- Download URL: opteryx_core-0.9.49-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
- Upload date:
- Size: 44.4 MB
- Tags: CPython 3.14, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
47c9a6d1007a10ef62d849f5eadac2c008db3f2097f6c0743af39c69b6f1d6d3
|
|
| MD5 |
2e4cadc1c9450139c5931e578aaeb096
|
|
| BLAKE2b-256 |
79eaecfbc40f063cef51934a18897edd1a96c22b9837b7fb36be0c2e582a0d8a
|
File details
Details for the file opteryx_core-0.9.49-cp314-cp314-macosx_11_0_arm64.whl.
File metadata
- Download URL: opteryx_core-0.9.49-cp314-cp314-macosx_11_0_arm64.whl
- Upload date:
- Size: 24.7 MB
- Tags: CPython 3.14, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
0fe2b5c6ea59cf0ba0c3f59e5ce34f2d86e51b56ac84a7bd49773ffc189c015d
|
|
| MD5 |
5720c1e0b75ba63601d073c2d7d1076a
|
|
| BLAKE2b-256 |
7be6dcc2177728aadfe7d02c2e6f847bc7a9827aedbd46d1fdd86133adbad21b
|
File details
Details for the file opteryx_core-0.9.49-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.
File metadata
- Download URL: opteryx_core-0.9.49-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
- Upload date:
- Size: 44.3 MB
- Tags: CPython 3.13, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
78a8e97c8a6651ec7e62cfb63e52bf4a064403674131f1c28c2ca29d013a2ebd
|
|
| MD5 |
739912c5ddc742caa60b12d9707b6e65
|
|
| BLAKE2b-256 |
3bf4181802966bf7a1aa63ce110ca60542c4e5dac0718f7b8e43872d4276ec09
|
File details
Details for the file opteryx_core-0.9.49-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.
File metadata
- Download URL: opteryx_core-0.9.49-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
- Upload date:
- Size: 44.4 MB
- Tags: CPython 3.12, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
c4c1f4c1b6095690cb41693f022758b10102d40b7c569e43576e06d1cbb0db5c
|
|
| MD5 |
e6368409a9888cfdb5f7ed34f98b1067
|
|
| BLAKE2b-256 |
c0919de6bc12ac896c23751523146f3b096c2a9949e2f749598ccfada4458975
|
