Opteryx Core

Opteryx Core is the SQL execution engine behind opteryx.app. It is a fork of Opteryx with a smaller, more opinionated API and configuration surface, shaped around the workloads used by the hosted service.

This library is designed for fast, read-heavy analytical queries over Parquet-backed data. It handles SQL parsing, planning, predicate pushdown, projection pruning, and execution so you can query datasets from Python without standing up a separate warehouse.

This project is opinionated toward the needs of opteryx.app. It is still useful as a standalone library if you want to query local Parquet, NDJSON, and CSV datasets, embed SQL into a Python service or notebook, or experiment with engine internals directly.

Requirements

  • Python 3.11 or later
  • A C/C++ toolchain for local source builds
  • Rust/Cargo for the Rust extension in src/

Install

pip install opteryx-core

Import it as:

import opteryx

Quick Start: Query Local Files

If your current working directory contains local Parquet data, the simplest way to use Opteryx Core is to register a local workspace and query it with dot-separated names.

import opteryx
from opteryx.connectors import DiskConnector

opteryx.register_workspace("data", DiskConnector)

session = opteryx.session()
result = session.execute_to_arrow(
    "SELECT id, name FROM data.planets WHERE id < 5"
)

print(result)

In this model, dataset names are resolved relative to the current working directory. For example, data.planets resolves to ./data/planets, and Opteryx Core reads the Parquet files it finds there.

What It Is For

  • Powering the execution layer used by opteryx.app
  • Running analytical SQL against local Parquet-backed datasets
  • Embedding a query engine inside Python applications, scripts, notebooks, and services
  • Working on engine internals such as planning, execution, and Parquet performance

Local Development

The supported local build path is the repository Makefile:

make dev-install
make compile
make q

Useful targets:

Target Purpose
make compile Clean in-place build of Cython, C++, and Rust extensions
make c Incremental extension build
make q Fast SQL shape smoke test
make test Full pytest suite after compiling
make dt Draken native unit tests
make check Ruff and import-order checks without modifying files

Do not use pip install . as the primary development build path; make compile matches the layout expected by this repository.

Repository Layout

Path Purpose
opteryx/ Python package, planner, operators, connectors, expression evaluation, and Cython modules
draken/ Native columnar vector substrate used by the execution engine
rugo/ Internal Parquet and JSONL reader used by scans and metadata paths
src/ Rust extension code, currently including the SQL dialect integration
tests/ Unit, integration, fuzzing, sqllogictest, and benchmark harnesses
testdata/ Local datasets and benchmark fixtures
dev/ Development, release, vendoring, and analysis scripts
scratch/ Experimental prototypes and one-off investigations
third_party/ Vendored native dependencies

Best With Opteryx Catalog

Opteryx Core works best when paired with the opteryx_catalog library. That is the intended model for named datasets, catalog-backed tables, and the general experience used in opteryx.app.

Typical setup:

import os

import opteryx

from opteryx import set_default_connector
from opteryx.connectors import OpteryxConnector
from opteryx_catalog import OpteryxCatalog

set_default_connector(
    OpteryxConnector,
    catalog=OpteryxCatalog,
    firestore_project=os.environ["GCP_PROJECT_ID"],
    firestore_database=os.environ["FIRESTORE_DATABASE"],
    gcs_bucket=os.environ["GCS_BUCKET"],
)

Once configured, you can query catalog-backed datasets using dot-separated names such as public.space.planets or opteryx.ops.billing.

For local data, Opteryx Core is typically used through registered workspaces such as testdata, scratch, or data. Queries refer to datasets by dot-separated names relative to the workspace root, for example testdata.planets, testdata.satellites, or scratch.signals.

Where It Fits

Opteryx Core is best thought of as an embedded analytical engine rather than a full end-user platform. If you want a hosted experience, multi-tenant service features, and the broader product workflow, use opteryx.app. If you want the core engine in your own environment, this package gives you that engine directly. If you want the intended table-resolution model, pair it with opteryx_catalog.

Contributing

If you use Opteryx-Core yourself, we want to hear from you.

  • Use it on your own datasets
  • Raise bugs when queries, schemas, or performance do not behave as expected
  • Open pull requests for fixes, tests, docs, or performance improvements
  • Share repro cases, failing queries, and edge-case Parquet files

This project is being actively built, and outside usage helps make it better.

Docs: https://docs.opteryx.app/ Source: https://github.com/mabel-dev/opteryx-core License: Apache-2.0

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.9.87)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for joocer from gravatar.com joocer

Credits

Author/Maintainer: Justin Joyce

Requires

Python >=3.11

Provides Extra

testing performance embeddings

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

opteryx_core-0.9.54.tar.gz (12.2 MB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

opteryx_core-0.9.54-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (42.2 MB view details)

Uploaded CPython 3.14manylinux: glibc 2.17+ x86-64

opteryx_core-0.9.54-cp314-cp314-macosx_11_0_arm64.whl (24.5 MB view details)

Uploaded CPython 3.14macOS 11.0+ ARM64

opteryx_core-0.9.54-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (42.1 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.17+ x86-64

opteryx_core-0.9.54-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (42.2 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ x86-64

opteryx_core-0.9.54-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (42.3 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ x86-64

File details

Details for the file opteryx_core-0.9.54.tar.gz.

File metadata

  • Download URL: opteryx_core-0.9.54.tar.gz
  • Upload date:
  • Size: 12.2 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for opteryx_core-0.9.54.tar.gz
Algorithm Hash digest
SHA256 9ef546d11fdce183ab5c7fb627be0e0a840950d82b68f3d2f330e61d3448c68f
MD5 333569d2799e7c942ed1e8f375fbffcd
BLAKE2b-256 fd00ee434b47aaedb667e172bacea7b95b7aca473b87ca270404e0d185103dc1

See more details on using hashes here.

File details

Details for the file opteryx_core-0.9.54-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for opteryx_core-0.9.54-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 5c4a764e5c967940fede5f5eb9271b5beef4349fa9a7ceb3778d3a1537d96792
MD5 2b153041d0cf5d1742a5c2b3d79c77a0
BLAKE2b-256 3027dbdac0f525d84dbcefee33ce39047fd18f43b025a5d4f0bb4401708df548

See more details on using hashes here.

File details

Details for the file opteryx_core-0.9.54-cp314-cp314-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for opteryx_core-0.9.54-cp314-cp314-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 9720cd0143fa07c4aec58950e69595710813ec9a3a4ca916634bd9bc206ffdf2
MD5 022e5bc55f30e3a3846c2804168c925c
BLAKE2b-256 43a1f9066b8389b5b79e93f780cfb5010eaed81cbbae8667cee3fa16c38a5ca5

See more details on using hashes here.

File details

Details for the file opteryx_core-0.9.54-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for opteryx_core-0.9.54-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 1a662e74f0a9c285040b11f0e6238f59a1d5a6e5c633bb68e0cf2203c0e5e97f
MD5 6e892e2c36dfcb6c977dc35fd3839f40
BLAKE2b-256 60084b5e9ae2e12e07849ef5d3c433f1b2f5d1f0df4f7a1b52c59d3557de5c0b

See more details on using hashes here.

File details

Details for the file opteryx_core-0.9.54-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for opteryx_core-0.9.54-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 451d2ce8f457461900c4c210cd78607f35bc710c5e85fbe5dc40bd4dc12b7f4c
MD5 065ecc8ef139320820e687514957ce7c
BLAKE2b-256 97f0daa7e779d009b226e64778c0ea587f8d21dad7e046195070f8a8e155ddeb

See more details on using hashes here.

File details

Details for the file opteryx_core-0.9.54-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for opteryx_core-0.9.54-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 17e81cae04e6ec8b64f57173deeb80e4e22a3c08871e80d977b2a047f779e0d8
MD5 9e6524c74c71e9f15eec6ba0b2316181
BLAKE2b-256 d5be9460cae45dd5f5db298789b0fbc4a2b2bda8c05a146d0cc349ae145e0dab

See more details on using hashes here.

Release history Release notifications | RSS feed

0.9.87

7 files

0.9.86

7 files

0.9.85

7 files

0.9.84

7 files

0.9.83

7 files

0.9.82

7 files

0.9.81

7 files

0.9.80

7 files

0.9.79

7 files

0.9.78

7 files

0.9.77

7 files

0.9.76

7 files

0.9.75

7 files

0.9.74

7 files

0.9.73

7 files

0.9.71

10 files

0.9.70

10 files

0.9.69

10 files

0.9.68

6 files

0.9.67

6 files

0.9.66

6 files

0.9.65

6 files

0.9.64

6 files

0.9.63

6 files

0.9.62

6 files

0.9.60

6 files

0.9.58

6 files

0.9.56

6 files

0.9.55

6 files

This release

0.9.54 This release

6 files

0.9.53

6 files

0.9.52

6 files

0.9.50

5 files

0.9.49

5 files

0.9.48

5 files

0.9.47

5 files

0.9.46

5 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page