opteryx 0.0.0a58

Distributed SQL Engine

A query engine for your data, no database required

Documentation | Examples | Contributing | Blog

---

NOTE Opteryx is an alpha product. Alpha means different things to different people, to us, being alpha means:

• Some features you may expect from a Query Engine may not be available
• Some features may have undetected bugs in them
• Some features may not be tuned for performance
• Some previously working features may break

---

Opteryx has no server component, Opteryx just runs when you need it making it ideal for deployments to platforms like Kubernetes, GCP Cloud Run, AWS Fargate and Knative.

• Bug Reports GitHub Issues
• Feature Requests GitHub Issues
• Source Code GitHub
• Discussions GitHub Discussions

How Can I Contribute?

All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.

If you have a suggestion for an improvement or a bug, raise a ticket or start a discussion.

Want to help build mabel? See the contribution guidance.

What Opteryx is

How is it different to SQLite or DuckDB?

Opteryx is solving a different problem in the same space as these solutions. Opteryx avoids loading the dataset into memory unless there is no other option, as such it can query petabytes of data on a single, modest sized node.

This also means that queries are not as fast as solutions like SQLite or DuckDB.

How is it different to MySQL or BigQuery?

Opteryx is an ad hoc database, if it can read the files, it can be used to query the contents of them. This means it can leverage data files used by other systems.

Opteryx is read-only, you can't update or delete data, and it also doesn't have or enforce indexes in your data.

How is it differnt to Trino?

Opteryx is designed to run in a serverless environment where there is no persistent state. There is no server or coordinator for Opteryx, the Engine is only running when it is serving queries.

When you are not running queries, your cost to run Opteryx is nil (+). This is particularly useful if you have a small team accessing data.

This also means the Query Engine can scale quickly to respond to demand, running Opteryx in an environment like Cloud Run on GCP, you can scale from 0 to 1000 concurrent queries within seconds - back to 0 almost as quickly.

(+) depending on specifics of your hosting arrangement.

Security

See the project security policy for information about reporting vulnerabilities.

License

The foundational technologies in Opteryx are:

• Apache Arrow memory model and compute kernels for efficient processing of data
• Parts of PyArrow_Ops by Tom Scheffers has been integrated into the codebase to support handling Arrow structured data
• sqloxide is used to parse SQL queries to syntax trees
• cython
• numpy
• orjson
• [cityhash] is used for non-cryptographic hashing