opteryx 0.3.0a4

Python SQL Query Engine for Serverless Environments

Opteryx

Query your data, no database required

Opteryx is a distributed SQL Engine written in Python, designed for cloud-native environments.

Documentation | Examples | Contributing

Features

Scalable

Designed to run in Knative and similar environments like Google Cloud Run, Opteryx can scale down to zero, and scale up to respond to thousands of concurrent queries within seconds.

High Availability

Shared nothing design means each query can run in a separate container instance making it nearly impossible for a rogue query to affect any other users.

If a cluster, region or datacentre is unavailable, if you have instances able to run in another location, Opteryx will keep responding to queries. (inflight queries may not be recovered)

Query In Place

With Opteryx, you don't need to migrate data to a common data platform to be able to query it. You can store some of your data in parquet files on disk or cloud storage, and some in MongoDB or Firestore and access all of these in the same query.

Bring your own Files

Opteryx supports many popular data formats, including Parquet, ORC, Feather and JSONL, stored on local disk or on Cloud Storage. You can mix-and-match formats, so one dataset can be Parquet and another JSONL, and Opteryx will be able to JOIN across them.

Consumption-Based Billing Friendly

Opteryx is perfect for deployments to environments which are pay-as-you-use, like Google Cloud Run. Great for situations where you low-volume usage, or many environments, where the costs of many traditional database deployment can quickly add up.

Python Native

Opteryx is an Open Source Python library, it quickly and easily integrates into Python code, including Jupyter Notebooks, so you can start querying your data within a few minutes.

Time Travel

Designed for data analytics in environments where decisions need to be replayable, Opteryx allows you to query data as at a point in time in the past to replay decision algorithms against facts as they were known in the past. (data must be structured to enable temporal queries)

Schema Evolution

Changes to schemas and paritioning can be made without requiring any existing data to be updated. (data types can only be changed to compatitble types)

Usage/Examples

Example usage, querying one of the internal datasets.

import opteryx

conn = opteryx.connect()
cur = conn.cursor()
cur.execute("SELECT * FROM $planets;")
for row in cur.fetchall():
    print(row["name"])

Installation

Install from PyPI (recommended)

This will install the latest release version.

pip install opteryx

Install from GitHub

The lastest version, including pre-release and beta versions can be installed from GitHub. This is not recommended for production environments.

pip install git+https://github.com/mabel-dev/opteryx

Community

How do I get Support?

For support join our Slack channel.

How Can I Contribute?

All contributions, bug reports, documentation improvements, enhancements, and ideas are welcome.

Want to help build Opteryx? See the Contribution and Set Up Guides.

Security

See the project Security Policy for information about reporting vulnerabilities.

License

Status

Opteryx is in beta. Beta means different things to different people, to us, being beta means:

• Core functionality has good regression test coverage to help ensure stability
• Some edge cases may have undetected bugs
• Performance tuning may be incomplete
• Changes are focused on feature completion, bugs, performance, reducing debt, and security