Skip to main content

RaySQL: DataFusion on Ray

This is a research project to evaluate performing distributed SQL queries from Python, using Ray and DataFusion.

Goals

  • Demonstrate how easily new systems can be built on top of DataFusion. See the design documentation to understand how RaySQL works.
  • Drive requirements for DataFusion's Python bindings.
  • Create content for an interesting blog post or conference talk.

Non Goals

  • Build and support a production system.

Example

Run the following example live in your browser using a Google Colab notebook.

import ray
from raysql.context import RaySqlContext
from raysql.worker import Worker

# Start our cluster
ray.init()

# create some remote Workers
workers = [Worker.remote() for i in range(2)]

# create a remote context and register a table
ctx = RaySqlContext.remote(workers)
ray.get(ctx.register_csv.remote('tips', 'tips.csv', True))

# Parquet is also supported
# ctx.register_parquet('tips', 'tips.parquet')

result_set = ray.get(ctx.sql.remote('select sex, smoker, avg(tip/total_bill) as tip_pct from tips group by sex, smoker'))
print(result_set)

Status

  • RaySQL can run 21 of the 22 TPC-H benchmark queries (query 15 needs DDL and that is not yet supported).

Features

  • Mature SQL support (CTEs, joins, subqueries, etc) thanks to DataFusion
  • Support for CSV and Parquet files

Limitations

  • Requires a shared file system currently

Performance

This chart shows the performance of RaySQL compared to Apache Spark for SQLBench-H at a very small data set (10GB), running on my desktop (Threadripper with 24 physical cores). Both RaySQL and Spark are configured with 24 executors.

Note that query 15 is excluded from both results since RaySQL does not support DDL yet.

Overall Time

RaySQL is ~65% faster overall for this scale factor and environment.

SQLBench-H Total

Per Query Time

Spark is much faster on some queries, likely due to broadcast exchanges, which RaySQL hasn't implemented yet.

SQLBench-H Per Query

Performance Plan

I'm planning on experimenting with the following changes to improve performance:

  • Make better use of Ray futures to run more tasks in parallel
  • Use Ray object store for shuffle data transfer to reduce disk I/O cost
  • Keep upgrading to newer versions of DataFusion to pick up the latest optimizations

Building

# prepare development environment (used to build wheel / install in development)
python3 -m venv venv
# activate the venv
source venv/bin/activate
# update pip itself if necessary
python -m pip install -U pip
# install dependencies (for Python 3.8+)
python -m pip install -r requirements-in.txt

Whenever rust code changes (your changes or via git pull):

# make sure you activate the venv using "source venv/bin/activate" first
maturin develop
python -m pytest

Benchmarking

Create a release build when running benchmarks, then use pip to install the wheel.

maturin develop --release

How to update dependencies

To change test dependencies, change the requirements.in and run

# install pip-tools (this can be done only once), also consider running in venv
python -m pip install pip-tools
python -m piptools compile --generate-hashes -o requirements-310.txt

To update dependencies, run with -U

python -m piptools compile -U --generate-hashes -o requirements-310.txt

More details here

Metadata

Release files for raysql 0.5.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for raysql 0.5.0
File Size Uploaded Provenance Actions
raysql-0.5.0.tar.gz 127.0 kB Without provenance Details for raysql-0.5.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for raysql 0.5.0
File Interpreter ABI Platform Actions
raysql-0.5.0-cp37-abi3-manylinux_2_34_x86_64.whl CPython 3.7 abi3 Linux glibc 2.34+ x86-64 Details for raysql-0.5.0-cp37-abi3-manylinux_2_34_x86_64.whl

Total release size: 14.0 MB

Release files / raysql-0.5.0.tar.gz

Download URL raysql-0.5.0.tar.gz
Size 127.0 kB
Tags Source
SHA-256 checksum
How to use checksums
e3aee31c2bb3240766d7597b4b3a211c7bf1ce0336c9808dc3ea865fad686ff2
BLAKE2b-256 checksum
How to use checksums
eb20f5fbc6c166c0109b149c688bff3cccf4ebd3eca7412983ae22cc14d803b8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via maturin/0.14.15

Release files / raysql-0.5.0-cp37-abi3-manylinux_2_34_x86_64.whl

Download URL raysql-0.5.0-cp37-abi3-manylinux_2_34_x86_64.whl
Size 13.9 MB
Tags CPython 3.7 Linux glibc 2.34+ x86-64 abi3
SHA-256 checksum
How to use checksums
2a85e9e54543c2d8c2baa1040c48349aca6cce6ec5a1eaa6aea7f53c0e74f949
BLAKE2b-256 checksum
How to use checksums
d20b9e0b88bfaee749f4bd3c64e4241464e35c5337e0c124652f8d57aa83ce4a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via maturin/0.14.15

Security for raysql 0.5.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page