PathsData Distributed

PathsData Distributed Query Engine - Python client for distributed SQL execution.

This project is versioned and released independently from the main Ballista project and is intentionally not part of the default Cargo workspace so that it doesn't cause overhead for maintainers of the main Ballista codebase.

Creating a SessionContext

[!IMPORTANT] Current approach is to support datafusion python API, there are know limitations of current approach, with some cases producing errors. We trying to come up with the best approach to support datafusion python interface. More details could be found at #1142

Creates a new context and connects to a Ballista scheduler process.

from pathsdata_distributed import BallistaBuilder
>>> ctx = BallistaBuilder().standalone()

Example SQL Usage

>>> ctx.sql("create external table t stored as parquet location './testdata/test.parquet'")
>>> df = ctx.sql("select * from t limit 5")
>>> pyarrow_batches = df.collect()

Example DataFrame Usage

>>> df = ctx.read_parquet('./testdata/test.parquet').limit(5)
>>> pyarrow_batches = df.collect()

Scheduler and Executor

Scheduler and executors can be configured and started from python code.

To start scheduler:

from pathsdata_distributed import BallistaScheduler

scheduler = BallistaScheduler()

scheduler.start()
scheduler.wait_for_termination()

For executor:

from pathsdata_distributed import BallistaExecutor

executor = BallistaExecutor()

executor.start()
executor.wait_for_termination()

Development Process

Creating Virtual Environment

python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt

Building

maturin develop

Note that you can also run maturin develop --release to get a release build locally.

Testing

python3 -m pytest