Sail

Build Status Codecov PyPI Release Static Slack Badge

Sail is a drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads on a distributed, multimodal compute engine.

  • Compatible with the Spark Connect protocol, supporting the Spark SQL and DataFrame API with no code rewrites required.
  • 100% Rust-native with no JVM overhead, delivering memory safety, instant startup, and predictable performance.
  • ~10× faster than Spark and 98% cheaper on infrastructure costs. See derived TPC-H benchmarks.
  • Proven on ClickBench, outperforming Spark, popular Spark accelerators, Databricks, and Snowflake.

Documentation

The documentation of the latest Sail version can be found here.

Installation

Quick Start

Sail is available as a Python package on PyPI. You can install it along with PySpark in your Python environment.

pip install pysail
pip install "pyspark-client"

Advanced Use Cases

You can install Sail from source to optimize performance for your specific hardware architecture. The detailed Installation Guide walks you through this process step-by-step.

If you need to deploy Sail in production environments, the Deployment Guide provides comprehensive instructions for deploying Sail on Kubernetes clusters and other infrastructure configurations.

Getting Started

Starting the Sail Server

Option 1: Command Line Interface. You can start the local Sail server using the sail command.

sail spark server --port 50051

Option 2: Python API. You can start the local Sail server using the Python API.

from pysail.spark import SparkConnectServer

server = SparkConnectServer(port=50051)
server.start(background=False)

Option 3: Kubernetes. You can deploy Sail on Kubernetes and run Sail in cluster mode for distributed processing. Please refer to the Kubernetes Deployment Guide for instructions on building the Docker image and writing the Kubernetes manifest YAML file.

kubectl apply -f sail.yaml
kubectl -n sail port-forward service/sail-spark-server 50051:50051

Connecting to the Sail Server

Once you have a running Sail server, you can connect to it in PySpark. No changes are needed in your PySpark code!

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost:50051").getOrCreate()
spark.sql("SELECT 1 + 1").show()

Please refer to the Getting Started guide for further details.

Spark Compatibility

Sail is designed to be compatible with Spark 3.5.x, Spark 4.x, and later versions. Existing PySpark code works out of the box once you connect your Spark client session to Sail over the Spark Connect protocol.

As a starting point, Sail ships with an experimental PySpark function compatibility check script that scans your codebase for PySpark functions and reports their Sail support status.

python -m pysail.examples.spark.compatibility_check <directory>

Experimental Use the script as a rough first pass only. The script checks whether referenced PySpark functions are implemented in Sail. It does not verify behavioral parity. It looks for functions used in DataFrame operations but does not cover Spark SQL strings.

See the Migration Guide for recommended migration practices.

Feature Highlights

Lakehouse Formats and Catalog Providers

Sail provides native support for the Delta Lake and Apache Iceberg table formats. It integrates with catalog providers including Apache Iceberg REST Catalog, AWS Glue, Unity Catalog, Hive Metastore, and Microsoft OneLake.

For more details on usage and best practices, see the Data Sources Guide and Catalog Guide.

Storage

Sail supports a variety of storage backends for reading and writing data, including:

  • AWS S3
  • Azure
  • Hugging Face
  • Cloudflare R2
  • Google Cloud Storage
  • HDFS
  • File systems
  • HTTP/HTTPS
  • In-memory storage

See the Storage Guide for more details.

Why Choose Sail?

For over 15 years, Spark has been the default engine for distributed data processing, powering ETL, machine learning, and analytics pipelines across nearly every industry.

But the JVM foundation that made Spark possible is now what holds it back. Sail is built to be a familiar, performant alternative without the JVM tax.

Sail is Spark-compatible

Sail offers a drop-in replacement for Spark SQL and the Spark DataFrame API. Existing PySpark code works out of the box once you connect your Spark client session to Sail over the Spark Connect protocol.

  • Spark SQL Dialect Support. A custom Rust parser (built with parser combinators and Rust procedural macros) covers Spark SQL syntax with production-grade accuracy.
  • DataFrame API Support. Spark DataFrame operations run on Sail with identical semantics.
  • Python UDF, UDAF, UDWF, and UDTF Support. Python, Pandas, and Arrow UDFs all follow the same conventions as Spark.

Sail’s Advantages over Spark

  • Rust-Native Engine. No garbage collection pauses, no JVM memory tuning, and low memory footprint.
  • Columnar Format and Vectorized Execution. Built on top of Apache Arrow and Apache DataFusion, the columnar in-memory format and SIMD instructions unlock blazing-fast query execution.
  • Lightning-Fast Python UDFs. Python code runs inside Sail with zero serialization overhead as Arrow array pointers enable zero-copy data sharing.
  • Performant Data Shuffling. Workers exchange Arrow columnar data directly, minimizing shuffle costs for joins and aggregations.
  • Lightweight, Stateless Workers. Workers start in seconds, consume only a few megabytes of memory at idle, and scale elastically to cut cloud costs and simplify operations.
  • Concurrency and Memory Safety You Can Trust. Rust’s ownership model prevents null pointers, race conditions, and unsafe memory access for unmatched reliability.

Ready to bring your existing workloads over? Our Migration Guide shows you how.

Benchmark Results

Derived TPC-H results show that Sail outperforms Apache Spark in every query:

  • Execution Time: ~10× faster across diverse SQL workloads.
  • Hardware Cost: 98% lower with significantly lower peak memory usage and zero shuffle spill.
Metric Spark Sail
Total Query Time 534.78 s 52.81 s
Query Speed-Up Baseline 176% – 2819%
Peak Memory Usage 72 GB 26 GB
Disk Write (Shuffle Spill) 115 GB 0 GB

These results come from a derived TPC-H benchmark (22 queries, scale factor 100, Parquet format) on AWS r8g.4xlarge instances.

Query Time Comparison

See the full analysis and graphs on our Benchmark Results page.

Further Reading

  • Architecture – Overview of Sail’s design for both local and cluster modes, and how it transitions seamlessly between them.
  • Query Planning – Detailed explanation of how Sail parses SQL and Spark relations, builds logical and physical plans, and handles execution for local and cluster modes.
  • SQL and DataFrame Features – Complete reference for Spark SQL and DataFrame API compatibility.
  • LakeSail Blog – Updates on Sail releases, benchmarks, and technical insights.

Project links

Data verified by PyPI on Aug 24, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for lakesail from gravatar.com lakesail

Credits

Author: LakeSail

GitHub Statistics

Data verified by PyPI on Aug 24, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache Software License

Requires

Python <3.15, >=3.10

Provides Extra

jdbc mcp test vortex

Tags

data big data sql spark pyspark arrow datafusion

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pysail-0.7.1.tar.gz (3.6 MB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

pysail-0.7.1-cp38-abi3-win_amd64.whl (63.5 MB view details)

Uploaded CPython 3.8+Windows x86-64

pysail-0.7.1-cp38-abi3-manylinux_2_24_aarch64.whl (54.2 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.24+ ARM64

pysail-0.7.1-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (57.3 MB view details)

Uploaded CPython 3.8+manylinux: glibc 2.17+ x86-64

pysail-0.7.1-cp38-abi3-macosx_11_0_arm64.whl (52.1 MB view details)

Uploaded CPython 3.8+macOS 11.0+ ARM64

pysail-0.7.1-cp38-abi3-macosx_10_12_x86_64.whl (56.1 MB view details)

Uploaded CPython 3.8+macOS 10.12+ x86-64

File details

Details for the file pysail-0.7.1.tar.gz.

File metadata

  • Download URL: pysail-0.7.1.tar.gz
  • Upload date:
  • Size: 3.6 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for pysail-0.7.1.tar.gz
Algorithm Hash digest
SHA256 7d7ad335e1c8542c0df98b54700751bfa10b51a5223f0831825b96522df4ac26
MD5 39d96c6735a6f901df593bdd53384b4d
BLAKE2b-256 cc00ce8f3508abcf06e2457eaf82f1ef7e66a38b18e4e570dc672de3d8223640

See more details on using hashes here.

Provenance

The following attestation bundles were made for pysail-0.7.1.tar.gz:

Publisher: release.yml on lakehq/sail

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file pysail-0.7.1-cp38-abi3-win_amd64.whl.

File metadata

  • Download URL: pysail-0.7.1-cp38-abi3-win_amd64.whl
  • Upload date:
  • Size: 63.5 MB
  • Tags: CPython 3.8+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for pysail-0.7.1-cp38-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 c59a352aa079280f766cf4881940b2edb1621e89477a05448405e10d6bd254fa
MD5 b8ca61db0ef8d10908cf8749349c6565
BLAKE2b-256 fd0495aa5db17c616a9fef7f751576057d70720eaa65d716c355eac7a353ea22

See more details on using hashes here.

Provenance

The following attestation bundles were made for pysail-0.7.1-cp38-abi3-win_amd64.whl:

Publisher: release.yml on lakehq/sail

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file pysail-0.7.1-cp38-abi3-manylinux_2_24_aarch64.whl.

File metadata

File hashes

Hashes for pysail-0.7.1-cp38-abi3-manylinux_2_24_aarch64.whl
Algorithm Hash digest
SHA256 61ffe5d970b2b273c326df1579c7bf262dc4e148013d6d0c592fb7c097ff9ae9
MD5 17b49e1f2a7b354aaf9d9c320cb3eba9
BLAKE2b-256 8d6eaba6026cc0916d9ed99fad9d584076d6a9652ccef71939b3b933be65ec55

See more details on using hashes here.

Provenance

The following attestation bundles were made for pysail-0.7.1-cp38-abi3-manylinux_2_24_aarch64.whl:

Publisher: release.yml on lakehq/sail

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file pysail-0.7.1-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for pysail-0.7.1-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 360410b168608effc5ac80af478861cc27f6dc17172c11b3200f346dd448d7c9
MD5 4e8e3f527483f6a57685f847b2530939
BLAKE2b-256 3a17940716d4eecc275e4caafa70f73033eb468317f63fe8784eb51e3a97a694

See more details on using hashes here.

Provenance

The following attestation bundles were made for pysail-0.7.1-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on lakehq/sail

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file pysail-0.7.1-cp38-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for pysail-0.7.1-cp38-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 ab265f8f9e10c44d5c533a071cee0938dc42b291341473b8441b16c727d52917
MD5 d9f156d3c88e2b0b3807906df6758c04
BLAKE2b-256 0373566755afe12f186fefb4aed8560e9ff31d80877e72b1a13b055996f06e02

See more details on using hashes here.

Provenance

The following attestation bundles were made for pysail-0.7.1-cp38-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on lakehq/sail

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file pysail-0.7.1-cp38-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for pysail-0.7.1-cp38-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 2117edb62a08b11321d7442b2b6f55cc785e064a8f89ed72074122ee47442eb6
MD5 c31dc4ab0dad90850b6574d6623819f7
BLAKE2b-256 eaa6006caf84465aa9c0d502b2b81157ed1c03ed6c5816a1f54902144bde1c58

See more details on using hashes here.

Provenance

The following attestation bundles were made for pysail-0.7.1-cp38-abi3-macosx_10_12_x86_64.whl:

Publisher: release.yml on lakehq/sail

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.7.1 This release

6 files

0.7.0

6 files

0.6.6

6 files

0.6.5

6 files

0.6.4

6 files

0.6.3

6 files

0.6.2

6 files

0.6.1

6 files

0.6.0

6 files

0.5.3

6 files

0.5.2

6 files

0.5.1

6 files

0.5.0

6 files

0.4.6

7 files

0.4.5

7 files

0.4.4

7 files

0.4.3

7 files

0.4.2

7 files

0.4.1

6 files

0.4.0

6 files

0.3.7

6 files

0.3.6

6 files

0.3.5

6 files

0.3.4

6 files

0.3.3

6 files

0.3.2

6 files

0.3.1

6 files

0.3.0

6 files

0.2.6

6 files

0.2.5

6 files

0.2.4

6 files

0.2.3

6 files

0.2.2

6 files

0.2.1

6 files

0.2.0

6 files

Pre-release

0.2.0.dev0

6 files

0.1.7

6 files

0.1.6

6 files

0.1.5

6 files

0.1.4

6 files

0.1.3

6 files

0.1.2

6 files

0.1.1

6 files

0.1.0

6 files

Pre-release

0.1.0.dev0

1 file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page