Spark Rust — multi-environment alpha

The Python distribution is spark-rust; import it as sparkrust. The alpha combines an embedded CPU SQL engine with compiled cluster runtimes, deployment assets, documentation, examples, browser WASM and language-package archives. It does not include the Rust engine source, install Apache Spark, or claim complete PySpark compatibility.

Install and check

The release candidate is 0.42.1a1, with engine version 0.42.1. Publication status is recorded in the release receipt; a local candidate is not proof of a public PyPI upload. Once published:

python3 -m pip install 'spark-rust==0.42.1a1'
sparkrust version
sparkrust doctor
sparkrust sql 'SELECT 40 + 2 AS answer'

For a locally supplied release wheelhouse instead:

python3 -m pip install --find-links /path/to/wheels 'spark-rust==0.42.1a1'

The main wheel requires exact-version companion wheels automatically. Native alpha scope is Linux x86_64, glibc >= 2.36, such as Debian 12 or Ubuntu 24.04. Other operating systems/architectures and GPU execution are not covered. The CPython extension uses the stable ABI from Python 3.9; consult release acceptance results for the interpreter versions actually tested. Python dependencies are installed by pip; the pandas extra is available with spark-rust[pandas].

In-process Python

import sparkrust

con = sparkrust.connect()
assert con.sql("SELECT 40 + 2 AS answer").collect() == [(42,)]
empty = con.sql("SELECT CAST(NULL AS INT) AS id WHERE false").arrow()
assert empty.num_rows == 0 and empty.column_names == ["id"]
con.close()

Use .arrow() for Arrow, .collect() for tuples and .toPandas() for pandas. sparkrust demo and sparkrust tour use bundled sample data.

Local server and native tools

sparkrust runtime list
sparkrust serve --port 15002 --ui-port 4040
# Run in a separate shell when desired:
sparkrust runtime run spark-sql -- --help

serve resolves the packaged matching Connect server; no Rust build is needed. The server's bind/security defaults are not a production exposure policy. Use private networks, authentication and TLS as described in the cluster guide. The native spark-sql shell is local, not a remote Connect client.

VM clusters and AKS

Export the installed binaries, installers, chart, documentation and notices:

sparkrust runtime export ./spark-rust-cluster-alpha
cd spark-rust-cluster-alpha
sha256sum -c SHA256SUMS
# Safe unprivileged installation check; does not create users or start services:
DESTDIR="$PWD/staged" bash vm/install.sh

For actual VMs, copy this verified bundle to each node and follow README.md for prerequisites, roles, addresses, shared storage, authentication, installation, health checks, upgrade and rollback. Installation and service startup remain explicit operator actions. The alpha installs under /opt/spark-rust/0.42.1a1.

The export also includes a runtime-only Dockerfile and Helm chart:

docker build --platform linux/amd64 -f Dockerfile -t spark-rust-cluster:v0.42.1a1 .
helm lint ./helm -f ./helm/values-aks.yaml

This builds locally. Push/import into your authorized registry separately, obtain its real image digest, and follow the AKS guide with ./helm as the chart path. No public container image is implied by PyPI publication. Real AKS networking, identity, storage and remote-query acceptance must be validated in your target cluster. pip install does not create Azure resources or deploy workloads.

Browser WASM

sparkrust wasm serve --demo bi --port 8000

Open http://127.0.0.1:8000/. The compiled engine and JavaScript are packaged; no Rust/wasm-bindgen build is needed. Other demos are browser, geospatial, mission-control, ml-classification and notebook-explorer. Some interfaces or optional language modes use external CDN/map/Pyodide dependencies and require network access; this is not a claim of a fully offline browser site.

Documentation, examples and language clients

sparkrust resources list
sparkrust resources path assets/docs/install-cluster.md
sparkrust resources export ./spark-rust-resources

Exports refuse to overwrite an existing directory. Documentation and example source live under assets/; Node, Ruby, TypeScript Connect, JVM/Scala, SparkR and WASM SDK archives live under clients/. Install archives with the corresponding language package manager. Node requires Node 20+, the native Ruby build targets Ruby 3.1 on Linux x86_64, and JVM/Scala examples require Java 17. SparkR also needs R and its JVM bridge. Notebook examples require their documented language kernels, data and service prerequisites; pip does not install those external runtimes.

These archives have their own API/engine versions recorded in the release manifest; their filenames do not imply independent registry publication. Small Rust example files and notebooks are teaching material, not engine source.

Release safety

Each wheel has a RECORD, license text and provenance; the release directory contains exact artifact hashes and acceptance evidence. Do not substitute old artifacts or retag an incompatible native binary. No pip installation step starts services, modifies kubeconfig, runs sudo, deploys Helm, or publishes a registry artifact. See the bundled release/cluster guides for limitations and security.