Spark On Kubernetes

Spark on Kubernetes is a python package that makes it easy to submit and manage spark apps on Kubernetes. It provides a Python client that can be used to submit apps in your API or scheduler of choice, and a CLI that can be used to submit apps from the command line, instead of using spark-submit.

It also provides an optional REST API with a web UI that can be used to list and manage apps, and access the spark UI through the reverse proxy. The optional Airflow 3.4+ task tab reuses the same React console components inside Airflow; see the Airflow guide.

Choose your route in the start-here guide and check the support boundaries before adopting a new integration. The release notes describe every published release and the release in preparation. Report vulnerabilities privately as described in the security policy.

Installation

Published releases

The following commands install a published package from PyPI. Choose the extra for the functionality you need, inside a virtual environment:

python -m pip install spark-on-k8s # Python client, CLI and helpers
python -m pip install "spark-on-k8s[sql]" # Also submit SQL files without Airflow
python -m pip install "spark-on-k8s[api]" # Also install the REST API and web UI
python -m pip install "spark-on-k8s[airflow]" # Also install the Airflow integration

Published releases can have different Python requirements and features from the unreleased main branch. Check the selected release's metadata on PyPI and its version of the documentation. The source requirements and examples below describe main; installing from PyPI does not install unreleased changes.

Current source

The current source requires Python 3.11 or newer. The core client and API are tested on Python 3.11–3.15. The Airflow integration is tested with Airflow 3.4 on Python 3.11–3.14 and Airflow 2.11.2 on Python 3.11–3.12. Airflow 2 is end-of-life and retains dependency versions with known security advisories; use Airflow 3 for new deployments. The optional airflow-ui extra adds a native task tab for Airflow 3.4 or newer, below Airflow 4. The airflow and airflow-ui extras are not supported on Python 3.15.

Clone the repository to use the source instructions and example files in this guide:

git clone https://github.com/hussein-awala/spark-on-k8s.git
cd spark-on-k8s

With uv, choose one environment profile. These commands use the committed lockfile and create a project-local .venv:

uv sync --locked --no-default-groups # Core package
uv sync --locked --no-default-groups --extra api # API and web UI
uv sync --python 3.12 --locked --no-default-groups --extra airflow # Airflow 3 integration
uv sync --python 3.12 --locked --no-default-groups --extra airflow-ui # Also install the native task tab

Run commands from the selected environment with uv run --no-sync spark-on-k8s, or activate it with source .venv/bin/activate before using the commands below. To install the API and Airflow integration together, select --extra api --extra airflow, or --extra api --extra airflow-ui to include the task tab. A later uv sync selects a new profile and can remove unselected extras.

Alternatively, install from the checkout with pip in an activated virtual environment:

python -m pip install . # Core package
python -m pip install ".[api]" # API and web UI
python -m pip install ".[airflow]" # Airflow integration
python -m pip install ".[airflow-ui]" # Airflow 3.4+ integration and native task tab

See CONTRIBUTING.md for development dependencies and the separate Airflow 2.11 compatibility environment. See the Airflow guide for provider installation, connections and DAG examples.

Usage

Set up the Kubernetes namespace

The client creates the Spark driver pod. The driver uses its Kubernetes service account to create and manage executor pods. The following examples use namespace spark and service account spark; the submitting client also needs permission to create the application resources in that namespace.

The setup helper creates the namespace if needed, a service account named spark, and a ClusterRoleBinding to the built-in edit ClusterRole. This grants that service account edit access across all namespaces. It is broader than a namespace-scoped Spark driver role. On a shared cluster, ask the cluster administrator to provision a suitably scoped service account instead.

Running the helper requires credentials that can list namespaces, service accounts and ClusterRoleBindings, create missing resources, and bind the edit role. Use it only when that cluster-wide grant is intended. See the Kubernetes RBAC documentation for the difference between RoleBindings and ClusterRoleBindings.

With Python:

from spark_on_k8s.utils.setup_namespace import SparkOnK8SNamespaceSetup

namespace_setup = SparkOnK8SNamespaceSetup()
namespace_setup.setup_namespace(namespace="spark")

With the CLI:

spark-on-k8s namespace setup -n spark

Python client

Replace the image and application path with an image accessible to your cluster and an application file included in that image:

from spark_on_k8s.client import SparkOnK8S

client = SparkOnK8S()
driver_pod_name = client.submit_app(
    image="my-registry/my-image:latest",
    app_path="local:///opt/spark/work-dir/my-app.py",
    app_arguments=["arg1", "arg2"],
    app_name="my-app",
    namespace="spark",
    service_account="spark",
    app_waiter="log",
    image_pull_policy="IfNotPresent",
    ui_reverse_proxy=True,
)

submit_app returns the driver pod name. The application ID is generated from the application name with a timestamp suffix and is available in the application list. For a private registry, configure image-pull credentials for the pods in your cluster.

CLI

The CLI can submit applications and manage applications created by either the CLI or the Python client. Submit an application and stream its logs:

spark-on-k8s app submit \
  --image my-registry/my-image:latest \
  --path local:///opt/spark/work-dir/my-app.py \
  --namespace spark \
  --service-account spark \
  --name my-app \
  --image-pull-policy IfNotPresent \
  --ui-reverse-proxy \
  --logs \
  param1 param2

List applications to find their generated IDs:

spark-on-k8s apps list -n spark

To kill a running application, copy its actual ID from that output. The following ID is illustrative; my-app alone is the application name, not the generated ID:

APP_ID="my-app-20261107120000" # Replace with the ID from the application list
spark-on-k8s app kill -n spark --app-id "$APP_ID"

Check the help for further commands and options:

spark-on-k8s --help
spark-on-k8s app --help
spark-on-k8s apps --help

Use --output json for versioned submission receipts and machine-readable list/status results. Waiting commands report failed workloads with nonzero exit codes; --timeout bounds observation. --max-runtime SECONDS instead makes Kubernetes stop the driver, and apps prune reports (or, with --delete, deletes) finished drivers left in a namespace. See the automation guide for exit codes and Python run handles.

REST API

Install the api extra before starting the REST API and web UI. The API serves only its current namespace unless SPARK_ON_K8S_API_NAMESPACES names others; outside a cluster, that is the namespace of the kubeconfig context. For local access to the spark namespace:

export SPARK_ON_K8S_API_NAMESPACES=spark
spark-on-k8s api start \
    --host "127.0.0.1" \
    --port 8080 \
    --workers 4 \
    --log-level error \
    --limit-concurrency 100

List applications in the spark namespace:

curl 'http://127.0.0.1:8080/apps/list_apps/spark' \
  -H 'accept: application/json'

The console provides a searchable application list, status filters, namespace selection, application details, and confirmed stop/delete actions. It refreshes in the background without resetting your filters. Counts and filters describe the applications loaded so far; use Load more when additional pages are available. Completed applications from the configured history server appear as archive entries without live-driver actions.

The driver-log view supports search, copying, download, a configurable tail, and pausing auto-scroll. It keeps the most recent 10,000 lines in the browser. Connection failures are shown separately from log text, and a history-server outage leaves live applications available. JavaScript and CSS are packaged with the Python distribution; no CDN is needed.

Open http://127.0.0.1:8080/webserver/apps?namespace=spark and use Open Spark UI for an application submitted with UI reverse proxy enabled. Its direct URL is http://127.0.0.1:8080/webserver/ui/spark/APP_ID; replace APP_ID with the actual application ID. The API process must be able to reach the driver's service.

Use --host 0.0.0.0 to listen on all network interfaces when deploying behind an appropriate service or proxy. Client URLs should use the server's reachable host or address, not the wildcard bind address 0.0.0.0.

When a gateway serves the API below a path such as /platform/spark and removes that prefix before forwarding, pass --root-path /platform/spark or set SPARK_ON_K8S_API_ROOT_PATH=/platform/spark. See the mounted path guidance.

Deploy the API with Helm

The chart deploys the API, its Service and RBAC resources. Ingress and the optional Spark History Server are controlled by chart values. The chart's env list sets environment variables for the API and its Kubernetes client.

The published chart is released independently from this source checkout. Inspect the selected published chart's values before deploying it; unreleased chart fixes and defaults described below may not be present:

helm repo add spark-on-k8s https://hussein.awala.fr/spark-on-k8s-chart
helm repo update
helm show values spark-on-k8s/spark-on-k8s
helm install spark-on-k8s-release spark-on-k8s/spark-on-k8s \
  --values examples/helm/values.yaml

To use the chart from this checkout, render it for review first, then install that local chart. Set API_IMAGE_TAG to a compatible, tested image from the current source; the old 0.12.1 API does not implement the readiness endpoint:

helm template spark-on-k8s-release ./chart --values examples/helm/values.yaml \
  --set-string image.tag="${API_IMAGE_TAG:?Set a qualified API image tag}"
helm install spark-on-k8s-release ./chart --values examples/helm/values.yaml \
  --set-string image.tag="$API_IMAGE_TAG"

Choose one chart source for a release. The commands above use example values from this checkout; adapt them to your cluster and selected chart version. The source chart defaults to readOnly: true, namespaces: "" (the release namespace), a ClusterIP Service on port 8000, and disabled ingress. Set namespaces to a comma-separated string such as "spark" or "spark,analytics" to serve those namespaces, or to "*" to serve every namespace through a chart-owned cluster role with the same permissions. The chart-created Kubernetes permissions follow the same value, and the API refuses requests, Spark UI proxy connections and log streams for other namespaces. Enable readOnly: false only if the API should be able to kill or delete applications; with "*", that also requires rbac.allowClusterWideWrite: true. The separate web UI button settings control visibility, not Kubernetes authorization. The API has no built-in user authentication; its namespace scope limits what it can reach, not who can use it. Expose the API through access-controlled networking or an authenticating proxy.

The source chart supplies in-cluster Kubernetes configuration and the API bind address and port. It requires an explicit API image tag or digest; a published chart defaults to its appVersion tag. Using a local chart does not build or install the Python source into an image. Set image.repository and image.digest to a verified production image digest (image.tag remains available for named releases), independently of Spark application images passed to submit_app. Digest selection takes precedence over the tag. See chart deployment guidance for scoped permissions, externally owned identities and release qualification.

Configuration

Set environment variables before starting the CLI/API process or importing the Python client: configuration values are read at module import time. The table below lists defaults in the current source, before chart or caller overrides. unset means no environment default (None); {}, [] and an empty string are distinct empty defaults. Use lowercase true or false for boolean settings.

Most client defaults also apply to the CLI. The six node-selector, label and annotation variables marked CLI are CLI defaults; Python callers should pass the corresponding driver_* or executor_* dictionaries to submit_app. If all three executor-count variables are unset, applications use two executors with dynamic allocation disabled. The two executor-template settings are mutually exclusive, including when supplied through environment variables.

JSON settings use JSON syntax, including double-quoted keys and strings. Packages and existing Secret names use comma-separated strings instead of JSON arrays:

export SPARK_ON_K8S_SPARK_CONF='{"spark.sql.shuffle.partitions": "8"}'
export SPARK_ON_K8S_APP_ARGUMENTS='["--input", "s3a://example/input"]'
export SPARK_ON_K8S_DRIVER_ENV_VARS_FROM_SECRET='spark-storage,spark-catalog'
export SPARK_ON_K8S_API_WEBSERVER_APPS_REFRESH_INTERVAL='10'

The named Secrets must exist in the application's namespace. Refresh values such as 10s are invalid because the setting is parsed as an integer.

Examples

Run the following commands from a source checkout. The examples use namespace spark and service account spark, which must already be configured as described above. See the examples guide for the application code, driver configuration and executor templates, and the Airflow guide for DAG examples. The ETL and Iceberg starters include real data checks and portable job definitions for shared-storage deployments. SQL jobs can be submitted from the CLI or Python without Airflow. Spark compatibility describes the Spark 3.5 and Spark 4 runtime matrix, image build arguments, and execution tests.

Python

Build the image from the repository root:

docker build -t pyspark-job examples/python

For a local cluster, load the image into one of the following cluster runtimes:

# Minikube
minikube image load pyspark-job
# Or kind
kind load docker-image pyspark-job

The supplied submission scripts use image_pull_policy="Never" (CLI: --image-pull-policy Never) for these preloaded images. For a remote cluster, replace the example registry below, push the tagged image, and update the script's image to that same name and its pull policy to IfNotPresent:

docker tag pyspark-job registry.example.com/team/pyspark-job:example
docker push registry.example.com/team/pyspark-job:example

Submit with the Python client or the CLI script:

python examples/python/submit.py
# Or
bash examples/python/submit.sh

Java

Build the image from the repository root:

docker build -t java-spark-job examples/java

For a local cluster, load the image into one of the following cluster runtimes:

# Minikube
minikube image load java-spark-job
# Or kind
kind load docker-image java-spark-job

The supplied submission scripts use image_pull_policy="Never" (CLI: --image-pull-policy Never) for these preloaded images. For a remote cluster, replace the example registry below, push the tagged image, and update the script's image to that same name and its pull policy to IfNotPresent:

docker tag java-spark-job registry.example.com/team/java-spark-job:example
docker push registry.example.com/team/java-spark-job:example

Submit with the Python client or the CLI script:

python examples/java/submit.py
# Or
bash examples/java/submit.sh

What's next

See the TODO list for future work. Contributions are welcome!

The governance policy describes maintainer roles and how contributors grow into them. If you use spark-on-k8s, add your organization to ADOPTERS.md.