airflow-provider-quanton-spark-k8s 0.0.2

Airflow provider for Onehouse QuantonSparkApplication Kubernetes CRDs

airflow-provider-quanton-spark-k8s

This airflow provider will make it easy to run a quanton spark job on kubernetes using airflow.

Requirements

Apache Airflow >= 2.7.0 Python >= 3.10

Installation

pip install airflow-provider-quanton-spark-k8s

Usage

Basic Example DAG

from datetime import datetime

from airflow import DAG
from quanton_provider.operators.quanton_spark_application import (
    QuantonSparkApplicationOperator,
)

with DAG(
    dag_id="test_quanton_spark_pi",
    start_date=datetime(2026, 3, 1),
    schedule=None,
    catchup=False,
    tags=["quanton", "spark", "test"],
) as dag:
    run_spark_pi = QuantonSparkApplicationOperator(
        task_id="run_spark_pi",
        name="quanton-spark-pi-java-example",
        namespace="spark-operator",
        timeout=600,
        delete_on_success=True,
        spark_application_spec={
            "type": "Java",
            "mode": "cluster",
            "image": "apache/spark:3.5.0",
            "imagePullPolicy": "IfNotPresent",
            "mainClass": "org.apache.spark.examples.JavaSparkPi",
            "mainApplicationFile": "https://repo1.maven.org/maven2/org/apache/spark/spark-examples_2.10/1.1.1/spark-examples_2.10-1.1.1.jar",
            "arguments": ["100"],
            "sparkVersion": "3.5.0",
            "restartPolicy": {
                "type": "Never",
            },
            "driver": {
                "cores": 1,
                "coreLimit": "1200m",
                "memory": "1024m",
                "labels": {"version": "3.5.0"},
                "serviceAccount": "spark-service-account",
            },
            "executor": {
                "cores": 1,
                "instances": 2,
                "memory": "512m",
                "labels": {"version": "3.5.0"},
            },
        },
    )

Testing

Install the package with test dependencies:

pip install -e .
pip install -r requirements-dev.txt

Run the test suite:

pytest

Run with coverage report:

pytest --cov=quanton_provider --cov-report=term-missing

The project enforces a minimum of 95% code coverage.