spark-logical-plan-capture 0.2.0

Python wrapper for Spark logical plan capture extension

spark-logical-plan-capture

Python package for connecting the Spark extension io.github.mt.logicplan.LogicalPlanCaptureExtension via spark-submit or from PySpark code.

Install

pip install spark-logical-plan-capture

Usage in PySpark

from pyspark.sql import SparkSession
from spark_logical_plan_capture import get_spark_conf, get_jar_path

conf = get_spark_conf()

spark = (
    SparkSession.builder
    .config("spark.sql.extensions", conf["spark.sql.extensions"])
    .config("spark.jars", conf["spark.jars"])
    .getOrCreate()
)

Usage with spark-submit

spark-submit \
  --conf "spark.sql.extensions=io.github.mt.logicplan.LogicalPlanCaptureExtension" \
  --jars "$(python -c 'from spark_logical_plan_capture import get_jar_path; print(get_jar_path())')" \
  your_job.py

Current package line is built for Spark 3.5.2, Scala 2.13.8, JVM 11.0.25.