jupyter-storage-explorer-pyspark-support

PySpark support for Storage Explorer, the Jupyter Storage Explorer plugin for PySpark as an engine: the data browser on Spark DataFrames of Delta Lake tables, Parquet, CSV and JSON Lines, and code loading them.

Installation

pip install "jupyter-storage-explorer[pyspark]"          # the plugin and its libraries
pip install "jupyter-storage-explorer[pyspark-support]"  # the plugin only, with the libraries you have

The libraries are not the plugin's dependencies, so that installing it never changes those of a shared environment. The first command installs them, as does jupyter-storage-explorer-pyspark-support[full]:

Without delta-spark, the other formats still work. For Databricks Connect, use databricks-support instead: with Databricks Connect installed, this plugin's engine is not offered.

Prerequisites

• The plugin expects Spark to reach your storage already, e.g. through spark-defaults.conf or a Databricks cluster's notebook.
• session_options in Engine options can add Spark properties. With a session already running, only those Spark can change at run time apply.

Plugin options

plugin_options gives the plugin's options:

StorageExplorer(
    "abfs://lake@myaccount.dfs.core.windows.net/raw",
    plugin_options={"pyspark-support": {"priority": 10}},
)

In the launcher's tabs, plugin_options() in launcher.py gives them:

def plugin_options():
    return {"pyspark-support": {"enabled": False}}

Engine options

The supports' options, in engine_options["pyspark"] (by format, or "*" for all), configure the Spark session and the reader, for the data browser and the code:

StorageExplorer(
    "abfs://lake@myaccount.dfs.core.windows.net/raw",
    engine_options={
        "pyspark": {
            "*": {
                "session_options": {
                    "spark.sql.extensions": "io.delta.sql.DeltaSparkSessionExtension",
                    "spark.sql.catalog.spark_catalog": "org.apache.spark.sql.delta.catalog.DeltaCatalog",
                },
            },
            "csv": {"null_value": "NA", "mode": "DROPMALFORMED"},
            "delta": {"version_as_of": 3},
        },
    },
)

Every format takes:

The other options are each format's reader options, from Spark's and Delta Lake's documentation, in snake case. Those left unset (None) are left to Spark, whose defaults are shown. Options that look like credentials are left out of the generated code.

CSV (csv)

From Spark's CSV Files.

JSON Lines (json-lines)

From Spark's JSON Files.

Parquet (parquet)

From Spark's Parquet Files.

Delta Lake (delta)

From Delta Lake's DataFrameReader options: they load the table as of a version, a snapshot that later changes to the table do not affect.