Skip to main content

Interactive visualization of Spark jobs

Project description

spark-board: interactive PySpark dataframes visualization

spark-board provides an interactive way to analize PySpark data frame execution plans as a static website displaying the transformations DAG.

Check out the examples for a quick overview of the features (and the corresponding examples source code here).

If you intend to develop spark-board or run from source, check out the documentation.

Usage

spark-board takes a PySpark data frame and inspects the operations to build the DAG. This usually is the final step of a PySpark script, right before writing it to disk.

Install spark-board

pip install spark-board

Run spark-board

from spark_board.html import dump_dataframe, DefaultSettings

# get the PySpark data frame that will be displayed
df = ...

dump_dataframe(
    df=df,
    output_dir="./spark_board_output",
    overwrite=True,  # overwrite output_dir if it already exists
    default_settings=DefaultSettings(),  # override default settings if desired
)

and that's it! spark-board will generate a static website in the defined output_dir folder. You can now serve the website using any web server and inspect the operations.

You can check out the available default settings here.

Serving

spark-board is intended to be a live documentation of PySpark scripts. Because of this, it's advisable to run it every time the source code is updated. For example, spark-board can be run as part of a CI pipeline and the generated website uploaded to a static website hosting service, like Github or Gitlab pages (we actually do this to update and serve the examples in this repository).

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

spark-board-1.0.0.tar.gz (235.0 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

spark_board-1.0.0-py3-none-any.whl (253.2 kB view details)

Uploaded Python 3

File details

Details for the file spark-board-1.0.0.tar.gz.

File metadata

  • Download URL: spark-board-1.0.0.tar.gz
  • Upload date:
  • Size: 235.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/4.0.2 CPython/3.11.6

File hashes

Hashes for spark-board-1.0.0.tar.gz
Algorithm Hash digest
SHA256 bdd5ae778f6d996793b2c812b852ec6dbcd00a4dfcbd98a54a0feadf9d6a1317
MD5 2d5af756f4903fd21fd2b923132081e4
BLAKE2b-256 56203e022e3be05f4e525fc848cd99265948fb69ae1ddbac6bafc00dba4a859d

See more details on using hashes here.

File details

Details for the file spark_board-1.0.0-py3-none-any.whl.

File metadata

  • Download URL: spark_board-1.0.0-py3-none-any.whl
  • Upload date:
  • Size: 253.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/4.0.2 CPython/3.11.6

File hashes

Hashes for spark_board-1.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 d60f15bb4cb6c6ae413c94496799aee67e5ca2ff5083a62a8e8599107c98e3bb
MD5 193759891bbedfe05651da1b2d8fdeff
BLAKE2b-256 85347cf1bc9d5240a72f76570334f5583e5e515cc366229842332b15cd20e71f

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page