A tool to visualize Apache Spark execution plans.
Project description
Spark Plan Viz ⚡
A lightweight, interactive tool to visualize PySpark execution plans using D3.js. It helps developers and data engineers debug complex queries when the textual result of df.explain() is not enough. It helps identifying bottlenecks (Sorts, Shuffles), and displaying runtime metrics (AQE).
Features
- Interactive Tree: Zoom, pan, and collapse nodes.
- Metric Insights: Click nodes to see runtime metrics (rows output, spill size, etc.).
- Jupyter Integration: Renders directly inside notebooks without external files.
- Zero Dependencies: Only requires Pyspark.
- AQE Support: Visualizes Adaptive Query Execution details.
Installation
pip install spark-plan-viz
Usage
In a Jupyter Notebook
from spark_plan_viz import visualize_plan
# Assuming 'df' is your PySpark DataFrame
visualize_plan(df, notebook=True)
Export to HTML
from spark_plan_viz import visualize_plan
# Generates a standalone HTML file
visualize_plan(df, output_file="my_query_plan.html", open_browser=True)
How to read the chart
- Red nodes: Exchange/Shuffle (Network heavy)
- Purple Nodes: Joins
- Green Nodes: Scans (Data Ingestion)
- Blue Nodes: Aggregations
Example visualization
Here's an example screenshot of the visualization
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file spark_plan_viz-0.2.0.tar.gz.
File metadata
- Download URL: spark_plan_viz-0.2.0.tar.gz
- Upload date:
- Size: 286.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: uv/0.9.11 {"installer":{"name":"uv","version":"0.9.11"},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
98fe1076fe7f3dee1cc044f00703ecaa101111c7b54c2a256ce5c345525c8c4a
|
|
| MD5 |
f01d784e2e0a59e1998f47dafb2fc331
|
|
| BLAKE2b-256 |
25175bba465312331108378e8839c7608b380ede19af763050abef8169234b69
|
File details
Details for the file spark_plan_viz-0.2.0-py3-none-any.whl.
File metadata
- Download URL: spark_plan_viz-0.2.0-py3-none-any.whl
- Upload date:
- Size: 8.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: uv/0.9.11 {"installer":{"name":"uv","version":"0.9.11"},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
aa3748a4630bc6077c8418e8c99f8cb2c001768bd09b3b364a7e0aa9058dce6c
|
|
| MD5 |
8789b72b604f90c0308ca16bc69b40a3
|
|
| BLAKE2b-256 |
f62ceeaae00bc5db4baef081968955691a185721d28e84dab95b8ec27e8314a9
|
