hi# PySpark MCP Server
Description
PySpark MCP Server is a lightweight server implementation of Model Context Protocol (MCP) for Apache Spark.
The primary purpose of this MCP server is to facilitate query optimization using AI systems. It provides both logical and physical query plans from Spark to AI systems for analysis, along with additional query plan information. Furthermore, the server exposes catalog and table information, enabling data discovery capabilities in data lakes powered by Spark.
Quick Start
Installation
pip install pyspark-mcp
Running the Server
After installation, use the pyspark-mcp command to start the server:
pyspark-mcp --master "local[*]" --host 127.0.0.1 --port 8090
The CLI automatically handles spark-submit configuration. All standard spark-submit options are supported:
# With additional Spark configuration
pyspark-mcp --master "local[*]" --conf spark.driver.memory=4g
# YARN cluster mode
pyspark-mcp --master yarn --deploy-mode client --num-executors 4
# With additional JARs
pyspark-mcp --master "local[*]" --jars /path/to/connector.jar
# Preview the spark-submit command without running
pyspark-mcp --master "local[*]" --dry-run
# With GraphFrames package
pyspark-mcp --master "local[*]" --packages io.graphframes:graphframes-spark3_2.12:0.10.1
CLI Options
| Option | Default | Description |
|---|---|---|
--master |
local[*] |
Spark master URL |
--host |
127.0.0.1 |
MCP server host address |
--port |
8090 |
MCP server port number |
--spark-submit |
spark-submit |
Path to spark-submit executable |
--dry-run |
- | Print command without executing |
All spark-submit options (--conf, --jars, --packages, --executor-memory, etc.) are passed through automatically.
Adding the running MCP to the Claude-code
# Must run one server on a different port per Claude instance
claude mcp add --transport http pyspark-mcp http://127.0.0.1:8090/mcp
Dependencies
- Python >=3.11,<4.0
- fastmcp >= 2.10.6
- loguru
- pyspark >= 3.5
Bundled MCP tools
The following tools are included in the PySpark MCP Server:
| MCP Tool | Description |
|---|---|
| Get the version of PySpark | Get the version number from the current PySpark Session |
| Get Analyzed Plan of the query | Extracts an analyzed logical plan from the provided SQL query |
| Get Optimized Plan of the query | Extracts an optimized logical plan from the provided SQL query |
| Get size estimation for the query results | Extracts a size and units from the query plan explain |
| Get tables from the query plan | Extracts all the tables (relations) from the query plan explain |
| Get the current Spark Catalog | Get the catalog that is the default one for the current SparkSession |
| Check does database exist | Check if the database with a given name exists in the current Catalog |
| Get the current default database | Get the current default database from the default Catalog |
| List all the databases in the current catalog | List all the available databases from the current Catalog |
| List available catalogs | List all the catalogs available in the current SparkSession |
| List tables in the current catalog | List all the available tables in the current Spark Catalog |
| Get a comment of the table | Extract comment of the table or returns an empty string |
| Get table schema | Get the spark schema of the table in the catalog |
| Returns a schema of the result of the SQL query | Run query, get the result, get the schema of the result and return a JSON-value of the schema |
| Read first N lines of the text file | Read the first N lines of the file as a plain text. Useful to determine the format |
2 maintainers
Metadata
2 maintainers
Release files for pyspark-mcp 0.0.6
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| pyspark_mcp-0.0.6.tar.gz | 13.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| pyspark_mcp-0.0.6-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 28.1 kB
Release files / pyspark_mcp-0.0.6.tar.gz
| Download URL | pyspark_mcp-0.0.6.tar.gz |
|---|---|
| Size | 13.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8c282f8af325a4a993284bb2ddfcee81d910a0ca32adfde98abe503bfc0f8a09
|
|
BLAKE2b-256 checksum How to use checksums |
fc2db0fe0479d0a55cdd26b50bae4c334cae15d7c6acea6d720363f87eed436e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 26, 2026.
Transparency logRelease files / pyspark_mcp-0.0.6-py3-none-any.whl
| Download URL | pyspark_mcp-0.0.6-py3-none-any.whl |
|---|---|
| Size | 14.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2a97867d81374b4997a010e5b4ad2cb4dc7c9f503d86be514b7b1f3808bbcd79
|
|
BLAKE2b-256 checksum How to use checksums |
5bf14c96104aa288886e924e1fb7d1781dfd3a6b37b6b11604f9b26fc535281e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 26, 2026.
Transparency log
