spark-advisor-cli
AI-powered Apache Spark job analyzer and configuration advisor. Standalone CLI tool.
Stop guessing Spark configs. Let data and AI tell you what's wrong.
Install
pip install spark-advisor-cli
Quick Start
# Analyze from event log file (rules-only, free)
spark-advisor analyze /path/to/event-log.json.gz --no-ai
# Analyze with AI recommendations
export SA_ANALYZER_AI__API_KEY=sk-or-...
spark-advisor analyze /path/to/event-log.json.gz
# Analyze from History Server
spark-advisor analyze app-20250101120000-0001 -hs http://yarn:18080
# Agent mode (multi-turn AI analysis)
spark-advisor analyze /path/to/event-log.json.gz --agent
Commands
analyze — analyze a Spark job
# From event log file
spark-advisor analyze /path/to/event-log.json.gz
# From History Server
spark-advisor analyze app-20250101120000-0001 -hs http://yarn:18080
# With AI analysis (default if SA_ANALYZER_AI__API_KEY is set)
spark-advisor analyze /path/to/event-log.json.gz
# Without AI (rules only)
spark-advisor analyze /path/to/event-log.json.gz --no-ai
# Agent mode (multi-turn AI with tool use)
spark-advisor analyze /path/to/event-log.json.gz --agent
# Verbose mode (per-stage breakdown)
spark-advisor analyze /path/to/event-log.json.gz --verbose
# JSON output
spark-advisor analyze /path/to/event-log.json.gz --format json
# Save suggested config to file
spark-advisor analyze /path/to/event-log.json.gz -o spark-defaults.conf
# Use specific LLM model
spark-advisor analyze /path/to/event-log.json.gz --model qwen/qwen3-coder:free
| Flag | Short | Default | Description |
|---|---|---|---|
source |
required | App ID (with -hs) or path to event log file |
|
--history-server |
-hs |
None |
Spark History Server URL |
--no-ai |
False |
Disable AI analysis (rules only) | |
--agent |
False |
Use agent mode (multi-turn AI with tool use) | |
--model |
-m |
qwen/qwen3-coder:free |
LLM model for AI analysis |
--output |
-o |
None |
Write suggested config to file |
--format |
-f |
text |
Output format: text or json |
--verbose |
-v |
False |
Show per-stage breakdown |
scan — list recent jobs from History Server
spark-advisor scan -hs http://yarn:18080 --limit 20
version
spark-advisor version
# spark-advisor v0.1.10
What it detects
11 deterministic rules: data skew, disk spill, GC pressure, shuffle partitions, executor idle, task failures, small files, broadcast join threshold, serializer choice, dynamic allocation, memory overhead.
All thresholds are configurable via Thresholds model.
See also
- Full documentation and architecture
- MCP Server setup (Claude Desktop / Cursor)
- Rules engine
- Analyzer
- Contributing
License
Apache 2.0
1 maintainer
Metadata
1 maintainer
Release files for spark-advisor-cli 0.1.21
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| spark_advisor_cli-0.1.21.tar.gz | 12.7 kB | Without provenance | Details for spark_advisor_cli-0.1.21.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| spark_advisor_cli-0.1.21-py3-none-any.whl | Python 3 | none | any | Details for spark_advisor_cli-0.1.21-py3-none-any.whl |
Total release size: 22.1 kB
Release files / spark_advisor_cli-0.1.21.tar.gz
| Download URL | spark_advisor_cli-0.1.21.tar.gz |
|---|---|
| Size | 12.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1b742760a4f1a59d045e7e3eb755f17df506564b0eb6afca4ec1f82e2300022e
|
|
BLAKE2b-256 checksum How to use checksums |
5530083bd3b8b3ea20bf4481dd6a83a5600a5bd99b720ca87f609b021b31f598
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.2 {"installer":{"name":"uv","version":"0.11.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / spark_advisor_cli-0.1.21-py3-none-any.whl
| Download URL | spark_advisor_cli-0.1.21-py3-none-any.whl |
|---|---|
| Size | 9.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8b38bc3e72eaf441c44551e52814baea426f84f3c563ee233e42c9e1fdcef849
|
|
BLAKE2b-256 checksum How to use checksums |
5484a6211ee7f33a6c1da956a8d9d1261f663c31bedcc5458908b7314f109ad1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.2 {"installer":{"name":"uv","version":"0.11.2","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for spark-advisor-cli 0.1.21
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
