AI-powered Spark job analyzer - NATS worker service
Project description
spark-advisor-analyzer
AI-powered Spark job analyzer — NATS worker service. Part of the spark-advisor ecosystem.
Install
pip install spark-advisor-analyzer
What it does
Combines deterministic rules engine with optional Claude AI analysis to provide actionable Spark configuration recommendations.
- Rules engine — 11 expert rules detecting data skew, GC pressure, disk spill, and more
- AI analysis — Claude API integration for prioritized recommendations with causal chains
- Agent mode — multi-turn Claude tool_use loop for autonomous job exploration
- NATS worker — FastStream subscriber for
analysis.runandanalysis.run.agent
Deployment
As a standalone NATS worker:
export SA_ANALYZER_NATS__URL=nats://localhost:4222
export SA_ANALYZER_AI__API_KEY=sk-or-...
spark-advisor-analyzer
Or use via spark-advisor CLI or spark-advisor-mcp server.
Configuration
| Variable | Default | Description |
|---|---|---|
SA_ANALYZER_NATS__URL |
nats://localhost:4222 |
NATS broker URL |
SA_ANALYZER_AI__ENABLED |
true |
Enable AI analysis |
SA_ANALYZER_AI__API_KEY |
— | LLM API key (required if AI enabled) |
SA_ANALYZER_AI__BASE_URL |
https://openrouter.ai/api/v1 |
LLM API base URL |
SA_ANALYZER_OTEL__ENABLED |
false |
Enable OpenTelemetry distributed tracing |
Links
License
Apache 2.0
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file spark_advisor_analyzer-0.1.19.tar.gz.
File metadata
- Download URL: spark_advisor_analyzer-0.1.19.tar.gz
- Upload date:
- Size: 23.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.11.0 {"installer":{"name":"uv","version":"0.11.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
41b88aa2b019ad4d242e105e2d546582ff0686e73cbfc129b0d0063f8ff24fa7
|
|
| MD5 |
5e57d623bf6d404e75c3dd39b99d82c0
|
|
| BLAKE2b-256 |
899538d2e34c177f4e0f8cd60ab6cc7d05531542c1aac129cc786cb22e13be02
|
File details
Details for the file spark_advisor_analyzer-0.1.19-py3-none-any.whl.
File metadata
- Download URL: spark_advisor_analyzer-0.1.19-py3-none-any.whl
- Upload date:
- Size: 24.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.11.0 {"installer":{"name":"uv","version":"0.11.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
283ed443d73ecea565df4ae4d1ddb7cb4aa5e233089217058422851a0612bcf5
|
|
| MD5 |
0efa9d2281b5b86a5dc3fbea3a811af9
|
|
| BLAKE2b-256 |
3ae1ba3231ec58643927cb648aa0c550ae1adafb3fd4f02b282935d7d172a03a
|
