Spark History Server connector - polls and publishes jobs to NATS
Project description
spark-advisor-hs-connector
Spark History Server connector — fetches job data and publishes to NATS. Part of the spark-advisor ecosystem.
Install
pip install spark-advisor-hs-connector
What it does
Connects to Apache Spark History Server REST API to fetch job metrics, stage details, executor info, and configuration. Operates in two modes:
- On-demand — NATS subscriber for
job.fetchrequests (returnsJobAnalysis) - Background polling — periodically scans History Server for new applications and publishes them to NATS
History Server endpoints used
GET /api/v1/applications/{app-id}— app metadataGET /api/v1/applications/{app-id}/environment— spark.* configGET /api/v1/applications/{app-id}/stages— stage metricsGET /api/v1/applications/{app-id}/stages/{id}/{attempt}/taskSummary— task distributionGET /api/v1/applications/{app-id}/executors— executor metrics
Deployment
export SA_HS_CONNECTOR_HISTORY_SERVER_URL=http://yarn:18080
export SA_HS_CONNECTOR_NATS__URL=nats://localhost:4222
spark-advisor-hs-connector
Configuration
| Variable | Default | Description |
|---|---|---|
SA_HS_CONNECTOR_HISTORY_SERVER_URL |
http://localhost:18080 |
Spark History Server URL |
SA_HS_CONNECTOR_NATS__URL |
nats://localhost:4222 |
NATS broker URL |
SA_HS_CONNECTOR_POLL_INTERVAL_SECONDS |
60 |
Polling interval |
SA_HS_CONNECTOR_BATCH_SIZE |
50 |
Apps per poll cycle |
SA_HS_CONNECTOR_OTEL__ENABLED |
false |
Enable OpenTelemetry distributed tracing |
Links
License
Apache 2.0
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file spark_advisor_hs_connector-0.1.16.tar.gz.
File metadata
- Download URL: spark_advisor_hs_connector-0.1.16.tar.gz
- Upload date:
- Size: 14.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.10.12 {"installer":{"name":"uv","version":"0.10.12","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
4521cebda5a804293530f10733385df6d44b4bcea40127d07fb540127b8123b7
|
|
| MD5 |
0b33e6f94c0d74169fffbb407cbf202e
|
|
| BLAKE2b-256 |
f061ad2f6586ff59aa94958a2b0b4f54d0bfda3c7cfeed9c36f70524a35881ee
|
File details
Details for the file spark_advisor_hs_connector-0.1.16-py3-none-any.whl.
File metadata
- Download URL: spark_advisor_hs_connector-0.1.16-py3-none-any.whl
- Upload date:
- Size: 13.1 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.10.12 {"installer":{"name":"uv","version":"0.10.12","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
b345b311ec8e430640aab8b988f4ffffce7c4555f123134eb2d2c347aeeb2d7b
|
|
| MD5 |
7a8bc31443a1126e2de226a6bf9407da
|
|
| BLAKE2b-256 |
66061224efdb96f2097c7db9d542f656670a2b5005757fd0f80e2f4d0e75b6e2
|
