arrakis-backend-kafka
Kafka backend for Arrakis server
Kafka backend for the Arrakis server, part of the LIGO next-generation data delivery (NGDD) project. Serves live, low-latency detector data over Apache Arrow Flight from timeseries published to Kafka topics, along with the recent past held in Kafka's retention window, and manages the publishers that write to those topics.
Resources
Installation
With pip:
pip install arrakis-backend-kafka
With Docker:
docker run --net=host -it containers.ligo.org/ngdd/backends/arrakis-backend-kafka:main
Features
- Serves live timeseries from Kafka topics over Apache Arrow Flight, with historical requests answered from Kafka's retention window
- Contiguous transition from historical to live data when a request spans the present
- Pooled Kafka consumers, so many clients subscribed to the same channels share a single set of readers and one polling thread
- Fast-forwards to the head of the stream when a subscription falls behind, logging the range that was skipped
- Publisher management: partition assignment, topic creation and channel
registration for publishers defined in TOML configuration, reloaded on
SIGHUP - Channel metadata kept in compacted Kafka topics and synchronized from remote publishers, so replicas serve a consistent channel table
- Gap-aware responses that return masked data for missing samples
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file arrakis_backend_kafka-0.14.0.tar.gz.
File metadata
- Download URL: arrakis_backend_kafka-0.14.0.tar.gz
- Upload date:
- Size: 80.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
Hatch/1.16.5 cpython/3.13.12 HTTPX/0.28.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
f6556196662fce17bc3d2f9300b1a6202ba3ae91907ed6e3917ce815a84a2f1f
|
|
| MD5 |
c2e1fa952eab9fbc09a0b775935a9345
|
|
| BLAKE2b-256 |
0d70ec8387815c07be1b6f223dc11b9375e296b090e38883b426f7c26a99a3b9
|
File details
Details for the file arrakis_backend_kafka-0.14.0-py3-none-any.whl.
File metadata
- Download URL: arrakis_backend_kafka-0.14.0-py3-none-any.whl
- Upload date:
- Size: 85.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
Hatch/1.16.5 cpython/3.13.12 HTTPX/0.28.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
165a3ad10309238655c1e82e29379e8729ee1bbd5480b39644703bfed7bb41a6
|
|
| MD5 |
ce394e5a88069366719139fe2c46760d
|
|
| BLAKE2b-256 |
1492196c6ac7690ead916b8a6780fd35ffc3ee30f660f4eb6ceb88dcd4ea1e6e
|
