astrocyte-ingestion-kafka 0.8.0

Kafka IngestSource adapter for Astrocyte

astrocyte-ingestion-kafka

Optional IngestSource implementation: Kafka consumer (AIOKafkaConsumer), wired from sources: with type: stream and driver: kafka.

Install

pip install astrocyte-ingestion-kafka

In this monorepo, astrocyte[stream] includes this package (see astrocyte-py/pyproject.toml; package lives under adapters-ingestion-py/).

The package registers kafka under the astrocyte.ingest_stream_drivers entry-point group.

Config sketch

sources:
  events:
    type: stream
    driver: kafka
    url: "localhost:9092"
    topic: "my-topic"
    consumer_group: "astrocyte"
    target_bank: "ingest"

Message values use astrocyte.ingest.payload.parse_ingest_kafka_value (see core docs).

Develop

cd adapters-ingestion-py/astrocyte-ingestion-kafka
uv sync --extra dev
uv run pytest