Provides a Kafka implementation of the ChannelInput/OutputPort in pypz.
Project description
Description
This package contains the Kafka implementation of the Input-/OutputPortPlugin interface of pypz. It enables the operators to send and receive data records in a real streaming fashion.
Check the documentation for more details.
Install
The python artifact is hosted on https://pypi.org/, so you can install it via pip:
pip install pypz-kafka-io
If you want to work on it locally, then you should install in editable mode:
pip install -e ./plugins/kafka_io
Test
You can run tests against an existing Kafka cluster, however, it is more convenient and safe to use a local test cluster.
- Run the confluent image to start a local cluster
docker run --name kafka-test-cluster -d --rm -p 9092:9092 confluentinc/confluent-local
- Install the subproject in editable more.
pip install -e ./plugins/kafka_io
- To run the tests locally, you need to execute the following command:
python -m unittest discover .\plugins\kafka_io\test\ -p "*.py"
- Delete cluster
docker stop kafka-test-cluster
Known issues
Due to the fact that the test execution is sometimes faster than the Kafka cluster, there might be failed tests. Although there are already several delays inserted into the test code, it still can happen. Try to rerun the test and give a notification to the maintainer(s). Thank you.
Build
Before you build, you will need to install the "build" package:
pip install build
Then from the subproject's root, you will need to execute the following command:
python -m build
It will create the source distribution and the wheel file in the "dist" folder.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file pypz_kafka_io-0.9.3.tar.gz
.
File metadata
- Download URL: pypz_kafka_io-0.9.3.tar.gz
- Upload date:
- Size: 12.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.1.0 CPython/3.12.4
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0705ffbb1bab40e76b02361d02c541c810bcd8c424ad6f2032cec95d1a7b31b4 |
|
MD5 | a95a607f7ac03a974a9ad3e2170bf3db |
|
BLAKE2b-256 | 43f0dc114a791af5d0600c72235037802c5458d7e3d0ec10351058380b878474 |
File details
Details for the file pypz_kafka_io-0.9.3-py3-none-any.whl
.
File metadata
- Download URL: pypz_kafka_io-0.9.3-py3-none-any.whl
- Upload date:
- Size: 12.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.1.0 CPython/3.12.4
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | dd6da0c0125a478ddc1e27b2341e10a0eaae8b7608599ab0322a3d412c740ebb |
|
MD5 | 8763a60640f0075fc8152ab5d5246a15 |
|
BLAKE2b-256 | 2ff7a9c7c24de6f4c7f3502b7b149a55b3ac8c26e039c14695906b59a4bd49c7 |