Skip to main content

Provides a Kafka implementation of the ChannelInput/OutputPort in pypz.

Project description

Description

This package contains the Kafka implementation of the Input-/OutputPortPlugin interface of pypz. It enables the operators to send and receive data records in a real streaming fashion.

Check the documentation for more details.

Install

The python artifact is hosted on https://pypi.org/, so you can install it via pip:

pip install pypz-kafka-io

If you want to work on it locally, then you should install in editable mode:

pip install -e ./plugins/kafka_io

Test

You can run tests against an existing Kafka cluster, however, it is more convenient and safe to use a local test cluster.

  1. Run the confluent image to start a local cluster
docker run --name kafka-test-cluster -d --rm -p 9092:9092 confluentinc/confluent-local
  1. Install the subproject in editable more.
pip install -e ./plugins/kafka_io
  1. To run the tests locally, you need to execute the following command:
python -m unittest discover .\plugins\kafka_io\test\ -p "*.py"
  1. Delete cluster
docker stop kafka-test-cluster

Known issues

Due to the fact that the test execution is sometimes faster than the Kafka cluster, there might be failed tests. Although there are already several delays inserted into the test code, it still can happen. Try to rerun the test and give a notification to the maintainer(s). Thank you.

Build

Before you build, you will need to install the "build" package:

pip install build

Then from the subproject's root, you will need to execute the following command:

python -m build

It will create the source distribution and the wheel file in the "dist" folder.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pypz_kafka_io-0.9.3.tar.gz (12.8 kB view details)

Uploaded Source

Built Distribution

pypz_kafka_io-0.9.3-py3-none-any.whl (12.8 kB view details)

Uploaded Python 3

File details

Details for the file pypz_kafka_io-0.9.3.tar.gz.

File metadata

  • Download URL: pypz_kafka_io-0.9.3.tar.gz
  • Upload date:
  • Size: 12.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.0 CPython/3.12.4

File hashes

Hashes for pypz_kafka_io-0.9.3.tar.gz
Algorithm Hash digest
SHA256 0705ffbb1bab40e76b02361d02c541c810bcd8c424ad6f2032cec95d1a7b31b4
MD5 a95a607f7ac03a974a9ad3e2170bf3db
BLAKE2b-256 43f0dc114a791af5d0600c72235037802c5458d7e3d0ec10351058380b878474

See more details on using hashes here.

File details

Details for the file pypz_kafka_io-0.9.3-py3-none-any.whl.

File metadata

File hashes

Hashes for pypz_kafka_io-0.9.3-py3-none-any.whl
Algorithm Hash digest
SHA256 dd6da0c0125a478ddc1e27b2341e10a0eaae8b7608599ab0322a3d412c740ebb
MD5 8763a60640f0075fc8152ab5d5246a15
BLAKE2b-256 2ff7a9c7c24de6f4c7f3502b7b149a55b3ac8c26e039c14695906b59a4bd49c7

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page