unistream 0.1.1

Library for creating stream data producer / consumer client library.

---

Welcome to unistream Documentation

📔 See Full Documentation HERE.

The library at hand provides a powerful abstraction for data producers and consumers—clients responsible for interfacing with stream processing systems. These clients handle the task of sending data to stream processing systems and pulling data from various streams, including but not limited to Apache Kafka, Apache Pulsar, AWS Kinesis, AWS SQS, and AWS CloudWatch logs.

This library offers a comprehensive set of business-critical features out of the box:

1. Efficient Record Buffering: It intelligently groups records into micro-batches, optimizing network bandwidth utilization for enhanced performance.

2. Data Integrity: To ensure data integrity, the library leverages a local write-ahead log mechanism, mitigating the risk of unexpected errors and data loss during the data transfer process.

3. Automatic Retrying: With an integrated automatic retry mechanism using exponential backoff strategies, transient errors are managed seamlessly, contributing to a robust and reliable data transfer process.

4. Checkpoint Management: The library automatically handles checkpoints, allowing for the storage of consumption progress and processing status for each record. This feature enhances traceability and fault tolerance, especially in critical business use cases.

Install

unistream is released on PyPI, so all you need is to:

$ pip install unistream

To upgrade to latest version:

$ pip install --upgrade unistream