Skip to main content

Python library for building stream processing applications with Apache Kafka

Project description

Quix - React to data, fast

GitHub Version PyPI License Docs
Community Slack YouTube LinkedIn X

Open source Python framework for reliable data engineering

Quix Streams is an end-to-end framework for real-time Python data engineering, operational analytics and machine learning on Apache Kafka data streams. Extract, transform and load data reliably in fewer lines of code using your favourite Python libraries.

Build data pipelines and event-driven microservice architectures leveraging Kafka's low-level scalability, resiliency and durability features in a lightweight library without server-side clusters to manage.

Key Features 💎

Quix Streams provides the following features to make your life easier:

Use Quix Streams to build simple Kafka producer/consumer applications or leverage stream processing to build complex event-driven systems, real-time data pipelines and AI/ML products.

Getting Started 🏄

Install Quix Streams

# PyPI
python -m pip install quixstreams

# or conda
conda install -c conda-forge quixio::quixstreams

Requirements

Python 3.9+, Apache Kafka 0.10+

See requirements.txt for the full list of requirements

Documentation

Quix Streams Docs

Example

Here's an example of how to process data from a Kafka Topic with Quix Streams:

from quixstreams import Application

# A minimal application reading temperature data in Celsius from the Kafka topic,
# converting it to Fahrenheit and producing alerts to another topic.

# Define an application that will connect to Kafka
app = Application(
    broker_address="localhost:9092",  # Kafka broker address
)

# Define the Kafka topics
temperature_topic = app.topic("temperature-celsius", value_deserializer="json")
alerts_topic = app.topic("temperature-alerts", value_serializer="json")

# Create a Streaming DataFrame connected to the input Kafka topic
sdf = app.dataframe(topic=temperature_topic)

# Convert temperature to Fahrenheit by transforming the input message (with an anonymous or user-defined function)
sdf = sdf.apply(lambda value: {"temperature_F": (value["temperature"] * 9/5) + 32})

# Filter values above the threshold
sdf = sdf[sdf["temperature_F"] > 150]

# Produce alerts to the output topic
sdf = sdf.to_topic(alerts_topic)

# Run the streaming application (app automatically tracks the sdf!)
app.run()

Tutorials

To see Quix Streams in action, check out the Quickstart and Tutorials in the docs:

Key Concepts

There are two primary objects:

  • StreamingDataFrame - a predefined declarative pipeline to process and transform incoming messages.
  • Application - to manage the Kafka-related setup, teardown and message lifecycle (consuming, committing). It processes each message with the dataframe you provide for it to run.

Under the hood, the Application will:

  • Consume and deserialize messages.
  • Process them with your StreamingDataFrame.
  • Produce it to the output topic.
  • Automatically checkpoint processed messages and state for resiliency.
  • Scale using Kafka's built-in consumer groups mechanism.

Deployment

You can run Quix Streams pipelines anywhere Python is installed.

Deploy to your own infrastructure or to Quix Cloud on AWS, Azure, GCP or on-premise for a fully managed platform.
You'll get self-service DevOps, CI/CD and monitoring, all built with best in class engineering practices learned from Formula 1 Racing.

Please see the Connecting to Quix Cloud page to learn how to use Quix Streams and Quix Cloud together.

Get Involved 🤝

  • Please use GitHub issues to report bugs and suggest new features.
  • Join the Quix Community on Slack, a vibrant group of Kafka Python developers, data engineers and newcomers to Apache Kafka, who are learning and leveraging Quix Streams for real-time data processing.
  • Watch and subscribe to @QuixStreams on YouTube for code-along tutorials from scratch and interesting community highlights.
  • Follow us on X and LinkedIn where we share our latest tutorials, forthcoming community events and the occasional meme.
  • If you have any questions or feedback - write to us at support@quix.io!

License 📗

Quix Streams is licensed under the Apache 2.0 license.
View a copy of the License file here.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

quixstreams-3.20.0-py3-none-any.whl (347.2 kB view details)

Uploaded Python 3

File details

Details for the file quixstreams-3.20.0-py3-none-any.whl.

File metadata

  • Download URL: quixstreams-3.20.0-py3-none-any.whl
  • Upload date:
  • Size: 347.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.8.10

File hashes

Hashes for quixstreams-3.20.0-py3-none-any.whl
Algorithm Hash digest
SHA256 d06167892a616010cda0b56d2577ed19ac462dacb219f59153b037a5da14c454
MD5 c882cd9b548fd3fac3238e03a0255c5a
BLAKE2b-256 0fc1a48395518b274a1c135f207591a6c5967e9b00e7ba928015d20005092c93

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page