Turbine

A high-performance stateful stream processor with a Rust core and a Python interface.

Vision

Turbine aims to bring Kafka Streams-level stateful stream processing to the Python ecosystem, with performance that matches or exceeds JVM-based solutions. By writing the core engine in Rust and exposing a Python SDK, Turbine bridges the gap between raw throughput and the ergonomics that AI/ML practitioners expect.

Goals

• Rust core — Low-latency, high-throughput stream processing engine with stateful operators (joins, aggregations, windowing).
• Python SDK — Idiomatic Python API so users can define topologies, transformations, and integrate with AI/ML tooling (LLMs, inference pipelines, etc.).
• Queryable state — Every node in the processing topology exposes its current state via an API. No more black-box pipelines.
• Built-in observability — Console UI and REST API to inspect the state of any node in real time, monitor throughput, lag, and topology health.
• Kafka-compatible — Consumes from and produces to Kafka and Kafka-compatible brokers (Redpanda).

Positioning

Turbine competes with:

• Kafka Streams — Powerful but JVM-only. Turbine targets similar semantics with better performance and a Python-native experience.
• Faust — Python stream processing, but slow, unmaintained, and limited tooling. Turbine offers Rust-level performance with superior observability and stability.

Why Python?

Python dominates the AI/ML landscape. Data scientists and ML engineers need to wire stream processing into inference pipelines, feature stores, and real-time AI applications. Turbine gives them a fast, stateful stream processor without leaving their ecosystem.