Turbine

A high-performance stateful stream processor with a Rust core and a Python interface.

Kafka I/O, JSON/Avro serialization, windowing, state (RocksDB) and checkpointing run in Rust. Your Python handler is called once per Apache Arrow RecordBatch — never per message — so the ergonomics stay Python and the hot path never does.

Documentation · Installation · Quick start · Windowing · Console & API

Install

uv init my-pipeline && cd my-pipeline
uv python pin 3.14t                        # free-threaded CPython is mandatory
uv add --prerelease=allow turbine-fast

The distribution is turbine-fast; the import name is turbine.

Free-threaded Python 3.14 is required. Turbine runs one partition per thread in a single process, which only scales without the GIL — so the published wheels are cp314t exclusively (Linux x86_64, manylinux_2_28). On a standard GIL interpreter there is no compatible wheel and no sdist fallback: the install fails rather than silently giving you a version that cannot scale. Other platforms build from source. Full details, including the source build, are in Installation.

--prerelease=allow is required while Turbine is published as alphas (0.2.0aN).

A first app

from turbine import KafkaBroker, RecordBatch, Turbine, TurbineState

kafka = KafkaBroker(bootstrap="localhost:9092")
app = Turbine(kafka)

@app.subscribe(kafka.topic("events"), output=kafka.topic("counts"))
def aggregate(batch: RecordBatch, state: TurbineState) -> RecordBatch:
    total = state.get("total") or 0
    total += batch.num_rows
    state.put("total", total)
    return batch

app.run()

Point it at any Kafka-compatible broker (Kafka or Redpanda) and run it like any Python script. State is checkpointed automatically and survives restarts; the node serves its REST API and the web console on port 8400 with nothing else to deploy.

What you get

• Rust on the hot path — broker I/O, serde, state and checkpointing. Python sees columnar Arrow batches, so per-message overhead never reaches it. Throughput grows roughly linearly as you add partitions; the absolute numbers depend on your handler.
• Stateful processing with checkpoints — one RocksDB per partition, snapshotted to a local directory or S3-compatible object storage. A restart restores the snapshot and resumes at the right offset.
• Event-time windowing — tumbling, sliding and session windows, persistent or in-memory, driven by a data-derived watermark, with late-data policies and mergeable accumulators (sum/mean/min/max, HLL, t-digest, …).
• Delivery guarantees you choose per subscription — at-least-once by default, opt-in exactly_once built on Kafka transactions, with dead-letter routing for bad payloads and failing handlers.
• Models in the pipeline — declare infer=ml.OnnxModel(...) on a subscription and every batch it publishes is scored in Rust, in the processing thread: no model server, no network hop. Train in PyTorch, scikit-learn, XGBoost or LightGBM, export to ONNX.
• High availability — embedded Raft replicates partition assignments; a node that dies has its partitions redistributed, and rejoins without manual intervention.
• Observability that ships with the package — /health, /ready, /metrics (Prometheus), a state-browsing REST API, and a web console served by every node at /console. Nothing extra to install or deploy.

Status

Alpha. The API is stabilising and may still change between alpha releases; state and internal topic formats can change in a way that requires dropping state and replaying. Published wheels: Linux x86_64, free-threaded CPython 3.14 only — macOS, Windows and arm builds are not published yet and need a source build.

Positioning

Turbine aims to bring Kafka-Streams-level stateful stream processing to the Python ecosystem, with performance that matches or exceeds JVM-based solutions.

• Kafka Streams — powerful, but JVM-only. Turbine targets similar semantics with a Python-native experience.
• Faust — Python stream processing, but slow, unmaintained and thin on tooling.
• FastStream and friends — better for per-message routing, request/reply and multi-broker work. Turbine is for high-volume aggregation, stateful pipelines and cluster-level HA. The trade-off is laid out in the overview.

Python dominates the AI/ML landscape, and the people who need stream processing wired into inference pipelines, feature stores and real-time applications are already there. Turbine gives them a fast, stateful stream processor without leaving their ecosystem.

License

Licensed under the Apache License 2.0. The full text ships in the distribution as LICENSE.