Python stream processing
Project description
Beavers
Documentation / Installation / Repository / PyPI
Beavers is a python library for stream processing, optimized for analytics.
It is used at Tradewell Technologies, to calculate analytics and serve model predictions, for both realtime and batch jobs.
Key Features
- Works in real time (eg: reading from Kafka) and replay mode (eg: reading from Parquet files).
- Optimized for analytics, using micro-batches (instead of processing records one by one).
- Similar to incremental, it updates nodes in a dag incrementally.
- Taking inspiration from kafka streams, there are two types of nodes in the dag:
- Stream: ephemeral micro-batches of events (cleared after every cycle).
- State: durable state derived from streams.
- Clear separation between the business logic and the IO. So the same dag can be used in real time mode, replay mode or can be easily tested.
- Functional interface: no inheritance or decorator required.
- Support for complicated joins, not just "linear" data flow.
Limitations
- No concurrency support. To speed up calculation use libraries like pandas, pyarrow or polars.
- No async code. To speed up IO use kafka driver native thread or parquet IO thread pool.
- No support for persistent state. Instead of saving state, replay historic data from kafka to prime stateful nodes.
Talks
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
beavers-0.14.0.tar.gz
(158.8 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
beavers-0.14.0-py3-none-any.whl
(32.8 kB
view details)
File details
Details for the file beavers-0.14.0.tar.gz.
File metadata
- Download URL: beavers-0.14.0.tar.gz
- Upload date:
- Size: 158.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.11.1 {"installer":{"name":"uv","version":"0.11.1","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
d84abc57cad0311ddc2bc6a43550572c787015454bc10cc0a99f0e871791fa52
|
|
| MD5 |
b0aee15a8718e896cdc3b6ac71a672d8
|
|
| BLAKE2b-256 |
0d0dcd6f3584ab7295dc45e9965c8e0f42d9ed4cb7c8b4202d61ee8950c41e9c
|
File details
Details for the file beavers-0.14.0-py3-none-any.whl.
File metadata
- Download URL: beavers-0.14.0-py3-none-any.whl
- Upload date:
- Size: 32.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.11.1 {"installer":{"name":"uv","version":"0.11.1","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
fba3f886fb0f15179256328f0e110220dd0f95a8efb854cf4d8b92beb4bd7473
|
|
| MD5 |
68e29220f5e146e3cb04e23015f7d403
|
|
| BLAKE2b-256 |
b7acdf49a6c741ee7ee679d3f10d6c77a619bcc29622b7bcec9c771aefa7995a
|
