Python stream processing
Project description
Beavers
Beavers is a python library for stream processing, optimized for analytics.
It is used at Tradewell Technologies, to calculate analytics and serve model predictions, in both realtime and batch jobs.
Key Features
- Works in real time (eg: reading from kafka) and replay mode (eg: reading from parquet)
- Optimized for analytics, it uses micro-batching (instead of processing records one by one)
- Similar to incremental, it updates nodes in a dag incrementally
- Taking inspiration from kafka streams, there are two types of nodes in the dag:
- Stream: ephemeral micro-batches of events (cleared after every cycle)
- State: durable state derived from streams
- Clear separation between the business logic and the IO. So the same dag can be used in real time mode, replay mode or can be easily tested.
- Functional interface: no inheritance or decorator required
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
beavers-0.0.4.tar.gz
(18.9 kB
view hashes)
Built Distribution
beavers-0.0.4-py3-none-any.whl
(19.5 kB
view hashes)
