Python stream processing
Project description
Beavers
Beavers is a python library for stream processing, optimize for analytics.
It is used at Tradewell Technologies, to calculate analytics and serve model predictions, in both realtime and batch jobs.
Key Features
- Works in real time (eg: reading from kafka) and replay mode (eg: reading from parquet)
- Optimized for analytics, it uses micro-batching (instead of processing records one by one)
- Similar to incremental, it updates nodes in a dag incrementally
- Taking inspiration from kafka streams, there are two types of nodes in the dag:
- Stream: ephemeral micro-batches of events (cleared after every cycle)
- State: durable state derived from streams
- Clear separation between the business logic and the IO. So the same dag can be used in real time mode, replay mode or can be easily tested.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
beavers-0.0.1.tar.gz
(9.4 kB
view hashes)