beavers 0.0.1rc0

Python stream processing

Beavers

---

Beavers is a python library for stream processing, optimize for analytics.

It is used at Tradewell Technologies, to calculate analytics and serve model predictions, in both realtime and batch jobs.

Key Features

• Works in real time (eg: reading from kafka) and replay mode (eg: reading from parquet)
• Optimized for analytics, it uses micro-batching (instead of processing records one by one)
• Similar to incremental, it updates nodes in a dag incrementally
• Taking inspiration from kafka streams, there are two types of nodes in the dag:
  • Stream: ephemeral micro-batches of events (cleared after every cycle)
  • State: durable state derived from streams
• Clear separation between the dag (which contains the business logic) and the IO (where the data comes from/goes). So the same dag can be used in real time mode, replay mode or can be easily tested.