bytewax 0.8.0b0

No project description provided

Bytewax

Bytewax is an open source Python framework for building highly scalable dataflows in a streaming or batch context.

Dataflow programming is a programming paradigm where program execution is conceptualized as data flowing through a series of operations or transformations.

At a high level, Bytewax provides a few major benefits:

• The operators in Bytewax are largely “data-parallel”, meaning they can operate on independent parts of the data concurrently.
• The ability to express higher-level control constructs, like iteration.
• Bytewax allows you to develop and run your code locally, and then easily scale that code to multiple workers or processes without changes.

Bytewax uses PyO3 to provide Python bindings to the Timely Dataflow Rust library.

Visit our documentation site for full documentation

Usage

Install the latest release with pip:

pip install bytewax

Example

Here is an example of a simple dataflow program using Bytewax:

# examples/simple.py
from bytewax import Dataflow, run


flow = Dataflow()
flow.map(lambda x: x * x)
flow.capture()


if __name__ == "__main__":
    for epoch, y in sorted(run(flow, enumerate(range(10)))):
        print(y)

Running the program:

python ./examples/simple.py
0
1
4
9
16
25
36
49
64
81

For a more complete example, and documentation on the available operators, check out the User Guide.

License

Bytewax is licensed under the Apache-2.0 license.