bytewax

Bytewax is an open source Python framework for building highly scalable dataflows in a streaming or batch context.

Get started

About

Bytewax lets you build Python based dataflows to process your data for augmentation, advanced analysis, machine learning and more. It is based on Timely Dataflow, which is a cyclic dataflow computational model. At a high-level, dataflow programming is a programming paradigm where program execution is conceptualized as data flowing through a series of operator based steps. Operators are the processing primitives of bytewax. Each of them gives you a “shape” of data transformation, and you give them functions to customize them to a specific task you need. The combination of each operator and their custom logic functions we call a dataflow step. You chain together steps in a dataflow to solve your high-level data processing problem.

At a high level, Bytewax provides a few major benefits:

The operators in Bytewax are largely “data-parallel”, meaning they can operate on independent parts of the data concurrently.
Bytewax offers the ability to express higher-level control constructs, like iteration.
Bytewax allows you to develop and run your code locally, and then easily scale that code to multiple workers or processes without changes.
Bytewax can be used in both a streaming and batch context
Ability to leverage the Python ecosystem directly

Community

Slack Is the main forum for communication and discussion.

GitHub Issues is reserved only for actual issues. Please use the slack community for discussions.

Code of Conduct

Usage

Install the latest release with pip:

pip install bytewax

Building From Source

To build a specific branch, you will need to use Maturin and have Rust installed on your machine. Once those have been installed run

maturin develop

And you will be able to import bytewax in your python file.

Example

Here is an example of a simple dataflow program using Bytewax:

from bytewax.dataflow import Dataflow
from bytewax.execution import run_main
from bytewax.inputs import ManualInputConfig
from bytewax.outputs import StdOutputConfig

def input_builder(worker_index, worker_count, resume_from):
    resume_from = None # Ignore recovery logic
    for e in range(10):
        yield resume_from, e

flow = Dataflow()
flow.input("input", ManualInputConfig(input_builder))
flow.map(lambda x: x * x)
flow.capture(StdOutputConfig())

if __name__ == "__main__":
    run_main(flow)

Running the program prints the following output:

For a more complete example, and documentation on the available operators, check out the User Guide.

For an exhaustive list of examples, checkout the /examples folder

License

Bytewax is licensed under the Apache-2.0 license.

Contributing

Contributions are welcome! This community and project would not be what it is without the contributors. All contributions, from bug reports to new features, are welcome and encouraged.

With ❤️ Bytewax

Metadata

Release files for bytewax 0.11.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for bytewax 0.11.1
File	Interpreter	ABI	Platform	Reset
bytewax-0.11.1-cp310-cp310-manylinux_2_31_x86_64.whl 21.8 MB Sep 7, 2022	CPython 3.10	CPython 3.10	Linux glibc 2.31+ x86-64	Details
bytewax-0.11.1-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl 7.8 MB Sep 7, 2022	CPython 3.10	CPython 3.10	macOS 10.9+ universal2 (ARM64, x86-64), macOS 11.0+ ARM64, macOS 10.9+ x86-64	Details
bytewax-0.11.1-cp39-cp39-manylinux_2_31_x86_64.whl 21.8 MB Sep 7, 2022	CPython 3.9	CPython 3.9	Linux glibc 2.31+ x86-64	Details
bytewax-0.11.1-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl 7.8 MB Sep 7, 2022	CPython 3.9	CPython 3.9	macOS 10.9+ universal2 (ARM64, x86-64), macOS 10.9+ x86-64, macOS 11.0+ ARM64	Details
bytewax-0.11.1-cp38-cp38-manylinux_2_31_x86_64.whl 21.8 MB Sep 7, 2022	CPython 3.8	CPython 3.8	Linux glibc 2.31+ x86-64	Details
bytewax-0.11.1-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl 7.8 MB Sep 7, 2022	CPython 3.8	CPython 3.8	macOS 11.0+ ARM64, macOS 10.9+ universal2 (ARM64, x86-64), macOS 10.9+ x86-64	Details
bytewax-0.11.1-cp37-cp37m-manylinux_2_31_x86_64.whl 21.8 MB Sep 7, 2022	CPython 3.7	CPython 3.7 pymalloc	Linux glibc 2.31+ x86-64	Details
bytewax-0.11.1-cp37-cp37m-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl 7.8 MB Sep 7, 2022	CPython 3.7	CPython 3.7 pymalloc	macOS 10.9+ x86-64, macOS 10.9+ universal2 (ARM64, x86-64), macOS 11.0+ ARM64	Details

Total release size: 118.4 MB

Release files / bytewax-0.11.1-cp310-cp310-manylinux_2_31_x86_64.whl

Download URL	bytewax-0.11.1-cp310-cp310-manylinux_2_31_x86_64.whl
Size	21.8 MB
Tags	CPython 3.10 Linux glibc 2.31+ x86-64
SHA-256 checksum How to use checksums	`2768abb07b43dd8a41444368c23473f5deafcee460a7f020fe55d8fad2c302c4`
BLAKE2b-256 checksum How to use checksums	`f39a52279cb9913d1312abb2c8c06c428ad0c5aaf008a3568fea5143fc34144b`
Upload date	Sep 7, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.1 CPython/3.9.13`

Release files / bytewax-0.11.1-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl

Download URL	bytewax-0.11.1-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Size	7.8 MB
Tags	CPython 3.10 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64
SHA-256 checksum How to use checksums	`70df637681e048cdcabfe90d7a1754e9786a23c8bef2cd94572d3e051fecdf0c`
BLAKE2b-256 checksum How to use checksums	`38baaea3d2275bb0c2a72b3f3b76d7273b0f92b78b69af6c5252b4fe3db538ab`
Upload date	Sep 7, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.1 CPython/3.9.13`

Release files / bytewax-0.11.1-cp39-cp39-manylinux_2_31_x86_64.whl

Download URL	bytewax-0.11.1-cp39-cp39-manylinux_2_31_x86_64.whl
Size	21.8 MB
Tags	CPython 3.9 Linux glibc 2.31+ x86-64
SHA-256 checksum How to use checksums	`755b03deaf00ee5e3b8d439e711afbdf9cbfd4d178721f2b8b27454c374cb209`
BLAKE2b-256 checksum How to use checksums	`db8f099a785bafc245c0a9086fd7c1a5d7a3e960a2ccb8e54cc65ef88d3ee5c8`
Upload date	Sep 7, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.1 CPython/3.9.13`

Release files / bytewax-0.11.1-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl

Download URL	bytewax-0.11.1-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Size	7.8 MB
Tags	CPython 3.9 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64
SHA-256 checksum How to use checksums	`f0ee8ca5c01827842702dd1f21dab1c6a19a937c48dc53b2486f9716f3d534fd`
BLAKE2b-256 checksum How to use checksums	`0acf19df96f7194f0d5d27162f1dcaf52b4b3ffcfd0d49891f38d54dee93bd99`
Upload date	Sep 7, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.1 CPython/3.9.13`

Release files / bytewax-0.11.1-cp38-cp38-manylinux_2_31_x86_64.whl

Download URL	bytewax-0.11.1-cp38-cp38-manylinux_2_31_x86_64.whl
Size	21.8 MB
Tags	CPython 3.8 Linux glibc 2.31+ x86-64
SHA-256 checksum How to use checksums	`f554e4512d82093b4cfe0c6453cad1edf8fa4c9dfb79ff761e73b3e1884cc421`
BLAKE2b-256 checksum How to use checksums	`270a5194d60bc74346389bbecd57e47c0407f7bde759fee73c06a82a2575005f`
Upload date	Sep 7, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.1 CPython/3.9.13`

Release files / bytewax-0.11.1-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl

Download URL	bytewax-0.11.1-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Size	7.8 MB
Tags	CPython 3.8 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64
SHA-256 checksum How to use checksums	`3d8e3e0e2e983388f490c0887e5ad6daa351861c36c2bdb575a05c40c3ca23d7`
BLAKE2b-256 checksum How to use checksums	`ae836d772ffa0328b4927c79f9bad1e1bdeb1f3561f28e8cb6ce26015fe24604`
Upload date	Sep 7, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.1 CPython/3.9.13`

Release files / bytewax-0.11.1-cp37-cp37m-manylinux_2_31_x86_64.whl

Download URL	bytewax-0.11.1-cp37-cp37m-manylinux_2_31_x86_64.whl
Size	21.8 MB
Tags	CPython 3.7 CPython 3.7 pymalloc Linux glibc 2.31+ x86-64
SHA-256 checksum How to use checksums	`f1fe2e0d5f5fe80ca4deddf04f45a31cab4c197dfb57f87afcb36811ab7a0bb6`
BLAKE2b-256 checksum How to use checksums	`1c04aabbe2debf42b1bc83400147867a87c7ed17c6618bf173853ae110c08a79`
Upload date	Sep 7, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.1 CPython/3.9.13`

Release files / bytewax-0.11.1-cp37-cp37m-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl

Download URL	bytewax-0.11.1-cp37-cp37m-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Size	7.8 MB
Tags	CPython 3.7 CPython 3.7 pymalloc macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64
SHA-256 checksum How to use checksums	`c6ed247c3d657179a41cc81becf95f5378414f4b8a755ce7ba1def437c678847`
BLAKE2b-256 checksum How to use checksums	`751c309422cbe42f2ffa55c6c78894e1a1eeac7752ee7002365bf087b6feae9c`
Upload date	Sep 7, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.1 CPython/3.9.13`