No project description provided
Project description
Bytewax is an open source Python framework for building highly scalable dataflows in a streaming or batch context.
Get started
Check out our getting started guide.
About
Bytewax lets you build Python based dataflows to process your data for augmentation, advanced analysis, machine learning and more. It is based on Timely Dataflow, which is a cyclic dataflow computational model. At a high-level, dataflow programming is a programming paradigm where program execution is conceptualized as data flowing through a series of operator based steps. Operators are the processing primitives of bytewax. Each of them gives you a “shape” of data transformation, and you give them functions to customize them to a specific task you need. The combination of each operator and their custom logic functions we call a dataflow step. You chain together steps in a dataflow to solve your high-level data processing problem.
At a high level, Bytewax provides a few major benefits:
- The operators in Bytewax are largely “data-parallel”, meaning they can operate on independent parts of the data concurrently.
- Bytewax offers the ability to express higher-level control constructs, like iteration.
- Bytewax allows you to develop and run your code locally, and then easily scale that code to multiple workers or processes without changes.
- Bytewax can be used in both a streaming and batch context
- Ability to leverage the Python ecosystem directly
Community
Slack Is the main forum for communication and discussion.
GitHub Issues is reserved only for actual issues. Please use the slack community for discussions.
Usage
Install the latest release with pip:
pip install bytewax
Example
Here is an example of a simple dataflow program using Bytewax:
from bytewax import Dataflow, run
flow = Dataflow()
flow.map(lambda x: x * x)
flow.capture()
if __name__ == "__main__":
for epoch, x in sorted(run(flow, enumerate(range(10)))):
print(x)
Running the program prints the following output:
0
1
4
9
16
25
36
49
64
81
For a more complete example, and documentation on the available operators, check out the User Guide.
For an exhaustive list of examples, checkout the /examples folder
License
Bytewax is licensed under the Apache-2.0 license.
Contributing
Contributions are welcome! This community and project would not be what it is without the contributors. All contributions, from bug reports to new features, are welcome and encouraged.
With ❤️ Bytewax
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for bytewax-0.10.0-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 75c64cc288892ff5d04a6930bb4b2122b7294c683b7267306d8cc040150baa88 |
|
MD5 | 414dc11c4766bdf5c9e3c8f9c9e585c7 |
|
BLAKE2b-256 | 71fae4399182f2fe4afa630b505b8e4809e081ff3db8aa858a831320fd900d13 |
Hashes for bytewax-0.10.0-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d62c1b49693bd0699f2c1f45b147b1961269d6fe6954530b94006cccc8dd6514 |
|
MD5 | c6dead2ec547113c9779169446452384 |
|
BLAKE2b-256 | 9fbe8c165275b1e82414484afb7e485e76a2d508e23c748e0c750b2ff9f74d2b |
Hashes for bytewax-0.10.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d456b7db481ef5519d976419260b0b6e63c1b1ff8a68a2df5b86621f6f8e6b1b |
|
MD5 | 0a548c29c99fd0eed03ab52ef34f76e5 |
|
BLAKE2b-256 | d4b834ac60842b9dc44122110e818f690d060955e69de449e8d14d002088c73c |
Hashes for bytewax-0.10.0-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d532056ec4dc219f28a62a1ab346ae2dd1d01ee9923aa5197cfdc2315debbb91 |
|
MD5 | 83eabe6b369564e4ac110b9eceecd33b |
|
BLAKE2b-256 | acdd39bcfbee93d899600e9225df7e5bf6c286bbfa445e7c90fd1023839853ff |
Hashes for bytewax-0.10.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4c7f3f081a118a798554fed93198e8421e3c2dba1312e3d343a28f60f45615ad |
|
MD5 | b237673f730306afc9ed75c43a9a6ac7 |
|
BLAKE2b-256 | 1ad307693552283c871ff4391b399b94e6c564463bbfed380029c5687a89605a |
Hashes for bytewax-0.10.0-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 881093f54aa9a04b68326d74d12e3f86d6aa6701476bd336f6a562b592d0a700 |
|
MD5 | bfb0b54cafb6a2ea0b036066c91b91bc |
|
BLAKE2b-256 | b30130663f942379af7e68b02ac4ebd818bf41c37e9a41d74bc456466c574166 |
Hashes for bytewax-0.10.0-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fcdcf51dc055162eb77ace657f742cf57840ed761f2c4ef886ea223fc508e8ae |
|
MD5 | 04e75d1e96c0897b072e5518ebac16c5 |
|
BLAKE2b-256 | 7e3f0f3ac5ec73f107a0789eade34b5264a647a3bd2b440cde6e821a1c5f6d51 |
Hashes for bytewax-0.10.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | caaf8327bdbc25258f70de17059d6d5abe9d5e789087564d32a95a9062ee0465 |
|
MD5 | f10bd257b35dec333f9d89ef073d5e72 |
|
BLAKE2b-256 | caaf61321b6e9b4dc0cc600904cd6d74f4523e46183a1abe8fc2d765fa07d3c7 |
Hashes for bytewax-0.10.0-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1bd1df28005b80c243a4f3a72ff7ab7f7ec56580d7d92836d0ab689d9601cc36 |
|
MD5 | 1a7182cc0cf12b0a8d756781a1f8e6d6 |
|
BLAKE2b-256 | 1c9e6f7e2e16435f3fd5e6fba35146bc79648572a4cc9de36acfe8cbab416563 |
Hashes for bytewax-0.10.0-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 839d22571a474ab6e52dfc070264ffc8430b76b1795b5d2c2bd1dd3d00d3a1af |
|
MD5 | 3c83e449e3f99f904d9139b855a9a61b |
|
BLAKE2b-256 | 72c2fd7e77110a1db271ab9bb4d53e0055cdb8a0f7bf2038f5260ea1114e34a8 |
Hashes for bytewax-0.10.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 76bdc5855a081fb98a84b175d8054e2088cf98db8b867ec130963d1f6194f3ea |
|
MD5 | b50c2a0c5a9bf534ff8cd48b1ecb5bd5 |
|
BLAKE2b-256 | 978937390eab675fba737a6eecb8189796f0cf63ff6e6bbe074720f21ef501ee |
Hashes for bytewax-0.10.0-cp37-cp37m-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 92d1ba19499c5611fce7ca17466df71e237668378b8e9d2b44216cd441259b8b |
|
MD5 | db85d141d041d4bacbb49f701bc6fb0c |
|
BLAKE2b-256 | f6e03b5719c70c22256660eb5c6a9fe087d842f0ebb3c1090dfa6ee173b6a4d4 |