Manage your dataflows seamlessly
Project description
Dataflow Awesome Managing Engine
The easiest dataflow managing framework - currently under construction.
DAME solves/facilitates:
- Building datasets from files / folders
- Transforming data in the right order
- Saving transformed data - once computed never compute it again
- Choosing the best transformation from a few configurations
Great for working with numpy, pyTorch and more.
Vision
Technically:
- Compute stages:
- Sources - get data element
- Transforms - compute something out of available data
- Reducers - compute something on the whole dataset
- Combining data sources
- Compute only what you need - optimized performance via DAGs
- Backup and cache, after stages, support for custom serializers
- Ranking various configurations
- (Optional) Parallel processing
Priorities:
- Easy to use
- Batteries included
- Little overhead - take advantage of fastest tools available
- Integrates seamlessly with other tools
- Expandable
Nice to have:
- Few python dependencies
- Integrate tqdm
- DAG output
Release history Release notifications
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
| Filename, size | File type | Python version | Upload date | Hashes |
|---|---|---|---|---|
| Filename, size dame-0.0.1-py3-none-any.whl (2.7 kB) | File type Wheel | Python version py3 | Upload date | Hashes View hashes |
| Filename, size dame-0.0.1.tar.gz (1.8 kB) | File type Source | Python version None | Upload date | Hashes View hashes |
