Python Data Libraries
Reason this release was yanked:
installer issue
Project description
mabel is a platform for authoring data processing systems.
What are its Key Features
- Programatically define data pipelines
- Treats datasets as immutable
- On-the-fly compression
- Automatic version tracking of processing operations
- Trace messages through the pipeline (random sampling)
- Automatic retry of failed operations
- Low-memory requirements, even with Tb of data
Where Can I Find Documentation
See the wiki
How Do I Get It?
From PyPI (recommended)
pip install --upgrade mabel
From GitHub
pip install --upgrade git+https://github.com/joocer/mabel
What Dependencies does mabel Have?
- UltraJSON (AKA
ujson
) is used whereorjson
is not available.orjson
is the preferred JSON library but is not available on all platforms and environments soujson
is a dependency to ensure a performant JSON library with broad support is available. - dateutil is used to convert dates received as strings
- zstandard is used for real-time compression
- mmh3 is used for non-cryptographic hashing
- bitarray is used for compacting data
There are a number of optional dependencies which are usually only required for specific features and functionality. These are listed in the requirements-test.txt file which is used for testing. The key exception is orjson
which is the preferred JSON library but not available on all platforms.
Can I Contribute?
Want to help build mabel? See the contribution guidance
What Platforms Does mabel Support?
mabel comes with adapters for the following services, or is tested to run on the following platforms:
Service | Support | |
---|---|---|
Google Cloud Storage | Read/Write | |
MinIO | Read/Write | |
S3 | Read/Write | |
MongoDB | Read Only | |
MQTT | Read Only | |
Docker | Hosting | |
Kubernetes | Hosting | |
Raspberry Pi | Hosting (1) |
Linux, MacOS and Windows (2) also supported.
Adapters for other data services can be written.
1 - Raspbian fully functional with ujson
2 - Multi-Processing not available on Windows
License
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.