mabel 0.4.12

Python Data Libraries

mabel is a platform for authoring data processing systems.

Documentation GitHub Wiki
Bug Reports GitHub Issues
Feature Requests GitHub Issues
Source Code GitHub
Discussions GitHub Discussions

Key Features

• Programatically define data pipelines
• Treats datasets as immutable
• On-the-fly compression
• Automatic version tracking of processing operations
• Trace messages through the pipeline (random sampling)
• Automatic retry of failed operations
• Low-memory requirements, even with Tb of data

Installation

From PyPI (recommended)

pip install --upgrade mabel

From GitHub

pip install --upgrade git+https://github.com/joocer/mabel

Guides

How to Write a Flow
How to Read Data

Dependencies

• UltraJSON (AKA ujson) is used where orjson is not available. (1)
• dateutil is used to convert dates received as strings
• zstandard is used for real-time compression

There are a number of optional dependencies which are usually only required for specific features and functionality. These are listed in the requirements-test.txt file which is used for testing. The key exception is orjson which is the preferred JSON library but not available on all platforms.

Can I Contribute?

All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.

Want to help build mabel? See the contribution guidance

Platform Support?

mabel comes with adapters for the following services, or is tested to run on the following platforms:

	Service	Support
	Google Cloud Storage	Read/Write
	MinIO	Read/Write
	S3	Read/Write
	MongoDB	Read Only
	MQTT	Read Only
	Docker	Hosting
	Kubernetes	Hosting
	Raspberry Pi	Hosting (1)

Linux, MacOS and Windows (2) also supported.

Adapters for other data services can be written.

1 - Raspbian fully functional with ujson
2 - Multi-Processing not available on Windows. Alternate indexing libraries may be used on Windows.

License

Apache 2.0