mabel 0.4.7

Python Data Libraries

mabel is a platform for authoring data processing systems.

What are its Key Features

• Programatically define data pipelines
• Treats datasets as immutable
• On-the-fly compression
• Automatic version tracking of processing operations
• Trace messages through the pipeline (random sampling)
• Automatic retry of failed operations
• Low-memory requirements, even with Tb of data

Where Can I Find Documentation

See the wiki

How Do I Get It?

From PyPI (recommended)

pip install --upgrade mabel

From GitHub

pip install --upgrade git+https://github.com/joocer/mabel

What Dependencies does mabel Have?

• UltraJSON (AKA ujson) is used where orjson is not available. orjson is the preferred JSON library but is not available on all platforms and environments so ujson is a dependency to ensure a performant JSON library with broad support is available.
• dateutil is used to convert dates received as strings
• zstandard is used for real-time compression
• mmh3 is used for non-cryptographic hashing
• bitarray is used for compacting data

There are a number of optional dependencies which are usually only required for specific features and functionality. These are listed in the requirements-test.txt file which is used for testing. The key exception is orjson which is the preferred JSON library but not available on all platforms.

Can I Contribute?

Want to help build mabel? See the contribution guidance

What Platforms Does mabel Support?

mabel comes with adapters for the following services, or is tested to run on the following platforms:

	Service	Support
	Google Cloud Storage	Read/Write
	MinIO	Read/Write
	S3	Read/Write
	MongoDB	Read Only
	MQTT	Read Only
	Docker	Hosting
	Kubernetes	Hosting
	Raspberry Pi	Hosting (1)

Linux, MacOS and Windows (2) also supported.

Adapters for other data services can be written.

1 - Raspbian fully functional with ujson
2 - Multi-Processing not available on Windows

License

Apache 2.0