Skip to main content

Python Data Libraries

Project description

overlapping arrows

mabel is a platform for authoring data processing systems.

License Regression Suite codecov Static Analysis PyPI Latest Release Maintainability Rating Security Rating mabel deepcode Downloads

Documentation GitHub Wiki
Bug Reports GitHub Issues
Feature Requests GitHub Issues
Source Code GitHub
Discussions GitHub Discussions

Key Features

  • Programatically define data pipelines
  • Treats datasets as immutable
  • On-the-fly compression
  • Automatic version tracking of processing operations
  • Trace messages through the pipeline (random sampling)
  • Automatic retry of failed operations
  • Low-memory requirements, even with Tb of data

Installation

From PyPI (recommended)

pip install --upgrade mabel

From GitHub

pip install --upgrade git+https://github.com/joocer/mabel

Guides

How to Write a Flow
How to Read Data

Dependencies

  • UltraJSON (AKA ujson) is used where orjson is not available. (Notice1)
  • dateutil is used to convert dates received as strings
  • zstandard is used for real-time compression

There are a number of optional dependencies which are usually only required for specific features and functionality. These are listed in the requirements-test.txt file which is used for testing. The key exception is orjson which is the preferred JSON library but not available on all platforms.

Can I Contribute?

All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.

Want to help build mabel? See the contribution guidance

Platform Support?

mabel comes with adapters for the following services, or is tested to run on the following platforms:

Service Support
GCP Storage Google Cloud Storage Read/Write
MinIo MinIO Read/Write
AWS S3 S3 Read/Write
MongoDB MongoDB Read Only
MQTT MQTT Read Only
Docker Docker Hosting
Kubernetes Kubernetes Hosting
Raspberry Pi Raspberry Pi Hosting (Notice1)

Linux, MacOS and Windows (Notice2) also supported.

Adapters for other data services can be written.

Notice1 - Raspbian fully functional with ujson
Notice2 - Multi-Processing not available on Windows. Alternate indexing libraries may be used on Windows.

License

Apache 2.0

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mabel-0.4.12.tar.gz (68.0 kB view hashes)

Uploaded Source

Built Distribution

mabel-0.4.12-py3-none-any.whl (97.1 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page