Skip to main content

Python Data Libraries

Project description

overlapping arrows

mabel is a Data Engineering platform designed to run in serverless environments.

mabel just runs when you need it, scaling to zero, making it efficient and ideal for deployments to platforms like Kubernetes, GCP Cloud Run, AWS Fargate and Knative.

License Status Regression Suite codecov Static Analysis PyPI Latest Release Maintainability Rating Security Rating mabel Downloads Code style: black commit_freq last_commit PyPI Latest Release

Focus on What Matters

We've built mabel to enable Data Analysts to write complex data engineering tasks quickly and easily, so they could get on with doing what they do best.

from mabel import Reader

data = Reader(dataset="test_data")
print(data.count())

Key Features

  • On-the-fly compression
  • Low-memory requirements, even with terabytes of data
  • Indexing and partitioning of data for fast reads
  • Cursors for tracking reading position between processes
  • Partial SQL DQL (Data Query Language) support
  • Schema and data_expectations validation

Installation

From PyPI (recommended)

pip install --upgrade mabel

From GitHub

pip install --upgrade git+https://github.com/mabel-dev/mabel

A preview release of mabel is available from PyPI

pip install --upgrade mabelbeta

You may need to manually uninstall mabel before the test version will install.

These versions are usually labelled with an a (signifying alpha status) in the library version. Alpha versions are more likely to have functional issues.

Guides

How to Read Data

Dependencies

  • orjson for JSON (de)serialization
  • bitarray for handling high density boolean data
  • siphashc for non-cryptographic hashing
  • pydantic to define internal data models
  • zstandard for real-time on disk compression
  • LZ4 for real-time in memory compression
  • simdjson for fast JSON deserialization
  • cython for precompilation

There are a number of optional dependencies which are usually only required for specific features and functionality. These are listed in tests/requirements.txt.

Integrations

mabel comes with adapters for the following data services:

Service
GCP Storage Google Cloud Storage
MinIo MinIO
AWS S3 AWS S3
Azure Azure Blob Storage
Local Local Storage

Mabel is extensible with adapters for other data services as required.

Deployment and Execution

mabel supports running on a range of platforms, including:

Platform
Docker Docker
Kubernetes Kubernetes
Windows Windows (Notice1)
Linux Linux (Notice2)

Notice1 - Some non-core features are not available on Windows.
Notice2 - Tested on Debian (WSL) and Ubuntu.

How Can I Contribute?

All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.

If you have a suggestion for an improvement or a bug, raise a ticket or start a discussion.

Want to help build mabel? See the contribution guidance.

License

Apache 2.0

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mabel-0.5.0.tar.gz (493.7 kB view hashes)

Uploaded Source

Built Distributions

mabel-0.5.0-cp39-cp39-win_amd64.whl (412.4 kB view hashes)

Uploaded CPython 3.9 Windows x86-64

mabel-0.5.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

mabel-0.5.0-cp39-cp39-macosx_10_14_x86_64.whl (490.3 kB view hashes)

Uploaded CPython 3.9 macOS 10.14+ x86-64

mabel-0.5.0-cp38-cp38-win_amd64.whl (412.5 kB view hashes)

Uploaded CPython 3.8 Windows x86-64

mabel-0.5.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

mabel-0.5.0-cp38-cp38-macosx_10_14_x86_64.whl (489.7 kB view hashes)

Uploaded CPython 3.8 macOS 10.14+ x86-64

mabel-0.5.0-cp37-cp37m-win_amd64.whl (399.7 kB view hashes)

Uploaded CPython 3.7m Windows x86-64

mabel-0.5.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.9 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

mabel-0.5.0-cp37-cp37m-macosx_10_14_x86_64.whl (480.6 kB view hashes)

Uploaded CPython 3.7m macOS 10.14+ x86-64

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page