metaxy 0.0.1rc2

Pluggable versioned metadata management for incremental multi-modal pipelines.

Metaxy

Metaxy is a metadata layer for multi-modal Data and ML pipelines. Metaxy tracks lineage and versioning across complex computational graphs for multi-modal datasets. Metaxy can cache every single sample and scale to handle millions of them.

Metaxy manages metadata while data typically lives elsewhere:

┌─────────────────────────────────┐          ┌─────────────────────────┐
│      Metadata (Metaxy)          │          │   Data (e.g., S3)       │
├──────┬──────────┬──────┬────────┤          │                         │
│  ID  │   path   │ size │version │          │  📦 s3://my-bucket/     │
├──────┼──────────┼──────┼────────┤          │                         │
│ img1 │ s3://... │ 2.1M │a3fdsf  │ ────────>│    ├─ img1.jpg          │
│ img2 │ s3://... │ 1.8M │b7e123  │ ────────>│    ├─ img2.jpg          │
└──────┴──────────┴──────┴────────┘          └─────────────────────────┘

The feature that makes Metaxy stand out is the ability to track field-level dependencies and detect prunable updates — updates that don't trigger change propagation through certain paths in the dependency graph because they modify fields that aren't dependencies of those downstream features. For example, updating audio upstream of a face recognition step allows pruning the face recognition branch since it only depends on video frames. This problem is specific to multi-modal pipelines and doesn't typically emerge in traditional data engineering.

Metaxy's goal is to provide a standard instrument for any kind of multi-modal (or just purely tabular) incremental pipelines, standardizing dependency specification, versioning, partial data dependencies, and manipulations over metadata. Or, in short, to be a universal glue for incremental data pipelines.

Metaxy is very reliable and is fanatically tested across all supported Python versions and platforms [^1].

Documentation

Read the docs to learn more.

Installation

[!WARNING] Metaxy hasn't been publicly released yet, but you can try the main branch:

pip install git+https://github.com/metaxy-dev/metaxy.git

Using Metaxy

Metaxy is highly pluggable and generally can be used with any kind of incremental pipelines, storage, metadata storage, and dataframe libraries.

Metaxy provides integrations with popular tools such as Dagster, Ray, ClickHouse, DeltaLake, SQLModel.

The full list can be found here.

Contributing

See CONTRIBUTING.md.

[^1]: The CLI is not tested on Windows yet.