metaxy 0.0.1.dev24

Pluggable versioned metadata management for incremental multi-modal pipelines.

Metaxy

Metaxy is a metadata layer for multi-modal Data and ML pipelines. Metaxy tracks lineage and versioning across complex computational graphs for multi-modal datasets.

Metaxy manages metadata while data typically lives elsewhere:

┌─────────────────────────────────┐          ┌─────────────────────────┐
│      Metadata (Metaxy)          │          │   Data (e.g., S3)       │
├──────┬──────────┬──────┬────────┤          │                         │
│  ID  │   path   │ size │version │          │  📦 s3://my-bucket/     │
├──────┼──────────┼──────┼────────┤          │                         │
│ img1 │ s3://... │ 2.1M │a3fdsf  │ ────────>│    ├─ img1.jpg          │
│ img2 │ s3://... │ 1.8M │b7e123  │ ────────>│    ├─ img2.jpg          │
└──────┴──────────┴──────┴────────┘          └─────────────────────────┘

The feature that makes Metaxy stand out is the ability to track partial data dependencies that are so common in multi-modal pipelines and skip downstream updates when they are not needed.

Metaxy's goal is to provide a standard instrument for any kind of multi-modal (or just purely tabular) incremental pipelines, standardizing dependency specification, versioning, partial data dependencies, and manipulations over metadata.

Read the docs to discover more about Metaxy.

Installation

Warning: Metaxy hasn't been publicly released yet, but you can try the latest dev release:

pip install --pre metaxy

Using Metaxy

Metaxy is highly pluggable and generally can be used with any kind of incremental pipelines, storage, metadata storage, and dataframe libraries.

Metaxy provides integrations with popular tools such as Dagster, ClickHouse, DeltaLake, SQLModel.

The full list can be found here.

Contributing

See CONTRIBUTING.md.