Skip to main content

Metaxy Logo

Metaxy

PyPI version Python versions PyPI downloads CI codecov JOSS paper Ruff Ty prek

Metaxy is a metadata layer for multimodal Data and ML pipelines. Metaxy tracks lineage and versioning across complex computational graphs for multimodal datasets. Metaxy can cache every single sample and scale to handle millions of them.

Metaxy manages metadata while data typically lives elsewhere:

┌─────────────────────────────────┐          ┌─────────────────────────┐
│      Metadata (Metaxy)          │          │   Data (e.g., S3)       │
├──────┬──────────┬──────┬────────┤          │                         │
│  ID  │   path   │ size │version │          │  📦 s3://my-bucket/     │
├──────┼──────────┼──────┼────────┤          │                         │
│ img1 │ s3://... │ 2.1M │a3fdsf  │ ────────>│    ├─ img1.jpg          │
│ img2 │ s3://... │ 1.8M │b7e123  │ ────────>│    ├─ img2.jpg          │
└──────┴──────────┴──────┴────────┘          └─────────────────────────┘

The feature that makes Metaxy stand out is the ability to track partial data dependencies and detect prunable updates — updates that don't trigger change propagation through certain paths in the dependency graph because they modify fields that aren't dependencies of those downstream features. For example, updating audio upstream of a face recognition step allows pruning the face recognition branch since it only depends on video frames. This problem is specific to multimodal pipelines and doesn't typically emerge in traditional data engineering.

Metaxy's goal is to provide a standard instrument for any kind of multimodal (or purely tabular) incremental pipelines, standardizing dependency specification, versioning, partial data dependencies, and manipulations over metadata. Or, in short, to be a universal glue for incremental data pipelines.

Metaxy is very reliable and is fanatically tested across all supported Python versions and platforms 1.

Documentation

Read the docs to learn more.

Metaxy is published in the Journal of Open Source Software.

Installation

Install Metaxy from PyPI:

uv add metaxy

Using Metaxy

Metaxy is highly pluggable and generally can be used with any kind of incremental pipelines, storage, metadata storage, and dataframe libraries.

Metaxy provides integrations with popular tools such as Dagster, Ray, ClickHouse, DeltaLake, SQLModel.

The full list can be found here.

Blog Posts

Contributing

See CONTRIBUTING.md.

  1. The CLI is not tested on Windows yet. ↩

Metadata

Release files for metaxy 0.1.12

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for metaxy 0.1.12
File Size Uploaded Provenance Actions
metaxy-0.1.12.tar.gz 256.5 kB Without provenance Details for metaxy-0.1.12.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for metaxy 0.1.12
File Interpreter ABI Platform Actions
metaxy-0.1.12-py3-none-any.whl Python 3 none any Details for metaxy-0.1.12-py3-none-any.whl

Total release size: 584.5 kB

Release files / metaxy-0.1.12.tar.gz

Download URL metaxy-0.1.12.tar.gz
Size 256.5 kB
Tags Source
SHA-256 checksum
How to use checksums
6717632e08e0a4fdf71586cffb3f6c98e7e05ea320c7f3a8c0ca07362d161467
BLAKE2b-256 checksum
How to use checksums
457578c353a6af942bea8c0cbfa1e22ef6b35bbc344bc3a63f490867d43d0b1b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / metaxy-0.1.12-py3-none-any.whl

Download URL metaxy-0.1.12-py3-none-any.whl
Size 328.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
5f76aff938b2ca60f433d902f00ff7121e8a672cf15da3a383a0b86e8fcaa757
BLAKE2b-256 checksum
How to use checksums
3ce939db5b0fb266f53e9d63ebfcbec8606c0844b398d695c6f3be7eb10c2021
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for metaxy 0.1.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.12 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page