Serverless Data Mesh

Federated data mesh lakehouse publication on AWS Lambda - with cryptographic proof, not just green job logs.

An open Python framework for domain-oriented, federated data mesh teams on AWS. Implements data as a product, self-serve write infrastructure, and federated computational governance at the Iceberg lakehouse layer.

• Producer domains publish governed data products
• Steward notaries enforce mesh contracts with VRP proofs
• Publisher zones expose consumer-ready Iceberg data products

Introduces the Vaquar Pattern (PVDM): Physical → Verify → Durable → Metadata. Invariant: commit_metadata ⟹ VRP = PASS.

Website: GitHub Pages · Paper: arXiv:2608.14643 · Reference gate: Proof-gated-publication-PVDM

Install

pip install serverless-data-mesh
serverless-data-mesh demo    # <60s local PVDM demo, no AWS
serverless-data-mesh new --template medallion --output my-mesh
serverless-data-mesh apply --contract my-mesh/mesh.yaml --output my-mesh/generated
pip install "serverless-data-mesh[rules]"   # + SparkRules on Lambda
pip install "serverless-data-mesh[spark]"   # + PySpark + SparkRules
pip install "serverless-data-mesh[all]"     # rules + spark
pip install "serverless-data-mesh[dev]"     # pytest, ruff, pre-commit

Works on Windows, Mac, and Linux. Uses veridata-recon cryptographic VRP when wheels are available; pure-Python fallback otherwise (same PASS/FAIL gate).

What it solves

Building blocks

Quick example

from serverless_data_mesh import (
    IceGuardDurableCoordinator,
    DomainTransactionBoundary,
    VRPProofGenerator,
)

boundary = DomainTransactionBoundary(
    domain_id="orders-domain",
    source_namespace="raw_orders",
    target_table="orders_curated",
    partition_spec={"dt": "2026-06-14"},
)

coordinator = IceGuardDurableCoordinator(
    durable_context=durable_ctx,
    lambda_context=lambda_ctx,
    proof_generator=VRPProofGenerator(),
    catalog_adapter=glue_adapter,
)
outcome = coordinator.run_workload(workload)

Optional extras

Documentation

• Website (GitHub Pages) — demos, walkthrough, developer path
• GitHub README
• PVDM paper (arXiv:2608.14643)
• PVDM reference gate
• Vaquar Pattern
• Getting started
• PyPI guide
• Full documentation

License

Apache-2.0