Serverless Data Mesh

Federated data mesh lakehouse publication on AWS Lambda - with cryptographic proof, not just green job logs.

An open Python framework for domain-oriented, federated data mesh teams on AWS. Implements data as a product, self-serve write infrastructure, and federated computational governance at the Iceberg lakehouse layer.

  • Producer domains publish governed data products
  • Steward notaries enforce mesh contracts with VRP proofs
  • Publisher zones expose consumer-ready Iceberg data products

Introduces the Vaquar Pattern (PVDM): Physical → Verify → Durable → Metadata. Invariant: commit_metadata ⟹ VRP = PASS.

Install

pip install serverless-data-mesh
serverless-data-mesh demo    # <60s local PVDM demo, no AWS
serverless-data-mesh new --template medallion --output my-mesh
serverless-data-mesh apply --contract my-mesh/mesh.yaml --output my-mesh/generated
pip install "serverless-data-mesh[rules]"   # + SparkRules on Lambda
pip install "serverless-data-mesh[spark]"   # + PySpark + SparkRules
pip install "serverless-data-mesh[all]"     # rules + spark
pip install "serverless-data-mesh[dev]"     # pytest, ruff, pre-commit

Works on Windows, Mac, and Linux. Uses veridata-recon cryptographic VRP when wheels are available; pure-Python fallback otherwise (same PASS/FAIL gate).

What it solves

Problem Answer
Silent data loss on backfill VRP FAIL blocks Iceberg snapshot
"Job succeeded" is not proof Cryptographic multiset proof per chunk
Lambda 15-minute limit Durable Execution + Step Functions (90+ min)
Retry duplicates data IceGuard rollback + workload_id checkpoints
Central ETL bottleneck Per-domain Lambda writer + transaction boundary
Federated blast radius Producer · Steward · Publisher accounts

Building blocks

Package Role
iceguard Physical SafeWriter, timeout rollback, S3 resume
veridata-recon VRP proof generation and validation
aws-durable-execution-sdk-python Cross-invocation step replay
pyiceberg Glue Iceberg REST metadata via GlueCatalogConnector
sparkrules Optional DRL business rules ([rules] extra)

Quick example

from serverless_data_mesh import (
    IceGuardDurableCoordinator,
    DomainTransactionBoundary,
    VRPProofGenerator,
)

boundary = DomainTransactionBoundary(
    domain_id="orders-domain",
    source_namespace="raw_orders",
    target_table="orders_curated",
    partition_spec={"dt": "2026-06-14"},
)

coordinator = IceGuardDurableCoordinator(
    durable_context=durable_ctx,
    lambda_context=lambda_ctx,
    proof_generator=VRPProofGenerator(),
    catalog_adapter=glue_adapter,
)
outcome = coordinator.run_workload(workload)

Optional extras

Extra Adds
rules SparkRules DRL on Lambda
spark PySpark + SparkRules
all rules + spark
dev pytest, ruff, mypy, pre-commit
publish build, twine

Documentation

License

Apache-2.0

Project links

Data verified by PyPI on Jul 21, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for vaquarkhan from gravatar.com vaquarkhan

Credits

Author: Serverless Data Mesh Contributors

GitHub Statistics

Data verified by PyPI on Jul 21, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

Apache-2.0
View SPDX License List

Requires

Python >=3.12

Provides Extra

all dev publish rules spark

Tags

aws bronze-silver-gold data-as-a-product data-mesh domain-oriented durable-execution etl federated-data-mesh federated-governance iceberg iceguard lakehouse lambda medallion metadata-driven pyiceberg rule-engine serverless sparkrules vaquar-pattern veridata-recon

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

serverless_data_mesh-1.2.0.tar.gz (47.1 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

serverless_data_mesh-1.2.0-py3-none-any.whl (102.6 kB view details)

Uploaded Python 3

File details

Details for the file serverless_data_mesh-1.2.0.tar.gz.

File metadata

  • Download URL: serverless_data_mesh-1.2.0.tar.gz
  • Upload date:
  • Size: 47.1 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for serverless_data_mesh-1.2.0.tar.gz
Algorithm Hash digest
SHA256 1c673199856ab18daccb4fcc7da8159440aac9b0244ec771b97c951572e90a00
MD5 7e70a2467fa3a918b6d9820ae7ee7f06
BLAKE2b-256 91c500c2ba3de3d5e1a72a47edc9fb9077139c6cc4551da5119a2cbf6ae0b449

See more details on using hashes here.

Provenance

The following attestation bundles were made for serverless_data_mesh-1.2.0.tar.gz:

Publisher: publish.yml on vaquarkhan/aws-serverless-datamesh-framework

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file serverless_data_mesh-1.2.0-py3-none-any.whl.

File metadata

File hashes

Hashes for serverless_data_mesh-1.2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 447e115f8be5995efbabc2e6122f5e9e97b8dca354c2e5dbeac49074d629d76c
MD5 295192115c5d54cf09522f98d982484e
BLAKE2b-256 47541ab7967a55713aa8e3006b4cdb28d9d0c66ece9bc4c183a2aa6d732f8903

See more details on using hashes here.

Provenance

The following attestation bundles were made for serverless_data_mesh-1.2.0-py3-none-any.whl:

Publisher: publish.yml on vaquarkhan/aws-serverless-datamesh-framework

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

1.2.0 This release

2 files

1.0.0

2 files

0.2.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page