lakelogic 1.41.0

A declarative, contract-driven medallion pipeline engine for data mesh architectures. Write once. Run on Spark, Polars, or DuckDB.

LakeLogic

Define data trust once. Enforce it from development to production.

LakeLogic is an open-source framework for executable data contracts. Describe a data product's schema, quality rules, PII handling, lineage, and materialization in YAML, then run that contract with Polars, DuckDB, or Spark.

Use the same contract to get fast feedback locally, check changes in CI/CD, and govern pipeline execution in your lakehouse. Records that fail row-level rules can be retained with their failure reasons instead of being silently discarded.

Run the five-minute Colab quickstart · Read the documentation · Browse the examples

Project status: LakeLogic is currently beta software. Test contracts and engine-specific behaviour against representative data before production adoption.

The Problem It Solves

Data teams repeatedly rebuild the same controls in notebooks and pipelines:

• schema checks and business rules;
• accepted and quarantined outputs;
• PII handling and lineage metadata;
• incremental processing and materialization;
• deployment checks for contract changes.

LakeLogic puts those expectations in a version-controlled contract and provides the execution machinery around it. Business meaning stays visible in YAML; complex transformations can remain in normal, testable Python or SQL.

Quick Start

Install the base package:

pip install lakelogic

The fastest complete introduction is the Google Colab quickstart. It creates sample data, executes a contract, and shows accepted and quarantined records without requiring a local Spark environment.

A contract starts with the fields and rules that matter to the data product:

version: 1.0.0
dataset: orders

info:
  title: E-Commerce Orders
  owner: [e-mail]
  target_layer: silver

model:
  fields:
    - name: order_id
      type: integer
      required: true
    - name: customer_email
      type: string
      required: true
      pii: true
      masking: partial
    - name: amount
      type: float
      required: true

quality:
  row_rules:
    - name: valid_email
      sql: "customer_email LIKE '%@%.%'"
    - name: positive_amount
      sql: "amount > 0"

Run it through the Python API:

from lakelogic import DataProcessor

processor = DataProcessor("orders_contract.yaml", engine="polars")
result = processor.run_source("orders.csv")

print(f"Accepted: {result.good_count}")
print(f"Quarantined: {result.bad_count}")
print(f"Quality score: {result.quality_score:.1f}")

For a validation run, LakeLogic returns the accepted and quarantined rows with counts and diagnostic context. Materialization, alerts, retention, and catalog behaviour depend on the contract, selected engine, and connected infrastructure.

To check a contract before deployment:

lakelogic validate \
  --contract orders_contract.yaml \
  --gates breaking_change,pii_classification,lineage_break

CI gates analyse contract declarations and the comparison context supplied to them. Add the command to your pull-request workflow to reject a change when a configured gate fails.

Core Capabilities

Capability	What LakeLogic provides
Executable contracts	Strictly parsed YAML for schemas, row rules, dataset rules, service levels, lineage, and materialization.
Quality and quarantine	Accepted and failed records, rule-level diagnostics, run counts, and quality scores.
Contract checks in CI/CD	Static gates for breaking schema changes, PII declarations, and lineage changes when the required comparison context is available.
Multiple execution engines	A common contract model across Polars, DuckDB, and Spark, with documented engine-specific boundaries.
Lakehouse patterns	Incremental processing, Delta or Iceberg outputs, merge strategies, SCD Type 2, dependencies, and external transformation logic.
Operational evidence	Structured run logs, execution metadata, lineage evidence, and optional notification integrations.

See the complete capability matrix before choosing an engine or storage format.

Engine Support

Engine	Best suited to	Installation and boundaries
Polars	Local development, notebooks, CI, and fast single-node processing	Included in the base package. Delta support uses delta-rs.
DuckDB	Local analytical SQL and embedded workflows	Included in the base package. Some catalog and materialization combinations differ from Spark.
Spark	Distributed lakehouse workloads and managed catalogs such as Unity Catalog	Install with pip install "lakelogic[spark]". Managed catalog features depend on the Spark platform and its configuration.

The contract model is shared, but engines are not identical. Review engine and format capabilities for supported combinations.

Where Data Mesh Fits

LakeLogic can provide shared contract machinery for a data mesh while domain teams retain ownership of business meaning.

Data-mesh principle	LakeLogic's role
Domain ownership	Domain teams version contracts alongside the data products they own.
Data as a product	Contracts make schemas, rules, service expectations, and dependencies explicit.
Self-service platform	Teams reuse common validation and execution interfaces across supported engines.
Federated governance	Platform standards can be expressed as shared defaults and checked alongside domain-specific rules.

LakeLogic does not create organisational ownership, access policies, alert delivery, or regulatory compliance by itself. It supplies contract declarations, runtime controls, and evidence that can participate in those wider systems.

Learn by Doing

Guide	Use it to explore
Quickstart	Your first contract, generated data, validation, and quarantine.
Data Quality and Trust	Schema rules, business rules, reconciliation, and medallion flows.
Compliance and Governance	PII-handling and governance patterns that must be combined with organisational controls.
Engine and Scale	Polars, DuckDB, Spark, incremental execution, and dimensional modelling.
Developer Experience	Validation, diagnostics, CI/CD, and development workflows.
Data Generation and AI	Synthetic test data and optional AI-assisted workflows.
Integrations	dbt, dlt, databases, streaming sources, and notifications.

Documentation

• Installation and optional dependencies
• Complete contract reference
• Capabilities and engine boundaries
• Pipeline concepts
• Reconciliation
• Notifications
• Full documentation site

Contributing

Contributions and issue reports are welcome. See CONTRIBUTING.md for the development workflow and developer installation for environment setup.

License

LakeLogic is available under the Apache 2.0 License.