LakeLogic
Define data trust once. Enforce it from development to production.
LakeLogic is an open-source framework for executable data contracts. Describe a data product's schema, quality rules, PII handling, lineage, and materialization in YAML, then run that contract with Polars, DuckDB, or Spark.
Use the same contract to get fast feedback locally, check changes in CI/CD, and govern pipeline execution in your lakehouse. Records that fail row-level rules can be retained with their failure reasons instead of being silently discarded.
Run the five-minute Colab quickstart · Read the documentation · Browse the examples
Project status: LakeLogic is currently beta software. Test contracts and engine-specific behaviour against representative data before production adoption.
The Problem It Solves
Data teams repeatedly rebuild the same controls in notebooks and pipelines:
- schema checks and business rules;
- accepted and quarantined outputs;
- PII handling and lineage metadata;
- incremental processing and materialization;
- deployment checks for contract changes.
LakeLogic puts those expectations in a version-controlled contract and provides the execution machinery around it. Business meaning stays visible in YAML; complex transformations can remain in normal, testable Python or SQL.
Quick Start
Install the base package:
pip install lakelogic
The fastest complete introduction is the Google Colab quickstart. It creates sample data, executes a contract, and shows accepted and quarantined records without requiring a local Spark environment.
A contract starts with the fields and rules that matter to the data product:
version: 1.0.0
dataset: orders
info:
title: E-Commerce Orders
owner: data-team@company.com
target_layer: silver
model:
fields:
- name: order_id
type: integer
required: true
- name: customer_email
type: string
required: true
pii: true
masking: partial
- name: amount
type: float
required: true
quality:
row_rules:
- name: valid_email
sql: "customer_email LIKE '%@%.%'"
- name: positive_amount
sql: "amount > 0"
Run it through the Python API:
from lakelogic import DataProcessor
processor = DataProcessor("orders_contract.yaml", engine="polars")
result = processor.run_source("orders.csv")
print(f"Accepted: {result.good_count}")
print(f"Quarantined: {result.bad_count}")
print(f"Quality score: {result.quality_score:.1f}")
For a validation run, LakeLogic returns the accepted and quarantined rows with counts and diagnostic context. Materialization, alerts, retention, and catalog behaviour depend on the contract, selected engine, and connected infrastructure.
To check a contract before deployment:
lakelogic validate \
--contract orders_contract.yaml \
--gates breaking_change,pii_classification,lineage_break
CI gates analyse contract declarations and the comparison context supplied to them. Add the command to your pull-request workflow to reject a change when a configured gate fails.
Core Capabilities
| Capability | What LakeLogic provides |
|---|---|
| Executable contracts | Strictly parsed YAML for schemas, row rules, dataset rules, service levels, lineage, and materialization. |
| Quality and quarantine | Accepted and failed records, rule-level diagnostics, run counts, and quality scores. |
| Contract checks in CI/CD | Static gates for breaking schema changes, PII declarations, and lineage changes when the required comparison context is available. |
| Multiple execution engines | A common contract model across Polars, DuckDB, and Spark, with documented engine-specific boundaries. |
| Lakehouse patterns | Incremental processing, Delta or Iceberg outputs, merge strategies, SCD Type 2, dependencies, and external transformation logic. |
| Operational evidence | Structured run logs, execution metadata, lineage evidence, and optional notification integrations. |
See the complete capability matrix before choosing an engine or storage format.
Engine Support
| Engine | Best suited to | Installation and boundaries |
|---|---|---|
| Polars | Local development, notebooks, CI, and fast single-node processing | Included in the base package. Delta support uses delta-rs. |
| DuckDB | Local analytical SQL and embedded workflows | Included in the base package. Some catalog and materialization combinations differ from Spark. |
| Spark | Distributed lakehouse workloads and managed catalogs such as Unity Catalog | Install with pip install "lakelogic[spark]". Managed catalog features depend on the Spark platform and its configuration. |
The contract model is shared, but engines are not identical. Review engine and format capabilities for supported combinations.
Where Data Mesh Fits
LakeLogic can provide shared contract machinery for a data mesh while domain teams retain ownership of business meaning.
| Data-mesh principle | LakeLogic's role |
|---|---|
| Domain ownership | Domain teams version contracts alongside the data products they own. |
| Data as a product | Contracts make schemas, rules, service expectations, and dependencies explicit. |
| Self-service platform | Teams reuse common validation and execution interfaces across supported engines. |
| Federated governance | Platform standards can be expressed as shared defaults and checked alongside domain-specific rules. |
LakeLogic does not create organisational ownership, access policies, alert delivery, or regulatory compliance by itself. It supplies contract declarations, runtime controls, and evidence that can participate in those wider systems.
Learn by Doing
| Guide | Use it to explore |
|---|---|
| Quickstart | Your first contract, generated data, validation, and quarantine. |
| Data Quality and Trust | Schema rules, business rules, reconciliation, and medallion flows. |
| Compliance and Governance | PII-handling and governance patterns that must be combined with organisational controls. |
| Engine and Scale | Polars, DuckDB, Spark, incremental execution, and dimensional modelling. |
| Developer Experience | Validation, diagnostics, CI/CD, and development workflows. |
| Data Generation and AI | Synthetic test data and optional AI-assisted workflows. |
| Integrations | dbt, dlt, databases, streaming sources, and notifications. |
Documentation
- Installation and optional dependencies
- Complete contract reference
- Capabilities and engine boundaries
- Pipeline concepts
- Reconciliation
- Notifications
- Full documentation site
Contributing
Contributions and issue reports are welcome. See CONTRIBUTING.md for the development workflow and developer installation for environment setup.
License
LakeLogic is available under the Apache 2.0 License.
1 maintainer
Release files for lakelogic 1.43.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| lakelogic-1.43.0.tar.gz | 4.5 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| lakelogic-1.43.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 5.3 MB
Release files / lakelogic-1.43.0.tar.gz
| Download URL | lakelogic-1.43.0.tar.gz |
|---|---|
| Size | 4.5 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
438faa2a5e3cfec99a61f0ebfab8798a75db274219dd078952990cc485e1ca37
|
|
BLAKE2b-256 checksum How to use checksums |
e31d7f88dfd221664e4cde5aee1e7b4955e3fcc0159f277213be3d8b5f61051a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.3 {"installer":{"name":"uv","version":"0.12.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / lakelogic-1.43.0-py3-none-any.whl
| Download URL | lakelogic-1.43.0-py3-none-any.whl |
|---|---|
| Size | 753.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
cd4943761751bde138df0aeb8f5f430d462056cd07ce7705ce4210b5e4fc14db
|
|
BLAKE2b-256 checksum How to use checksums |
9c3ef09594945e6d353a92926798a8b9e1de6250a0716b393ff23a6966371ca0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.3 {"installer":{"name":"uv","version":"0.12.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
