LakeLogic Core

Run data pipelines from YAML contracts.

Define your sources, transformations and quality rules in an Open Lakehouse Contract (OLC). LakeLogic Core runs the supported steps with Polars, DuckDB or Spark, separating accepted rows from failed rows and recording why they failed.

Test locally, check contract changes in CI and run pipelines in your supported environment.

OLC defines the contract. LakeLogic Core executes it. Core is the open-source reference runtime; supported features vary by engine and storage format. See the capability matrix.

Run the Colab quickstart · Documentation · Examples

LakeLogic Architecture

The Problem It Solves

Data teams repeatedly rebuild the same controls in notebooks and pipelines:

schema checks and business rules;
accepted and quarantined outputs;
PII handling and lineage metadata;
incremental processing and materialization;
deployment checks for contract changes.

LakeLogic puts those expectations in a version-controlled contract and provides the execution machinery around it. Business meaning stays visible in YAML; complex transformations can remain in normal, testable Python or SQL.

Quick Start

Install the base package:

pip install lakelogic

The fastest complete introduction is the Google Colab quickstart. It creates sample data, executes a contract, and shows accepted and quarantined records without requiring a local Spark environment.

A contract starts with the fields and rules that matter to the data product:

version: 1.0.0
dataset: orders

info:
  title: E-Commerce Orders
  owner: data-team@company.com
  target_layer: silver

model:
  fields:
    - name: order_id
      type: integer
      required: true
    - name: customer_email
      type: string
      required: true
      pii: true
      masking: partial
    - name: amount
      type: float
      required: true

quality:
  row_rules:
    - name: valid_email
      sql: "customer_email LIKE '%@%.%'"
    - name: positive_amount
      sql: "amount > 0"

Run it through the Python API:

from lakelogic import DataProcessor

processor = DataProcessor("orders_contract.yaml", engine="polars")
result = processor.run_source("orders.csv")

print(f"Accepted: {result.good_count}")
print(f"Quarantined: {result.bad_count}")
print(f"Quality score: {result.quality_score:.1f}")

For a validation run, LakeLogic returns the accepted and quarantined rows with counts and diagnostic context. Materialization, alerts, retention, and catalog behaviour depend on the contract, selected engine, and connected infrastructure.

To check a contract before deployment:

lakelogic validate \
  --contract orders_contract.yaml \
  --gates breaking_change,pii_classification,lineage_break

CI gates analyse contract declarations and the comparison context supplied to them. Add the command to your pull-request workflow to reject a change when a configured gate fails.

Core Capabilities

Capability	What LakeLogic provides
Executable contracts	Strictly parsed YAML for schemas, row rules, dataset rules, service levels, lineage, and materialization.
Quality and quarantine	Accepted and failed records, rule-level diagnostics, run counts, and quality scores.
Contract checks in CI/CD	Static gates for breaking schema changes, PII declarations, and lineage changes when the required comparison context is available.
Multiple execution engines	A common contract model across Polars, DuckDB, and Spark, with documented engine-specific boundaries.
Lakehouse patterns	Incremental processing, Delta or Iceberg outputs, merge strategies, SCD Type 2, dependencies, and external transformation logic.
Operational evidence	Structured run logs, execution metadata, lineage evidence, and optional notification integrations.

See the complete capability matrix before choosing an engine or storage format.

Engine Support

Engine	Best suited to	Installation and boundaries
Polars	Local development, notebooks, CI, and fast single-node processing	Included in the base package. Delta support uses delta-rs.
DuckDB	Local analytical SQL and embedded workflows	Included in the base package. Some catalog and materialization combinations differ from Spark.
Spark	Distributed lakehouse workloads and managed catalogs such as Unity Catalog	Install with `pip install "lakelogic[spark]"`. Managed catalog features depend on the Spark platform and its configuration.

The contract model is shared, but engines are not identical. Review engine and format capabilities for supported combinations.

Where Data Mesh Fits

LakeLogic can provide shared contract machinery for a data mesh while domain teams retain ownership of business meaning.

Data-mesh principle	LakeLogic's role
Domain ownership	Domain teams version contracts alongside the data products they own.
Data as a product	Contracts make schemas, rules, service expectations, and dependencies explicit.
Self-service platform	Teams reuse common validation and execution interfaces across supported engines.
Federated governance	Platform standards can be expressed as shared defaults and checked alongside domain-specific rules.

LakeLogic does not create organisational ownership, access policies, alert delivery, or regulatory compliance by itself. It supplies contract declarations, runtime controls, and evidence that can participate in those wider systems.

Learn by Doing

Guide	Use it to explore
Quickstart	Your first contract, generated data, validation, and quarantine.
Data Quality and Trust	Schema rules, business rules, reconciliation, and medallion flows.
Compliance and Governance	PII-handling and governance patterns that must be combined with organisational controls.
Engine and Scale	Polars, DuckDB, Spark, incremental execution, and dimensional modelling.
Developer Experience	Validation, diagnostics, CI/CD, and development workflows.
Data Generation and AI	Synthetic test data and optional AI-assisted workflows.
Integrations	dbt, dlt, databases, streaming sources, and notifications.

Documentation

Contributing

Contributions and issue reports are welcome. See CONTRIBUTING.md for the development workflow and developer installation for environment setup.

License

LakeLogic is available under the Apache 2.0 License.

ai all api aws-messaging azure azure-messaging azuresql bigquery cli cloud databases delta demo dev dlt dlt-mssql dlt-postgres docs documents duckdb engines enterprise extraction extraction-ocr gcp-messaging integrations kafka local mongodb mysql nlp notebook notifications notify pdf pii polars postgresql profiling review sftp snowflake spark sql sse streaming synthetic websocket

Metadata

Release files for lakelogic 1.84.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for lakelogic 1.84.0
File	Size	Uploaded	Provenance
lakelogic-1.84.0.tar.gz	7.3 MB	Oct 24, 2026	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for lakelogic 1.84.0
File	Interpreter	ABI	Platform	Reset
lakelogic-1.84.0-py3-none-any.whl 1.2 MB Oct 24, 2026 Without provenance	Python 3	none	any	Details

Total release size: 8.5 MB

Release files / lakelogic-1.84.0.tar.gz

Download URL	lakelogic-1.84.0.tar.gz
Size	7.3 MB
Tags	Source
SHA-256 checksum How to use checksums	`78857c4fea8221c1c5b27120bca76e4612d2aa627605cdddcfccbfa0d25ea74a`
BLAKE2b-256 checksum How to use checksums	`9caf638aff4cd72114bd742ec0a88bc4a7f6d8acfe095c3e484e79bb45524cdf`
Upload date	Oct 24, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.13.5 {"installer":{"name":"uv","version":"0.13.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}`

Release files / lakelogic-1.84.0-py3-none-any.whl

Download URL	lakelogic-1.84.0-py3-none-any.whl
Size	1.2 MB
Tags	Python 3
SHA-256 checksum How to use checksums	`82075cd6b0af6b846b776dc35c678670b94fadb116badd54b6da2c2bc848dc81`
BLAKE2b-256 checksum How to use checksums	`ab94352cccebe97dfd8ec94e2de08bb5f2006800e9936cfb742cc3f9fa5edb73`
Upload date	Oct 24, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.13.5 {"installer":{"name":"uv","version":"0.13.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}`

Security for lakelogic 1.84.0

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.