clade-iceberg-extension 0.1.0

An upstream layout reflection and FinOps utility wrapper for pyiceberg.

CLADE Iceberg Layout Guard Extension

A lightweight, zero-compute utility wrapper built to expose the physical layout properties of Apache Iceberg tables to developers and automated AI agents before a query string is executed.

The Problem It Solves

When a software engineer or an autonomous AI text-to-SQL agent issues a standard database query, they are completely blind to how data is physically sorted across cloud object storage disks. If their SQL WHERE clause accidentally omits a table's physical partition keys, the downstream database cluster is forced to pull petabytes of raw files out of storage to resolve the request. This is called a Blind Scan, and it causes massive unexpected cloud bills and severe resource starvation across development teams.

This wrapper implements the CLADE (Context/Layout-Aware Agentic Data Engineering) philosophy. It intercepts queries upstream at the application layer. It inspects Iceberg's metadata tree directly in local client memory and explicitly tells the caller exactly which data columns must be filtered on to trigger fast, cheap partition pruning.

---

Installation

This package is managed inside a localized monorepo layout. Depending on your system access rules, choose one of the three deployment options below:

Option 1: Local Development (Editable Mode)

If you are developing inside this monorepo workspace, install the library directly to your active virtual environment:

cd packages/clade-iceberg-extension/
python3 -m pip install -e .
Option 2: Enterprise Deployment (Private Git Network)
To deploy this library internally across your team pipelines without publishing your proprietary code to the public internet, engineers can install it directly from your GitHub tracking registry:

Bash
pip install git+[https://github.com/YOUR_GITHUB_USERNAME/clade-workspace.git#subdirectory=packages/clade-iceberg-extension](https://github.com/YOUR_GITHUB_USERNAME/clade-workspace.git#subdirectory=packages/clade-iceberg-extension)
Option 3: Public Open-Source Installation
Once published to the global Python Package Index (PyPI), users worldwide can install the utility via a standard string execution command:

Bash
pip install clade-iceberg-extension
How to Use It in Your Code
Using the utility wrapper requires zero architectural rewrites. You load your standard PyIceberg catalog, load your active table structure, and pass that table instance directly into the IcebergLayoutGuard proxy object.

Here is a complete, production-ready script:

Python
import os
import json
from pyiceberg.catalog.sql import SqlCatalog
# Import the layout guard package asset
from clade_utils import IcebergLayoutGuard

# 1. Connect to your active Iceberg catalog registry
WAREHOUSE_PATH = os.path.abspath("../local_iceberg_warehouse")
SQLITE_DB_PATH = os.path.abspath("../local_catalog.db")
catalog = SqlCatalog("sandbox", **{"uri": f"sqlite:///{SQLITE_DB_PATH}", "warehouse": f"file://{WAREHOUSE_PATH}"})

# 2. Load the target system table
table = catalog.load_table("telemetry.system_logs")

# 3. Encapsulate the table inside the Clade Layout Guard
guarded_table = IcebergLayoutGuard(table)

# 4. Extract the physical layout context map instantly
layout_context = guarded_table.explain_layout()

# 5. Output the structural blueprint properties
print(json.dumps(layout_context, indent=2))
Understanding the Metadata Payload
When you execute .explain_layout(), the wrapper outputs a clean, token-dense JSON mapping payload. This data structure is optimized to be injected directly into an LLM agent's system prompt context or printed to an engineer's log console:

JSON
{
  "table_identifier": "telemetry.system_logs",
  "status": "PARTITIONED",
  "mandatory_filter_targets": [
    "tenant_id",
    "event_timestamp"
  ],
  "structural_blueprint": [
    {
      "storage_directory_key": "tenant_id",
      "query_filter_target": "tenant_id",
      "transform_rule": "identity",
      "is_hidden_illusion": false
    },
    {
      "storage_directory_key": "event_day",
      "query_filter_target": "event_timestamp",
      "transform_rule": "day",
      "is_hidden_illusion": true
    }
  ]
}
Key Blueprint Attributes:
mandatory_filter_targets: A clean, array list of the exact data columns that must be targeted inside an SQL WHERE statement to protect the infrastructure from a high-cost full table scan.

is_hidden_illusion: A safety flag exposing Hidden Partitioning. For example, the physical folder on disk is named event_day, but that attribute does not exist inside the data file schemas. The wrapper flags this as an illusion, warning the caller that they must filter on the original source data column event_timestamp instead for the optimization to succeed.