IceFrame (Alpha)

A DataFrame-like library for working with Apache Iceberg tables using REST catalogs with local execution.

IceFrame provides a simple, intuitive API for creating, reading, updating, and deleting Iceberg tables, as well as performing maintenance operations and exporting data.

IceFrame 0.14.0 requires Python 3.10+, upgrades the optional MCP server to the official Python SDK 2.1 protocol stack, and completes full-package static type checking. It also includes the correctness, UX, packaging, and performance hardening listed in CHANGELOG.md.

Features

• DataFrame API: Familiar interface for working with tables — read_table, to_arrow, to_pandas, lazy, head, describe, count_rows, scan_batches
• Local Execution: Uses PyIceberg, PyArrow, and Polars for efficient local processing
• Catalog Support: REST catalogs (Dremio, Tabular, Polaris, …) with credential vending, plus PyIceberg's sql (SQLite/Postgres), memory, glue, hive, and dynamodb catalogs by pass-through
• CRUD Operations: Create, Read, Update, Delete tables and data
• Native upserts and transactions: ice.upsert(...) on PyIceberg's atomic Table.upsert, and with ice.transaction(tbl) as txn: for multi-operation atomic commits
• Predicate, projection and limit pushdown: the query builder pushes WHERE, SELECT and LIMIT into the Iceberg scan when it is sound to do so, and re-applies anything it cannot push locally so results stay correct
• Metadata tables: ice.inspect(tbl).snapshots() / .files() / .partitions() / .manifests() / .history() / .refs() as Polars frames
• Maintenance: Expire snapshots, remove orphan files (dry-run by default), compact data files (bin-pack, sort, approximate z-order)
• Data quality: constraint validation where nulls fail by default, wired into append_to_table(validators=[...]) as a write gate
• Typed: ships py.typed, so annotations are visible to downstream type checkers
• Structured logging: standard logging throughout the library; no print() to stdout
• Export: Export data to Parquet, CSV, and JSON

Documentation

Getting Started

• Creating Tables
• Reading Tables
• Updating Tables
• Deleting Tables
• CLI Usage
• Dependencies
• Environment Variables

Data Ingestion

• Native File Ingestion (CSV, JSON, Parquet, ORC, Avro)
• Optional File Ingestion (Excel, Delta, Google Sheets)
• Advanced File Ingestion (SQL, XML, SAS/SPSS)
• API Ingestion
• HuggingFace Ingestion
• HTML Ingestion
• Clipboard Ingestion
• Folder Ingestion
• Bulk Ingestion
• Incremental Ingestion

Querying & Processing

• Query Builder API
• SQL Support (DataFusion)
• Lazy Reading
• Distributed Processing (Ray)
• Async Operations
• Notebook Integration
• Scalable Updates

Table Management

• Namespace Management
• Schema Evolution
• Partition Management
• Branching & Tagging
• Views
• Catalog Operations
• Catalog Support Matrix
• Transactions & Upserts
• Metadata Tables

Maintenance & Quality

• Table Maintenance
• Native Maintenance
• Safe Compaction
• Streaming Auto-Compaction
• Data Quality
• Enhanced Data Quality
• Rollback & History

Advanced Features

• Visualization
• Incremental Processing
• Table Statistics
• Scalability Overview
• AI Agent
• MCP Server
• Pydantic Integration

Recipes

• ETL Pipeline
• SCD Type 2
• Data Quality Gate

Installation

IceFrame requires Python 3.10 or newer and supports Python 3.10-3.13.

pip install iceframe

For cloud storage support:

pip install "iceframe[aws]"   # AWS S3
pip install "iceframe[gcs]"   # Google Cloud Storage
pip install "iceframe[azure]" # Azure Data Lake Storage

Install every optional runtime integration with pip install "iceframe[all]". For a smaller environment, prefer only the extras listed in Dependencies.

Quick Start

1. Create a .env file with your catalog credentials (see .env.example):

ICEBERG_CATALOG_URI=https://catalog.dremio.cloud/api/iceberg
ICEBERG_TOKEN=your_token
ICEBERG_WAREHOUSE=your_warehouse
ICEBERG_CATALOG_TYPE=rest

2. Use IceFrame in your code:

from iceframe import IceFrame, col, lit, load_catalog_config_from_env
import polars as pl

# Initialize
config = load_catalog_config_from_env()
ice = IceFrame(config)

# Create an EMPTY table (note: as of 0.12, create_table never writes data
# even if you pass a DataFrame as the schema — use append_to_table afterwards)
schema = {
    "id": "long",
    "name": "string",
    "created_at": "timestamp",
}
ice.create_table("my_table", schema)

# Append data
data = pl.DataFrame({
    "id": [1, 2],
    "name": ["Alice", "Bob"],
    "created_at": [pl.datetime(2024, 1, 1), pl.datetime(2024, 1, 2)],
})
ice.append_to_table("my_table", data)

# Read data
df = ice.read_table("my_table")
print(df)

# Query Builder API — col, lit, IceFrame, QueryBuilder all importable from
# the package root as of 0.12.
from iceframe.functions import sum as ice_sum

df = (ice.query("my_table")
      .select("name", ice_sum(col("id")).alias("total_id"))
      .group_by("name")
      .execute())
print(df)

Filters: two dialects, named explicitly

filter_expr= historically accepted both an IceFrame Expression (pushed to Iceberg) and a string (Polars SQL, evaluated locally) — the same parameter name meaning two languages with opposite performance. As of 0.13.0 both are named:

# Pushed into the Iceberg scan: only matching files are read.
ice.read_table("my_table", filter=col("age") > 30)

# Evaluated locally after the scan, in Polars SQL.
ice.read_table("my_table", filter_sql="age > 30")

filter_expr= still works and dispatches on type, but the explicit parameters are preferred.

Projection and limit pushdown

# `selected_fields=("id",)` and `limit=10` are pushed into the scan, so a
# billion-row table is not materialised just to trim it.
ice.query("events").select("id").limit(10).execute()

Pushdown is skipped when it would change the answer — an ORDER BY, a join, an aggregation, or a predicate that can't be fully pushed all force the limit to be applied after the fact.

Native upsert and transactions

# Atomic, incremental MERGE — only files containing matching rows are rewritten.
ice.upsert("users", new_rows, join_cols=["id"])

# Schema change + append in a single commit.
with ice.transaction("events") as txn:
    txn.set_properties({"owner": "data-eng"})
    txn.append(arrow_table)

Metadata tables

ice.inspect("events").snapshots()   # Polars DataFrame
ice.inspect("events").files()
ice.inspect("events").partitions()

Maintenance

# Compaction scoped by a filter rewrites ONLY matching rows. Before 0.13.0 this
# replaced the entire table with the matching subset.
ice.compact_data_files("events", filter_expr="event_date >= '2026-01-01'")

# Snapshot expiry, on PyIceberg's native maintenance API. Branch/tag heads are
# always protected.
ice.expire_snapshots("events", older_than_days=30, retain_last=5)

# Orphan-file cleanup is DRY-RUN BY DEFAULT: it deletes files permanently.
candidates = ice.remove_orphan_files("events", older_than_days=7)
ice.remove_orphan_files("events", older_than_days=7, dry_run=False)

Data quality (nulls fail by default)

from iceframe.quality import DataValidator

# A null `age` FAILS "age > 0". Before 0.13.0 it passed, because Polars'
# three-valued logic filtered null rows out of the violations frame.
DataValidator().validate(df, ["age > 0"])
DataValidator().validate(df, ["age > 0"], null_policy="pass")   # opt out

# As a write gate:
ice.append_to_table("users", df, validators=["age > 0"])   # raises ValidationError

Error handling

Every deliberate IceFrame error derives from IceFrameError, so infrastructure failures are distinguishable from bad input. For backwards compatibility the input-shaped errors are still ValueError subclasses:

from iceframe import IceFrameError, CatalogError, ValidationError

try:
    ice.append_to_table("users", df, validators=["age > 0"])
except ValidationError:
    ...          # your data was bad
except CatalogError:
    ...          # the catalog was unreachable

Development

pip install -e ".[dev]"
pytest                 # runs offline against a local SQLite catalog
pytest --live          # additionally runs the live REST-catalog tests
ruff check iceframe/ tests/
ruff format --check iceframe/ tests/
mypy iceframe/

The core suite needs no credentials and no network: a session-scoped fixture builds a PyIceberg sql (SQLite) catalog with a file:// warehouse. Before 0.13.0 the read, write, query-builder, schema and stats tests only ran against the author's live Dremio catalog, so 61 of 179 tests skipped everywhere else.

Feature Comparison: IceFrame vs PyIceberg

IceFrame builds on top of PyIceberg, adding high-level abstractions and missing features.

What IceFrame is not

Being straight about the limits:

• Merge-on-read delete writes are not supported. PyIceberg 0.11 has no public delete-file writer, so MoRWriter.write_position_deletes / write_equality_deletes raise UnsupportedOperationError. Deletes are copy-on-write. Reading tables that already contain delete files works fine.
• QueryBuilder.merge with column-level update rules reads the whole target table and overwrites it. Use ice.upsert(...) (native, incremental, atomic) whenever "replace matched rows wholesale" is what you want.
• Joins read each joined table in full. Only the driving table gets pushdown.
• Z-order is an approximation — a hierarchical sort, not a bit-interleaved Z-curve. The returned strategy says so.
• Window functions run locally in Polars, after the scan.
• Joins and column-level merge rules materialize locally. Call query.explain() before running an unfamiliar plan at scale; use native ice.upsert() or a distributed SQL engine for large mutations and joins.

Project policies

• Current roadmap
• Contributing
• Security policy
• Support and releases
• Performance benchmarks
• Apache-2.0 license