IceFrame (Alpha)
A DataFrame-like library for working with Apache Iceberg tables using REST catalogs with local execution.
IceFrame provides a simple, intuitive API for creating, reading, updating, and deleting Iceberg tables, as well as performing maintenance operations and exporting data.
IceFrame 0.14.0 requires Python 3.10+, upgrades the optional MCP server to the official Python SDK 2.1 protocol stack, and completes full-package static type checking. It also includes the correctness, UX, packaging, and performance hardening listed in
CHANGELOG.md.
Features
- DataFrame API: Familiar interface for working with tables —
read_table,to_arrow,to_pandas,lazy,head,describe,count_rows,scan_batches - Local Execution: Uses PyIceberg, PyArrow, and Polars for efficient local processing
- Catalog Support: REST catalogs (Dremio, Tabular, Polaris, …) with credential vending, plus PyIceberg's
sql(SQLite/Postgres),memory,glue,hive, anddynamodbcatalogs by pass-through - CRUD Operations: Create, Read, Update, Delete tables and data
- Native upserts and transactions:
ice.upsert(...)on PyIceberg's atomicTable.upsert, andwith ice.transaction(tbl) as txn:for multi-operation atomic commits - Predicate, projection and limit pushdown: the query builder pushes
WHERE,SELECTandLIMITinto the Iceberg scan when it is sound to do so, and re-applies anything it cannot push locally so results stay correct - Metadata tables:
ice.inspect(tbl).snapshots() / .files() / .partitions() / .manifests() / .history() / .refs()as Polars frames - Maintenance: Expire snapshots, remove orphan files (dry-run by default), compact data files (bin-pack, sort, approximate z-order)
- Data quality: constraint validation where nulls fail by default, wired into
append_to_table(validators=[...])as a write gate - Typed: ships
py.typed, so annotations are visible to downstream type checkers - Structured logging: standard
loggingthroughout the library; noprint()to stdout - Export: Export data to Parquet, CSV, and JSON
Documentation
Getting Started
- Creating Tables
- Reading Tables
- Updating Tables
- Deleting Tables
- CLI Usage
- Dependencies
- Environment Variables
Data Ingestion
- Native File Ingestion (CSV, JSON, Parquet, ORC, Avro)
- Optional File Ingestion (Excel, Delta, Google Sheets)
- Advanced File Ingestion (SQL, XML, SAS/SPSS)
- API Ingestion
- HuggingFace Ingestion
- HTML Ingestion
- Clipboard Ingestion
- Folder Ingestion
- Bulk Ingestion
- Incremental Ingestion
Querying & Processing
- Query Builder API
- SQL Support (DataFusion)
- Lazy Reading
- Distributed Processing (Ray)
- Async Operations
- Notebook Integration
- Scalable Updates
Table Management
- Namespace Management
- Schema Evolution
- Partition Management
- Branching & Tagging
- Views
- Catalog Operations
- Catalog Support Matrix
- Transactions & Upserts
- Metadata Tables
Maintenance & Quality
- Table Maintenance
- Native Maintenance
- Safe Compaction
- Streaming Auto-Compaction
- Data Quality
- Enhanced Data Quality
- Rollback & History
Advanced Features
- Visualization
- Incremental Processing
- Table Statistics
- Scalability Overview
- AI Agent
- MCP Server
- Pydantic Integration
Recipes
Installation
IceFrame requires Python 3.10 or newer and supports Python 3.10-3.13.
pip install iceframe
For cloud storage support:
pip install "iceframe[aws]" # AWS S3
pip install "iceframe[gcs]" # Google Cloud Storage
pip install "iceframe[azure]" # Azure Data Lake Storage
Install every optional runtime integration with pip install "iceframe[all]".
For a smaller environment, prefer only the extras listed in
Dependencies.
Quick Start
- Create a
.envfile with your catalog credentials (see.env.example):
ICEBERG_CATALOG_URI=https://catalog.dremio.cloud/api/iceberg
ICEBERG_TOKEN=your_token
ICEBERG_WAREHOUSE=your_warehouse
ICEBERG_CATALOG_TYPE=rest
- Use IceFrame in your code:
from iceframe import IceFrame, col, lit, load_catalog_config_from_env
import polars as pl
# Initialize
config = load_catalog_config_from_env()
ice = IceFrame(config)
# Create an EMPTY table (note: as of 0.12, create_table never writes data
# even if you pass a DataFrame as the schema — use append_to_table afterwards)
schema = {
"id": "long",
"name": "string",
"created_at": "timestamp",
}
ice.create_table("my_table", schema)
# Append data
data = pl.DataFrame({
"id": [1, 2],
"name": ["Alice", "Bob"],
"created_at": [pl.datetime(2024, 1, 1), pl.datetime(2024, 1, 2)],
})
ice.append_to_table("my_table", data)
# Read data
df = ice.read_table("my_table")
print(df)
# Query Builder API — col, lit, IceFrame, QueryBuilder all importable from
# the package root as of 0.12.
from iceframe.functions import sum as ice_sum
df = (ice.query("my_table")
.select("name", ice_sum(col("id")).alias("total_id"))
.group_by("name")
.execute())
print(df)
Filters: two dialects, named explicitly
filter_expr= historically accepted both an IceFrame Expression (pushed to
Iceberg) and a string (Polars SQL, evaluated locally) — the same parameter name
meaning two languages with opposite performance. As of 0.13.0 both are named:
# Pushed into the Iceberg scan: only matching files are read.
ice.read_table("my_table", filter=col("age") > 30)
# Evaluated locally after the scan, in Polars SQL.
ice.read_table("my_table", filter_sql="age > 30")
filter_expr= still works and dispatches on type, but the explicit parameters
are preferred.
Projection and limit pushdown
# `selected_fields=("id",)` and `limit=10` are pushed into the scan, so a
# billion-row table is not materialised just to trim it.
ice.query("events").select("id").limit(10).execute()
Pushdown is skipped when it would change the answer — an ORDER BY, a join, an
aggregation, or a predicate that can't be fully pushed all force the limit to be
applied after the fact.
Native upsert and transactions
# Atomic, incremental MERGE — only files containing matching rows are rewritten.
ice.upsert("users", new_rows, join_cols=["id"])
# Schema change + append in a single commit.
with ice.transaction("events") as txn:
txn.set_properties({"owner": "data-eng"})
txn.append(arrow_table)
Metadata tables
ice.inspect("events").snapshots() # Polars DataFrame
ice.inspect("events").files()
ice.inspect("events").partitions()
Maintenance
# Compaction scoped by a filter rewrites ONLY matching rows. Before 0.13.0 this
# replaced the entire table with the matching subset.
ice.compact_data_files("events", filter_expr="event_date >= '2026-01-01'")
# Snapshot expiry, on PyIceberg's native maintenance API. Branch/tag heads are
# always protected.
ice.expire_snapshots("events", older_than_days=30, retain_last=5)
# Orphan-file cleanup is DRY-RUN BY DEFAULT: it deletes files permanently.
candidates = ice.remove_orphan_files("events", older_than_days=7)
ice.remove_orphan_files("events", older_than_days=7, dry_run=False)
Data quality (nulls fail by default)
from iceframe.quality import DataValidator
# A null `age` FAILS "age > 0". Before 0.13.0 it passed, because Polars'
# three-valued logic filtered null rows out of the violations frame.
DataValidator().validate(df, ["age > 0"])
DataValidator().validate(df, ["age > 0"], null_policy="pass") # opt out
# As a write gate:
ice.append_to_table("users", df, validators=["age > 0"]) # raises ValidationError
Error handling
Every deliberate IceFrame error derives from IceFrameError, so infrastructure
failures are distinguishable from bad input. For backwards compatibility the
input-shaped errors are still ValueError subclasses:
from iceframe import IceFrameError, CatalogError, ValidationError
try:
ice.append_to_table("users", df, validators=["age > 0"])
except ValidationError:
... # your data was bad
except CatalogError:
... # the catalog was unreachable
Development
pip install -e ".[dev]"
pytest # runs offline against a local SQLite catalog
pytest --live # additionally runs the live REST-catalog tests
ruff check iceframe/ tests/
ruff format --check iceframe/ tests/
mypy iceframe/
The core suite needs no credentials and no network: a session-scoped
fixture builds a PyIceberg sql (SQLite) catalog with a file:// warehouse.
Before 0.13.0 the read, write, query-builder, schema and stats tests only ran
against the author's live Dremio catalog, so 61 of 179 tests skipped everywhere
else.
Feature Comparison: IceFrame vs PyIceberg
IceFrame builds on top of PyIceberg, adding high-level abstractions and missing features.
| Feature | PyIceberg (Native) | IceFrame (Enhanced) |
|---|---|---|
| Table CRUD | Low-level API | Simplified create_table, drop_table |
| Data Writing | Arrow/Pandas integration | Polars integration, Auto-schema inference |
| Branching | Basic support (WIP) | create_branch, fast_forward, WAP Pattern |
| Compaction | rewrite_data_files (limited) |
bin_pack, sort strategies (Polars-based) |
| Views | Catalog-dependent | Unified ViewManager abstraction |
| Maintenance | expire_snapshots |
GarbageCollector, Native remove_orphan_files |
| SQL Support | None | Fluent Query Builder (select, filter, join) |
| Ingestion | add_files |
add_files wrapper + Incremental Ingestion recipes |
| Rollback | manage_snapshots |
rollback_to_snapshot, rollback_to_timestamp |
| Async | None | AsyncIceFrame with its own bounded thread pool |
| Upsert / MERGE | Table.upsert |
ice.upsert(...) wrapper + QueryBuilder.merge for column-level update rules |
| Transactions | Table.transaction() |
with ice.transaction(tbl) as txn: |
| Metadata tables | Table.inspect.* (Arrow) |
ice.inspect(tbl).* returning Polars frames |
| Data Quality | None | DataValidator constraints, null-failing by default, usable as a write gate |
| Agent surface | None | Read-only MCP server with row/byte caps, plus a multi-provider LLM agent |
What IceFrame is not
Being straight about the limits:
- Merge-on-read delete writes are not supported. PyIceberg 0.11 has no public delete-file writer, so
MoRWriter.write_position_deletes/write_equality_deletesraiseUnsupportedOperationError. Deletes are copy-on-write. Reading tables that already contain delete files works fine. QueryBuilder.mergewith column-level update rules reads the whole target table and overwrites it. Useice.upsert(...)(native, incremental, atomic) whenever "replace matched rows wholesale" is what you want.- Joins read each joined table in full. Only the driving table gets pushdown.
- Z-order is an approximation — a hierarchical sort, not a bit-interleaved Z-curve. The returned
strategysays so. - Window functions run locally in Polars, after the scan.
- Joins and column-level merge rules materialize locally. Call
query.explain()before running an unfamiliar plan at scale; use nativeice.upsert()or a distributed SQL engine for large mutations and joins.
Project policies
Metadata
Release files for iceframe 0.14.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| iceframe-0.14.0.tar.gz | 189.5 kB | Without provenance | Details for iceframe-0.14.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| iceframe-0.14.0-py3-none-any.whl | Python 3 | none | any | Details for iceframe-0.14.0-py3-none-any.whl |
Total release size: 370.3 kB
Release files / iceframe-0.14.0.tar.gz
| Download URL | iceframe-0.14.0.tar.gz |
|---|---|
| Size | 189.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ee7e7a94fff83cd78cc097118f561c8ce2f2de74b7df3a4e5a8662003eeca269
|
|
BLAKE2b-256 checksum How to use checksums |
1d2cb846a08ad2825c036115ea3f5f8b4be5a274edd245af24e60214195bbc48
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.3
|
Release files / iceframe-0.14.0-py3-none-any.whl
| Download URL | iceframe-0.14.0-py3-none-any.whl |
|---|---|
| Size | 180.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e582cea7a0fcd4c2f4f7e85f9a962356cac4f555a516c4f8de0b75ff6c8cd378
|
|
BLAKE2b-256 checksum How to use checksums |
7d598bc41c93c7c94ce5a54a5a1872801088117c0d68530dfcaca4762c71e7b0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.3
|
Security for iceframe 0.14.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.