iceberg-loader 0.1.1

A convenience wrapper around PyIceberg for simplified data loading into Apache Iceberg tables

iceberg-loader

📚 Documentation

A convenience wrapper around PyIceberg that simplifies data loading into Apache Iceberg tables. PyArrow-first, handles messy JSON, schema evolution, idempotent replace, upsert, batching, and streaming out of the box.

Status: Actively developed and under testing. PRs are welcome! Currently tested against Hive Metastore; REST Catalog support is planned.

Why iceberg-loader?

• Messy JSON friendly: auto-serializes dict/list/mixed fields to strings so writes don't fail.
• Schema evolution: add columns on the fly (opt-in), preserves field IDs.
• Safe writes: append/overwrite, idempotent replace via replace_filter, upsert.
• Stream friendly: commit intervals, batches, IPC streams.
• Single config: LoaderConfig sets defaults; override per-call if needed.

Install

pip install "iceberg-loader[all]"

Or with uv:

uv add "iceberg-loader[all]"

Quickstart

from iceberg_loader import LoaderConfig, load_data_to_iceberg
from iceberg_loader.utils.arrow import create_arrow_table_from_data

catalog = load_catalog("default")
table_id = ("default", "comparison_complex_json")

data = [
    {"id": 1, "complex_field": {"a": 1, "b": "nested"}, "signup_date": "2023-01-01"},
    {"id": 2, "complex_field": {"a": 2, "b": "another", "c": [1, 2]}, "signup_date": "2023-01-02"},
    {"id": 3, "complex_field": [1, 2, 3], "signup_date": "2023-01-02"},
]

arrow_table = create_arrow_table_from_data(data)

config = LoaderConfig(write_mode="append", partition_col="day(signup_date)", schema_evolution=True)
load_data_to_iceberg(arrow_table, table_id, catalog, config=config)

Which function to use?

Function	Use when...	Input Format
load_data_to_iceberg	You have a single pa.Table in memory.	pyarrow.Table
load_batches_to_iceberg	You have a generator/iterator of batches (memory efficient).	Iterator of pyarrow.RecordBatch
load_ipc_stream_to_iceberg	You are reading from an Arrow IPC stream file/socket.	File-like object or path

Preparing Data

Use helpers to convert Python dictionaries to Arrow format (handling messy types automatically):

from iceberg_loader.utils.arrow import create_arrow_table_from_data, create_record_batches_from_dicts

# 1. Convert list of dicts -> pa.Table
arrow_table = create_arrow_table_from_data(data_list)

# 2. Convert iterator of dicts -> Iterator[pa.RecordBatch]
batches = create_record_batches_from_dicts(data_generator(), batch_size=10000)

Alternatively, use standard PyArrow conversion: pa.Table.from_pylist(data).

Public API & Stability

• Public surface: LoaderConfig, load_data_to_iceberg, load_batches_to_iceberg, load_ipc_stream_to_iceberg.
• Everything else is internal and may change without notice; always pass options via LoaderConfig.
• Avoid legacy positional arguments—use the config parameter only.
• LoaderConfig validates partition expressions and rejects unsafe combos (e.g., replace_filter with upsert, identity partition on _load_dttm).

How we version

• Semantic Versioning starting at 0.1.x: MINOR for compatible features, PATCH for fixes, MAJOR for breaking API changes.
• Breaking changes only happen on the public surface noted above.
• Prefer partition transforms for timestamps (day(ts), hour(ts)), especially when using load_timestamp.

Release checklist

• Bump version in pyproject.toml and src/iceberg_loader/__about__.py (they must match).
• Update RELEASE.md with highlights and breaking notes.
• Run uv lock --locked and commit uv.lock if it changes.
• Run uv run ruff check ., uv run mypy src/iceberg_loader tests, and uv run python -m pytest.
• Tag and push (git tag -a vX.Y.Z ...), then let CI publish.

Contributing

We welcome contributions! See CONTRIBUTING.md for setup, coding style, and PR guidelines.

hatch run lint
hatch run test

Contributors

Thanks to all contributors who have helped make this project better!

Made with contrib.rocks.

License

iceberg-loader is distributed under the terms of the MIT license.