iceberg-loader 0.0.6

A convenience wrapper around PyIceberg for simplified data loading into Apache Iceberg tables

iceberg-loader

A convenience wrapper around PyIceberg that simplifies data loading into Apache Iceberg tables. PyArrow-first, handles messy JSON, schema evolution, idempotent replace, upsert, batching, and streaming out of the box.

Status: Actively developed and under testing. PRs are welcome!
Currently tested against Hive Metastore; REST Catalog support is planned.

Why iceberg-loader?

• Messy JSON friendly: auto-serializes dict/list/mixed fields to strings so writes don't fail.
• Schema evolution: add columns on the fly (opt-in), preserves field IDs.
• Safe writes: append/overwrite, idempotent replace via replace_filter, upsert.
• Stream friendly: commit intervals, batches, IPC streams.
• Single config: LoaderConfig sets defaults; override per-call if needed.

Install

pip install "iceberg-loader[all]"

Or with uv:

uv pip install "iceberg-loader[all]"

Quickstart

import pyarrow as pa
from pyiceberg.catalog import load_catalog
from iceberg_loader import LoaderConfig, load_data_to_iceberg, create_arrow_table_from_data

catalog = load_catalog("default")
table_id = ("default", "comparison_complex_json")

data = [
    {"id": 1, "complex_field": {"a": 1, "b": "nested"}},
    {"id": 2, "complex_field": {"a": 2, "b": "another", "c": [1, 2]}},
    {"id": 3, "complex_field": [1, 2, 3]},
]

arrow_table = create_arrow_table_from_data(data)

config = LoaderConfig(write_mode="append", partition_col="signup_date", schema_evolution=True)
load_data_to_iceberg(arrow_table, table_id, catalog, config=config)

Documentation

Full usage guide, API reference, and examples: docs or run mkdocs serve locally. For runnable demos see docs/examples.md; to try locally, cd examples && docker-compose up -d.

Contributing

We welcome contributions! See CONTRIBUTING.md for setup, coding style, and PR guidelines.

hatch run lint
hatch run test

Contributors

Thanks to all contributors who have helped make this project better!

Made with contrib.rocks.

License

iceberg-loader is distributed under the terms of the MIT license.