iceguard 1.0.0

Reliability library for Spark-on-AWS-Lambda lakehouse writes with timeout rollback, checkpoints, and orphan cleanup.

IceGuard

Published on PyPI: pypi.org/project/iceguard · Download stats: pepy.tech/project/iceguard (aggregated; PyPI does not expose per-version counts in the API)

Reliability library for Spark-on-AWS-Lambda (SoAL) lakehouse writes. Chunked writes with timeout rollback, S3 checkpoints, orphan cleanup, and optional CloudWatch or OpenTelemetry metrics.

Capability	Out of the box	You provide
Timeout rollback between chunks	Yes	—
Checkpoint resume (S3)	Yes	S3 bucket
S3 path cleanup on rollback	Yes (track_paths)	—
Iceberg / Delta / Hudi adapters	Yes (S3 fallback)	Catalog / commit client for metadata
Orphan scan CLI	Yes	IAM on table path
Blocking df.write.save() in protect() only	No	Use write_dataframe

Install

pip install iceguard

# With optional extras
pip install "iceguard[spark,iceberg,hudi,otel]==1.0.0"

# From source (specific tag)
pip install "git+https://github.com/vaquarkhan/IceGuard.git@v1.0.0"

Extras: [spark], [iceberg], [hudi], [otel], [dev]

Quick start

import iceguard

with iceguard.protect(context, s3_bucket="my-checkpoints") as writer:
    writer.write(
        path="s3://lake/db/table",
        total_records=10_000,
        batch_writer=lambda s, e: write_chunk(s, e),
        track_paths=lambda s, e: new_paths(s, e),
    )

Spark: iceguard.write_dataframe(writer, df, path, write_format="iceberg")

CLI: iceguard orphans scan s3://lake/db/table --json

Repository layout

Path	Purpose
docs/	Full documentation
examples/	Python, SAM, CDK samples
terraform/	Modular production IaC
infra/cloudwatch/	Dashboard JSON
src/iceguard/	Library source

Development

pip install -e ".[dev]"
pytest tests --cov=iceguard
python validation/run_all.py

Release status

See docs/STATUS.md for capability checklist. v1.0.0 is published via GitHub Actions trusted publishing (OIDC) on release tags.

Documentation

• API reference (complete)
• Installation
• Architecture
• Terraform
• Formal verification
• Publishing

License

MIT — see LICENSE.