delta-audit 0.1.0

A lightweight, Spark-free Python library for inspecting, validating, and diagnosing Delta Lake tables across local and cloud storage.

Delta-audit

A Spark-free Python library for inspecting, validating, and diagnosing Delta Lake tables across local and cloud storage.

delta-audit is a production-oriented, read-only Python library that enables data engineers, platform engineers, and platform teams to inspect, validate, and diagnose Delta Lake tables without requiring Apache Spark.

Instead of launching a Spark cluster just to inspect a Delta table, Delta-audit reads the Delta transaction log directly from storage and provides meaningful diagnostics about table health, metadata, protocol compatibility, checkpoints, and transaction history.

---

Features

• Spark-free Delta Lake inspection
• Read Delta transaction logs (_delta_log)
• Parse Delta checkpoint metadata
• Validate Delta table structure
• Detect missing transaction log versions
• Validate Delta protocol compatibility
• Inspect Delta metadata
• Calculate overall table health score
• Support Local Filesystem
• Support Amazon S3
• Support Azure Data Lake Storage Gen2
• Support Google Cloud Storage
• Lightweight and read-only
• Modular and extensible architecture

---

Why Delta-audit?

Debugging Delta Lake tables typically requires running Apache Spark, even for simple metadata inspection.

Delta-audit removes that dependency by providing a lightweight library capable of reading and validating Delta Lake metadata directly from storage.

It helps answer questions such as:

• Is this a valid Delta table?
• What is the latest Delta version?
• Does the table contain a valid checkpoint?
• Are transaction log versions missing?
• Is the Delta protocol supported?
• Is metadata valid?
• Are referenced Parquet files missing?
• What is the overall health score?

---

Installation

Install from source

git clone https://github.com/SurenderElangovan/delta-lens.git

cd delta-lens

pip install -e .

Install from PyPI

pip install delta-lens

Optional Cloud Storage Support

Amazon S3

pip install "delta-lens[aws]"

Azure Data Lake Storage Gen2

pip install "delta-lens[azure]"

Google Cloud Storage

pip install "delta-lens[gcp]"

Install All Optional Dependencies

pip install "delta-lens[all]"

---

Quick Start

from delta_lens import scan

report = scan("/path/to/delta/table")

print(report.summary())

Example output:

Table Health      : 96%

Latest Version    : 154

Checkpoint Version: 150

Missing Versions  : None

Metadata          : Valid

Protocol          : Reader=3 Writer=7

Warnings          : 0

Errors            : 0

---

Supported Storage

Storage	Supported
Local Filesystem	Yes
Amazon S3	Yes
Azure Data Lake Storage Gen2	Yes
Google Cloud Storage	Yes

---

Project Structure

Delta-audit/
│
├── src/
├── tests/
├── pyproject.toml
└── README.md

The project follows the modern src layout recommended for Python packages.

• src/ – Delta-audit source code
• tests/ – Unit and integration tests
• pyproject.toml – Project metadata, dependencies, and build configuration
• README.md – Project documentation

---

Project Goals

Delta-audit is designed to become a production-ready diagnostics library for Delta Lake.

Primary goals include:

• Inspect Delta Lake tables without Apache Spark
• Detect common Delta Lake issues
• Validate transaction logs and checkpoints
• Analyze metadata and protocol compatibility
• Provide meaningful diagnostics
• Calculate table health scores
• Support multiple cloud storage providers
• Remain lightweight and read-only

---

Roadmap

Version 0.1

• Project foundation
• Modern package structure
• Storage abstraction
• Public API

Version 0.2

• Local filesystem support
• Storage adapter architecture

Version 0.3

• Delta transaction log parser
• Checkpoint parser

Version 0.4

• Metadata validation
• Protocol validation

Version 0.5

• Diagnostics engine
• Health score calculation

Version 1.0

• Production-ready release
• Complete documentation
• Comprehensive testing
• Stable public API

---

Contributing

Contributions are welcome.

If you would like to contribute:

1. Fork the repository.
2. Create a feature branch.
3. Add or update tests.
4. Submit a pull request.

Please ensure that new code includes appropriate documentation and follows the existing project structure.

---

License

Delta-audit is licensed under the MIT License.

See the LICENSE file for additional information.

---

Author

Surender Elangovan

Email: [e-mail]

GitHub: https://github.com/SurenderElangovan

---

Support

If Delta-audit helps you in your work, consider:

• Starring the repository
• Reporting bugs
• Suggesting new features
• Contributing improvements

Community feedback and contributions help improve the project for everyone.