hydro 💧

hydro is a collection of Python-based Apache Spark and Delta Lake extensions.

See Key Functionality for concrete use cases.

Warning ⚠️

hydro is well tested but not battle hardened, yet. Use it at your own risk.

Installation

pip install delta-hydro

Docs 📖

https://christophergrant.github.io/delta-hydro

Key Functionality 🔑

• De-duplicate a Delta Lake table, in-place, without a full overwrite - hydro.delta.deduplicate
• Correctly perform Slowly Changing Dimensions (SCD) (types 1 or 2) on Delta Lake tables - hydro.delta.scd and hydro.delta.bootstrap_scd2
• Issue queries against Delta Log metadata, quickly and efficently getting things like partition sizes on huge tables - hydro.delta.partition_stats
• Other quality of life improvements like hydro.delta.detail_enhanced and hydro.spark.fields

Contributions ✨

Contributions are welcome.

Please create an issue and discuss before starting work on a feature to make sure that it aligns with the future of the project.

Naming 🤓

hydro is short for hydrologist, where a hydrologist is a person who studies water and its movement. Delta Lake, Data Lake, Lakehouse => water.

ChatGPT and LLMs 🤖

Some of this project's code and documentation was generated by a Large Language Model(LLM), namely ChatGPT.

We are proud prompt engineers, so we display the prompt that gave us the code in hydro's source (example).

APIs

The topic of SQL vs DataFrames is a hot one in the data space.

SQL certainly has its place in analytic and other ad-hoc use cases, but it is missing the expressive power of an imperative language.

This project is a testament to the power of the mix of imperative and declarative expression that DataFrames give. A lot of this code would be very verbose or impossible to express with SQL.