delta-hydro 0.2.1

Advanced Delta-Lake related tooling based on Apache Spark

hydro 💧

hydro is a collection of Python-based Apache Spark and Delta Lake tooling.

See Key Functionality for concrete use cases.

Installation

pip install delta-hydro

Docs 📖

https://christophergrant.github.io/delta-hydro

Key Functionality 🔑

• De-duplicate a Delta Lake table, in-place, without a full overwrite - hydro.delta.deduplicate
• Correctly perform Slowly Changing Dimensions (SCD) (types 1 or 2) on Delta Lake tables - hydro.delta.scd and hydro.delta.bootstrap_scd2
• Issue queries against Delta Log metadata, quickly and efficently getting things like partition sizes on huge tables - hydro.delta.partition_stats
• Other quality of life improvements like hydro.delta.detail_enhanced and hydro.spark.fields

Contributions ✨

Contributions are welcome.

However, please create an issue before starting work on a feature to make sure that it aligns with the future of the project.

Naming 🤓

Originally this project was going to be hydrologist but that's way too long and pretentious, so we shortened to hydro.

A hydrologist is a person who studies water and its movement. Delta Lake, Data Lake, Lakehouse => water.

ChatGPT and LLMs 🤖

Some of this project's code was generated by a Large Language Model(LLM), namely ChatGPT.

We are proud prompt engineers, so we display the prompt that gave us the code in hydro's source (example).

Our take is that the model is very impressive, but not sophisticated enough to be able to write this whole program (yet). A lot of this stuff is very context-dependent and would be difficult to explain to an AI. Plus, ChatGPT isn't aware of newer APIs as it was trained on an older corpus.

We are excited for the future of humanity given recent advancements in artificial intelligence and hope that the technology is used to liberate, rather than accelerate.