spark-hydro 0.1.1

Advanced Delta-Lake related tooling based on Apache Spark

hydro 💧

hydro is a collection of Python-based Apache Spark and Delta Lake extensions.

See Key Functionality for concrete use cases.

Who are hydro's intended users?

hydro is intended to be used by developers and engineers who interact with Delta Lake tables and Spark DataFrames with Python. It can be used by those of all skill levels.

hydro works on the Databricks platform as well as on other platforms where PySpark and Delta Lake can be installed: laptops for example.

Warning ⚠️

hydro is well tested but not battle hardened, yet. Use it at your own risk.

Installation

pip install spark-hydro

Docs 📖

https://christophergrant.github.io/hydro

Key Functionality 🔑

• Correctly perform Slowly Changing Dimensions (SCD) on Delta Lake tables - hydro.delta.scd and hydro.delta.bootstrap_scd2
• Issue queries against Delta Log metadata, quickly and efficiently retrieving file-level metadata even on Petabyte-scale tables - hydro.delta.file_stats, hydro.delta.partition_stats
• Infer the schema of JSON columns - hydro.spark.infer_json_schema
• Drop nested fields from a Spark DataFrame hydro.spark.drop_fields
• Quality of life improvements like hydro.delta.detail_enhanced and hydro.spark.fields
• And more... check the docs!

Contributions ✨

Contributions are welcome.

Please create an issue and discuss before starting work on a feature to make sure that it aligns with the future of the project.

Naming 🤓

hydro is short for hydrologist, where a hydrologist is a person who studies water and its movement. Delta Lake, Data Lake, Lakehouse => water.

ChatGPT and LLMs 🤖

Some of this project's code and documentation was generated by a Large Language Model (LLM), namely ChatGPT. Some code was even taken verbatim from the AI (with 0 adaptations). Their outputs have been vetted by humans and tested, so don't worry about quality.

Otherwise, we are proud prompt engineers, so we display the prompt that gave us the code in hydro's source (example).