spark-frame 0.0.3

A library containing various utility functions for playing with PySpark DataFrames

Spark-frame

What is it ?

Spark-frame is a library that brings several utility methods and transformation functions for PySpark DataFrames. These methods were initially part of the karadoc project used at Younited, but they don't rely on karadoc, so it makes more sense to keep them as standalone library.

Several of these methods were my initial inspiration to make the cousin project bigquery-frame, which is why you will find similar methods in transformations and data_diff for both spark_frame and bigquery_frame, except the former runs on PySpark while the latter runs on BigQuery (obviously).

Installation

spark-frame is available on PyPi.

pip install spark-frame

Release notes

v0.0.3

• New transformation: spark_frame.transformations.convert_all_maps_to_arrays.
• New transformation: spark_frame.transformations.sort_all_arrays.
• New transformation: spark_frame.transformations.harmonize_dataframes.