spark-frame 0.1.0

A library containing various utility functions for playing with PySpark DataFrames

Spark-frame

What is it ?

Spark-frame is a library that brings several utility methods and transformation functions for PySpark DataFrames. These methods were initially part of the karadoc project used at Younited, but they don't rely on karadoc, so it makes more sense to keep them as standalone library.

Several of these methods were my initial inspiration to make the cousin project bigquery-frame, which is why you will find similar methods in transformations and data_diff for both spark_frame and bigquery_frame, except the former runs on PySpark while the latter runs on BigQuery (obviously).

Getting Started

Visit the official Spark-frame website documentation for use cases examples and reference.

Installation

spark-frame is available on PyPi.

pip install spark-frame

Compatibilities and requirements

This library does not depend on any other library. Pyspark must be installed separately to use it. It is compatible with the following versions:

• Python: requires 3.8.1 or higher (tested against Python 3.9, 3.10 and 3.11)
• pyspark: requires 3.3.0 or higher

This library is tested against Mac and Linux.

Release notes

v0.1.0

• Added a new amazing module called spark_frame.nested, which makes manipulation of nested data structure much easier! Make sure to check out the reference and the use-cases.

• Also added a new module called spark_frame.nested_functions, which contains aggregation methods for nested data structures (See Reference).

• New transformations:

  • spark_frame.transformations.transform_all_field_names
  • spark_frame.transformations.transform_all_fields
  • spark_frame.transformations.unnest_field
  • spark_frame.transformations.unnest_all_fields
  • spark_frame.transformations.union_dataframes

v0.0.3

• New transformation: spark_frame.transformations.convert_all_maps_to_arrays.
• New transformation: spark_frame.transformations.sort_all_arrays.
• New transformation: spark_frame.transformations.harmonize_dataframes.