pyspark-test 0.1.0

Check that left and right spark DataFrame are equal.

pyspark-test

Check that left and right spark DataFrame are equal.

This function is intended to compare two spark DataFrames and output any differences. Is is mostly intended for use in unit tests. Additional parameters allow varying the strictness of the equality checks performed.

Installation

pip install pyspark-test

Usage

assert_pyspark_df_equal(left_df, actual_df)

Additional Arguments

• check_dtype : To compare the data types of spark dataframe. Default true
• check_column_names : To compare column names. Default false. Not required of we are checking data types.
• check_columns_in_order : To check the columns should be in order or not. Default to false
• order_by : Column names with which dataframe must be sorted before comparing. Default None.

Example

from pyspark_test import assert_pyspark_df_equal

df_1 = spark_session.createDataFrame(
    data=[
        [datetime.date(2020, 1, 1), 'demo', 1.123, 10],
        [None, None, None, None],
    ],
    schema=StructType(
        [
            StructField('col_a', DateType(), True),
            StructField('col_b', StringType(), True),
            StructField('col_c', DoubleType(), True),
            StructField('col_d', LongType(), True),
        ]
    ),
)

df_2 = spark_session.createDataFrame(
    data=[
        [datetime.date(2020, 1, 1), 'demo', 1.123, 10],
        [None, None, None, None],
    ],
    schema=StructType(
        [
            StructField('col_a', DateType(), True),
            StructField('col_b', StringType(), True),
            StructField('col_c', DoubleType(), True),
            StructField('col_d', LongType(), True),
        ]
    ),
)

assert_pyspark_df_equal(df_1, df_2)