pyspark-testframework 1.0.0

Testframework for PySpark DataFrames

pyspark-testframework

⏳ Work in progress

The goal of the pyspark-testframework is to provide a simple way to create tests for PySpark DataFrames. The test results are returned in DataFrame format as well.

Example

Input DataFrame:

primary_key	email	number
1	[e-mail]	123
2	infowoonstadrotterdam.nl	01
3	@woonstadrotterdam.nl	-45
4	[e-mail]	1.0
5	Null	Null

from testframework.tests import RegexTest, IsIntegerString

# test for valid email addresses
email_regex = r"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$"

mail_tester = RegexTest(
    name="ValidEmail",
    pattern=email_regex
)

test_result_email = mail_tester.test(
    df=df,
    col="email",
    nullable=False
)

# test for integer strings
integer_string_tester = IsIntegerString()

test_result_number = number_tester.test(
    df=df,
    col="number",
    nullable=True
)

test_result_email.show()
test_result_number.show()

Output for ValidEmail:

primary_key	email__ValidEmail
1	True
2	False
3	False
4	True
5	False

Output for IsIntegerString:

primary_key	number__IsIntegerString
1	True
2	False
3	True
4	True
5	True