pyspark-testframework 1.0.1

Testframework for PySpark DataFrames

pyspark-testframework

⏳ Work in progress

The goal of the pyspark-testframework is to provide a simple way to create tests for PySpark DataFrames. The test results are returned in DataFrame format as well.

Example

Input DataFrame:

primary_key	email	number
1	[e-mail]	123
2	infowoonstadrotterdam.nl	01
3	@woonstadrotterdam.nl	-45
4	[e-mail]	1.0
5	Null	Null

from testframework.tests import RegexTest, IsIntegerString

# test for valid email addresses
email_regex = r"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$"

mail_tester = RegexTest(
    name="ValidEmail",
    pattern=email_regex
)

test_result_email = mail_tester.test(
    df=df,
    col="email",
    primary_key="primary_key",
    nullable=False
)

# test for integer strings
integer_string_tester = IsIntegerString()

test_result_number = number_tester.test(
    df=df,
    col="number",
    primary_key="primary_key",
    nullable=True
)

test_result_email.show()
test_result_number.show()

Output for ValidEmail:

primary_key	email	email__ValidEmail
1	[e-mail]	True
2	infowoonstadrotterdam.nl	False
3	@woonstadrotterdam.nl	False
4	[e-mail]	True
5	Null	False

Output for IsIntegerString:

primary_key	number	number__IsIntegerString
1	123	True
2	01	False
3	-45	True
4	1.0	True
5	Null	True