pyspark-testframework 0.1.0

Testframework for PySpark DataFrames

pyspark-testframework

⏳ Work in progress

The goal of the pyspark-testframework is to provide a simple way to create tests for PySpark DataFrames. The test results are returned in DataFrame format as well.

Example

Input DataFrame:

primary_key	email
1	[e-mail]
2	infowoonstadrotterdam.nl
3	@woonstadrotterdam.nl
4	[e-mail]
5	Null

from testframework.tests import RegexTest

email_regex = r"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$"

mail_tester = RegexTest(
    name="ValidEmail",
    pattern=email_regex
)

test_result = mail_tester.test(
    df=df,
    col="email",
    nullable=False
)

test_result.show()

primary_key	email__ValidEmail
1	True
2	False
3	False
4	True
5	False